如何用 Truvec 基準測試及優化 RAG 檢索
RAG retrieval benchmarking and optimization
Abdesslem Amri 示範使用 Truvec 為 RAG 檢索建立固定問題集、比較模型及策略,並逐項優化設定。示例中兩款模型的 hit rate 同為 90%,MRR 分別為 0.900 和 0.850;較佳者每 1M tokens 收費 $0.020,另一款為 $0.100。優化時應每次只改一項,檢視相對基準修復或破壞的問題,並可在文件變更或切換模型前透過 API 執行測試。
Your RAG is only as good as its retrieval. Here's how to benchmark it.
A retrieval miss doesn't look like an error. When the right passage isn't retrieved, the LLM still answers from the wrong data. Nothing in your logs points at retrieval.
In this article I will show you how to use Truvec to debug and optimize your retrieval.
The 5 knobs
Everything before the LLM changes which chunks come back:
- Chunking
- Embedding model
- Dense or hybrid search (BM25 + vectors)
- Reranker
- Top k
Your contracts, support articles and code have their own vocabulary. You can not rely on other people setup to build your own retrieval system.
1. Get a golden dataset
Questions about your docs, each tied to the chunk that answers it.
UI: Data Upload → Golden Set → Create Dataset → Generate Dataset. Review the questions, fix the bad ones, then Commit Dataset. Committed means locked: every run uses the same questions.
API:
API=https://app.truvec.dev
AUTH="Authorization: Bearer $TOKEN" # from POST /api/auth/login
curl -s -X POST "$API/api/v1/documents/upload?project_id=$PROJECT" -H "$AUTH" \
-F "file=@help-center.pdf" -F "chunking_strategy=recursive" \
-F "chunk_size=1000" -F "chunk_overlap=200"
DATASET=$(curl -s -X POST $API/api/v1/golden-datasets -H "$AUTH" \
-H "Content-Type: application/json" \
-d "{\"project_id\": $PROJECT, \"name\": \"v1\", \"number_of_questions\": 20}" | jq .id)
curl -s -X POST $API/api/v1/golden-datasets/$DATASET/commit -H "$AUTH"
2. Benchmark the models
UI: Comparison → Compare models → pick the dataset, tick the models, Run Comparison.
Look at that table. Same hit rate: 90% for both. MRR splits them: 0.900 vs 0.850. And the winner is the one priced at $0.020 per 1M tokens, not $0.100.
Hit rate says the right chunk is in the top k. MRR says how high. If you pass 3 chunks to the LLM, that's the one to watch.
API:
curl -s -X POST $API/api/v1/benchmarks -H "$AUTH" \
-H "Content-Type: application/json" \
-d "{\"project_id\": $PROJECT, \"golden_dataset_id\": $DATASET,
\"name\": \"small vs large\", \"model_ids\": [1, 2], \"top_k\": 5}"
3. Optimize: one change at a time
Keep one baseline. Add hybrid search in one configuration, a reranker in another. Same questions.
UI: Comparison → Compare strategies. The Hybrid search and Reranker buttons copy the baseline and change one thing.
API: the first variant is the baseline.
curl -s -X POST $API/api/v1/benchmarks -H "$AUTH" \
-H "Content-Type: application/json" -d "{
\"project_id\": $PROJECT, \"golden_dataset_id\": $DATASET,
\"name\": \"Does a reranker help?\", \"top_k\": 5,
\"variants\": [
{\"model_id\": 1},
{\"model_id\": 1, \"pipeline\": {\"retrieval\": \"hybrid\"}},
{\"model_id\": 1, \"pipeline\": {\"reranker\": {\"provider\": \"cohere\", \"model\": \"rerank-v3.5\"}}}
]
}"
Then read each one against the baseline:
curl -s $API/api/v1/benchmarks/$BENCHMARK -H "$AUTH" \
| jq '.results.comparison[] | {config: .label, mrr,
mrr_delta: .vs_baseline.mrr_delta,
fixed: .vs_baseline.fixed, broken: .vs_baseline.broken}'
fixed: questions the baseline missed and this setup found.
broken: the opposite.
Fixing 4 and breaking 3 is not the same as fixing 1 and breaking none, even if the averages match.
What to expect
- Hybrid shines on product names, error codes and SKUs. On plain prose it can add nothing, or hurt.
- A reranker can make a cheap model beat an expensive one. It can also cost more per query than the embedding.
- Read the questions a change broke before you ship it.
Put it in CI
It's all API calls, so run it when your documents change or before you switch models. Retrieval first, generation after.
Free plan, no card, no provider keys to set up. About 15 minutes to a first leaderboard on your own docs.
👉 https://truvec.dev
API docs: https://docs.truvec.dev
來源:DEV Community · dev.to




