跳到正文
DEV Community· diogo·· 9 天前AI 評分60

Haystack 讓 AI Agent 的 Markdown 筆記庫可檢索,並把修正轉為回歸測試

I made my AI agent's notes vault retrievable: 35 MB embedder, llama.cpp reranker, corrections as regression tests

AI 導讀

作者開源 Haystack,讓 AI Agent 透過語義路由檢索 Markdown 筆記庫,並把每次檢索修正轉成永久規則及 QA 配對。系統以嵌入向量和語料 IDF 詞項統計混合檢索,經 RRF 融合及 llama.cpp cross-encoder 重排;審核會重建索引並執行 QA 配對,遇到回歸便以狀態碼 1 結束。

正文

diogo

I made my AI agent's notes vault retrievable: 35 MB embedder, llama.cpp reranker, corrections as regression tests

Ok, I made a thing! LLMs have their... quirks. Namely re-reading the same notes, wandering back and forth, and spending 10x more tokens than needed because I can't be explicit.

So I built Haystack — a semantic router for a markdown vault. Instead of the agent digging through files, it queries: hybrid retrieval (embeddings + corpus-IDF term stats, RRF-fused, cross-encoder reranking via llama.cpp), all local, zero cloud, one 35 MB model, ~10 ms per query. MIT, one-command setup: github.com/diogocccarvalho/haystack.

The part I don't see elsewhere

Your corrections become regression tests. Every retrieval miss you fix becomes a permanent rule AND a QA pair. The audit reindexes and runs every pair — it exits 1 on a regression, so the same question can't silently fail twice.

And the learned rules only pin when they actually beat retrieval: a stored phrasing must outrank the corpus' own best hit for the same query. Scale-free, not absolute-threshold — so swapping embedders doesn't break your corrections. I A/B'd models against frozen QA sets and rejected two (a reranker that was 2x faster but -16pp Recall@3; a 97M embedder at -7pp). Numbers in docs/RESEARCH.md.

The numbers, honestly

Corpus Recall@3 MRR
English vault (68 pairs, nickname-style) 1.000 0.917
Portuguese vault (14 pairs, synthetic) 1.000 1.000
Foreign corpus (llama.cpp docs, 33 pairs) 1.000 0.970

The foreign-corpus row matters more than the home-vault row: the chunker generalizes beyond the corpus it was tuned on. Also: llama-server does both the embeddings (bge-small-en GGUF) and the reranking (--reranking flag on ms-marco MiniLM-L12) — one binary, tested with llama.cpp v1.0.0.

The honest part

The agent that uses Haystack daily also built most of it with me. Every gate in the repo was run by the agent against its own index — that's how the setup-script bugs got caught. And docs/SHORTCOMINGS.md is the failure catalog: ten real failures (a reranker overriding user corrections live; a rebuild that once deleted every correction; an over-context chunk killing a whole batch), each one now a design rule.

Try it

./setup.sh --start          # model + config + embeddings server, one command
python3 haystack.py reindex
python3 haystack.py query "that thing about the invoices"

Any OpenAI-compatible endpoint works instead (Ollama, LM Studio, remote API). Python 3.11+, stdlib + numpy only — no framework, no vector DB daemon, no cloud.

The open question I'd genuinely love answers to: this was tuned entirely on my vault. I want to know how the default config holds up on other people's notes. If you run it, tell me what breaks.

來源:DEV Community · dev.to