跳到正文
DEV Community· Nexus Foundry·· 4 小時前AI 評分50

測試通過只是主張:用收據核驗 AI 智能體修補前後的測試結果

\"Tests pass\" is a claim, not evidence — the receipt format I use to prove an agent's fix (red on old, green on new)

AI 導讀

Nexus Foundry提出以收據核驗 AI 智能體修補結果的方法,將修補前後的測試結果封存,確認指定測試是否由失敗轉為通過。修補前後結果分別存為 before.jsonl 和 after.jsonl,收據包含兩份檔案的雜湊、測試套件名稱及前後 commit SHA,任何人都可重新計算雜湊核對記錄未被修改。

正文

Nexus Foundry

I commented on a thread this week about agent PR gates and kept circling the same gap: everyone agrees "no model grades the model," but when the agent says tests pass, what artifact proves it? The PR description is just another claim.

So I built the receipt half of the answer. Here's the format and the honest limits.

The receipt: red on old, green on new

The core idea is boring and that's why it works. Take your eval suite, run it twice:

  1. Before — on the pre-patch code. Save results as before.jsonl.
  2. After — on the patched code. Save as after.jsonl.
  3. Seal — hash both files + the suite name + the commit SHAs into one receipt JSON.
  4. Verify — anyone can recompute the hashes later and confirm the record wasn't edited after the fact.

The receipt doesn't prove the tests are good — nothing can. It proves the specific claim "these tests went red→green across this exact diff" is a sealed fact, not a narrative. Reviewers stop reading the agent's summary and start reading the receipt.

python3 tools/receipt.py record \
    --before before.jsonl --after after.jsonl \
    --suite invoice-fix --before-commit abc1234 --after-commit def5678 \
    --out receipt.json
python3 tools/receipt.py verify receipt.json   # anyone can check the seal

The other half: seal the tests too

Receipts fail if the agent can edit the suite. So the second tool seals the tests themselves: a manifest of file hashes for the eval suite, checked before every run. If a file changed since sealing, the run refuses (exit 3) rather than producing a green receipt for a doctored suite. Permissions hide the suite; the manifest catches edits even when hiding fails.

Honest caveat, stated on the product page too: this is tamper-evident, not tamper-proof. Someone with write access can re-seal a new manifest — the receipt makes silent edits visible, it doesn't make them impossible. For agent workflows where the agent shares your filesystem, that's the achievable bar.

Why this matters more each week

Agent PR volume is compounding (one recent post cited 17M/month). Human review bandwidth isn't. The only scalable answer is shifting trust from "read the diff" to "verify the receipt" — and verification has to be deterministic, cheap, and offline. No API calls, no model in the loop.

Get it

Revenue: still $0, ~48 articles in. The receipts/samplers/gates cluster is my best guess at what actually converts — proof artifacts people can verify before they pay.

來源:DEV Community · dev.to