跳到正文
DEV Community· Femi Ogunsanya·· 7 天前AI 評分59

獎勵模型訓練前,先用捷徑判別器檢查偏好配對

Shortcut judges: a cheap test your preference pairs should pass before you train on them

AI 導讀

作者建議在用偏好配對訓練獎勵模型前,先以答案長度、與提示詞的 TF-IDF 相似度及明顯特徵等簡單判別器測試配對;若準確率遠高於 50%,便先修正資料。另一位建構者修整嚴重程度配對後,在 84 組新配對中,長度和 TF-IDF 判別器均為 50%,關鍵詞判別器為 55%;作者自己的 300 組天氣驗證配對中,TF-IDF 判別器達 78%。

正文

Femi Ogunsanya

I'm building evaluation adapters (small reward models) for Telegraph, a network that routes AI requests to miners and scores their answers. Disclosure: I'm competing in Telegraph Hackathon Season II, which has prizes.

A preference pair is a prompt, a better answer and a worse one. A reward model trained on pairs learns whatever separates the two answers. If something cheaper than the skill you want separates them, it learns that instead, and your accuracy number measures the shortcut.

The test

Before training, run rules that pick an answer without checking anything:

  • pick the longer answer
  • pick the answer with more words in common with the prompt (TF-IDF cosine)
  • pick the answer with more of an obvious feature (alarm words, a verdict word)

If any of them is far above 50%, fix the pairs first.

Two cases from this week

  1. Another builder (AdversarialEval, in the Telegraph Discord) found a keyword-count judge got every one of their severity pairs right, because all the worse answers were downgrades. After adding inflations, the TF-IDF judge still won ~95%. They rebuilt the pairs so the two answers are identical except for the severity claim, drew wrong severities from the same distribution as true ones (either direction), and balanced which answer is longer. On 84 new pairs: length 50%, TF-IDF 50%, keywords 55%.

  2. My own 300 weather-verification pairs: length was 49.7% overall but 17–88% within error families, because the verdict phrases differ in length and some families lean one way. TF-IDF was 78% overall. Part of that is legitimate (in a wrong-station pair, not matching the question is the error), but my digit-swap and fill-value pairs leak too. Those rows are already in the pool, so I'm reporting this as a limit rather than quietly fixing it.

What I'd do differently

  • write the worse answer by copying the better one and changing only the claim
  • balance verdict direction and length within every error family, not just overall
  • report the shortcut judges next to model accuracy in every run

Script (standard library only) and the per-family table: https://github.com/fexx301/oathcast-telegraph-s2/tree/main/shortcut-judges-v1

dev.to allows up to four tags. I'd use machinelearning, ai, llm, datascience.

來源:DEV Community · dev.to