跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 26 天前AI 評分35

Jev 跨查詢原語一致性評估與改進

Cross-Primitive Consistency in Jev

AI 導讀

Jev 的跨查詢原語一致性在四軸基準 PrimaCons 上接受評估,三種 Jev 規模均出現四個軸向的一致性違規;跨原語差異預測決策失誤的 ROC-AUC 為 0.91。研究提出 Cross-Primitive Residual Calibrator(CPRC),在留一領域外測試中將不一致性降低 58.57%,並將準確率提升 +2.97%。

正文

Published on Oct 28

·

Submitted by

Xu

on Nov 3

Authors:

,

,

Abstract

Jev's cross-primitive consistency is benchmarked and improved via a residual calibrator that reduces inconsistency and boosts decision accuracy.

Jev enables efficient decision-making with large language models (LLMs) through typed querying primitives, allowing the same semantic state to be expressed in different formats, such as binary and multiple-choice queries. However, the consistency of decisions across primitives has not been systematically evaluated. This work introduces PrimaCons, a four-axis benchmark for assessing cross-primitive consistency while preserving the semantics of the query. Evaluations across three Jev sizes reveal consistency violations across all four axes. Our analysis shows that these violations follow systematic patterns and that cross-primitive discrepancies predict decision failure with a ROC-AUC of 0.91. To mitigate these inconsistencies, we propose the Cross-Primitive Residual Calibrator (CPRC), which attaches a lightweight residual network to a frozen backbone and jointly optimizes predictive accuracy and cross-primitive consistency. Under a leave-one-domain-out protocol, CPRC reduces cross-primitive inconsistency by 58.57% and improves accuracy by +2.97%., demonstrating its effectiveness in both in-domain and out-of-domain settings. Our findings establish cross-primitive consistency as a key dimension for evaluating Jev and provide a practical approach to improving the reliability of its decisions. Code and benchmark is available at https://github.com/67L1/PrimaCons-CRPR.

View arXiv page View PDF GitHub Add to collection

Get this paper in your agent:

hf papers read 2610.38623

Don't have the latest CLI?

curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.38623 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.38623 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.38623 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co