CRM-Agent-Bench:企業有狀態環境中工具使用智能體的基準評測
CRM-Agent-Bench: Benchmarking Tool-Using Agents in Stateful Enterprise Environments
CRM-Agent-Bench 評估智能體在企業多輪業務任務中的執行能力,並以嚴格的合約式標準檢查任務結果、執行過程、可行性及合規性。基準涵蓋 9 個業務範疇的 555 項任務及 36 個能力標籤;16 個系統的嚴格成功率介乎 64.32% 至 83.60%。領先系統的成本及各範疇可靠性仍有明顯差異。
Published on Oct 23
Authors:
,
Abstract
The study introduces an execution-grounded benchmark for evaluating deployable language agents on multi-turn business tasks using strict contract-based success criteria across outcome, trajectory, viability, and compliance.
Deploying language agents in real applications demands more than the right answer: an agent must execute a multi-step business process correctly, follow domain-specific rules, and interact with a user across a dialogue, where the gap between saying and doing is the gap between a satisfied customer and a real loss. We argue that a deployable-agent benchmark should jointly assess four facets: the outcome an agent reaches, the trajectory it takes, its operational viability, and its policy compliance. Customer relationship management (CRM), where mutable records meet operational rules and consequential effects, exercises all four. We introduce CRM-Agent-Bench, an execution-grounded benchmark of 555 stateful, multi-turn tasks across nine business domains and 36 capability tags. Each task supplies a task-specific executable contract drawn from checks over actions, arguments, order, response, target state, and protected state; success is the strict conjunction of the checks declared for that task. We evaluate the complete agent system through these contracts and report scored coverage, latency, and cost. Across sixteen systems evaluated on the audited final release, strict success spans 64.32% to 83.60%; the closely grouped leading systems nonetheless diverge sharply in cost and per-domain reliability. CRM-Agent-Bench offers an auditable framework for measuring contract-grounded business-transaction integrity and adherence to task-declared benchmark policies, with a portable structure designed for extension as models improve. Github repository: https://github.com/salesforce-misc/crm-agent-bench
View arXiv page View PDF Project page GitHub Add to collection
Get this paper in your agent:
hf papers read 2610.32004
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash
Models citing this paper 0
No model linking this paper
Cite arxiv.org/abs/2610.32004 in a model README.md to link it from this page.
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2610.32004 in a dataset README.md to link it from this page.
Spaces citing this paper 0
No Space linking this paper
Cite arxiv.org/abs/2610.32004 in a Space README.md to link it from this page.
Collections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.
來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co