跳到正文
InfoQ· Leela Kumili·· 11 天前AI 評分44

JioHotstar 對 AI 編碼智能體知識庫方案進行基準測試

JioHotstar Benchmarks Knowledge Bases for AI Coding Agents

AI 導讀

JioHotstar 評測六種協助 AI 編碼智能體瀏覽大型程式碼庫的方案,發現結構化程式碼搜尋適合明確工程任務,語義檢索則較能處理含糊需求。測試採用 SWE-bench 和 CodeScaleBench 子集,並以 Claude Sonnet 作為解題模型;加入檢索層未能持續改善任務完成情況或減少 token 消耗。團隊建議保留檔案系統工具作為基準,按任務特性選用檢索方式。

正文

JioHotstar evaluated six approaches to helping AI coding agents navigate large codebases, finding that structural code search performed well on clearly defined engineering tasks, while semantic retrieval helped more with ambiguous requirements. The evaluation also found that adding retrieval layers did not consistently improve task completion or reduce token consumption.

Abhishek Bansal, Senior Staff Engineer at JioHotStar, compared filesystem tools, Sourcebot, Graphify, CocoIndex Code, Repowise, and an in-house retrieval-augmented generation (RAG) pipeline. The team evaluated the approaches using subsets of SWE-bench and CodeScaleBench, alongside a custom question-answering harness for JioHotstar's codebase. The experiments used Claude Sonnet as the solving model and measured task accuracy, token consumption, agent turns, and tool usability.

JioHotstar’s benchmark setup for evaluating code retrieval approaches for AI coding agents. (Source: JioHotstar Blog Post)

The filesystem baseline, which relied on locally checked out repositories and standard search and file operations, remained competitive on concrete tasks. Sourcebot, which provides indexed text search, symbol definitions, references, and file reads, performed better than the baseline on harder SWE-bench Verified tasks while also costing less. On simpler SWE-bench Lite tasks, retrieval generally added overhead without consistently improving results.

CodeScaleBench showed a different pattern. Semantic retrieval helped more when agents had to interpret ambiguous problem descriptions and map them to implementation details that were not obvious from exact text or symbol searches. The findings suggest that retrieval approaches should be selected according to task characteristics rather than treated as interchangeable solutions.

The report also highlighted the importance of tool design. Agents frequently preferred familiar developer operations such as searching text, reading files, and locating symbols over higher-level abstractions that required additional verification.

Tool call breakdown by category across coding agent knowledge base approaches. (Source: JioHotstar Blog Post)

Siddharth Vaithinathan raised a broader architectural question in a LinkedIn post discussing the study:

Whether coding agents should understand not only the code but also the company that produced it. He argued that repository indexing alone cannot capture why architectural decisions were made, which constraints shaped them, or what earlier incidents taught the engineering team.

Vaithinathan suggested extending agent knowledge beyond code and documentation to include architecture, constraints, decisions, rationale, incidents, and outcomes. Such context could help agents evaluate existing design choices before proposing changes, particularly in systems where historical operational experience influences architectural decisions.

Tarun Bhargava raises a relevant question for benchmark interpretation, particularly when evaluations use limited subsets and probabilistic models.

Cost/accuracy numbers with multiple runs of the same setup.

JioHotstar's evaluation did not identify a single retrieval approach that performed best across all tasks. Instead, JioHotstar recommends retaining filesystem tools as a baseline, using structural search for concrete, code-focused problems and semantic indexing for ambiguous or cross-cutting requirements. JioHotStar engineering team also emphasizes measuring tool-use traces alongside task accuracy to determine whether retrieval reduces unnecessary exploration. The findings highlight the importance of matching retrieval capabilities to the programming task and how the agent navigates the codebase.

About the Author

Leela Kumili

Show moreShow less

來源:InfoQ · infoq.com