跳到正文
HuggingFace Daily Papers(社區熱門論文)·· 27 天前AI 評分40

TAO:面向 LLM 智能體服務的工具感知聯合排程

TAO: Tool-Aware Joint Scheduling for LLM-Based Agentic Serving

AI 導讀

TAO 是一套面向 LLM 智能體服務的工具感知聯合排程框架,透過動態規劃,在容量限制下共同選擇及配置等候中的工作階段。其價值介面納入工具即將就緒的狀態、隨時間衰減的節點快取權重及 token 用量。在 12 組模型、工作負載及並發設定中,與各項指標的最佳受評基準相比,TAO 的延遲最多降低 26.8%、平均首 token 時間最多降低 36.3%,解碼吞吐量最多提升 29.5%。

正文

Published on Oct 27

Authors:

,

,

,

Abstract

TAO improves scheduling for LLM agent services by considering tool readiness and using dynamic programming to place sessions, reducing latency and increasing throughput.

Agentic services alternate between model inference on clustered GPUs and external tools on distributed CPUs. In this scenario, existing systems (1) overlook tool-return readiness in service evaluation, wasting limited resources, and (2) use heuristic strategies to decide admission and placement on GPUs, resulting in suboptimal scheduling decisions. To bridge this gap, we present TAO, a tool-aware joint scheduling framework for LLM-based agentic serving. Its value interface combines near-term tool readiness, node-specific cache credit with time-aware decay, and token footprint; a dynamic program jointly selects and places waiting sessions under capacity constraints. Across the 12 reported model-workload-concurrency settings, TAO reduces latency by up to 26.8%, reduces mean time to first token (TTFT) by up to 36.3%, and increases decode throughput by up to 29.5% relative to the best evaluated state-of-the-art baseline for each metric.

View arXiv page View PDF Add to collection

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.37230 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.37230 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2610.37230 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.

來源:HuggingFace Daily Papers(社區熱門論文) · huggingface.co