跳到正文
DEV Community· Muhammad Arshad·· 4 天前AI 評分65

智能體失控循環事故應變手冊

Runaway agent loop: incident runbook

AI 導讀

這份事故手冊為失控智能體循環提供止損、事件重建、永久防護及結案流程。文中列出的公開案例中,一名客戶的智能體在凌晨 1:12 至 6:50 間重試失敗工具 31,000 次,產生 $4,700 的模型 token 費用,當時方案月費為 $400。永久控制包括每次執行及每個租戶的預算上限、按錯誤類別設定最多 3 次重試,以及相同工具與正規化參數重複 3 次時停止並升級處理。

正文

Muhammad Arshad

Runaway agent loop: incident runbook

RB-01 · v0.1 · Muhammad Arshad · CC BY 4.0

When to use this

  • Token spend for one run, customer or agent is far above its normal rate.
  • The same tool is called again and again with near-identical arguments.
  • The agent keeps producing output but makes no progress toward its goal.

Public example: a 3-person B2B team reported that a customer's agent retried a failing tool 31,000 times between 1:12am and 6:50am. It spent $4,700 in tokens on a $400/month plan. They had a global rate limit and total-spend alerts, but nothing per customer. (r/SaaS, October 2026)

Source: https://www.reddit.com/r/SaaS/comments/1xh0cq5/an_agent_got_stuck_in_a_loop_and_burned_4700_of_a/

Severity

  • SEV-1: spend is still rising, or the loop touched customer data or side-effecting tools.
  • SEV-2: contained, but the cost is unbilled or someone has to absorb it.
  • SEV-3: caught by existing caps; no customer impact.

1. Contain (first 15 minutes)

  1. Stop the run: kill switch, pause the queue, or disable that tenant's agent key.
  2. Freeze, don't delete: keep traces, prompts, tool responses and billing exports.
  3. Add a temporary hard cap per tenant in front of the model call.
  4. Write down every containment action with the time and who did it. Mark each one TEMPORARY.

2. Reconstruct: 11 questions

Label every answer VERIFIED (from logs or billing), PROPOSED (a hypothesis) or UNKNOWN. Unknowns stay unknown until evidence arrives.

# Step Question
1 Deviation event What exactly happened, and between which timestamps?
2 Root driver What made the agent keep going? (error text, retry policy, missing budget)
3 Discovery How was it found, by whom, and how long after it started?
4 Observation What did the traces and the bill show?
5 Escalation Who was told, and how fast?
6 Tacit knowledge What did the on-call person know that isn't written down anywhere?
7 Alternatives and trade-offs Which other fixes were considered, and why were they rejected?
8 Authorization Who approved the containment and the fix?
9 Loss and impact Tokens, money, customer trust, side effects.
10 Containment What stopped it? Is that change temporary or permanent?
11 Prevention Which control would have stopped it at step 3 instead of step 31,000?

3. Usual causes

  • Retries on errors that can never succeed (validation, auth, not found).
  • Tool errors returned as plain text that the model reads as "try a slightly different prompt".
  • No per-run budget for steps, tokens or wall-clock time.
  • Spend checked only in total, not per tenant or per run.
  • Loop detection that only catches exact repeats, while the agent varies its prompt slightly.
  • A tool timeout that triggers a re-call while the first call is still running, duplicating side effects.

4. Permanent controls

  • Per-run budget: max steps, max tokens and max wall-clock time, checked before every model call.
  • Per-tenant budget: a soft alert at 1x plan and a hard stop at 2x (the fix the team in the example adopted).
  • Retry policy by error class: retry only 429s, 5xx and timeouts, with backoff, at most 3 times.
  • No-progress detector: the same tool with the same normalised arguments 3 times stops the run and escalates.
  • Tool errors returned as structured observations with retryable: false.
  • Idempotency keys on every side-effecting tool.
class Stop(Exception): ...
class Escalate(Exception): ...

def guard(run, tool, args, err=None):
    if run.tokens > run.budget.tokens or run.steps > run.budget.steps:
        raise Stop("run budget exceeded")
    key = (tool, normalize(args))
    run.seen[key] = run.seen.get(key, 0) + 1
    if run.seen[key] >= 3:
        raise Escalate(f"no progress: {tool} called 3x with the same arguments")
    if err is not None and not err.retryable:
        raise Escalate(f"{tool} failed with a non-retryable error: {err.code}")

5. Label the actions: automate, never automate, escalate

Score each action on four dimensions: safety and compliance, deterministic heuristic, reversibility, authorization. A fail on safety or reversibility rules out automation.

Action Label Why
Stop a run that exceeds its budget Automate Deterministic and reversible
Retry a timeout or 5xx up to 3 times Automate Deterministic, bounded, reversible
Raise a tenant's budget Escalate Needs a person to authorise it
Credit or refund the customer Escalate Money and customer relationship
Delete data or rotate production keys to "fix" the loop Never automate Fails reversibility

6. A workaround is not a fix

The emergency cap you added at 3am does not become permanent agent behaviour until it has been reviewed, tested and authorised. Track it like any other change.

7. Close-out checklist

  • [ ] Spend stopped and confirmed in the provider's billing
  • [ ] Traces and prompts preserved
  • [ ] 11 questions answered and labelled VERIFIED / PROPOSED / UNKNOWN
  • [ ] Per-run and per-tenant budgets in the code path, with tests
  • [ ] Retry policy by error class
  • [ ] No-progress detector, with a test that replays the incident
  • [ ] Temporary workarounds reviewed and either removed or approved
  • [ ] Customer told, by a person

Need a second pair of eyes?

I'm Muhammad Arshad, a forward-deployed AI engineer and former EVP Engineering at BITLOGIX. If an agent loop or bill spike hit you this month, I run a 48-hour AI bill / agent audit for $500. I reconstruct the incident from your own logs, rank the causes and hand you the fixes. Your data and code stay with you.

Book it by LinkedIn DM: https://www.linkedin.com/in/marshadofficial/

來源:DEV Community · dev.to