智能體失控循環事故應變手冊
Runaway agent loop: incident runbook
這份事故手冊為失控智能體循環提供止損、事件重建、永久防護及結案流程。文中列出的公開案例中,一名客戶的智能體在凌晨 1:12 至 6:50 間重試失敗工具 31,000 次,產生 $4,700 的模型 token 費用,當時方案月費為 $400。永久控制包括每次執行及每個租戶的預算上限、按錯誤類別設定最多 3 次重試,以及相同工具與正規化參數重複 3 次時停止並升級處理。
Runaway agent loop: incident runbook
RB-01 · v0.1 · Muhammad Arshad · CC BY 4.0
When to use this
- Token spend for one run, customer or agent is far above its normal rate.
- The same tool is called again and again with near-identical arguments.
- The agent keeps producing output but makes no progress toward its goal.
Public example: a 3-person B2B team reported that a customer's agent retried a failing tool 31,000 times between 1:12am and 6:50am. It spent $4,700 in tokens on a $400/month plan. They had a global rate limit and total-spend alerts, but nothing per customer. (r/SaaS, October 2026)
Source: https://www.reddit.com/r/SaaS/comments/1xh0cq5/an_agent_got_stuck_in_a_loop_and_burned_4700_of_a/
Severity
- SEV-1: spend is still rising, or the loop touched customer data or side-effecting tools.
- SEV-2: contained, but the cost is unbilled or someone has to absorb it.
- SEV-3: caught by existing caps; no customer impact.
1. Contain (first 15 minutes)
- Stop the run: kill switch, pause the queue, or disable that tenant's agent key.
- Freeze, don't delete: keep traces, prompts, tool responses and billing exports.
- Add a temporary hard cap per tenant in front of the model call.
- Write down every containment action with the time and who did it. Mark each one TEMPORARY.
2. Reconstruct: 11 questions
Label every answer VERIFIED (from logs or billing), PROPOSED (a hypothesis) or UNKNOWN. Unknowns stay unknown until evidence arrives.
| # | Step | Question |
|---|---|---|
| 1 | Deviation event | What exactly happened, and between which timestamps? |
| 2 | Root driver | What made the agent keep going? (error text, retry policy, missing budget) |
| 3 | Discovery | How was it found, by whom, and how long after it started? |
| 4 | Observation | What did the traces and the bill show? |
| 5 | Escalation | Who was told, and how fast? |
| 6 | Tacit knowledge | What did the on-call person know that isn't written down anywhere? |
| 7 | Alternatives and trade-offs | Which other fixes were considered, and why were they rejected? |
| 8 | Authorization | Who approved the containment and the fix? |
| 9 | Loss and impact | Tokens, money, customer trust, side effects. |
| 10 | Containment | What stopped it? Is that change temporary or permanent? |
| 11 | Prevention | Which control would have stopped it at step 3 instead of step 31,000? |
3. Usual causes
- Retries on errors that can never succeed (validation, auth, not found).
- Tool errors returned as plain text that the model reads as "try a slightly different prompt".
- No per-run budget for steps, tokens or wall-clock time.
- Spend checked only in total, not per tenant or per run.
- Loop detection that only catches exact repeats, while the agent varies its prompt slightly.
- A tool timeout that triggers a re-call while the first call is still running, duplicating side effects.
4. Permanent controls
- Per-run budget: max steps, max tokens and max wall-clock time, checked before every model call.
- Per-tenant budget: a soft alert at 1x plan and a hard stop at 2x (the fix the team in the example adopted).
- Retry policy by error class: retry only 429s, 5xx and timeouts, with backoff, at most 3 times.
- No-progress detector: the same tool with the same normalised arguments 3 times stops the run and escalates.
- Tool errors returned as structured observations with
retryable: false. - Idempotency keys on every side-effecting tool.
class Stop(Exception): ...
class Escalate(Exception): ...
def guard(run, tool, args, err=None):
if run.tokens > run.budget.tokens or run.steps > run.budget.steps:
raise Stop("run budget exceeded")
key = (tool, normalize(args))
run.seen[key] = run.seen.get(key, 0) + 1
if run.seen[key] >= 3:
raise Escalate(f"no progress: {tool} called 3x with the same arguments")
if err is not None and not err.retryable:
raise Escalate(f"{tool} failed with a non-retryable error: {err.code}")
5. Label the actions: automate, never automate, escalate
Score each action on four dimensions: safety and compliance, deterministic heuristic, reversibility, authorization. A fail on safety or reversibility rules out automation.
| Action | Label | Why |
|---|---|---|
| Stop a run that exceeds its budget | Automate | Deterministic and reversible |
| Retry a timeout or 5xx up to 3 times | Automate | Deterministic, bounded, reversible |
| Raise a tenant's budget | Escalate | Needs a person to authorise it |
| Credit or refund the customer | Escalate | Money and customer relationship |
| Delete data or rotate production keys to "fix" the loop | Never automate | Fails reversibility |
6. A workaround is not a fix
The emergency cap you added at 3am does not become permanent agent behaviour until it has been reviewed, tested and authorised. Track it like any other change.
7. Close-out checklist
- [ ] Spend stopped and confirmed in the provider's billing
- [ ] Traces and prompts preserved
- [ ] 11 questions answered and labelled VERIFIED / PROPOSED / UNKNOWN
- [ ] Per-run and per-tenant budgets in the code path, with tests
- [ ] Retry policy by error class
- [ ] No-progress detector, with a test that replays the incident
- [ ] Temporary workarounds reviewed and either removed or approved
- [ ] Customer told, by a person
Need a second pair of eyes?
I'm Muhammad Arshad, a forward-deployed AI engineer and former EVP Engineering at BITLOGIX. If an agent loop or bill spike hit you this month, I run a 48-hour AI bill / agent audit for $500. I reconstruct the incident from your own logs, rank the causes and hand you the fixes. Your data and code stay with you.
Book it by LinkedIn DM: https://www.linkedin.com/in/marshadofficial/
來源:DEV Community · dev.to