語音 Agent 的 800 ms 等待問題如何以智能回合偵測改善(45 行 Python 模擬)
Your Voice Agent Makes Every Caller Wait 800 ms. Smart Turn Detection Fixes It (Simulated in 45 Lines of Python)
文章以 45 行 Python 模擬展示,按已説詞語調整等待時間,可改善語音 Agent 固定靜音計時帶來的延遲與打斷問題。固定 800 ms 計時在 5 組通話中打斷 3 組,其餘回覆延遲 800 ms;簡單詞表規則打斷 1 組,其餘回覆延遲 250 ms。
Twilio made Smart Turn Detection generally available for ConversationRelay last week. Its changelog describes it as a semantics-aware end-of-turn model that replaces "rigid silence timers", and you switch it on per call with turnDetection="true".
That one feature fixes the most common complaint about AI voice agents: they either talk over you, or they sit in silence for a second after you finish. Both come from the same place. The agent has to guess when you are done talking, and most agents guess with a stopwatch.
Here is why the stopwatch fails, and a small simulation you can run that shows the fix, in 45 lines of Python.
The stopwatch problem
The classic voice agent waits for a fixed amount of silence, say 800 ms, and then treats the caller as finished. That forces a bad trade:
- Set it short (300 ms) and the agent jumps in every time someone pauses to think, reads a number slowly, or says "um".
- Set it long (800 ms or more) and every single reply starts with an awkward gap, even after "Yes, Friday works."
People do not pause at random. They pause in the middle of things: between the digits of an order number, after "and", after "my". Those pauses can be a full second long. A timer cannot tell them apart from the end of a sentence.
The idea: wait based on what was said
A smart turn detector looks at the words, not just the silence. If the sentence sounds complete, wait a little. If it obviously is not ("My order number is..."), wait much longer. That gives you fast replies and fewer interruptions at the same time.
The simulation
Five realistic callers, each word with its start and end time. Two of them pause for a second mid-sentence, one reads a 4 digit number slowly, one says "um".
import statistics
# Each call: (text, [(word, start_ms, end_ms), ...]). The caller is done when the last word ends.
# Long gaps inside a sentence are where fixed silence timers cut people off.
CALLS = [
("Yes, Friday works.", [("yes",0,300),("friday",400,700),("works",750,1000)]),
("My order number is ... 4 4 7 1", [("my",0,200),("order",250,500),("number",550,800),("is",850,1000),
("4",2000,2150),("4",2300,2450),("7",2600,2750),("1",2900,3050)]),
("I want to book ... um ... a haircut", [("i",0,150),("want",200,400),("to",450,550),("book",600,900),
("um",1900,2100),("a",2900,3000),("haircut",3050,3400)]),
("Can you call me back tomorrow?", [("can",0,200),("you",250,350),("call",400,600),("me",650,750),
("back",800,950),("tomorrow",1000,1400)]),
("It's for two people and ... my mom", [("its",0,200),("for",250,350),("two",400,600),("people",650,900),
("and",950,1150),("my",2000,2150),("mom",2200,2500)]),
]
UNFINISHED = {"and", "but", "or", "is", "to", "a", "the", "my", "for", "um", "uh", "so", "because", "with"}
def sounds_done(heard):
last = heard[-1]
if last in UNFINISHED: return False
digits = [w for w in heard if w.isdigit()]
if last.isdigit() and len(digits) < 4: return False # a 4 digit number is still coming
return True
def simulate(words, wait_for):
"""Agent replies once the silence after a word is longer than wait_for(words heard so far)."""
true_end = words[-1][2]
for i, (_, _, end) in enumerate(words):
heard = [w for w, _, _ in words[: i + 1]]
silence = words[i + 1][1] - end if i + 1 < len(words) else float("inf")
wait = wait_for(heard)
if silence > wait:
reply_at = end + wait
return ("CUT OFF", 0) if reply_at < true_end else ("ok", reply_at - true_end)
fixed = lambda heard: 800 # classic silence timer
smart = lambda heard: 250 if sounds_done(heard) else 1500 # quick when complete, patient when not
for name, policy in [("fixed 800 ms", fixed), ("smart turn", smart)]:
results = [simulate(words, policy) for _, words in CALLS]
cut = sum(r == "CUT OFF" for r, _ in results)
delays = [d for r, d in results if r == "ok"]
print(f"{name}: {cut}/{len(CALLS)} callers cut off, avg reply delay {statistics.mean(delays):.0f} ms")
for (text, _), (r, d) in zip(CALLS, results):
print(f" {r:<8}{d:>5} ms {text}")
The result
$ python turns.py
fixed 800 ms: 3/5 callers cut off, avg reply delay 800 ms
ok 800 ms Yes, Friday works.
CUT OFF 0 ms My order number is ... 4 4 7 1
CUT OFF 0 ms I want to book ... um ... a haircut
ok 800 ms Can you call me back tomorrow?
CUT OFF 0 ms It's for two people and ... my mom
smart turn: 1/5 callers cut off, avg reply delay 250 ms
ok 250 ms Yes, Friday works.
ok 250 ms My order number is ... 4 4 7 1
CUT OFF 0 ms I want to book ... um ... a haircut
ok 250 ms Can you call me back tomorrow?
ok 250 ms It's for two people and ... my mom
The fixed timer cuts off three of five callers and still makes everyone else wait 800 ms. The text-aware policy replies in 250 ms and cuts off only one caller.
Look at the failure
The one it gets wrong is "I want to book ... um ... a haircut". After "book" the sentence sounds complete to a word list, so the agent jumps in. A human hears that "book" needs an object. That is exactly the gap between a rule like UNFINISHED and a trained end-of-turn model such as Twilio's, which reads meaning rather than the last word.
So the lesson is not "use my word list". It is:
- Turn detection is a decision about meaning, not silence. Silence is one input.
- Rules get you surprisingly far, and they are a good baseline to measure a model against.
- Measure both failure modes. Average reply delay alone looks great for an agent that interrupts everyone. Track cut-offs too.
Taking it to production
- Use a semantic turn model from your voice platform or an open-source one, and keep a rule layer for things you know: digits, emails, addresses and spelled names.
- Make the wait depend on context. If the agent just asked for a phone number, expect pauses and wait longer. If it asked a yes or no question, reply fast.
- Handle barge-in anyway. No detector is perfect, so when the caller keeps talking, stop the agent's audio within a couple hundred milliseconds.
-
Turn real calls into test cases. Every call where the agent interrupted someone becomes a new entry in your list, exactly like
CALLSabove, and you run it on every change. - Log turn events. Twilio can send StartOfTurn, EagerEndOfTurn and EndOfTurn events in the WebSocket payload. Store them with timestamps so you can see where turns go wrong.
Good turn taking is most of what makes a voice agent feel human. The model can be brilliant, but if it interrupts people reading their order number, callers will hang up.
If you are building a voice agent and want it to feel natural on real calls, that is what we do at Axionry, an AI product development studio. We ship in weeks.
Source
來源:DEV Community · dev.to
