BotOfMouth 如何檢查 AI 助理會否提及企業
How we check whether AI assistants name a business
BotOfMouth介紹一套檢查方法,按問題、AI 助理及多次執行統計企業被提及的比例,並分開記錄自家網站是否獲引用。文章建議保留名稱別名、留意同名企業,並保存每次原始回答及日期;示例中,一間泰國島嶼的度假屋租賃公司在 3 種語言的 23 條非品牌問題、共 69 個回答中均未被提及。作者亦指出,API 結果可能與應用程式答案不同,應視為可重複的代理指標並清楚説明。
This article was written with AI assistance. Every fact in it links to its source.
A café owner asks you: "Does ChatGPT recommend us?"
You open ChatGPT, type "best café near the station", and the café isn't there. You try again. Now it is. So what do you tell the owner?
This is the measurement problem we work on at BotOfMouth. Below is the method, with a small Python harness you can run. Our question sets and prompts aren't in it, and you don't need them to follow along.
The unit: question × assistant × run
One answer is an anecdote. The unit that means something is a rate: of n answers to this question from this assistant, how many named the business?
Three choices define a check:
- Questions. What a customer would really type, without the business name: "Which dentist in Whitby would you recommend for a nervous patient?" Branded questions ("Is Harbour Street Dental any good?") are a separate set. They test facts, not discovery.
- Assistants. Each vendor has an API that runs a live web search and returns the answer with its sources.
- Runs. The same question, several times, on different days.
Ask through the search-grounded APIs
Here are three providers. Request shapes follow each vendor's docs as of November 2026, so check them before you copy: they change often.
import re
from urllib.parse import urlparse
import anthropic
from google import genai
from google.genai import types
from openai import OpenAI
host = lambda url: urlparse(url).netloc.removeprefix("www.")
def ask_openai(question):
r = OpenAI().responses.create(
model="gpt-6-astra",
tools=[{"type": "web_search"}],
tool_choice="required", # always search
input=question,
)
cited = [
host(a.url)
for item in r.output if item.type == "message"
for part in item.content
for a in (getattr(part, "annotations", None) or [])
if a.type == "url_citation"
]
return r.output_text, cited
def ask_gemini(question):
r = genai.Client().models.generate_content(
model="gemini-3.8-flash",
contents=question,
config=types.GenerateContentConfig(
tools=[types.Tool(google_search=types.GoogleSearch())]
),
)
meta = r.candidates[0].grounding_metadata
chunks = (meta.grounding_chunks or []) if meta else []
# web.uri can be a Google-hosted link; web.title carries the site's domain
return r.text or "", [c.web.title for c in chunks if c.web]
def ask_claude(question):
r = anthropic.Anthropic().messages.create(
model="claude-opus-5-5",
max_tokens=4096,
tools=[{"type": "web_search_20250305", "name": "web_search", "max_uses": 5}],
messages=[{"role": "user", "content": question}],
)
texts = [b for b in r.content if b.type == "text"]
cited = [
host(c.url)
for b in texts for c in (b.citations or [])
if c.type == "web_search_result_location"
]
return "".join(b.text for b in texts), cited
A few notes on those calls:
-
OpenAI: the tool type is
web_search(web_search_previewis the older name). Sources arrive asurl_citationannotations on the output text. Auser_locationon the tool lets you ask from a given country or city. -
Gemini: this is the
generateContentpath, where sources sit ingroundingMetadata. That object also haswebSearchQueries: the searches the model actually ran. Log it. It shows you how one customer question fans out into several searches. Google's docs now default to the newer Interactions API, where sources come back asurl_citationannotations instead. -
Claude:
web_search_20250305is the basic version of the tool. Newer versions add dynamic filtering and run the search inside code execution, which puts more blocks in the response. For a plain "what did it say and cite" check, the basic one is the easiest to parse. -
Perplexity: its Agent API takes the same kind of
web_searchtool and returns asearch_resultslist. Per its docs, support for Sonar chat completions ended in September 2026.
Decide "named" and "linked" in code
Two booleans per answer cover most of what an owner wants to know.
def norm(s):
return re.sub(r"[^a-z0-9]+", " ", s.lower()).strip()
def score(answer, cited_hosts, business):
text = norm(answer)
return {
"named": any(norm(n) in text for n in business["names"]),
"own_site_cited": any(h.endswith(business["domain"]) for h in cited_hosts),
"cited": sorted(set(cited_hosts)),
}
Name matching is where most of the mistakes hide:
- Keep a list of aliases ("Harbour St Dental", the name without "Ltd").
- Watch for name collisions. In one check, three unrelated businesses on neighbouring islands had almost the same name as our client. A string match would have counted their mentions as a win.
- "Cited" and "named" are different results. An assistant can read your site, cite it, and still recommend someone else.
Run it many times
BUSINESS = {"names": ["Harbour Street Dental", "Harbour St Dental"],
"domain": "harbourstreetdental.example"}
QUESTION = "Which dentist in Whitby would you recommend for a nervous patient?"
RUNS = 5
for name, ask in [("openai", ask_openai), ("gemini", ask_gemini), ("claude", ask_claude)]:
results = [score(*ask(QUESTION), BUSINESS) for _ in range(RUNS)]
named = sum(r["named"] for r in results)
linked = sum(r["own_site_cited"] for r in results)
print(f"{name}: named in {named} of {RUNS}, own site cited in {linked} of {RUNS}")
Why repeat? Because the answers really do move. SparkToro and Gumshoe had 600 volunteers run 12 prompts 2,961 times and found a less than 1 in 100 chance of getting the same list of brands twice. You can't switch it off either: Google's docs say Gemini 3.6+ models ignore custom temperature values, and every call runs a fresh search.
The same study has the useful half of the story: presence is steadier than position. One hospital appeared in 69 of 71 answers but came first in only 25. So we count how often a business comes up, and we don't report rank.
Store every raw answer with its date, model, question and run number. When a rate changes, you'll want to read the answers, not only the totals.
What the number does not tell you
The API is not the app. Perplexity says so directly: the API and the web interface "share search infrastructure, but their tools, configuration, and models differ, so results can differ". A Surfer study that compared scraped app answers with API answers for 1,000 prompts found the overlap was "just 30% at best". Treat API results as a consistent proxy you can repeat, and say so wherever you show them.
Location and language change the answer. Ask in the languages customers use, and set a location where the API allows it.
Small samples lie. "Named in 1 of 5" and "named in 0 of 5" are the same result. Look for changes that hold up on a second check, on a different day.
What a result looks like
One real check, with the client's details removed: a villa rental company on a Thai island. We asked 23 unbranded questions in three languages through three assistants, 69 answers in all. The company was named in none of them. The most-named competitor showed up in 16 of the 30 answers to one group of questions. Another competitor, with only four Google reviews, was in all 10 Google AI Mode answers to that group. Its Google Maps category matched the question. Our client's didn't.
That last detail is why we keep the raw answers and sources. The rate tells you where you stand. The citations tell you what to fix.
Two honest limits on that example: each question ran once, and the assistants were queried through APIs and Google AI Mode, not through the ChatGPT or Perplexity apps.
Cost
Search tools are billed per call on top of tokens. Listed prices today: OpenAI $10 per 1,000 calls, Claude $10 per 1,000 searches, Gemini $14 per 1,000 search queries after a free monthly allowance, Perplexity $2.50 per 1,000. A model can run several searches for one question, so measure before you schedule anything weekly.
This is the check BotOfMouth runs for small businesses. If you'd rather not build it, there's a free website check at botofmouth.com.
來源:DEV Community · dev.to