AI 搜尋引用網頁前的三項快速檢查
Three quick checks before you expect AI search to quote your page
作者提出三項可自行完成的檢查,讓網站管理者核對 AI 爬蟲規則、llms.txt 與結構化資料是否和頁面一致。檢查包括查看 robots.txt 及 CDN/WAF 是否阻擋爬蟲、確認 llms.txt 內容準確,以及比對 JSON-LD 的作者、更新日期和標題與可見頁面。這些做法不能保證頁面獲引用,只能排除可能令頁面被跳過或誤讀的問題;頁面是否回答用戶提問,以及首段寫得是否清楚更重要。
I've been building small Claude/Cursor skills that review pages for AI-search readiness. Running them on public pages kept turning up the same three gaps. You can check all three yourself in about ten minutes with curl and view-source. You don't need a tool.
1. Does robots.txt say anything about AI crawlers?
curl -s https://yoursite.com/robots.txt | grep -iE 'gptbot|claudebot|perplexitybot|google-extended|ccbot'
No output means you have no AI-specific rules, so these bots follow your User-agent: * group. That's fine if it's on purpose. The problems I see are leftovers: a blanket Disallow someone added years ago, or a CDN/WAF bot rule that blocks crawlers the robots.txt says are allowed. Check both.
2. Do you have an llms.txt, and is it accurate?
curl -s -o /dev/null -w '%{http_code}\n' https://yoursite.com/llms.txt
llms.txt is a proposed convention, not a standard, and no AI company has promised to read it. It's still cheap to add. Put a one-line summary of what you do at the top, then short sections linking your key pages. When I checked on Nov 13, stripe.com returned 200 with exactly that layout, and anthropic.com returned 404. If you add one, keep it current. A stale llms.txt is worse than none.
3. Does your structured data agree with the visible page?
Open view-source, search for application/ld+json, and compare it with what a reader sees:
- author: is it null or missing while the page names a person or company?
- dateModified: the page says "Last updated March 2026", but is that date in the JSON-LD?
- headline: does it match the H1, or carry an extra "| Brand" suffix?
On one well-known payments explainer, all three were off: author was null, dateModified was missing even though the page showed a date, and the headline had a brand suffix. These take minutes to fix. Don't add FAQ or rating markup the page doesn't actually show.
What these checks won't tell you
None of this guarantees a citation. Nobody outside the AI companies knows exactly how sources get chosen. These checks only remove the cheap reasons a page might be skipped or misread. Whether people are asking the question your page answers, and whether you answer it well in the first paragraph, matters more.
Disclosure: I make HubDev, a set of Claude/Cursor skills that run checks like these. If you'd rather have an agent score a page for you, there's a free starter scorecard (no purchase needed): https://payhip.com/b/TvxJH. Paid packs exist, but nothing in this post needs them.
來源:DEV Community · dev.to