跳到正文
DEV Community· Ashish sinha·· 2026-10-24AI 評分57

一個月公開基準測試結果呈現 schemagate text-to-SQL 存取層的表現

What one month of public benchmarks says about my text-to-SQL access layer

AI 導讀

作者公佈開源 text-to-SQL 權限存取層 schemagate 的公開基準測試結果,Spider-ACL 的 refusal F1 為 0.984,BIRD Dev 執行準確率為 77.1%。

正文

Cover image for What one month of public benchmarks says about my text-to-SQL access layer

Ashish sinha

schemagate is an open-source layer for text-to-SQL that starts from one question: what is this caller allowed to see? It shows the model only the tables the current user may read, before any SQL is written, reads the permissions from your database's own grants, and refuses a query that touches anything else before it reaches the database.

This month I ran it on public benchmarks. Every number below comes from that benchmark's own evaluator, and the predictions are in the repo.

Access control

  • RBAC Text-to-SQL (SIGMOD '27): 90.1 AC-F1, against 88.1 for the paper's best setup with the same model.
  • Spider-ACL: 0.984 refusal F1 over all 19,624 question-and-role pairs; best published 0.933.
  • 0 queries read data the role was not granted, out of 8,128 answered.

BIRD

  • Dev set, all 1,534 questions: 77.1% execution accuracy with one model call per question. No candidate voting, no agent loop.
  • Mini-Dev: 81.0 / 72.6 / 72.4 on SQLite / PostgreSQL / MySQL, above the current listed leaders. The leaderboard pull request is waiting on the maintainers.
  • Hidden test set: submitted; the BIRD team runs it.

Spider 2.0

  • SQLite questions (135): 71.9% with one call per question. The best published agent on the same subset reports 57.8% Pass@1.
  • The BigQuery and Snowflake runs for a full leaderboard entry are under way.

Oracle

  • UniQL, Oracle dialect, 1,534 questions: 64.5% on the first try, against a best published baseline of 49.6. Two prompt rules about Oracle syntax did most of it.

Adoption

About 10,000 downloads on PyPI so far (pypistats, mirrors excluded). Apache-2.0. Works with Oracle, PostgreSQL, SQL Server, MySQL and SQLite, as an MCP server, a LangChain retriever or a CLI.

pip install schemagate

What these numbers are not: the Mini-Dev rows are self-reported until BIRD merges them, single runs vary by about a point, and the Spider 2.0 figure is one third of that benchmark. The details and every caveat are on the benchmarks page.

GitHub: https://github.com/ashishsinha1602/schemagate · a star helps other people find it.

來源:DEV Community · dev.to