Repository navigation
BIRD: linking tables, inferred joins, lenient plan intent and contact columns - #33
Merged
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Schema retrieval matched keywords of the question only and added only the tables that matched tables point to. A linking table (hero_power between superhero and superpower, results between drivers and races) rarely shares a word with the question, so the model was not shown it and answered "cannot be answered from this schema". Now a table whose foreign keys point at two or more selected tables is added, and the user's definitions take part in choosing tables. Measured on BIRD (gold tables all shown): Mini-Dev 94.8% -> 100%, dev 97.1% -> 99.4%, about one more table per prompt.
The plan's intent is only a label shown with the plan and recorded in the trace; nothing decides on it. A strict Literal rejected the whole reply when the model wrote an unknown one (often the prompt's own list, "lookup|aggregate"), throwing away valid SQL and costing a repair call. On the BIRD Mini-Dev run this hit 9 questions; 4 ended as errors. An unknown intent is now "other". The test that expected such a reply to be rejected now expects it to be kept, with the intent as "other".
Database dumps often keep some foreign keys and drop others. BIRD's card_games declares rulings.uuid -> cards.uuid but not foreign_data.uuid or legalities.uuid, and set_translations.setcode -> sets.code but not cards.setcode. Join inference only knew <name>_id -> primary key, so the model was shown no join for those tables and guessed one (foreign_data via multiverseid), giving empty or wrong results. A column named like a declared single-column foreign key now joins what that key references, unless the name is declared with different targets. On the 11 BIRD databases this adds exactly those three joins.
The sensitive-column policy hid california_schools' phone and admemail1-3 and student_club's member phone and email, so 6 Mini-Dev and 14 dev questions that ask for them could only be misses. Other systems on the leaderboard see every column, and this is long-published public data. BIRD connections now allow them, like the four birth-date columns already allowed. The app's default policy is unchanged. A test checks that every allowed name is one the policy would hide.
The integrity check counts the rows of all 75 BIRD tables after each run. It used the read-only account's statement timeout (the agent's 5 s), and once timed out on a busy server, ending a run with a traceback after its answers were saved. It now sets the ground-truth timeout (60 s) for its own session, still read-only.
Parth-Vasave
force-pushed
the
bird-schema-linking
branch
from
October 14, 2026 05:45
79eb444 to
bc2912a
Compare
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second of six, on top of #32.
other, instead of being rejected. The test that expected rejection now expects the reply to be kept.Testing
Run locally on the top of the stack (
readme-bird-results):ruff check,ruff format --check,mypyclean; backend 340 passed, scripts + evaluation 115 passed (database integration tests skipped locally, run in CI); frontend lint and 57 tests pass. CI runs on each PR.