Repository navigation
BIRD Mini-Dev run results, hand review and official-script export - #35
Merged
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Parth-Vasave
force-pushed
the
bird-sql-correctness
branch
from
October 14, 2026 05:47
a226871 to
08544fa
Compare
From reading the misses of the full Mini-Dev test run on Cerebras (20261012-094139), each checked against the BIRD database with a corrected query: - B0234: missing parentheses around an OR, and atoms _1 / _2 where the hint names TR009_12 (1041 instead of 6 connections). - B1107: the outer ORDER BY date discards the ranking by crossing, so it returns the latest record instead of the first date with the highest crossing. - B0474 and B0486: the hint contradicts the question and BIRD's own SQL (baseSetSize < 10 instead of < 100; a percentage divided by the sum of mana costs instead of the number of cards). B0424, also suspected, follows its own hint and is not listed. The review stays beside the official score and never changes it.
All 15 BIRD runs of the project, from the first partial run to the final test run (284/500 official EX on the PostgreSQL Mini-Dev, against 258/500 for the baseline), so the numbers in the docs can be checked with python -m evaluation.report. BIRD_RUNS.md lists each run with its code commit, what it measured and what was kept. The result files contain BIRD questions, hints and data values, so they are shared under CC BY-SA 4.0 with attribution to BIRD, as stated in BIRD_RUNS.md; the rest of the repository stays MIT.
python -m evaluation.bird export-official writes a run in the input format of evaluation_ex.py from bird-bench/mini_dev: the predictions in question order, the canonical ground truth as one line per question and the questions as JSON lines. BIRD_RUNS.md gives the steps to score it with BIRD's script against the local database. Done for the final test run: BIRD's script reports EX 56.80 (simple 70.95, moderate 52.80, challenging 46.08), the same 284/500 as the project's scorer. With the July 2025 package's older ground-truth file it reports 56.60: that file still has BIRD's earlier SQL for B0879. mypy no longer looks into evaluation/results/, which holds run outputs.
Parth-Vasave
force-pushed
the
bird-minidev-results
branch
from
October 14, 2026 05:48
cea3d0f to
d50964b
Compare
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fourth of six, on top of #34.
evaluation/results/BIRD_RUNS.mdindexes them. The result files hold BIRD questions and data, so they are shared under CC BY-SA 4.0 with attribution; the rest of the repository stays MIT.python -m evaluation.bird export-officialwrites a run in the input format of BIRD'sevaluation_ex.py. BIRD's script reports EX 56.80 on the final run, the same 284/500 as the project's scorer.evaluation/results/.Testing
Run locally on the top of the stack (
readme-bird-results):ruff check,ruff format --check,mypyclean; backend 340 passed, scripts + evaluation 115 passed (database integration tests skipped locally, run in CI); frontend lint and 57 tests pass. CI runs on each PR.