Repository navigation
BIRD: floating-point ratios and a joined-columns result check - #34
Merged
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Parth-Vasave
force-pushed
the
bird-schema-linking
branch
from
October 14, 2026 05:45
79eb444 to
bc2912a
Compare
To avoid integer division the model casts to NUMERIC, and the ratio then comes back as an arbitrary-precision decimal. BIRD's answers are floats, and its execution accuracy compares values exactly, so the same number scored as wrong (26 of the 28 "value format" misses on the Mini-Dev run). The validator now rewrites a CAST to NUMERIC/DECIMAL without precision to DOUBLE PRECISION when it only serves a division, and records how many it rewrote in the trace. A cast with a chosen precision, or one outside a division, is left alone. The prompt (sql-generator/8) asks for DOUBLE PRECISION in such calculations. Replayed read-only on earlier answers: Mini-Dev 16 fixed, 1 broken; dev sample 1 fixed, 1 broken. Both broken ones are gold queries that use a 1.0 literal and so return a decimal themselves.
The rule asked for DOUBLE PRECISION in "ratios, percentages and averages you divide yourself". On the dev sample the model also cast AVG(...) results to float (D1210), which then no longer matched an answer computed with AVG. The validator's rewrite of NUMERIC casts in divisions does the intended part on its own and never touches AVG; it is the variant measured on earlier answers (Mini-Dev 16 fixed, 1 broken). Prompt is sql-generator/9.
On the dev runs, answers missed where the model kept its own reading of the question over the definition given with it: it counted atoms where the formula divides bonds, filtered subtypes where the definition names the language, summed ids for SUM(id where ...) and left NULL rows out of a percentage's denominator. A percentage computed as (a / b) * 100 also differs in the last digit from a * 100 / b, which exact comparison counts as a different value. The rule now says to follow the definition where it and the question seem to differ, to use the columns and values it names, that COUNT(condition) and SUM(condition) count rows, and to compute a percentage as CAST(numerator AS DOUBLE PRECISION) * 100 / denominator over every row of the group. The examples are invented, not taken from BIRD. Prompt is sql-generator/10.
This reverts commit eec2715. Measured on dev with Cerebras gpt-oss-120b: on 60 questions whose hints contain a formula it scored 28/60 against 33/60 before (1 fixed, 6 broken), and 65/100 against 66/100 on the fixed dev sample. Three of the broken answers come from the percentage rule itself: an answer the gold computes in single precision was recomputed in double precision, the gold for another question computes (a / b) * 100, which the rule replaced by a * 100 / b, and a plain average was multiplied by 100. The row-count reading of COUNT(condition) also made the model count rows where the gold counts distinct patients. Prompt is back to sql-generator/9.
The prompt already asks for each stored column as its own result
column, yet the model still sometimes returns first_name || ' ' ||
last_name as one value. A new result check, joined_columns, reports an
output column that concatenates two or more columns (||, CONCAT,
CONCAT_WS) and sends it back through the repair loop with a hint to
return the columns separately. A single column with a label ('$' ||
price), a concatenation inside an aggregate or a subquery, and a key
built inside a CTE are not reported.
Replayed on every recorded BIRD answer without an LLM, it fires on the
three Mini-Dev test answers that concatenate (all misses) and on none
of the 491 dev answers. Repair prompt is sql-repair/4.
Parth-Vasave
force-pushed
the
bird-sql-correctness
branch
from
October 14, 2026 05:47
a226871 to
08544fa
Compare
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third of six, on top of #33.
CAST(... AS NUMERIC/DECIMAL)without a precision toDOUBLE PRECISIONwhen it only serves a division, and records the rewrite count in the trace. Replayed on earlier answers: Mini-Dev 16 fixed, 1 broken. The prompt rule that asked for this was dropped again, since it also made the model castAVG(sql-generator/9).Testing
Run locally on the top of the stack (
readme-bird-results):ruff check,ruff format --check,mypyclean; backend 340 passed, scripts + evaluation 115 passed (database integration tests skipped locally, run in CI); frontend lint and 57 tests pass. CI runs on each PR.