[Project] RL on Text to SQL with Qwen 3B
I have been getting into RL post training recently, and I wanted a small project that would test my understanding of the implementation side rather than the theory: build an environment, write a reward, generate the data, train a model, and read the results. This is that project. I trained a 3B code model with GRPO to write SQL for one real bank database, using execution feedback as the only reward, over two runs and about 3 days. The interesting part turned out to be why the first run stalled, and what I found in the benchmark while working out why.
The task
Text to SQL maps a natural language question about a database to a SQL query that answers it. The following test question from this project illustrates the setting:
How many accounts who choose issuance after transaction are staying in East Bohemia region?
The gold query returns 13:
SELECT COUNT(T2.account_id)
FROM district AS T1
INNER JOIN account AS T2 ON T1.district_id = T2.district_id
WHERE T1.A3 = 'east Bohemia' AND T2.frequency = 'POPLATEK PO OBRATU'Producing it requires four pieces of knowledge that the question does not contain: that “issuance after transaction” corresponds to the value POPLATEK PO OBRATU in the frequency column, that “region” is the column A3, that the stored value is east Bohemia in lowercase, and that accounts join to districts on district_id. The difficulty of the task is the distance between how a question is phrased and how the data is stored.
That distance is also what makes the task well suited to reinforcement learning with verifiable rewards. The reward is exact and cheap to compute: execute the model’s query and the gold query, and compare the result sets. No judge model or rubric is involved. And because the knowledge being acquired is specific to one schema, a small model trained on that schema has a plausible path to matching a much larger model that has never seen it.
The financial database
The database is financial from the BIRD benchmark, the PKDD 1999 Czech bank dataset. It has eight tables covering clients, accounts, the dispositions that link them, transactions, loans, credit cards, standing orders, and district demographics.
Three properties of this schema distinguish it from the other BIRD databases I considered, and each one is a distinct thing the model has to learn.
The schema does not describe itself. In most databases the column names and values carry meaning a model can read directly: a column called salary holds salaries. Here the demographic columns are named with codes, and the categorical values are abbreviations in Czech. The mapping from an English phrase to a column or a stored value has to be learned from experience with the database or supplied in the prompt; it cannot be inferred from the schema text. This is the knowledge that separates a model that has trained on this database from one that has only read the schema.
The relationships between entities are indirect. Clients and accounts are not linked to each other. Both link to a dispositions table that records each client’s role on each account, so any question that connects a person to their transactions or loans passes through a table the question never mentions. Clients and accounts also each have their own district, and for 494 clients the two differ. The correct join path has to be reasoned out from the structure rather than read off the question, and a model that has not learned the structure tends to invent the direct link it expects to find.
Exact values matter. String comparisons in the database are case sensitive, and questions are written in ordinary English capitalization while values are stored as they were entered. A query that is right in every other respect returns nothing if a region name is capitalized differently. The reward gives no credit for almost right.
Why this database. Before choosing a database, I scanned five BIRD databases with two models: Claude Sonnet 5 as a frontier reference, and the untrained 3B model sampled eight times per question so that I could measure not just accuracy but how many questions had mixed outcomes, the quantity GRPO learns from.
| Database | Sonnet, hints / none | 3B, hints / none | 3B mixed groups of 40 |
|---|---|---|---|
| financial | 78 / 53 | 15 / 7 | 17 / 10 |
| card_games | 60 / 55 | 45 / 28 | 17 / 15 |
| codebase_community | 75 / 65 | 47 / 47 | 16 / 23 |
| european_football_2 | 72 / 68 | 53 / 25 | 22 / 18 |
| toxicology | 60 / 57 | 38 / 12 | 18 / 13 |
financial had the largest gap between the frontier model and the small one, and the largest gap between the hinted and unhinted scores for the frontier model, which is the signature of a database where the missing ingredient is schema knowledge rather than SQL skill. Reading the 3B model’s failures on it confirmed that: wrong join keys, invented columns, the reserved word order left unquoted. Those are learnable errors. It also executes quickly, which matters when the reward loop runs hundreds of thousands of queries.
Setup
Everything in this section is shared by both runs: how a reply is scored, what it is tested on, and how the model is trained. The two runs differ only in the prompt and the training data, which each run describes for itself.
The reward
The model gets the schema and a question and has to return one SQL query in a code block. The verifier runs that query against a read only copy of the database, runs the gold query, and compares the two result sets. The reward is 1 if they match and 0 otherwise. The rules that fill in “match”:
- Row order is ignored unless the question asks for a ranking, in which case it is compared as a list.
- Duplicate rows are collapsed. This is how BIRD’s own execution accuracy metric works.
- The number of columns has to match. Returning the right answer plus an extra column scores 0.
- Numbers compare numerically, so 1 and 1.0 are equal. Strings compare exactly, case included.
- An empty result never scores. Neither does a result that is all NULLs, a query that errors, or one that runs past 30 seconds.
- More than one code block, or none, scores 0. There is no partial credit for anything.
I also added a rule that any result over 1,000 rows scores 0, to keep giant accidental queries from slowing the trainer. That rule was a mistake, and I will come back to it.
The environment is written against Prime Intellect’s verifiers library and trained with their prime-rl trainer, GRPO with vLLM generating rollouts on one GPU and the update on another.
The test set
The test questions come from BIRD, the benchmark most text to SQL work has reported on since 2023. BIRD’s authors took 95 real databases across 37 domains and had annotators write questions about them along with the SQL that answers each one, about 12,000 pairs in all. Two things set it apart from earlier benchmarks. The databases are messy in the way real ones are, with coded columns, inconsistent values, and tables large enough that a query has to be right rather than approximately right. And each question comes with a hint, a sentence of external knowledge the annotator thought a solver would need, such as “A3 contains the data of region” or “when the account type = ‘OWNER’, it’s eligible for loans.” Systems are normally evaluated with the hint supplied.
BIRD’s public dev split has 1,534 questions across 11 databases, and 106 of them are about financial. Those 106 are the test set for everything in this post. They come labeled by difficulty, 62 simple, 37 moderate, and 7 challenging, and I score every checkpoint on them twice, once with the hints and once without. The dropdown has one question from each difficulty, with its hint and gold query, to give a sense of what the annotators’ English looks like.
The questions are exactly as the annotators wrote them, typos and all. Two later corrections of these same questions exist, and from run 2 on I score on those as well; the benchmark section near the end explains them. A leak filter removed any training row whose result set or SQL shape matched one of the 106, so nothing in the pool is a paraphrase of a test question.
Model and training loop
I chose Qwen2.5-Coder-3B-Instruct. I wanted a code model rather than a general one because its base accuracy on SQL is higher, and under GRPO that matters directly: the algorithm only learns from questions where some samples in a group succeed and others fail, so a model that already gets some answers right has more to learn from. I used the instruct checkpoint so that it would follow the output format without a supervised stage, and I stayed at 3B because a training pass fits on one GPU and a specialization result at that size is more interesting than at 30B.
I trained with LoRA at rank 32 rather than fine tuning the full model, for memory and speed. RL needs very little capacity, since each episode delivers roughly one bit of information, and Schulman et al. showed in 2025 that LoRA matches full fine tuning for policy gradient methods even at very low rank when applied to every layer.
The training loop is prime-rl, which runs as three concurrent processes: an inference server on one GPU generating replies with the current policy, an orchestrator that draws 48 questions per step, samples 8 replies for each at temperature 1, scores them with the verifier, and packs the batch, and a trainer on the second GPU that updates the adapter and broadcasts the new weights. I used its default GRPO configuration unchanged. The advantage is the reward minus the group mean, groups where every reply scored the same are dropped, and the asynchronous pipeline and that zero variance filtering are the parts of the recipe the ScaleRL paper recommends. AdamW at a learning rate of 2e-5, 250 steps, and an evaluation on the 106 test questions every 25 steps, without hints and again with hints.
Run 1
The first run was deliberately minimal: the instruct model, no supervised stage, and a prompt that withheld everything except the schema, so that anything the model learned about this database could be attributed to execution feedback alone.
Prompt
Each training example is a single prompt and a single reply. The prompt has three parts: a fixed system instruction, the schema as CREATE TABLE statements with three sample rows per table, and the question. The full prompt is about 1,900 tokens; the dropdown shows it for one question, with the schema cut down to one of the eight tables.
The model replies with one SQL query in a code block, and the verifier does the rest.
The prompt contains the schema but nothing about what its coded columns or Czech values mean, and no hint. In the standard BIRD setting a system gets all three, and I withheld them on purpose: that knowledge is what distinguishes a model trained on this database from one that has only read its schema, and I wanted to know whether the execution reward alone could teach it. Run 2 revisits this.
Training data
Since the only human written questions about this database are the 106 I use as the test set, the training questions had to be generated, and this is where most of my effort went. In RL with a verifiable reward, the training set effectively is the environment. Whatever shapes and phrasings it contains are what the model can learn, and any incorrect gold it contains gets learned as well. Two decisions set the design, and the rest of this section describes how I carried each one out and checked it. First, the gold SQL would come from templates rather than from a model, so that its correctness is guaranteed by construction. Second, difficulty would be organized as a curriculum over SQL structure, which is the standard axis in text to SQL work and the one I could control directly.
The semantic layer. Because the schema does not describe itself, my first step was to write down what everything means: each coded district column, each Czech value in each categorical column, and a one line description of each table. This legend is the source of truth for every English question the generator produces, and it is what the rewriting model sees when it rephrases a question. In run 1, I deliberately kept it out of the training prompt.
Tiers. Before writing any templates, I defined four tiers according to the SQL structure a question requires. Each template family belongs to exactly one tier.
| Tier | Structure | Example |
|---|---|---|
| 1 | One table: a filter, a count, an aggregate, or a top k | Which 5 districts have the highest number of committed crimes in 1996 per 1000 inhabitants? |
| 2 | One join: a filter on one table, output or aggregate from the other | How many female clients in the region ‘north Bohemia’ were born in the 1950s? |
| 3 | Two or three joins with grouping, a HAVING clause, or a ranking | What percentage of female clients in the region ‘south Bohemia’ were born before 1950? |
| 4 | Subqueries, set operations, date arithmetic, conditional aggregation | Which loans in ‘north Moravia’ have a higher monthly payment than the average for loans in their own district? |
The tiers form a curriculum in the sense that matters for GRPO. A question only contributes a gradient when the model is sometimes right on it, so the pool needs questions at the edge of what the untrained model can already do, along with harder ones that become learnable later.
Templates. A template is a function that samples real values from the live database, builds a SQL query from those choices, and renders an English question from the same choices, so the query and the question cannot disagree. Because the values come from the database itself, every filter is guaranteed to match something. For example, one tier 3 template picks a region, a transaction category, and a year:
SELECT SUM(T1.amount)
FROM trans AS T1
INNER JOIN account AS T2 ON T1.account_id = T2.account_id
INNER JOIN district AS T3 ON T2.district_id = T3.district_id
WHERE T3.A3 = 'west Bohemia' AND T1.k_symbol = 'DUCHOD' AND STRFTIME('%Y', T1.date) = '1997'What was the total amount of old-age pension transactions in 1997 for accounts in the region ‘west Bohemia’?
I ended up with 34 families across the four tiers. Each family is a single generator with several variants inside it. They were written in three waves: a base set for each tier, an extension set covering shapes the base set had missed, and a final set added after I compared the pool’s SQL features against the test questions, which I describe at the end of this section.
Every generated query is executed before it can enter the pool, and it is kept only if it passes a fixed set of filters.
Pilot. Before scaling up, I generated 200 rows, 50 per tier, and read 80 of them myself. That alone turned up three generator bugs. The legend had no table descriptions, so questions about permanent orders came out as questions about orders. One tier 4 template could return a NULL row, and described a decrease as an increase. One tier 2 template about average salary was ambiguous between the per loan and per district readings. I fixed all three before running the full generation.
Model written SQL. Templates guarantee correctness but offer limited variety, so I also tried having Claude Haiku write SQL directly from questions for the harder tiers, hoping to cover shapes the templates did not. Between half and ninety percent of what it wrote executed, which looked promising at first. But when I audited a sample, executing each query and checking the result against the question, only 72 percent was actually correct. A plausible looking gold query that is wrong is worse than no gold at all, because the policy learns the mistake and the reward confirms it. I dropped all 804 rows and kept only the templates.
Rewriting the English. Template questions are stilted and all sound alike, and that uniformity is its own kind of leakage, since a model could learn the template grammar instead of the task. To break it, I had Haiku rewrite each canonical question in one of six registers: formal, concise, interrogative, imperative, colloquial, or descriptive. The rewriting uses the question synthesis prompt published with OmniSQL, with my legend supplied as the column documentation. In the imperative register, the tier 3 example above became:
Could you please calculate the total amount of all old-age pension transactions that took place in 1997 for accounts located in the west Bohemia region?
My first design filtered the rewrites by whether Haiku could solve them from the rewritten question alone. I noticed only after building it that this would cap the training set at Haiku’s own ability. I replaced it with a reading check, in which a separate pass scores each rewrite on six criteria using a four level scale, and a row is kept only if the four gating criteria are all rated Good or Excellent. I kept the solvability signal as a difficulty tag and never used it as a filter.
Audits. Correct by construction turned out to be a weaker guarantee than it sounds. I ran three rounds of audits in which eight agents each executed 25 generated rows against the database and independently compared the results to the English, and I merged their verdicts so that only findings agreed on by two or more agents counted. The rounds found 18, then 8, then 7 template bugs. I turned every bug class into a mechanical check that now runs on every generated row.
Matching the test set. Finally, I compared the SQL features of the pool against the 106 test questions. Three kinds of question that BIRD asks often were missing from the templates: lookups that chain three or four joins to reach a single fact, “what percentage of” questions, and second or third highest rankings. I added families for each, chain_lookup, chain_count, percentage, and nth in the inventory above, built from the database’s own entities and phrasings rather than from any test question. A leak filter then removed any row whose result set or SQL skeleton matched one of the 106.
The first run trained on 2,252 rows: 420 in tier 1, 771 in tier 2, 661 in tier 3, and 400 in tier 4.
Result
The chart below is the test accuracy every 25 steps: the fraction of the 106 human questions whose query returned the gold result, one sampled reply per question, without hints and with them. The dashed lines are Claude Sonnet 5 with no training, scored the same two ways.
Without hints the model went from 9 percent to 34 percent. With hints, from 28 to 46. Sonnet scores 53 and 78.
The endpoints are less interesting than the shape. Everything the no hint score gained, it gained by step 50, and it then sat between 31 and 36 for two hundred more steps. Nine evaluations in a row inside a five point band is not noise. My first assumption was that the model was still improving on its training data and the test set had simply stopped rewarding that, so I plotted the two together.
The mustard line is the reward on the training pool at every step, the fraction of the 384 sampled replies that matched gold, with the raw values faint behind a ten step average. It tracks the test curve almost exactly: a climb to about 35 percent by step 50, then a slow drift to 38 by the end. The model was not mastering its own training data either. Sixty percent of the pool, questions whose gold query I had generated and verified, was still being answered wrong at the end of training. Whatever was blocking the test questions was blocking the training questions too, and the second run’s version of this chart, later in the post, is the clearest single picture of what changed.
BIRD labels each of its questions simple, moderate, or challenging. Splitting the test accuracy by that label, at every evaluation, with and without hints, shows where the flat line comes from.
The seven challenging questions were never solved without hints. The 37 moderate ones crawled into the twenties by step 50 and wandered there. The plateau is the 62 simple questions: a jump to 48 percent by step 50, then nothing. For the second half of training the model was failing more than half of the questions BIRD considers easy.
Comparing the two conditions separates two kinds of failure. At the final step, hints lift the moderate questions from 24 to 46 percent, so those failures are largely vocabulary, the model not knowing which column or value a phrase refers to. The simple questions move only from 44 to 52. Whatever is wrong with the thirty or so simple questions the model cannot solve, telling it which column to use does not fix it.
A model that has run out of capacity does not stall on sixty percent of its own training pool, so something more specific was going on. I went through the failures one at a time before deciding what to change, and that is where run 2 starts.
Run 2
Post mortem
Two things run 1 clearly did learn set the baseline for reading its failures. Execution errors, replies that were not even valid queries against this schema, fell from 52 percent of answers to 17 percent, and the Czech values were learned from the sample rows alone: at step 0 the model wrote frequency = 'POPLATEK TYDENNE', a guess at the spelling, and by the end it wrote POPLATEK TYDNE. So the failures that remained were not about syntax, and not about values that appear in the sample rows.
I read all 70 answers the model got wrong at step 250 and tracked every one of the 106 questions across the nine evaluations from step 50 onward. 47 questions were never solved in any of them. My first guess was that the model could not infer the coded columns, so I checked whether those 47 were solved when the hint was present. 38 of them were not. Whatever was blocking them, it was not the legend.
Read by hand, the 47 sort into a small number of causes.
| Cause | Questions | Example |
|---|---|---|
| A join path the training pool never contained | ~15 | trans.client_id and disp.district_id, columns the model invented because no training question joined clients to transactions |
| BIRD’s phrasings, absent from the pool | ~12 | “eligible for loans” means disposition type OWNER; “high level card” means gold |
| Capitalization | 3 | the question says North Bohemia, the database stores north Bohemia |
| Output column conventions | ~8 | “which district” expects the name, the model returned the id |
| Coded column confusion | ~6 | A10 for salary instead of A11 |
| The verifier’s row cap | 5 | the gold result has more than 1,000 rows, so no answer could score |
The pool had 2,252 questions and not one of them joined the transaction table through a client. Every question that needed the path from a person to their transactions had to go client, then disp, then account, then trans, and the model had never once been rewarded for producing it. Instead it invented one, a client_id on the transaction table or a district_id on the disposition table, and the query failed to execute. That was the block on the simple questions. They were simple in SQL terms and impossible in terms of what the pool taught.
The row cap was my mistake. I had capped results at 1,000 rows to keep runaway queries from slowing the trainer, without checking whether any gold query exceeded it. Five did. One of them had the model’s query identical to the gold in every single evaluation, scoring zero each time.
Two other checks shaped what I did next. I had assumed the unsolved questions were the ones with more joins, and they were not: the never solved questions averaged 2.55 tables and the always solved ones 2.41. And of the roughly 100 wrong answers that did execute, 71 returned a result with nothing in common with the gold. There was nothing for a partial credit reward to work with, which settled a question I had been considering, and it pointed at the same thing as the missing join path: when the model takes the wrong path through the schema, the whole answer is wrong.
This also explained why more steps could never have helped. GRPO learns from the difference between replies in a group, so a question the model gets wrong all eight times contributes nothing, and prime-rl drops those groups before they reach the trainer. Every question needing the client to transaction path was in that state for the entire run. It was never going to see a gradient for it. The fixes had to change what the model could sometimes get right, which means the prompt and the pool.
Prompt and data
The training recipe itself, the model, the adapter rank, the batch, the learning rate, the number of steps, stayed identical to run 1. I wanted the second run to be about the environment, so that whatever changed could be attributed to the prompt, the data, and the reward.
Prompt. The legend went into the prompt: what each coded district column means and what every Czech value stands for. Withholding it in run 1 had been a deliberate bet, and the failure read settled it. The knowledge that makes a specialized model valuable is the schema structure and the phrasings, which no prompt supplies, rather than sixteen column codes that can simply be provided. So the legend became part of the input, as it is for every BIRD system. With it went the join structure written out in plain sentences, one per foreign key and one saying that clients reach accounts only through disp, since the CREATE TABLE statements had that information and a 3B model plainly could not use it in that form. The exact stored values of every categorical column went in too, so that capitalization is given rather than guessed, along with a line on how years are compared and an instruction borrowed from the small model text to SQL papers to only output the information that is asked.
The prompt is about 2,800 tokens, up from 1,900. The dropdown shows the parts that are new, with the schema itself omitted since it is unchanged from run 1.
Data. New template families, derived from the schema graph rather than from the failures, so that the fix is the database’s structure and not the test set. For every foreign key path at one, two, and three hops there is a family of plain lookups with a single filter and no aggregate: the accounts a client owns, the transactions on those accounts, the district of the account a card was issued on. The idea is that the model has to produce the disp path in its easiest form before it can learn the versions with aggregates and conditions stacked on top. Alongside those, a family for date phrasings with BIRD’s year semantics, one for output conventions, and later one for rankings through a join, which the pool had at half BIRD’s rate. The pool grew to 3,246 rows, through the same English rewrite and reading check as before, and three more audit rounds found a single bug, a code that means two different things in two columns.
Verifier. I removed the rule that scored any result over 1,000 rows as zero. It had been there to keep runaway queries from slowing the trainer, but five of the test questions have gold results larger than that, so the rule made them impossible rather than protecting anything. BIRD’s own execution accuracy metric compares full result sets with no cap, and the verifier now does the same.
Evaluation. Four sampled replies per question instead of one, since a single reply on 106 questions moves by six points from noise and the run 1 curve had been hard to read. And a third version of the test set. In 2024, Wretblad et al. went through these exact 106 questions by hand and found problems in 52 of them: 23 with spelling or grammar errors, 17 that were ambiguous, 22 whose gold SQL was wrong, 7 where the question’s capitalization did not match the stored values, and a few others. They published a corrected version with the questions and the gold queries fixed. I had picked this database before finding that paper, so from run 2 on I evaluate on their corrected 106 as well as the original, and treat the corrected number as the one that measures the model rather than the annotators.
Pilot tests
Two questions I did not want to settle by intuition were whether to add a supervised warm up before RL, and whether to let the model reason before answering. Both are standard. The two small model papers closest to this project, SLM-SQL and FINER-SQL, do both, and their models reach the high sixties on full BIRD. But both cost something. A supervised stage narrows the distribution that RL then explores from, and a thinking format makes every reply several times longer. So before renting the training pod I ran a gate: the untrained model, eight replies per question, under each candidate configuration, scored with the training verifier so the numbers would mean what the run’s numbers mean. It took half an hour on one consumer GPU.
| Condition | Mean accuracy | Solved at least once in 8 | Mixed groups |
|---|---|---|---|
| Run 1 prompt, 106 questions | 6.7 | 25 | 25 |
| Run 2 prompt, 106 questions | 20.3 | 42 | 40 |
| Run 2 prompt with hints | 21.9 | 51 | 51 |
| Run 2 prompt, think then answer | 15.4 | 52 | 51 |
| New hop families, run 2 prompt | 42.1 | 85 | 83 |
Three results. The prompt alone tripled the untrained model, from 6.7 to 20.3, and it absorbed nearly the whole hint gap: hints had been worth 19 points on top of the run 1 prompt and were worth under 2 on top of the new one. The hop families had mixed outcomes on 83 percent of their questions, which is the regime GRPO learns from, so the model could climb them without a supervised stage and I skipped it. And the thinking format lost. It started five points lower per reply, and 38 percent of its replies never closed the thinking tag. It did solve more questions at least once in eight, which is a real point in its favor, but I had committed in advance to a rule of at least as good per reply, and it was not.
Result
The chart below is every saved checkpoint scored after the run on the original 106 with and without hints and on the corrected 106, eight replies per question, with run 1 shown faintly for comparison.
The final checkpoint scored 50 without hints, 52 with, and 58 on the corrected set. The best checkpoints were 175 and 225 rather than 250, at 51 without hints, 55 with, and 59 on the corrected set, all within noise of each other. Sonnet 5 zero shot, with the run 1 prompt, is 53 and 78.
Almost all of the gain came in the first fifty steps again, and this time the training pool shows why.
In run 1 the pool and the test set plateaued together, at roughly the same value. Here they separate. Reward on the pool kept climbing until about step 150 and saturated at 93 percent, which is close to the fraction of the pool that is solvable at all, while the test set stopped at step 50. Everything the pool could teach about this schema had transferred by then. The remaining hundred steps taught the model more of the pool, and none of it moved the test number. In run 1 the model could not learn its pool; in run 2 it learned the pool completely, and the test set stopped anyway.
The sweep also showed what training past that point does to the model. The fraction of test questions solved at least once in eight replies peaked at step 25 and drifted down while the mean rose. The policy was getting more consistent and slightly less diverse, which is what RL at low entropy does.
Analysis
Did it learn the database
How do we know the model learned the database rather than the test? The training questions are template generated and the test questions are human written, no test question or a paraphrase of one is in the pool, and the reward never touches a test question. So a gain on the 106 is a gain on questions of a kind the model was never trained on. The sweep shows what kind of gain it was, comparing the untrained model to the step 225 checkpoint with eight replies per question.
The untrained model solved one question reliably; step 225 solves 42 reliably. Twenty one questions moved out of never solved, and of the questions the untrained model solved at least sometimes, two regressed. Replies that failed to execute at all, invented columns and bad joins, went from nearly half to under a tenth.
The fourteen questions that went from never solved to reliably solved show what was learned. Two of them, with the untrained model’s answer and the step 225 answer.
Who placed the order with the id 32423? The untrained model wrote a four table join through trans and card and referenced card.account_id, a column that does not exist. Step 225:
SELECT client.client_id
FROM client
JOIN disp ON client.client_id = disp.client_id
JOIN `order` ON disp.account_id = `order`.account_id
WHERE `order`.order_id = 32423How much, in total, did client number 617 pay for all of the transactions in 1998? The untrained model wrote WHERE account_id = 617, treating a client id as an account id. Step 225 goes through disp to find the client’s accounts and sums the transactions on those.
Both are the client to disp to account path, the one no training question in run 1 contained. What the model learned is the structure of the database, which tables connect to which and through what, and that is exactly what the hop families were built to teach.
Analyzing the errors
At step 100, 45 of the 106 questions were still wrong in all four sampled replies, and 33 of those had been wrong in every evaluation since step 25. I read them and sorted them by what would have to change for the model to get them right.
| What would fix it | Questions | Example |
|---|---|---|
| A definition my legend does not contain | 14 | “eligible for loans” means disposition type OWNER in BIRD’s convention; the model reads it as having a loan and joins the loan table. “Running contract” means status C or D; the model uses only C. |
| A correct gold query | ~10 | the question says 1998 and the gold query says 1997; a question about 2021 on a database that ends in 1998 |
| The right output columns | 6 | the model returns a client’s gender when the question asks who, or every column when it asks for the transaction id |
| A join the model still invents | 6 | account.client_id, district.A11 without joining district; down from 18 such errors in run 1 |
| Aggregate logic the model cannot yet do | 4 | growth rates between two years, conditional sums inside a percentage |
| Nothing available | 5 | questions with more than one reasonable reading, where two published corrections give different answers |
The largest group is the fourteen phrasings. They are all ordinary English for concepts the database has a code for, and the templates never used them, so the model never learned that “eligible for loans” and “owner” mean the same thing here. The pool phrases each concept exactly one way. A wider spread of phrasings for the same concepts would close most of those fourteen.
One result I found strange was that the hints did not help. In run 1 they were worth up to 19 points; in run 2 the with hints score sits a few points above no hints and at some checkpoints below it, even though a hint literally names the column or value the question needs. I think I know why. The model was never trained with hints. Not one prompt in the pool contains a hint line, so during training it never once saw a sentence like “A3 contains the data of region” and never had a reason to read one. At evaluation a hint is unfamiliar text tacked onto the question, and a policy sharpened on hint free prompts does not know to lean on it. Hints helped in run 1 because they filled a vocabulary gap the model had no other way to fill. In run 2 the legend fills that gap, and what is left in the table above is mostly not the kind of failure a hint fixes. Sonnet, which reads instructions natively, gains 25 points from the same hints. The fix is to put hints in some of the training prompts so the model learns what they are for.
The benchmark
About a third of the questions the model still gets wrong are questions no model could get right, because the published answer is wrong or because the question has more than one reasonable reading. This section is the evidence.
Two corrections. After run 1 it was clear that many of the published gold queries were wrong, so I looked for corrected versions and found two, built in different ways.
| Wretblad et al., 2024 | Arcwise and UIUC, 2026 | |
|---|---|---|
| What they went through | the 106 financial questions | BIRD’s 500 question mini dev set, 30 of them on financial |
| How | by hand | a database probing agent, then expert review |
| What they found | problems in 52 of 106: 23 spelling or grammar, 17 ambiguous, 22 wrong gold SQL, 7 capitalization | errors in 52.8 percent |
| What they published | corrected questions and gold for all 106 | corrected gold for the 30 |
Three answer keys. The 30 questions Arcwise corrected are among the harder ones, 20 moderate and all 7 challenging, so scores on them run well below the full set whichever gold is used. The clean comparison is the step 225 checkpoint on exactly those 30 questions, scored against each of the three sets of gold queries.
| Gold queries from | Mean accuracy on the 30 | Solved at least once in 8 |
|---|---|---|
| the original BIRD annotators | 36 | 13 |
| Wretblad’s correction | 43 | 17 |
| Arcwise’s correction | 44 | 18 |
Same model, same questions, and the corrected keys score it 6 to 8 points higher. That difference is the size of the annotation problem on these questions. For comparison, the other 76 questions score 56.
Where they disagree. Since both groups had corrected the same 30 questions, I ran both groups’ gold queries against the database and compared the results. On 17 of the 30 they return the same answer. On 13 they do not. Two teams of experts, each trying to write the right query for the same English sentence, wrote queries that return different results.
| Question | Wretblad’s query returns | Arcwise’s query returns |
|---|---|---|
| Which are the top ten withdrawals (non-credit card) by district names for the month of January 1996? | district ids | district names |
| List out the account numbers of clients who are youngest and have highest average salary | a client id, joining through the client’s district | an account id, joining through disp |
Those questions do not have a single right answer. A model scored against either key is marked wrong for reading the sentence the other way, and no correction to the key can change that, because the problem is the sentence. Part of this benchmark therefore measures whether the model reads a question the way one particular annotator did.
At frontier scale. A week before this project started, Zhu, Jin, Choi, and Kang, the UIUC group behind the second correction, published with Thinking Machines a paper on RL for text to SQL. They fine tuned Kimi K2.6 with LoRA and reached 91 percent on the corrected mini dev set, above every frontier model and level with a human proxy. They checked 2,500 of BIRD’s training examples, found that 52 percent had wrong gold SQL, rebuilt the training set by hand, and that alone was their largest gain. Two other findings match this project. A third of their execution match rewards went to queries that were not equivalent to the gold, which I saw on the first day when a query that skipped a join matched anyway. And a quarter of their failures came from the model ignoring the supplied hint, which is the with hints result in run 2. The same three problems, at 3B and at a trillion parameters.
What I would change
Four things I would do differently.
More phrasings. Fourteen of the 45 unsolved questions come down to ordinary English the templates never used for concepts the database has a code for: “eligible for loans,” “running contract,” “high level card.” Each template phrases its concept one way, and the rewriting pass changes the register of a sentence but not the words for the concept inside it. Generating several phrasings per concept, the way a person might actually ask, is a change to the generator and to the legend it reads from, and it costs nothing to train.
Hints in training. The model never saw a hint during training, so it never learned to use one, and the with hints score measures almost nothing as a result. Zhu et al. go further and reward the model for honoring each hint as a constraint on the query, which is the better design because it cannot be satisfied by guessing.
Check the rewards. A third of Zhu et al.’s execution match rewards went to queries that were not equivalent to the gold, and I saw the same thing on day one, a query that skipped a join and matched anyway because every card in the database happens to be on an owner’s account. The cheap version of their check is to re-execute every rewarded query on a perturbed copy of the database and count how many stop matching. I have not run it, and until I do the 93 percent training pool number carries an asterisk.
No caps. The row cap made five questions unwinnable for a whole run. BIRD’s metric has no cap, and now neither does mine.
Beyond those, the frontier comparison should be rerun with the run 2 prompt, since the Sonnet numbers above were measured with the run 1 prompt and are a floor on what it would score. A third run would take roughly half the GPU hours of the second, and the first two items above are where I would spend it.
Code and weights
The trained adapters, one from every 25 steps of run 2, and the evaluation traces are on Hugging Face at Induction/qwen2.5-coder-3b-t2s-v2-lora. The environment, the data pipeline, the measurement scripts, the configs, and the per step results are on GitHub at Induction1/rl/text2sql.