I Fine-Tuned a 9B Model Into a Godot 4 Specialist, and the Judge Was the Engine Itself

Ask a small local LLM for Godot code and watch what comes back: yield instead of await, export var instead of @export, signals connected with strings, a File class that hasn’t existed for years. It compiles in the model’s imagination and nowhere else. When I measured it properly, stock Qwen3-8B produced parse-valid Godot 4 code on 1 of 10 basic tasks. One.

The models aren’t stupid. Their training data is contaminated. Godot 4.0 broke compatibility with a decade of tutorials, Stack Overflow answers and open-source repos, and that pre-4.0 corpus outweighs everything written since. The model has seen far more Godot 3 than Godot 4, so that’s what it writes — confidently.

That diagnosis matters, because it means this is a distribution problem, not a capability problem — and shifting a distribution is exactly what fine-tuning is good at. So I spent a summer of evenings finding out how far a solo dev with a mid-range GPU and ~$150 could push it.

The result is Estragon-9B: a fine-tune of Qwen3.5-9B that scores 248/300 (82.7%) on a 300-task GDScript benchmark where every answer is judged by an actual headless Godot binary. The base model I started the project with scores 115/300 on the same tasks. The weights, the benchmark, the training pipeline and every decision record — including the failed experiments — are public.

About the name: Godot (the engine) is named after Beckett’s Waiting for Godot, as a joke that it would never be finished. Estragon is one of the two tramps who wait for a Godot who never arrives. This model is the reference-to-the-reference, with the inversion as the point: Estragon stopped waiting.

Model: huggingface.co/Blugart/estragon-9b · GGUF quants
Everything else: github.com/blugart-dev/estragon

The one decision that carried the whole project

Before touching any training code, I built the judge: a pinned, headless Godot 4.7 binary that parses — and for harder tasks, actually runs — every piece of GDScript in the project. Not an LLM grading another LLM’s homework. The engine itself.

Everything downstream leans on it:

No training example enters the dataset without parsing in headless Godot. All ~20,000 pairs, no exceptions, including the hand-written ones.
The benchmark is engine-verified. A task passes only if the generated file parses and its runtime assertions hold: a test scene instantiates the node, calls the methods, checks the state, reports back.
Reinforcement learning got a reward signal that can’t be sweet-talked. More on that below.

If you take one transferable idea from this post, take this one. If your domain has a compiler, an interpreter or an engine with a CLI, you have a verifier — and a verifier turns “the model seems better” into a number you can defend. It’s also what kept the project honest in a second way I didn’t fully appreciate at the start: I never had to trust generated code or generated training data. Only the engine’s verdict on it.

~20,000 validated pairs, five streams

Everything targets Godot 4.7, pinned to the exact stable binary so “correct” has a fixed meaning. The mix:

Stream	Share	What it is
Docs-derived	~30%	The official class-reference XML, parsed into Q&A about methods, signals, properties
Real code	~20%	38 MIT/Apache Godot 4 repos, turned into explain / implement / fill-in-the-middle tasks
Synthetic	~30%	Claude-generated tasks grounded in doc excerpts or real code — every output through the judge
Migration	~8%	Valid Godot 4 code mechanically corrupted into Godot 3 idioms, trained corrupt→fixed
General	~15%	A slice of dolly-15k so the model doesn’t forget how to be an assistant

The migration stream is my favorite, and it cost nothing. I wrote a corruptor that rewrites correct Godot 4 code backwards into Godot 3 idioms — await becomes yield, @export becomes export var, Callable connections become string connections — and the model trains on fixing it. It attacks the core failure mode directly, and the “answers” are real, already-validated code.

One rule I set early and am glad I kept: the eval tasks are sacred. Nothing from the benchmark may appear in training data, in generation prompts, or even pasted into a data-generation context. Decontamination runs mechanically (shingle-level text overlap) at every dataset assembly, and it caught real overlap more than once — 140 pairs dropped the first time it ran with teeth.

The $2.60 training run

My own GPU is an RTX 3060 Ti with 8 GB of VRAM — the same card I run local agents on. I measured local QLoRA training honestly: 766 seconds per step. Technically possible, practically absurd. A rented RTX 4090 did the full first run in ~3.5 hours for $2.60.

That set the pattern for the whole project: my machine authors data and validates everything; rented pods train and evaluate; every pod gets deleted the same day. Total GPU spend across every experiment, including all the failed ones: about $50.

The first fine-tune took the 10-task smoke test from 1/10 to 8/10, with zero Godot 3 idioms. On the first real 100-task benchmark: base 39/100, fine-tune 74/100. (Claude Opus scored 100/100 on the same benchmark — a useful ceiling to keep me humble about what a 9B model is.)

Building a benchmark that can’t lie to me

The 100-task eval had a problem I only understood after wasting a cycle on it: with n=100, the 95% confidence interval is about ±9 tasks. I was trying to steer training with a compass that swings nine degrees at rest. One full data-iteration cycle produced “71 vs 74” — statistically a tie, and I couldn’t tell whether anything had improved.

So: measurement first. The benchmark grew to 300 tasks across 10 categories (signals, tweens, file I/O, physics, node lifecycle…), each with a reference solution and a self-test, each judged by the engine. That shrinks the noise floor enough to read a 5-task delta. Every training experiment after that point ran against a fixed promotion gate: beat the incumbent by ≥5 tasks on the same rented GPU, same day, plus behavioral checks — or don’t ship.

Two measurement lessons that cost me real money to learn:

Re-run your baseline on the same hardware, same day. Different GPU, different library version, different day — I saw the same adapter move a few tasks. The gate compares against a fresh baseline, not a remembered number.
The system prompt is part of the model. Ablations showed the deployment prompt is worth about +15 tasks — and, beautifully, the base model scores worse with that same prompt than without it. The instruction tuning is what makes the prompt pay. So the prompt ships with the weights and every published number includes it.

The chapter where everything fails

This is the part most model-release posts skip, and it’s the part I most want on the record. After supervised fine-tuning plateaued, I ran four training cycles against that promotion gate. Three failed.

DPO, cycle 1: failed, badly. Direct Preference Optimization — generate several candidates per prompt, have the judge rank them, train on the preferences. Result: 222/300, versus its own starting point at 229. The model got worse. The autopsy found my ranking rules had a hidden length bias, and the model learned to compress code until correctness broke.

DPO, cycle 2: failed, differently. I removed the length pressure and anchored preferences in 154 new runtime-verified training tasks. Result: +1. A statistical tie — while the style noise the ranking was supposed to suppress came back. Two failures with two different causes. I closed the DPO line and wrote up why.

GRPO, cycle 1: failed by exactly one task. GRPO is reinforcement learning with the judge as the reward function: the model generates, the engine scores, pass = reward. The on-pool learning curve was beautiful — batch pass rate 51% → 68% during training — but on the held-out benchmark it landed at +4 against a gate of +5. The autopsy found a transfer gap: the model genuinely improved at the file I/O tasks it trained on while getting worse at file I/O on the benchmark. A nine-prompt task family can’t teach a topic. Noted.

GRPO, cycle 2: passed, convincingly. I expanded the training pool with 100 new tasks aimed at exactly what the autopsy said was missing. Result: +13 (240 vs 227), the first statistically significant delta of the entire phase (McNemar p=0.019), with the file I/O regression healed.

The meta-lesson sits above any single technique: most cycles fail, and the gate is the product. A fixed threshold, measured same-pod against a re-run baseline, is what kept three consecutive failures from shipping under deadline-brain (“it’s probably fine, the curve looked good”). The failed cycles all have their own post-mortems in the repo’s decision records, because the autopsies are where the actual learning happened.

Reward hacking is real, and it’s creative

Training against a judge means the model will eventually try to cheat the judge. My defenses: a canary set of already-solved tasks (if those regress, the model is gaming something), a style-noise metric, and degeneracy detectors. Over the project those caught four distinct flavors of degenerate output — including one late entry that cycled through variable declarations in a pattern specifically shaped to evade the first two detectors I’d written. Nothing hacked its way through a gate, but only because something was watching. If you RL against a verifier, budget for this.

Swapping the base model mid-project

Halfway through, Qwen3.5-9B came out and scored 219/300 on my benchmark out of the box — within reach of my best fine-tune at the time. Sunk-cost said keep going on the old lineage; the numbers said re-run the whole recipe on the new base. The numbers won:

Model	gdeval_v2	%
Qwen3-8B (original base)	115/300	38.3%
granite-4.1-8b	123/300	41.0%
phi-4 (14B)	146/300	48.7%
gemma-4-12B-it	211/300	70.3%
Qwen3.5-9B (new base)	219/300	73.0%
Estragon-9B (SFT + GRPO on it)	248/300	82.7%
Claude Opus 4.8 (the ceiling)	300/300	100%

The bake-off rows were a late addition for the release table, and they surprised me: phi-4 at 14B and gemma-4 at 12B both score below the raw 9B base I tuned. If you want small-model GDScript today, the Qwen3.5 lineage is simply where it’s at.

SFT alone on the new base tied my old champion. SFT + one GRPO cycle — run 007 of the project, now wearing the Estragon name — cleared the gate at +8.

Knowing when to stop

I tried one more GRPO cycle on top. It returned exactly +0. Five tasks fixed, five broken, perfect symmetry, p=1.0.

Diagnostics said the remaining headroom is real but out of this method’s reach: with 8 sampling attempts the model solves 285/300 — the knowledge is in the weights — but my training-task pool couldn’t convert sampling luck into greedy reliability anymore. That’s a clean stopping signal, and for once in my life I took it. The honest framing: the last 52 failures cluster in rare-API precision, long spec chains and timer/tween edge cases, and getting them needs new ideas, not more compute through the same pipe.

Shipping honestly: every quant measured

Two things in the release I haven’t seen done often enough:

Every quantization ran the full 300-task benchmark. Not “quantization quality loss is usually minimal” — numbers:

File	Size	Score	vs bf16
Q5_K_M	6.5 GB	249/300	+1 — statistically identical. The one to download.
Q8_0	9.5 GB	248/300	±0
Q4_K_M	5.6 GB	229/300	−19 — fits an 8 GB card, and now the price of that is a number

And a pre-release smoke test on the actual artifacts caught two shipping bugs. Running the published GGUF through Ollama on my own machine — the exact download-and-run path a user would take — surfaced a Modelfile parse error (one missing space) and a subtler one: Ollama’s Qwen3.5 runtime manages reasoning blocks itself, and Estragon is a non-thinking fine-tune. The flag matters, measurably (−15 tasks with thinking on):

ollama run hf.co/Blugart/estragon-9b-gguf:Q5_K_M --think=false \
  "A 2D platformer jump with coyote time, CharacterBody2D"

Test the thing you’re actually telling people to download, on the hardware you claim it runs on. Both bugs took an hour to find and would have been the first-user experience.

What it cost

What	Cost
Claude API (synthetic data, mostly batch)	~$100
Rented GPUs (every run, all experiments, export)	~$50
My own hardware	the 3060 Ti I already had
Wall clock	~9 days of evenings, plus this write-up

How this was made

I want to be precise here, because “AI-assisted” can mean anything. I used Claude throughout this project: it generated the synthetic training data (grounded in docs and real code), helped author benchmark tasks, wrote most of the pipeline code with me, and co-drafted the decision records. I made the calls, set the gates, paid the bills, and learned each piece by reviewing what got built — this was my first fine-tuning project, and using AI to help build an AI specialist was half the point of the experiment.

What makes me comfortable publishing the numbers anyway is the architecture: the verifier isn’t an AI. Every training example parsed in a real Godot binary. Every benchmark score is a count of programs that ran correctly in the engine, on held-out tasks mechanically decontaminated from the training set. The one thing in this project that was never generated is the judge’s verdict, and every claim above traces back to one.

Limitations, so you don’t have to discover them

Godot 4.7 only. It will not write Godot 3, and may “fix” valid Godot 3 code on principle. Untested against future 4.x releases.
GDScript only. No C#, no non-trivial shaders.
It’s a 9B specialist, not a general assistant. Frontier models are still simply better — Opus scores 300/300 on my own benchmark. If you have API access and no privacy constraints, use one. Estragon is for the local, offline, 8-GB-of-VRAM crowd.

What I’d tell you to steal

Find the verifier in your domain. Compiler, interpreter, engine CLI, schema validator — anything that turns “seems right” into pass/fail. It’s worth more than a bigger GPU.
Fix your measurement before optimizing. A ±9 noise floor ate one of my cycles whole. Size the benchmark to the deltas you need to read.
Set the promotion gate before the run, and let it kill your work. Mine killed three of five cycles. All three deserved it.
Autopsy failures in writing. The cycle that finally worked was aimed by the post-mortem of the one that failed by one task.
Smoke-test the shipped artifact on the hardware you name in the README.

Estragon stopped waiting. You can too: the model · the benchmark and everything else