Frontier models write CAD from a spec. Give them a drawing and they fall apart.
Kimi K3 gets 95% of our held-out parts right from a text description. Hand it the same part as a four-view drawing sheet and it drops to 61.8%. That gap is narrow, it matters, and it can be graded exactly.
From a public dataset to a model that checks its own work
Public CAD data is noisy, so every stage either checks the data or removes a way for the result to be wrong.
- 1
Audit every reference program
Execute all 82,659 programs in CAD-Coder and check each resulting solid against the dimensions its own spec states. This is where we found the data was wrong.
- 2
Build clean splits
Hold 500 parts out by source part, then drop every training part whose geometry signature matches a benchmark part. 1,886 were removed this way.
- 3
Render each part as a drawing
Front, top and right at one shared scale, plus an isometric view, rendered from the part's own reference program so the drawing and the answer always agree. 30,260 sheets.
- 4
LoRA fine-tune Qwen3-VL-4B
Rank 64 on Baseten Training, adapters on the language model with the vision tower frozen, loss on the generated code only. Two epochs: 2.9 hours on one H100, then 55 minutes on four.
- 5
Serve from the training checkpoint
Merged weights are pulled straight out of the training job and served with vLLM. No copy step between training and serving.
- 6
Sample, then check its own work
Draw eight programs, render each candidate the way the input sheet was drawn, and keep whichever matches the drawing best. The verifier never sees the reference solid.
- 7
Grade by executing the geometry
Run the program in a sandbox and compare its solid to the reference by volumetric intersection over union. No model judges another model anywhere in this project.
Look at your data
We ran all 82,659 reference programs before training anything, and checked each solid against the size its own specification claimed.
of the official test split contradicts its own specification, on single-part rows where the stated size is checkable. The hand-checked train_high split came in at 2.3%. Anyone reporting accuracy against that test split is reporting partly on noise.
References apply the specs' stated rotations and translations inconsistently, so the metric aligns both solids first, taking the best overlap across the 24 axis rotations.
Alignment forgives placement, not shape or size. A part built 10% too large scores 0.75 and fails, well short of the 0.9 bar.
Some reference programs export files while being evaluated. The sandbox strips export calls, allows only three imports, and runs in a throwaway directory.
How a part is scored
A prediction is correct when its program runs and the solid it produces overlaps the reference by 0.9 or more. Everything else is a miss, including code that builds something plausible but wrong.
IoU = volume(A ∩ B) / volume(A ∪ B)
Exact boolean volumes from OpenCascade, not a mesh approximation and not a sampled estimate. We take the best value over all 24 axis-aligned rotations after centering both solids.
To pick between candidate programs at inference time, each one is built and re-rendered from the same three orthographic views as the input sheet. The score is silhouette overlap across those views, multiplied by how closely the candidate's bounding box matches the one stated in the prompt.
separating correct programs from incorrect ones across 689 real frontier-model answers spanning 237 parts. Given several answers for one part it picks a correct one 98% of the time, against 76% for a random pick. It is not specific to our model: the same verifier lifts Kimi K3 from 28% to 48% on hard parts.
Every lane on the same 497 parts
Bootstrap 95% confidence intervals on every row. A request the API never answered is excluded and listed, never counted as a wrong answer.
| Model | Correct | 95% CI | Simple | Medium | Complex | Multi | $ / 1K | p50 |
|---|---|---|---|---|---|---|---|---|
| Cadabra 4B, best of 8 | 80.7% | 77–84 | 95.6% | 61.3% | 39.0% | 58.1% | $1.19 | 2.3 s |
| Cadabra 4B, 1 sample | 74.8% | 71–79 | 92.5% | 52.1% | 25.4% | 53.5% | $0.22 | 2.0 s |
| GLM-5.3 Flash (high) | 66.2% | 62–70 | 85.9% | 37.0% | 18.6% | 39.5% | $1.00 | 3.7 s |
| Kimi K3 (high) | 61.8% | 58–66 | 81.5% | 30.3% | 18.6% | 39.5% | $36.82 | 10.4 s |
| Base model (untuned) | 14.7% | 11–18 | 19.4% | 8.4% | 1.7% | 16.3% | $1.38 | 20.3 s |
Tiers are the reference solid's face count: simple ≤ 6, medium 7 to 12, complex ≥ 13. Frontier lanes get two worked examples and high reasoning effort. Ours gets neither.
Is it memorising the benchmark?
The obvious objection to any fine-tune, so here is the evidence rather than an assurance.
The 500 benchmark parts are held out by source part, with zero shared, and every training part carrying a benchmark part's exact geometry signature was dropped. That removed 1,886 of them.
0.212 to 0.158 across the first epoch, 0.160 to 0.149 across the second, with token accuracy reaching 94.7%. Neither pass shows the upturn you would expect from overfitting.
| Model | Near-duplicate (57) | No near-duplicate (443) |
|---|---|---|
| Cadabra, 1 sample | 94.7% | 72.2% |
| Cadabra, best of 8 | 94.7% | 78.8% |
| Kimi K3 (high) | 73.7% | 60.3% |
| GLM-5.3 Flash (high) | 84.2% | 63.9% |
57 of 500 benchmark parts have a training part within 1% on every bounding-box axis with the same face and part count. None within 0.1%. Those parts are easier for every model, including ones that never saw our data.
is our margin over the best frontier model on the 443 parts with no near-duplicate in training, slightly wider than the +14.5 we hold overall. Removing the parts we could plausibly have memorised makes the result better, not worse.
What didn't work
The grader is an exact reward, so reinforcement learning on it looked obvious. It made the model worse.
We ran GRPO on top of the supervised model, 8 samples per sheet and 64 rollouts per step, for 60 steps on four H100s. Training reward rose from 0.89 to 1.2 and the share of correct rollouts went from 39% to 70%, so by its own training signal it was working.
Held-out accuracy fell anyway. We graded every saved checkpoint with the same harness, including the supervised checkpoint as a control, which reproduced its own score to within two parts. The loss lands almost entirely on medium-complexity parts: 56.3% down to 48.7% after ten steps, then flat near 42%.
We shipped the supervised model and published the checkpoint sweep, because a negative result with the evidence attached is worth more than quietly dropping the branch.
The data, and what it was built on
Everything derives from CAD-Coder, which is Apache-2.0 and itself derived from Text2CAD and DeepCAD.
| Set | Parts | What it is |
|---|---|---|
| Audited | 82,659 | Every reference program run and checked against its own spec |
| Benchmark, held out | 500 | 457 single-part, 43 multi-part, held out by source part and by geometry |
| Training, train_high | 5,698 | What remains after removing benchmark sources and geometries |
| Training, train_middle | 24,868 | Two batches, weighted toward medium, complex and multi-part |
| Sheets trained on | 30,260 | Plus 306 validation. 72% medium or complex, 35% multi-part |
- Model APIs for the Kimi K3 and GLM-5.3 Flash baselines, at high reasoning effort.
- Training Jobs for the LoRA fine-tune. One H100 for the first epoch, four with DDP for the second.
- Deployments pulling merged weights from the training job's checkpoints, served with vLLM.
- Switch, which ran Claude Code on open models while we built this.