Making small models reliable for repetitive tasks, without expensive training runs.

The form changes what the model has to generate. That alone saves money: output tokens are billed at a premium, often 3-5× the cost of input tokens, so cutting what the model produces is the highest-leverage cost reduction available. Same model, same task, fewer tokens, lower bill.

The more interesting consequence is what becomes possible after the task is simplified. Generating a paragraph of free-form English is a different problem than filling three string fields. Filling out a form is bounded, structurally constrained, and well within the competence of a much smaller model.

Once the form is in place, the question shifts from "how do we prompt the frontier model better" to "do we still need the frontier model at all." Sonnet is priced at $18/M blended tokens; Qwen3-8B serves at $0.20/M. The price gap is roughly 90×.

We already know the small model is cheaper, so the problem shifts to whether we can make it reliable enough for production. Prefill is the first step of the answer and the first run on the optimization ladder.

Small models do not win everywhere. They win when the work is bounded: known environment, constrained actions, structured output, and an objective scorer. That is why this benchmark uses workflow execution instead of an open-ended chat task.

Raw Qwen3-8B was already close on the action-level scorer: 0.9560. But it was a bad production route. It used 1.39M eval tokens and had 32.6s p50 latency. The model could often do the work, but it spent too much time and too many tokens getting there.

That distinction matters. A model can know enough to complete a bounded task and still be unusable because it rambles, thinks in the wrong mode, emits awkward structure, or burns tokens before taking the action.

The first useful intervention was JSON prefill. It does not change weights. It starts the model's answer in the structure the system needs. Raw Qwen3-8B without prefill scored 0.9560 and used 1.39M eval tokens. With JSON prefill, the same 8B model scored 0.9667 and used 74k eval tokens.

That is the key result: 18.8× fewer eval tokens, a small score lift, and no training. The first big win did not come from making the model smarter. It came from putting the model in the output mode the system needed.

Qwen3 also supports `/no_think`, which tells the model not to spend tokens on extended thinking before producing the answer. Raw `/no_think` scored 0.9598 and used 38k eval tokens: 36.5× fewer tokens than raw no-prefill with near-equivalent quality.

This is not a claim that reasoning is bad. It is a claim that this specific Operations slice did not need long visible deliberation for every action.

Fine-tuning still mattered, but not as the whole story. SFT + JSON prefill did not beat raw JSON prefill in aggregate. The more useful role for SFT was sparse repair: raw `/no_think` scored 0.9598; SFT + `/no_think` reached 0.9733 while staying around 39k eval tokens.

The right order is: do not fine-tune first. Find the cheapest control that works, then train only on the residual failures.

Same 30-task Operations holdout, 25 samples per task for the methodology rows. The Fireworks row is the production-style serving validation at 90 trajectories.

The holdout spans scheduling, compliance, facilities, legal documents, inventory, and cross-system execution across workplace tools.

The remaining misses were specific, not random. Raw `/no_think` repeatedly missed lease archival, Jira/Confluence incident, and Pipefy vendor onboarding. SFT mostly repaired the incident workflow and partially improved lease archival. Pipefy vendor onboarding remains the stable repair target.

That is exactly the signal understudy wants: not just a score, but a repair map. The next training batch should focus on the residual workflow clusters instead of blindly adding more examples.

The easiest way to overfit an eval is to report one score and stop. Here, the strongest signal was not that SFT won everywhere. It did not. JSON prefill reproduced the SFT+prefill aggregate score without changing weights, SFT produced a modest repair rather than a magical rewrite, and the remaining misses stayed clustered enough to become a repair map.

The production-style comparison is Sonnet API versus the optimized Fireworks 8B route. On the same 90-trajectory scale, Sonnet scored 1.0000 with 1.935s p50 latency and about $0.040 measured token/eval cost. The Fireworks route scored 0.9630 with 369ms p50 latency and $0.0066 token cost.

That is the clean cost-and-latency story: roughly 5.2× lower p50 latency and 6.0× lower measured token cost, while preserving most of the frontier score on this bounded slice.

The Fireworks route used 33,111 measured tokens across 90 trajectories. At the Qwen3-8B serving basis of $0.20/M blended tokens, that is $0.0066 in model-token cost for the validation slice.

The normalized work-per-dollar number is still directionally useful, but the direct cost and latency are the cleaner public claim. The short-lived Fireworks deployment had a loaded validation cost of about $1.56 because cold-start GPU time dominated the tiny token bill. That should stay separate from steady-state serving economics.

We do not need the small model to beat the frontier. We need to find the work where the frontier is overkill, then make the small model reliable enough to take that work off the frontier path.

The ladder is the product: find bounded work, make the cheap model reliable, validate it on holdouts, and promote only the routes that survive. Sometimes that is prefill. Sometimes it is `/no_think`. Sometimes it is SFT. The point is not to train by default. The point is to run the ladder.