What this study doesn't show

What the result covers

• One GPU, one server setup. Everything ran on one RTX 5060 Ti under vLLM 0.30.0, without CUDA graphs, with the prefix cache off, at most 64 requests in a batch and 4,096 tokens per step. The rerun noise belongs to that setup. Another GPU, CUDA graphs, another vLLM version or the prefix cache could each change it.
• Two compressed models, as deployed. The models differ at once in size, maker, training, compression and chat format. The main comparison compares two deployments, and nothing here attributes the gap to any one of those differences.
• What 541 prompts can show. The smallest gap this design reliably detects is 5.7 points for this pair, and about 6 for two models closer together. The interval describes prompts like IFEval's. IFEval's prompts aren't a random sample of anything, so the result doesn't carry over to other kinds of instruction-following.
• Rerun noise was measured mostly on Qwen. Only at 32 requests at a time was it measured on both models. The loads of 1, 8 and 64, and batch-invariant mode, are Qwen only.
• Gemma and Qwen weren't batched the same way. Gemma's server preempted requests 857 to 866 times per pass, and Qwen's never did. The difference in their rerun noise is reported, and nothing here explains it.
• One request at a time used 200 prompts. The probe predicted 2.0005 hours against a 2-hour cutoff, so those comparisons use a different prompt set from the 541-prompt batched passes.
• A rerun means a second pass on the same server. A new run on a freshly started server, with the model loaded again, wasn't measured on its own. The one comparison across servers also changes the load. With two passes per setting, each rerun number rests on one comparison.
• Prompt order was fixed. Every pass sent the prompts in the same order. A shuffled order might batch differently, and that wasn't tested.
• The GPU window isn't recorded. The launch script claimed it before each run, but neither the window nor whatever else was on the GPU is in the run records. The general GPU queue also counted each run at 1,024 MiB of GPU memory instead of the 12,452 it declared, because memory declarations didn't reach the queue until the day after these runs.
• The grader is the official one, bugs included. Prompts 1122 and 1129 are checked for the letter y instead of # and !. Prompt 2785 asks for one placeholder while the grader requires three. A text the language detector can't classify passes the language test. None of these is corrected, so the scores stay comparable with other IFEval results, and all of them are disclosed (part 5).
• Batch-invariant mode may not have turned on. vLLM's source history shows no line confirming the mode even when it's active, so the server gives no signal either way, and the rule fixed in advance gives no conclusion (part 4).
• The prompts as the models saw them aren't stored. The runs don't record each prompt after the chat format was applied. It can be rebuilt from pinned inputs, but that hasn't been done. No number depends on it.

How far the evidence goes

Every measured number here comes from one results file, written by version 2 of the analysis code, or is quoted from the run records. The running-interval numbers under Fig. 2 of the overview are the exception: this site's scripts compute them from the main runs' per-prompt results, exported from the same runs and checked to reproduce that file's main comparison exactly. The containers are pinned by digest and the weights and dataset by version and hash, and the analysis code writes byte-identical output each time, so the results can be regenerated from the stored responses. The code and results file live in a private repository on my homelab, and the raw responses in its object store.

Pinned inputs