How the study was set up

Every choice that could bend the result was committed as code before any run: the question, the formulas, the settings and which runs would count. Two later changes, listed at the end, changed two server settings before any counted run and filled in parts of the analysis.

The question

The study asks two things. Do Qwen2.5-7B and Gemma-4-12B score differently on IFEval, with an error bar that accounts for how IFEval is built? And how much of any difference could come from the inference server giving different answers on identical passes?

Only the first is a yes-or-no test.

The benchmark and the two models

IFEval is graded by a program, not by another model. Its 541 prompts carry 834 instructions between them: 305 prompts have one instruction, 179 have two and 57 have three, drawn from 25 kinds of instruction. Under strict grading, a prompt counts as right only if all its instructions are followed. The prompt file is pinned to one published version, and the grader is Google's reference code, copied into the project unmodified. Part 5 covers what it took to make that grader repeatable.

These are the two models already deployed on this GPU, so the study compares two deployments. They differ in size, maker, training, compression and chat format at once. Qwen's chat format also adds its own default system message when the caller sends none, and Gemma's doesn't. Isolating compression would mean comparing each model with its own uncompressed version, and isolating size would need two models that differ only in size. This study does neither.

What was fixed before any run

Where the methods come from. The estimators follow Evan Miller's Adding Error Bars to Evals (2024): the paired standard error for the gap between the models (its section 4.2), clustered standard errors for instructions that share a prompt (2.2), the split of variance between prompts and repeated answers that shows whether resampling helps (3.1), and a power analysis for sizing the next eval (5). The rerun measurements apply the same split to passes with randomness off on a batched server, which the paper doesn't cover.

The main comparison. For each prompt, take Gemma's result minus Qwen's: +1 if only Gemma got it right, −1 if only Qwen did, 0 if they agreed. The gap is the average of those numbers, its standard error is their standard deviation divided by √541, and the 95% interval follows from that. The models count as different only if the interval excludes zero. The data is the first pass of each model's main run (randomness off, 32 requests at a time), chosen before any run.

Reported beside it, but never used to decide: the interval you'd get by treating the two scores as unrelated, how closely the two models' results track each other prompt by prompt, how much rerun noise contributes, and an exact McNemar test, a standard check for paired right/wrong data, to confirm the interval's approximation holds.

The prompt is the unit. Each prompt counts as one independent observation, never each instruction, each sampled answer or each run. The interval describes how much the gap would move on another set of prompts like IFEval's. IFEval's prompts aren't a random sample of anything, and its near-duplicate prompts each count separately.

Five measurements, none of them a test:

• Instructions graded together. Instructions in one prompt share a response, so they can be followed or missed together, and an error bar that treats them as independent comes out too narrow. The study measures how much too narrow, and predicted little, because most prompts carry one instruction.
• Strict versus loose grading. How many more prompts count as right when the grader forgives formatting around an otherwise correct answer.
• Answers per prompt. With sampling randomness on and 8 answers per prompt, how the variance splits between prompts and between answers, and how many answers per prompt are worth collecting.
• Benchmark size. The smallest gap this design detects 4 times out of 5, and how many prompts it would take to detect 2 and 5 points.
• Rerun noise. Nine pairs of passes chosen in advance. For each: how many responses matched exactly, how many prompts flipped between right and wrong, how far the score moved, and how big the noise is next to the error bar.

How many chances to be wrong. Only the main comparison is a test. The results include 26 explicit 95% intervals, and by construction about one in twenty of them misses its true value. The nine score-change checks in the rerun measurements are reported as standard errors and aren't in that count.

Also fixed in advance: the models, the kinds of run and how many passes each makes; the nine rerun pairs; the number of prompts, answers and concurrent requests; a 1,200-second timeout and the retry rule; every sampling setting and server flag (a later change altered two of the flags); the fallback to a 200-prompt subset for passes one request at a time; and which run counts. The first run of each kind that clears its self-checks is the one used. A run that doesn't is listed and never read, and a second run that clears them would be reported as a reproduction without replacing the first.

The four kinds of run

Each run is one job on the homelab's general GPU queue for one model, and all of a run's passes go through one server process.

No hidden defaults. The server starts with --generation-config vllm, so no model's own default sampling settings apply. Qwen's would otherwise add top_k=20, top_p=0.8 and repetition_penalty=1.05, even with randomness off. Every request also sends the full set of sampling settings, a 1,280-token response limit, the model's own chat format and a single user message. Gemma's step-by-step thinking mode is off. Only the response text is graded, with nothing stripped, and the exact output tokens are stored, because re-tokenizing the text doesn't reliably recover them.

Order and caching. vLLM's prefix cache is off, so no pass reuses work from another. Prompts go in the same order on every pass, so two passes at one load differ only in what the server does unpredictably. "Requests at a time" is how many the client keeps outstanding. That caps the batch the server runs, and part 3 reports the batch each pass actually reached.

Each run had its own server

The homelab already serves Qwen on demand, behind a gateway, with its own settings and with batches that mix whoever is calling. A study of how batching changes results can't use it. Instead, each run starts its own vLLM server in its own container and loads the weights from the cluster's storage with downloads disabled. Once the server is healthy, the run records the vLLM version, the full command line, the exact weight revision and the default settings vLLM reported. All six counted runs record vLLM 0.30.0 and the pinned weights. A sampler reads the server's metrics once a second: requests running, requests waiting, and preemptions, where the server evicts a request to free memory and recomputes it later. At the end the server is stopped and its log is stored.

The server flags were the same for both models apart from one line each: --max-model-len 4096, --max-num-batched-tokens 4096, --max-num-seqs 64, --gpu-memory-utilization 0.76, --no-enable-prefix-caching, --enforce-eager (no CUDA graphs), --kv-cache-dtype auto, --seed 0, and no speculative decoding. Qwen adds --quantization fp8. Gemma switches off image and audio input, which keeps its vision and audio parts off the GPU.

Each run went through the general GPU queue (Argo Workflows and Kueue) at the priority used for training jobs, because an eval run can't resume partway through. The GPU is shared with models that wake on request and with a few small always-on services. Before each run, a launch script claimed the homelab's GPU window, which blocks the on-demand models from waking while it's held, and released it afterwards. The run records don't capture the window, or what else was on the GPU.

Each run checked itself

Before any of its numbers could be used, each run checked nine things against its own records. A failed check is recorded on the run as its result. None of the checks looks at accuracy.

1. Its settings exactly match one of the six planned runs: four kinds, two of them run for each model.
2. It ran through the general GPU queue, with its container, workflow, retries and image recorded.
3. Its server started, loaded the planned weights, and recorded its version and default settings.
4. The client kept the planned number of requests going. It reached the target, and averaged at least 90% of it over the first 90% of requests.
5. Every prompt and sample has exactly one final answer, and infrastructure failures stayed at or under 1%.
6. At most 1% of responses are empty, which would point to a chat-format problem.
7. The stored responses have exactly the planned number of rows.
8. The grader cleared its self-test inside the container.
9. Even the slowest request of 64 tokens or more produced at least 3.2 tokens a second, which keeps the 1,200-second timeout at least three times the longest possible response.

Every counted run cleared all nine. Three weaknesses turned up after the runs had recorded their results, so the checks were left as they were and the weaknesses were noted. Check 9 counts the end-of-response token, which overstates the rate by at most one part in 64. Check 5 duplicates check 7, so the nine are really eight independent checks. And check 3's comparison of the command line against the plan only confirms the wiring: both come from the same settings, so it can't prove on its own that the flags reached vLLM.

Changes to the plan

All eleven changes were made on the day the study ran. Only the last two came after any run had started, and change 10 still came before every counted run.

Two more items are recorded as deviations from the plan. The runs don't store each prompt exactly as the chat format rendered it. It can be rebuilt from pinned inputs, but that hasn't been done yet, and no number depends on it. And a sign convention in the table of rerun pairs was found in review and fixed in code before any run.

Neither late change altered a formula, the main comparison's rule or a self-check, and change 11 fixed the rule for reading the batch-invariant run before that run existed. Part 6 has both in full.