Every run with randomness off sent its prompts through twice at each load, with nothing changed in between, to measure how much a rerun moves the result.
Share of responses that came out exactly the same in both passes
Lines show 95% intervals. One request at a time gave exactly the same responses, and so did 64 at a time with batch-invariant mode requested. With batching in vLLM's default mode, most responses came out different somewhere.
Why identical passes can differ
With randomness switched off, a model always picks its most likely next word, so in exact arithmetic two passes would match. On a GPU, the arithmetic for one request depends slightly on how the server batches it with others, because many GPU kernels add numbers up in a different order for different batch shapes. A request the server preempts (evicts to free memory) is also recomputed later, which changes the arithmetic again. When two candidate next words are nearly tied, a difference in the last digits picks a different one, and the rest of the response follows from there.
The nine pairs
The pairs were fixed before any run.
- Rows 1 to 6 repeat a pass exactly: the second pass against the first, with the same model and load.
- Rows 7 to 9 change the load on purpose: Qwen's first pass at a batched load against its first pass one request at a time.
| # | Pair | Responses identical | Prompts | Flipped right/wrong | Score change ± 1 SE | Noise vs error bar |
|---|---|---|---|---|---|---|
| 1 | Qwen, 32 at a time (main runs) | 13.5% (10.9–16.6) | 541 | 7.2% (5.3–9.7) | −0.9 ± 1.155 | 42.2% |
| 2 | Gemma, 32 at a time (main runs) | 30.7% (26.9–34.7) | 541 | 2.4% (1.4–4.1) | +0.2 ± 0.667 | 33.3% |
| 3 | Qwen, 1 at a time | 100.0% (98.1–100.0) | 200 | 0.0% (0.0–1.9) | 0, nothing changed | 0.0% |
| 4 | Qwen, 8 at a time | 33.6% (29.8–37.7) | 541 | 3.7% (2.4–5.6) | −0.4 ± 0.827 | 30.6% |
| 5 | Qwen, 64 at a time | 14.4% (11.7–17.6) | 541 | 7.0% (5.2–9.5) | +1.5 ± 1.139 | 41.1% |
| 6 | Qwen, 64, batch-invariant mode requested | 100.0% (99.3–100.0) | 541 | 0.0% (0.0–0.7) | 0, nothing changed | 0.0% |
| 7 | Qwen, 8 at a time vs 1 | 11.0% (7.4–16.1) | 200 | 10.0% (6.6–14.9) | +3.0 ± 2.232 | † |
| 8 | Qwen, 32 at a time vs 1, separate servers | 18.0% (13.3–23.9) | 200 | 8.0% (5.0–12.6) | +2.0 ± 2.000 | † |
| 9 | Qwen, 64 at a time vs 1 | 14.0% (9.9–19.5) | 200 | 8.0% (5.0–12.6) | +1.0 ± 2.004 | † |
Score changes are in points. Ranges in brackets are 95% intervals (Wilson intervals, a standard choice for proportions). † Rows 7 to 9 change the load on purpose, so they don't measure rerun noise. Any row involving one request at a time ran on the 200-prompt subset (part 6 explains why), so its intervals are about 1.6 times wider and it isn't compared with the 541-prompt rows. Row 8's two passes also came from different runs, on different servers. No prompt needed a retry in any run, so nothing was left out.
Where responses split. Among responses that changed, the first different token came at a median position of 25 for Qwen at 32 requests at a time, 80 for Gemma at 32, 91 for Qwen at 8 and 23 for Qwen at 64. In most batched pairs, some responses differed from the very first token, so batching can change a model's first word, not only drift in late in a long answer. The exceptions were Qwen at 8 and the 8-versus-1 comparison. Rows 3 and 6 had no changed responses at all.
The changes don't lean either way
How far the score moved between two identical passes, in points, ± 2 standard errors
Every interval crosses zero: the flips went both ways, with no detectable lean toward right or wrong. A lean would need to be about 2.8 standard errors to show 4 times in 5.
The share of flipped prompts says how often results change. The score change says whether those changes lean toward right or toward wrong. Every pair stayed within two standard errors. Across the four repeat pairs with any flips, the change runs from −0.80 to +1.30 standard errors, and across the three load comparisons from +0.50 to +1.34, so no lean was detectable. To show up 4 times in 5, a lean would need to be about 2.8 standard errors: 1.9 to 3.2 points for the repeat pairs, and 5.6 to 6.3 for the load comparisons.
The main comparison's error bar assumes rerun noise is independent from prompt to prompt, and noise of that kind is already inside it. These checks are consistent with that assumption, but they can't prove it. They're also reported as standard errors, so they aren't among the 26 intervals counted as the study's chances to be wrong.
How big the noise is next to the error bar
The last column of the table is √(f / (2p̂(1 − p̂))), where f is the share of flipped prompts and p̂ is the accuracy. It is the standard deviation an identical rerun contributes, relative to the score's standard error. The score's error bar already contains this noise. Squared, the column gives the noise's share of the variance.
- More prompts don't shrink it. Both the noise and the error bar shrink with the square root of the number of prompts, so their ratio stays put.
- It is the sampling runs' measure, with randomness off. Two passes with randomness off are like two answers per prompt, and half the flip rate is the variance between those two answers.
- Each value rests on one pair. With two passes per load, there is one comparison behind each number.
At every batched load in vLLM's default mode it was 31 to 42%: 30.6% for Qwen at 8, 42.2% at 32 and 41.1% at 64, and 33.3% for Gemma at 32. As shares of the variance those are 9.3%, 17.8%, 16.9% and 11.1%. Averaging many passes of each prompt would remove that part and narrow one model's error bar by about 5 to 9%. A deterministic server wouldn't: it would freeze one answer per prompt, and that answer still varies from prompt to prompt. In the gap between the models the share is larger, 21.7%, because comparing prompt by prompt cancels out how hard each prompt is and can't cancel rerun noise.
Batched passes against one at a time
Share of responses that came out exactly the same in both passes
Qwen's first pass at each batched load against its first pass one request at a time, on the same 200 prompts. Lines show 95% intervals. One request at a time gave exactly the same responses, and so did 64 at a time with batch-invariant mode requested. With batching in vLLM's default mode, most responses came out different somewhere.
Against the pass one request at a time, only 11.0 to 18.0% of batched responses were identical, and 8 to 10% of prompts flipped between right and wrong. These comparisons use 200 prompts, so their intervals are wide and they aren't compared with the 541-prompt pairs. They change the load on purpose, so they aren't used to measure rerun noise either.
Why Gemma and Qwen aren't comparable here
Gemma's results changed less than Qwen's at 32 requests at a time: 30.7% of responses identical against 13.5%, and 2.4% of prompts flipped against 7.2%. But the two weren't batched the same way.
| Run | Requests at a time | Passes | Requests actually running, average | Preemptions per pass |
|---|---|---|---|---|
| Qwen, main run | 32 | 2 | 24.3–28.0 | 0 |
| Gemma, main run | 32 | 2 | 22.8–23.0 | 857–866 |
| Qwen, sampling run | 32 | 8 | 24.3–28.9 | 0 |
| Gemma, sampling run | 32 | 8 | 22.6–23.4 | 782–855 |
| Qwen, concurrency run | 1 · 8 · 64 | 2 each | 1.0 · 7.8 · 49.8–51.0 | 0 |
| Qwen, batch-invariant run | 64 | 2 | 47.9–49.7 | 0 |
Every pass reached its full load at its busiest. Qwen's server never preempted a request. Gemma's preempted requests hundreds of times in every pass at 32 requests at a time, evicting them and recomputing them later, and ran smaller batches on average. vLLM preempts when it runs out of KV-cache memory, and recomputing a request changes its arithmetic. So Gemma's rerun noise was measured with preemption in the mix and Qwen's without, and the two models also use different compression code on the GPU. These runs can't tell which of those, if either, explains the difference. The metrics are sampled once a second, so a preemption that starts and ends between two samples at the edge of a pass can be missed.
What it means
For the main comparison, rerun noise at 32 requests at a time is 21.7% of the variance of the gap. It is already inside the error bar, and no lean was detectable. For anyone reading single-pass scores from a batched vLLM server on a setup like this one, an identical rerun moved a model's score by up to 1.5 points, with standard errors of 0.7 to 1.2.