LLM cold start

The lab's main GPU, a 16 GB RTX 5060 Ti, is shared by four always-on models and a queue of experiments. The cluster sees it as eight time-sliced shares, scheduling tokens that guarantee no compute or memory. A Qwen2.5-7B with fp8 weights and one fine-tuned LoRA adapter, served by vLLM, used to be started by hand and left running, holding about 12 GiB of the card. I wanted three things: keep the card free for experiments, let any app's request wake the model, and make that wake as short as possible.

How it was measured

Engine changes ran first on a twin: a disposable copy of the production server with the same model, flags, declared GPU memory and memory limit, so experiments never touched the live model. A harness split each run's start into named phases from the vLLM log and pod events, and wrote a ledger row before anyone interpreted it. Most arms ran two or three times on the same card. Ready is the pod's readiness probe passing, checked every 10 seconds on the twin and every 5 in production, so Ready totals land on those steps. The adoption rule was set before the first measurement: keep a change if it cuts the cold start by at least 20% and costs at most 5% of batch-1 decode speed. The deciding number moved twice. Step 5 replaced time to Ready with wake to first answer, a 256-token request timed to its last token. Step 10 replaced that with a real client's wait through the gateway for its first streamed token.

The engine: 141 seconds to 40

The baseline took 141 seconds to Ready, and 104 of them were engine work around CUDA graphs: 52 seconds profiling their memory and 52 capturing them (step 2). Pinning the KV cache size skipped the profiling pass but saved only ten seconds, because the pass had been paying first-use kernel costs that then moved into the rest of the engine start (3). Capturing fewer graphs, or none, cut the start by up to 38%, and appeared to wreck batch-1 decode (4).

A warm-up request before the decode benchmark showed steady decode within 2% in every arm. The graph-less arms had only moved about 40 seconds from startup to their first request, so a caller's wake to first answer improved by 7%, not 38% (5).

The 40 seconds looked like framework kernel compilation repeated on every pod start. Keeping Triton's and FlashInfer's caches on a persistent volume changed nothing, and Triton compiled only one kernel during inference (6). The engine logged nothing for about 40 seconds before answering in 5, and a neighbouring pod held a cache from the CUDA driver itself. On this GPU generation the driver compiles part of the image's kernels from PTX on first use and caches the result in the container's home directory, lost with every pod. Keeping that 69 MB cache on the volume cut the start from 80 seconds to 40 without graphs and the first request from 44.7 seconds to 5.1; with graphs, capture fell from 52 seconds to 12–13 (7). The first-use costs that had moved around since step 3 were all this compile: the run that filled the cache paid them, and the runs after it didn't. Only which kernels it compiles is inferred.

Production took the driver cache, the pinned KV cache and eager mode, which drops both CUDA graphs and torch.compile. The real pod went from 145 seconds to 40 to Ready, with a first 256-token answer in 5.2 (8).

The caller: 151 seconds to 36.5

Verifying the production change turned up the LoRA adapter loading 21 seconds after the model, because its loader backed off between health checks. A fixed two-second check now has it ready first (9).

Every number so far stopped at the model. Timing a real client through the gateway showed the model Ready at 41 seconds and a well-behaved client still waiting until 150.3, because the gateway's Retry-After: 150 had been sized for the old start. A client polling every five seconds got its first token at 45.6 (10). The gateway now holds a cold request for up to 60 seconds and forwards it the moment the model is Ready, so the caller sees one slow answer and no 503: 41.4 seconds (14). A start that outruns the hold, such as a driver-cache refill or a wait for quota behind an experiment, still gets a 503.

Since the driver cache, Python before the engine had been the largest phase. An import profile attributed 89% of it to imports, paid twice because vLLM spawns its engine process and the engine imports everything again (11). Forking it instead cut about 14% of the start, under the 20% rule; I adopted it anyway, since decode didn't change and the API server never initializes CUDA. A cold caller's first token now arrives in 36.5 seconds, median of three back-to-back wakes, probably with the weights still in the host's page cache, and with no 503s (15).

Sleep mode, and the other GPU services

vLLM's sleep mode freed about 11.5 GiB of the card and served again in two seconds or less with a warm page cache, but a sleeping model keeps its Kueue quota. Level 1 offloads the weights to host memory and added 14.1 GiB there, on a host that has hung under memory pressure. Level 2, which discards them, becomes a third state when the keep-alive window is chosen (12). The driver cache setting moved into the shared Helm chart for every vLLM service. The speech and vision services' caches were 112 KB or less, so they were left alone (13).

Every step

1. 01AdoptedMake the model wakeableFirst wake to answer: 151 s
2. 02MethodBuild the measurement rigBaseline to Ready: 141 s
3. 03AdoptedSkip vLLM's memory-profiling passEngine start: 110 → 99.5 s
4. 04Suspect resultCapture fewer CUDA graphs, or noneTime to Ready: 130 → 80 s
5. 05MethodMeasure decode fairlyWake to first answer: ~135 → ~125 s
6. 06FalsifiedPersist the Triton and FlashInfer kernel cachesTime to Ready: 130 → 130 s
7. 07AdoptedPersist the CUDA driver's kernel cacheReady, no graphs: 80 → 40 s
8. 08AdoptedAdopt it in productionProduction time to Ready: 145 → 40 s
9. 09FixedThe adapter lagged the modelAdapter ready after the model: +21 s → −1 s
10. 10MethodMeasure what a caller waitsObeys Retry-After: 150.3 s
11. 11MethodAttribute the 22 seconds before the engine startsImports: 89% of pre-engine
12. 12Not adoptedSleep mode as a middle tierGPU memory freed: ~11.5 GiB
13. 13AdoptedGive every vLLM service the driver cacheThe 7B's driver cache: 69 MB
14. 14AdoptedHold the cold requestClient obeying Retry-After: 150.3 → 41.4 s
15. 15AdoptedFork the engine processProduction time to Ready: 40 → 35–36 s

Limits

The numbers describe this card, this vLLM version and this host, from two or three runs per arm. The twin missed two things production showed: the adapter's lag and a probe that kept waking the model. GPU-hours returned to experiments aren't measured yet, and that needs a week of real traffic.

Still open: choosing the keep-alive window from real traffic, waking only for inference requests so an automated probe can't keep the model up, attributing the 12 seconds between the engine and Ready, a third of today's start, and measuring the driver cache's gain on the embedding server and the lab's other on-demand model, a 12B.