Step 8 of 15Adopted

Adopt it in production

Driver cache, pinned KV cache, no CUDA graphs. The production pod went from 145 s to 40 s to Ready, for a 0.8% batch-1 decode cost on the twin.

Production time to Ready
145 → 40 s
First 256-token answer
5.2 s
Batch-1 decode
0.8% slower
on the twin

The production pod, before and after · Time to Ready

Before tuning
145 s
First start after the change
81 s, once
Every start after that
40 s

The first start after the change fills the driver's cache, so it pays the old compile cost once.

The real pod, against its own log from the day before.

Problem

Two configurations passed the adoption rule: graphs on, about 56 seconds to a first answer, and graphs off, about 45.

Change

Production got the persistent driver cache, the pinned KV cache, and eager mode: no CUDA graphs and no torch.compile. Eager won on wake to first answer, at a 0.8% cost to batch-1 decode on the twin, and the goal was the shortest cold start.

Result

On the real pod, time to Ready went from 145 seconds to 40, with the first 256-token answer in 5.2. The first start after the change takes 81 seconds, once, while it fills the cache.

Decision

Shipped.