Step 12 of 15Not adopted

Sleep mode as a middle tier

Both sleep levels freed about 11.5 GiB of the card and served again in 0.8 s or 2 s, but a sleeping model still held its quota, and level 1 added 14.1 GiB to the pod's host memory.

GPU memory freed
~11.5 GiB
Back to serving
0.8 s / 2.0 s
level 1 / level 2, warm
Host memory added
14.1 GiB
level 1

Sleep mode against a full cold start · Back to serving

Off, cold start at the time
45.6 s
Level 2, page cache reclaimed
16.9 s
Level 2, page cache warm
2.0 s
Level 1
0.8 s

The resident models hold about 3.2 GiB of the card in every bar.

Three views of the trade: how fast it comes back, what it frees on the card, and what it adds on the host.

Problem

Between up and off, vLLM's sleep mode might give a fast wake without holding the GPU.

Change

Both sleep levels on a twin, one fresh pod per level, measuring GPU memory, host memory, the time back to serving, and whether answers matched before and after. Level 1 offloads the weights to host memory. Level 2 discards them and reloads them on wake.

Result

Both levels freed about 11.5 GiB of the card and answered identically after waking. Level 1 came back in 0.8 seconds, but the pod's host memory rose 14.1 GiB, more than the card freed, so probably part page cache, and stayed there after waking. The host has hung under memory pressure before. Level 2 added nothing and served again in 2.0 seconds with a warm page cache, or 16.9 with a cold one.

A sleeping pod keeps its Kueue quota. A training job still takes the card from it: against level 1, Kueue preempted the sleeping twin and admitted the job in four seconds, as it would from a model that's up. Lower-priority work, which can't preempt the 7B, waits for a sleeping model just as it waits for one that's up. Level 2 holds the same quota.

Decision

Not adopted as it stands. Level 2 becomes a third state, beside up and off, when the keep-alive window is chosen from real traffic.