Step 13 of 15Adopted

Give every vLLM service the driver cache

The driver-cache setting moved into the shared Helm chart, covering the embedding server and the on-demand 12B. The speech and vision services' caches were too small to matter.

The 7B's driver cache
69 MB
Speech and vision services
≤ 112 KB

Each service's CUDA driver cache · Size

The 7B (fp8, LoRA kernels)
69 MB
Embedding server
6.5 MB
Text-to-speech
112 KB
Speech-to-text
4 KB
Object detection
none

The shared Helm chart gives every vLLM server the cache. The speech and vision services compile too little to need one.

The vLLM servers' driver caches run to megabytes; the speech and vision services' to almost nothing.

Problem

Every other GPU pod on the card kept the driver's cache in a temporary directory too, so whatever it compiled was compiled again on every start.

Change

The setting moved into the shared Helm chart template that every vLLM service uses.

Result

It now covers the embedding server, whose cache is 6.5 MB, and the lab's other on-demand model, a 12B. The speech and vision services were measured first. Their driver caches were 112 KB, 4 KB and none, so they were left alone.

Decision

Adopted. The gain on the embedding server and the 12B isn't measured; the 12B's shows on its next wakes.