Step 1 of 15Adopted

Make the model wakeable

The gateway wakes the 7B on the first request, and an idle reaper turns it off after an hour. The first clean drill took 151 s to a first answer.

First wake to answer
151 s
Idle window
60 min
Outage in the first drill
~16 min
  1. Off0 replicasThe card is free for experiments.
  2. A requestThe LLM gateway scales the model 0 → 1 and answers 503 until it's up.
  3. WakingReady in about 145 sKueue admits the pod on the one share the large-model queue may borrow.
  4. ReadyA client retrying every 10 s got its first answer at 151 s.
  5. ServingAbout 12 GiB of the cardOpenAI-style requests from any app or agent.
60 idle minutes The idle reaper scales it back to 0.
The 7B's life cycle as first built. A request that finds it off makes the gateway scale it up, Kueue admits the pod on the one GPU share its queue may borrow, and the idle reaper scales it back to zero after an hour without requests.

Problem

The 7B held about 12 GiB of the card around the clock. Turning it off broke every app that called it until someone turned it back on.

Change

The API gateway in front of the model scales it from zero to one replica when a request finds it cold. An idle reaper scales it back to zero after 60 minutes, counted from Ready and reset by every request. Sixty minutes is a starting guess until real traffic can set it. Fine-tuned adapters persist across restarts.

Result

The first live drill crashed the gateway: it couldn't handle the scaling API's response for a deployment at zero replicas, and the model was unavailable for about 16 minutes. Tests built from captured real responses fixed it, and the drill script now restores the model if a drill fails. The next drill, with a client retrying every 10 seconds, woke cleanly: 151 seconds from request to first answer.

Decision

Wake-on-request stays. Everything after this is about making the wake short.