From the request to the first token, seconds
the model reports Ready. a 503 sent to the caller. the first token. Nine 503s, then the first token at 45.6 s: Ready, plus up to one polling interval, plus the first token. Fast, but only for a client written to retry.
Problem
Every number so far stopped at the model. Callers go through the gateway, which
answered a cold request with a 503 and a Retry-After header.
Change
An end-to-end mode for the harness. It scales the model to zero, sends a streaming request through the gateway, follows the 503s in one of two client styles, and times the first token. Three runs per client.
Result
The model was Ready at 41 seconds. A client that obeyed Retry-After: 150, a
value sized for the old start, got its first token at 150.3 seconds. A client
retrying every five seconds got it at 45.6. A warm request took 0.06 seconds,
and runs agreed within a few hundredths of a second.
Decision
The deciding number became a real client's wait. The stale retry hint was costing a well-behaved caller about 109 seconds.