A job that declares 11,000 MiB of GPU memory, submitted while two small jobs run
submitted, waiting. running, from the job's own start stamp to its end. The queue had 10,556 MiB free while the two small jobs ran, 444 short of the declaration. The large job waited in the queue, then started once a small job ended.
Problem
Sharing the card depends on the queue knowing what each job needs, and it didn't. The cluster syncs the lab's job templates from a pinned commit, so a template change reaches the cluster only when the pin moves. The pin predated the change that lets a job declare its GPU memory, and the template ignored the extra argument. An eval job that asked for 12,452 MiB was counted at the 1,024 default, as was every job that asked for more since that default went live; before it, jobs declared nothing.
Change
Moving the pin to the cache change from step 2 carried the memory declaration with it. Each template file passed a server-side dry run against the cluster before the pin moved.
Result
A job declaring 11,000 MiB, submitted while two small jobs ran, passed the lock
and waited in the queue, which gave its own reason, insufficient unused quota for homelab.dev/vram-mib … 444 more needed. The arithmetic matches: 15,888 MiB
in all, less 3,284 held by the services and 2,048 by the two small jobs, leaves
10,556. Within four seconds of the first small job ending, the queue admitted
it, and it ran normally. It used far less of the card than it declared. The
queue accounts declarations, and nothing on the card enforces them.
Decision
Fixed. Large jobs now reserve what they declare, so a model-loading job that declares most of the card also keeps small jobs off it.