The general GPU queue's quota, with 4 small jobs
Each small job takes one share, declares 1,024 MiB of GPU memory and requests 2 GiB of host memory. Shares and the host memory quota both stop at four; GPU memory would allow more. A share is a scheduling token, with no guaranteed compute or memory. These are the queue's own numbers, read from the cluster; the card itself, services included, peaked at 3,636 MiB in use with four jobs running.
Problem
The lock should stop at the most jobs the queue can actually admit, so that the queue, which accounts for GPU shares and memory, decides admission below it. I had been reasoning from a queue of four shares, which would have meant three.
Change
I read the queue's live quota before setting anything. The card is time-sliced into eight shares, the number the node advertises; a share is a scheduling token, with no guaranteed compute or memory. The four always-on models (speech in and out, object detection and embeddings) hold one each, which leaves four free while the on-demand 7B is off. The queue's 12 GiB host-memory quota, with 4 GiB used by the services and 2 GiB requested per job, also stops at four. Declared GPU memory would allow more: 12,604 MiB free against 1,024 declared per job. The lock went to four.
Result
Four small jobs fit by every measure the queue keeps. The lock stops a fifth before the queue sees it, so a fifth job waits on the lock (step 5). The way back is unchanged: set the lock to one.
Decision
Adopted, and kept once the first concurrent runs were in. Four is tied to today's services: adding another always-on model to the card changes the number.