The lab's main GPU, a 16 GB RTX 5060 Ti, runs four always-on models and a queue of experiment jobs. Until this series the jobs ran one at a time by design: eight 5-second jobs took 2.7 minutes, most of it spent handing the card from one job to the next. Four jobs can now run at once, the most the queue's quota admits, and a fifth waits its turn.
When each job waited and ran, from the moment the batch was submitted
submitted, waiting. running, from the job's own start stamp to its end. the cancel request. Eight jobs submitted together, each holding the GPU idle for 30 s after its work. They ran in turn, with 16 to 19 s between one ending and the next starting. The third was cancelled mid-run.
How the card is shared
Every GPU job is a step in an Argo Workflows workflow, and the shared template that step runs through takes an Argo semaphore before it starts. That semaphore is the lock, and it had room for one. Behind it, Kueue, the cluster's job queueing system, holds each GPU pod until its queue has quota for it. The card is time-sliced into eight shares, scheduling tokens that guarantee no compute or memory, and each job takes one. The general GPU queue owns all eight, and the four always-on models hold four of them. It also accounts for the host memory each pod requests and the GPU memory each pod declares, a number the queue counts but nothing on the card enforces. A Qwen2.5-7B served by vLLM is off until a request wakes it. Up, it runs through the large-model queue, which owns no shares and borrows one, with 12,200 MiB of GPU memory, from the general queue. Every GPU workload also carries a priority: from the top, the always-on models, training (a GPU step's default), the on-demand language models, and batch, the lowest.
The general GPU queue's quota, with 4 small jobs
Each small job takes one share, declares 1,024 MiB of GPU memory and requests 2 GiB of host memory. Shares and the host memory quota both stop at four; GPU memory would allow more. A share is a scheduling token, with no guaranteed compute or memory. These are the queue's own numbers, read from the cluster; the card itself, services included, peaked at 3,636 MiB in use with four jobs running.
The test job
Every test used the lab's smallest GPU job: a 4096×4096 matrix multiply and a tiny in-process language model, each checked against a CPU reference. It takes about five seconds from start to finish, a fraction of a second of that on the GPU. Most runs then held the GPU idle for a fixed 30 or 60 seconds: one at a time, to tell overhead from work, and in the concurrency tests, so that jobs would overlap. Each job was its own submission, and the measured batches ran at batch priority, which never preempts anything. The numbers come from the lab's run records: when each job was submitted, started and ended, its own GPU timings, and the card's memory use sampled during the run. Two things were read from the cluster instead, during the runs: the hand-off breakdown in Fig. 3, from pod and queue timestamps, and the queue's quota.
One at a time: 16 seconds between 5-second jobs
The gap between one job ending and the next starting, by part
Hover or tap a part. The middle three parts come from pod and queue timestamps read from the cluster during the run, at one-second resolution, not from the run record. The last runs from the container's start to the job's own start stamp, and the first is the gap less the other four, so it also carries their rounding. Most of every gap passed before the next job's pod existed, while Argo released the one-job lock and started the next waiting workflow's step.
Eight short jobs submitted together drained in 2.7 minutes, 2.96 jobs a minute. The jobs ran for about five seconds each, and the gaps between one ending and the next starting, 16 seconds at the median, took most of the time. In pod and queue timestamps, 8 to 11 seconds of each gap passed before the next job's pod existed, while Argo released the lock and started the next waiting workflow's step; these runs don't show why that takes Argo so long. The queue admitted each pod in the same second it appeared. Container start took 2 to 3 seconds, and the job's own start-up, Python, PyTorch, CUDA and part of opening its run record, another 3 to 4. Cancelling a job mid-run closed it in 5.3 seconds, and the job behind it started 15.7 seconds later, no slower than after a normal finish (step 1).
Raising the lock
GPU jobs mounted the lab's shared Hugging Face model store read-write, since the shared GPU template has no read-only switch. The store is root-owned and the lab's image runs as an unprivileged user, so jobs can't write to it. Before raising the lock, I added a cache-free submission that skips the mount, for jobs that load no model, and gave jobs that keep it a cache lock with room for one. That lock guarded a write no job could make, so after the series I removed it. Two jobs that mount the store, submitted together, then started within milliseconds of each other and ran side by side (step 2).
I sized the lock from the queue's live quota. Of the general queue's eight shares, the always-on models hold four, which leaves four while the 7B is off, and the 12 GiB host-memory quota, 4 GiB of it held by the services, at 2 GiB a job, also stops at four. So the lock went to four, and below it the queue's own quota decides admission (step 3).
The GPU-memory quota only works if declarations reach the queue, and they hadn't. The cluster syncs the lab's job templates from a pinned commit, and the pin predated the change that lets a job declare its GPU memory. Every job since the 1 GiB default went live, including an eval job that declared 12,452 MiB, had been counted as 1,024 MiB. Moving the pin made declarations real for the first time (step 4).
Four at once: 4.0× one job's rate, on mostly idle jobs
Jobs completed per minute, by how many were submitted together
one job alone's rate times the number of jobs. the range across batches. Each batch's rate is its completed jobs over the time from submission to the last job's end. Every job held the GPU idle for 60 s after a fraction of a second of GPU work, so these rates count jobs side by side, with almost no GPU work to share. One batch of two is left out of the rate: the 7B held it for 30 minutes. At five, the lock let four run and the fifth followed alone, so the batch took two rounds.
With the lock at four, I submitted one to four cache-free jobs together, three batches each, and one batch of five, every job holding the GPU idle for 60 seconds after its work. The completion rate rose in a straight line: 0.81 jobs a minute for one job alone, 1.69 with two, 2.47 with three and 3.24 with four, 4.0× in all, or 3.7× against the faster lone run. That compares one batch of mostly idle jobs with one such job alone, so a near-linear rise is close to built in. The costs showed at the edges: the last of four started up to 3 seconds after the first, five jobs took two rounds at 1.92 a minute, below three, and no backlog longer than one batch was drained at four.
Median GPU timings at each level stayed within 7% of a lone job's. Jobs that started within 0.1 seconds of another ran the matrix multiply 5% slower at the median, which hints at a time-slicing cost; with a fraction of a second of GPU work per job, these runs can't measure it. Cancelling one of three running jobs closed it in 11 seconds, twice the time alone, and the other two finished with correct outputs (step 5). Two slowdowns in the jobs' own bookkeeping are unexplained: in the step 2 check, run while CI was building the lab's images, opening each run record took 16 to 17 seconds instead of one or two, and one lone job later stalled for 33 seconds after its work.
While the 7B is up, lowest-priority jobs wait
Two lowest-priority jobs submitted while the on-demand 7B was up
submitted, waiting. running, from the job's own start stamp to its end. Both jobs waited in the queue for 30 minutes without failing. Once I scaled the 7B to zero by hand, the queue admitted both within 7 s, and they ran normally.
Up, the 7B borrows a share and 12,200 MiB of GPU memory, which leaves the general queue 404 MiB, so no lowest-priority job is admitted at all. During the test, a request woke it (the cold-start series covers the wake), and two jobs submitted 27 seconds later sat in the queue for 30 minutes. They didn't fail: the job time limit only counts once a pod starts, and a pod the queue holds never does. When I scaled the 7B down by hand, the queue admitted both within seven seconds. Left alone, it stays up until an hour after its last request, so a steady trickle of requests could hold these jobs off indefinitely. A job at training priority doesn't wait: the general queue reclaims the borrowed share and evicts the 7B, which took four seconds against a test copy of the 7B in the cold-start series. In the other direction, the queues are configured to let a waking 7B evict running lowest-priority jobs, which Argo then retries. That direction wasn't tested (step 6).
Every step
- 01MethodMeasure jobs run one at a timeThroughput, one at a time: 2.96 jobs/min
- 02AdoptedLet jobs that load no model skip the cacheJobs running together: 3
- 03AdoptedRaise the lock from one to fourLock size: 1 → 4
- 04FixedGet memory declarations to the queueMemory the queue counted per job, before: 1,024 MiB
- 05AdoptedSubmit one to five jobs at onceCompletion rate, four at once: 3.24 jobs/min
- 06Open problemWhile the 7B is up, lowest-priority jobs waitLowest-priority jobs admitted, 7B up: 0
Where it stands
Sharing stays on at four. The cost falls on the lab's vLLM benchmark workflow, which measures a model server's throughput and got the card to itself by holding the same lock: now up to three small jobs can run beside it unless it declares most of the GPU memory. Exclusive work needs a guarantee of its own, and that comes next. Not tested here: a job that crashes or runs out of GPU memory beside others. Time-slicing gives jobs no memory isolation from each other, and declarations are accounting rather than limits, so that is the open risk of sharing. The next experiment measures the other kind of concurrency: one model server with many requests in flight, and where its throughput stops scaling.