Jobs completed per minute, by how many were submitted together
one job alone's rate times the number of jobs. the range across batches. Each batch's rate is its completed jobs over the time from submission to the last job's end. Every job held the GPU idle for 60 s after a fraction of a second of GPU work, so these rates count jobs side by side, with almost no GPU work to share. One batch of two is left out of the rate: the 7B held it for 30 minutes. At five, the lock let four run and the fifth followed alone, so the batch took two rounds.
Problem
With the lock at four, the remaining questions were how throughput scales with the number of jobs, whether sharing slows each job's GPU work, and what happens at the limit and on failure.
Change
None to the platform. Cache-free jobs at the lab's lowest priority, each holding the GPU idle for 60 seconds after its work, were submitted together in batches of one, two, three and four, three batches per level, and one batch of five. Before each batch I checked that the lab's own CI was idle and no GPU job was still running.
Result
| Jobs submitted | Jobs at once | Jobs per minute (median) | vs one job alone |
|---|---|---|---|
| 1 | 1 | 0.81 | 1.0× |
| 2 | 2 | 1.69 | 2.1× |
| 3 | 3 | 2.47 | 3.0× |
| 4 | 4 | 3.24 | 4.0× |
| 5 | 4, then 1 | 1.92 | two rounds |
Each rate is one batch's completed jobs over the time from submission to its last job's end. These jobs hold the GPU idle for nearly all of their 63 seconds, so a near-linear rise is close to built in. Against the faster lone run, 0.87 a minute, four at once is 3.7×. The four started up to 3 seconds apart, five jobs took two rounds at a rate below three's, and no longer backlog was drained at four.
Median GPU timings for the matrix multiply and the language model stayed within
7% of a lone job's at every level. Split by start time, they differ: the 16 jobs
that started within 0.1 seconds of another ran the matrix multiply 5% slower at
the median and the slowest 25% slower, where lone jobs varied by up to 17%. The
16 that started apart matched a lone job. That hints at a time-slicing cost;
with a fraction of a second of GPU work per job, these runs can't measure it. At
five, four ran and the fifth waited with
Waiting for … gpu-semaphore/gpu lock, then ran alone. Every output matched its
CPU reference.
Three jobs running at once; the second to start is cancelled mid-run
submitted, waiting. running, from the job's own start stamp to its end. the cancel request. The cancelled job closed 11.1 s after the request, twice the 5.3 s of the one cancel one at a time. The other two ran on and finished with correct outputs.
Cancelling one of three running jobs closed it in 11.1 seconds, as cancelled
and not retried, twice the 5.3 seconds a cancel took one at a time, though each
is a single cancel. The other two finished normally.
The cache-lock check in step 2, the first time three jobs ran together, was much slower: opening each run record took 16 to 17 seconds instead of about one, and writing results afterwards about 13 more, both bookkeeping against the results database and object store. It ran while CI was building the lab's container images, and the slow start didn't come back in any later batch. One job running alone later stalled for about 33 seconds after its work, though, and neither slowdown is explained.
Decision
Adopted, and kept. Four small jobs at once is the platform's limit for now, set by the lock, which is sized to the queue's quota; with four running, the whole card peaked at 3,636 MiB in use. A job that crashes or runs out of GPU memory beside others wasn't tested: time-slicing gives jobs no memory isolation, and declarations aren't limits.