Step 2 of 6Adopted

Let jobs that load no model skip the cache

A job submitted as cache-free skips the shared model store; jobs that kept it, the default, also took a cache lock with room for one. In the first live check, three jobs ran together and the second cache job waited. Since no job can write the store, the cache lock was later removed.

Jobs running together
3
the first time three experiment jobs shared the card
Cache-mounting jobs at once
1 → 4
after the cache lock was removed

Two jobs that mount the model cache and two that don't, submitted together

Job 1mounts cache
65 s
Job 2no cache
67 s
Job 3no cache
64 s
Job 4mounts cache
48 s

submitted, waiting. running, from the job's own start stamp to its end. The three that could run together did. The second cache-mounting job waited for the first to release the cache lock, then ran.

Two jobs that mount the model cache (green) and two that don't, submitted together. The second cache job waits for the cache lock while the other three run.

Problem

Every GPU job mounts the lab's shared Hugging Face model store as its model cache, read-write, because the shared GPU template has no read-only switch. The store is root-owned and the lab's image runs as an unprivileged user (uid 1000), so a job can't write to it, and a write the Hugging Face library attempts fails harmlessly. Under the one-job lock, two mounts never overlapped. Raising the lock meant deciding what overlapping jobs do with the mount, and the test jobs load no model at all.

Change

What shipped:

  • A submit flag, --no-hf-cache, for jobs that load no model. Their pods mount nothing.
  • Jobs that keep the cache (the default) hold a second lock, the cache lock, with room for one, for their whole run, as a backstop behind the store's ownership. They run one at a time among themselves.

Cache-free jobs take only the main lock, not a lock of their own: with two separate keys, setting the main lock back to one would still let one job of each kind overlap. The cache lock is always taken before the main one, so the two can't deadlock, and setting the main lock back to one restores one job at a time.

Result

Two cache-mounting and two cache-free jobs submitted together. Three ran at once for about a minute. The second cache job's step waited with Argo's own message, Waiting for … gpu-semaphore/hf-cache lock. Lock status: 0/1, and started 19.5 seconds after the first cache job ended. The pods mounted the cache or not as intended, and each job's spec and outputs came through intact. The jobs' own bookkeeping was slow in this batch, 16 to 17 seconds to open each run record, which didn't come back in later batches (step 5).

Decision

Adopted, and the cache lock removed after the series. It guarded a write no job could make, and it kept every model-loading job waiting on the one before it. Two cache-mounting jobs submitted together without it started within milliseconds of each other, ran side by side and both succeeded. The cache-free flag stays for jobs that load no model; only the command-line submit offers it so far. Every concurrency test in step 5 used cache-free jobs.