ML Orchestration Lab

This lab is a small ML platform that I use to learn how large GPU platforms work. It runs on the cluster described in The Homelab and has one GPU in service, a 16 GB RTX 5060 Ti.

The problems it exists to practice show up anywhere GPUs are shared and expensive. Serving and training compete for the same accelerators. Work has to queue fairly, with priorities that hold up under pressure. Distributed jobs need all their workers at once or none at all. A higher-priority job has to be able to take a card away from a lower one, and the job that loses it has to recover cleanly. And when a run is slow or fails, someone has to be able to see why, from the model down to the hardware.

One card is enough to make every one of those real, because the card is genuinely contended. The tools are the ones used at scale, Kueue, Volcano, Argo Workflows and Slurm, so the lab is oversized for one card on purpose. The patterns are meant to carry over to a cluster with many GPUs.

The experiments that run on it are written up on Experiments: two series that take one question a step at a time, and a study that fixed its analysis before the first run.

One card, many claimants

The card has no hardware partitioning, so it is shared by time-slicing: the cluster sees it as eight shares, and a pod may hold one. The shares are scheduling tokens: they count pods, not memory or a fixed slice of compute. Every share sees all 16 GB of memory, and a workload that runs past it fails at run time rather than being refused up front.

These are the workloads that want it:

• Resident models, always on: an embedding server, speech-to-text, text-to-speech and object detection.
• Two language models, a 7B and a 12B, off by default. The 7B wakes on its first request, and both are set to go back to zero after an hour without one. Only one fits at a time.
• A GPU notebook, started when someone needs it.
• Experiments and benchmarks, submitted as workflows.
• Slurm jobs, on a GPU worker that exists only while it's asked for.
• Fine-tuning, which needs as much of the card as it can get.
• GPU CI, build steps meant to test code on the real card.

That mix, latency-sensitive serving next to throughput-hungry batch work, is the same contest a company's GPU cluster runs every day, at a smaller size.

The lab didn't start with a scheduler in charge. GPU work used to run in several containers that all had the card mounted, with a script that stopped the language model before a heavy job and restarted it afterwards. The script could only see part of what was on the card, so it would report the card free while another model still held most of the memory, and the heavy job would run out of memory. Patching the script wasn't the answer. Every GPU consumer moved into Kubernetes instead, Slurm included, so that one scheduler decides who gets the card. The script no longer arbitrates anything, and removing it from the host is the last step.

Deciding whether, then where

Two schedulers run at different layers, and the split between them is the core of the design.

Kueue decides whether a workload may run. It is the admission layer: it holds each GPU pod at a gate until the pod's queue has quota for it, compares priorities, and preempts lower-priority work when something more important needs the card. It never chooses a node. Workflow steps pass one gate before it: an Argo semaphore that caps how many GPU steps run at once.

Volcano decides where a group of pods lands. A distributed training job is useless with half its workers, so Volcano places a gang all at once or not at all. It will preempt the extra workers a job holds above its minimum, but never take a gang below the minimum it needs to run.

Neither is enough alone. Kueue admits pods one at a time, so it could admit half a gang that would then wait forever. Volcano doesn't see Kueue's quota. Labelling a gang's pods as one group makes Kueue wait until every member exists and admit them together, and Volcano then binds them together. That gives one all-or-nothing guarantee at each layer: quota in Kueue, placement in Volcano.

Priority is policy, written down. Every GPU workload carries a priority class. From the top: the in-cluster container registry, because evicting it to make room for a GPU pod would leave that pod unable to pull its own image; then the always-on models; then training; then the on-demand language models and the notebook; then batch work such as benchmarks, CI and low-priority experiments.

Quota is policy too. Two queues share a pool. The general queue owns all eight shares. The large-model queue owns none and may borrow one, which is how "only one language model at a time" is enforced: by quota, not by a script. When training needs that share back, Kueue reclaims it, the language model is preempted, and it comes back on its own once the training finishes. An experiment that needs the whole card claims a GPU window instead: the on-demand models are cleared, and no request can wake them until the window is released or lapses.

Memory is accounted for, but not enforced. Shares count pods, and memory is what actually runs out, so the pool also counts GPU memory. Every GPU workload declares the memory it needs, CI refuses a workload that doesn't, and Kueue holds work back when the memory isn't there, even with shares free. What nothing does is stop a process from using more than it declared. Time-slicing has no memory isolation, so a declaration is a promise. The real fix, Kubernetes' Dynamic Resource Allocation with shareable memory, is still alpha in every component it would need.

Slurm inside the cluster

Much of the ML world still schedules on Slurm, and its idioms are worth knowing on their own terms: partitions, a QOS ladder where higher tiers preempt lower ones by requeueing them, job accounting, and a REST API for submitting work. So Slurm runs inside the cluster through Slinky, SchedMD's Kubernetes operator, with its controller, accounting database and REST API as ordinary pods.

Its GPU worker is the interesting part. A Slurm worker is a long-lived pod. If it held a share of the card permanently, Kueue would admit it once and every Slurm job after that would bypass the quota. So the GPU worker sits at zero replicas. Asking for it scales it up, Kueue admits it against shares and memory like anything else, and Slurm's own scheduler places jobs on it from there. With the 7B serving, a request for the Slurm GPU worker waits in Kueue's queue rather than squeezing in beside it.

Slurm used to run in two containers beside the cluster. That setup is stopped, after its drills were re-run on the new one. Partitions, the QOS ladder, preemption by requeue, job arrays and accounting passed. A two-node exclusive preemption and a multi-rank PyTorch run couldn't run yet, and that is the gap: Slurm jobs can't yet run real ML containers. Only a GPU check has run through Slurm on the card, and a container per job is next.

Experiments that don't know the scheduler

The experiment code lives in its own repository. It never names a Kueue, Volcano or Slurm object or asks for a GPU directly, and a test fails the build if it does. It reaches the card only through the platform's GPU step template, passing a priority and a memory figure as arguments. A study is a function that takes a spec, and the platform's templates decide everything else about where and how it runs. Swapping Volcano for something else, or moving to a cloud cluster, is meant to be a platform change that touches no experiment.

Each run writes a record of what it was: its config, a fingerprint of the code that produced it, and the environment it ran in, down to the driver and the card. The same record says how the run ended.

Preemption, a node drain and a user's cancel all reach a running process the same way, as SIGTERM. The process can't tell them apart, so a cancel marks the record before it asks for the stop. On the way out, the run closes its record as cancelled if the mark is there, and interrupted if not. failed is kept for code that raised or a process that died, so a failure count never mixes in work that was simply preempted. When the workflow retries an interrupted run, the new attempt links back to the one it replaces, and a study that writes checkpoints can resume from that attempt's last one. A run that is killed outright can't close its own record, so a reaper fails records that have gone quiet for too long.

In a preemption drill, the interrupted run's record closed within a tenth of a second of the preemptor arriving. The retry waited in Kueue's queue until the card was free, then finished.

A tracker sits on top of these records: plans for experiments not yet run, what was submitted and when, and every run with its outcome, provenance and retry chain. It is the seed of a cluster portal, the front door for every GPU job on the homelab, from any repository: submit it, watch it, see why it's waiting and what preempted it, and keep its record. People and agents will both submit through it, and work an agent proposes will run on its own when it fits its project's budget. The portal is decided and not built.

Telemetry across the stack

A training or inference run touches every layer at once: what the model computes, the framework and kernels that compute it, the process and the card underneath, the requests or steps the run is made of, and the scheduler and cluster around it. Each layer has its own tools and its own format. The goal is one continuous picture of a run across all of them, so that a slow step can be traced from the scheduler's decision down to the kernel and back.

Scheduler and cluster. Kubernetes events, the scheduler's admission and preemption metrics, and the workflow engine's metrics are shipped to SigNoz, alongside container logs and node metrics. This is the layer that answers "why is my job waiting?" and "what took my card away?". Alerts here cover failed workflows, work stuck waiting for quota, and the health of the scheduler itself.

Run and requests. A run emits a span for each phase of its work and a heartbeat as it makes progress, under its experiment's name and tagged with its id and pod. The model servers emit a span per request, and the gateway in front of the 7B tags each request with the fine-tuned adapter it used, which the server's own metrics can't do. It also logs every wake and every request it held. Alerts on stalled or failed runs are queries over the run records rather than over telemetry, because a process that is killed outright stops sending anything, and its record is still there to be found.

Process and card. The host reads the card every 15 seconds: memory, utilization, power, temperature and clocks, plus memory per process, traced through its cgroup to the pod. The run record carries the pod's name, so the two can be joined. That answers who is using the card, but it is too coarse to show what happens inside a step. The next piece is a sampler inside the run at ten samples a second, written as a table attached to the run, with memory percentiles and energy as summary numbers. Checking that sampler against the host's readings is the first study planned on the platform.

The card itself sets a limit here. NVIDIA's data-center monitoring stack, which reports how busy the compute units and tensor cores are, doesn't support consumer cards like this one. That gap is why lower-level options are on the table: kernel profiling with NVIDIA's own tools, which needs a driver setting on the host, and, as an idea still to test, eBPF probes on the driver's calls, which would show what a process asks of the card without touching its code.

Kernels and framework. PyTorch's profiler records which kernels ran, how long they took, and where the GPU sat idle waiting on the CPU. Its traces have their own format, and they belong in a trace viewer, not a metrics backend. A self-hosted Perfetto viewer and a capture helper exist from before the cluster. The work is to capture traces from runs on the cluster and attach them to the run they came from.

Model internals. The experiment code already uses nnsight to read a model's activations for research. The same hooks can produce cheap per-layer summaries while a model trains or serves, such as activation scales or attention statistics, and turn them into telemetry rather than results.

One rule holds across every layer. Telemetry is for operating the system and is never read back as a result. Results live in the run's database. When a study needs a time series as data, like the sampler's table, it is written as an artifact of the run, not fetched from the monitoring system afterwards. The portal is designed the same way from the other side: a run's scheduling and runtime context will be joined from the scheduler, the logs and the telemetry when someone looks at it, and never copied into the record.

Experiment rounds

The first series made the wake-on-request 7B worth using. Turning the model off by default is meant to give the card back to experiments, though how much it gives back is still to be measured, and the first wake took 151 seconds. Fifteen measured steps brought a cold caller's wait to 36.5 seconds, and the gateway now holds a cold request instead of answering with errors. The biggest single gain came from keeping the CUDA driver's compiled kernels between starts instead of letting every pod compile them again. Each step has its own short page: LLM cold start.

The second round used the lab to measure an LLM eval: two models compared on IFEval with error bars, and how much of those error bars comes from the inference server's own run-to-run noise. Gemma's 15.7-point lead over Qwen carried about ±4 points of error, mostly from which prompts the benchmark happens to contain. Batching in vLLM flipped up to 7% of one model's results between identical passes, yet made up only about a fifth of the lead's variance (Error bars on an LLM eval).

The third round measured the platform itself. Experiment jobs used to run one at a time. With the limit raised to four, four 60-second test jobs that mostly held the GPU idle finished in about the time one takes alone. Heavy GPU work sharing the card isn't tested yet. While the on-demand 7B is up, no job at the lowest priority is admitted; a training job evicts the 7B instead (Many small jobs on one GPU).

The next rounds measure what sharing one card costs each tenant, how a request becomes GPU time, and how a model could be gated before it's promoted. Distributed training gets its own rounds. The lab's second GPU is in a laptop, an older NVIDIA card several generations behind the main one. The laptop runs hot and is a pain to keep connected, and it has been offline since the summer. Rather than run it all the time, I plan to bring it back as a second GPU node for distributed rounds. The two cards need different PyTorch builds and no NCCL build spans both, so gradients will travel over Gloo across the home network, so communication cost, stragglers, and workers dropping out and rejoining become part of the experiment.

What broke

What's next

• Slurm jobs in real containers, one container per job, so ML workloads can run through Slurm and not just reach the card.
• The GPU node as its own VM, with the card passed through.
• Waking only for inference, so an automated probe can't keep the model up, and a keep-alive window chosen from a week of real traffic.
• The cluster portal, starting with who submitted what, scheduler views, and submitting through its API.
• The telemetry layers above, starting with the in-run GPU sampler.
• Run events on the lab's event log, so a finished study can start its own analysis.
• GPU memory the card enforces, once Dynamic Resource Allocation is ready.