Experiments
Experiment rounds
Rounds of experiments on the ML Orchestration Lab, and projects designed in public. A series takes one question from a baseline to a result, one measured step at a time. A study fixes its question and analysis in code before the first run. A project starts from its design review, and every published version of its design documents stays readable.
4 experiments
LLM cold start
A 7B language model on a shared GPU, off by default and woken by its first request. Fifteen measured steps took a cold caller's wait from 151 seconds to 36.5, and the gateway now holds a cold request instead of answering 503.
vllmcudakueueargo workflowspytorchError bars on an LLM eval
I put error bars on an IFEval comparison of two open models, Gemma and Qwen, following Evan Miller's Adding Error Bars to Evals, on one GPU in my homelab. Gemma's 15.7-point lead carries about ±4 points. An identical rerun flipped 7% of Qwen's prompts between right and wrong, yet that noise is only about a fifth of the error bar's variance. The rest comes from which prompts the benchmark happens to contain.
vllmifevalstatisticsargo workflowskueueMany small jobs on one GPU
Experiment jobs on my homelab's GPU ran one at a time, with a 16-second hand-off from each job to the next. With the limit raised to four, four 60-second jobs that mostly hold the GPU idle finished at four times one job's rate, and while the on-demand 7B model is up, no lowest-priority job is admitted at all.
kueueargo workflowskubernetesgpu time-slicingpytorchJoining mouse brain datasets for AI agents
The design for a library, an MCP server and Skills that let AI agents answer questions spanning Allen Institute and Janelia mouse brain datasets. Each join checks that both sides use compatible atlas and taxonomy versions, and each answer carries a record that a verifier reruns. A pre-registered experiment measures what each part adds, in Claude Science and in other agent harnesses.
mcppythonslurmapptainerclaude agent sdk