Engineering design review: mouse-brain-ccf-join

Summary

• Problem: Many questions about the mouse brain need several datasets joined in one shared 3D atlas. Each crossing between datasets has a trap that gives a wrong answer without an error, and nobody can check what an agent joined.
• Design: A Python library translates each source, through its own adapter, into one model of atlas versions, cell-type taxonomies and observations. A join runs only between compatible versions. Every answer records what it read and every fix it applied.
• Agent surface: An MCP server and two Skills give agents tools that find data, plan joins, refuse invalid ones, and run code in a sandbox next to the data. They work in Claude Science and in Claude Code.
• Infrastructure: Batch work runs as Slurm jobs in Apptainer containers, which is how Claude Science computes on a cluster. The same job scripts run on the homelab's Slurm, on a Slurm cluster in AWS at the end, and on a partner's cluster. The MCP server runs as a container service.
• Evaluation: A pre-registered experiment runs four tool conditions in the Agent SDK, and subsets of them in Managed Agents and Claude Science, against gold answers from published sources. A component stays only if it measurably helps.

The problem

A question that spans datasets

Here is a question a neuroscientist might ask an agent:

Which cell types sit in the brain region under this recording probe, and how do the neurons there respond to what the animal sees?

No single dataset answers it. v1 answers two families of question. The first is the one above. The second starts from one traced neuron and asks which cell types sit where its axon ends, and how the neurons there respond. Four public datasets hold the parts of these two questions:

The Allen Common Coordinate Framework (CCF) links them. The CCF is a shared 3D reference atlas of the mouse brain. Each dataset places its observations in CCF coordinates and labels them with CCF regions. In software terms, the CCF is a coordinate reference system, like WGS84 for maps, and its annotation is a lookup table from a coordinate to a region id. Answering the question is a spatial join across datasets on that shared reference.

Where the joins go wrong

A spatial join is correct only when both sides agree on the axes, the units, the key and the version of the lookup table. GIS engineers know the failure: points geocoded against different datums land in the wrong place, and nothing errors. The v1 joins cross five traps of this kind. Each was checked in the public data on 2026-10-15 and 2026-10-16:

Why agents make it harder

• Size: The MERFISH brain's expression matrices take 15 GB as measured and 50 GB with imputed genes. An agent cannot read that. It calls tools or writes code, and each step can pick up a default it never sees.
• No signal: A trap returns a plausible result, so the agent has nothing to react to.
• No record: The answer does not say which files, versions and fixes it used, so a reviewer cannot rerun it.

What a trustworthy answer needs

A cross-dataset answer can be trusted only when four things hold:

1. Every observation sits in one reference space, at a named atlas version, with known axes and units.
2. Every label is an id in a named version of its ontology or taxonomy.
3. Every join connects observations at compatible versions: the same version, or two versions linked by a recorded crosswalk. A crosswalk is a mapping table between two versions.
4. The answer records what was read, every fix applied, and the code and environment that computed it, so a second person can rerun it.

The design enforces all four in code.

Constraints

Who it is for

Goals and non-goals

The goals, in priority order:

1. An agent answers v1's two question families with valid joins, or refuses and says why.
2. A verifier reruns any answer from its record and gets the same result.
3. A new source needs one adapter. The core and the agent surface stay as they are.
4. An experiment measures what the MCP server, the Skills and the join rules each add. The results are published whatever they show.
5. The system runs end to end on the homelab and on AWS, and its batch work runs unchanged on a partner's Slurm cluster.

v1 leaves these out:

• Patch-seq recordings: They use an older taxonomy with no crosswalk to the current one, and they have no CCF coordinates.
• MICrONS: It has no CCF registration.
• CELLxGENE Census: It excludes MERFISH, so neither v1 join needs it.
• Search over cell types in plain language: The Brain Knowledge Platform's AI helper already does this.
• Human or controlled-access data: Only open data is read.
• Re-hosting data: Data is read in place. The slice cache stays private to the cluster.
• Proposing findings: The system returns joined data, computed numbers and their records. It does not suggest biological conclusions.
• GPUs: No v1 step needs one.

Requirements

The 28 cases on the Evidence page motivate the provenance and identity requirements. The Evidence page also maps each case to its requirements.

Proposed design

Overview

The design follows the ports-and-adapters pattern. The core holds the shared model and every rule. It defines five ports, one per capability a source can offer: spatial observations, taxonomy, physiology, morphology and expression. Each source sits behind an adapter that implements the ports it can serve. On the other side, the MCP server, the Skills and the CLI call into the core.

The example question flows through the system like this:

1. Claude calls resolve_term, which turns the region's name into a region id in a named CCF version.
2. Claude calls find_observations for the units recorded in that region. The DANDI adapter reads the session's units in place, takes channel positions from Allen's channel table, mirrors left-right back, and returns a handle and a count.
3. Claude calls find_observations for the MERFISH cells in that region. The ABC Atlas adapter maps parcellation_index to region ids and converts millimetres to micrometres.
4. Claude calls plan_join with the two handles. The planner checks that both sides use the same CCF version, or finds a crosswalk. It returns the plan and the code that runs it, or it refuses and says why.
5. Claude calls run_plan. The code runs as a Slurm job, in an Apptainer container next to the data, and a small table comes back with a record id.
6. A reviewer, or Claude Science's Reviewer, calls verify on the record. The verifier reruns the plan and compares the result.

Components

Three external stores hold the data: Allen's S3 bucket, DANDI's S3 bucket and Janelia's NeuronBrowser.

How a join is kept valid

The model: The model has four kinds of thing.

• Reference spaces: each CCF version, with its axes, unit, resolution and region ontology.
• Taxonomies: each version of a cell-type taxonomy, with its levels.
• Genes: each gene at a named annotation release, with its symbol history.
• Observations: cells, units, channels and traced points. Each one has coordinates in a named reference space and labels at named versions.

Crosswalks link versions. The model is a schema. Adapters read observations on demand and return a handle to an Arrow table. No source is loaded whole.

The adapters: Each one fixes its source's traps on the way in and writes every fix to the record.

The planner's rules: The join planner applies five rules before any join runs.

1. Both sides must be in the same reference space version, or linked by a crosswalk in the crosswalk tables. Otherwise the join is refused.
2. Labels joined at a taxonomy level must come from the same taxonomy version, or be crosswalked.
3. Joins run on ids from the region ontology. A source's own index never reaches a join.
4. Unlabelled observations are counted and reported on the record.
5. Every fix an adapter applied is listed on the record.

What the rules miss: Adapters fix the axis and unit traps on the way in. Rules 1, 3 and 4 catch the version, key and missing-label traps, and rule 2 does the same for taxonomy versions. The rules cannot judge whether a question makes scientific sense, and they cannot catch a trap nobody has written a rule for. Code that an agent writes without the planner is also outside their view. The experiment counts the misses: an attempt that fails on an unlisted trap gets the failure code missed_caveat, and each such trap becomes a candidate rule with an adapter test.

Tools

Each tool declares input and output schemas and returns structured content, so a code step in Claude Science and a model calling the tool by name read the same fields. Each is marked readOnlyHint. A refused join returns the reason and the crosswalk that would allow it. Tests hold each response under a size budget.

Design principles

Anthropic's engineering guidance gives two principles for agent tools. The design accepts the cost of each and adds a third.

Data model

Reference rows are keyed by id and version together, so the same region id in two CCF versions is two rows, and a crosswalk row links them. Each observation belongs to one release and one reference space. A record lists what an answer read, each join step with its verdict, and the code and environment. A record cites only releases in the source catalogue, which is the Sources group.

One question, end to end

The first family's Skill carries this workflow, and it is the template for that family's join questions.

Infrastructure

The same job scripts and Apptainer images run on three Slurm clusters:

• The homelab: Slurm runs on k3s through Slinky, SchedMD's operator. No v1 step needs a GPU.
• AWS: a Slurm cluster built from infrastructure code, in P5.
• A partner's cluster: partitions, accounts and paths are settings.

Claude Science runs on a laptop and connects to a Slurm login node over SSH, so its data work runs as Slurm jobs too. In the Agent SDK and Claude Science, agent-written code runs on the agent partition, so the place code runs is the same in both harnesses.

Design decisions

Trade-offs

Evaluation plan

This section is the pre-registration. The question set, the gold answers, the graders and this plan are tagged in the product repository before the first full run, and the results report links to that tag.

Hypotheses

• H1, the MCP server: On leg questions, and separately on join questions, Claude with code, Allen's libraries and the MCP server (T2) answers correctly more often than Claude with code and Allen's libraries alone (T1).
• H2, the Skills: On join questions, adding the Skills to the MCP server (T3 against T2) raises correctness.
• H3, the join rules: On trap questions, T2 and T3 each give the required fix or refusal more often than T1.

T0 is reported as a floor, with no hypothesis.

Three comparisons are exploratory. They are reported with intervals and carry no decision rule:

• E1, the harness: Claude Science against the Agent SDK, with the same tools, on the Claude Science subset.
• E2, the Reviewer: In Claude Science, whether the Reviewer criterion catches wrong joins.
• E3, a hosted agent: Managed Agents against the Agent SDK, with the same tools, in P5.

Conditions

T1 is the baseline. It gives Claude what a capable agent has today. Every condition uses one model and effort setting, named in the pre-registration tag, and the same system prompt apart from the tool description. Each answer ends with a JSON block with fixed fields, so every grader reads one format.

Question set

Leg questions check one dataset at a time against Allen's notebooks and two dataset papers: the AllenSDK Neuropixels tutorials, the abc_atlas_access MERFISH-to-CCF tables, Siegle et al. 2021 and Winnubst et al. 2019. No published source answers a join question, so join questions are graded on form. The report marks each form-only question as a place where expert review would go.

Size: The decision rules compare conditions within one kind, so the count per kind sets the precision. Each question is scored as its mean correctness over five runs, and conditions are compared question by question. If the per-question differences have a standard deviation near 0.5, then at 80% power and a 5% two-sided test, 30 questions detect a difference of about 25 percentage points, and 20 detect about 31. Each extra question costs a reference script or a sourced fix.

Pilot: A pilot of ten questions per kind checks the 0.5 assumption before the set is frozen. A larger spread adds questions. Questions that every condition gets right, or every condition gets wrong, are replaced. A leg question whose published number does not reproduce at the pinned release moves to form-only, and the report lists it.

Procedure and grading

Each question runs five times under each Agent SDK condition, for 1,600 attempts. Managed Agents adds 800 in P5. The Claude Science subset is 25 questions, stratified by kind and run once per cell by hand, for 75 attempts. Run order is randomised, and every transcript is stored with its token counts.

The leg and trap graders follow the aind-data-mcp benchmark's two modes: a match and an LLM judge. Every condition is graded from the same JSON block, so no condition needs this system's tools to pass. Every wrong attempt also gets one failure code: wrong_version, wrong_axes_or_units, wrong_key, dropped_unlabelled, wrong_level, ungrounded_number, fabricated_citation, missed_caveat or tool_call_error.

Analysis and decision rules

The primary metric is correctness per kind and condition. Each comparison reports the paired difference with a 95% interval, from a bootstrap that resamples questions and keeps each question's runs together, following Adding Error Bars to Evals. Secondary metrics are the share of answerable questions refused, the count of each failure code, and tokens per question. The report leads with the failure codes.

A component stays only if the evidence shows it helps. An interval that includes zero counts as no help.

No target score is set in advance. The results are published with their intervals whatever they show.

Security, data handling and operations

The threat model maps every data flow and trust boundary, and records each STRIDE threat with its mitigations and status.

• Read-only: The system reads open data and holds no data credentials.
• Untrusted text: Free text from metadata and NWB descriptions comes back in quoted fields, and the tool descriptions mark it as data. Agent code runs only in the sandbox, so text that does steer an agent reaches no credentials.
• Sandboxed code: Agent code runs only as Slurm jobs on the agent partition, in Apptainer containers, as an unprivileged user whose network access is limited to the data buckets.
• Licences: Every record names the licence of each source. Anything beyond reading in place would need the data owner's agreement.
• Adopting an upstream release: A new release becomes a change to an adapter's pinned manifest in a pull request. The contract tests run against it, and a new crosswalk is a reviewed table.
• Contract tests: They read small pinned metadata files and run in CI on every change.
• Public endpoint: It runs only in the AWS build. It is read-only and rate-limited, and it depends on no homelab service.
• Hand-off: The project is done when someone new stands up the AWS build from its infrastructure code and reruns one condition from the documentation alone.

Milestones

Risks

Questions for reviewers

1. Crosswalks between atlas versions: For MouseLight, is a crosswalk between region labels enough, or must traced points be re-registered between CCF 2.5 and CCF 3?
2. Join granularity: v1 joins units and cells that share a region. Should it also offer joins by distance, such as the cells within 100 µm of a channel, given the registration error of each dataset?
3. Precision: Is detecting differences of about 25 points on leg and join questions, and 31 on trap questions, enough for these decisions?
4. The AWS build: Should it run a Terraform-built Slurm cluster on EC2, a pattern Claude Science has already driven, or AWS Parallel Computing Service, AWS's managed Slurm?
5. An instructions-only condition: Should the experiment add a condition in which Claude is told to state its versions but has no tools? It would separate the effect of the instruction from the effect of the tools, for a quarter more attempts.

Supporting material

1. What an answer about the brain depends on: The versioned inputs behind an answer, in depth.
2. What already exists: The MCP servers and the assistant that already work with Allen data.
3. Same input, different answer: Documented version changes that moved results on identical inputs.
4. Fluent, confident, unverifiable: AI deployments in science and medicine that failed on what could not be checked.
5. Threat model: A data flow diagram with its trust boundaries, and the STRIDE threats against each element and crossing.

The full-screen data model and every verified case, with its source, are on the Data model and Evidence pages.