How well does AI understand healthcare facilities?

Medical AI answers "what is the appropriate treatment for condition X". Healthcare system intelligence answers a different question: which facility performed this procedure, what did its authoritative data show, during which reporting period, under what measure, and what can legitimately be concluded from those observations. zBench tests the second question. The first vertical is IVF: 5,144 tasks generated from CDC's clinic level ART file, every answer key re derived from a named row, scored against GPT-5, Gemini and Claude.

The nine layer ladder. Layers 1 and 2 are retrieval; layers 3 to 9 are the claim: misses that survive perfect retrieval. This page covers layers 1 to 7. Layer 9 has its own page; layer 8 is in preparation with physicians.

The gap

Each model answered the same questions twice. Browsing: web search on, the way a patient or an analyst would ask. Record: the CDC row served to the model as a tool (find the clinic, get the record, get the profile), so the model has only to read it. The two conditions differ in tool access, so the difference between the columns is an evidence access gap: what a model does with the authoritative row in hand against what it does when it has to reach that row itself. It does not, by itself, assign each miss a single cause; the failure modes below say what the misses look like.

Exact = graded Pass. An abstention ("CDC does not publish that cell", when it does not) is neither a pass nor a miss; it is counted separately. Runs: browsing: run 5, October 10, 2026 · record: run 8, October 11, 2026 · layer 7 browsing: run 12, October 13, 2026 · layer 7 record: run 10, October 13, 2026 · released set, record: run 9, October 11, 2026. Grader 5k: the harness version every stored run is relabeled under. A release carries the grader version at its freeze (zbench-ivf-1.2: 5i); 5j and 5k changed the harness only, no item, key or rule in any release changed.

By layer

Layers 1 to 6: 179 items of the ivf-v1 set, every item in both conditions. Layer 7: 30 items of the released set (ivf-v2), the same items in both conditions. Abstentions while browsing: GPT-5 13, Gemini 0, Claude 173. Layer 7 with the record: GPT-5 30 Pass, 0 Half, 0 Fail of 30; Claude 28 Pass, 2 Half, 0 Fail of 30. Released set, layers 1 to 6, with the record: GPT-5 28 Pass, 2 Half, 0 Fail of 30; Claude 27 Pass, 3 Half, 0 Fail of 30. Layer 6 is graded to CDC's standard: a bounded pick with the small sample caveat is a Half, not a Pass (with the record: GPT-5 24 of 29, Claude 20 of 29 answers graded Half); exact requires declining to rank.

Two answers, verbatim

Clinic names are withheld on this page; every released item names the clinic, the CDC dataset id and the row. The answers are reproduced as returned (a markdown link shows as its text), with the grader's label under each.

What the models do when they miss

Every failed answer carries a category and, when the wrong number can be found on the table, the row it came from: another reporting year, another age band, another measure, or the national row. That label is deterministic: the grader compares the figure against every neighbouring cell of the same clinic and against the national cell.

A data trap any lab can reproduce

CDC's clinic level file on data.cdc.gov carries two value columns. The printed value is text; the numeric column is what a programmatic reader takes. For every cell CDC prints as a fraction (a denominator under 20) the numeric column reads 0, so a reader that takes the numeric column scores every small clinic band at 0%. About 150,000 rows per reporting year are affected. zBench items for those cells grade "0%" as a Fail and "7 of 16, 43.8%, small sample" as a Pass.

What the models cited while browsing

Both browsing runs together (7 layers). A citation is any source the answer names; the record condition cites only the three tools. Gemini's citations arrive as grounding redirects (vertexaisearch), so the sites behind them are not visible.

What zBench IVF is

zReports assembled authoritative healthcare facility tables and built a system that turns them into traceable evaluation tasks at scale. The released benchmark is the proof; the generator, the ground truth tables, the provenance and the grading are the asset. Three numbers, kept apart:

Every item is generated, never hand written: a template picks a clinic, a measure, an age band and a reporting year from the tables, writes the question, and writes the answer key, the pass, half and fail rules and the row locator from the same cells. The same seed reproduces the same items; a new seed on untouched cells produces a fresh set that no model has seen. Layers 3 to 7 are built the same way from joins across cells: the national row, the neighbouring years, a second clinic, the clinic's profile record.

Projection: our own estimates from the row counts of each loaded table, before validation of reporting periods, missing values and deduplication per category. Only the IVF figure stands behind a release.

Releases and downloads

A release is a frozen copy: the items as generated, with a manifest that records counts, CDC dataset ids, the grader version and the SHA-256 of each file, so a download can be checked against it. A release id is never reused and never re frozen; the live set keeps evolving into the next id. The public split downloads here without a key; the held out split is withheld. The grader column is the version at freeze; the model results above are relabeled under the current harness version (Grader versions, below).

public.jsonl

one item per line: id, layer, question, answer key, pass, half and fail text, the grading rules, the probe (failure mode) label, the clinic's own cells and the row locator (ClinicId, dataset, filter, question, breakout) that re derives the key from CDC's file.

neighbours.jsonl

for each public item, the neighbouring cells of the same clinic with their substitution labels: what the grader compares a wrong figure against.

held out split

1,005 items withheld in full, questions and keys. The manifest records the hash of the held out question file, so a graded submission can be tied to the exact set; the questions are sent to a lab for a private evaluation (below), the keys never leave the server, and held out scores are never published here.

manifest.json

counts by layer, form, template and probe; CDC dataset ids per year; grader version; license; SHA-256 per file.

Mirror and code. The current release is mirrored on Hugging Face under release prefixed names (zbench-ivf-1.2-public.jsonl and so on), with the same SHA-256 digests. The grader, a Python port checked answer by answer against the reference grader on 717 stored answers, an Inspect AI task and a HELM scenario are published at github.com/zReports-Health/zbench under Apache-2.0; the item set keeps the terms below. A static leaderboard carries the results table.

Method

Source

CDC's National ART Surveillance System clinic tables, reporting years 2020 to 2022 (Success Rates, Services and Profiles, Patient and Cycle Characteristics), loaded as published. Each clinic year holds every measure for both patient types and every age band, with the national row beside it. Source and year are held constant within an item.

Conditions

Browsing: the model's own web search, no other tool. Record: three tools over the same tables (find_clinic, get_record, get_profile), the record feed that will be offered to AI systems as zAPI. The record condition is the control for access, not a controlled experiment on causes: the two conditions differ in tool access and in what is answerable with a tool, so their difference is read as an evidence access gap. Every item ran in both conditions with the same prompt and the same grader.

Grading

Grading is deterministic. Each item carries its own rules, written by the generator from the cells: pass (every pass pattern present), half (the figure without its qualifier, or the qualifier without the figure), fail, and hard fail (a figure the record cannot support). Verdict order: hard fail → clarifying question or abstention → Pass → Half → Fail → Review. A Half that carries another row's figure is a Fail. An answer that stops at "CDC does not publish that cell" is an Abstain when the cell is published and a Pass when it is not. Review marks the answers no rule reached; a reviewer can override any verdict in the console, and the stored verdict records who set it.

Substitution labels. Every figure in an answer is compared against the clinic's neighbouring cells (other years, bands and measures) and the national cell. A wrong figure found on the table is labeled by the row it came from; a figure quoted as context ("the national rate was 37%") and labeled as such by the answer is attributed, not a miss; a figure found nowhere on the table is labeled unknown.

Standards, as stated choices

• Layer 6 does not rank. CDC's own standard: clinic figures are not comparable without the patient mix. Pass = both figures, the caveat, and declining to rank; a bounded pick with the caveat = Half; a ranking without the caveat = Fail.
• Cell forms are part of the answer. A fraction (denominator under 20), a suppressed count (*), a true zero and "no cycles" are four different records. "Not reported" for a printed fraction is a Fail; a rate computed from a fraction without the small sample note is a Half.
• Counts the record does not print are hard fails. A rate cell carries no per band count; stating one is a Fail unless the answer labels it as its own estimate, which caps the answer at a Half.
• Layer 9 prices are premises. The price per cycle is stated in the question; the model is never asked to know a price, only to multiply the record and to stop when the record stops (layer 9 page).

Grader versions

Rules live on the items. A grader change regenerates the live set with the same ids and seed, every stored run is relabeled, and the next freeze carries the new version. A harness only change (5j, 5k) relabels the stored runs without regenerating the set, so the frozen releases keep their stamped version. Released items never change.

Failure modes the ladder was built to catch

• Reputation over record. From memory and from the web, a well known clinic's rate is guessed above the record and a small clinic's below it.
• Substitution. A directory's or the national figure presented as the clinic's CDC cell; another year's cell presented as the year asked; SART's figure presented as CDC's.
• Invented sample sizes. A per band count stated where CDC prints a rate, or a fraction's denominator read as a count.
• Encoding. Fractions, suppressed counts and "no cycles" cells read as zero, as missing, or as a rate.
• Ranking. Two clinics ranked on figures CDC publishes with a do not rank notice.

A data finding

The numeric column of CDC's clinic level file reads 0 for every fraction cell (the table above). The loader behind zBench parses the printed value; suppressed cells load as null, never zero; a complete fraction loads as a rate with its numerator, denominator and a small sample flag.

Limits

Model runs are samples: 179 items for layers 1 to 6, 30 for layer 7, one run per condition. Gemini lost 101 of 179 answers in the record run to a daily request cap on our key; GPT-5 lost 1 of 30 answers in the layer 7 browsing run to a request timeout; those are reported of answered, not as misses. Verdicts are automatic unless a reviewer sets them; the agreement figure between the automatic grader and a hand graded run will be published with the next release. The held out split is never scored on this page.

Private evaluation

A lab or a healthcare AI team can send outputs for the public split, or ask for the held out questions and send outputs for those, and receive a graded run: scores by layer and by probe, every verdict, the failure taxonomy, the substitution analysis, the evidence analysis and the error corpus with each answer's reason. The held out keys stay on the server. For a repeat evaluation the generator takes a new seed and is checked against the cells and question patterns the releases have exposed, so a model can be fixed and tested again without re reading a public set. The same engine runs as an interactive environment: the record tools, provenance on every cell, the deterministic grader as the reward, about 80,000 gradable cells behind the release; environment pilots are arranged the same way. Write to shawn@zreports.org.