Data pipeline¶
In one paragraph, for a non-engineer
Nothing here is a real patient. The records are generated by an open-source simulator, and the health information the assistant cites comes from public, government-run pages that anyone can open. Both are rebuilt from scratch by one command; neither is stored in this repository.
Four sources¶
| Need | Source | Licence | In the repo? |
|---|---|---|---|
| Patient records | Synthea | Apache-2.0 | No — generated by make data |
| Lab reference ranges | data/lab_reference_ranges.csv |
Original work, each row cited | Yes, small |
| Education content | MedlinePlus + RxNav/RxNorm (NLM) | US Government work, public domain. Free, no key, no registration | No — fetched at build time |
| Safety policy rules | generate_policy_rules.py |
Original work, red flags cited | Generated |
Why the citations are real¶
Generating education text with a language model and attaching a citation_url
to it would produce an app whose citations point at sources that did not say the
thing being cited — in an app whose entire argument is that every claim carries
a citation. That is a category error, not a shortcut.
MedlinePlus Connect resolves a LOINC code to the matching lab-test page and an RxCUI to the matching drug page. The LOINC table is deliberately not a dependency: Synthea's observations already carry the code, so the licensed terminology download never enters the pipeline.
Where no vetted page exists, the assistant says so. It does not substitute a similar test and it does not generate content. In a citation-grounded system a visible coverage gap is a feature.
Two gaps Synthea has, and how they are filled¶
| Gap | Fix |
|---|---|
| No free-text clinical notes | render_notes.py composes them deterministically from the structured record using templates. Stated plainly as rendered, not authored |
| No reference ranges or flags | lab_reference_ranges.csv — small, hand-curated, LOINC-keyed, three bands per analyte, every row citing its published source |
Build¶
Five steps, in order: fetch the population, build the typed store, render notes, fetch education, generate the rule table, bind the scenarios.
The build fails loudly rather than degrading:
| Condition | Result |
|---|---|
| A reference band with no cited source or no verbatim quote | Build fails |
| A critical band asserted without a citation | Build fails — leave it empty instead |
| An escalation rule with no published source URL and quote | Build fails |
| An emitted citation URL that does not resolve | Build fails |
| More or fewer than one planted injection fixture | Build fails |
It never coerces a bad value into a plausible one.
What the build produced¶
| Measure | Value |
|---|---|
| Patients | 25, selected deterministically toward covered analytes |
| Rendered notes | 150, byte-stable for a fixed store |
| Reference-range rows | 27, every band quoted from its published source |
| Rows with a cited critical band | 8 |
| Safety rules | 14, every escalation rule citing and quoting a source |
| Education pages | 321 across 149 distinct URLs, all verified reachable |
| Declared coverage gaps | 18 |
See the data profile for the coverage chart.
Two honesty notes¶
Reference ranges are quoted, not paraphrased. Each row carries the sentence its band came from, so an answer can show a reader the published text rather than asking them to trust a number. Where no published threshold could be quoted — a critical high for sodium, a critical low for calcium — the band is left empty rather than estimated.
The education cache is a build artifact, not a corpus. NLM permits linking to and displaying what its services return, and recommends caching for 12–24 hours; it does not permit copying MedlinePlus pages. So the cache holds only returned fields with a retrieval timestamp, it is never committed, and the committed test fixture carries citation metadata with the page bodies stripped.