nairobi-longitudinal-healthcare-utilization

Synthetic population data for predicting healthcare demand across time.

100% synthetic. No real patient records. Every citizen, visit, condition, coverage record and household in this dataset was generated by the STRIPED DONKEY / FCTE city-twin synthetic data platform for the city of Nairobi. No real individual's data was used to produce it.

What is this?

A genuine balanced longitudinal panel: the same 10,000 synthetic Nairobi residents, tracked at 3 annual observation dates (2023-09-01 / 2024-09-01 / 2025-09-01 = Train / Validation / Test) — 30,000 person-period rows, 38 columns. Each row carries up to 24 months of prior health, insurance, household and employment history as of that date, plus what happened to that same person over the following 12 months. This links the same synthetic person across health, insurance, household, education and employment domains, over time — not a single disease table.

Marketplace question: Who is most likely to use healthcare — or require an inpatient admission — in the next 12 months?

• Look back: what prior health and life-history patterns preceded increased healthcare use?
• Understand now: which residents currently carry the greatest healthcare-utilization burden?
• Look ahead: who is likely to use healthcare, or need inpatient care, over the next 12 months?

from datasets import load_dataset
ds = load_dataset("STRIPEDDONKEY/nairobi-longitudinal-healthcare-utilization")
train = ds["train"]        # 10,000 rows, 2023-09-01
validation = ds["validation"]  # 10,000 rows, 2024-09-01
test = ds["test"]          # 10,000 rows, 2025-09-01

Targets

Full column-by-column roles (identifier vs. predictor vs. target, and which columns to drop) are in docs/DATA_DICTIONARY.md.

This is a real balanced panel — not three independent samples

Earlier previews of this dataset sampled each split independently, so the citizens in Train, Validation and Test only partially overlapped. This release fixes that. All 10,000 citizens in train.csv are the same 10,000 citizens in validation.csv and test.csv — verified identical, not approximate. This makes the dataset usable for genuine per-citizen longitudinal analysis (e.g. "how did this person's prior-encounter trend change from 2023 to 2024, and did that predict their 2025 outcome?"), not just three cross-sectional snapshots that happen to share a schema.

This does introduce a real, disclosed trade-off — read docs/LIMITATIONS.md before using this dataset, in particular: - a structural survivorship bias (anyone who died 2023-2025 cannot appear in this panel), and - a structural age floor (nobody born after 2023-09-01 can appear — the youngest citizens currently under ~2 years old are entirely absent from this sample).

Splits are temporal, not random

data/longitudinal_panel/train.csv       -- 2023-09-01 observation date (10,000 rows)
data/longitudinal_panel/validation.csv  -- 2024-09-01 observation date (10,000 rows)
data/longitudinal_panel/test.csv        -- 2025-09-01 observation date (10,000 rows)

Do not re-shuffle these into a random split — SplitName encodes real calendar time, and the whole point of this release is to measure how well a model trained on earlier periods generalizes to a later one. See temporal drift, below.

Temporal drift (disclosed)

Healthcare-utilization prevalence rises noticeably at the Test (2025) date:

This pattern is real (it also holds in the full 12.18M-row certified population) but it is a consequence of the synthetic health-visit generator's volume growing over its generation timeline (see HLT-VISITFREQ-001 in Limitations) — not a genuine Nairobi epidemiological trend. Treat this dataset as a temporal-generalization / distribution-shift benchmark, not as a source of real-world trend estimates.

Feature groups

• Demographics: age, sex, ward (WardId + human-readable WardName), sub-county, household size
• Household / economic: employment status (formal/informal), K-12 school-record existence, tertiary-enrollment history
• Insurance: coverage status as of the observation date, acquisition type, plan tier (79.1% of rows are uninsured as of their observation date — a measured synthetic-panel rate, not a real Nairobi estimate)
• Prior health history (12/24-month windows before the observation date): encounter counts, inpatient/emergency counts, diagnosis counts, prescription counts, chronic-condition count, recent maternity-record flag

Repository contents

README.md                      -- this file (dataset card)
LICENSE                        -- CC-BY-4.0
CITATION.cff                   -- how to cite this dataset
data/
  longitudinal_panel/
    train.csv                    -- 10,000 rows, 2023-09-01
    validation.csv                -- 10,000 rows, 2024-09-01
    test.csv                      -- 10,000 rows, 2025-09-01
companion/
  README.md                      -- describes the 3 files below
  health_encounters.csv          -- 39,207 encounter-level rows for these same 10,000 citizens
  patient_journeys.csv           -- 32,877-event longitudinal log for the 750 richest histories
  health_population_profile.csv  -- 94-condition synthetic prevalence table (population-level)
docs/
  DATA_CARD.md                   -- dataset summary, config, task, split methodology pointer
  FEATURE_DICTIONARY.csv         -- machine-readable column dictionary
  DATA_DICTIONARY.md            -- full column-by-column roles and usage notes
  CERTIFICATION.md               -- what was verified, and how
  GENERATION_AND_CURATION.md     -- how this data is generated, certified and curated
  SPLIT_METHODOLOGY.md           -- the temporal (not random) split, cohort-selection formula
  TEMPORAL_LEAKAGE.md            -- point-in-time feature guards, disclosed simplifications
  PROVENANCE.md                  -- source views, scripts, generation lineage
  LIMITATIONS.md                 -- disclosed limitations, all independently verified
  MODELING_GUIDE.md              -- recommended baseline, common mistakes, companion-file usage
  LINEAGE.md                     -- logical + physical lineage, what's certified vs. companion
  RELEASE_NOTES.md               -- what changed, release over release
notebooks/
  01_quickstart.ipynb             -- fastest path to loading the panel and inspecting targets
  02_citizen_health_journey.ipynb -- one real citizen's full 51-year trajectory, worked example
  03_ml_baseline.ipynb            -- a working baseline (logistic regression + gradient boosting)
                                      for both primary targets, with ROC-AUC / PR-AUC / Brier score
  04_cohort_utilization_trends.ipynb -- ward/coverage-level descriptive cohort analysis
metadata/
  dataset_manifest.json          -- full file manifest (hashes, bytes, row counts)
  release_manifest.json          -- release-level summary
  certification_summary.json     -- 16-gate certification summary
  grounding_manifest.csv          -- machine-readable grounding tiers
  sample_manifest.csv            -- pointers to preview samples
  sha256sums.txt                 -- flat hash list

A sibling package, huggingface_dense_encounter/, is a scaffold only — see below.

Companion files: the clinical system behind the panel

train.csv / validation.csv / test.csv summarize each citizen-period as longitudinal features and forward outcomes. The companion/ folder provides the encounter-level history, longitudinal patient journeys and population health profile for the same 10,000 citizens — join on CitizenId. Visit-derived targets and prescription counts reconcile exactly to the released panel. See companion/README.md for scope, column definitions and limitations.

Dense Encounter companion (not yet available)

A separate sibling package, huggingface_dense_encounter/ (ships alongside this package, not inside it), is planned to deliver a deeper, per-encounter-grain companion (symptoms, procedures, lab results, claims, facility detail). It currently ships as a scaffold only, with no citizen data — gated on a real, tracked platform WorkItem (HLT-TRACELINEAGE-001). See that package's own README.md for the full explanation; do not expect data there yet.

Known limitations (disclosed, not hidden)

See docs/LIMITATIONS.md for the full list, independently verified against the released files: structural survivorship bias, the age floor, temporal drift, the HLT-UTILCAL-001 external utilization-calibration gap, HouseholdSize missingness (7.6%), and the scope of what this preview's own certification does and does not cover.

Full commercial dataset

This Hugging Face preview is a genuine but small (30,000-row) balanced-panel sample. The full certified dataset (12,177,749 rows, all 3 observation dates, the complete resident population) is available as a STRIPED DONKEY commercial product (ml-10), which passed a 16-gate structural and statistical certification (BuildRunId=28, re-certified BuildRunId=29) — see docs/CERTIFICATION.md.

License

CC-BY-4.0. Fully synthetic — no privacy, consent or re-identification concerns, since no real individuals are represented.