diversa-biomedical-v1

Diversa Biomedical Research Intelligence v1

A curated, normalized, enriched and packaged biomedical research intelligence dataset derived from OpenAlex metadata, provided in AI-ready formats (Parquet + JSONL) with full provenance and reproducible build tooling.

This is a derivative dataset: it is not a re-upload of OpenAlex. The value added here is biomedical filtering by taxonomy, cleaning, normalization, relational restructuring, quality control and derived analytical features.

What it is

• Source: OpenAlex work metadata, retrieved through the OpenAlex REST API.
• Scope: biomedical and health-related research only, selected by OpenAlex taxonomy IDs (fields 13, 24, 27, 28, 29, 30, 35, 36), not keyword search.
• Coverage: publication years 2015-2025.
• Selection: stratified random sample drawn per (year x field) shard with a fixed random seed, so every year and every biomedical field is represented proportionally and the result is reproducible.
• Records: 999,619 works, 999,619 unique OpenAlex IDs, 862,633 unique DOIs.
• Languages / OA: see metadata/dataset_statistics.json.
• Package size (uncompressed): 0.0B.

Intended uses

Machine-learning and analytical research on the biomedical literature: trend and growth analysis, collaboration-network analysis, bibliometric modeling, topic classification, retrieval/ranking experiments, and as a ready-to-load table for LLM and data-science tooling.

Formats and layout

data/       works + dimensions + relations (Parquet), JSONL exports, splits
analytics/  pre-aggregated time-series and collaboration tables (Parquet)
metadata/   manifest, statistics, schema, source queries, build info

Parquet is the primary format. JSONL exports (gzip) are provided for streaming/LLM use. See DATA_DICTIONARY.md for every column.

Key derived features

paper_age_years, citations_per_year, citation_velocity, citation_percentile (within year x field), high_impact_flag, publication_recency_score, international_collaboration, multi_institutional, research_growth_score, topic_growth_score, emerging_topic_flag. Definitions are in METHODOLOGY.md.

Train / validation / test splits

Chronological to avoid leakage: train <= 2021, validation [2022, 2023], test >= 2024. The complete unsplit dataset is retained separately.

Limitations

See QUALITY_REPORT.md. In short: metadata-only (no full text), citation counts are OpenAlex's at retrieval time, institutional/author disambiguation is inherited from OpenAlex, and derived growth signals are estimates that scale with sample size.

Rights

Base metadata originates from OpenAlex (CC0). Derived fields and packaging are original. No copyrighted full text, figures or supplementary files are included. See LICENSE_AND_RIGHTS.md.

Reproduce

pip install -r requirements.txt
python run-all.py

Retrieval dates and exact filters are recorded in PROVENANCE.md and metadata/source_queries.json.

Generated 2026-10-06 by pipeline v1.0.0.

Loading this dataset

Parquet is the primary format. Load the full table directly:

from datasets import load_dataset

works = load_dataset(
    "parquet",
    data_files="hf://datasets/skyleee/diversa-biomedical-v1/data/works.parquet",
    split="train",
)
# relation / dimension tables
work_authors = load_dataset("parquet", data_files="hf://datasets/skyleee/diversa-biomedical-v1/data/work_authors.parquet", split="train")
authors      = load_dataset("parquet", data_files="hf://datasets/skyleee/diversa-biomedical-v1/data/authors.parquet", split="train")

The chronological train / validation / test splits are gzip JSONL under data/ (biomedical_works_train.jsonl.gz, ..._validation.jsonl.gz, ..._test.jsonl.gz); datasets does not auto-detect the .gz extension, so read them explicitly, e.g.:

import gzip, json
with gzip.open("data/biomedical_works_train.jsonl.gz", "rt") as fh:
    records = [json.loads(line) for line in fh]