Scientific-Statistics-QA

RegalFire Statistical methods QA

RegalFire — AI Data Foundry

89 compact source-grounded statistical QA records. Accepted answers are distinguished from community-score fallbacks.

Verified scope

Records: 89; distinct source threads: 89; unique answers represented: 130. Domain thread counts: {"statistics": 89}. Actual record splits: {"train": 59, "test": 11, "validation": 8, "holdout": 11}.

Each record includes a canonical thread, all source answers, author attribution, original HTML, mechanically converted text, revisions and raw hash/API provenance. record_id is the primary key; thread_id is site-qualified, while original numeric question/answer IDs are retained. Citation record counts count edges; answer/thread totals are deduplicated within the product.

Sources and licensing

Official Stack Exchange API 2.3 source snapshots from Cross Validated (stats), Computational Science (scicomp) and Biology (biology), as applicable to the recorded domain distribution. Native new capture windows used the 100 most recently created answered questions for Stats and Scicomp. Biology reuses a deterministic 100-thread sample from the previously validated local capture; the original public Biology repo is unchanged. After source and additional cross-post sensitivity exclusions, the family source corpus contains 288 threads.

All retained current question and answer contributions explicitly returned CC BY-SA 4.0. This is verified per post, not assumed for historical posts. Bionic raw contained 2.5/3.0 contributions but lacked sufficient independently captured provenance/revisions; those inputs are excluded. Historical revision license fields, when supplied, are retained. See SOURCE_LICENSE_AUDIT.json and official source licensing.

Keep question titles, Biology/Computational Science/Cross Validated source identity, contributor names and available profile links, original post URLs, license links and transformation notices when redistributing. Each thread includes revision contributors and their source identifiers. Null profiles mean unavailable/deleted information; no profile is invented. Share adaptations under the applicable same or compatible license and impose no additional restrictions. Packaging does not replace source ownership or licensing. Third-party quotations and referenced images/papers may have separate rights; external assets are not downloaded or repackaged.

Deterministic product rules

QA chooses an accepted answer when present, otherwise maximum community score >=1 with lowest answer ID as tie breaker. accepted_answer remains null for a score-based fallback. Ranking requires at least two answers. RAG signals describe counts, acceptance, scores, references and time spans: they are not semantic disagreement annotations, verified difficulty or correct-answer labels. Code selection uses preformatted blocks of at least 20 characters with a documented code-syntax screen. Language is explicitly unspecified; snippets may be incomplete/pseudocode and are not executed. Citation extraction uses literal external URLs/DOI-like strings; context is a literal ±100-character text window or null when only HTML contains the reference. No DOI resolution, paper download or citation correctness claim.

Transformations and privacy

No LLM rewriting. Original API HTML remains intact. Text mechanically decodes HTML entities, preserves source text, code whitespace and TeX strings, adds literal link destinations and image-reference markers, and flattens superscript/subscript formatting. Code blocks retain exact decoded preformatted contents with SHA-256. Treat HTML/links/code as untrusted; sanitize rendered HTML and never execute snippets without a sandbox.

Patient-specific/personal medical histories, direct treatment requests and email-like identifiers are screened across both questions and answers and excluded by an English heuristic. General scientific medical discussions can remain with sensitivity flags. This is not comprehensive PII detection or clinical review. No medical advice. Rejections contain IDs/URLs/reasons without the excluded bodies. SENSITIVITY_REPORT.json describes the family screen.

Splits and overlap

One shared source-thread grouping is used by all five family products: normalized title/body/exact answer hashes and exhaustive question word-trigram Jaccard >=0.80 unite obvious duplicates. SHA-256 buckets assign 70/10/10/10 train/validation/test/holdout; observed proportions vary. Every occurrence of a thread has the same split across products. Products overlap intentionally; do not add their volumes as unique source content or concatenate one product’s held-out records into another’s training. These are publicly available forum posts: holdout is not a contamination-resistant private benchmark and lexical checks do not guarantee absence of semantic paraphrases.

Validation and reproducibility

Independent source verifiers check schema, UTF-8, author/revision attribution, actual license fields, HTTP/raw hashes, HTML/text preservation, unique IDs, complete answers and accepted consistency. A separate product engine checks eligibility, code/citation extraction, source equality, duplicates, exact split files and leakage. Five product-specific validator entry points use this engine; no builder is imported. The family passed 15 controlled corruption tests and byte-identical regeneration. validation_report.json records this product’s results; ROBUSTNESS_REPORT.json records family tests.

Run python validate_dataset.py --source-root <scientific_product_factory> with the local capture archive and source pipelines. Full raw/source validation requires that archive. Public release files omit raw bodies containing excluded data; SOURCE_MANIFEST.json records their hashes, API URLs and collection times. Fresh API collection can differ from the original snapshot. MANIFEST.json hashes every release file except itself.

Intended use and limitations

Statistical search assistants, internal analytics documentation and provenance-aware QA. No expert scientific correctness labels, model accuracy claims, validated code execution, clinical review or resolved-citation guarantees. Community votes are not scientific truth. This is a bounded starter evaluation corpus; expansion depends on source completeness, licensing and available API quota. Statistical QA/code subsets are small and recent; do not treat them as representative of a whole research discipline.

Custom / Private Dataset Work

RegalFire builds custom AI datasets, evaluation sets and reproducible data pipelines for research and production systems.

Client-specific modelling domains, internal statistical documentation integration, custom schemas and private QA/holdout suites.

Contact: [email protected] Hugging Face: RegalFire