granite-decisions-synthetic

Granite Decisions synthetic datasets

Original, deterministic English fixtures for Adam Pippert's personal Granite Decisions project. The original default config has 162 examples: 54 train, 54 calibration, and 54 test. These exercise the pipeline; they are not a representative quality benchmark.

Source and license

The source is the project's original template generator, published here as make_smoke_data.py, from release v0.1.0, commit 543345ea033370484ca226424afd73164d48ca35. All dataset content and the generator are MIT licensed; see LICENSE. No third-party dataset or model output was used to generate labels. Jev was used separately for evaluation, never as a source of training labels.

Schema

Each JSONL record has an id, state.request containing an English software-work request, and labels with three fields:

• route: code, retrieve, or review.
• python: Boolean indicating whether Python is explicitly named.
• scope: 0 for one function, 1 for one service, or 2 for several services.

questions.json defines the corresponding Choice, Noul, and Score questions. Labels are assigned by construction from the templates. Each split has 18 examples per route, 27 per Boolean value, and 18 per scope.

Reproduce

python make_smoke_data.py --out regenerated

The output directory must not already exist. This reproduces the three JSONL files and question definitions byte for byte. No model or network is required. SHA256SUMS records the published source-file checksums.

Limitations and intended use

Use these fixtures to test training, calibration, serialization, and evaluation plumbing. Templates are grouped by split, but vocabulary, languages, and topics overlap. Construction labels have not been independently human-adjudicated. Performance on these fixtures does not establish reliability on natural user requests, unseen domains, or adversarial inputs. Replace them with reviewed, use-case-specific data before making model-quality claims.

The separate calibration split is intended for fitting calibration parameters; keep the test split out of training and calibration. The project includes evaluation evidence and training instructions.

Score interval experiment (2026-09-17)

The score_interval_v1 configuration adds 2,816 original MIT examples: 2,048 train and 256 each validation, calibration, and test. Each record contains an observed integer count, a question with 3–7 interval descriptions, and the correct zero-based Score level. Labels are computed from the intervals, not from Jev or any language model. Rubric boundary sets are unique across all splits; instruction wording also varies by split. These are arithmetic and schema-following fixtures, not representative real-world safety judgments.

Reproduce without downloads:

python score-interval-v1/generate.py --synthetic-only --out regenerated-scores

The first/default configuration remains the original 162-row v0.1.0 fixtures. The new configuration is used in a bounded local ROCm training experiment; no improvement is claimed until evaluation is complete. External BANKING77 and BoolQ data used in that experiment are not included in this MIT dataset.

IT/business weekend pilot (September 18, 2026)

New config it_business_weekend_v2: 2,304 policy-based synthetic decisions plus 24 separate challenge cases, with a source ledger, reproducible generator and schema, quality report and weekend plan. Original fictional scenarios are MIT; cited works retain their own rights. No source articles, book chapters or X posts are used as training text.

Eight Choice, two Noul and two Score task families cover IT recovery/releases, security, vendor governance, experiments, procurement, cost evidence and delegated decisions. All linked variants remain in the same split. Policy instances are disjoint; templates and source inspirations are shared. This is an experimental pilot, not a validated real-world benchmark.

For this config, decode state_json, question_json and label_json with Python json.loads to recover typed native records. Jev was used only as a blinded quality check on 24 training examples, never to generate or replace their labels. The deterministic gate rejected 120 deliberately corrupted controls; that does not measure Jev's semantic false-accept rate. No independent human review or Granite performance test has yet been completed on this corpus.

Frozen IT/business v3 corpus (September 18, 2026)

it_business_weekend_v3 contains 5,740 training rows, 1,152 each validation/calibration/test, and 480 challenge rows. it_business_weekend_v3_sft contains the same records as conversational prompt/completion pairs using the actual numeric-code inference prompt. These are two formats of one corpus, not additional independent examples.

V3 expands to 24 main families and 48 substantive policies, with arithmetic, units, strict boundaries, ordered exceptions, proportionate currency scaling, three evidence renderings and genuine option-code permutations. Six new inspiration events and policy families are reserved for challenge only. Main split policy instances and scenario groups are disjoint; main policy structures and renderings are shared. V3 supersedes v2 for this domain: do not concatenate their training/evaluation folds.

Dataset card and reproduction · Source ledger · Quality report · Frozen hashes

Every record is deterministically checked; 96 preselected training rows received blinded Jev review. Its top label agreed on 95/96 and 91 passed the preset confidence gates. Five entire groups (20 rows) are published separately as quarantined, excluded from all configured splits. No labels were replaced by Jev. Semantic probes had 0/40 false accepts and 4/40 false rejects at the fixed threshold; these are authored probes, not real-world validation.

All 9,676 accepted SFT pairs passed pinned-Granite token-prefix/target checks. The data are frozen for experimental fine-tuning; no independent human adjudication or new Granite performance claim is made. Original synthetic data and code are MIT. Linked source works retain their own rights and are not included as training text. Publication uses Adam Pippert's personal account.

For native fields, decode state_json, question_json, and label_json. For standard SFT, use only prompt and completion as model input, with completion-only loss; group/family/source/label metadata must stay outside prompts. The native categorical objective and a standard SFT objective differ; see the card before selecting a training route.