laya-typed-decisions

Non-autoregressive System 1 decision model, fine-tuned on the typed-decisions workflows: agent-trace observability, customer service, invoice processing and security incidents.

Part of the Laya family.

Benchmark

400 test cases, 2,000 decisions, measured on the official test split.

+3.9 points over Jev's published 0.727, above the 0.735 teacher ceiling, with 2.4x better Brier and 1.6x better score MAE.

Jev figures are third-party published, not measured here — there is no TypeSafe API access in this project, and sample sizes and prompts differ. Treat the comparison as indicative.

By workflow

By primitive

Quickstart

pip install laya

import laya

agent = laya.load("convaiinnovations/laya-typed-decisions")
result = agent.predict(state, questions)

Or route to it explicitly:

from laya import Router

router = Router()
router.predict(state, questions, model="typed-decisions")

Router will not select this checkpoint automatically unless you construct it with auto_task_detection=True — it is specialised to four synthetic workflows and should not be a silent default.

If this checkpoint is on a hot path, keep it resident rather than loading it per request:

router = Router()
router.preload(["typed-decisions"])          # or router.attach("typed-decisions", agent)

preload fetches and builds the checkpoints you name once; attach registers an Agent you already hold, so nothing is loaded twice.

Since laya 0.3.13, laya.load() builds the model without a throwaway random initialisation, so loading is about 10x faster with identical answers.

agent.predict_batch(states, questions) scores many states in shared forward passes, with answers identical to one call per state, which suits workflow triage over a queue.

If laya.load() hangs: transformers probes for TensorFlow at import, and when TF is installed its abseil runtime can deadlock model construction. Run with USE_TF=0.

Training

Fine-tuned from convaiinnovations/laya on the benchmark's 1,200-case training split (6,000 decisions) with RLCD: the policy reports a distribution, exploration adds zero-mean Gaussian noise to the logits, and the reward is a strictly proper scoring rule (log + spherical, plus ranked probability score for ordinal questions), so expected reward is maximised only by honest probabilities. Updates are REINFORCE with a group-mean baseline, alongside soft cross-entropy against the teacher's distributions.

Reproduce it: laya_finetune_typed_decisions_2xT4_kaggle.ipynb — about 4–5 hours on Kaggle's free 2xT4.

Limits

• This is a specialist. It was fine-tuned on four specific synthetic workflows. Expect it to behave like the base laya checkpoint, or worse, on anything else.
• Soft accuracy trails Jev (0.471 vs 0.580): its argmax is better, but its probability distributions match the teacher less well.
• Still over-confident (ECE 0.213 vs Jev's 0.144). Its temperature_by_options was inherited from the base checkpoint and overrides the per-type temperatures fitted for this model — refit on your own held-out data before relying on the probabilities.
• Its per-type temperatures were fitted on training data. The fine-tuning run fitted [1.0148, 1.0374, 1.0575] on a slice of the same items it had just trained on, which is why they sit so close to 1.0 (#186). The notebook now holds that slice out of training. Until this checkpoint is refit, treat its confidence as uncalibrated.
• English only. Use laya-multilingual for other languages.
• Keep choice questions under ~20 options. Options share a fixed 256-token head budget, so a large label space leaves few tokens per label and accuracy falls off sharply.

Links

• Hub / family https://huggingface.co/convaiinnovations/laya
• GitHub https://github.com/NandhaKishorM/laya · full benchmark data on the research branch
• PyPI https://pypi.org/project/laya/
• Benchmark dataset https://huggingface.co/datasets/LocalLLaMA/typed-decisions

Apache 2.0 · Convai Innovations