laya

Multilingual, non-autoregressive System 1 decision model. Give it a state (text, email, ticket, or JSON) and typed questions; it returns typed answers with mathematically calibrated probabilities in a single forward pass (~33 ms) across 100+ languages. Trained with reinforcement learning against strictly proper scoring rules (RLCD), so reporting honest probabilities is the only way to maximise reward. It never generates text, so there is nothing to parse and nothing to hallucinate.

This repo holds all three checkpoints and is the hub for the family. The English checkpoint is at the repo root; the other two are bundled subfolders, and only the one you request is downloaded:

What's new in laya 0.3.13

pip install -U laya for all of this; everything below is new since 0.3.6. The checkpoints themselves are unchanged.

• About 10x faster loading. Checkpoints are built without the throwaway random weight initialisation, so laya.load() drops from about 22 s to about 2 s on CPU, with bit-identical answers. This also skips the pass that crashed on Windows with Python 3.14.
• import laya no longer loads torch. Routing, language detection and e-mail cleaning work in lightweight processes.
• Batch scoring. agent.predict_batch(states, questions) scores many states in shared forward passes, with answers identical to calling predict one state at a time.
• Routed batches. Router.predict_batch(requests) routes each request, groups them by checkpoint and question set, and scores each group in shared forward passes, with answers identical to one predict call per request, including requests whose options arrive in a different order.
• Prediction hooks. Opt-in hooks run around every decision, to audit, trace, redact, cache or gate results. With no hooks set, answers are unchanged.
• Schema-driven decisions. agent.decide(state, schema=...) takes a JSON schema or pydantic model and returns typed values in one forward pass.
• Per-language calibration. lang_temperatures= applies your own per-language temperatures, and every answer also reports answer_confidence (max(p)) next to the unchanged confidence.
• Sturdier fast path. On a CUDA out-of-memory error the fast path switches off before falling back to CPU, and single-option choice questions work.
• transformers 4.x and Apple GPUs. Checkpoints load with the right RoPE settings on transformers 4.x, and predict() no longer crashes on MPS builds without an autocast backend.
• Faster paths, all opt-in. laya.load(..., fast=True) uses a TileLang GPU fast path that matches the stock bf16 forward within rounding. Agent(compile=True) enables torch.compile, and laya.onnx_agent.ONNXAgent runs an exported model on ONNX Runtime.
• Run it your way, locally. A self-hosted Jev-compatible HTTP server (pip install "laya[serve]", then laya-serve, see Self-hosting), a laya command for quick local tests, an optional MCP server (pip install "laya[mcp]"), LangChain and LangGraph integrations (pip install "laya[langchain]"), and laya-ts, a TypeScript package for Node and the browser that gives the same answers as the Python package.
• Better routing. Plain-ASCII Spanish, Italian, Portuguese, French and German, Brazilian Portuguese support text, CJK text with Latin brand names, romanized Bangla and Azerbaijani now reach the multilingual checkpoint. Checked on 20,000 English texts: at most 5 English sentences move, all quoting long native-script names.
• Router defaults and hooks. Router() keeps two checkpoints resident, you can pass your own language guess with lang_guess=, and Router/Agent work as context managers.
• Correctness and clearer errors. Long conversation lists keep the newest turn when truncated, non-ASCII instructions reach the model as text, and a malformed question is rejected with a message naming the question and what to fix. A noul criteria dict keyed anything other than true/false is rejected rather than silently replaced; use labels to change the wording.

Quickstart: Route Mode (Recommended)

Laya's built-in Router is the recommended way to use Laya in production. It evaluates any state in any language, automatically detects scripts and languages in sub-milliseconds, and dispatches to the optimal checkpoint in a single forward pass.

pip install laya

import laya
from laya import Router

# Preload checkpoints into memory for instant sub-35ms routing
router = Router(preload=True)

state = {
    "from": "[email protected]",
    "subject": "Duplicate charge on invoice #4411",
    "body": "Hi, we were billed twice for March. Please refund the duplicate today or we will cancel our plan."
}

questions = {
    "department": {
        "type": "choice",
        "instructions": "Which department should handle this request?",
        "criteria": {
            "billing": "invoices, payments, refunds",
            "technical": "bugs, outages, system errors",
            "sales": "pricing, new contracts",
            "other": "everything else"
        }
    },
    "urgency": {
        "type": "score",
        "instructions": "How urgent is this request?",
        "criteria": ["not urgent", "soon", "critical deadline or blocking issue"]
    },
    "churn_risk": {
        "type": "noul",
        "instructions": "Does the user threaten to cancel or leave?"
    },
    "refund_requested": {
        "type": "noul",
        "instructions": "Does the user explicitly request a refund?"
    }
}

# 1. English state -> automatically routed to ModernBERT-large (39.5 ms)
res_en = router.predict(state, questions)
print("Department :", res_en["answers"]["department"]["choice"])  # -> billing (confidence: 0.94)
print("Routing    :", res_en["routing"]["model"])                 # -> english

# 2. Hindi state -> automatically routed to mmBERT-base (100+ languages, 32.8 ms)
res_hi = router.predict({"body": "मुझसे दो बार शुल्क लिया गया, कृपया पैसे वापस करें।"}, questions)
print("Department :", res_hi["answers"]["department"]["choice"])  # -> billing (confidence: 0.86)
print("Routing    :", res_hi["routing"]["model"])                 # -> multilingual

# 3. Explicit override when you already know the checkpoint
res_td = router.predict(state, questions, model="typed-decisions")

Every result carries full routing metadata explaining why the choice was made:

res_hi["routing"]
# {
#   'model': 'multilingual',
#   'repo': 'convaiinnovations/laya/multilingual',
#   'reason': 'non-Latin script (devanagari, 100% of letters); the English checkpoint cannot read it'
# }

Why Route: The Evidence

On a shared benchmark (17,416 questions, one T4 GPU, identical questions per model):

The English checkpoint collapses on non-Latin scripts (Khmer scores 0.000 accuracy at 0.952 confidence). Because the model stays confident while being wrong, confidence gating cannot save you. Router detects the script in <0.5 ms pure Python before the forward pass.

Supplying your own language detection

If you already run a language-identification model, pass its answer instead of relying on the built-in heuristic. lang_guess takes a language code or a callable, is checked after an explicit lang= and before detection, and a callable that returns None falls through to detection:

router.predict(state, questions, lang_guess="ro")              # a code you already know
router = Router(preload=True, lang_guess=my_lid)               # or install one for every request

Production Preload & Memory

A cold checkpoint build costs seconds; language detection costs microseconds. Since laya 0.3.13 the lazy default keeps two checkpoints resident (english and multilingual, the only two automatic routing chooses between), so after each language's first load a switch costs detection only. A single-language deployment never builds the second. max_loaded=1 rebuilds on every switch (measured at a 7.4 s median reload on CPU and 10.3 s on T4).

For a server or a demo, preload:

# Every checkpoint resident in memory; language flips cost detection only (<1 ms)
router = Router(preload=True)
router = Router(preload=True, device="cuda")

# Or preload only the specific checkpoints you serve:
router.preload(["english", "multilingual"])

# If your app already built an agent, attach it to avoid duplicate VRAM:
router.attach("english", existing_agent)

# Manage resident memory (default keeps two hot: english + multilingual, LRU eviction)
router = Router(max_loaded=3)       # all three hot, e.g. with auto_task_detection
router = Router(max_loaded=1)       # memory-constrained host, reloads on every switch
router.unload()                     # free memory

with Router() as r:                 # releases the models when the block ends
    r.predict(state, questions)

Single-Model Mode (Direct SDK)

If you only need a single checkpoint for a dedicated pipeline:

import laya

# 1. Load from the repo root or subfolders (downloads only the requested weights)
agent = laya.load("convaiinnovations/laya")                           # English root (~808 MB)
agent_ml = laya.load("convaiinnovations/laya", subfolder="multilingual") # 100+ languages (~647 MB)
agent_td = laya.load("convaiinnovations/laya", subfolder="typed-decisions")

# 2. Run all questions in ONE single forward pass (~35 ms on GPU)
result = agent.predict(state, questions)
answers = result["answers"]

print("Department :", answers["department"]["choice"])   # -> billing (confidence: 0.94)
print("Urgency    :", answers["urgency"]["score"])        # -> 1.84 / 2.0
print("Churn Risk :", answers["churn_risk"]["noul"])       # -> 0.892 (89.2% probability)

If laya.load() hangs: transformers probes for TensorFlow at import, and when TF is installed its abseil runtime can deadlock model construction. Run with USE_TF=0.

Self-hosting: Jev-compatible HTTP server

laya-serve exposes the Router on the same POST /v1/systemone request and response shape as TypeSafe Jev, so existing TypeSafe clients work by changing their base URL:

pip install "laya[serve]"
LAYA_DEVICE=cuda LAYA_PRELOAD=1 laya-serve        # 0.0.0.0:8000, preloads the checkpoints

curl -s localhost:8000/v1/systemone -H 'Content-Type: application/json' -d '{
  "state": {"document": "I was charged twice. Please fix this ASAP."},
  "questions": {"billing": {"type": "noul", "instructions": "Is this ticket about billing?"}}
}'

It accepts every question shape the Jev API does (for example criteria as a list), ignores unknown fields, and returns a 422 naming the problem for a malformed question. It binds 0.0.0.0 with no authentication unless LAYA_API_KEY is set, in which case it requires Authorization: Bearer <key>.

Architecture

• Backbone: ModernBERT-large (395M, bidirectional, fully fine-tuned) + a decision head trained from scratch: 2 transformer layers, an option-marker scorer, and an act/escalate head. 421M total. (Multilingual uses mmBERT-base, 22 layers, 256k vocab, 322M total).
• Option markers: Every option is scored at its own [MASK] token, then softmaxed over that question's options. The answer space is defined at request time, so new schemas need no retraining.
• Budget: 512 tokens per question for English (head_max_len = 192); 1024 tokens for multilingual (head_max_len = 256).
• Batching: Every question in a call is answered in one single forward pass.

Training

RLCD (Reinforcement Learning for Calibrated Decisions). The policy reports a distribution; exploration adds zero-mean Gaussian noise to the logits; the reward is a strictly proper scoring rule (log + spherical, plus ranked probability score for ordinal questions). Expected reward is maximised only by reporting honest probabilities. Updates are REINFORCE with a group-mean baseline (GRPO-style). Multi-turn conversations use TD(λ=1.0) over prefix slices.

Benchmarks

Measured on a Tesla T4; every checkpoint answered byte-identical questions in the same run.

Speed

103–332 questions/sec batched on a single T4. For reference, TypeSafe Jev has been independently measured at 236–276 ms p50 (AbdelStark, nibzard), so Laya answers a single question roughly 6–8× faster.

Laya (with routing) vs TypeSafe Jev

Every Laya figure is what Router().predict(...) returns — the checkpoint the router selects for that input. Jev figures are third-party published, never measured here (no TypeSafe API access); sample sizes and prompts differ.

On DAIR Emotion, Jev assigned zero probability to the true label on 16% of examples.

Where Jev leads

• High-cardinality label spaces (>20 options at default settings): On Banking77, Jev scores 0.870 (on 72 labels) while Laya scores 0.425 (on 77 labels at default 256-token head budget). Options share a fixed head_max_len budget (192 tokens on English, 256 on multilingual), so 77 options receive only ~3 to 4 tokens per label, causing text to become indistinguishable. Jev supports up to 255 options out-of-the-box. While laya-multilingual supports 1,024 context (and up to 8,192 in the encoder) and you can raise agent.cfg["head_max_len"] = 512 at runtime, Jev is currently better suited for 50+ options in a single prompt without tuning.
• Soft distribution matching: On typed-decisions, while Laya achieves higher argmax accuracy (0.766 vs 0.727), Jev achieves higher soft accuracy (0.580 vs 0.471) against the teacher's full probability distributions.
• Out-of-the-box raw calibration: Before temperature scaling, the base checkpoint has higher raw ECE (0.213 vs 0.144). Laya achieves its 0.081 ECE after domain temperature fitting.

Full report: BENCHMARKS.md.

typed-decisions, measured on all three checkpoints

400 cases, 2,000 decisions, four workflows — measured here.

The fine-tuned checkpoint clears the teacher ceiling and wins all four workflows: invoice processing 0.804, security incidents 0.766, customer service 0.764, agent-trace observability 0.730. By primitive: noul 0.857, choice 0.733, score 0.723.

The base checkpoints sit below the majority-class baseline here — the capability on this benchmark comes from fine-tuning, which is what the fine-tuning notebook is for.

Honest Limits

• Base checkpoints are near chance on typed-decisions zero-shot — 0.362 here and 0.352 for multilingual, against a 0.318 random and 0.461 majority-class baseline. The 0.766 belongs to the checkpoint fine-tuned on that benchmark's own training split. Laya is a fast base to specialise, not a zero-shot decision engine.
• High-cardinality choice questions and token budgets: Sequences split into an option prompt budget (head_max_len) and the remaining document/state budget (max_len - head_max_len):
• laya (English) defaults to 512 context (head_max_len = 192, ~320 tokens for state).
• laya-multilingual and laya-typed-decisions default to 1,024 context (head_max_len = 256, ~768 tokens for state; mmBERT-base encoder supports up to 8,192 with RoPE). At default settings, a 77-option question like Banking77 allocates only (256 - 16) // 77 ≈ 3–4 tokens per label, causing accuracy to fall off sharply (0.425 vs Jev's 0.870). If evaluating 50+ options in a single question: 1. Raise agent.cfg["head_max_len"] = 512 and agent.cfg["max_len"] = 1024 (or up to 2048 / 4096 / 8192) so every option has enough tokens to remain distinct. 2. Or split large option sets into a two-step coarse-to-fine hierarchical choice.
• Ordinal score questions are the weakest primitive (SST-5 0.372).
• noul can follow its option labels instead of the state, most strongly on this English checkpoint. noul renders its two options as false: / true:, and here that label pair can dominate the answer, returning a confident "no" for clearly positive input (#156). Check noul answers on your own data. If they look stuck, ask the same question as a two-option choice with neutral keys and your yes/no wording as the descriptions:

python {"type": "choice", "instructions": "Is this review positive?", "criteria": {"A": "yes, the review is positive", "B": "no, the review is negative"}} - action.act_probability carries no usable signal yet (#185). It reads 1.0 for almost every input, and its raw logits run against correctness (AUROC 0.30 on 396 labelled decisions). Gate on confidence instead, which reaches an AUROC of 0.77 on the same items. - Ships over-confident: Refitting one temperature per (question type, option count) moves mean ECE 0.466 → 0.081 (laya) and 0.314 → 0.106 (laya-multilingual). Do this on your own data before trusting the probabilities. - English only on root: Use laya-multilingual for anything outside English.

Links

• GitHub: https://github.com/NandhaKishorM/laya
• PyPI: https://pypi.org/project/laya/
• Live Demo: https://huggingface.co/spaces/convaiinnovations/laya-demo
• Write-up: Read on Dev.to

Apache 2.0 · Convai Innovations