siddhanto-bangla-english-banglish

Siddhanto — a Bangla / English / Banglish typed-decision model

Siddhanto (সিদ্ধান্ত, "decision") is a fine-tune of convaiinnovations/laya-multilingual (mmBERT-base, 322M params) for open-domain typed decisions over Bangla, English and Banglish (romanized Bangla / code-switched) conversational text.

It is not a fixed-label classifier. It implements Laya's typed choice / noul / score interface: at request time you supply the question and its options (any schema, any domain), and the model returns a calibrated probability distribution over exactly those options — no retraining or head-swapping needed to add a new label set.

from laya import Agent

agent = Agent("kazalbrur/siddhanto-bangla-english-banglish")
result = agent.predict(
    state={"message": "amar order ta ekhono pai nai, ki obostha eta?"},
    questions={
        "intent": {
            "type": "choice",
            "instructions": "What is the customer asking about?",
            "criteria": {
                "order_status": "asking where their order is",
                "complaint": "complaining about a problem",
                "refund_request": "asking for a refund",
            },
        },
        "escalate": {
            "type": "noul",
            "instructions": "Should a human agent take over this conversation?",
        },
    },
)
print(result["answers"])

Architecture

Training data

Fine-tuned on ~268k typed-decision examples drawn from:

• An internal 56-intent OTA (travel-agency) synthetic corpus
• Public Bangla intent datasets: BanglaAirlineIntent, BankChatIntent, bangla-ecom-voice-decisions (order-confirm / intent / escalation)
• LocalLLaMA/typed-decisions — English rehearsal data (choice/noul/score, teacher-ensemble soft targets)
• A curated pool of ~49 real-text sources (social media, e-commerce reviews, moderation/hate-speech datasets, task-oriented dialogue corpora, QA datasets) reframed as typed decisions, spanning Bangla, English and Banglish

Calibration

Post-training, per-type temperature scaling was fit by golden-section search on a held-out calibration split (temperatures bounded to [0.5, 5.0]):

Fitted temperatures: choice 1.454, score 1.448, noul 1.626.

Evaluation

Trained for 3 epochs (17,610 optimizer steps), selected on validation selection_metric = 0.654. Accuracy on the project's standing eval suites:

Compared against other models

Siddhanto was benchmarked head-to-head against the untuned base checkpoint and three independent third-party models — nafiullah/laya-multilingual-bn-ecom-voice (same architecture, narrow ecom-only fine-tune), cmul8-hf/nirnaya (Qwen3-4B + LoRA + custom decision heads), and VTXAI/VTX-JEV-3 (zero-shot static-embedding routing model) — across all 10 eval suites:

Bold = best score on that row. Siddhanto wins 7 of 10 suites outright and is within 0.03 on two more. The only clear losses are narrow specialist wins: nirnaya edges ahead on real_b2b (+0.03), and nafiullah/nirnaya tie just above Siddhanto on unseen_schema — all cases of a model doing better on a task closely matching its own narrow fine-tuning domain. VTX-JEV-3 (a zero-shot embedding-similarity router with no learned classifier, sub-millisecond inference) is last on every suite, confirming it optimizes for speed rather than decision quality.

¹ cmul8-hf/nirnaya cannot score omni_golden_eval at all: that suite's intent question has 54 options, and nirnaya's own formatting code silently truncates long option lists to fit its 1024-token budget, then indexes results using the original (untruncated) option count — an out-of-bounds crash on every single record. This is a hard architectural limitation of that model (confirmed by direct debugging), not a quality result, so it is reported as not-applicable rather than as a 0.0 accuracy.

Two internal production systems were also used as honest external yardsticks: on the same real B2B gold messages, an existing production intent classifier scores 0.503 vs. Siddhanto's 0.764; on a separate production e-commerce golden-eval file, that system's own specialist classifier (trained narrowly for that one domain) scores 0.764/0.766 (accuracy/macro-F1) vs. Siddhanto's 0.650 — Siddhanto is open-domain and was not fine-tuned for that specific schema, so this is an expected and honestly-reported loss to a narrow specialist, not a general capability gap.

Limitations

• Not a fixed-taxonomy classifier — every question's options are supplied at request time; there is no built-in label set.
• Loses to narrow, single-domain specialist models on their home task (see above). If you have a large amount of labeled data for one fixed schema, a dedicated fine-tune may still beat this general-purpose model on that schema.
• Real-traffic accuracy (real_zero_shot 0.707, real_b2b 0.764) is meaningfully below in-domain synthetic accuracy (0.889) — treat outputs on unfamiliar real-world text with appropriate caution, especially for high-stakes decisions (e.g. escalation).
• Choice questions with very large option sets (dozens of options with long descriptions) consume a large share of the 1024-token head budget; extremely large schemas may need chunking.

Credits

• Base architecture and runtime: Laya by NandhaKishorM, and the base checkpoint convaiinnovations/laya-multilingual.
• LocalLLaMA/typed-decisions (Apache-2.0) for English rehearsal data.

@misc{siddhanto-bangla-english-banglish,
  title     = {siddhanto-bangla-english-banglish},
  author    = {Kazal Chandra Barman},
  title     = {Siddhanto — a Bangla / English / Banglish typed-decision model},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/kazalbrur/siddhanto-bangla-english-banglish}
}