bargein-classifier-ettin-150m

When a caller starts talking while a voice agent is still speaking, should the agent stop?

This model makes that decision from text, in a few milliseconds. It reads one moment of a call: what the agent had already said, what it was about to say, and the caller's speech-recognised words. It then returns the probability that the agent should stop (INTERRUPT) or keep talking (CONTINUE).

The CPU-friendly model: 38 ms per decision on 4 CPU threads. It matches the 400m on held-out business types and real calls, and gives up about 2 points under distribution shift.

Live demo · 400m model · Collection

Have a GPU? bargein-classifier-ettin-400m is the flagship: more robust to distribution shift, at 8.5 ms on a GPU.

Why barge-in matters

Callers talk over voice agents all the time, and most of it should not stop the agent: "mm hmm", "okay", a cough, a word to someone in the room. Some of it must stop it at once: "wait, which doctor?", "that's the wrong address", "stop calling me". Rules based on voice activity or keywords get this wrong in both directions. An agent that stops on every "uh huh" feels jumpy and loses its place, and one that talks over a real question feels rude.

The right answer depends on what the agent was saying when the caller spoke. "Yes" right after a question the caller heard is an answer (INTERRUPT); "yeah" in the middle of a statement is a backchannel (CONTINUE). That is why the model reads the agent's line split at the moment the caller started talking.

Input format

format_input.py, with textnorm.py, turns the three fields into one string. It is the exact code used in training:

[BOT SAID] your appointment is on tuesday at
[BOT NOT YET SAID] three pm with dr . lee . does that still work ?
[CALLER] wait which doctor

• Normalisation: text is lowercased and numbers are written as words. Agent punctuation is kept as tokens, because a question mark is a strong cue. Caller text is mapped to the ASR's output style.
• Length: the budget is 192 tokens. If the input is longer, the formatter trims agent text and never the caller's words.
• Always build inputs with format_input.py. A hand-built string is a train/inference mismatch. The inference widget on this page takes an already formatted string.

Labels

Quickstart

pip install torch "transformers>=4.51" huggingface_hub

import sys
from huggingface_hub import snapshot_download

path = snapshot_download("ozonetg/bargein-classifier-ettin-150m")
sys.path.insert(0, path)
from predict import BargeInClassifier       # predict.py ships in this repo

clf = BargeInClassifier(path)                # GPU (bf16) if available, else CPU (fp32)
clf.predict(agent_said="Your appointment is on Tuesday at",
            agent_unsaid="three pm with Dr. Lee. Does that still work?",
            caller_said="wait which doctor")
# {'label': 'INTERRUPT', 'p_interrupt': 0.99..., 'model_called': True}

python predict.py in a local copy of the repo runs the built-in examples. Use --agent-said, --agent-unsaid and --caller-said to classify your own input.

import sys, torch
from huggingface_hub import snapshot_download
from transformers import AutoTokenizer, AutoModelForSequenceClassification

path = snapshot_download("ozonetg/bargein-classifier-ettin-150m")
sys.path.insert(0, path)
import format_input                          # the training-time formatter

tok = AutoTokenizer.from_pretrained(path)
model = AutoModelForSequenceClassification.from_pretrained(
    path, dtype=torch.float32).eval()        # transformers < 4.56: torch_dtype=...; on 4.x add reference_compile=False

text = format_input.build_input(
    "Your appointment is on Tuesday at",
    "three pm with Dr. Lee. Does that still work?",
    "wait which doctor",
    count_tokens=lambda s: len(tok(s).input_ids), max_tokens=192).text
enc = tok(text, truncation=True, max_length=192, return_tensors="pt")
with torch.no_grad():
    p = model(**enc).logits.softmax(-1)[0, 1].item()
print(model.config.id2label[int(p >= 0.5)], round(p, 3))   # INTERRUPT 1.0

import { AutoTokenizer, AutoModelForSequenceClassification } from "@huggingface/transformers";
import { buildInput } from "./format_input.js";    // from this repo

const id = "ozonetg/bargein-classifier-ettin-150m";
const tok = await AutoTokenizer.from_pretrained(id);
const model = await AutoModelForSequenceClassification.from_pretrained(id);   // fp16 ONNX by default

const { text } = buildInput("Your appointment is on Tuesday at", "three pm with Dr. Lee. Does that still work?",
  "wait which doctor", { countTokens: (s) => tok.encode(s).length, maxTokens: 192 });
const { logits } = await model(tok(text, { truncation: true, max_length: 192 }));
const [c, i] = logits.data;
const p = 1 / (1 + Math.exp(c - i));
console.log(model.config.id2label[p >= 0.5 ? 1 : 0], p.toFixed(3));   // INTERRUPT 1.000

Examples

These are this model's outputs on the demo inputs.

Recommended runtime use

1. Treat an empty transcript as CONTINUE without calling the model. On real calls, 44% of the moments when the caller made sound over the agent had no recognised words (breath, noise, a cough). Training data never had an empty caller turn, so if you do call the model on one, it stops the agent 13.1% of the time. predict() applies this rule for you.
2. Use the 0.5 threshold. Every number on this page uses it, and thresholds tuned on small human-labelled sets were unstable. The outputs are sharp: about 1% fall between 0.2 and 0.8.
3. Run it on each new ASR partial or final while the agent is speaking. One reasonable pattern is to debounce partials, then act on the first result ≥ 0.5 or at the end of the caller's speech. That loop is a runtime design choice and was not evaluated here.
4. Feed it the right inputs. - agent_said is the TTS text actually played, sliced by word timestamps, with its original casing and punctuation. Digits, $ and a.m. are fine. - agent_unsaid may be empty, or partial while the LLM is still streaming. - caller_said is the raw ASR text. Training covered caller turns of 1 to 49 words (median 5).

Evaluation

All accuracies use threshold 0.5 and come from the single published model of each size. Both models are shown, so you can choose between them. Except for the human-labelled set and the real-call rule cases, test labels come from the same LLM-judge pipeline as the training labels (see Data). Those scores therefore measure agreement with the labelling rubric on inputs the model has never seen.

Unseen business types and human labels

On the human-labelled set, one case is worth 0.63 points, so the two sizes cannot be told apart there. The rubric was written with these human labels in view, so read this score as optimistic. The human labels themselves disagree with each other on about 5% of such cases.

Robustness

Distribution shift (OOD-10k)

This test has 10,888 new exchanges, generated and frozen before any model was scored. They cover ten deliberate shifts of about 1,090 exchanges each.

Real calls

This test uses 2,525 barge-in moments from 1,477 real outbound sales calls from a single campaign. - Sampling: 2,000 calls were drawn at random, stratified by week and call length, with at most 2 moments kept per call. The agent's own echo on the caller channel was removed. - Transcription: both channels were transcribed by the telephony ASR. The agent's line was split into heard and not-yet-said at the caller's onset, using word timestamps, and its punctuation was restored by a simple rule. - Privacy: no call content left the servers and no humans labelled these moments. There are two references instead.

• Intervals: 95% bootstrap intervals over calls.
• The judge is a second opinion, not ground truth. It agrees with the human labels 97.5% of the time, one case below the models.
• Clear cases are right. The 400m keeps talking over 209 of 213 pure backchannels and stops on 28 of 29 explicit stop requests.
• Agreement holds where the judge is confident. There it is close to the models' score on human labels, so real recogniser text does not break them.
• Most of the remaining gap has two causes the synthetic data never had: empty transcripts, which runtime rule 1 handles, and a lone "hello" while the agent's opener plays. The 150m stops less often than the 400m on the opener "hello" (52% vs 79%), which is why its all-moments agreement is higher.

Latency

• Setup: batch 1, end to end per decision (formatting, tokenisation and the forward pass).
• Hardware: an NVIDIA RTX 6000 Ada GPU and an Intel Xeon Gold 5412U CPU.
• Run: both models were measured side by side in one run with exclusive use of the GPU. The 400m peaks at 0.75 GB of GPU memory.

Training

How R-Drop works: each batch is run twice with different dropout masks. The loss is the mean cross-entropy of the two passes plus α × the symmetric KL divergence between their predictions. The model is penalised when two dropout samples disagree, which makes it depend less on small details of the wording. That is also why it holds up better on noisy caller text.

Why the 150m uses a lighter dropout. On the small model, the 400m's recipe (dropout 0.1 + R-Drop) improved almost every test but broke one behaviour: telling a caller who is leaving from one who says goodbye and then comes back (87% → 61% over 3 seeds). Dropout 0.02 with full-strength R-Drop keeps most of it (83.7% vs 87.0% for a plain fine-tune) while keeping the gains: distribution shift +0.55 pt (p < 0.001), third party +2.6 pt, TTS → phone → ASR +1.1 pt, all against the plain fine-tune over 3 seeds.

Data

The training data is synthetic. It contains no real call audio or transcripts, and it is not released.

1. Rubric. About 325 exchanges were labelled by a human, the voice-agent operator, under their own stop / keep talking policy. Where the labeller's own answers contradicted each other, the newer answer with a written note won. The labels were then distilled into a written rubric of interrupt and continue categories (see Labels).
2. Generation. An LLM wrote 70,609 exchanges from sampled specifications: - business type (166 of them); - agent moment (25: pitch, price reveal, yes/no question, appointment details, read-back, …); - caller act and who the caller is addressing; - persona, where the agent's line is cut, and caller length. About a third come as minimal pairs, where one small change flips the label.
3. ASR style. Caller text is written the way the target telephony recogniser prints speech: lowercase, no punctuation, numbers as words. The ASR noise mix is 75% clean, 21% light and 3% heavy errors.
4. Labelling. An LLM judge applied the rubric, with 167 of the human labels as worked examples. Under cross-validation it agrees with held-out human labels 96–97% of the time.
5. Quality control. - A second, different LLM blind-relabelled 40k rows, and every disagreement went to three adjudicators. - An ensemble of trained models flagged about 3% of rows as likely wrong, and those were dropped. - Blind audits estimate the remaining label error at 1–3%.
6. Result. 64,474 exchanges, 56.6% CONTINUE and 43.4% INTERRUPT. Caller turns are 1–49 words (median 5) and 87% are unique. Accuracy on human labels stopped improving at around 64–66k rows.

Limitations and risks

• English only, telephone style. Training data is US-English phone conversations. Other English locales held up on the shift test (90.2%), but other languages are out of scope.
• One exchange, text only. The model sees no dialogue history and no audio (prosody, loudness). It answers "should the agent stop now?", not "has the caller finished their turn?".
• Third parties are the weakest area. 82.5% (400m) and 75.4% (150m) on the shift test. An interpreter relaying the caller's answer is often missed.
• A lone "hello" while the agent's opener plays (usually the caller picking up) makes the model stop: this model stops on 52% of them on real calls, where the judge keeps going. The training data had no single pickup "hello", only "hello? hello?" (INTERRUPT). If your agent should talk over a pickup greeting, handle it in the runtime.
• Stopping rate. On real calls with words, this model stops the agent on 31.2% of caller turns; the LLM judge stops it on 22%. Most of the difference is the opener "hello".
• Short accept words right after a question. "okay" or "mm hmm" just after the agent asks something is read as an answer (INTERRUPT) by the rubric. Human labels split on this too.
• Leaving vs. goodbye-and-back. This model is weaker than the 400m on telling "I have to go" from a goodbye followed by more talk (87.8%, 36 of 41 on the shift test). Use the 400m if this matters.
• The policy is opinionated. The labels follow one operator's policy:
• voicemail greetings are INTERRUPT (low confidence);
• urgent talk in the caller's room that is not addressed to the agent is CONTINUE.

Check these against your product before use. - Not a compliance control. Explicit stop and do-not-call requests are INTERRUPT (28 of 29 on real calls), but opt-out handling should not depend on a classifier. Detect and log those requests separately. - Single real-call campaign. The real-call test covers one outbound sales campaign with a recorded-prompt agent. Its agent text came from ASR with rule-restored punctuation, not the exact TTS text. - Validate before going live. Check on a sample of your own calls, especially if your callers, recogniser or agent style differ from the above.

Citation

@misc{ozonetg2026bargein,
  title        = {Barge-in classifier for voice agents (Ettin-150m)},
  author       = {ozonetg},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/ozonetg/bargein-classifier-ettin-150m}}
}

@misc{weller2025seqvsseqopen,
  title         = {Seq vs Seq: An Open Suite of Paired Encoders and Decoders},
  author        = {Orion Weller and Kathryn Ricci and Marc Marone and Antoine Chaffin and Dawn Lawrie and Benjamin Van Durme},
  year          = {2025},
  eprint        = {2507.11412},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2507.11412}
}

@inproceedings{liang2021rdrop,
  title     = {R-Drop: Regularized Dropout for Neural Networks},
  author    = {Xiaobo Liang and Lijun Wu and Juntao Li and Yue Wang and Qi Meng and Tao Qin and Wei Chen and Min Zhang and Tie-Yan Liu},
  booktitle = {Advances in Neural Information Processing Systems},
  volume    = {34},
  year      = {2021},
  url       = {https://arxiv.org/abs/2106.14448}
}

Related

• bargein-classifier-ettin-400m: the flagship GPU sibling.
• Live demo: try both models.
• parakeet-tdt-ctc-110m-call-audio-adapter: the telephony ASR adapter whose output style the caller text follows.