nitzotz

What it is. Nitzotz reads a Hebrew message and answers questions you type about it: pick one of several options, give a score on a scale, or say yes or no to a claim. For every answer it gives a probability you can trust, so you know when it is sure and when it is guessing. It does not write text, so it cannot make things up. It runs on a normal laptop, with no internet connection and no cost per question.

What it is for. Deciding what to do with incoming messages: is this a scam, what kind of message is it, which department should get it, how urgent is it. It is not a chatbot and it is not built for long documents (see Limitations).

In numbers. On 298 Hebrew messages it says correctly whether a message is a scam 91.3% of the time. For context: 70% of those messages are not scams, so a model that always says "not a scam" would score 70.5%. The number that matters more is the ranking: it gives real scams a higher probability than legitimate messages 98% of the time (AUC 0.98).

What changed in version 1.1

The same model, with the same size, files and way of use. It was trained again with more data: reading questions, the same questions in many different wordings, and harder scam examples (see Training data). On the same frozen tests as version 1.0:

• Reading. The biggest change. Questions of the form "which of these is NOT said in the passage" were at the level of a blind guess (25.3%); now 51.9%.
• Scams. Better overall and on the hard cases (scams written to look legitimate, and real messages that look like scams). The scam threshold was chosen again on held-out training messages: 0.515 (version 1.0: 0.517).
• Wording. Asking the same question in other words now changes the answer much less often. In version 1.0 one everyday wording of the paying-customer question made it answer "yes" for all 30 tickets; now no wording gives one answer to every ticket. The urgency question is still unstable (see Limitations).
• No test got clearly worse (paired exact McNemar against version 1.0 on the same questions: no drop with p < 0.05).

Try it

Python (the laya library, pip install laya):

import laya
agent = laya.load("BrainboxAI/nitzotz")
q = {"scam": {"type": "noul",
              "instructions": "ההודעה מנסה לגרום לנמען ללחוץ על קישור, לשלם או למסור פרטים בלי סיבה לגיטימית.",
              "criteria": {"true": "כן, זה ניסיון מרמה",
                           "false": "לא, זו הודעה לגיטימית, גם אם יש בה קישור או בקשת תשלום"}}}
print(agent.predict("החבילה שלך מעוכבת. לשחרור שלם 12.90 בקישור", q)["answers"]["scam"]["noul"])

Output: 0.8638, the probability that the claim ("the message tries to make the reader click a link, pay or hand over details for no legitimate reason") is true.

The wording of the question matters. This is the exact question the scam test used, and the numbers on this card are for it. In our tries a shorter wording ("the message is a scam attempt") gave clearly worse answers, for example a high scam probability for a plain verification code. If you change the wording, check it on your own messages first.

laya.exe (the standalone binary from ggmlc releases, no Python needed). Download the Q8 file, start the daemon, then send one JSON request per line; each answer comes back on one line:

hf download BrainboxAI/nitzotz nitzotz-q8_0.gguf --local-dir .
laya.exe daemon nitzotz-q8_0.gguf --device vulkan
{"id":"1","state":"היי, הפגישה מחר ב-10 עדיין בתוקף?","questions":{"scam":{"type":"noul","instructions":"ההודעה מנסה לגרום לנמען ללחוץ על קישור, לשלם או למסור פרטים בלי סיבה לגיטימית.","criteria":{"true":"כן, זה ניסיון מרמה","false":"לא, זו הודעה לגיטימית, גם אם יש בה קישור או בקשת תשלום"}}}}

{"status":"ready","model":"laya"}
{"model": "nitzotz", "family": "nitzotz", "route": "forced nitzotz", "answers": {"scam": {"type": "noul", "action": {"act_probability": 0.6666}, "confidence": 0.9708, "noul": 0.0292}}, "usage": {"input_tokens": 66, "output_tokens": 0, "latency_ms": 239.876}, "id": "1"}

noul is the probability that the claim is true. Several questions in one request are answered together in one pass. Use --device cpu on a machine without a GPU.

Benchmarks

8 frozen test sets, 3,990 questions in total, locked by checksum before any training data existed. All three models answered exactly the same questions. The two reference points are the other open laya models that read Hebrew: RoeiG/laya-hebrew (licence CC-BY-NC-SA-4.0, non-commercial use only) and laya-multilingual (Apache-2.0).

Full table. Accuracy, the best in each row in bold. The last two columns say whether Nitzotz's difference from that model is real or could be luck (a paired exact McNemar test on the same questions): "better" or "worse" means p < 0.05, "tie" means the difference could be chance.

On Belebele reading comprehension Nitzotz scores 62.7%, below RoeiG/laya-hebrew (75.4%); Nitzotz is built for message decisions, not long-passage comprehension.

How to read it: - MASSIVE (voice commands): Nitzotz was trained on MASSIVE's training commands. The test commands are different, but written by the same people in the same style, so this test is easier for Nitzotz than for the others. - HeQ: Nitzotz was trained on HeQ's training passages. The test passages are different ones, and training passages that overlapped a test passage were removed. - The support-ticket rows have only 30 questions. No difference there is reliable. - The spam and scam test has a lean: 70% of its messages are not scams. The "chance" line (50%) is a coin flip, not the best blind strategy.

Why you can trust it

The probabilities mean something. Take all the answers where Nitzotz said it was about 70 to 80% sure, and count how many were right: the chart above does that for every confidence level, over 3,990 test questions. Where the dots sit above the line, Nitzotz is more often right than it claims (it is modest); where they sit below, it is over-confident. This is what lets you set thresholds (see the next section). The per-question-type temperatures were fitted on 4,550 held-out training items, never on the test sets (calibration.json).

It reads the text. With the message removed and only the question left, its accuracy falls to 70.5% on the scam question and 7.7% on message type. So the answers come from the message, not from the wording of the question.

It is fast on ordinary hardware. On one laptop (Intel Core Ultra 9 285H laptop, built-in Arc 140T GPU, Windows 11), one question at a time:

The short and long columns repeat one fixed input 30 times after 5 warm-up calls. Timings on this laptop change a lot from one session to another (an earlier measurement of the same setup was several times slower), so treat these as rough.

The GGUF files give almost the same answers as the Python model. Compared with the full-precision Python model on the CPU:

A gap of 0.01 means, for example, 0.83 against 0.84. The few questions where the top answer changes are ones where the two best answers were almost tied. Over both spam questions the Q8 file on the GPU is right 83.6% of the time, against 83.7% for the Python model; the F16 file is closer (83.7%).

Use it in your business

The idea ("Ramzor", traffic light). Every incoming message gets one or more quick questions. The probability decides what happens next:

• Green (confident it is fine): handle automatically, file it, tag it, route it.
• Yellow (not sure): send it to a person, or to a large language model if you use one.
• Red (confident it is a scam): block or quarantine it.

The drawing is a worked example on the 298 test messages. The upper threshold, 0.52, is the one chosen for the scam question on 1000 held-out training messages (never on the test); the lower one, 0.35, was picked by hand. With them, 207 messages go to green (11 of them are in fact scams), 5 to yellow (3 scams), and 86 to red (12 of them are in fact legitimate). So red should mean "quarantine and check", not "delete". Pick your own thresholds on a sample of your own messages, and decide how many mistakes in green and red you can live with. At the 0.52 threshold alone, the scam answer is right 91.3% of the time overall and 80.0% on the 65 hard cases (at 0.50: 91.3% and 80.0%).

The model is cheap enough to run on every message. The person (or the LLM) only sees the yellow part. Do not use it as the only line of defence for decisions that can hurt someone.

Training data and transparency

Two training stages, on a rented GPU:

1. Learning to read. HalleluBERT-large was first trained to find the answer to a question inside a passage, on 27,085 HeQ training questions (CC BY 4.0). Passages that overlapped a test passage were removed.
2. Learning to decide. A laya decision head was put on top and the whole model was trained on 106,723 items (102,173 for training, 4,550 held out to pick the best of 2 passes and to fit the temperatures).

• Synthetic messages (35,000 items). DeepSeek V4.1 Flash (MIT) wrote Israeli-style SMS, WhatsApp and email messages from a plan (intended label, topic, tone, varied fake phone numbers and links). DeepSeek and Gemma 4 31B (Apache-2.0) then each labelled every message on their own. A message was kept only if both agreed; the two models agreed on 94% of them, and 93.7% of the 37,429 written messages were kept. Both ran through DeepInfra (via OpenRouter), with data retention off and "thinking" mode off. The synthetic data is not published.
• New in version 1.1. 6,825 reading questions on Hebrew web passages from FineWeb-2 (ODC-By 1.0): "which of these is NOT true or NOT mentioned" (each paired with a positive question on the same passage), answers said in other words, and answers that need several sentences. DeepSeek V4.1 Flash wrote them and Gemma 4 31B checked each one twice, with the passage and without it; a question that could be answered without the passage was dropped. 6,000 messages that are hard to tell apart (disguised scams, and real messages that look suspicious), written by DeepSeek and kept only if both labellers agreed with each other and with the writer. 4,000 passages labelled with a topic, 575 support messages for the paying-customer question, and 4,750 repeated HeQ items so that the older skills are not lost.
• Written with GPT. Part of the training data was written with OpenAI's GPT, through our ChatGPT subscription: 376 alternative wordings of the questions and 34 sets of alternative answer options, used in 30,488 training items (including all 575 paying-customer items), and 120 short scenario outlines from which DeepSeek V4.1 Flash wrote 6,000 messages. GPT wrote no message and no label. In total 33,148 of the 106,723 training items (31%) use text written with GPT. GPT output is covered by OpenAI's terms of use, not by an open licence.
• Open data. HeQ v1.1 (CC BY 4.0; about half of its questions are on Geektime articles, shared by the HeQ authors under the same licence) and MASSIVE he-IL (CC BY 4.0), training splits only. FineWeb-2 Hebrew (ODC-By 1.0) passages for the new reading and topic items.
• No leaks from the tests. Every training item was compared with every test question; anything sharing an 8-word run with a test text was dropped, and so were MASSIVE commands equal to a test command. The new items were checked with stricter rules: every new passage against every test text (any shared 6-word run), and every new question, option and wording against every test question and every reworded test question (exact match and character similarity).
• Not used: no output of any other closed commercial chatbot, no DICTA model, no non-commercial or share-alike data. GPT, a closed commercial model, was used only as described above.
• Size: encoder 357.1M parameters (HalleluBERT-large, MIT, fine-tuned); decision head 26.5M parameters, trained from scratch.

About the tests.

Caveats that change how much to trust the numbers: - The spam and business test was written by an AI model and checked by an AI model, not by a person. Real inboxes will look different. It was frozen after that review (4 labels changed, 2 messages removed). - Three training runs, three random seeds; this is one of them. It was picked on held-out training items (95.8% against 94.9% and 95.3%), not on the tests. The three runs are close on the large tests: scam or not 91.3% here against 91.3% and 90.9%, news topic 80.4% against 80.9% and 82.3%, reading 62.7% against 63.1% and 63.4%. This run and each of the other two give the same answer on 90.6% to 91.0% of all test questions. On the 30-ticket tests they differ more: ticket urgency 80.0% here against 66.7% and 66.7%. - The training labels come from two AI models. Where both are wrong in the same way, Nitzotz learned their mistake. - HeQ's wrong answers in the test were picked by code, not checked by a person.

Limitations

• Reading comprehension of longer passages is better, but still limited. Belebele: 62.7%, where a blind guess gets 25%, and still below the best other laya model on this test (see Benchmarks). When the right answer is written word for word in the passage it gets 79% (252 questions); when the answer is said in other words it gets 56% (648 questions); on "which of these is NOT" questions 52% (158 questions). Do not ask it whether a long document supports a claim.
• Checking an answer against a passage (HeQ) works. 94.8% with the passage; with the passage removed it falls to 50.3%, a coin flip. So on this kind of question it really reads the passage.
• Numbers, dates, amounts and rules: not trained and not measured. Compute them in code and pass the result in.
• Hard cases are still the weak spot: scams written to look legitimate (a "supplier" changing bank details, the "CEO" asking for a transfer) and real messages that look like scams (a real bank alert with a link, a real verification code). On the 65 hard cases the scam question is right 80.0% of the time (always answering "not a scam" there would give 70.8%), with AUC 0.88. At the chosen threshold it still misses 5 of the 19 scams written to look legitimate, and flags 8 of the 46 real messages that look like scams.
• Urgency is subjective, and its answer depends on the wording. Even the two teacher models matched the intended urgency only about 62 to 64% of the time. When the urgency question is asked in other words, the answer changes on 43% of the 30 test tickets.
• The in-house tests and much of the training data were written by AI models, not by people. This includes the spam and business test, the support-ticket test and the new reading questions. Real messages will look different.
• The support-ticket tests are small: 30 tickets per question, so one ticket moves a score by 3.3 points.
• Sarcasm and irony are probably read literally. Not measured.
• 512 tokens (roughly 300 to 400 Hebrew words) per question. A longer message is cut from the end without a warning.
• Hebrew only. Not trained or tested on English or Arabic.
• Not a safety system on its own. It makes mistakes in both directions. Keep a person in the loop for anything that can hurt someone.

Licence and attribution

Apache-2.0 for the weights, the GGUF files and the code. Commercial use is allowed. Built on:

• HalleluBERT-large: the encoder, MIT.
• laya (NandhaKishorM, Convai Innovations): the decision-head architecture and the runtime, Apache-2.0. No laya weights are used.
• HeQ: CC BY 4.0, by Webiks for MAFAT and the Israeli National NLP Program (NNLP-IL); includes Geektime passages.
• MASSIVE: CC BY 4.0, Amazon (FitzGerald et al., 2022).
• DeepSeek V4.1 Flash (MIT) and Gemma 4 31B (Apache-2.0), used through DeepInfra as data writer and labellers.
• FineWeb-2 (Hebrew): ODC-By 1.0, Hugging Face; the passages of the new reading and topic items.
• OpenAI GPT, through a ChatGPT subscription (OpenAI terms of use): question wordings and scenario outlines, as described in the training data section.
• ggmlc for the GGUF files.
• Belebele and SIB-200 (CC BY-SA 4.0) were used only to test, never to train.

The full notice is in NOTICE.

import laya
agent = laya.load("BrainboxAI/nitzotz")
q = {"scam": {"type": "noul",
              "instructions": "ההודעה מנסה לגרום לנמען ללחוץ על קישור, לשלם או למסור פרטים בלי סיבה לגיטימית.",
              "criteria": {"true": "כן, זה ניסיון מרמה",
                           "false": "לא, זו הודעה לגיטימית, גם אם יש בה קישור או בקשת תשלום"}}}
print(agent.predict("החבילה שלך מעוכבת. לשחרור שלם 12.90 בקישור", q)["answers"]["scam"]["noul"])

hf download BrainboxAI/nitzotz nitzotz-q8_0.gguf --local-dir .
laya.exe daemon nitzotz-q8_0.gguf --device vulkan
{"id":"1","state":"היי, הפגישה מחר ב-10 עדיין בתוקף?","questions":{"scam":{"type":"noul","instructions":"ההודעה מנסה לגרום לנמען ללחוץ על קישור, לשלם או למסור פרטים בלי סיבה לגיטימית.","criteria":{"true":"כן, זה ניסיון מרמה","false":"לא, זו הודעה לגיטימית, גם אם יש בה קישור או בקשת תשלום"}}}}

{"status":"ready","model":"laya"}
{"model": "nitzotz", "family": "nitzotz", "route": "forced nitzotz", "answers": {"scam": {"type": "noul", "action": {"act_probability": 0.6666}, "confidence": 0.9708, "noul": 0.0292}}, "usage": {"input_tokens": 66, "output_tokens": 0, "latency_ms": 239.876}, "id": "1"}