jev-opentwbench-results

--- license: cc-by-4.0 language: - zh tags: - benchmark - evaluation - taiwan - zh-tw - multiple-choice - twinkle-eval - opentwbench - self-reported task_categories: - question-answering pretty_name: Jev 1.13 results on OpenTWBench size_categories: - 10K<n<100K configs: - config_name: index data_files: - split: train path: results/index.jsonl - config_name: formosa_predictions data_files: - split: train path: results/jev-1.13.0/tw-formosa-bench/predictions_run.jsonl - config_name: exam_sample_predictions data_files: - split: train path: results/jev-1.13.0/tw-exam-bench-sample/predictions_run.jsonl

Jev 1.13 on OpenTWBench (self-reported)

Per-question predictions and summaries for TypeSafe Jev (jev-1.13.0, requested as jev-latest) on the two OpenTWBench sets:

Status: self-reported. This run was not executed by the OpenTWBench maintainers and the weights are not public, so it cannot be an "Official" leaderboard row.

Read this before comparing to other rows

• Jev is not a chat model. It is TypeSafe's "System One" classifier: it takes a state and a typed choice question and returns a probability for every option. Twinkle Eval's \boxed{X} / OpenAI-compatible harness cannot call it, so the runner in this repo (run_eval.py) was used instead. Nothing is generated and nothing is parsed, so the parse rate is 100% by construction and says nothing about format-following.
• Hosted API, not open weights. OpenTWBench's board records full-precision weights and where they were served; here the model is served by the vendor (api.typesafe.ai, POST /v1/systemone) and its precision is unknown.
• Prompt. One request per question. state = the question followed by A.–D. options; one choice question with criteria A–D and the instruction "Select the single correct answer to the multiple-choice question in the state." The predicted letter is the arg-max of probabilities.
• Option order. Formosa: run 0 uses the published order, runs 1–2 use a seeded shuffle (seed jev-otb-v1; the permutation is stored in each row as perm, and pred is already mapped back to the original letters). Exam sample: a single run in the published order, so no run-to-run variance is available.
• Prompt choice was tuned on a tiny pilot. Two phrasings (options in state vs. options as criteria descriptions) were compared on 155 evenly strided Formosa questions (72.3% vs 73.5%, within noise); the state form was kept. Those 155 questions are part of the final set.
• Formosa caveat inherited from the dataset card: its items were drafted by gpt-oss-120b, and the set was difficulty-selected against a frozen pool of open models; absolute numbers are not directly comparable to the exam sample.
• Vendor docs state English is Jev's primary language and CJK is "supported but less accurate", and that it tends to favour the first option. Predicted-letter distribution here is mildly skewed toward B (Formosa A/B/C/D = 27/30/22/21%, gold ≈ 25% each).

Calibration

Jev also returns a confidence. Over 10 equal-width bins: ECE = 0.054 (exam sample) and 0.145 (Formosa, over-confident). Full bin tables are in each results.json.

Layout

results/
  index.jsonl                                   one row per (model, benchmark)
  jev-1.13.0/
    tw-formosa-bench/
      results.json                              summary, per-category accuracy, calibration
      predictions_run{0,1,2}.jsonl
    tw-exam-bench-sample/
      results.json                              summary, per-subject accuracy (148), calibration
      predictions_run0.jsonl
run_eval.py                                     the exact runner (reads TYPESAFE_API_KEY from env)

Prediction row fields: qid (<set>/<category-or-subject>/<index or row_id>), qh (first 16 hex of sha256(question); for the exam sample this equals the dataset's own qh), group, run, perm (original option shown at A–D), gold, pred_shown, pred, correct, confidence, probs (keyed by original letter), input_tokens, output_tokens, model. Question and option text is deliberately not copied; join on qid/qh against the source datasets. In results.json, per-group n for Formosa counts question-runs (3 × questions).

Reproduce

python download.py            # fetches both sets (stdlib only)
export TYPESAFE_API_KEY=...
python run_eval.py --dataset formosa --runs 3 --out runs/formosa
python run_eval.py --dataset exam    --runs 1 --out runs/exam

Total usage for the published runs: ~12.1M input / ~1.2M output tokens, ~4 minutes wall clock at 24–32 concurrent requests; no request failed after retries.

Licensing

Predictions, summaries and code: CC BY 4.0. They reference but do not redistribute the Apache-2.0 exam questions or the CC BY-SA 4.0 Formosa questions.