visrl-bench-results

VisRL benchmark eval results — VisPlotBench per-sample outputs

Companion to github.com/weixuzhang/visrl and weixu-zhang/visrl-data. This repo holds the per-sample outputs of VisPlotBench runs for 5 trained checkpoints, so the GPT-4.1 visual judge or the custom HTML/vegalite/lilypond stats path can be re-run from these artifacts without regenerating the model outputs.

Layout

{model}/
├── {lang}.json                                      summary stats from the bench
├── {lang}/
│   ├── results_{ckpt}_{lang}_head_0.json            per-sample model output (16-25 MB each)
│   └── benchmark_stat.jsonl                         GPT-4.1 judge mean/median/good-rate

5 models × 5 languages each:

Languages covered for each (where bench eval ran cleanly): python, html (per-sample only — bench's stats step crashes), svg, vegalite, lilypond. Mermaid + LaTeX + asymptote not yet included because the original bench host lacks the needed renderers.

A 6th model — phase3b_fixedrubric (the rerun on the new server with rubrics correctly nested under extra_info, judge upgraded to Qwen2-VL-7B) — has bench eval armed but waiting on free GPU; will be uploaded as phase3b_fixedrubric/ once it lands.

Notes

• Notebook outputs (*.ipynb with embedded base64 PNGs) are not included here — they're large and the same plots can be regenerated by re-rendering the per-sample code field. The summary {lang}.json and the per-sample results_*.json are sufficient for redoing the GPT-4.1 judge pass or computing custom stats.
• For HTML/vegalite/lilypond, the bench's nbclient-based stats path crashes; use data/render_3langs.py to compute alternative pass-rate stats from these results JSONs.

Quick load

from huggingface_hub import snapshot_download
import pandas as pd

local = snapshot_download("weixu-zhang/visrl-bench-results", repo_type="dataset")
df = pd.read_json(f"{local}/phase3b/python/results_global_step_574_hf_python_head_0.json")
print(df.columns.tolist())
# ['id', 'task__plot_description', ..., 'raw_response', 'code',
#  'error', 'has_plot', 'score_vis', 'score_task', ...]