AutoResearchEval

Agent trajectory data for AI-for-Science tasks, collected across 8 frontier models (glm-5.2, claude-sonnet-5, opus-4.8, deepseek-v4-pro, qwen3.7-max, minimax-m3, gemini-3.5-flash, gpt-5-mini) under real agentic harnesses (Claude Code / Codex / Gemini CLI), each running in an isolated container with held-out data and a host-side evaluation service.

Two subsets capture two distinct task shapes:

open-ended — scientific discovery

70 tasks built from published frontier papers, each framed as premise → tension → (what step to take). The agent must make an open-ended move (new regime / mechanism / method-correction …); rollouts are classified by discovery regime (search / retrieval / discovery / unverified) under Wang & Buehler Def. 6. 70 tasks × 8 models = 559 rollouts.

• rollouts.jsonl: one row per (task, model). Fields include regime_move_verified, reward, novelty_move, observable, conclusion, report_md, decision_json, trajectory_path.
• tasks.jsonl: task spec — premise, tension, conclusion_gt, domain, novelty_move, source_paper.
• trajectories/<model>/<task>.jsonl: full agent trajectory.

optimization — Target-anchor SOTA optimization

31 tasks where the agent must reproduce a paper's method and beat or match its reported SOTA metric — configure the run, execute, submit predictions to the eval service, iterate. 31 tasks × 8 models = 247 rollouts (195 success / 52 non-success).

• rollouts.jsonl: one row per (task, model). Fields include status (success/timeout/agent_error/killed), duration_s, best_aggregate_improvement (vs paper SOTA; >0 beats SOTA), n_evaluations, trajectory_path.
• tasks.jsonl: task metadata from the source papers (domain, methodology_paradigm, source_paper).
• trajectories/<model>/<task>.jsonl: full agent trajectory.

Honest notes

• Credentials are redacted: API keys, auth tokens and other secrets that agents printed (e.g., in environment dumps) are replaced with [REDACTED]; the trajectories were scanned for e-mail addresses, home-directory paths and credentials before release.
• Failure trajectories are included on purpose (for failure-mode analysis): filter by status in the optimization rollouts.
• gpt-5-mini (codex) trajectories are weak: many end after a shallow single pass without producing/submitting predictions; those that submit score below SOTA. They are kept as faithful records, not as strong baselines.
• best_aggregate_improvement = null means the model never obtained a valid score (never submitted, or submissions failed to parse).

Citation

Anonymous submission — citation withheld during review.