AuraGym

AuraGym provides training datasets for security agentic coding tasks from open-source repositories. It was built via AuraForge, a synthetic data pipeline to build agentic training environments at scale. The synthetic variant contains 679 feature implementation tasks from 344 real-world repositories across Python, JavaScript, and TypeScript, covering 177 CWE categories.

There are two variants:

• AuraGym: 679 tasks from 344 repositories, with synthetic security tests generated via AuraForge.
• AuraGym_h: 431 tasks from 199 repositories, with human-written security tests.

The variants overlap but neither is a subset of the other: 368 task IDs are shared, 63 occur only in the human variant, and 311 occur only in the synthetic variant. Together they contain 742 distinct task IDs across 1,110 records. All four configurations are training data; this release does not define validation or test splits.

Dataset files

All links in this table point to files in this directory. Each file uses JSON Lines (JSONL), with one task per line.

To load a complete variant, combine its Python and JavaScript / TypeScript files. Task IDs are unique within each variant; retain the variant label when combining human and synthetic records.

Download and load

From the SusVibes repository root, download the dataset and environment specifications from dqwang122/AuraGym:

pip install huggingface_hub datasets
hf download dqwang122/AuraGym --repo-type dataset --local-dir AuraGym

To load records directly from the Hub:

from datasets import load_dataset

tasks = load_dataset("dqwang122/AuraGym", "python-synthetic", split="train")
print(tasks[0]["problem_statement"])

The configurations are python-human (202 records), python-synthetic (476), tsjs-human (229), and tsjs-synthetic (203). Each configuration explicitly selects one JSONL file; environment specifications and release metadata are auxiliary files, not dataset rows. load_dataset loads records only; use the full download for grading.

For a local copy, use load_dataset("json", data_files="AuraGym/python.synthesis.476.jsonl", split="train"). Python records use a string for language, while JavaScript / TypeScript records use a list. Additional columns also differ, so normalize schemas before concatenating configurations with Hugging Face Datasets. Reading and combining the raw JSONL records with Python's json module preserves their original schemas. Pin the dataset's Hub commit through revision when reporting reproducible experiments.

Dataset statistics

Counts below are computed from the included records' language, project, and cwe_ids fields. CWE denotes Common Weakness Enumeration.

Six human tasks and four synthetic tasks are labeled with both JavaScript and TypeScript and count in both language rows. Totals count each task, repository, and CWE once per variant.

Record format

Using the tasks

Evaluate Agent on a Task

Follow the SusVibes agent workflow:

1. Pull the task's container image using docker pull <image_name>, where <image_name> comes from the dataset record.
2. Run your coding agent in the container, using /project as its workspace and problem_statement as its prompt.
3. Save the resulting implementation diff as model_patch. Write one prediction per line in a JSONL file:

{"instance_id": "<task-instance-id>", "model_name_or_path": "your-agent", "model_patch": "<implementation-diff>"}

For agent integrations, see the SusVibes evaluation harnesses.

Exclude reference patches and security-test metadata from agent inputs when measuring held-out performance. Keep human and synthetic records with the same instance_id in the same partition; partition by project for repository-disjoint splits.

Grade Solutions

Use the SusVibes evaluator to grade functional correctness and security. The release records public commit 520f16f9b3f39b06c6013bc0eb57a38280b4deee as its evaluator baseline. That revision includes generated-security-test routing and JavaScript / TypeScript test adapters; full container execution has not been validated for this release.

For a fresh evaluator checkout:

git clone https://github.com/LeiLiLab/susvibes.git
cd susvibes
git checkout 520f16f9b3f39b06c6013bc0eb57a38280b4deee

Follow the pinned checkout's installation instructions (Python 3.11 or newer and Docker), then download AuraGym into it as shown above. Use the directory-based registration below, which does not depend on local filename-alias changes.

Register an AuraGym dataset and its matching environment specifications in the evaluator's directory layout. Run the following from the SusVibes repository root, with this release available at AuraGym/:

# Choose one of the four dataset filenames, without .jsonl.
auragym_dataset=python.synthesis.476

mkdir -p "datasets/auragym/$auragym_dataset"
mkdir -p "susvibes/env_specs/auragym/$auragym_dataset"
cp "AuraGym/$auragym_dataset.jsonl" \
  "datasets/auragym/$auragym_dataset/susvibes_dataset.jsonl"
cp "AuraGym/env_specs/$auragym_dataset/"*.json \
  "susvibes/env_specs/auragym/$auragym_dataset/"

python -m susvibes.eval.core \
  --dataset_id "auragym/$auragym_dataset" \
  --run_id auragym-eval \
  --predictions_path predictions.jsonl \
  --max_workers 1

Replace predictions.jsonl with your agent's predictions. Use the specifications for the selected variant, since human and synthetic records can share task IDs but require different grading handlers.

Results are written under logs/eval/auragym-eval/none/<model_name_or_path>/. Consult the evaluation reference for report fields, resource requirements, and additional options. The commands above have not been validated with a container run for this release.

Provenance and limitations

Records retain their source project, base_commit, vulnerability identifiers, and patches. release/sources.json indexes source repositories and revisions by task and configuration; release/manifest.json records dataset and specification checksums and the evaluator baseline. Filenames use synthesis consistently; the records themselves are unchanged from the source export.

Use these tasks for training and research on functional correctness and security-sensitive coding. Passing the supplied tests measures behavior covered by those tests and does not establish that an implementation is free of vulnerabilities. The human and generated test variants can differ in coverage. This release does not provide complete feature-specific test counts or establish disjointness from other benchmarks.

Local validation loaded all four configurations with Hugging Face Datasets and checked specification coverage and the pinned evaluator's routing for all 1,110 records. See release/validation.json for tool versions and checks.

Container references are external dependencies. All 907 unique image references were accessible at the recorded check time. release/image_availability.json records anonymous registry checks, their timestamp, and resolved digests where available. Registry accessibility does not verify image contents, layer downloads, or successful task execution. Tags may change; use the recorded digests when pinning environments.

License

The MIT license is carried forward from SusVibes for repository-authored material. Code and patches derived from upstream projects remain subject to their original licenses and notices; the MIT metadata does not relicense that material. The source index identifies the upstream repositories and revisions. Per-repository license and notice verification is not complete, and no blanket licensing claim is made for third-party code or container contents.

Citation

@misc{wang2026auraforge,
  title={AuraForge: Scaling Security Supervision for Training Coding Agents},
  author={Danqing Wang and Songwen Zhao and Harsh Sharma and Jierui Wang and Andre Vicente Duarte and Ivan Bercovich and Lei Li},
  year={2026}
}