FrameSWE

A repository-level software engineering dataset for deep learning frameworks.

Hugging Face Dataset · Browse Tasks

Release status: Task data and reference artifacts are available now. Dataset construction scripts and the unified evaluation pipeline will be released in a future update.

Overview

FrameSWE contains 102 real-world software engineering tasks drawn from the PaddlePaddle deep learning framework. The dataset covers bug fixes, enhancements, implementations, and API changes, with an emphasis on framework-specific code understanding and repository-level reasoning.

Each task provides a problem description and supporting artifacts, including a reference patch and task-specific tests. FrameSWE is intended for research on coding agents, automated program repair, and software engineering benchmarks.

Dataset Statistics

The enhancement category includes one task previously labeled Unspecified in an earlier summary.

Affected components (tasks may span multiple components): API / layers (51), Graph / runtime (23), Kernels / operators (21), Distributed (15), Operator metadata (14), I/O / utilities (9), Bindings (4), and Test infrastructure (4).

Reference-patch scope: 1 file (40 tasks), 2–3 files (36), 4–10 files (18), and 11–20 files (8).

Dataset Structure

FrameSWE/
└── tasks/
    └── <task_id>/
        ├── README.md
        ├── instruction.md
        ├── proposal.md
        ├── environment/
        │   └── README.md
        ├── solution/
        │   └── code.patch
        └── tests/
            ├── test.patch
            └── test.sh

• instruction.md — Task description for the coding agent.
• README.md / proposal.md — Task metadata, context, and supporting documentation.
• environment/ — Environment and setup notes.
• solution/code.patch — Reference implementation patch.
• tests/ — Task-specific test patch and execution script.

Task artifacts can vary; consult the corresponding task README for details.

Getting Started

Download the dataset using the Hugging Face CLI:

pip install -U huggingface_hub
hf download yuxuanie/FrameSWE --repo-type dataset --local-dir ./FrameSWE

Or clone the repository:

git clone https://huggingface.co/datasets/yuxuanie/FrameSWE

The tasks/ directories contain the complete task artifacts. For convenience, data/benchmark.jsonl contains a lightweight, viewer-friendly index with task IDs, instructions, proposals, and relative task paths. The Hugging Face test split is a display/index convention; this release does not yet provide an official evaluation protocol or hidden test set.

Evaluation and Release Plan

To evaluate a coding agent, start from the upstream repository revision specified for a task, provide the task instruction, apply the generated patch, and run the relevant tests in the required environment. Do not expose reference solutions to the evaluated agent.

The current release does not include a standardized evaluation harness or official baseline scores. The task-construction and evaluation scripts are planned for a subsequent release. Until then, results from custom execution setups should report their environment, test protocol, and dataset revision.

Limitations and License

FrameSWE currently focuses on PaddlePaddle, so results may not generalize to other software ecosystems. Because tasks originate from public development artifacts, potential training-data contamination should be considered when interpreting model performance. Some tasks may require substantial build dependencies or specialized hardware.

The dataset repository is published under Apache-2.0. Upstream source code and third-party artifacts remain subject to their respective licenses.