smol-software-harbor

1,002 small, verifiable software-engineering tasks in the Harbor format, for training and evaluating small coding agents.

• Codebases: 576 distinct hand-written mini-codebases with at most 2 tasks each, so the set is varied rather than templated.
• Languages: 12 programming languages (Python first) and instructions in 5 languages.
• Reward: 1 only when the hidden tests, the visible project tests and collateral-damage checks all pass.
• Size: most tasks are small. The agent reads 1–3 files, edits 1–2 and runs the tests, in about 3–10 tool calls. About a fifth are search tasks: find the right place in a larger repository with distractor files, then act.

Contents

Task types (26): fix_bug, add_feature, locate_fix, edge_case, implement_stub, search_answer, add_validation, find_and_change, add_test, read_answer, fix_shell_script, add_cli_flag, fix_sql_query, rename_symbol, handle_missing_file, traceback_fix, config_change, update_call_sites, write_transform_script, fix_build, refactor_extract, add_make_target, fix_regex, remove_dead_code, bump_version, and more.

Splits (manifest.jsonl, field split)

Splits are made by codebase, never by task.

Layout

tasks/<codebase>--<slug>/
  instruction.md          the request (en / it / fr / de / es)
  task.toml               Harbor task config; metadata: language, task_type, difficulty, split, search, ...
  environment/workdir/    the starting codebase (copied to /workdir)
  solution/solve.sh       the oracle
  tests/                  test.sh + check.json + the verifier (python3 only)
manifest.jsonl            one row per task (id, codebase, language, domain, task_type, instruction_language,
                          difficulty, split, files_touched, oracle_steps, n_attacks, search, ...)
registry.json, build_report.json   the index, plus build hashes (deterministic builds)
base_image/Dockerfile     the shared offline image
generators/               the hand-written codebase and task specs and the builders / self-checks

Task names. task.toml names are small-software-harbor/<id> (the build name of this set).

Images. Wave-1 tasks use harbor-swe-base:1. Wave-2 tasks use harbor-swe-base:2, which is :1 plus perl (with prove) and lua 5.4. The image is offline: python 3.11 with pytest, node 18, go 1.19, gcc 12 and make, ruby 3.1 with minitest, php 8.2, OpenJDK 17, rust 1.63, sqlite3, git, jq and ripgrep. Nothing is downloaded at task time.

Verification. Each task's checker enforces: - protected files keep their recorded hashes; - no stray source files; - text checks; - an answer file for search / answer tasks; - the visible project tests; - hidden checks run in a copy of the working directory.

Validation

• Every task: the oracle scores 1, doing nothing scores 0, and every documented attack scores 0. The attacks include editing or deleting tests, hardcoding outputs, and blind edits that skip the search.
• Wave 1: checked in Docker on its image.
• Wave 2: checked with the real toolchains and in Docker on harbor-swe-base:2. In the Docker check, 1,002 of 1,003 built tasks passed every mode. The failing task (a shell task whose reference output depended on the awk implementation) is not included.
• Determinism: builds are byte-for-byte reproducible (build_report.json).

Limitations: - Attacks reduce reward hacking but cannot rule it out. - The tasks are deliberately small: they teach and measure basic agentic coding habits (read, locate, edit, test, stop), not large-repository engineering. - Tasks use two palingenesis HarborEnv extensions, environment/workdir/ and setup.sh. Under the plain Harbor CLI these need the same conversion as other HarborEnv task sets.