wc3env-AgentEnv

Warcraft III (wc3env) on AgentEnv

DeepSeek V4.1 Flash's Undead army meets an identical one played by Warcraft's own attack-move. The plan it wrote at that moment is over the fight. It wins in 54 seconds, keeping 79% of its army. This is the game's own picture, rendered from the run's replay. Every run here plays like this in the Space, with the env's map beside it.

LLM agents play the real Warcraft III: The Frozen Throne (1.29) through wc3env, as an AgentEnv environment: the agentenv-wc3 plugin. Each game is graded per player. A task is played in one of three styles, each its own env or agent: - raw (env wc3): one model orders units by id through general tools: it reads the game with get_state, list_units and lookup, gives orders with act and moves game time on with advance. - commander (env wc3-commander): one model plays through wc3agent's interface, from wc3agent's own code: named units (peasant1), a turn page that says what it can and can't do yet, wc3agent's order language (command), its code reflexes, and its unit menus (fight, choose). - wc3agent (agent wc3-macro-micro on env wc3): wc3env's own agent, as Peter Wang built it: a macro model, and a micro model answering each unit group's menu.

This dataset holds three v1 task sets and every run of them from 2026-10-09 and 2026-10-10, played on the real game. Each set is a bundle per style, and a task is the same game in every style:

Every run comes with: - its video in the game's own picture (all but the six drill-shopping runs and one commander duel, see How the videos were made); - its replay in the browser, with the video beside the env's map; - its timeline; - the game's own .w3g replay; - the agent's own record: from env v34, wc3-llm's untrimmed transcript; for wc3agent's seed-2 duels, its report and session.

Watch them in the Space. The dataset and the Space make up the collection wc3env on AgentEnv.

Bring your own game. Warcraft III is a trademark of Blizzard Entertainment, and the videos show its picture. This dataset holds no game files or activation files, and isn't affiliated with or endorsed by Blizzard. The .w3g replays need your own game to watch. To play a task you need your own Warcraft III: Reforged (it includes the classic game) and an x86-64 Linux host; see Play a task.

Results

The same three low-cost models played every set in the raw and commander styles, and wc3agent played six of the duels. The plugin's styles reads the styles side by side.

The drills (passed, of 25):

The commander style lifts the two weaker models. Its page says what can be built now and what is still short, and its order language builds, trains and gathers in a line each. It does worse in the combat drills (a mean reward of 0.23 against 0.44): it has no micro model, and the models seldom take a fight over. On the raw tools the cheapest model plays the drills best, and the checks most models miss are idle_worker_seconds (a worker idle for more than 30 seconds) and average_unspent_gold.

The ladder (Human against the Orc AI, 30-minute games; each model played all 12 games in each style):

No model beats any level in either style, not even the easy AI. On the raw tools the low-cost models bank their money, rarely cut lumber, and send armies of four to seven Footmen at the AI's base. In the commander style they last longer and kill more: DeepSeek 28 units a game against 12, and a draw against the normal AI. Claude Sonnet 5.5 drew and Claude Opus 5.5 lost at 27.5 minutes, each in one raw game against the normal AI on Echo Isles ($1.23 and $3.14): they spend what they mine and act on the env's feedback, but fall behind after 12 to 18 minutes.

The duels:

The budgets played 26 of the 48 raw duels and 40 of the commander ones. On the 26 both played, GPT-5.4 mini won 5 against 0: in the commander style it groups its whole army and attack-moves it at the enemy base, as Warcraft does. DeepSeek won 3 against 3, and Haiku 1 against 2 and a draw. wc3agent, the only player with a model for the fights, wins the most, at $1.85 a duel against DeepSeek's $0.07 raw and $0.13 commander. Read the duels with two caveats, both in Limitations: six wins came from razing the opponent's only building, and 13 commander duels ended early.

The plugin's docs/evals read these runs game by game: ladder, drills, duels, frontier and styles.

Tables

Every config has one split, eval. Two kinds of tables share the repo:

• WC3's own: what each task stages and checks, and each run's outcome, score and cost.
• agentenv-hf's, one pair per bundle, as agent-env hf publish writes them: the tasks' steps and prompts, and each run's record and chat transcript (messages).

A run has the same episode_id in both, so you can join them. WC3's tables are one pair per set, with every style's runs: a task is the same game in every style, so <set>_tasks has one row per task, and each <set>_episodes row says who played it: bundle, style (raw, commander or wc3agent), agent, model (wc3agent's macro model), micro_model (wc3agent's) and max_cost_usd, the cap it played under. Each of WC3's episode rows, and each reference duel, also names its files: video, replay, timeline, w3g and transcript (empty when the run has none). video_in_sync says whether the run's playback stayed in step with the game. A playback that drifted isn't kept: that run has no video, and its replay shows the env's map alone.

Beside the tables, each run's files are under its episode_id (<bundle>/<run>, or duels-references/<run>):

And: - bundles/<bundle>/: the bundles that agent-env hf run plays. - raw/<bundle>.jsonl: each run's record and trajectory, as agentenv-hf writes them. - runs/<sweep>/: each sweep's spec (sweep.json), its results (results.jsonl) and its report. - assets/: the clip at the top of this card, and the Space's thumbnail.

from datasets import load_dataset

repo, rev = "earakely-scale/wc3env-AgentEnv", "v0.5.0"
episodes = load_dataset(repo, "ladder_episodes", split="eval", revision=rev).to_pandas()
transcripts = load_dataset(repo, "wc3-v1-ladder_episodes", split="eval", revision=rev).to_pandas()
games = episodes.merge(transcripts[["episode_id", "messages"]], on="episode_id")
print(games.groupby("model")[["reward", "score_share", "cost_usd"]].mean())

A run's files by the paths in its row:

import json
from huggingface_hub import hf_hub_download

run = games.iloc[0]
timeline = json.load(open(hf_hub_download(repo, run["timeline"], repo_type="dataset", revision=rev)))
replay = hf_hub_download(repo, run["replay"], repo_type="dataset", revision=rev)   # open it in a browser

Play a task

You need: - Warcraft III: Reforged on Battle.net. In the Battle.net app on Windows, choose Warcraft III - Legacy TFT 1.29 in the Game Version dropdown and install it, then copy its folder (about 1.2 GB) to the Linux host. - An x86-64 Linux host with Docker. The game runs under Wine; Apple Silicon can't run it. - A model endpoint for agent-env: [model] in .agentenv/config.toml, or LITELLM_BASE_URL and LITELLM_API_KEY.

pip install "agentenv-wc3 @ git+https://github.com/earakely-scale/[email protected]"
agent-env wc3 build-worker "<game folder>"     # the worker image from your copy, about five minutes
agent-env wc3 license import "<game folder>"   # your activation files, into agent-env's secret store
agent-env wc3 setup --agent                    # the env as "wc3", and the agents
agent-env hf run earakely-scale/[email protected] --task drill-opening --model anthropic/claude-haiku-4-5

agent-env hf run downloads the bundle at that revision and checks that the plugin is installed and that the env and agents are set up, before it plays. It plays wc3-v1-drills unless --bundle names another. A style's bundles need its env or agent: agent-env wc3 setup --style commander builds and registers wc3-commander, and setup --agent registers wc3-macro-micro with the other agents.

agent-env hf run earakely-scale/[email protected] --bundle wc3-v1-ladder \
    --task ladder-echoisles-vs-easy-orc-s1 --model <your model>
agent-env hf run earakely-scale/[email protected] --bundle wc3-v1-duels --eval duels --model <your model>
agent-env hf run earakely-scale/[email protected] --bundle wc3-v1-drills-commander --eval drills-economy \
    --model <your model>
agent-env hf run earakely-scale/[email protected] --bundle wc3-v1-duels-wc3agent --task mirror-orc-s1

Without --model, a task plays Claude Haiku 4.5; wc3-v1-duels-wc3agent's play DeepSeek V4.1 Flash as wc3agent's macro model, as its runs did, with Claude Haiku 4.5 as its micro. --dry-run shows what would run, and --yes skips the question. The plugin's README has the setup in full. It also covers watching a game live and getting its video and replay, and agent-env wc3 sweep for running a task set across models.

How the runs were made

Each set was played as a sweep, by wc3-llm, the plugin's agent (any chat model, with the style's tools), or by wc3-macro-micro (wc3agent). The sweeps ran two or three games at a time on one Linux host, under a spend budget. runs/ has each sweep's spec, results and report.

The raw env changed during 2026-10-09 (plugin env versions v32 to v36). The fixes report why the game dropped an order, keep a dead hero in view, and check each order on its own; each fix reached the games started after it. On 2026-10-10 the commander sweeps played wc3-commander v1, and wc3agent's duels the raw env v37 (seed 1) and v38 (seed 2), which play them alike. A sweep's task names carry the model (ladder-gpt-5.4-mini-echoisles-vs-easy-orc-s1); here each run is filed under its v1 task.

How the videos were made

None of these games was played with the game's picture on, since drawing makes stepping about three times slower. Each video was rendered afterwards from the run's replay with agent-env wc3 render, in the env's own image with the picture on: - The replay plays the game again. wc3env plays the .w3g back: the players' orders and the AI come from the recording. - Staging is applied again. A drill's or a duel's setup (its armies, levels and mana) was made with debug commands, which a replay doesn't record. The render applies the task's staging again before the playback, step for step as the live game did, and leaves the staging's orders (a hero's skills) to the recording. - The camera follows the agent with the env's director, as a live game's picture does: its fights first, then its key moments (a hero, a tier, an expansion, a building lost), then its army. - The agent's plans show in the game at the game time the agent wrote them. - One frame per fixed slice of game time, at 20 frames a second: 400 ms a frame for the ladder (8×), 100 ms for drills (2×), 50 ms for duels (1×).

Every published video is checked against the game. Its playback must end on the score the live game's own timeline last showed for the run's lead player; both are read from the game's observations, so the check is exact. 313 of the 314 rendered do. A playback that drifted is not published, since it would show a game the agent didn't play. - The AI level, fixed. Six playbacks first drifted, all of games where the easy AI was set: wc3env saves a replay's startup without an AI level of 0, so they played back against the normal AI. With the level restored from the task, all six end in sync. - One playback drifts, cause unknown. Commander Haiku's orc duel on seed 3 plays back to the same victory but ends half a second early, at 31.65 game seconds against 32.15, with 26,528 points against the live game's 26,535. A second render ended the same way. Its replay page shows the env's map alone. - The render's own record. videos/<run>.mp4.json, next to each video (and for the drifted run, alone), has the render's pace and final scores.

The six drill-shopping runs have no video, three in each style. Their staging lets the game run 460 seconds first, a minute at a time, and the game plays those minute-long steps of a replay back short: about 422 seconds instead of 460. The playback therefore can't reach the drill's start in step with the game that was played, so they weren't rendered. Their replay pages show the env's map alone.

Limitations

• Few runs. One game per model and task, so a model's ladder record is 12 games, and the frontier models played one ladder task each. The raw duels stopped at 26 of 48 and the commander duels at 40, and wc3agent played six duels: seed 1 of each race and seed 2 of the human and night elf mirrors.
• A duel can be won by the base. On odd seeds each army starts nearer the enemy's base than the enemy army is. Six wins (five commander, one raw, all in the orc mirror) came from attack-moving at the opponent's Great Hall, its only building: the hall fell, and Warcraft's melee rule defeated the player with no buildings. In five of them the enemy army was untouched. The duel's goal asks for the enemy army, but the grade counts the game's result.
• Some players stopped early. In 13 commander duels the player stopped giving orders more than 15 game seconds before the end, and lost each. Haiku reached the $0.50 cap in 8 of its 13: each commander turn returns the whole page. DeepSeek's turns came back empty: wc3-llm allowed 4,096 output tokens a turn, DeepSeek's reasoning ran past them before it called a tool, and the agent stopped at a game's fourth reply without a tool call. Plugin v0.5.0 allows 32,768 tokens and stops only at the fourth in a row; these runs were played before.
• The commander style has no micro model. A group fights by Warcraft's attack-move unless the model takes the fight over with fight and choose, which the models seldom do.
• wc3agent's micro is Haiku, not Jev. wc3agent's own micro model needs a TypeSafe key. With Haiku a duel costs about $1.85 (Haiku costs about $0.03 a game second), so a 30-minute ladder game would cost about $50, and wc3agent played duels only. Its four seed-1 duels were played before a fix: it kept running after the env decided the duel, so their report and session weren't kept.
• Duels on Echo Isles. wc3agent ran its duels on a flat arena built from Echo Isles, which needs Windows tools to build. Here the armies meet in Echo Isles' middle, with the creeps in sight cleared and odd seeds swapping the sides. Warcraft against Warcraft scores 15 of 24 for player 0 in these games, and 59% over the 48 since the staging fix.
• A draw is the time limit. A ladder game nobody wins in 30 minutes is a draw (0.5), however far behind.
• Cost caps need a LiteLLM proxy. wc3-llm prices each call from LiteLLM's x-litellm-response-cost header. Through another endpoint it can't see spend, so its cap never ends a game; set a limit at the endpoint.

Licence and credits

The tasks, records and tables are Apache-2.0, like the plugin. The transcripts are the models' outputs.

• wc3env (MIT), by Peter Wang: the game under Wine, the hook that observes and orders it, and the fake game the plugin's tests use.
• wc3agent (MIT), in wc3env's repository: the scenarios the drills are built from, the mirror duel and the heroes' skill builds; the commander style's interface (its pages, order language, reflexes and unit menus), and the agent that wc3-macro-micro runs.

This dataset and its Space are built by scripts/hub_dataset.py in the plugin's repository, from the sweeps' folders and the agent-env store they wrote to.