PortSimEnv-AgentEnv

PortSimEnv on AgentEnv

_{GPT-6.1 Sol plays the marine week dock-24B-w07x1-busy-0. The week opens with a quay closure; an unscheduled call, an emergency, late ships, a crane outage, a gale and bunched arrivals come in as bulletins, and the tug company and the pilot station announce 2 tugs and 2 pilots out from hour 54. It ends at the hindsight optimum (reward 1.0), where a naive re-plan breaks the rules. ▶ Watch this week on the replay Space. Twin © OpenStreetMap contributors (ODbL).}

An agent runs one container quay at the Port of Barcelona for a week of 2024 that has just gone wrong: late and bunched ships, closed quay sections, crane breakdowns, gales, emergencies. It decides when, where along the quay and with how many cranes every ship docks, and the week is graded deterministically against a plan CP-SAT proved optimal. This dataset holds the weeks and the recorded runs of that environment on the AgentEnv Framework, in three versions:

• v3, the marine port (env portsim-marine): v2's live week, where every berthing and departure also takes a pilot and tugs in its hour, from port-wide pools shared with the rest of the port's real 2024 traffic. The tug company and the pilot station announce cuts during the week, and a gale adds a tug to most movements.
• v2, the live port (env portsim-live): the week played as it unfolds. AgentEnv's virtual clock runs it watch by watch, the ships, the harbour master, terminal ops and the line desk send their news through triggers, windows about to start are frozen, and the agent confirms berths as it goes. It is graded on what it confirmed, against the week as it really happened.
• v1, a week planned in one go (env portsim): the agent reads the situation, checks drafts and submits one plan, as in PortSimEnv itself.

The environment is the plugin earakely-scale/agentenv-portsim-plugin, built on PortSimEnv, Adithya S Kolavi's environment in FineEnvs. Its own tasks, including the 1,050 training weeks, and rollouts are in FineEnvs/PortSimEnv.

Tables

Every week is an eval week. The 15 live weeks are the one-week eval weeks on which the rolling re-planner reaches the hindsight optimum, so a perfect score is reachable on what the agent could know; with pilots and tugs, the same 15 qualify. A task id reads dock-<quay>-w<week>x<weeks>-<difficulty>-<seed>: dock-24B-w07x1-busy-0 is week 7 of 2024 at quay 24B, one week long, busy tier. A v1 task can span up to three weeks (x2, x3); a v2 or v3 week is always one. dock-v1 is the name of PortSimEnv's task pack, which all three versions use, not this dataset's version.

Terms

• Week: one task, a quay with its ships and what goes wrong. Watch (v2, v3): watch 0 opens the week at hour 0, and a new watch starts at each news bulletin; the agent re-plans each watch, and a window starting within 6 hours (the freeze) can no longer change.
• Pilots and tugs (v3): a pilot is the local mariner who boards to guide a ship in or out of the port, and tugs are the boats that push and pull it alongside. Every berthing and departure of a ship of 45 m or more takes a pilot, and tugs by length (0 under 120 m, up to 3 from 300 m), in its hour, from 7 pilots and 8 tugs on duty; inside an announced gale window most movements take one more tug (not Ro-Ro ships and ferries under 200 m, nor yachts). The port's other 2024 calls use the same pools. In any hour the quay's ships may need no more than are free; a short hour is a rule break.
• Optimum: the cost of the best plan CP-SAT, a constraint solver, finds for the week as it really happened. Rolling: a CP-SAT re-planner that only knows what has been announced. Naive: a simple policy that pushes ships later (in v2 and v3, it keeps each confirmed window that still fits and moves the rest; it knows nothing of pilots and tugs, so in v3 it often breaks their rule). Unavoidable: the cost no plan avoids. Regret: cost above the optimum. Excused cost: cost that news put on a frozen window, which is not charged.
• Reward: 0.2 + 0.8·e^(−gap/0.5) for a valid plan, where gap = max(0, cost − optimum) / (max(0, optimum − unavoidable) + 100), so the optimum scores 1.0. A plan that breaks a rule scores under 0.2 (0.2 times the share of ships placed cleanly), and no plan scores 0. The confidence intervals below are bootstrap intervals over weeks.
• Sweep: one batch of runs (models × weeks), recorded in runs/; the sweeps named …-pilot-… were small first batches of two weeks, unrelated to harbour pilots. Bundle: a folder of agent-env tasks.

Use

import json
from datasets import load_dataset

weeks = load_dataset("earakely-scale/PortSimEnv-AgentEnv", "v3_tasks", split="eval", revision="v0.3.1")
week = weeks[0]
print(week["system_prompt"], week["situation"], sep="\n\n")   # what the agent gets at hour 0
for watch in json.loads(week["watches"]):                       # what arrives later
    print(watch["hour"], [n["text"] for n in watch["notices"]])

episodes = load_dataset("earakely-scale/PortSimEnv-AgentEnv", "v3_episodes", split="eval", revision="v0.3.1")
messages = json.loads(episodes[0]["messages"])                  # the whole conversation

Columns holding JSON (watches, task, plans, steps, messages) are strings; json.loads them.

Play a week yourself

You need Docker (running, usable without sudo, with buildx), uv and git. Install the plugin, build the images, and check it all works with a free task that calls no model:

uv tool install agentenv-framework \
    --with "agentenv-portsim @ git+https://github.com/earakely-scale/[email protected]"
agent-env portsim setup --agent        # build the env and agent images and register them (a few minutes)
agent-env run portsim --task smoke     # no model: grades a known optimal plan, prints "passed"

Then play a week with a model. You need a Hugging Face token with the "Make calls to Inference Providers" permission, on an account with credits; a GLM-5.3-Flash week costs a few cents, and each task stops an episode before it could spend $5.

export HF_TOKEN=hf_...
uvx --from huggingface_hub hf download earakely-scale/PortSimEnv-AgentEnv --repo-type dataset --revision v0.3.1 \
    --include "bundles/*" --local-dir portsim-weeks

export LITELLM_BASE_URL=https://router.huggingface.co/v1 LITELLM_API_KEY=$HF_TOKEN
agent-env run portsim-weeks/bundles/dock-v1-eval-marine --task dock-24B-w07x1-busy-0 \
    --model zai-org/GLM-5.3-Flash:baseten   # v3
agent-env run portsim-weeks/bundles/dock-v1-eval-live --task dock-24B-w07x1-busy-0 \
    --model zai-org/GLM-5.3-Flash:baseten   # v2
agent-env run portsim-weeks/bundles/dock-v1-eval --task dock-24B-w07x1-busy-0 \
    --model zai-org/GLM-5.3-Flash:baseten   # v1

To replay your own runs on the same viewer as the Space, play them as a sweep, which records each run under results/runs/<name>/, then serve them:

agent-env portsim sweep run --marine --name mine --models zai-org/GLM-5.3-Flash:baseten \
    --tasks dock-24B-w07x1-busy-0 --episode-cap-usd 1 --cap-usd 3
agent-env portsim view mine            # http://127.0.0.1:8237/viewer/

The open models priced for the router are Qwen/Qwen3.8-2.4T-A95B:together, Qwen/Qwen3.8-27B:ovhcloud, zai-org/GLM-5.3-Flash:baseten and zai-org/GLM-5.3:together. If a step fails, the plugin README's troubleshooting covers the usual causes (agent-env not on the PATH, port 5000 taken by macOS AirPlay); it also covers sweeps, other model endpoints and filming a run.

Dataset structure

Tasks (v3_tasks, v2_tasks, v1_tasks), one row per week:

• task_id, quay, terminal, difficulty (standard, busy, storm or extreme), week, week_start_utc, num_ships, num_disruptions.
• system_prompt, situation: the rules and the opening message, exactly as the agent receives them. In v3 the situation ends with a "Pilots and tugs" section: the pools, the rule, the tugs each ship takes, and the pilots and tugs free for the quay's ships each hour.
• optimal_cost: the cost of the best plan CP-SAT finds for the week as it really happened. naive_cost: v1, a plan that pushes ships later; v2 and v3, the naive policy's cost, empty when its plan breaks a rule (naive_feasible).
• v2 and v3: num_watches, watches (each watch's hour and its notices: who sends it, the hour it is about, the text), rolling_cost (the rolling CP-SAT re-planner) and unavoidable_cost (what no plan avoids).
• v3 only: pilots and tugs (the pools on duty), other_movements (the berthings and departures of the port's other traffic the pools also serve), v2_optimal_cost (the same week's optimum without pilots and tugs).
• v1 only: proven_optimal, optimal_plan, naive_plan. A plan is a list of {"ship": id, "berth_hour": h, "section": s, "cranes": c}.
• task: the full week as the task pack has it, including every disruption (and in v3 the pools and the other traffic's movements); in v2 and v3 the agent learns the disruptions only as their notices arrive.

Episodes (v3_episodes, v2_episodes, v1_episodes), one row per model and week:

• model, model_name, task_id, difficulty, quay, num_ships.
• reward (0 to 1), feasible, cost (of the plan, or of the confirmed windows in v2 and v3), optimal_cost.
• v2 and v3: num_watches, regret (cost above the optimum), excused_cost (cost that news put on a frozen window, not charged). v1: submitted, checks.
• turns, tool_calls, input_tokens, output_tokens, cost_usd (model spend at prices pinned in October 2026), end_reason, sweep (the run it comes from).
• final_plan, steps (each tool call with its plan), messages (the full transcript).

References (v3_references, v2_references), one row per one-week eval week: qualifies, optimal_cost, unavoidable_cost, watch_hours, the rolling re-planner's and the naive policy's cost, reward, feasible and excused_cost, the rolling plans per watch, and the solver settings; v3 adds v2_optimal_cost.

Files beside the tables: bundles/ holds the weeks as agent-env tasks (used above), references/ the references also as JSONL, and runs/ the raw sweeps the replay Space reads.

Results

v3, the marine port, one run per model and week:

v3 is harder than v2 in a way a naive planner can't see: the pools are shared with the rest of the port, the cuts arrive mid-week, and a plan that ignores them breaks the rule. The naive policy, which knows nothing of pilots and tugs, is infeasible in 9 of the 15 weeks (0.187, against 0.202 in v2). The two models stay close to their v2 scores: GPT-6.1 Sol's mean is 0.957 without its one infeasible week, its v2 mean, and Claude Sonnet 5.5 averaged 0.901 on the same 14 weeks in v2. With one run per week, neither change is beyond noise.

GPT-6.1 Sol's infeasible week is dock-24B-w35x1-busy-0: at its last watch it confirmed windows before checking them and advanced with a pilot-short hour in its plan. Claude Sonnet 5.5's unscored week is dock-36A-w10x1-standard-0: on the first attempt its second reply never came and the run ended after 15 minutes with no spend recorded, and the second attempt was stopped by hand after about 4 minutes.

v2, the live port, one run per model and week:

Every v2 run reached the end of its week with a feasible plan and passed the validity audit.

v1, a week planned in one go, on ten eval weeks, next to PortSimEnv's published eval of the same weeks:

One Sonnet episode, on the 56-ship storm week dock-36A-w15x2-storm-0, spent both of its turns' 32,000 output tokens without calling a tool and scored 0; PortSimEnv's harness ends such an episode the same way.

Limitations

• Small samples: one run per model and week, two closed models, 15 weeks in v2 and v3 and 10 in v1. The intervals above are wide, and a single week can swing by a lot.
• Simulated pressure on real data: the ships, their calls, the cranes, the port's rules and its other traffic are real; the workloads and the disruptions are simulated, and the parties send scripted news without answering the agent.
• v3's assumptions: some numbers have no public source, among them the 7 pilots on duty (calibrated to the 2024 traffic), the tugs per ship length, and the cuts. Ships don't wait for a tug: a short hour is a rule break. The plugin README's rules and their sources marks each one.
• Eval weeks only: live and marine weeks are qualified from the eval pack; the training weeks are in PortSimEnv's dataset and play on v1.
• Spend is computed from token usage at prices pinned in October 2026, not billed amounts.

Licence and attribution

CC BY-SA 4.0. The weeks come from PortSimEnv's dock-v1 task packs, which Adithya S Kolavi built from the Port of Barcelona's 2024 container calls and shared under CC BY-SA 4.0. v3's other traffic is derived from the port's record of its 2024 calls of every ship type (Barcelona_2024.csv in alberto-santini/berth-allocation-problems, CC BY-SA 4.0). Contains data from the Port de Barcelona open data portal. The tables, the references and the transcripts, which carry the task text, are derived from them under the same licence. The verifier scripts in bundles/ are from the plugin, under the Apache License 2.0. scripts/hub_dataset.py in the plugin builds this dataset.

This dataset is independent: neither the Port de Barcelona nor PortSimEnv's author is affiliated with it or endorses it.