airmail-runs

Airmail

Human gameplay from a first-person rooftop-courier game, recorded at the simulation tick rather than from the screen. One row per tick: the exact input the player gave, and the full world state that input produced.

The action labels and the world state are ground truth read out of the simulator, not inferred from video. No pose estimation, no key-press detection, no frame differencing, no interpolation. in_jump is the jump bit the game consumed on that tick; vel_y is the number the physics step wrote. Every row is exact to the byte the recorder stored.

Every session in this dataset has been replay-verified: re-simulated from its recorded seed and input stream and compared field-by-field against the recorded state trace, on every tick. See Replay verification.

The game

A continuous first-person courier game on a district of flat rooftops. There is no end state and no score screen. You pick up a stack of one to three crates, carry them — they hang from your hands on a damped spring and swing when you turn, jump or get hit by a gust — cross the roofs, and drop them into a delivery beam before the timer runs out. Delivering raises a streak multiplier, which raises the difficulty: heavier stacks and longer throws. Dropping a crate, falling to the street, or running out the clock breaks the streak. Falling costs seconds, never the session.

Controls: WASD move, mouse look, Space jump, left mouse button throw the held stack, E grab or release a zipline.

• Fixed 60 Hz simulation tick. dt is exactly 1/60 s and is never variable.
• Movement is Quake-style ground acceleration and friction with an air-control cap, coyote time and a jump buffer. Cargo is a Rapier rigid body on a spring joint, not an attachment.
• A recorded, deterministic wind field applies horizontal acceleration to the player and, more strongly, to the carried crates. The acceleration that acted on each tick is in that tick's row (wind_x, wind_z, wind_gust, wind_exposure) — it never has to be inferred.
• Weather (sun position, fog density, rain, cloud cover) is drawn once per session and changes what the player could see. It has no effect on the physics. It is on every row because a log that omits visibility describes decisions it cannot explain.

How the data was produced

The game is a browser build. Each tick, after the simulation step, the recorder writes one fixed-size binary record (128 bytes) into a preallocated buffer, plus any events the tick emitted (24 bytes each). Every 3,600 ticks (60 s) the buffer is closed into a self-contained chunk:

[ header ][ tuning constants ][ keyframe ][ 3600 × tick record ][ N × event record ]

The header carries the session id, the session seed, the sim and format versions, the level hash, the session's weather, and a full copy of every tunable simulation constant in force while the chunk was recorded. The keyframe is the complete simulation state at the chunk's first tick. The chunk is gzipped in the browser and streamed to this repository. Nothing is ever written to the player's disk, and the recording path allocates nothing per tick.

The tables here are produced from those chunks by npm run export in the game's repository, which decodes them through the game's own binary reader. There is no second implementation of the format.

Raw chunks stay in the repository under chunks/<session-id>/<index>.bin.gz, so every table below can be regenerated, and any claim in it can be checked against the bytes the browser wrote.

Loading it

from datasets import load_dataset
ticks    = load_dataset("arjvnv/airmail-runs", split="train")               # one row per tick
sessions = load_dataset("arjvnv/airmail-runs", "sessions", split="train")   # one row per session

"events" is the third config. With pandas instead:

import pandas as pd
ticks = pd.read_parquet("hf://datasets/arjvnv/airmail-runs/data/ticks/")

Layout

data/sessions.jsonl              one row per session — the index
data/events.jsonl                one row per event
data/ticks/<session-id>.parquet  one row per tick, one file per session
chunks/<session-id>/*.bin.gz     the raw recorded chunks the tables were built from
index/<session-id>/*.json        one sidecar per chunk; its file name carries the weather kind,
                                 useful-tick count, tick count and byte size, so the corpus totals
                                 can be rebuilt from a file listing alone
frames/manifest.json             what the rendered frames are and how they were made
frames/<session-id>/*.bin.gz     rendered frames, raw uint8, packed flat, one shard per file
frames/<session-id>/*.index.jsonl  one line per frame: session, tick, chunk, shard, byte offset
frames-wx/manifest.json          the weather-matrix set: which skies, which sessions, which ticks
frames-wx/<weather>/<session-id>/*.bin.gz    the same trajectories re-rendered under a forced sky
frames-wx/<weather>/<session-id>/*.index.jsonl
quarantine/                      chunks that failed replay verification, kept for inspection

Parquet is partitioned one file per session, so nothing is enormous and the viewer streams it.

Schema — data/ticks/*.parquet (one row per tick)

65 columns. 60 rows per second of play. tick is 1-based and monotonic within a session; join to sessions.jsonl on session.

Axes: x east, y up, z south, metres, right-handed. Positions are the capsule centre, which sits 0.9 m above the feet. Velocities are m/s, angles radians.

The eight weather columns are constant within a session and are repeated on every row on purpose, so the tick table stands alone without a join.

Schema — data/sessions.jsonl (one row per session)

Schema — data/events.jsonl (one row per event)

Sparse: only the ticks where something happened. Every row has session, tick, type (the numeric code), event (the name), the decoded payload fields listed below, and raw (a, b, f0, f1, f2 — the record exactly as stored).

The last seven types exist so the data is honest about its own gaps. PAUSE / RESUME / TIME_DROP mark real-time discontinuities: simulation ticks stay contiguous across them, but wall time does not. DEBUG_OVERRIDE and CONST_CHANGE mark a session where a simulation constant was edited live; filter those sessions out if you need one fixed dynamics model across the corpus.

Known confound — on-rail zipline ticks

Riding a zipline is on rails. While attached, the character controller is suspended and the player's position is driven along the cable by the along-cable component of gravity. The inputs recorded on those ticks — including movement keys the player kept holding — did not move the player. A model trained on them will learn that actions have no effect.

Those ticks are flagged. Exclude them from inverse dynamics and from any action-conditioned dynamics model:

ticks = ticks.filter(lambda r: not r["on_zipline"])   # or: df[~df.on_zipline]

zipline_id says which cable, so rides can be grouped, and the ZIP_GRAB / ZIP_RELEASE events bracket each ride exactly. on_rail_fraction in sessions.jsonl reports the share per session. The flag is written by the simulation on every attached tick, including the grab tick and the release tick.

The comparable filter for dead time is idleness: no input bit the simulation honours, and a player speed below 0.25 m/s. That is the definition the ingest endpoint uses to decide whether a chunk was worth keeping, and idle_ticks / idle_fraction in sessions.jsonl report it per session. In the tick table it is input_bits == 0 and (vel_x**2 + vel_y**2 + vel_z**2) < 0.0625. Looking around while standing still counts as idle: a mouse delta with no movement is not gameplay.

Neither filter is applied to the data — both are recorded so you can apply the one you want.

Replay verification

The simulation is deterministic: the same seed and the same input stream produce the same trajectory, bit for bit, under the same simulation version and constants. Everything a replay needs is in the chunk header — the seed, the level hash, and a full snapshot of every tunable constant that was in force.

npm run verify-dataset re-simulates every session in this repository from its recorded seed and inputs and compares all 37 recorded fields against the recorded state trace, tick by tick, and reports the first divergent tick and the largest per-field error. A session that reproduces exactly cannot have been edited, resampled, interpolated or inferred: the state trace is a pure function of the inputs, and the inputs are in the file.

AIRMAIL DATASET — REPLAY VERIFICATION

session                            chunks    ticks  verdict     first div  max field error
-------------------------------------------------------------------------------------------
19c74a1bf07a3e87cdab828bc3dfca62        2    4,200  MATCH               —  0
348253747137121a8508bde5e7b544dd        1    2,400  MATCH               —  0
87f6081f5a72b5d4b3ab4d5cd68d3a55        2    4,200  MATCH               —  0
97a4430ddc6ca0578ef93c1095cd23b4        2    4,200  MATCH               —  0
eebb9d951ada1ffda7f6d27375f9be9d        2    4,200  MATCH               —  0

5 session(s), 9 chunk(s), 19,200 tick(s) re-simulated from the recorded seed and inputs.
5 MATCH · 0 DIVERGE · 0 ERROR · 0 UNVERIFIED
Every recorded tick reproduced bit-for-bit. The dataset is replay-verified.

A session that does not reproduce is moved to quarantine/ rather than deleted, so it can be opened and understood. Anything under quarantine/ is outside the dataset configs and is not loaded by load_dataset.

Two verdicts other than MATCH and DIVERGE can appear, and neither is a claim about the data being wrong:

• UNVERIFIED — the session's chunk 0 is missing, so the replay can only resume from a later chunk's keyframe. A keyframe does not carry the physics solver's internal state, so contact-heavy stretches drift and an exact comparison is not available. contiguous_from_zero in sessions.jsonl marks the sessions that can be verified exactly.
• ERROR — the chunk cannot be replayed by the current build at all: its level_hash or sim_version names a map or a simulation this build no longer contains. Those sessions are real recordings, but they are not comparable trajectories and are not part of the training table.

Frames (rendered offline)

The recorder never stored video. It stored the inputs and the state, which is strictly more information, and images are regenerated from them on demand by npm run render-frames: every recorded tick is re-simulated, drawn by the game's own renderer in a headless GPU browser, and streamed straight into frames/ here. Nothing about a frame is inferred — it is the same renderer, the same camera, the same weather, the same tick.

frames/ may not exist, or may cover only some sessions. frames/manifest.json is the authority on what has been rendered, at what resolution, and under which settings.

The frame ↔ tick join

Frame i of a session is the image drawn after the simulation consumed tick record i's input. It shows the state that tick record describes — the same pos, yaw, pitch, cargo pose and wind that the row in data/ticks/<session>.parquet carries. So for inverse dynamics, the pair (frame at tick t−1, frame at tick t) is explained by the action in the row at tick t.

The join is written down, not implied by ordering. Every shard has a sidecar with one line per frame:

{"frame":0,"session":"00df…","tick":1,"chunk":0,"shard":"frames/00df…/00000.bin.gz","offset":0}

tick is the simulation's own tick counter, read at the instant the pixels were captured, and joins to the tick table on (session, tick). offset is the byte offset of the frame inside the uncompressed shard. An entry is self-contained: you can shuffle the index file and the join still resolves.

Shard format

A shard is frames × height × width × channels bytes of uint8, row-major, top-down, packed flat in capture order, then gzipped. No image container, no per-frame header. frames/manifest.json records the frame count, the uncompressed byte length and the SHA-256 of the uncompressed bytes of every shard, so a truncated shard is an error rather than a short read.

import gzip, json, numpy as np, pandas as pd
from huggingface_hub import hf_hub_download

REPO = "arjvnv/airmail-runs"
man  = json.load(open(hf_hub_download(REPO, "frames/manifest.json", repo_type="dataset")))
h, w, c = man["height"], man["width"], man["channels"]

rows = []
for s in man["shards"]:
    raw = gzip.decompress(open(hf_hub_download(REPO, s["path"], repo_type="dataset"), "rb").read())
    assert len(raw) == s["raw_bytes"], "truncated shard"
    imgs = np.frombuffer(raw, np.uint8).reshape(s["frames"], h, w, c)
    idx  = [json.loads(l) for l in
            open(hf_hub_download(REPO, s["index_path"], repo_type="dataset"))]
    for e in idx:
        rows.append((e["session"], e["tick"], imgs[e["offset"] // (h * w * c)]))

# join to the action labels
ticks = pd.read_parquet(f"hf://datasets/{REPO}/data/ticks/")

The frames deliberately contain no HUD

The rendered frames show the world only. No HUD, no crosshair, no overlays.

This is the single most important thing about them. The game's HUD displays a speedometer, a wind dial and the streak multiplier — which are exactly the quantities a model would be asked to infer from the pixels. A frame containing them is a frame with the answer printed on it, and any result trained on such frames would be measuring OCR, not dynamics. The renderer never attaches the HUD's DOM root for training frames, and the capture reads the WebGL drawing buffer directly, where DOM cannot reach.

npm run render-frames -- --hud exists for demo video and is not used here. Frames rendered that way are marked "hud": true in the manifest; do not train on them.

What they look like

Grayscale (Rec.709 integer luma of the sRGB output), at whatever resolution the manifest states. The default is 64×64, which preserves ego-motion, the horizon, roof planes and building massing but reduces a carried crate to a handful of pixels. 128×128 makes crates, stair treads and the delivery beam clearly legible for roughly four times the bytes and about the same render time.

Reproducibility

Every visual term in the renderer is a pure function of simulation state and tick — camera feel, wind dust, clouds, rooftop props and weather — and the render is driven with no mouse residual, no requestAnimationFrame, no wall clock, and alpha = 1 (the tick's settled state). Re-rendering the same chunk on the same machine produces byte-identical shards. Across a different GPU or driver, expect the same image but not necessarily the same bytes: rasterisation and multisample resolve are not specified to the bit. The manifest records the GPU, the browser and the three.js revision the frames were made with.

frames-wx/ — the same trajectories under four different skies

Because the images are regenerated rather than recorded, a run can be re-rendered under a sky it was never played in. frames-wx/ holds a subset of the corpus drawn four times over — once as clear, once as overcast, once as fog, once as rain — from the same recorded inputs, with the same seed and the same simulation.

frames-wx/<weather>/<session-id>/<00000>.bin.gz      identical format to frames/
frames-wx/<weather>/<session-id>/<00000>.index.jsonl
frames-wx/manifest.json

What it is for. Train an inverse-dynamics model on one weather and test it on another, and read the 4×4 matrix. Each off-diagonal cell is a controlled measurement of how much action inference depends on appearance rather than on structure, because the only thing that differs between the training set and the test set is the light and the air. A dataset scraped from video cannot produce this comparison at all: the trajectory and its appearance arrive welded together.

The trajectory is the same run, not a similar one. Weather in this game is presentation only and is drawn from its own random stream, never the simulation's, so forcing it cannot move the physics by a bit. Every frame here comes from the same replay the canonical frames/ render performs — same seed, same recorded inputs, same per-tick comparison against the recording, same refusal to emit a frame whose state no longer matches its label. The frames differ; the states behind them are bit-identical.

Coverage is identical across the four skies, and the manifest says so. Rendering happens one sky per run, so frames-wx/manifest.json is merged across runs and re-checks the whole set every time: by_weather.<kind>.coverage lists the exact (session, first_tick, last_tick, frames) for each sky, coverage is the shared list, and identical_across_weathers is the verdict. If a sky were ever short by a single frame the matrix would still compute and would quietly be comparing different trajectories, so that flag — not an assumption about the renderer — is what the comparison rests on. Tick ranges are contiguous, never subsampled: inverse dynamics needs consecutive pairs.

The manifest also records which sky each session was actually played under, so the diagonal of the matrix can be told apart from the three cells that re-light it.

man = json.load(open(hf_hub_download(REPO, "frames-wx/manifest.json", repo_type="dataset")))
assert man["identical_across_weathers"], man["coverage_problems"]
for wx in man["weathers"]:                       # clear, overcast, fog, rain
    for s in man["by_weather"][wx]["shards"]:    # e.g. frames-wx/fog/00df…/00000.bin.gz
        ...                                      # read exactly as frames/ above

frames/ is untouched by any of this: it holds each session under the sky it was really played in, and is the release to use when appearance should not be manipulated.

Baseline results — inverse dynamics from frames alone

notebooks/airmail-inverse-dynamics.ipynb in this repo runs the whole measurement end to end. It is shipped with its outputs, so the numbers below are readable without re-running it.

The task. Given consecutive frames and nothing else, predict which keys were held on that tick — the problem faced by anyone training on gameplay video without recorded inputs. Because this corpus carries the true action labels, the guessing can be graded, which is the thing a video-only corpus cannot do for itself.

Setup. 4 frames plus the 3 consecutive differences as extra channels (7 input channels), a small residual CNN (2.86M parameters) with its first stage kept at 48x48, one-cycle LR, no positive weighting. Held out by session, never by random tick — adjacent frames are near-identical and a random split leaks the answer into the test set. Chosen on validation sessions only.

The floor to compare against

in_forward is held on ~77% of ticks, so raw accuracy is not a usable headline: a constant "forward" predictor scores ~94% while predicting nothing else. Macro-F1 across the action bits is reported instead, and against the right floor:

The second is the correct floor for a threshold-tuned metric and is the one used throughout. With tuned thresholds a constant predictor is not pinned at F1 = 0 on a rare action — always answering "yes" scores 2p/(1+p) — so the majority-class figure understates chance. This correction came out of the shuffled-label control below, which initially looked like a leak and was not one.

Result

2.6x the floor. The model also beats the majority baseline on plain accuracy (0.956 vs 0.943), which an earlier positively-weighted variant did not — it bought rare-class recall with false alarms on the common classes.

Per action, on the held-out sessions, with support stated so nothing is over-read:

in_back and in_right are thin; in_throw and in_use are not reliable at this support and should not be quoted. in_use has zero validation positives, so its threshold was never tuned.

Two controls

A stronger number is only worth having if it is a number about the thing it claims to be about.

Shuffled labels land below the floor. Trained on nonsense the model recovers nothing, so nothing in the pipeline leaks the answer. (Against the majority-class baseline the same run reads +0.048, which is precisely why that baseline is the wrong yardstick here.)

Removing motion costs 58%. The model is reading movement, not memorising which roof it is on. The +0.016 the static model keeps over the floor is what the scene alone is worth — posture, camera angle and place.

Visual robustness — the same trajectories under four skies

The measurement frames-wx/ exists for. Train on frames rendered under one sky, test on the identical trajectories rendered under another: same seed, same recorded inputs, same physics, bit-identical state, verified byte-identical labels. Only the pixels differ.

Macro-F1, rows trained on, columns tested on (Colab T4):

• in-distribution (diagonal) 0.302 · transfer (off-diagonal) 0.210
• gap 0.092, a 30.5% relative drop
• diagonal spread 0.038 — the gap is 2.4x the spread, so it clears the variation between four separately-trained models rather than sitting inside it
• reproduced on Apple M4 / MPS at 0.300 / 0.201, a 33.2% drop

Action inference here is substantially a function of appearance, not only of structure. The comparison is only valid because the four renders are the same runs — which is why the headline model uses no photometric or geometric augmentation: brightness jitter teaches invariance to lighting by construction and would shrink the very gap being measured. Run deliberately as a secondary experiment, per-window photometric jitter closes 77% of the gap (diagonal 0.331, off-diagonal 0.308) while raising in-distribution performance 0.030.

Reproducing it

Open notebooks/airmail-inverse-dynamics.ipynb in Colab, select a T4, and run all. It prompts for a HuggingFace read token and takes roughly 2.5-3.5 hours. Seeds are set and reported. The numbers above came from two independent runs on different hardware.

Limitations

• One map. All sessions are the same rooftop district (level_hash identifies it). The layout of parcels and drop zones varies with the session seed; the geometry does not.
• One player. Recorded by the game's author. This is not a diverse population of policies.
• Frames are rendered, not recorded. No video was ever captured. frames/ holds images regenerated from the recorded inputs and state (see above); if frames/ is absent, this release is state and action only. The frames are byte-reproducible on one machine, not across GPUs.
• sim_version is a hard boundary. Trajectories recorded under different simulation versions are not the same dynamics. Group by sim_version before fitting anything.
• Constants can move. They are recorded per chunk, and a live edit emits CONST_CHANGE. Almost every session uses the shipped defaults, but check before pooling.
• The mouse is quantised. mouse_dx / mouse_dy are the integer pixel deltas the simulation consumed; the sub-pixel residual stayed outside the simulation and is not recorded.
• Idle and on-rail ticks are included and must be filtered by the consumer (see above).
• The reported model is a floor, not a ceiling. The epoch budget was set to fit the available compute, and the selected arms were still improving on validation when they stopped. Longer training would very likely raise both the headline and the diagonal of the robustness matrix.
• The 96x96 resolution choice is an inference, not a direct test. Three ablation arms showed that early feature-map resolution, not input pixels, was the binding constraint, so no 128x128 render was run. The direct test would be a stride-1 stem or the re-render itself.
• Rare actions are not evaluable. in_throw (125 positives) and in_use (85) do not have the support to carry a number. Their metrics are reported for completeness and flagged, not claimed.

Licence

CC BY 4.0.