Matrix-Game-4.0

Source code | Project website

Scene-specialized causal video renderers for Medieval Castle, Ancient Egypt, and City Racing. An executable game determines gameplay, physics and input, while the model re-renders clean low-poly RGB. HUD and sound remain separate, frame-aligned presentation layers.

Contents

Each scene checkpoint has 5,001,016,512 BF16 parameters. The weights require the Matrix-Game 4.0 inference source; they are not drop-in Diffusers pipelines. Training code and training data are not included. Shared encoders/codecs are installed once for all scenes. See game asset details.

Download and Layout

Use the source release's manifest downloader with a pinned Hub commit revision and --repo-id Skywork/Matrix-Game-4.0 --repo-type model for both model and game-asset manifests. See releases/README.md in the source repository for exact commands. A scene selection downloads that scene plus its shared dependencies.

The shared root config.json is installed as resources/release-config.json and verified by the model manifest. It describes this release; it is not a Diffusers/Transformers pipeline configuration or the optional DiT resources/base/config.json. Game-asset-only downloads do not fetch it.

The T5 encoder, tokenizer and Wan2.2 VAE are fetched directly from the official base-model repository. The manifest pins that revision and verifies every file's SHA256; no manual base-model download is needed. These shared files are not duplicated here.

The downloader maps Hub scene folders to the runtime paths resources/models/castle/, resources/models/ancient-egypt/ and resources/models/racing/. Do not infer the local directory layout from the Hub folder names alone. The repo_path field in releases/models.json describes this mapping. Both length and SHA256 are checked before a file is installed.

Inference Contract

• Output resolution: 480x864 RGB. One latent chunk corresponds to four RGB frames after startup; the startup chunk contains one RGB frame.
• Castle/Egypt use an independent identity sink with fixed-relative RoPE. Racing uses the supplied image as output frame zero and an absolute-position sink. These policies are not interchangeable.
• Quality preset: Wan VAE encoding/decoding, two denoising steps (1000, 500), native DiT. Realtime preset: TAEHV encoding/decoding, one step (1000), optimized single-GPU DiT and CUDA graphs.
• CFG is 1 and the scheduler shift is 5. The attention cache contains 21 latent frames including the sink. Racing skips denoising on its supplied first frame.
• Shooter captions are fixed. Racing supports fixed or per-chunk dynamic engine captions. Clean RGB, not the HUD or keyboard overlay, is the model input.

Use base/taew2_2.pth for the full streaming fast codec; a standalone decoder file is not a replacement for its encoder. Use the exact shared files in the manifest, not an arbitrary upstream version or Wan2.1's 16-channel VAE.

Limitations

The models are specialized to the supplied low-poly scenes and camera profiles. Unseen geometry, rapid camera motion, occlusion and long sessions can cause appearance drift or geometry errors. Neural output does not change collision or gameplay. One-step/TAEHV and two-step/Wan have different quality and cost. There is no guaranteed frame rate, consumer-GPU memory fit, or exact scene reconstruction. This model card does not claim a performance benchmark.

License and Attribution

Project scene weights, sink images and game assets: Apache-2.0, Skywork AI. The shared Wan/umT5 resources retain their upstream Apache-2.0 terms: Wan2.2-Fun-5B-Control, umT5. TAEHV is by Ollin Boer Bohan under MIT; see licenses/TAEHV-LICENSE and its upstream repository. Racing audio includes CC0 and CC BY 3.0 material; retain racing/assets/audio/CREDITS.md. See THIRD_PARTY_NOTICES.md for code dependencies.