procgen-renderformer

Procgen RenderFormer Dataset

Procedurally generated indoor scenes with ground-truth path-traced renders and precomputed 3-slat VAE latents, built for training RenderFormer-style neural renderers. Each sample is one scene observed from 14 camera poses along an orbit.

Configs

Sample layout

This is a WebDataset. Each sample carries:

<key>.000.webp .. <key>.013.webp   GT renders, 512x512 uint8 sRGB, lossless WebP
<key>.latents.safetensors          coords + per-role feats for all 14 frames
<key>.json                         scene provenance, camera, lighting, materials

The latents hold three roles sharing one coordinate grid per frame (a unified-proxy encode):

Frames are concatenated along the token axis with a frame_offsets index rather than stored separately, so frame i is coords[frame_offsets[i]:frame_offsets[i+1]]. Token count varies slightly per frame (~2750) because light proxies are voxelized in per frame.

Loading

from datasets import load_dataset

ds = load_dataset("eternity304/procgen-renderformer", "validation", split="validation", streaming=True)
sample = next(iter(ds))

Fields arrive as raw bytes. Decode them with:

import io, json
import numpy as np
from PIL import Image
from safetensors.torch import load as st_load

ROLES = ("shape", "slat1", "slat2", "slat3")

def decode(sample):
    frames = np.stack([
        np.array(Image.open(io.BytesIO(sample[k])).convert("RGB"))
        for k in sorted(k for k in sample if k.endswith(".webp"))
    ])                                              # uint8 [14, 512, 512, 3]
    tensors = st_load(sample["latents.safetensors"])
    return frames, tensors, json.loads(sample["json"])

def frame_latents(tensors, i):
    """coords [N,3] int32 and {role: feats [N,32] fp16} for frame i."""
    a, b = tensors["frame_offsets"][i], tensors["frame_offsets"][i + 1]
    return tensors["coords"][a:b], {r: tensors[f"{r}_feats"][a:b] for r in ROLES}

coords are integer latent-voxel indices in [0, latent_grid) where latent_grid = resolution // VAE_SPATIAL_DOWNSAMPLE. Map them to [-0.5, 0.5] centres at model time.

Provenance

Scenes compose meshes from Objaverse-LVIS with PBR materials from PolyHaven, ambientCG, and cgbookcase (all CC0). Every scene records the source asset UIDs under assets[].uid in its JSON, so individual renders remain traceable to their source meshes.

Renders are derivative works of the source meshes. Objaverse licenses are per-asset and mixed; consult assets[].uid against the Objaverse annotations for the terms covering any particular scene.

Generation

Rendered at 64 spp, 512x512, 14 frames per scene at 12 fps, Blender/OpenGL camera convention. Full generator configuration is embedded per scene under generator_config and _dataset.