LastKV-NVS-DL3DV

LastKV for scene-level novel-view synthesis, trained on DL3DV with a three-stage progressive-resolution curriculum: 128 × 128 → 256 × 256 → 512 × 512. This repository releases the third-stage checkpoint at 512 × 512.

A 24-layer Transformer with hidden width 768, patch size 8, and 184.8M parameters. LastKV shares the final layer's historical KV cache across attention layers. Training uses BF16 autocast; the released model weights are stored in FP32.

Training curriculum

Stage 1 starts from random initialization. Its perceptual-loss branch continues from the 4,000-pass L2 checkpoint, preserving optimizer, scheduler, and step-counter states.

At each resolution transition, only model weights are inherited; the optimizer, learning-rate scheduler, and step counters are reset.

Checkpoint steps are stage-local forward/backward pass counters (fwdbwd_pass_step); successful optimizer-update counts can be lower when updates are skipped. The selected checkpoints define the ancestry of the released weights. The configured budgets determine the cosine schedule and should be preserved when reproducing this training recipe, even when an earlier checkpoint is selected.

Training settings

• Optimizer: AdamW, betas (0.9, 0.95), weight decay 0.05.
• Schedule: 2,500-step learning-rate warmup followed by cosine decay in each stage.
• Supervision: pixel L2 plus 0.5 × VGG19 PerceptualLoss.
• Loss warmup: the first 4,000 passes of stage 1 use only L2. Stages 2 and 3 enable the perceptual term from the start.
• Training: 12 views per sequence, with teacher-forced sequential conditioning.

The VGG19 training perceptual loss and the LPIPS evaluation metrics use different definitions.

Checkpoint and use

Download model.pt. This is a PyTorch training checkpoint containing model, optimizer, lr_scheduler, fwdbwd_pass_step, and param_update_step.

The released checkpoint contains 6,950 stage-3 passes and 6,950 successful parameter updates. Its optimizer and scheduler states belong to stage 3.

To download and extract the model weights:

import torch
from huggingface_hub import hf_hub_download

path = hf_hub_download("ZefanCai/LastKV-NVS-DL3DV", "model.pt")
checkpoint = torch.load(path, map_location="cpu", weights_only=True)
state_dict = checkpoint["model"]

Rendering requires the matching LastKV NVS implementation, the 512 × 512 configuration, and compatible camera preprocessing. See checkpoint_info.json for the file size, SHA-256, and tensor metadata.

Evaluation of the released checkpoint

Evaluation uses the DL3DV benchmark, with 140 scenes, 512 × 512 images, 24 fixed views per scene, and seed 4242. Sequential conditioning uses ground-truth preceding views. Metrics are averaged over the 23 target views within each scene, then across scenes.

Project page · Paper