CoVisco-L-14

CoVisco-L-14 (Vision Encoder)

CoVisco-L-14 is the Stage 1 vision encoder of the CoVisco pipeline (Codec-Native Vision Encoder with Native Token Compression for Efficient Unified Visual Understanding). It encodes images and videos in a unified way and is trained by SigLIP-style distillation against two teacher branches.

This repository releases only the vision-encoder weights in the safe .safetensors format, together with the minimal model code and a self-contained loader. The model is a transformers.PreTrainedModel and is loadable via from_pretrained (see below).

Architecture

• Backbone: ViT-L/14, image_size=224, patch_size=14, layers=24, width=1024, heads=16
• Native token compression: num_query_per_seg=100 learnable abstract tokens, segment_t_size=32
• 4D RoPE (rope_theta=10000) for spatio-temporal positions
• Three projection heads (each LayerNorm + Linear, output L2-normalized):
• to_image_caption → 1536-d, SigLIP2 space (image classification / retrieval)
• to_image → 1536-d
• to_video_caption → 4096-d, Qwen3-VL-Embedding space (video classification / retrieval)
• Independent logit_scale / logit_bias per contrast pair

Files

• model.safetensors — model weights (fp32)
• config.json — encoder config + head dims + preprocessing config; includes architectures / model_type / auto_map so the repo is HF-loadable
• covisco_vit.py, covisco_model.py, reconstruction_decoder.py — model code
• load_covisco.py — self-contained loader + image-embedding example
• codec_visidx.py — codec-based token filtering (see below)
• requirements.txt, LICENSE

Load via from_pretrained

The model is a transformers.PreTrainedModel, so you can load it with the standard HF API (needs trust_remote_code=True because the modeling code ships in this repo via auto_map):

from transformers import AutoModel

REPO = "ernie-research/CoVisco-L-14"
model = AutoModel.from_pretrained(REPO, trust_remote_code=True).eval()
# or, importing the class directly (no trust_remote_code needed):
# from covisco_model import CoViscoModel
# model = CoViscoModel.from_pretrained(REPO).eval()

The only preprocessing knobs a user adjusts are the resolution and (for video) segment_t_size. The raw parameters (size / mean / std / interpolation) live in model.config.preprocess_cfg, and load_covisco.py ships ready-made helpers so you do not hand-roll any transform. The same per-frame transform serves both images and video — the only extra step for video is sampling frames and stacking them into a (B, C, T, H, W) clip, which encode_video does for you:

• Per-frame transform (shared): build_preprocess(cfg, size=...) for a fixed/square resolution (see Programmatic usage), or build_native_preprocess(cfg, max_side=...) for native, aspect-preserving resolution (see Native resolution).
• Image end-to-end: encode_image(model, preprocess, img) → single image.
• Video end-to-end: read_video_frames(path, num_frames) → encode_video(model, preprocess, frames), with build_model(..., segment_t_size=...) setting the temporal segment size (see Video usage).

Resolution is decided entirely by the preprocessing (the H x W of the pixel_values you feed in); there is no resolution knob to set on the model. The one runtime parameter that is not fixed by preprocessing is segment_t_size (how video frames are split into temporal segments). If you stay on the pure HF API (no helpers), override it directly on the live encoder (no weight shape depends on it):

model = AutoModel.from_pretrained("ernie-research/CoVisco-L-14", trust_remote_code=True)

model.encoder.segment_t_size = 16          # video frame count T must be a multiple of it
model.encoder.config.segment_t_size = 16   # keep config in sync (optional)

The load_covisco.py helpers below wrap this with the torchvision preprocessing and apply such overrides at build time.

Quick start

pip install -r requirements.txt
# smoke test (blank 224x224 image)
python load_covisco.py
# real image, square 224 preprocessing
python load_covisco.py --image your.jpg
# native resolution (keeps aspect ratio, long side <= 1400 px)
python load_covisco.py --native_resolution --image your.jpg
# native resolution with a smaller memory cap
python load_covisco.py --native_resolution --max_side 896 --image your.jpg

CLI flags: --config, --weights, --image, --device, --native_resolution, --max_side, --segment_t_size.

Programmatic usage (image)

from load_covisco import build_model, build_preprocess, encode_image
from PIL import Image

model, cfg = build_model("config.json", "model.safetensors", device="cuda")
preprocess = build_preprocess(cfg, size=224)  # square 224; explicit resolution
emb = encode_image(model, preprocess, Image.open("your.jpg"), device="cuda")
# emb: [1, 1536], L2-normalized (SigLIP2 branch, to_image_caption)

# 4096-d Qwen3-VL-Embedding-space embedding instead. For images this head is
# keyed `to_image_video_caption` (it reuses the video-caption head weights;
# `to_video_caption` only exists under modality="video").
emb = encode_image(model, preprocess, Image.open("your.jpg"), device="cuda",
                   branch="to_image_video_caption")

Native resolution (images)

The encoder is RoPE-only (no learned position table) and derives its patch grid as h = H // 14, w = W // 14 at runtime, so it accepts any H x W that is a multiple of the patch size — including non-square inputs:

from load_covisco import build_model, build_native_preprocess, encode_image
from PIL import Image

model, cfg = build_model("config.json", "model.safetensors", device="cuda")
preprocess = build_native_preprocess(cfg, max_side=1400)

emb = encode_image(model, preprocess, Image.open("doc_page.png"), device="cuda")

build_native_preprocess keeps the original aspect ratio and only scales the long side down to max_side (memory cap) and rounds each side to a multiple of patch_size. This is the recommended mode for text-heavy document pages, where the square 224 center-crop destroys detail.

Because a batch is a single dense tensor (no resolution packing), encode one image at a time when using native resolution.

Video usage

Videos are passed as a (B, C, T, H, W) tensor (channel before time). T must be a positive multiple of segment_t_size; the encoder splits the clip into T // segment_t_size temporal segments. segment_t_size is a pure runtime parameter (RoPE positions are computed from the runtime frame grid; no weight shape depends on it), so override it when loading — no manual config rebuild needed:

from load_covisco import (build_model, build_preprocess,
                          read_video_frames, encode_video)

NUM_FRAMES = 16      # frames uniformly sampled per clip
SEGMENT_T_SIZE = 16   # frames per temporal segment (must divide NUM_FRAMES;
                      # 16 -> 1 segment here; config.json default is 32)
IMAGE_SIZE = 224      # square resolution per frame (any multiple of 14 works)

# segment_t_size is runtime-only (RoPE, no weight shape depends on it):
# same weights, override the config.json default (32).
model, cfg = build_model("config.json", "model.safetensors", device="cuda",
                         segment_t_size=SEGMENT_T_SIZE)

# Same per-frame transform as images. For aspect-preserving native-resolution
# frames use build_native_preprocess(cfg, max_side=1400) instead.
preprocess = build_preprocess(cfg, size=IMAGE_SIZE)

# read_video_frames needs PyAV (optional): pip install av
frames = read_video_frames("video.mp4", NUM_FRAMES)   # list of PIL images
video_emb = encode_video(model, preprocess, frames, device="cuda")
# video_emb: (1, 4096), L2-normalized (to_video_caption, Qwen3-VL-Embedding space)

encode_video applies the shared per-frame preprocess, stacks the frames into the (B, C, T, H, W) tensor the encoder expects, and runs the forward pass. If you already have a (T, C, H, W) clip tensor you can pass it directly instead of a list of PIL frames.

On load, the effective segment_t_size is printed (e.g. [INFO] effective segment_t_size = 16 (video frame count T must be a multiple of it)). A frame count that is not a multiple of the effective segment_t_size raises an AssertionError at forward time; a clip shorter than segment_t_size is also accepted (single segment, T frames) — positions stay consistent because RoPE is computed from the runtime grid.

Codec-based token filtering (video)

The encoder natively supports sparse patch selection: passing visible_indices (visidx) makes get_segments compute only the selected patches and derive 4D-RoPE from them — a codec-style token budget without any mosaic packing. codec_visidx.py generates these indices from video codec signals, with the same scoring math as the training-time precompute_codec_visidx pipeline:

• Backend 1 — cv_reader (exact, matches training): install Compressed-Video-Reader (C++ extension + patched FFmpeg) to read H.264/HEVC macroblock motion vectors and DCT residuals. The scoring functions are embedded verbatim, so the visidx is numerically identical to the training-time precompute.
• Backend 2 — PyAV (fallback, no extra deps): FFmpeg AV_FRAME_DATA_MOTION_VECTORS side data for motion; luminance frame differences approximate the residual.
• Selection (both backends): per-segment equal top-K (keep_ratio of segment_t_size * patches_per_frame), first frame of each segment always kept in full, indices sorted ascending — the layout get_segments requires.

Two usage modes:

End-to-end (scoring runs inline, right before the forward pass):

from codec_visidx import compute_codec_visidx, sample_frame_ids, _decode_frames
from load_covisco import build_model, build_preprocess, encode_video

NUM_FRAMES, SEGMENT_T_SIZE, KEEP_RATIO = 64, 16, 0.57

visidx, meta = compute_codec_visidx("video.mp4", num_frames=NUM_FRAMES,
                                    segment_t_size=SEGMENT_T_SIZE,
                                    keep_ratio=KEEP_RATIO)

model, cfg = build_model("config.json", "model.safetensors", device="cuda",
                         segment_t_size=SEGMENT_T_SIZE)
preprocess = build_preprocess(cfg, size=224)   # codec mode needs square preprocessing

# Decode the SAME frames the scoring used. sample_frame_ids samples over
# meta["total_frames"] (the probed stream length), so this guarantees patch
# index i still refers to the same patch — don't substitute read_video_frames
# here, it samples over the decoded-frame count instead.
_, rgbs, *_ = _decode_frames("video.mp4",
                             sample_frame_ids(meta["total_frames"], NUM_FRAMES))

# Pass visidx into encode_video to run in codec (sparse-patch) mode.
video_emb = encode_video(model, preprocess, rgbs, device="cuda",
                         visible_indices=visidx)   # (1, 4096)

Pre-extracted (batch datasets; output layout matches the SFT precompute_codec_visidx pipeline):

# videos.jsonl: one {"video": "/path.mp4", "key": "unique_id"} per line
python codec_visidx.py --jsonl videos.jsonl --out_root cache/ \
    --num_frames 64 --segment_t_size 16 --keep_ratio 0.57
# writes cache//{visidx.npy, frame_ids.npy, meta.json}

import numpy as np
from codec_visidx import load_visidx, _decode_frames
from load_covisco import encode_video

# Reuse the exact frame ids saved alongside the visidx.
frame_ids = np.load("cache/<key>/frame_ids.npy").tolist()
_, rgbs, *_ = _decode_frames("video.mp4", frame_ids)
video_emb = encode_video(model, preprocess, rgbs, device="cuda",
                         visible_indices=load_visidx("cache/<key>"))

Constraints: the clip fed to encode_video must use the same num_frames linspace sampling, the same square image_size preprocessing, and the same segment_t_size as the visidx computation (meta.json records these for verification). With keep_ratio=0.57 and a 224² grid, each 16-frame segment keeps ~2329 of its 4096 patches (~57%), cutting encoder FLOPs correspondingly.

Model interface

model(pixel_values, modality=..., run_decoder=False) returns a dict:

modality selects which projection heads are computed: "image" → to_image / to_image_caption / to_image_video_caption, "video" → to_video_caption only, None → all. The 4096-d image head is keyed to_image_video_caption (not to_video_caption) even though it shares the same projection weights.

Extracting token-level representations

Pass return_intermediates=True to also get the pre-pooling token outputs. This is what the CoVisco SFT stage consumes as visual inputs (e.g. as the vision tower of an LLM):

from load_covisco import build_model, build_preprocess
from PIL import Image

model, cfg = build_model("config.json", "model.safetensors", device="cuda")
preprocess = build_preprocess(cfg, size=224)
px = preprocess(Image.open("your.jpg")).unsqueeze(0).cuda()   # (B, C, H, W)

out = model(px, modality="image", run_decoder=False,
            return_intermediates=True)

out["pooled_output"]      # (B, 1024)          pooled representation
out["query_tokens"]       # (B, 1, 100, 1024)  per-segment abstract/query tokens
out["vit_tokens"]         # (B, 1, 256, 1024)  per-segment ViT patch tokens
# 224x224 / patch 14 -> 16x16 = 256 patches per frame/segment
# (the first 100 tokens of every segment are the learnable query tokens)

For video the layout is (B, S, ...) with S = T // segment_t_size segments, e.g. 16 frames at segment_t_size=16 (one segment of 16 frames) → query_tokens (B, 1, 100, 1024) and vit_tokens (B, 1, 16*256, 1024) (256 patches per frame, all frames of the segment).

Notes:

• The projection heads consume pooled_output; query_tokens / vit_tokens are the uncompressed sequence outputs if you need token-level features.
• return_attentions=True additionally returns per-layer attention weights (requires eager attention, not flash-attn).

Classification / retrieval needs a teacher text encoder

This repo ships only the vision tower. To run zero-shot classification or text↔image/video retrieval you must match embeddings against the frozen teacher text encoder of the corresponding branch:

• to_image_caption (1536-d) ↔ SigLIP2 text tower (timm/ViT-gopt-16-SigLIP2-384, pretrained=webli)
• to_video_caption (4096-d) ↔ Qwen/Qwen3-VL-Embedding-8B text encoder

The embedding spaces are branch-specific and not interchangeable.

Notes

• The encoder is stored twice in the state dict, under encoder.* and its alias visual.* (the model sets self.visual = self.encoder). Both resolve to the same module; a clean load reports no missing/unexpected keys. This is why the file is ~2.4 GB for a ~320 M-parameter encoder.
• image_size in config.json is informational only — it does not affect any weight shape or forward computation (patch embedding is a conv and positions are RoPE), so the same weights run at any resolution.
• flash_attn is optional; the code falls back to standard attention if it is not installed.
• use_reconstruction=False for the released weights (no decoder parameters); pass run_decoder=False to skip the decoder path entirely.
• The training-time data-augmentation paths (uniform_sample_frames / frame_concat) have been removed from this release; inference always uses the get_segments path with the frame count and segment_t_size you configure.

License

MIT License.

Citation

@article{liu2026covisco,
  title={CoVisco: Codec-Native Vision Encoder with Native Token Compression for Unified Image-Video Understanding},
  author={Liu, Yulong and Han, Xiaotian and Shang, Junyuan and Ding, Yuchen and Zhang, Zhenyu and Wang, Shuohuan and Zhu, Guibo and Han, Sirui and Yu, Dianhai},
  journal={arXiv preprint arXiv:2609.39924},
  year={2026}
}