Xiaomi-Lens-0 is a 4B multimodal language model built on Qwen3.5-4B. It adds a lightweight pixel-mask decoder and is trained with joint language and mask supervision. Each generated <|mask|> token identifies an image region that the decoder turns into a segmentation mask.
The model supports referring expression comprehension and segmentation, multi-instance grounding, reasoning segmentation, embodied referring, and fine-grained visual perception. It can produce masks in its final answer or during reasoning.
This repository contains the model weights, configuration, tokenizer, processor, and chat template. Inference uses the XiaomiLensQwen3_5ForConditionalGeneration class from the accompanying Xiaomi-Lens-0 code repository. Install that package to load the model and decode masks.
News
Overview video
Watch on the project page · Download the MP4
Task showcases
Recorded model outputs across six task domains. Each task includes four examples from the GitHub project.
Referring Expression Comprehension
Dark brown bear: “dark brown bear at left” |
Boy with a phone: “boy looking at phone” |
Smallest spoon: “smallest wooden spoon of the group” |
Runner behind the pitcher: “the player shown running behind the pitcher” |
Referring Expression Segmentation
Grapefruit wedge: “second wedge grapefruit from left to right” |
Laptop with a sign: “laptop with small white windows sign” |
Kite on the grass: “a red, black, and white kite lying on the grass” |
Brown cow: “a large brown cow standing in the grass” |
Multi-Instance Grounding
Bottles: “all bottles” |
Oranges: “all oranges” |
People: “all people” |
Surfboards: “all surfboards” |
Reasoning Segmentation
Tangy condiment: “What can be squeezed onto seafood as a tangy flavor enhancer?” |
Gift packaging: “What could be used to package a gift?” |
Lawn care: “What would be useful for keeping a lawn well-groomed?” |
Orbiting object: “What object orbits the Earth to complete missions?” |
Embodied Referring
Old book: “Hand me the old book with a black and white cover” |
Facial cleanser: “Hand me the facial cleanser on the left” |
White cup: “Pass me the white cup on the left” |
Pepsi can: “Hand me the can of Pepsi on the right” |
Fine-Grained Perception
Climbing stones: “How many climbing stones are there?” → 5 |
Leaves: “How many leaves are there?” → 6 |
Chair and bicycles: “Where is the chair relative to the bicycles?” → To the left |
Trailer color: “What's the color of the trailer?” → Orange |
Quick start
Inference requires Python 3.10 or later and a CUDA GPU with bfloat16 and FlashAttention-2 support. From a checkout of the Xiaomi-Lens-0 inference repository, install the dependencies and package:
pip install -r requirements.txt
pip install -e .
Run the inference script with the downloaded checkpoint directory and an image:
python infer.py \
--checkpoint /path/to/Xiaomi-Lens-0 \
--image /path/to/image.jpg \
--text 'Locate all instances in the image that match the description "the cat on the left". Report bbox coordinates and masks.' \
--output result.jpg --draw-bbox
The script prints the generated text and saves an image with the predicted masks and bounding boxes. --text accepts a complete instruction; the checkpoint's chat template supplies the image tokens and assistant prefix.
Each grounded instance is represented as <bbox>[x1, y1, x2, y2]</bbox><mask><|mask|></mask>, with box coordinates normalized to 0–1000. Mask decoding uses the contextual state of each <|mask|> token and the vision encoder's features. The inference script excludes masks inside <think> from the final-answer overlay.
Serving with vLLM
Use a separate environment for vLLM serving. From the inference repository, install the serving dependencies:
pip install -e ".[vllm]"
Run text generation and mask decoding in separate terminals:
# Terminal 1: text generation on port 8000
bash src/xiaomi_lens/integrations/vllm/run_vllm.sh /path/to/Xiaomi-Lens-0
# Terminal 2: mask decoding on port 8001
bash src/xiaomi_lens/integrations/vllm/run_vllm_mask.sh /path/to/Xiaomi-Lens-0
# Terminal 3: query both services
python -m xiaomi_lens.integrations.vllm.client \
--image /path/to/image.jpg --query "the cat on the left" --output result.png
The client must run on the same machine as the services because it sends the image path to the mask server. The launchers prepare temporary checkpoint views for vLLM. If you override MAX_PIXELS, use the same value for both services.
Training and evaluation
Training first pretrains the mask decoder, then fine-tunes the full model with language and pixel-level supervision. The inference repository's README provides the benchmark tables and recorded examples across the six capability domains.
Predictions can miss objects or produce inaccurate boxes and masks, particularly for small, occluded, or ambiguous regions. Masks are decoded on the image canvas produced by the processor; preserve the image resizing and coordinate mapping when consuming them. The supplied inference path supports images.
License and attribution
The model is released under the Apache License 2.0.
Xiaomi-Lens-0 derives from Qwen3.5-4B, whose weights are also licensed under Apache 2.0. Xiaomi-Lens-0 adds the pixel-mask decoder and fine-tunes the model for grounded language and segmentation tasks.
- Qwen3.5-4B: Copyright 2026 Alibaba Cloud.
- Xiaomi-Lens-0 modifications: Copyright 2026 Xiaomi Corporation.
Citation
@misc{xiaomilens0,
title = {Xiaomi-Lens-0 Technical Report},
author = {Pei Fu and Yike Liu and Yue Sun and Hanmo Wang and
Xiuwen Xi and Ruoceng Zhang and Shaojie Zhang and
Chao Jiang and Hang Li and Bingrui Liu and Zhenbo Luo and
Heng Qu and Changjian Xu and Ming Xu and Jian Luan},
year = {2026},
eprint = {2610.23279},
archivePrefix = {arXiv},
primaryClass = {cs.CV},
url = {https://arxiv.org/abs/2610.23279}
}
