sceneseg-p150

SceneSeg (Autoware VisionPilot): the scene segmentation network of the Autoware Foundation's VisionPilot camera stack, as its scene_seg_model node deployed it, ported to one Tenstorrent Blackhole p150 with tt-nn. One camera image of any size in (converted to BGR8, as the node receives it); the VisionPilot foreground mask (mono8, 255 = foreground, at the input's size) and a 3-class map (background / foreground / road) out, with the node's exact pre- and post-processing. Weights: weights/ of this repo @ a6829b24 (VisionPilot's SceneSeg_FP32.onnx and its .pth as safetensors, re-hosted from the upstream Google Drive links) · Paper: none (model card AutowareFoundation/SceneSeg) · Autoware package: VisionPilot models (node scene_seg_model) · Training code: autowarefoundation/vision_pilot Models/ @ ca58cb50 · Port: code/

Runs on p150 (mesh P150). Configuration: dispatch on the ETH cores, 1 command queue, 12×10 compute grid, one metal trace per frame. Two serve profiles, the input channel order the network is fed: bgr (the deployed VisionPilot order, the default) and rgb (the training order). Numerics: backbone weights as two bf16 terms (hi + lo) with float32 activations fed as two bf16 terms; the decoder's 3x3 convs, its sub-pixel up-blocks, the class-margin conv and SceneContext with one bf16 weight term at HiFi3; biases added in fp32, fp32 accumulation. All numbers on this card were measured in this configuration (bgr unless stated).

Second release (optimized): 12.58 ms per frame on the device (61.06 ms in the first release, 4.9x), 22.5 ms per model() call on the shipped sample (90.9 ms before). Five custom ttnn.generic_op kernel families do most of the work: conv / linear epilogues (which also take the class decision), the depthwise convs, the sub-pixel decoder blocks, the SE MLPs and the stem. Every accuracy gate of the first release still passes, unchanged. Details: OPT_REPORT.md.

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

Quickstart (Python)

Prerequisite: a tt-metal / ttnn environment at tt-metal 44d66500520 with patches/tt-metal-eth-dispatch.patch and patches/tt-metal-reshape-rm-sys1419.patch applied. ttnn is not on PyPI.

hf download changh95/sceneseg-p150 --exclude "image/*" --exclude "weights/*" --local-dir sceneseg-p150 && cd sceneseg-p150
pip install -e .                        # adds numpy<2, pillow, pyyaml, onnx, safetensors, huggingface_hub; ttnn and torch come from tt-metal
pip install -e ".[server,test]"         # optional: the HTTP server and the tests

Run the snippet from the model repo root: code/tt_sceneseg/samples/highway_normal_1.png is a path relative to it.

from tt_sceneseg import SceneSeg

with SceneSeg.from_pretrained(device_id=0) as model:      # weights -> your HF cache, trace captured
    out = model("code/tt_sceneseg/samples/highway_normal_1.png")   # path, bytes, PIL image or RGB uint8 HxWx3 array of any size

mask = out.mask                    # uint8 (H, W) at the input size: 255 = foreground (the VisionPilot topic)
print(out.class_counts())          # pixels per class of out.class_map (background / foreground / road)

• from_pretrained downloads weights/SceneSeg_FP32.onnx (+ LICENSE, NOTICE, provenance; 194 MB) of this repo at the pinned commit a6829b246a0 to your HF cache, opens the chip, builds the graph and captures the metal trace. With a warm kernel cache the load takes about 19 s; the first load on a machine also compiles the kernels (4.6 min here with an empty kernel cache, measured before R8.1, which saves about 21 s).
• The trace is captured during the load, so no call compiles anything: the first call took 28-30 ms here, the later ones 22.5-24 ms.
• The with block releases the trace and closes the chip. Without with, call model.close().

• The API gives the same output as the HTTP server /predict: both share the decoders, the device trace and the host post-processing (checked on the device by test_api_equals_server).
• One model uses one chip; calls from several threads are serialised.
• Full reference: PYTHON.md. Runnable example: examples/quickstart.py (also writes quickstart_mask.png and quickstart.png, the class map over the image).

Serving (HTTP)

tt-model pull  changh95/sceneseg-p150 --with-weights
tt-model serve changh95/sceneseg-p150       # or with tt-cli: tt serve changh95/sceneseg-p150; --profile rgb for the training order
python3 code/tt_sceneseg/server/client.py --image CAM_FRONT=code/tt_sceneseg/samples/highway_normal_1.png --out req.json
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json
tt model stop changh95/sceneseg-p150

• The image does not contain the weights. --with-weights puts them in your HF cache.
• The server uses port 20000 (or the next free port). It is ready when the log shows Application startup complete (20 s with a warm kernel cache, measured with uvicorn on the host).
• Two serve profiles (tt-model profiles changh95/sceneseg-p150): bgr (default) and rgb.
• client.py builds the request with the standard library only; add --url http://127.0.0.1:20000 to send it.
• POST /predict: images (one entry: base64 PNG / JPEG of the camera image, any size, decoded to BGR8 like the node's input), optional params (class_map true), output_format. Also GET /health, GET /info, GET /v1/models (stub). Contract: SERVING.md section 3.

The served body for the shipped sample (bgr profile; PNG data elided):

{"model": "sceneseg-p150", "frame_id": "camera",
 "meta": {"channel_order": "bgr", "source_hw": [414, 727], "network_hw": [320, 640], "foreground_class": 1, "mask_value": 255, "foreground_fraction": 0.040156},
 "mask": {"format": "png", "key": "mask", "dtype": "uint8", "shape": [414, 727], "data": "<base64 PNG>"},
 "class_names": ["background", "foreground", "road"],
 "class_counts": {"background": 166581, "foreground": 12086, "road": 122311},
 "class_map": {"format": "png", "key": "class_map", "dtype": "uint8", "shape": [414, 727], "data": "<base64 PNG>"},
 "timing_ms": {"preprocess": 1.976, "device": 13.236, "postprocess": 0.749, "total": 44.344, "decode": 18.425, "model_call": 15.978}}

• mask is what the VisionPilot node publishes on /autoseg/scene_seg/mask: mono8 at the input's size, 255 where the per-pixel argmax is class 1 (foreground), else 0, resized back from 640x320 with nearest neighbour.
• class_map is the 3-class argmax on the same grid (0 background incl. sky, 1 foreground objects, 2 drivable road), a by-product the node does not publish; class_counts counts its pixels.

Demo

The p150 output next to the fp32 CPU reference on the same image; the bottom panel marks in white the pixels where the two class maps differ (25 of 300,978):

All nine shipped VisionPilot tutorial images, and three of the shipped comma10k images against their ground truth:

On public driving datasets (p150 outputs; the frames themselves are not in this repository). The caption of each image gives its agreement with the fp32 CPU reference (KITTI: the foreground IoU against the ground truth).

PandaSet renders: contains data from PandaSet (Scale AI and Hesai), https://pandaset.org, licensed under CC BY 4.0 and the PandaSet Dataset Terms; resized and annotated with model outputs; Scale AI and Hesai do not endorse this work. The nuScenes render is non-commercial (CC BY-NC-SA 4.0): rendered from the nuScenes dataset, © Motional AD Inc., nuScenes Terms of Use; Motional does not endorse this work. The KITTI render is non-commercial (CC BY-NC-SA 3.0): images and labels from the KITTI Vision Benchmark Suite. Sources, changes and the full attributions: media/ATTRIBUTION.md.

Demo & Performances

Warm, batch 1, 2026-10-13 (this release) next to the first release (2026-10-09). Latency: the stage bench (code/scripts/bench.py, 100 iterations per stage, 5 x 50 back-to-back replays) on the shipped sample code/tt_sceneseg/samples/highway_normal_1.png (727x414) and on a PandaSet front-camera frame (1920x1080, not shipped); the served rows come from uvicorn on the host (the app the container runs) and a loopback client, 50 requests of the shipped sample. The host is shared with other jobs, so the host stages move with its load (about ±0.8 ms between sessions); the device rows repeat to within 0.003 ms. Accuracy: the p150 output against the fp32 CPU reference of the same network (same weights, same pre- and post-processing) on the shipped samples, on 23 public gate frames and on 415 held-out public frames.

The rgb profile measures the same device times (b2b 12.577 ms) and an e2e p50 of 22.66 ms on the sample (29.13 ms on PandaSet); served, its timing_ms.total median was 41.2 ms (20 requests). Measured on the same code in the same week: a second command queue does not help (ETH 2 CQs: 13.03 ms per replay), and WORKER dispatch (11×10 grid) replays in 13.23 ms. The device profile of one replay: 380 programs, 12.49 ms of kernel time, 0.22 ms of gaps between programs (12.71 ms span; OPT_BASELINE.md, "After optimization"). All numbers in this table were measured with dispatch on the ETH cores, 1 command queue and a 12×10 compute grid on one p150, with the published numerics. Accuracy is agreement with the fp32 CPU reference of the same network; SceneSeg has no paper and no published benchmark, so the dataset metrics below are this port's own measurements. Details: OPT_REPORT.md, OPT_BASELINE.md, VERIFICATION_OPT_2026-10-13.md, VERIFICATION_2026-10-09.md.

Channel order. VisionPilot's deployed C++ pre-processing feeds B, G, R planes (with the ImageNet constants reordered to match) to a network trained on R, G, B: a channel-order bug in the deployment. The default bgr profile reproduces the deployed node; rgb feeds the training order. Both run the same device graph (the order is folded into the first conv's weights) and both are validated against the fp32 CPU reference of their own order. The training order is clearly better, most of all on two-wheelers (foreground IoU unless stated; the datasets are held out except comma10k):

The p150 column scores the served trace's class maps of all 952 public-frame runs (this release, 2026-10-13) with the same research protocols as the CPU (research/sceneseg/public_data, ss_sanity.py --cls-dir); the dataset labels are mapped to SceneSeg's three classes (vehicles, two-wheelers and people are foreground). The LiDAR rows are a sparse proxy (points 1-40 m projected into the camera), and the comma10k validation split is reconstructed from the upstream code.

Use bgr to reproduce the Autoware VisionPilot output, rgb for better masks.

No GPU comparison: no GPU was available on the host where this port was built and measured, so this card makes no GPU speed claim. The reference rows are the port's own fp32 CPU reference on the same host (a correctness baseline, not a speed target). VisionPilot ran this network as a TensorRT FP16 engine built from SceneSeg_FP32.onnx (autoseg.yaml: backend tensorrt, precision fp16); the upstream model README (Models/model_library/SceneSeg/README.md @ ca58cb50) reports 18.1 FPS FP32 / 26.7 FPS FP16 on an RTX 3060 Mobile (not like-for-like). p150 power was not measured, so no efficiency comparison is made.

Caveats

• Second release: the optimized port (first release: the baseline, 61.06 ms per replay). The device graph is one metal trace of 380 programs, kernel-bound (12.49 ms of kernels in a 12.71 ms span). The largest remaining items are the decoder tail at 320x640 (decode_layer_9 + class margins, about 1.5 ms with its DRAM slicing) and the depthwise kernel (about 1 ms); OPT_REPORT.md lists the backlog.
• Experimental code, off: code/tt_sceneseg/tt/mkt.py (knob SCENESEG_TAIL_MK, pinned to 0) is a megakernel attempt for that decoder tail. Its first form hung the chip during bring-up, and its reworked form has never run on a device. Do not enable it.
• Deployment status in Autoware: SceneSeg is not part of Autoware Universe. It was deployed by the Autoware Foundation's VisionPilot stack (formerly autoware.privately-owned-vehicles) through its generic ROS 2 models package, node scene_seg_model (TensorRT FP16; ONNX Runtime and Zenoh wrappers shared the same pre- and post-processing). Upstream removed that deployment code on 2026-06-15 (last tree 04fa3e80, which this port follows) and the PyTorch model code on 2026-07-06 (last tree ca58cb50); the current VisionPilot 1.0 ships AutoSpeed, AutoSteer and AutoDrive, not SceneSeg. This bundle is not a ROS 2 node (Python API and HTTP) and not a certified Autoware component; do not use it for safety-critical driving decisions.
• Channel order: the default bgr reproduces the deployed node, including its channel-order bug (above). rgb is the training order, offered as a load-time option; it is not what VisionPilot ran.
• Numerics: validated against the fp32 CPU reference of the same graph (ONNX Runtime on the deployed ONNX), not against VisionPilot's TensorRT FP16 engine.
• Precision policy of this release: the backbone runs its conv and linear weights as two bf16 terms (hi + lo; stacked into one conv where that is bit-identical) and keeps float32 activations, fed to each layer as two bf16 terms; its depthwise convs run in fp32 on the SFPU (a custom kernel). The decoder 3x3 convs, its sub-pixel up-blocks, the class-margin conv and SceneContext use one round-to-nearest bf16 weight term at HiFi3 with bf16 activations; this map was tuned on a separate 58-frame set and then gated on all frames. Every bias is added in fp32 (a bias fused into a ttnn conv or linear truncates its fp32 result toward zero); fp32 accumulation. One bf16 weight term in the backbone expand 1x1 convs as well passed the gates but lowered the worst held-out frame from 99.86 % to 99.60 %, and was rejected. Plain bf16 weights everywhere missed the 99.5 % agreement gate on a nuScenes frame (0.9949, before fix round 1), and bf16 backbone activations miss it on 3 held-out frames in the CPU emulation (OPT_REPORT.md, "Rejected"). The hardest held-out frame agrees on 99.85 % of pixels (99.75 % in the first release).
• Documented deviations from the node: the squash to 640x320 (bit-exact with cv::resize INTER_LINEAR) and the INTER_NEAREST resize back run on the host, the normalisation, the network and the class decision (argmax, lowest index on ties, as the node's strict > scan) on the device; the 3-class map is returned as well; the mask is a PNG (or npz) in JSON, not a ROS image message; inputs are RGB arrays or encoded files (converted to BGR8 inside). SceneSeg's Python tooling also offered a bicubic resize; it is not offered here, because the deployed node never used it. The INT8 QAT ONNX (never deployed) and SceneSegLite (a different 19-class network) are not part of this package.
• Domain: SceneSeg was trained on ACDC, MUSES, IDDAW, Mapillary Vistas and comma10K (roughly 2:1 views without the ego hood) for a front camera with 52-55° HFOV. On the public frames used here the fp32 CPU reference itself shows the weights' limits, and the p150 output agrees with it: small and distant people and two-wheelers are the weak spot (above all in the deployed bgr order), parking areas and a third of sidewalk points come out as road, trams stay background, and wider cameras (nuScenes 65°) or frames with the ego hood (raw comma10k) score lower.
• Does not scale to multiple p150 in a mesh configuration. The build uses a 12×10 compute grid of Tensix cores: the dispatch functions move from one Tensix column to the ETH cores (patches/tt-metal-eth-dispatch.patch), so this build assumes that you do not need chip-to-chip ethernet communication. The image also carries patches/tt-metal-reshape-rm-sys1419.patch (a guard against a Blackhole ROW_MAJOR reshape hang under ETH dispatch; this release was measured with it); a ttnn environment for the Python quickstart should apply both.
• dispatch="worker" (server: SCENESEG_DISPATCH=worker) is an A/B opt-in. On a p150 it gives an 11×10 grid (13.23 ms per replay instead of 12.58 ms); if ETH dispatch is not available (tt-metal without the patch), the model falls back to it with a warning. The numbers on this card do not apply to that mode.
• Batch 1, one frame per request; requests are serialised on the chip. The network input is fixed at 640x320 (the SceneContext reshape hard-codes the 10x20 stride-32 grid); any source image size is accepted, squashed on the host without keeping the aspect ratio, and the mask is resized back with nearest neighbour.
• Not an OpenAI-compatible API; GET /v1/models is a stub so the tt-model ready card does not 404.
• p150 power was not measured, so no efficiency comparison is made.

Licensing

• Weights: weights/ of this repo at commit a6829b246a05dba55c356323bbf75f7b1d148499: VisionPilot's SceneSeg_FP32.onnx (Google Drive 1l-dniunvYyFKvLD7k16Png3AsVTuMl9f, sha256 8e509094…c973) and its .pth (Drive 1vCZMdtd8ZbSyHn1LCZrbNKMK7PQvJHxj, sha256 9482f131…89fc) converted to safetensors, re-hosted unchanged with the upstream Apache-2.0 LICENSE, NOTICE and provenance.json (the upstream README covers the model weights with Apache-2.0; the card AutowareFoundation/SceneSeg holds no weights). The upstream model card lists the training data: ACDC, MUSES, IDDAW, Mapillary Vistas and comma10K (BDD100K held out); check those datasets' terms before commercial use.
• Pre- and post-processing ported from autowarefoundation/vision_pilot VisionPilot/middleware_recipes/ROS2/models and middleware_recipes/common/backends @ 04fa3e80 (Apache-2.0; removed upstream on 2026-06-15).
• Port and serving code (code/): Apache-2.0. patches/tt-metal-eth-dispatch.patch and patches/tt-metal-reshape-rm-sys1419.patch modify tt-metal (Apache-2.0).
• Sample data (only redistributable data ships; the public-dataset frames of the accuracy tables are not in this repository):
• code/tt_sceneseg/samples/*.png: the 9 images of the upstream SceneSeg tutorial (autowarefoundation/vision_pilot @ ca58cb50, Models/tutorials/assets/images/), Apache-2.0 (samples/LICENSE-vision_pilot.txt); the original photo source is not stated upstream.
• code/tt_sceneseg/samples/comma10k/: six comma10k images with their masks (commaai/comma10k @ 6c205fe), MIT, © 2020 Comma.ai, Inc. (samples/comma10k/LICENSE-comma10k.txt); five of them are SceneSeg training images.
• Demo media (media/, sources and changes in media/ATTRIBUTION.md):
• PandaSet renders: Contains data from PandaSet (Scale AI and Hesai), https://pandaset.org, licensed under CC BY 4.0 and the PandaSet Dataset Terms. Changes: resized, annotated with model outputs. Scale AI and Hesai do not endorse this work. Cite: P. Xiao et al., PandaSet: Advanced Sensor Suite Dataset for Autonomous Driving, ITSC 2021.
• nuScenes render (media/sceneseg_nuscenes_scene0103_k28_front_tt_NC.jpg), non-commercial, CC BY-NC-SA 4.0: Rendered from the nuScenes dataset, © Motional AD Inc., CC BY-NC-SA 4.0 and the nuScenes Terms of Use (https://www.nuscenes.org/terms-of-use). Non-commercial use only; adaptations under the same license. Motional does not endorse this work. Cite: H. Caesar et al., nuScenes: A Multimodal Dataset for Autonomous Driving, CVPR 2020.
• KITTI render (media/sceneseg_kitti_heldout_tt_NC.jpg), non-commercial, CC BY-NC-SA 3.0: Contains images and labels from the KITTI Vision Benchmark Suite, semantic segmentation benchmark 2015 (A. Geiger, P. Lenz, C. Stiller, R. Urtasun; H. Alhaija et al.), https://www.cvlibs.net/datasets/kitti/. Non-commercial use only; adaptations under the same license. Changes: cropped to 2:1, resized, prediction and ground truth overlaid. Cite: A. Geiger et al., CVPR 2012; H. Alhaija et al., IJCV 2018; M. Menze and A. Geiger, CVPR 2015.
• Tutorial-image renders: Apache-2.0; comma10k renders: MIT.

Provenance

These are the exact sources the container image was built from: