bobcat-flash-1.2-fp8

Technical write-up: foxl.ai/blog/bobcat-typed-decisions

Bobcat Flash 1.2 · Try it in the Space · Code on GitHub

A pre-quantized FP8 checkpoint of Bobcat Flash 1.2, the fast tier of the typed-decision model Bobcat: you send a state and questions whose answers you name, and it returns a probability for every answer you named; it never generates text. Evaluation, training data, limitations and the full license notes are on the main card, sanghwa-na/bobcat-flash-1.2.

This repository, sanghwa-na/bobcat-flash-1.2-fp8, is the BF16 weights of sanghwa-na/bobcat-flash-1.2 quantized to FP8 (27.2 GB, FP8_DYNAMIC, every expert included): float8_e4m3fn weights with one scale per output channel, activations quantized per token at run time, in the compressed-tensors format that vLLM loads as it is. It is the checkpoint the Bobcat Flash Space serves.

Serve

curl -LsSf https://astral.sh/uv/install.sh | sh && export PATH="$HOME/.local/bin:$PATH"   # uv
git clone https://github.com/foxl-ai/bobcat && cd bobcat
export UV_PYTHON_PREFERENCE=only-managed
uv venv --python 3.12 .venv-serve
uv pip install --no-config --python .venv-serve/bin/python vllm==0.30.0 fastapi uvicorn scipy jinja2 \
  "tokenizers>=0.21" huggingface_hub typesafe-sdk==0.7.1
export PYTHONPATH=$PWD/src PY=.venv-serve/bin/python

$PY -c "from huggingface_hub import snapshot_download as s; s('sanghwa-na/bobcat-flash-1.2-fp8', local_dir='bobcat-flash-1.2-fp8')"
VLLM_USE_FLASHINFER_SAMPLER=0 $PY -m bobcat.flash_server --engine vllm --model bobcat-flash-1.2-fp8 \
  --compiler-model bobcat-flash-1.2-fp8/compiler --quantization none \
  --temperature choice=0.594604,noul=0.353553,score=0.529732 \
  --name bobcat-flash-1.2 --release-date 2026-10-07 --max-num-seqs 64 --max-model-len 98368 \
  --schedule all --engine-arg max_num_batched_tokens=16384 --engine-arg attention_backend=TRITON_ATTN \
  --host 127.0.0.1 --port 8000 --local

Then call it with the official typesafe-sdk as on the main card (TYPESAFE_BASE_URL=http://127.0.0.1:8000, TYPESAFE_DEFAULT_MODEL=bobcat-flash-1.2).

• --quantization none: vLLM reads the FP8 scheme from config.json (compressed-tensors).
• --temperature and --max-model-len 98368 are as on the main card: one calibration temperature per question type, and states up to about 96K tokens read in full; longer inputs are refused with HTTP 422 and never truncated.
• --compiler-model .../compiler points the Bobcat compiler at the base model's pinned tokenizer, template and configuration; run the server from the repository root, where the identifier list lives, or pass --identifiers bobcat-flash-1.2-fp8/bobcat-identifiers.json.

Precision

FP8 minus BF16: -0.19 points [-0.54, +0.15]; 41 answers changed, mostly near-ties. Served by bobcat.flash_server over localhost HTTP with the command above, TypeSafe's 20 workflow cases agreed with the reference on 88.4% of 329 questions (no failed request), at a median 0.33 s per case; the BF16 weights of the main repository, served in FP8 at load (--quantization fp8), agreed on 90.3% and 90.6% in two runs. Where TypeSafe-style workflows matter, serve the main repository. One decision (512 tokens, 8 candidates) takes 24.6 ms at p50 in the vLLM engine.

Provenance

• Weights: the BF16 checkpoint of sanghwa-na/bobcat-flash-1.2 (two Bobcat LoRA adapters merged into Gemma 4 26B-A4B-it at revision 4d7ae4984b7db7de8f8457170b3f1a419ee76d52, combined as 0.6 / 0.4), checked against its SHA256SUMS.json.
• Quantization: scripts/fp8_quantize.py with llm-compressor 0.14.0 (FP8_DYNAMIC, no calibration data), as for the earlier Flash FP8 checkpoint: the 30 fused expert blocks are linearized into one Linear per expert; lm_head, the embeddings, the 30 routers and the vision tower stay BF16.
• Hashes: model-00001-of-00002.safetensors sha256 8d041fdb…7d99, model-00002-of-00002.safetensors 8a495d45…0a97, as recorded in the release manifest (bobcat-release-manifest.json, serving_artifacts.fp8); every file is listed in SHA256SUMS.json.
• Labels computed by code; teacher distributions from GLM-5.3-Flash. No output of Jev was used; see the main card.

License and attribution

Apache-2.0. This checkpoint is a derivative of Gemma 4 26B-A4B-it by Google DeepMind (Apache-2.0); the license text is included as LICENSE, and NOTICE lists what was changed. Training data keep their own licenses (main card and THIRD_PARTY.md in github.com/foxl-ai/bobcat). Bobcat is an independent project, not affiliated with or endorsed by Google, the Qwen team, Z.ai, TypeSafe AI or any other company named here. Provided as is, without warranty.

Citation

@misc{bobcatflash12,
  title        = {Bobcat Flash 1.2},
  author       = {{The Bobcat Authors}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/sanghwa-na/bobcat-flash-1.2}}
}