snif-qwen-deepseek

SNIF: StateGate Neural Inference Fabric (Qwen2.5-0.5B + DeepSeek-R1-1.5B)

SNIF (StateGate Neural Inference Fabric) is an open-source composite causal language model that dynamically connects Qwen2.5-0.5B (Edge Prefill and Fast Decode) and DeepSeek-R1-1.5B (Cognitive Reasoning) via a continuous 2048-dimensional latent interconnect.

Author: CS Hari Krishna ([email protected])
Repository: csharikrishna/stategate
License: MIT

Architecture Highlights

• Continuous Latent Deliberation: Zero intermediate text tokens transferred across models during routing.
• Asymmetric Compute Efficiency: Confident queries complete via lightweight edge prefill; complex queries escalate dynamically to deep reasoning.
• Low Hardware Footprint: Active memory footprint is ~1,980 MiB VRAM, enabling execution on consumer GPUs and free Google Colab environments.
• Concise Decode: Bounded conclusion decoding avoids repetitive multi-paragraph derivations, delivering up to 3.5x faster decode latency.

Quickstart

1. Installation

pip install torch transformers safetensors accelerate sentencepiece tiktoken

2. Inference

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "csharikrishna/snif-qwen-deepseek"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if torch.cuda.is_available() else torch.float32

# Load tokenizer and composite model
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    torch_dtype=dtype,
    device_map="auto" if torch.cuda.is_available() else None,
)
if device == "cpu":
    model = model.to("cpu")
model.eval()

# 1. Factual Query (Fast exit locally via Qwen)
prompt1 = "What is the boiling point of water at sea level?"
inputs1 = tokenizer(prompt1, return_tensors="pt").to(model.device)
out1 = model.generate(**inputs1, max_new_tokens=40, do_sample=False)
print("Answer 1:", tokenizer.decode(out1[0][inputs1.input_ids.shape[-1]:], skip_special_tokens=True).strip())
print("Telemetry 1:", model.last_routing_info)

# 2. Logic Query (Escalates via 2048D NeuralBus to DeepSeek-R1 with Dynamic CHOP)
prompt2 = "If all bloops are razzies and some razzies are snarks, are all bloops snarks? State conclusion concisely."
inputs2 = tokenizer(prompt2, return_tensors="pt").to(model.device)
out2 = model.generate(**inputs2, max_new_tokens=40, do_sample=False)
print("Answer 2:", tokenizer.decode(out2[0][inputs2.input_ids.shape[-1]:], skip_special_tokens=True).strip())
print("Telemetry 2:", model.last_routing_info)
# Example Telemetry:
# {
#   'route_taken': 'ESCALATE_CHOP',
#   'p_uncertainty': 0.4888,
#   'reasoning_layers_executed': 13,
#   'reasoning_layers_saved': 15,  # 53.6% compute reduction!
#   'chop_stability': 0.8769,
#   'intermediate_text_tokens': 0,
#   'elapsed_ms': 3058.4
# }

System Architecture

• Small Model (Edge): Qwen/Qwen2.5-0.5B-Instruct (896D, 24 layers, 494M params)
• Big Model (Reasoning): deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B (1536D, 28 layers, 1.78B params)
• Latent Interconnect: Universal 2048D Lingua Franca Hub
• Routing Policy: Pre-generation latent routing evaluating intermediate activations at Layer 9.

Citation

@misc{stategate2026,
  author = {CS Hari Krishna},
  title = {StateGate: Pre-Generation Latent Meta-Routing for Cost-Optimal Language Model Inference},
  year = {2026},
  publisher = {GitHub},
  howpublished = {\url{https://github.com/csharikrishna/stategate}}
}

Note: Research paper and extended theoretical evaluation in preparation.

License and Trademark Notice

• Software and Model Weights License: MIT License
• Copyright: Copyright (c) 2026 CS Hari Krishna. All rights reserved.
• Trademark Notice: The names "StateGate", "SNIF", and "NeuralBus" are project and research identifiers belonging to CS Hari Krishna. This open-source license grants permission to use the software and weights, but does not grant rights to use these names to endorse or promote derivative products without attribution.