Q-TensorFormer

Q-TensorFormer: Information-Value-Driven Adaptive Tensor-Network Transformer for Constrained & Edge Inference

Official Technical Documentation & Research Portal: https://premchan369.github.io/Q-TensorFormer/
Jump to topics: Architecture | Methodology | Benchmarks | Edge Inference | Training | API Reference | Related Research | Limitations | Contributing

Executive Summary for Technical Leaders & Hiring Managers

Q-TensorFormer is an open-source, publication-grade research architecture and inference testbed designed by Premchand Yadav. It investigates a fundamental question in efficient deep learning: Can a Transformer dynamically allocate its computational capacity, tensor rank, attention complexity, and KV-cache precision according to the marginal informational value of each token under explicit hardware constraints?

What This Repository Demonstrates

┌─────────────────────────────────────────────────────────────────────────────────────────────────┐
│                                   CORE COMPETENCIES AT A GLANCE                                 │
├───────────────────────────────┬─────────────────────────────────┬───────────────────────────────┤
│    DEEP ML SYSTEMS DESIGN     │  APPLIED MATHEMATICS & CONTROL  │    PRODUCTION ENGINEERING     │
├───────────────────────────────┼─────────────────────────────────┼───────────────────────────────┤
│ • Custom Tensor-Train autograd│ • Zero-SVD dynamic rank slicing │ • Turnkey Hugging Face Hub    │
│   layers with batched einsum  │   r ∈ {1, 2, 4, 8, 16}          │   AutoModel / AutoConfig      │
│ • Zero-copy Grouped-Query     │ • Lagrangian dual-subgradient   │ • Lossless UTF-8 256-subword  │
│   Attention (GQA) with RoPE   │   resource allocation           │   deterministic tokenizer     │
│ • Bitwise-packed INT4 & INT8  │ • Closed-loop PID controller &  │ • GGUF v3 & ONNX export       │
│   quantized KV cache          │   hysteresis deadband (τ = 0.15)│ • 101 unit tests passed (100%)│
│ • Streaming Attention-Sink    │ • 8D normalized token info      │ • Strict serialization audits │
│   preservation (no collapse)  │   state vector (z_t)            │   (strict=True enforced)      │
└───────────────────────────────┴─────────────────────────────────┴───────────────────────────────┘

Key Empirically Measured Results (RUN-20261013-QTF-HARDENING & RUN-20261019-FRONTIER)

Table of Contents

1. Project Overview & Core Research Question
2. Layman's Explanation: How Q-TensorFormer Works End-to-End
3. Edge AI & Cold-Start Motivation
4. Complete Conceptual Architecture Loop
5. Feature Implementation & Verification Status Table
6. Canonical Model & Configuration Families
7. Installation & Environment Setup
8. Quickstart Inference Examples
9. Training, Benchmarking & Evaluation Commands
10. Checkpoint Integrity Status (strict=True Audit)
11. Empirical Benchmark Summary & Evidence Classification
12. Cold-Start & Model-Loading Benchmark Table
13. Steady-State Inference & Tensor-Train Compression Tables
14. Adaptive KV-Cache Scaling & Attention-Sink Evaluation Table
15. Controller Stability, Hysteresis & Ablation Summary
16. Optional Quantum Pathway Status & Overhead
17. Honest Latency, Memory & Energy Interpretation
18. Related Work & Novelty Positioning
19. Known Limitations & Failure Modes
20. Reproducibility & One-Command Verification Instructions

1. Project Overview & Core Research Question

Q-TensorFormer is an experimental PyTorch and Hugging Face research architecture that investigates closed-loop, information-value-driven resource allocation for Transformer inference under explicit memory, latency, bandwidth, and quality constraints.

Central Research Question

Can a Transformer dynamically allocate computational resources—Tensor-Train rank, layer depth, attention pathway, and KV-cache precision/retention—according to their marginal informational value rather than applying a static computational strategy to every token?

Rather than claiming that dynamic tensor networks automatically accelerate every workload on every processor, Q-TensorFormer is built as a reproducible experimental testbed to measure and determine which benefits are real, under which hardware conditions, and at what runtime cost.

2. Layman's Explanation: How Q-TensorFormer Works End-to-End

Imagine how a human reads a book: 1. Standard Dense Transformers read every word with maximum effort: Whether the next word is an obvious filler word ("the", "and") or a complex medical diagnosis, a standard LLM multiplies every token through 100% of its giant weight matrices and stores every past token in full high-precision memory. 2. Q-TensorFormer acts like an adaptive reader with a strict budget: - Step 1 — Sense Token Complexity & Device Pressure (Information State): As each token enters a layer, a lightweight probe measures how uncertain or ambiguous the token is alongside how full the device's memory and latency budgets are. - Step 2 — Estimate Marginal Value (Resource Allocator): A controller estimates whether spending extra matrix rank or memory precision on this token will meaningfully improve prediction quality relative to its cost. - Step 3 — Slice Nested Tensor Cores (Zero-SVD TT-FFN): Instead of storing one monolithic weight matrix, Q-TensorFormer stores weights as a chain of compact Tensor-Train (TT) cores. For easy tokens or tight memory budgets, it slices only the leading rank channels ($r \in {1, 2, 4, 8, 16}$) on the fly without running an expensive Singular Value Decomposition (SVD). - Step 4 — Protect Key Context & Compress Old Memory (Adaptive KV Cache): It keeps the first 4 "anchor" tokens (Attention Sinks) and the most informative recent tokens while compressing older KV states into 8-bit (INT8) or packed 4-bit (INT4) integers. - Step 5 — Regulate Stability (PID + Hysteresis): A Proportional-Integral-Derivative (PID) loop adjusts resource shadow prices to track target budgets, while a hysteresis deadband ($\tau = 0.15$) prevents rapid rank oscillation ("chattering") between consecutive tokens.

3. Edge AI & Cold-Start Motivation

Deploying language and multimodal models on edge hardware—point-of-care medical devices, mobile phones, autonomous drones, robotics controllers, and embedded computers—faces two distinct bottlenecks:

1. Cold-Start & Model-Loading Latency: - Developer Motivation: This research direction was directly motivated by the repository author's practical experience when deploying a 1.5B-parameter medical model (MedGemma 1.5B) on a VRAM-constrained system, where staging and loading the model into memory required approximately 200 seconds before the first token could be generated. (Note: This ~200s observation is cited strictly as the author's motivating developer experience, not as a benchmark executed inside this repository.) - When a model checkpoint exceeds available fast RAM/VRAM, disk I/O and PCIe paging dominate Time-To-First-Token (TTFT). Reducing static checkpoint byte volume via Tensor-Train factorization and FP16/INT8 serialization directly shrinks the payload that must be read from flash storage.
2. Long-Context KV-Cache Memory Growth: - During autoregressive decoding, the Key-Value (KV) cache grows linearly with sequence length $L$ and can exhaust edge DRAM even when static weights fit. Combining attention-sink protection with INT8/INT4 quantization and utility-guided eviction bounds peak KV memory to a fixed byte ceiling.

4. Complete Conceptual Architecture Loop

flowchart LR
    A["Input Token IDs"] --> B["8D Token Information State z_t\n(S_t, H_t, U_t, A_t, R_t, L_t, M_t, B_t)"]
    B --> C["Marginal-Value Estimator\n& Lagrangian Allocator"]
    C --> D["Hysteresis Deadband\n(tau = 0.15)"]
    D --> E["Hardware-Aware Execution\n- Nested TT-FFN (r in 1..16)\n- GQA + RoPE / Optional QKSAM\n- Adaptive KV Cache (FP16/INT8/INT4 + Sinks)"]
    E --> F["Output Logits & Telemetry"]
    F --> G["PID Dual-Subgradient Feedback\n(Updates lambda_L, lambda_M, lambda_E, lambda_B)"]
    G --> C

Mathematical Formulation of Core Subsystems

1. 8D Normalized Token Information State (src/information_state.py): $$z_t = \sigma!\left(\frac{\tilde{z}t - \mu{\text{EMA}}}{\sqrt{\sigma^2_{\text{EMA}} + \epsilon}}\right) \odot m_{\text{ablate}} \in [0, 1]^8, \quad \tilde{z}_t = [S_t, H_t, U_t, A_t, R_t, L_t, M_t, B_t]$$ where $S_t$ is surprise/loss proxy, $H_t$ is predictive entropy, $U_t$ is attention dispersion, $A_t$ is cumulative attention, $R_t$ is current rank ratio, and $L_t, M_t, B_t$ measure latency, memory, and bandwidth pressure.

2. Nested Tensor-Train Linear Contraction (src/tensor_layers.py): $$W_{(i_1,\dots,i_d),(o_1,\dots,o_d)}(r) = \sum_{\alpha_1=1}^{r} \cdots \sum_{\alpha_{d-1}=1}^{r} \mathcal{G}^{(1)}{1, o_1, i_1, \alpha_1} \mathcal{G}^{(2)}{\alpha_1, o_2, i_2, \alpha_2} \cdots \mathcal{G}^{(d)}{\alpha{d-1}, o_d, i_d, 1}$$ Zero-SVD Rank Slicing: Cores $\mathcal{G}^{(k)}$ are trained with nested rank ordering such that slicing $\mathcal{G}^{(k)}[:r', :, :, :r']$ for $r' < r$ immediately scales down capacity with zero decomposition overhead.

3. Lagrangian Marginal-Utility Selection with Hysteresis (src/resource_allocator.py): $$U(a \mid z_t, \boldsymbol{\lambda}) = \widehat{\Delta Q}(a \mid z_t) - \sum_{k \in {L, M, E, B}} \lambda_k \widehat{\Delta C}k(a \mid z_t)$$ $$a_t = \begin{cases} a^ & \text{if } U(a^) - U(a{t-1}) \ge \tau \ a_{t-1} & \text{otherwise} \end{cases}$$ where $\tau = 0.15$ guarantees that rank transitions occur only when marginal gains exceed the hysteresis threshold.

4. Dual-Subgradient PID Shadow-Price Update: $$\lambda_{k, t+1} = \max!\left(0, \lambda_{k, t} + K_p e_{k, t} + K_i \sum_{s=1}^t e_{k, s} + K_d (e_{k, t} - e_{k, t-1})\right)$$ where $e_{k, t} = C_{k, t} - C_{k, \text{budget}}$ drives shadow prices up when budgets are exceeded and down when capacity is available.

5. Feature Implementation & Verification Status Table

Every advertised capability is audited in reports/CLAIM_TO_CODE_MAP.md and classified below:

6. Canonical Model & Configuration Families

As documented in reports/ARCHITECTURE_FAMILY_MAP.md and reports/CANONICAL_MODEL_DECISION.md, the repository provides two canonical active APIs alongside two preserved historical prototype scripts:

7. Installation & Environment Setup

git clone https://huggingface.co/Premchan369/Q-TensorFormer
cd Q-TensorFormer
pip install -r requirements.txt

Verified Test Environment (results/metadata/run_metadata.json): - OS: Windows 10 (10.0.26200-SP0), Intel 20-thread CPU - Python: 3.11.9 | PyTorch: 2.10.0+cpu | Transformers: 4.41.2 | NumPy: 2.4.6 | Pytest: 9.1.1

8. Quickstart Inference Examples

8.1 Turnkey Hugging Face Hub Loading (AutoModelForCausalLM)

import torch
from transformers import AutoConfig, AutoModelForCausalLM
from src.tokenizer import QTensorFormerTokenizer

# 1. Load custom deterministic tokenizer
tokenizer = QTensorFormerTokenizer.load("checkpoints/tokenizer")

# 2. Load trained model from checkpoint directory or Hugging Face Hub
model = AutoModelForCausalLM.from_pretrained(
    "checkpoints/hf_canonical_trained",
    trust_remote_code=True,
)
model.eval()

# 3. Generate autoregressively
prompt = "Tensor networks reduce memory footprint"
input_ids = torch.tensor([tokenizer.encode(prompt)])

with torch.no_grad():
    output_ids = model.generate(
        input_ids,
        max_new_tokens=32,
        do_sample=False,
        use_cache=True,
    )

print("Generated text:", tokenizer.decode(output_ids[0].tolist()))

8.2 Canonical Research Model with 8D Telemetry (src.models.QTensorFormer)

import torch
from src.config import ModelConfig
from src.models import QTensorFormer
from src.kv_cache import AdaptiveKVCache, KVPrecision

cfg = ModelConfig(
    d_model=128,
    n_heads=4,
    n_layers=2,
    ff_multiplier=4,
    max_seq_len=128,
    vocab_size=10000,
    tt_rank=8,
    tt_min_rank=2,
    use_quantum=False,
)
model = QTensorFormer(cfg, preset="QTF_EDGE")
model.eval()

input_ids = torch.randint(0, 10000, (1, 16))
kv_caches = [AdaptiveKVCache(max_capacity=128, default_precision=KVPrecision.INT8) for _ in range(cfg.n_layers)]

with torch.no_grad():
    logits, layer_stats = model(input_ids, kv_caches=kv_caches, return_stats=True)

print("Logits shape:", logits.shape)
print("Layer 0 chosen rank & KV precision:", layer_stats[0]["chosen_rank"], layer_stats[0]["chosen_kv_precision"])

8.3 Lossless Deterministic UTF-8 Tokenizer (src.tokenizer.QTensorFormerTokenizer)

from src.tokenizer import QTensorFormerTokenizer

tok = QTensorFormerTokenizer()
text = "Q-TensorFormer: dynamic rank slicing r in {1, 2, 4, 8, 16}"
tokens = tok.encode(text)
recovered = tok.decode(tokens)
assert text == recovered, "Lossless roundtrip guaranteed across all UTF-8 strings"
print(f"Token count: {len(tokens)} | Roundtrip verified: True")

9. Training, Benchmarking & Evaluation Commands

# 1. Run deterministic end-to-end smoke test
python scripts/smoke_test.py

# 2. Run full unit & scientific guardrail test suite (94 passed, 0 failed)
python -m pytest -v

# 3. Train canonical model and evaluate perplexity reduction
python scripts/train_and_evaluate.py

# 4. Run full reproducible scientific benchmark suite (generates results/raw/*.json)
python scripts/run_scientific_benchmark.py

# 5. Run automated scientific integrity & evidence-label checker
python scripts/scientific_integrity_checker.py

# 6. Run automated SEO & discoverability quality gate
python scripts/seo_audit.py

Training Verification Results (results/raw/training_and_quality_validation.json)

Trained on data/qtensorformer_corpus.txt with identical batching, optimizer, and learning rate schedule:

10. Checkpoint Integrity Status (strict=True Audit)

Full forensic details: reports/CHECKPOINT_INTEGRITY.md and results/raw/checkpoint_integrity.json.

1. Legacy File qtensorformer_small.pt (FAIL_STRICT_LOAD): - qtensorformer_small.pt (5,432,305 bytes, 113 tensors, 1,347,747 parameters) was committed on May 6, 2026 from an external TensorLy container (/app/qtensorformer) that used 6-core TensorLy keys (blocks.i.ffn.layer.weight.factors.factor_0..5). - Tested under model.load_state_dict(sd, strict=True), it has 0/113 (0.0%) key overlap with both src.models.QTensorFormer (185 keys) and modeling_qtensorformer.QTensorFormerForCausalLM (52 keys). - Scientific Integrity Policy: We never use strict=False to silently ignore mismatched keys, nor do we alter checkpoint tensors to fake compatibility. qtensorformer_small.pt is quarantined as a historical artifact, and qtensorformer_to_gguf.py enforces strict=True when loading checkpoints.
2. Canonical Models (PASS strict=True Round-Trip): - Both src.models.QTensorFormer (185/185 keys) and modeling_qtensorformer.QTensorFormerForCausalLM (52/52 keys) pass strict=True save/reload round-trip verification with < 1e-6 max logit difference.

11. Empirical Benchmark Summary & Evidence Classification

All benchmark tables below come from run RUN-20261013-QTF-HARDENING executed via scripts/run_scientific_benchmark.py. Every metric is explicitly classified as: - MEASURED — Directly timed or byte-counted on the local 20-thread Intel CPU host (PyTorch 2.10.0+cpu). - SIMULATED — Computed via deterministic mathematical simulation (e.g., counterfactual policy rollouts or quantum statevector fidelity math). - ESTIMATED — Calculated via analytical FLOP/byte hardware cost models (LEVEL 1/2/3). - BLOCKED / UNVERIFIED — Cannot be measured on the current host (e.g., LEVEL 4 NVML/RAPL power sensors, physical CUDA GPU kernels, or physical QPU execution).

12. Cold-Start & Model-Loading Benchmark Table

Source: results/raw/cold_start_and_edge.json & reports/COLD_START_EVALUATION.md (d_model=256, n_layers=4, n_heads=8, vocab_size=10000, 5 repeats).

13. Steady-State Inference & Tensor-Train Compression Tables

13.1 Model-Level Prefill (TTFT) & Autoregressive Decode (TPOT)

Source: results/raw/cold_start_and_edge.json & reports/EDGE_INFERENCE_EVALUATION.md.

13.2 Layer-Level Tensor-Train Rank Sweep (512 -> 2048 Projection)

Source: results/raw/tt_compression_audit.json (40 timed runs per rank; core slice overhead 98.29 μs).

13.3 Frontier Research Empirical Extensions (2026 Hardening)

Source: results/raw/frontier_research_benchmarks.json & reports/FRONTIER_RESEARCH_BENCHMARKS.md (RUN-20261019-FRONTIER-EXTENSIONS).

14. Adaptive KV-Cache Scaling & Attention-Sink Evaluation Table

Source: results/raw/kv_cache_audit.json ($B=1, H=4, d_{\text{head}}=32$, 4 sink tokens protected, eviction capacity 128).

15. Controller Stability, Hysteresis & Ablation Summary

Source: results/raw/controller_audit.json & reports/BENCHMARK_RESULTS.md.

1. Hysteresis Anti-Chattering (MEASURED over 64 oscillating token steps): - Without hysteresis ($\tau = 0.00$), InformationValueAllocator switches TT rank on 25.40% of token steps (mean_rank = 3.484). - With hysteresis ($\tau = 0.15$), rank-switching churn drops to 0.00% (mean_rank = 4.000), compared to 11.11% for a simple entropy threshold and 63.49% for random rank selection.
2. 8D Information State LOFO Ablation (MEASURED): - Zeroing each individual feature in $z_t$ via set_ablation(feature, False) confirmed non-zero L2 sensitivity across all 8 signals, led by uncertainty ($U_t$, L2 impact 4.3095), predictive_entropy ($H_t$, 3.5497), and hardware pressure signals ($L_t, M_t, B_t$, 1.8681 each).
3. Counterfactual Policy Evaluation (SIMULATED across 100 token states): - Joint learned marginal-value allocation achieves the lowest regret vs the retrospective oracle (0.1135 regret, 0.4495 avg reward, 0.655 MB avg memory) compared to fixed-rank (0.1422 regret, 1.120 MB), heuristic (0.1545 regret), independent subsystem control (0.1854 regret), and entropy-only (0.1962 regret).

16. Optional Quantum Pathway Status & Overhead

Source: results/raw/quantum_pathway_audit.json.

• Status: No physical QPU was used (real_qpu_hardware_executed: false) and pennylane is optional (not installed in the default CPU environment). Even with ClassicalSurrogateUnitary, forcing QKSAM on adds +6.75% CPU latency overhead. Consequently, edge presets (QTF_EDGE, QTF_CLASSICAL_ONLY) disable quantum execution by default.

17. Honest Latency, Memory & Energy Interpretation

Detailed analysis: reports/EDGE_INFERENCE_EVALUATION.md and reports/ENERGY_MEASUREMENT_METHODOLOGY.md.

1. Parameter & Memory Compression vs CPU Wall-Clock Speed (MEASURED): - TTLinear achieves massive parameter compression (3.95x at $r=16$ up to 56.96x at $r=1$) and reduces active latency by 69.1% when slicing from $r=16$ (13.49 ms) down to $r=2$ (4.17 ms). - However, on a CPU where dense weights already fit in cache, a single MKL/BLAS nn.Linear(512, 2048) call (0.20 ms decode, 3.59 ms prefill) is faster in wall-clock time than 4 sequential einsum/bmm core contractions. - When TT Compression Wins: TT compression improves deployment latency when (a) a dense model exceeds edge SRAM/VRAM and incurs disk/PCIe paging stalls during cold start or decoding, or (b) fused GPU kernels eliminate intermediate core materialization.
2. Energy Measurement Honesty (ESTIMATED vs UNVERIFIED): - Because this CPU host lacks hardware power sensors (NVML / privileged RAPL / inline wattmeter), direct sensor energy (LEVEL 4) is marked UNVERIFIED / N/A. - All energy metrics in src/hardware_cost_model.py (LEVEL 1 analytical, LEVEL 2 memory-traffic, and LEVEL 3 measured runtime $\times$ nominal TDP) are strictly labeled ESTIMATED.

18. Related Work & Novelty Positioning

Full literature review: reports/RELATED_WORK_AND_NOVELTY.md.

Q-TensorFormer builds upon established prior work without claiming priority over individual primitives: - Tensor-Train Networks: Oseledets (2011) (Tensor-Train Decomposition); Novikov et al. (2015) (Tensorizing Neural Networks). - Adaptive Computation & Early Exit: Graves (2016) (ACT); Schuster et al. (2022) (CALM); Raposo et al. (2024) (Mixture-of-Depths). - Attention Sinks & KV Quantization: Xiao et al. (2024) (StreamingLLM); Zhang et al. (2023) (H2O); Liu et al. (2024) (KIVI). - Quantum Feature Maps & Entanglement Metrics: Schuld & Killoran (2019); Meyer & Wallach (2002); Sim et al. (2019).

What Q-TensorFormer Contributes: An integrated open-source architecture and benchmark suite that couples an 8D token/hardware state vector ($z_t$), zero-SVD nested TT rank slicing, sink-preserving multi-precision KV caching, and hysteresis-stabilized PID control into a single closed-loop inference framework.

19. Known Limitations & Failure Modes

Full list: reports/KNOWN_LIMITATIONS.md.

1. Unfused CPU Contraction Latency: Multi-core einsum contraction on CPU has higher wall-clock latency than dense MKL GEMM for small/medium matrices that fit in cache.
2. CPU float16 Emulation Penalty: While float16 checkpoints cut disk size by 49.3%, running float16 einsum contractions on x86 CPUs without native FP16 tensor cores incurs software emulation overhead (TTFT = 335.66 ms vs 25.82 ms in float32).
3. Legacy Checkpoint Incompatibility (qtensorformer_small.pt): Cannot be loaded with strict=True (0/113 key overlap) because it was exported from an external TensorLy container. It is preserved for provenance.
4. Canonical Trained Checkpoint Scope: The verified trained canonical checkpoint (checkpoints/qtensorformer_canonical_trained.pt and checkpoints/hf_canonical_trained/) was trained on a compact technical educational corpus (data/qtensorformer_corpus.txt), reducing perplexity by 87.2% (from 352.35 to 45.09). Scale pretraining is recommended for open-domain generation.
5. Hardware Blockers on Current Host: CUDA/Triton GPU execution, PennyLane statevector simulation, and LEVEL 4 hardware power sensor readings are BLOCKED / UNVERIFIED on the current CPU-only test environment.

20. Reproducibility & One-Command Verification Instructions

Full checklist: reports/REPRODUCIBILITY_CHECKLIST.md.

To independently reproduce the entire training, benchmark, and verification suite:

# 1. Run deterministic unit test suite (101 passed, 0 failed)
pytest -v

# 2. Run fast deterministic smoke test
python scripts/smoke_test.py

# 3. Train canonical model and evaluate perplexity reduction
python scripts/train_and_evaluate.py

# 4. Execute full scientific benchmark suite
python scripts/run_scientific_benchmark.py

# 5. Execute 6 frontier research empirical benchmarks
python scripts/run_frontier_benchmarks.py

# 6. Run automated scientific integrity checker
python scripts/scientific_integrity_checker.py

# 7. Run automated SEO & discoverability quality gate
python scripts/seo_audit.py

@software{yadav2026qtensorformer,
  author       = {Premchand Yadav},
  title        = {Q-TensorFormer: Information-Value-Driven Adaptive Tensor-Network Transformer for Constrained and Edge Inference},
  year         = {2026},
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/Premchan369/Q-TensorFormer}
}