knivesysl-typed-2b

local, typed decisions from qwen3.5-2b. one shared state is prefetched once, each question is isolated, every allowed answer is scored as a complete token sequence, and python returns validated choice, score, and noul results.

this is an inference system, not rlcd training and not a clone of typesafe jev. it never calls typesafe. the published qwen checkpoint is unchanged; fp8 changes execution precision only. probabilities are normalized support over the candidates you provide, not calibrated correctness probabilities.

this repository republishes the unchanged Qwen/Qwen3.5-2B weights at pinned upstream revision 15852e8c16360a2fea060d615a32b45270f8a8fc together with the knivesysl-typed model card and retained benchmark evidence. it is not a fine-tune. the fp8 and compiled modes are applied at runtime and are not stored in these weights.

• source and runnable service: github.com/SRSWTI/knivesysl-typed
• upstream model: Qwen/Qwen3.5-2B
• full benchmark summary: benchmarks/precision_final/summary.json

demo and implementation media

how it works

flowchart LR
  A[structured state] --> B[shared qwen prefix]
  B --> C1[isolated question 1]
  B --> C2[isolated question 2]
  B --> C3[isolated question n]
  C1 --> D[complete candidate sequences]
  C2 --> D
  C3 --> D
  D --> E[sum token log likelihoods]
  E --> F[softmax over allowed answers]
  F --> G[typed choice score and noul answers]

unlike ordinary autoregressive json generation, the model does not write a response token by token. it scores only the values supplied by the caller. complete-sequence scoring handles multi-token values and candidates that begin with the same token. each question gets its own prompt, so one answer cannot silently become context for another.

qwen3.5 uses hybrid attention, recurrent, and convolution state. knivesysl-typed copies the full hybrid cache for each candidate batch; copying attention kv alone is incorrect for this architecture.

sequenceDiagram
  participant client
  participant api
  participant qwen as qwen3.5-2b
  client->>api: state plus typed questions
  api->>qwen: prefill shared state once
  loop each isolated question
    api->>qwen: question and allowed values
    qwen-->>api: complete-sequence log likelihoods
  end
  api-->>client: validated answers probabilities usage metadata

models

the live api intentionally exposes one model id, knivesysl-typed-2b, backed by pinned qwen3.5-2b revision 15852e8c16360a2fea060d615a32b45270f8a8fc. bf16, fp16, fp8, fp8_row_compiled, and int8 are runtime execution profiles of that unchanged checkpoint, not separately trained or serialized model weights.

preferred runtime

recommended: cuda fp8_row_compiled. it uses torchao e4m3fn fp8 activations and per-row fp8 weights for 150 large backbone linear layers, then compiles those linears. embeddings, the lm head, and small recurrent gates remain at base precision. zero activation rows are clamped before scaling to avoid nan values created by padded hybrid-attention rows.

uv sync --extra fp8
KNIVESYSL_BACKEND=sdpa \
KNIVESYSL_PRECISION=fp8_row_compiled \
uv run python -m knivesysl_typed --host 127.0.0.1 --port 8765

use --host 0.0.0.0 only when remote clients need access. set KNIVESYSL_API_KEY before non-loopback binding; clients send it as Authorization: Bearer ....

supported execution profiles:

sdpa is the portable backend. cute_sm120 is an inference-only flashattention-4 cute backend for nvidia sm120 and requires cuda with a 16-bit base dtype. intel accelerated runs use fla with native torch convolution; intel float32 uses the torch reference recurrence. the implementation uses one accelerator per process; it does not shard across two intel cards.

measured rtx 5090 results

checkpoint: Qwen/Qwen3.5-2B, revision 15852e8c16360a2fea060d615a32b45270f8a8fc. workload: 12 hand-written diagnostic states, 36 labeled fields, 6 unlabeled presets, 2 warmups, and 5 measured repeats. runs were serialized on one rtx 5090. repeats are latency samples, not independent quality samples.

fp16 was the fastest measured profile. we prefer compiled per-row fp8 because it retained the best observed diagnostic accuracy while reducing peak allocated gpu memory by about 31% versus fp16. it was about 1.48x slower than fp16 on this workload. the one-field fp8 difference on this small diagnostic set is not evidence that quantization generally improves model accuracy.

xychart-beta
  title "rtx 5090 mean server latency — lower is better"
  x-axis [bf16, fp16, fp8_tensor, fp8_tensor_compiled, fp8_row, fp8_row_compiled]
  y-axis "milliseconds" 0 --> 85
  bar [34.74, 32.82, 78.67, 51.71, 78.72, 48.46]

xychart-beta
  title "rtx 5090 peak allocated gpu memory — lower is better"
  x-axis [bf16, fp16, fp8_tensor, fp8_tensor_compiled, fp8_row, fp8_row_compiled]
  y-axis "gib" 0 --> 4.5
  bar [4.07, 4.07, 2.82, 2.80, 2.82, 2.80]

full retained measurements and limitations are in benchmarks/precision_final/summary.json. measurements were taken with a desktop and an existing idle demo running, so they are not isolated laboratory results. no int4 or nvidia int8 result was recorded.

api

start the service, then open:

• playground: http://127.0.0.1:8765/
• illustrated implementation: http://127.0.0.1:8765/?view=how-it-works
• openapi ui: http://127.0.0.1:8765/docs
• openapi json: http://127.0.0.1:8765/openapi.json
• health: GET /health
• models: GET /v1/models
• presets: GET /presets
• inference: POST /v1/systemone

curl --fail-with-body http://127.0.0.1:8765/v1/systemone \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "knivesysl-typed-2b",
    "state": {"message": "my card was charged twice. please refund the duplicate."},
    "questions": {
      "department": {
        "type": "choice",
        "instructions": "which team should handle this message?",
        "criteria": {
          "billing": "payments, charges, invoices, and refunds",
          "technical": "bugs and integration failures",
          "other": "requests outside billing and technical support"
        }
      },
      "refund_requested": {
        "type": "noul",
        "instructions": "does the customer explicitly request a refund?"
      },
      "urgency": {
        "type": "score",
        "instructions": "how urgent is the request?",
        "criteria": ["no deadline", "prompt response requested", "immediate resolution requested"]
      }
    }
  }'

response fields:

• answers.<id>.choice: selected option; probabilities contains normalized support for every supplied option.
• answers.<id>.score: probability-weighted level index; probabilities and legend preserve the level distribution and descriptions.
• answers.<id>.noul: normalized support for yes. noul intentionally has no separate confidence field.
• confidence: distribution concentration, defined as 1 - entropy(probabilities) / log(option_count); it is not empirical calibration.
• usage: input and scored token accounting. output_tokens is zero because no tokens are generated.
• metadata: latency, backend, checkpoint, precision, candidate count, forward-call count, and scoring definition.

python client

from knivesysl_typed import Choice, KnivesyslClient, Noul, Score

with KnivesyslClient(base_url="http://127.0.0.1:8765") as client:
    result = client.system_one(
        state={"message": "my card was charged twice. please refund the duplicate."},
        questions={
            "department": Choice(
                instructions="which team should handle this message?",
                criteria={
                    "billing": "payments, charges, invoices, and refunds",
                    "technical": "bugs and integration failures",
                    "other": "anything else",
                },
            ),
            "refund_requested": Noul(
                instructions="does the customer explicitly request a refund?"
            ),
            "frustration": Score(
                instructions="how frustrated does the customer sound?",
                criteria=["calm", "frustrated", "very angry"],
            ),
        },
    )
    print(result.choices["department"].choice)
    print(result.nouls["refund_requested"].noul)
    print(result.scores["frustration"].score)

AsyncKnivesyslClient exposes the same system_one contract for async callers. remote callers only need http access; model weights and a gpu remain on the server.

important limits

• maximum request body: 2 mib.
• choice: 2–255 options; score: 2–10 levels.
• state must be a string, object, or array; all models reject unknown fields and non-finite numbers.
• candidate wording, tokenization, and length affect likelihoods. score results can favor shorter level descriptions.
• independent questions do not model dependencies between answers. use staged requests or explicit joint alternatives when fields depend on one another.
• gpu execution is serialized by a process lock. candidates within a request are batched; separate http requests are not cross-request batched.
• output_tokens: 0 does not mean zero model work. every candidate continuation is still evaluated.
• no fixed latency, calibration, semantic correctness, safety, or unattended-action guarantee is implied.

development

uv sync --extra fp8
HF_HUB_OFFLINE=1 TOKENIZERS_PARALLELISM=false \
  uv run pytest tests/test_engine.py tests/test_contract.py tests/test_live_precision.py

model revisions are pinned in benchmarks/results/model_revisions.json. generated benchmark logs, virtual environments, caches, and downloaded references are not source dependencies.

weights and attribution

model architecture, tokenizer, configuration, and weights are from Qwen/Qwen3.5-2B, revision 15852e8c16360a2fea060d615a32b45270f8a8fc, under the Apache 2.0 license. knivesysl-typed changes the inference procedure, not the stored model parameters. use the GitHub implementation for typed complete-candidate scoring; ordinary Transformers generation from this repository behaves as the upstream Qwen checkpoint.