gemma-4-12b-it-coreml-128k

Gemma 4 12B IT — Core ML, 128K Context Ladder

Run Google's Gemma 4 12B IT locally on an Apple silicon Mac with Core ML. One int4 bundle supports conversations up to 131,072 tokens, switching from its 32K mode to its 128K mode without reloading weights or copying the conversation cache.

Swift runtime & app · Model guide · Watch the X demo · Article: English / Japanese

Who this is for

• For: developers trying long-context chat on a 24 GB+ Apple silicon Mac, or studying a multifunction Core ML bundle with shared MLState.
• Not for: iPhone/iPad or Neural Engine deployment; this bundle keeps its conversation cache on the Mac GPU.
• Not for: temperature / top-p sampling, or fast 128K decoding; the measured 128K speed is ~3.3 tok/s under the conditions in Key numbers.

Requires: macOS 26+ · Xcode 26 / Swift 6.2 · ~11 GB disk · ~12 GB inference memory, 24 GB+ recommended. See Requirements for measured peaks.

Quick start

Use the published companion CLI:

git clone https://github.com/oka-yuji/coreml-llm-samples.git
cd coreml-llm-samples
./scripts/download-model.sh
swift run -c release --package-path CoreLLMKit corellm-chat --model ./models/gemma-4-12b-it-coreml-128k --stats

The script downloads this bundle to ./models/gemma-4-12b-it-coreml-128k. The model guide also covers one-shot prompts and CLI flags. For the app, open CoreMLSamples.xcodeproj at the repository root and run the CoreMLSamples scheme on My Mac.

For a standalone download, the existing Hub command is:

hf download okayuji/gemma-4-12b-it-coreml-128k --local-dir ./gemma-4-12b-it-coreml-128k

The runtime supplies the tokenizer, prompt template, host inputs and speculative decoding. The Swift ladder example below shows the shared-state mechanism; it is intentionally only a partial example.

Sample output

M4 Max (128 GB) / macOS 26.5.2, CPU_AND_GPU, greedy decode, 32 prompt tokens (BOS + template), one process per run, via the companion CLI.

Prompt (Japanese — "What is the capital of Japan? Answer concisely, with one sentence of reasoning."):

日本の首都はどこですか?理由も一文添えて簡潔に答えてください。

Output:

日本の首都は東京です。政治、経済、文化の中心地として機能しているためです。

("The capital of Japan is Tokyo. It serves as the country's political, economic and cultural center.")

The two runs produced byte-identical output — that is what "lossless speculation" means here: only the speed changes, never the text. The speculative run's larger TTFT is a one-time cost inside its first prefill, not a per-token cost.

Key numbers

Measured on an M4 Max (128 GB) / macOS 26.5.2, coremltools 9.0, compute units CPU_AND_GPU, single-token decode (S=1), one process per condition.

The speculative-decode multipliers span the range ×1.05 (verbatim) → ×1.27 (quoting) → ×1.47 (enumeration), all lossless, and match the standalone 32K bundle to within ±0.03.

Timing clarification (2026-10-23): the earlier card attached ~0.64 s to the copy-zero promotion row. That value belongs to the 2026-07-13 copy-based prototype (state mapping plus write_state), not the shared-state bundle shipped here. The 2026-07-15 multifunction build switches functions without state export/import; its benchmark found no measurable additional switch cost. The old timing remains a result for the earlier design, not a latency claim for this bundle.

Benchmark sources: the author's 2026-07-15-phasec-c0v1-build.md (decode and shared-state gates), 2026-07-13-phasec-c0-ctx-ladder.md (copy-based prototype), and 2026-07-19-12b-ladder-drafter-regime.md (speculative speedups). These lab records are not public; the public model guide summarizes the benchmark conditions, and the verification guide describes the equality checks.

How the Context Ladder works

The bundle is an MLProgram + MLState graph that co-hosts two functions — ctx32k and ctx128k — in one weight set. The full-attention KV state is declared once at the maximum width [1, 131072, 512] and shared by both functions, so moving from the 32K to the 128K regime is a copy-zero promotion: the same MLState buffer is simply read by a wider function. Promotion is bit-for-bit lossless (the zero-padded KV slots are masked with -inf, contributing exactly 0 to the softmax), so decode output is identical before and after promotion.

The result: short and medium conversations (≤32K tokens) decode at the fast 32K speed, and the model transparently steps up to 128K only when the context actually exceeds 32K — without a re-load and without a KV copy.

Swift ladder example

import CoreML

// 1. Compile once. Each .mlpackage becomes one .mlmodelc containing BOTH functions.
//    Measured: ~1.2 s for the whole bundle on an M4 Max (~0.3 s per chunk). Cache and reuse it.
let compiled = try await MLModel.compileModel(at: bundleURL.appending(path: "chunk_0_12.mlpackage"))

// 2. Open the same .mlmodelc twice, once per function. No recompile, weights are deduplicated.
func open(_ function: String) async throws -> MLModel {
    let cfg = MLModelConfiguration()
    cfg.computeUnits = .cpuAndGPU        // required: the KV MLState is GPU-resident
    cfg.functionName = function          // "ctx32k" | "ctx128k"
    return try await MLModel.load(contentsOf: compiled, configuration: cfg)
}
let chunk32  = try await open("ctx32k")
let chunk128 = try await open("ctx128k")

// 3. ONE state, made from either model — it is physically [1, 131072, 512] in both cases.
let state = chunk32.makeState()

// 4. Decode through ctx32k; once the write position reaches 32768, decode through ctx128k
//    instead. Same `state` object, no read_state/write_state — that is the copy-zero promotion.
let model = (position < 32768) ? chunk32 : chunk128
let out = try model.prediction(from: inputs, using: state)

Prompt template

From manifest.json (promptPrefix / promptSuffix). These are not the stock Gemma <start_of_turn> markers — a loader that assumes them will silently produce a different token sequence. A single turn is BOS + promptPrefix + user text + promptSuffix:

<bos><|turn>user
{user text}<turn|>
<|turn>model
<|channel>thought
<channel|>

The model continues from there. For multi-turn, close the assistant's text with <turn|>\n and append the next user turn with the same prefix/suffix:

<bos><|turn>user
{turn 1 user}<turn|>
<|turn>model
<|channel>thought
<channel|>{turn 1 model}<turn|>
<|turn>user
{turn 2 user}<turn|>
<|turn>model
<|channel>thought
<channel|>

The exact strings, verbatim from manifest.json (no trailing newline after <channel|>):

"promptPrefix": "<|turn>user\n",
"promptSuffix": "<turn|>\n<|turn>model\n<|channel>thought\n<channel|>"

Use tokenizer.json from this repository, and prepend BOS if the tokenizer does not. With BOS, the sample prompt shown earlier encodes to 32 tokens.

Files

Total download ~10.2 GiB (~11 GB). The four chunk_*.mlpackage folders plus lmhead.mlpackage are the 6.71 GiB Core ML model bundle; the drafters, embedding table, and tokenizer account for the remaining ~3.5 GiB.

Size units corrected on 2026-10-23: the earlier card labeled these binary estimates as GB/MB. The Hub file inventory at revision 40488bbd72275753c25a80b009c25882cc49a6b0 contains 10,945,351,416 bytes of assets (excluding README.md and .gitattributes), including 7,207,132,566 bytes for the chunks and LM head. File sizes below use GiB/MiB; RAM observations retain the units of their original measurements.

The chunks and lm_head are shipped as .mlpackage only. The Swift loader compiles them to .mlmodelc on first load and caches the result, so no pre-compiled .mlmodelc copies are included. The two drafters are shipped as .mlmodelc (the loader detects those directly).

Requirements

• Apple Silicon Mac (M-series). This bundle is effectively Mac-only: the KV cache lives in an MLState that is GPU-resident, so the Apple Neural Engine (ANE) cannot be used and compute units must be CPU_AND_GPU.
• macOS 26 (Tahoe) or later (multifunction models + MLState function-specialized load).
• Memory: ~12 GB at inference; 24 GB+ recommended. The full-attention KV state (~2.1 GB) is resident from the very first token, and process RSS reaches ~10 GB. Note that the two sample runs above reported a peak memory figure of 18.0 GB (no speculation) / 20.5 GB (speculation on, which keeps a drafter resident) on a 128 GB machine — the peak is well above the steady-state RSS, so treat 24 GB as a practical floor rather than a comfortable one.
• Disk: ~11 GB for the downloaded bundle.
• GPU is required. There is no CPU-only or ANE fallback for this configuration.

Operational notes & troubleshooting

• First load compiles the model once. Each .mlpackage is compiled to .mlmodelc on first use and cached to disk; subsequent loads are warm. This step is cheap: ~1.2 s for the whole bundle on an M4 Max (about 0.3 s per chunk).
• "It hangs on the first token." The lm_head's GPU specialization runs before the first inference. A 2026-07-05 measurement put this at ~40 s per process; a 2026-07-21 re-check on macOS 26.5.x instead found it cached across processes — consistent with the Sample-output table above, where fresh processes reached the first token in 0.26 s (drafter off) / 3.83 s (drafter on). Budget the full ~40 s only for a first-ever run (no cache, or after clearing the E5RT cache); later runs are fast. Pre-warm at launch if you embed this in an app.
• Disk fills up, or loads get flaky after many runs. Core ML's E5RT cache (~/Library/Caches/com.apple.e5rt.e5bundlecache) grows by several GB per run. Deleting it is safe and recovers the space; it regenerates, and the next load is just slower.
• Out of memory / heavy swapping. Close other large apps. See the memory note in Requirements — below 24 GB this configuration is not realistic.
• "bundle has no drafter — running without speculation." drafter_ring.mlmodelc / drafter_ring32k.mlmodelc were not found next to manifest.json. Re-run the download.
• Loading your own way: point the loader at the bundle directory; manifest.json and convert_config_ladder.json describe the chain, prompt template, and ladder functions.

Limitations

• 128K mode is slow by design. 300.96 ms/tok (~3.3 tok/s) versus 90.51 ms/tok in 32K mode — a fixed wide-context tax of 300.96 / 90.51 ≈ 3.3× at int4. The ladder exists to avoid paying it below 32K; it does not make a genuinely 128K-deep context fast.
• Greedy argmax only. The lm_head emits an argmax token id, not logits. There is no temperature / top-k / top-p sampling — output is deterministic. Sampling would require re-converting with a logits head.
• Mac only. The stateful chain keeps its KV in GPU-resident MLState; CPU-only and ANE configurations are not usable with multiple MLState instances. There is no iOS build.
• Speculation's edge shrinks with context. The fixed-width verify tax means the largest wins are on short, structured prompts; free-form prose is near break-even (×1.05).
• First-ever run pays ~40 s of one-time GPU specialization (measured 2026-07-05). On macOS 26.5.x this is cached across processes (re-checked 2026-07-21) — later launches reach the first token in seconds — so budget ~40 s only for the first run or after clearing the E5RT cache.

Verification

These gates are listed because "it runs" and "it computes the same thing as the reference" are different claims, and this bundle makes the second one.

Correctness was gated against the PyTorch (Hugging Face) reference at the bit level. The 128K conversion passed all 13 correctness gates (RoPE range at pos 131071, schema/runtime, self-consistency, cross-generation int4/fp16 exact matches, ring-boundary, deep-feed to 40960); the Context Ladder passed all four ladder gates exact, including the copy-zero promotion gate (feed 2000 tokens in ctx32k, switch functions on the same state with zero read/write, then match the ctx128k reference greedy output with #diff=0). The rare near-tie cases (fp16 summation-order differences near the 32768 boundary) were attributed by an arbitration panel and never produced a hard failure — every near-tie had the oracle token within the chain's top-2.

Related samples

This bundle runs with the Core ML runtime in coreml-llm-samples. For separate Core AI / Foundation Models conversions and examples, see coreai-samples. That project is a related collection for comparing the frameworks; its runtime uses different model assets and does not load this Core ML bundle.

License & attribution

• Base model: google/gemma-4-12B-it. The artifacts here are derived from Gemma 4 12B IT: the original files have been modified — the weights converted to a Core ML graph and quantized (int4 AWQ matmul / int8 lm_head).
• License: Apache 2.0. Google releases Gemma 4 under the Apache License 2.0, and these derivative artifacts are redistributed under the same license. The full text is in the bundled LICENSE.
• No source code here. This repository contains model weights and configuration only — no source code. The companion Swift package on GitHub is separately licensed under MIT.
• Built with Gemma.
• "Core ML", "MLState", and "Apple Neural Engine" are trademarks of Apple Inc.; "Gemma" is a trademark of Google LLC. Provided as-is, without warranty. Not affiliated with, sponsored by, or endorsed by Google or Apple.