RoPE-free long-range attention, trained at 4k with zero-shot length extrapolation. FS-B searches full-prefix and norm-filtered forests of chunk mean keys, reranks their candidates, and reads four chunks. Its attention is superlinear and subquadratic. This 743M research base model was trained on FineWeb-Edu at a 4,096-token context. The short retrieval example asks for the hidden number 5437923. Start with CUDA-enabled PyTorch; Usage gives the tested installation and speed-replication command. FreeSolo learns simple chunk mean keys that support search, norm pruning, and hierarchical summarization. RoPE stays in the local window; long-range attention reads selected original tokens without…
FreeSolo-743M is an open-weight model for text generation from Zekun Shi, released under MIT License. It has 743M parameters. At 16-bit it needs about 1.8 GB of GPU memory, which fits on 1x MI300X from $1.85 an hour, at the lowest prices in the SAVRN Index.
RoPE-free long-range attention, trained at 4k with zero-shot length extrapolation. FS scores all completed chunk mean keys and reads four selected chunks per query and head. This 743M research base model was trained on FineWeb-Edu at a 4,096-token context.
Parameters743M
Context—
Weights3.0 GB
Licensemit
AccessOpen weights
Monthly Downloads—
Runs On
What it takes to serve FreeSolo-743M (743M parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them.
| Precision | Weights | Memory needed | Cheapest setup | Per hour | Also fits |
|---|---|---|---|---|---|
| 16-bit | 1.5 GB | 1.8 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x H100 $1.99 · 1x MI325X $2.00 |
| 8-bit | 0.7 GB | 0.9 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x H100 $1.99 · 1x MI325X $2.00 |
| 4-bit | 0.4 GB | 0.4 GB | 1x MI300X (192 GB) Vultr |
$1.85 | 1x H100 $1.99 · 1x MI325X $2.00 |
Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the SAVRN Index, read Oct 16, 2026.
FreeSolo-743M on every accelerator the SAVRN Index prices, at every precision
Model Card
By Zekun Shi, published under mit, revision 14d2137f3e1a.
RoPE-free long-range attention, trained at 4k with zero-shot length extrapolation. FS scores all completed chunk mean keys and reads four selected chunks per query and head. This 743M research base model was trained on FineWeb-Edu at a 4,096-token context. The short retrieval example asks for the hidden number 5437923. Start with CUDA-enabled PyTorch; Usage gives the tested installation and speed-replication command. FreeSolo learns simple chunk mean keys that support search, norm pruning, and hierarchical summarization. RoPE stays in the local window; long-range attention reads selected original tokens without rotating their queries and keys. 12 tasks at 2k/4k; 13 at 8k/16k, where QA-2 is…
Read Zekun Shi's full model card
Paper · GitHub · Models · Usage
RoPE-free long-range attention, trained at 4k with zero-shot length extrapolation.
FS scores all completed chunk mean keys and reads four selected chunks per query and head. This 743M research base model was trained on FineWeb-Edu at a 4,096-token context.
Try it
hf download zekun-shi/FreeSolo-743M freesolo-source.zip --local-dir ./FreeSolo-743M
python -m pip install ./FreeSolo-743M/freesolo-source.zip
freesolo-generate --model zekun-shi/FreeSolo-743M --output generation.json
The short retrieval example asks for the hidden number 5437923. Start with CUDA-enabled PyTorch; Usage gives the tested installation and speed-replication command.
The idea
FreeSolo learns simple chunk mean keys that support search, norm pruning, and hierarchical summarization. RoPE stays in the local window; long-range attention reads selected original tokens without rotating their queries and keys.
Results
| Context | 2k | 4k | 8k | 16k |
|---|---|---|---|---|
| RULER mean | 39.61 | 34.07 | 27.79 | 23.23 |
12 tasks at 2k/4k; 13 at 8k/16k, where QA-2 is available. 100 examples per task, seed 42. These are checkpoint evaluation results; the figures show the paper's model-family comparison.
This is a base model without instruction tuning. Retrieval is task dependent, especially when several facts or values must be combined. Training details, artifact checksums, and citation are in Usage.
Configuration
- Vocabulary size
- 32,000
- Model type
- freesolo
Identity and Version
- Repository
- zekun-shi/FreeSolo-743M
- Publisher
- Zekun Shi
- Task
- Text generation
- Modality
- Text
- Library
- pytorch
- Parameters
- 743M parameters
- Languages
- en
- Revision
- 14d2137f3e1ab44bd7825abc775ef464015c8c13
- First published
- 2026-10-15
- Last updated
- 2026-10-16
Files and Weights
10 files, 3.0 GB in total. The weights are 1 file totalling 3.0 GB in safetensors.
Weights1 file · 3.0 GB
Configuration1 file · 1.2 KB
Documentation3 files · 8.0 KB
Other4 files · 1.7 MB
Repository1 file · 1.7 KB
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| model.safetensors | Weights | 3.0 GB | e2e7f78a4287 |
| config.json | Configuration | 1.2 KB | — |
| LICENSE | Documentation | 1.1 KB | — |
| README.md | Documentation | 2.2 KB | — |
| USAGE.md | Documentation | 4.8 KB | — |
| assets/figure-1.png | Other | 122.7 KB | f1937f7b6ee3 |
| assets/figure-2.png | Other | 296.9 KB | 192b3bd396e9 |
| assets/figure-3.png | Other | 254.8 KB | 055de9b7fd7f |
| freesolo-source.zip | Other | 987.5 KB | 7d3c4c6a9174 |
| .gitattributes | Repository | 1.7 KB | — |
License and Download
- License
- mit
- Access
- Open weights, no gate
- Download size
- 3.0 GB
Download from Zekun Shi
Released by Zekun Shi through its official repository on Hugging Face. Read the license.
Built From
- Described by arXiv:2610.17191
- Trained on (disclosed) HuggingFaceFW/fineweb-edu
Memory Requirements
| Precision | Weights in memory |
|---|---|
| As published | 3.0 GB |
| 16-bit | 1.5 GB |
| 8-bit | 0.7 GB |
| 4-bit | 0.4 GB |
Weights only, from the published parameter count; the key-value cache and runtime add to this.
Built on This Model
- Derived fromFreeSolo-B-743M
- Derived fromFreeSolo-L-743M
Questions About FreeSolo-743M
How much GPU memory does FreeSolo-743M need?
About 1.8 GB at 16-bit and 0.4 GB at 4-bit: the weights (743M parameters) plus a working margin. A long context needs more.
What is the cheapest GPU to run FreeSolo-743M on?
At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists.
Can I use FreeSolo-743M commercially?
Yes. FreeSolo-743M is released under MIT License. The MIT License is a short permissive license. It permits commercial use, modification and redistribution, provided the copyright notice and permission notice are included.
Similar Models
RoPE-free long-range attention, trained at 4k with zero-shot length extrapolation. FS-L retains a bounded pool of chunks by key norm and recency, then reads four selected chunks. Its attention scales linearly with sequence length. This 743M research base model was trained on FineWeb-Edu at a 4,096-token context. The short retrieval example asks for the hidden number 5437923. Start with CUDA-enabled PyTorch; Usage gives the tested installation and speed-replication command. FreeSolo learns simple chunk mean keys that support search, norm pruning, and hierarchical summarization. RoPE stays in the local window; long-range attention reads selected original tokens without rotating their queries…
Qwen3 is the latest generation of large language models in Qwen series, offering a comprehensive suite of dense and mixture-of-experts (MoE) models. Built upon extensive training, Qwen3 delivers groundbreaking advancements in reasoning, instruction-following, agent capabilities, and multilingual support, with the following key features: - Uniquely support of seamless switching between thinking mode (for complex logical reasoning, math, and coding) and non-thinking mode (for efficient, general-purpose dialogue) within single model, ensuring optimal performance across various scenarios. - Significantly enhancement in its reasoning capabilities, surpassing previous QwQ (in thinking mode) and…
A 0.6B parameter model built in two stages: knowledge distillation from a 30B Thinking teacher to establish a structured reasoning backbone, then supervised fine-tuning on legal instruction data. 50x compression. Under 500MB quantized. Runs on a phone. The training order is the thesis: teach the model how to reason first (distillation from Thinking teacher), then teach it what to reason about (legal SFT). The Thinking teacher's extended deliberation traces transfer deeper reasoning structure than an Instruct teacher — critical when the student has only 0.6B parameters to work with. Qwen3-0.6B distilled from Qwen3-30B-A3B-Thinking-2507 — a Mixture-of-Experts model with 30B total parameters…
A 0.6B parameter model distilled from Qwen3-30B-A3B-Thinking on 6,122 STEM chain-of-thought samples. 50x parameter compression. The Thinking variant teacher produces richer extended reasoning traces than the Instruct variant, transferring deeper deliberation structure into the smallest possible student. The result: a model under 500MB quantized that produces structured STEM derivations because a 30B thinking model showed it how to reason. Two key differences from standard small-model distillation: 1. Thinking teacher, not Instruct teacher. The Qwen3-30B-A3B-Thinking variant generates extended internal reasoning before committing to an answer. Its softmax distributions are higher-entropy…
Qwen3-Coder-Next quantized to 4-bit with Coded Quantization (CQ). CQ is a calibration-free 4-bit quantization method designed to preserve the behavior of the original model without requiring a calibration dataset. Because CQ does not optimize the quantization around a fixed calibration corpus, it is designed to generalize beyond calibration-specific data and workloads. This repository contains Qwen3-Coder-Next quantized with Coded Quantization (CQ) 4-bit. Calibration-free. CQ does not require a representative calibration dataset before quantization. This avoids tying the quantization procedure to a particular calibration corpus and is intended to provide better generalization beyond the…