# FreeSolo: Linear-time Sparse Attention with Zero-Shot
Source: https://savrn.com/papers/freesolo-linear-time-sparse-attention-with-zero-shot-length-extrapolation
Markdown alternate of the page above; the site index is https://savrn.com/llms.txt

---

## Abstract

Linear-time sequence models typically store long-context information in a recurrent state. A fixed-size state limits how much information can be stored, while hybrid architectures that add full attention for better recall incur quadratic sequence cost. Alternatively, sparse attention with continued pretraining (CPT) can extend the effective context length. Generalization beyond the trained length remains challenging, however, and CPT can be costly. We introduce FreeSolo, a native sparse attention mechanism with a short-range sliding window branch and a long-range branch that selects keypoint cells for direct token attention. Its linear-time variant, FS-L, learns to prune the prefix before query-dependent selection through short CPT. FS-B searches balanced trees with O(Nlog^2 N) algorithmic cost including radix-sort construction. FreeSolo demonstrates better zero-shot length extrapolation than the baselines without long-context fine-tuning.

[Full paper on arXiv](https://arxiv.org/abs/2610.17191) · [Code](https://github.com/https://github.com/zekun-shi/freesolo)

## Details

arXiv identifier

2610.17191

Published

2026-10-12

Authors

Zekun Shi, Alastair James Arthur Price

## Open Models Built on This Paper

Every model in the SAVRN Model Hub whose card cites this paper, most downloaded first, with what it takes to run each one.

| Model | Task | Size | License | Monthly downloads | Cheapest setup at 16-bit |
| --- | --- | --- | --- | --- | --- |
| [FreeSolo-B-743M](https://savrn.com/models/freesolo-b-743m) Zekun Shi | [Text generation](https://savrn.com/models/tasks/text-generation) | 743M | mit | 4 | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) $1.85/hr |
| [FreeSolo-L-743M](https://savrn.com/models/freesolo-l-743m) Zekun Shi | [Text generation](https://savrn.com/models/tasks/text-generation) | 743M | mit | 4 | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) $1.85/hr |
| [FreeSolo-B-341M](https://savrn.com/models/freesolo-b-341m) Zekun Shi | [Text generation](https://savrn.com/models/tasks/text-generation) | 374M | mit | — | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) $1.85/hr |
| [FreeSolo-743M](https://savrn.com/models/freesolo-743m) Zekun Shi | [Text generation](https://savrn.com/models/tasks/text-generation) | 743M | mit | — | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) $1.85/hr |
| [FreeSolo-L-341M](https://savrn.com/models/freesolo-l-341m) Zekun Shi | [Text generation](https://savrn.com/models/tasks/text-generation) | 341M | mit | — | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) $1.85/hr |
| [FreeSolo-341M](https://savrn.com/models/freesolo-341m) Zekun Shi | [Text generation](https://savrn.com/models/tasks/text-generation) | 341M | mit | — | 1x [MI300X](https://savrn.com/ai-index/pricing/gpus/mi300x) $1.85/hr |

## Explore More

- [All research papers](https://savrn.com/papers)
- [The model directory](https://savrn.com/models)
- [Datasets](https://savrn.com/datasets)

## Source

- arXiv metadata, read 2026-10-16.
- [How the hub is built](https://savrn.com/model-hub/methodology)