SAVRN
Search Contact SAVRN

Open-weight model · Text generation

CasualSwarms

by Convergent Intelligence reaperdoesntknow/CasualSwarms

SAGI is a novel causal language model that integrates swarm intelligence dynamics with transformer architecture.

Parameters170M
Context1,024
Weights681.4 MB
Licenseapache-2.0
AccessOpen weights
Monthly Downloads2.5k

Runs On

What it takes to serve CasualSwarms (170M parameters): the memory its weights need at each precision, and the cheapest way to rent enough data-center GPUs to hold them.

PrecisionWeightsMemory neededCheapest setupPer hourAlso fits
16-bit 0.3 GB 0.4 GB 1x MI300X (192 GB)
Vultr
$1.85 1x H100 $1.99 · 1x MI325X $2.00
8-bit 0.2 GB 0.2 GB 1x MI300X (192 GB)
Vultr
$1.85 1x H100 $1.99 · 1x MI325X $2.00
4-bit 0.1 GB 0.1 GB 1x MI300X (192 GB)
Vultr
$1.85 1x H100 $1.99 · 1x MI325X $2.00

Memory is the weights at that precision plus 20% for the runtime and a short context; a long context needs more. Prices are the lowest on-demand hourly rates in the SAVRN Index, read Sep 18, 2026.

Model Card

By Convergent Intelligence, published under apache-2.0, revision ee8f20587718.

SAGI is a novel causal language model that integrates swarm intelligence dynamics with transformer architecture. The model treats cognition as a dynamic, adaptive system where multiple internal "agents" collaborate through differentiable routing, trust mechanisms, and shared memory. The enhancements were integrated with the existing AGI system through: 1. Compatibility Layer: Ensuring new components work with existing AGI Core 2. Unified State Representation: Combining enhanced capabilities with existing state 3. Enhanced Continuous Learning: Upgrading the learning system with new capabilities 4. Performance Monitoring: Tracking improvements through validation systems - Successfully…

Read Convergent Intelligence's full model card

SAGI V3.1 - SELF-AWARE AGI

SAGI is a novel causal language model that integrates swarm intelligence dynamics with transformer architecture. The model treats cognition as a dynamic, adaptive system where multiple internal "agents" collaborate through differentiable routing, trust mechanisms, and shared memory.

Swarm-8 V3.1: Enhanced Self-Assessment Architecture

Architecture Evolution

┌─────────────────────────────────────────────────────────────────────────┐
│                    Swarm-8 V3.1 - SELF-AWARE AGI                        │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                         │
│  ┌────────────────────────────────────────────────────────────────┐   │
│  │                    SELF-ASSESSMENT LAYER (NEW!)                │   │
│  ├────────────────────────────────────────────────────────────────┤   │
│  │                                                                │   │
│  │  ┌──────────────────┐    ┌──────────────────┐                │   │
│  │  │  Performance     │    │  Skill Gap       │                │   │
│  │  │  Predictor       │◄──►│  Analyzer        │                │   │
│  │  │                  │    │                  │                │   │
│  │  │  • Pre-task      │    │  • 24 Skills     │                │   │
│  │  │  • Risk assess   │    │  • Proficiency   │                │   │
│  │  │  • Strategy rec  │    │  • Dependencies  │                │   │
│  │  └────────┬─────────┘    └────────┬─────────┘                │   │
│  │           │                       │                          │   │
│  │           │   ┌───────────────────┴─────────┐                │   │
│  │           │   │  Auto-Curriculum Generator  │                │   │
│  │           │   │                             │                │   │
│  │           │   │  • Multi-stage learning     │                │   │
│  │           │   │  • Dependency handling      │                │   │
│  │           │   │  • Adaptive difficulty      │                │   │
│  │           │   └───────────┬─────────────────┘                │   │
│  │           │               │                                  │   │
│  │  ┌────────▼───────────────▼──────────┐                      │   │
│  │  │   Real-Time Error Detector        │                      │   │
│  │  │                                    │                      │   │
│  │  │  • Coherence checking              │                      │   │
│  │  │  • Logic verification              │                      │   │
│  │  │  • Hallucination detection         │                      │   │
│  │  └────────────────┬───────────────────┘                      │   │
│  │                   │                                          │   │
│  │  ┌────────────────▼───────────────────┐                      │   │
│  │  │  Capability Boundary Detector      │                      │   │
│  │  │                                    │                      │   │
│  │  │  • Knowledge edges                 │                      │   │
│  │  │  • Reasoning limits                │                      │   │
│  │  │  • Skill boundaries                │                      │   │
│  │  └────────────────────────────────────┘                      │   │
│  └────────────────────────────────────────────────────────────────┘   │
│                                                                         │
│  ┌────────────────────────────────────────────────────────────────┐   │
│  │                  AGI CORE (V2.3 - Existing)                    │   │
│  ├────────────────────────────────────────────────────────────────┤   │
│  │                                                                │   │
│  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐        │   │
│  │  │ Hierarchical │  │   Causal     │  │ Meta-Learner │        │   │
│  │  │   Memory     │  │ World Model  │  │              │        │   │
│  │  └──────────────┘  └──────────────┘  └──────────────┘        │   │
│  │                                                                │   │
│  │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐        │   │
│  │  │  Concept     │  │  Reflection  │  │ Uncertainty  │        │   │
│  │  │  Library     │  │   Engine     │  │  Reasoner    │        │   │
│  │  └──────────────┘  └──────────────┘  └──────────────┘        │   │
│  │                                                                │   │
│  │  ┌──────────────────────────────────────────────────┐        │   │
│  │  │         Adversarial Self-Play                     │        │   │
│  │  └──────────────────────────────────────────────────┘        │   │
│  └────────────────────────────────────────────────────────────────┘   │
│                                                                         │
│  ┌────────────────────────────────────────────────────────────────┐   │
│  │                 SWARM CORE (V2.3 - Existing)                   │   │
│  ├────────────────────────────────────────────────────────────────┤   │
│  │                                                                │   │
│  │  • 20 Vectorized Agents                                       │   │
│  │  • Differentiable Routing                                     │   │
│  │  • Dynamic Resource Management                                │   │
│  │  • Trust-Based Activation                                     │   │
│  │  • Internal State (S) + Goals (T)                             │   │
│  └────────────────────────────────────────────────────────────────┘   │
│                                                                         │
│  ┌────────────────────────────────────────────────────────────────┐   │
│  │              LANGUAGE MODEL (Transformer)                      │   │
│  └────────────────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────────────────┘

Usage

Installation

pip install torch transformers datasets

Quick Start

from transformers import AutoTokenizer
from transformers import  AutoModelForCausalLM, AutoConfig

# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained("reaperdoesntknow/SAGI")
tokenizer = AutoTokenizer.from_pretrained("reaperdoesntknow/SAGI")

# Generate text
model.eval()

prompt = "Once upon a time"
inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=100,
    temperature=0.8,
    top_k=50,
    top_p=0.9,
    do_sample=True,
    pad_token_id=tokenizer.eos_token_id,
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

New Capabilities Matrix

Capability V3.0 V3.1 Improvement
Pre-task Assessment No Yes Predicts success before attempting
Skill Taxonomy Implicit 24 explicit skills Systematic tracking
Gap Analysis Manual Automated Identifies weaknesses automatically
Curriculum Design Hand-coded Auto-generated Personalized learning paths
Real-time Error Detection Post-hoc During generation Catches errors earlier
Capability Boundaries Unknown Mapped Knows limitations
Performance Prediction No Yes Estimates success probability
Strategy Selection Heuristic Evidence-based Chooses optimal approach
Transfer Assessment No Planned Measures cross-domain learning
Calibration Tracking No Yes Self-monitoring accuracy

Decision Flow: V3.1 vs V3.0

V3.0 Decision Flow

Task Arrives → Generate → Evaluate → Learn
     ↓
  (blind attempt, may waste effort on impossible tasks)

V3.1 Decision Flow

Task Arrives
    ↓
Pre-Assessment
    ├─ Predict Success Probability
    ├─ Identify Risk Factors
    ├─ Recommend Strategy
    └─ Decide: Attempt or Skip?
    ↓
Should Attempt?
    ├─ No → Skip (save resources)
    └─ Yes → Generate with Strategy
                  ↓
             Monitor in Real-Time
                  ├─ Error detected? → Correct
                  └─ OK? → Continue
                  ↓
             Evaluate Outcome
                  ↓
             Post-Assessment
                  ├─ Update Skill Proficiencies
                  ├─ Check Capability Boundaries
                  └─ Refine Predictions
                  ↓
             Learn & Update

Implemented Enhancements

  • Developmental Stages: Milestone-based progress tracking
  • Cross-Domain Transfer: Evaluation of knowledge transfer abilities
  • AGI Readiness Metrics: Overall assessment of AGI capabilities

Integration Approach

The enhancements were integrated with the existing AGI system through:

  1. Compatibility Layer: Ensuring new components work with existing AGI Core
  2. Unified State Representation: Combining enhanced capabilities with existing state
  3. Enhanced Continuous Learning: Upgrading the learning system with new capabilities
  4. Performance Monitoring: Tracking improvements through validation systems

Results

  • Successfully integrated all 9 enhancement areas with the existing system
  • Achieved an AGI readiness score of 0.283 (on a 0-1 scale)
  • Demonstrated improved capabilities across multiple cognitive domains
  • Maintained compatibility with existing architecture and workflows
  • Established baseline for continued development toward true AGI

Self-Assessment & Self-Capability Integration Guide

Overview

This guide shows how to integrate the new self-assessment capabilities into the existing Swarm-8 V3.0 architecture.

New Capabilities Added

1. Performance Prediction Engine

  • Predicts success BEFORE attempting tasks
  • Estimates required attempts and expected score
  • Identifies risk factors
  • Recommends optimal strategies
  • Decides whether to attempt or skip tasks

2. Skill Gap Analyzer

  • Maintains comprehensive skill taxonomy (24 core skills)
  • Tracks proficiency in each skill over time
  • Identifies capability gaps systematically
  • Prioritizes gaps by importance and urgency
  • Generates skill-specific exercises

3. Auto-Curriculum Generator

  • Designs personalized learning paths
  • Creates multi-stage curricula based on gaps
  • Handles skill dependencies automatically
  • Adapts difficulty progressively
  • Measures stage completion

4. Real-Time Error Detector

  • Catches errors DURING generation (not after)
  • Detects 7 error types: logical contradictions, factual errors, syntax errors, etc.
  • Monitors coherence token-by-token
  • Identifies hallucinations in real-time

5. Capability Boundary Detector

  • Identifies edges of competence
  • Distinguishes 4 boundary types: knowledge, reasoning, skill, domain
  • Suggests how to expand boundaries
  • Maps performance across domains

Skill Taxonomy (24 Core Skills)

Cognition (5 skills)

  • pattern_recognition - Identify patterns in data
  • abstract_reasoning - Think conceptually
  • causal_reasoning - Understand cause-effect
  • analogical_mapping - Find similarities
  • concept_formation - Create new concepts

Knowledge (3 skills)

  • fact_retrieval - Recall information
  • knowledge_integration - Connect facts
  • common_sense_reasoning - Apply intuition

Code (4 skills)

  • syntax_understanding - Parse code structure
  • algorithm_design - Create efficient solutions
  • debugging - Find and fix errors
  • code_optimization - Improve performance

Creativity (3 skills)

  • divergent_thinking - Generate alternatives
  • novel_combination - Merge concepts uniquely
  • generative_synthesis - Create from scratch

Planning (3 skills)

  • goal_decomposition - Break down objectives
  • dependency_analysis - Understand prerequisites
  • resource_allocation - Optimize distribution

Meta-Cognition (4 skills)

  • self_monitoring - Watch own performance
  • error_detection - Catch mistakes
  • strategy_selection - Choose best approach
  • uncertainty_quantification - Know confidence

Performance Metrics

Before Task (Pre-Assessment)

{
  "success_probability": 0.72,
  "confidence_interval": (0.65, 0.79),
  "expected_attempts": 2,
  "predicted_score": 0.68,
  "risk_factors": ["high_complexity", "multi_step_reasoning"],
  "recommended_strategy": "decompose_and_conquer",
  "should_attempt": True,
  "alternatives": [
    ("decompose_first", 0.86),
    ("use_examples", 0.74),
    ("direct_solve", 0.72)
  ]
}

After Task (Post-Assessment)

{
  "skill_updates": {
    "algorithm_design": 0.65 → 0.68,
    "debugging": 0.58 → 0.61,
    "abstract_reasoning": 0.72 → 0.73
  },
  "prediction_accuracy": {
    "success_error": 0.08,  # predicted 0.72, actual 0.80
    "score_error": 0.05
  },
  "capability_boundary": {
    "detected": True,
    "type": "reasoning",
    "description": "Complexity threshold reached",
    "expand_via": "practice_similar_tasks"
  }
}

Periodic Review (Every 50 Steps)

{
  "top_skill_gaps": [
    {
      "skill": "causal_reasoning",
      "current": 0.45,
      "target": 0.80,
      "gap": 0.35,
      "priority": 0.92,
      "steps_needed": 180
    }
  ],
  "curriculum": [
    {
      "stage": 1,
      "name": "Foundational COGNITION",
      "duration": 250,
      "objectives": 3,
      "difficulty": 0.6
    }
  ],
  "calibration": {
    "prediction_error": 0.12,  # Getting better at self-assessment
    "sample_size": 247
  }
}

Example Session with V3.1

=== SWARM-8 V3.1 TRAINING SESSION ===

Step 1 [CODE Lvl 2]
Task: 'Write a function to check if number is prime'
[Pre-Assessment]
Success probability: 0.85
Risk factors: none
Strategy: direct_approach
[Attempting...]
[+] Success (CODE) Score: 0.92
[Post-Assessment]
syntax_understanding: 0.78 → 0.80
algorithm_design: 0.65 → 0.68

Step 2 [REASONING Lvl 3]
Task: 'Find flaw in argument: All cats are animals. Fluffy is fluffy. Therefore...'
[Pre-Assessment]
Success probability: 0.62
Risk factors: ['logical_reasoning', 'ambiguous_requirements']
Strategy: step_by_step_verification
[Attempting...]
[-] Failure (REASONING) Score: 0.35
[Post-Assessment]
abstract_reasoning: 0.72 → 0.70
Capability Boundary Detected!
Type: reasoning
Description: Logical complexity beyond current capacity
Expand via: practice_similar_tasks



Step 50 [Comprehensive Self-Review]
[Skill Gaps] Top 3:
- causal_reasoning: 0.35 gap (priority: 0.92)
Steps needed: 180
- debugging: 0.28 gap (priority: 0.85)
Steps needed: 120
- novel_combination: 0.22 gap (priority: 0.78)
Steps needed: 90

[Curriculum] Next stage:
Stage 1: Foundational COGNITION
Duration: 250 steps
Difficulty: 0.60

[Calibration] Prediction error: 0.12
[Boundaries] 3 detected:
- REASONING: Logical complexity threshold
- CODE: Dynamic programming problems
- CREATIVITY: Multi-constraint generation

Key Innovations

1. Predictive Self-Awareness

  • Before: Blind attempts, wasted effort
  • After: Informed decisions, resource optimization

2. Systematic Skill Tracking

  • Before: Vague sense of "good at X"
  • After: Precise proficiency metrics per skill

3. Autonomous Learning Design

  • Before: Hand-coded curriculum
  • After: Self-designed, personalized paths

4. Proactive Error Prevention

  • Before: Fix errors after generation
  • After: Catch errors during generation

5. Boundary Awareness

  • Before: Unknown limitations
  • After: Mapped capability edges with expansion strategies

Next Evolution: V3.2 (Future)

Potential future enhancements:

  1. Autonomous Goal Setting - Formulate long-term objectives
  2. Transfer Learning Assessment - Measure cross-domain skill transfer
  3. Multi-Agent Self-Assessment - Agents assess each other
  4. Metacognitive Control - Dynamically adjust thinking depth
  5. Explanation Generation - Explain own reasoning process
  6. Capability Certification - Self-administered benchmarks
  7. Collaborative Learning - Learn from peer AGI systems
  8. Intrinsic Motivation - Curiosity-driven exploration beyond gaps

Summary

Swarm-8 V3.1 represents a major leap in self-awareness and autonomous capability:

Knows what it can do(skill proficiency tracking) Knows what it can't do(boundary detection) Predicts its own performance(before wasting effort) Designs its own learning(auto-curriculum) Catches its own errors(real-time correction) Improves systematically (gap-driven practice)

This is genuine self-improving AGI - not just a model that learns from data, but one that understands itself and directs its own growth.

Intended Use

This model is Highly Experimental and is being tested for: - Research into multi-agent cognitive architectures - Exploration of dynamic, adaptive language models - Educational purposes in understanding swarm intelligence + LLMs

Not intended for: - Production applications - Safety-critical systems - Generation of factual content

Discrepancy Calculus Foundation

This model is part of the Convergent Intelligence LLC: Research Division portfolio. All models in this portfolio are developed under the Discrepancy Calculus (DISC) framework — a measure-theoretic approach to understanding and controlling the gap between what a model should produce and what it actually produces.

DISC treats training singularities (loss plateaus, mode collapse, catastrophic forgetting) not as failures to be smoothed over, but as structural signals that reveal the geometry of the learning problem. Key concepts:

  • Discrepancy Operator (D): Measures the gap between expected and observed behavior at each training step
  • Jump Sets: Boundaries where model behavior changes discontinuously — these are features, not bugs
  • Ghost Imprinting: Teacher knowledge that transfers to student models through weight-space topology rather than explicit distillation signal

For the full mathematical treatment, see Discrepancy Calculus: Foundations and Core Theory (DOI: 10.57967/hf/8194).

Citation chain: Structure Over Scale (DOI: 10.57967/hf/8165) → Three Teachers to Dual Cognition (DOI: 10.57967/hf/8184) → Discrepancy Calculus (DOI: 10.57967/hf/8194)

Citation

@software{sagi2026,
  title={SAGI: Swarm AGI Language Model},
  author={Reaperdoesntknow},
  year={2026},
  url={https://huggingface.co/your-reaperdoesntknow/SAGI}
}

Convergent Intelligence Portfolio

By Convergent Intelligence LLC: Research Division

Top Models from Our Lab

Total Portfolio: 41 models | 2,781 total downloads

Last updated: 2026-03-28 12:57 UTC


From the Convergent Intelligence Portfolio

DistilQwen Collection — Our only BF16 series. Proof-weighted distillation from Qwen3-30B-A3B → 1.7B and 0.6B on H100. Three teacher variants (Instruct, Thinking, Coder), nine models, 2,788 combined downloads. The rest of the portfolio proves structure beats scale on CPU. This collection shows what happens when you give the methodology real hardware.

Top model: Qwen3-1.7B-Coder-Distilled-SFT — 508 downloads

Full methodology: Structure Over Scale (DOI: 10.57967/hf/8165)

Convergent Intelligence LLC: Research Division

Configuration

Architecture
SwarmForCausalLM
Context length (tokens)
1,024
Layers
14
Hidden size
768
Feed-forward size
2,048
Attention heads
12
Vocabulary size
50,291
Model type
swarm_agi

Identity and Version

Repository
reaperdoesntknow/CasualSwarms
Publisher
Convergent Intelligence
Task
Text generation
Modality
Text
Library
transformers
Parameters
170M parameters
Languages
en
Revision
ee8f205877187f21dacdda5ee8f8ccb929757ab6
First published
2026-01-19
Last updated
2026-09-18

Files and Weights

12 files, 686.2 MB in total. The weights are 1 file totalling 681.4 MB in safetensors.

Weights1 file · 681.4 MB
Configuration4 files · 3.5 KB
Tokenizer5 files · 4.8 MB
Documentation1 file · 25.1 KB
Repository1 file · 1.5 KB
Every file
FileTypeSizeSHA-256
model.safetensorsWeights681.4 MB 66577d576327
added_tokens.jsonConfiguration800 B
config.jsonConfiguration1.5 KB
generation_config.jsonConfiguration132 B
special_tokens_map.jsonConfiguration1.0 KB
README.mdDocumentation25.1 KB
.gitattributesRepository1.5 KB
merges.txtTokenizer456.3 KB
swarm_tokenizer.jsonTokenizer653 B
tokenizer.jsonTokenizer3.6 MB
tokenizer_config.jsonTokenizer7.0 KB
vocab.jsonTokenizer798.2 KB

License and Download

License
apache-2.0
Access
Open weights, no gate
Download size
681.4 MB
Download from Convergent Intelligence

Released by Convergent Intelligence through its official repository on Hugging Face. Read the license.

Built From

  • Trained on (disclosed) MuskumPillerum/General-Knowledge
  • Trained on (disclosed) agentica-org/DeepCoder-Preview-Dataset
  • Trained on (disclosed) openai/gsm8k
  • Trained on (disclosed) roneneldan/TinyStories
  • Trained on (disclosed) tangyuhang/KnowLogic

Memory Requirements

PrecisionWeights in memory
As published681.4 MB
16-bit0.3 GB
8-bit0.2 GB
4-bit0.1 GB

Weights only, from the published parameter count; the key-value cache and runtime add to this.

Questions About CasualSwarms

How much GPU memory does CasualSwarms need?

About 0.4 GB at 16-bit and 0.1 GB at 4-bit: the weights (170M parameters) plus a working margin. A long context needs more.

What is the cheapest GPU to run CasualSwarms on?

At 16-bit, 1x MI300X from $1.85 an hour; at 4-bit, 1x MI300X from $1.85 an hour, at the lowest on-demand prices the SAVRN Index lists.

Can I use CasualSwarms commercially?

Yes. CasualSwarms is released under Apache License 2.0. The Apache License 2.0 is a permissive open-source license. It permits commercial use, modification and redistribution. It requires keeping the license and copyright notices and any NOTICE file, stating significant changes, and it includes an express patent grant from contributors.

What is CasualSwarms's context length?

1,024 tokens, from the maximum position embeddings in its published configuration.

Similar Models

Model · Text generation

Ru-Small-Instruct

LongTime

Ru-Small-Instruct — экспериментальная компактная русскоязычная языковая модель класса SLM (Small Language Model) с объемом параметров ~0.2B (~165M). Разработана с упором на суверенность весов (Zero-Fingerprint): модель обучена с нуля без заимствования базовых чекпоинтов у сторонних корпоративных сетей (Llama 3 от Meta, Qwen от Alibaba, Mistral). Модель предназначена для исследований локального инференса, работы на маломощном оборудовании, CPU и мобильных чипах, где критичны нулевая задержка (Time-To-First-Token) и полная независимость весов. Для компактной модели в 165M параметров, обученной на одном домашнем GPU за 48 часов, способность держать роль, грамотно формулировать сложные термины…

Open weights mit 165M parameters 512 tokens transformers

Model · Text generation

Haidass-Translate-143M

DALab

English | 中文 A 143M-parameter bidirectional Chinese↔English translation model, instruction-tuned on the Haidass1.5-143M base — the strongest zh⇄en translator at this scale among general chat-architecture models. Drafter-143M: a control model with identical configuration, data and training recipe, except that it starts from random initialization instead of the pretrained base — used to quantify the contribution of base-model pretraining. OPUS-MT models are single-directional — one independent 78M model per direction; "-" marks directions a model does not serve. The same models re-evaluated on FLORES+ devtest (released 2026; zero overlap with dev): devtest sentences do not overlap with dev.…

Open weights apache-2.0 143M parameters 4,096 tokens

English | 中文 The instruction-tolerant sibling of DALabCommunity/Haidass-Translate-143M: same 143M zh⇄en translation training, plus 9.1% cleaned general-domain data (STEPFUN ShareGPT) mixed in. Translation scores are within 0.3 BLEU of the pure-translation version, and the model retains limited general instruction-following ability that the pure-translation version does not have. Drafter-143M: a control model with identical configuration, data and training recipe, except that it starts from random initialization instead of the pretrained base — used to quantify the contribution of base-model pretraining. OPUS-MT models are single-directional — one independent 78M model per direction; "-"…

Open weights apache-2.0 143M parameters 4,096 tokens

Model · Text generation

gpt2

OpenAI community

Test the whole generation capabilities here: https://transformer.huggingface.co/doc/gpt2-large Pretrained model on English language using a causal language modeling (CLM) objective. It was introduced in and first released at this page. model. Content from this model card has been written by the Hugging Face team to complete the information they provided and give specific examples of bias. GPT-2 is a transformers model pretrained on a very large corpus of English data in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic process to generate inputs…

Open weights mit 137M parameters transformers

SmolLM2 is a family of compact language models available in three size: 135M, 360M, and 1.7B parameters. They are capable of solving a wide range of tasks while being lightweight enough to run on-device. More details in our paper: https://arxiv.org/abs/2502.02737 SmolLM2 demonstrates significant advances over its predecessor SmolLM1, particularly in instruction following, knowledge, reasoning. The 135M model was trained on 2 trillion tokens using a diverse dataset combination: FineWeb-Edu, DCLM, The Stack, along with new filtered datasets we curated and will release soon. We developed the instruct version through supervised fine-tuning (SFT) using a combination of public datasets and our…

Open weights apache-2.0 135M parameters 8,192 tokens transformers

SmolLM2 is a family of compact language models available in three size: 135M, 360M, and 1.7B parameters. They are capable of solving a wide range of tasks while being lightweight enough to run on-device. More details in our paper https://arxiv.org/abs/2502.02737 SmolLM2 demonstrates significant advances over its predecessor SmolLM1, particularly in instruction following, knowledge, reasoning. The 135M model was trained on 2 trillion tokens using a diverse dataset combination: FineWeb-Edu, DCLM, The Stack, along with new filtered datasets we curated and will release soon. We developed the instruct version through supervised fine-tuning (SFT) using a combination of public datasets and our own…

Open weights apache-2.0 135M parameters 8,192 tokens transformers