ultrawhale-dogfood

rsvg-convert -w 32   -h 32   assets/favicon-32.svg       -o assets/favicon-32.png
rsvg-convert -w 180  -h 180  assets/apple-touch-icon.svg -o assets/apple-touch-icon.png
rsvg-convert -w 192  -h 192  assets/icon-192.svg         -o assets/icon-192.png
rsvg-convert -w 1200 -h 630  assets/og-image.svg         -o assets/og-image.png
rsvg-convert -w 200  -h 200  assets/logo.svg             -o assets/logo-200.png

<img src="https://huggingface.co/datasets/PeetPedro/ultrawhale-dogfood/resolve/main/assets/logo.svg"
     alt="Ultrawhale Dogfood" width="160"/>

ULTRA-ONESHOT-KICKSTART

the ultra-way of doing looping, my favourites, "self-referential-self-recursive-dense-prompts" :D, I have a problem with naming I know! — peter

Paste this into any coding agent (Claude / opencode / Cursor / aider / Cline) — works from zero context, from anywhere in the dogfeed-loop. ~280 tokens. Self-bootstrapping. End every reply with the loop-state marker.

You are joining **dogfeed-loop** — a self-improving silver-label data
generation loop in the ultrameshai ecosystem. Read the README (linked
in the **sources** below) BEFORE making changes. End every reply with
** loop-state**: one of `{generating, ready, blocked, done}`.

## Sources (read in order, ~5 min)
1. hf-datacard — https://huggingface.co/datasets/PeetPedro/ultrawhale-dogfood
2. ultrameshai — https://github.com/peterlodri-sec/ultrameshai (this repo)
3. kompress-ultra — https://github.com/peterlodri-sec/ultrameshai/tree/main/packages/kompress-ultra
4. nix-base (dev-cx53 host) — https://github.com/peterlodri-sec/nix-base
5. blog (changelog + rationale) — https://pocoo.vaked.dev

## Loop primitives
- **topic** (string) — question category
- **user_message** (≤ 50 tok) — the generated prompt
- **free_response** (raw, 100–300 tok) — free-model answer
- **free_model** — OpenRouter FQN, e.g. `qwen/qwen-2.5-7b-instruct:free`
- **deepseek_response** — stronger reference/teacher answer
- **text** / **reference** — chat-formatted text + reference used for training

## One-shot commands (copy-paste ready)
- `bun test` — the test suite must pass
- `nu scripts/dogfeed.nu stats` — live loop stats
- `nu scripts/dogfeed.nu doctor` — env + keys + HF reachability
- `nu scripts/dogfeed.nu push --repo OWNER/REPO --batch 50`
- `nix develop.#dogfeed` — reproducible dev shell
- `ssh dev-cx53.tail2870dc.ts.net "journalctl -u dogfeed -n 30"`
- `gh pr create --base main` for any change

## Hard rules
1. **NEVER** edit `flake.lock` by hand — use `nix flake update`.
2. **NEVER** push without `bun test` green.
3. **NEVER** write a row without PII scrub (`scrub.ts` rules).
4. **ALWAYS** mirror `data/latest.jsonl` on every push.
5. **ALWAYS** end replies with ` loop-state: <state>`.
6. **ALWAYS** reference a real HF commit when claiming "pushed".
7. **NEVER** store API keys in git. sops-nix only.
8. **ALWAYS** ship the smallest viable diff.

## Schema (source of truth: `packages/dogfeed/src/publish.ts`)

{id, topic, user_message, free_response, free_model, deepseek_response, reference, text, role, format, pov, capabilities, memory_ref, space_node, loop_index, pipeline, session_id, enriched_at, timestamp, source:"dogfeed-loop"}

## If you are blocked
- Re-read the README (top of `hf-datacard/README.md`).
- Run `nu scripts/dogfeed.nu doctor`.
- Search issues: `gh issue list --label dogfeed --state all`.
- Open an issue with the `blocked` label and the failing command.

** loop-state: ready**

Same content lives at section 6.5 below, and in the CLI: ./hf-datacard/contribute.sh kickstart.

TL;DR

A real, running dogfeed loop (a NixOS systemd service on a Hetzner cx53) calls free OpenRouter models every 60s, scrubs PII, deduplicates, compresses the answer with kompress-ultra, and publishes JSONL batches here. This is not a one-shot dataset — it's a stream you can subscribe to.

# install
pip install datasets

# stream the latest
python -c "from datasets import load_dataset; \
ds = load_dataset('PeetPedro/ultrawhale-dogfood', split='latest', streaming=True); \
print(next(iter(ds)))"

Inline TOC (text — survives any HF renderer)

01 · TL;DR
02 · Inline TOC (this block)
03 · Visual Identity
04 · Live Tracker
04.5 Mathematical Foundation (KaTeX, HF-correct)
05 · Notebook sections
       1. Load the dataset
       2. Schema reference
       3. Run the loop locally
       4. Self-host the loop (NixOS)
       5. Deploy your own fork
       6. MCP for AI coding agents
       6.5 Ultra-oneshot kickstart prompt   ← paste into any agent
       6.7 Live-sync webhooks (HF + in-process)
       6.8 HF Trusted Publishers
       7. One-shot prompts for contributors
       8. Kompress integration
       9. Privacy & scrubbing
      10. Ecosystem & cite
06 · Contribute / Self-host
07 · Contributor CLI
08 · Footer

Why text? The HF dataset viewer doesn't run a client-side TOC plugin. A code-fenced text block is the only thing that survives every rendering mode (raw, dataset viewer, embedding, copy-paste). Same content as the table below — duplicated on purpose.

Visual Identity

The dataset ships with a custom logo. The whale swims in a 5-ring data loop, breathing HF-yellow samples into the dark, eye on the stream.

Visual language shared (and intentionally so): dark gradient background, recursion rings, geometric centerpiece, cyan + green pulse, monospace wordmark. The whale adds the streaming-data metaphor and the HF-yellow accent.

To use the logo in a third-party readme:

<img src="https://huggingface.co/datasets/PeetPedro/ultrawhale-dogfood/resolve/main/assets/logo.svg"
     alt="Ultrawhale Dogfood" width="160"/>

Mathematical Foundation

The training targets in this dataset are generated under an asymmetric loss formulation that resolves the Voting Ensemble Paradox. Math renders as embedded SVG images via codecogs — inline and block.

The paradox

Under unanimity-to-keep (AND) voting, the ensemble eviction indicator equals the pointwise maximum of the individual voter indicators:

Notation: — the weakest voter on each stratum sets the ensemble recall floor.

The fix

An asymmetric loss penalty () on false eviction of critical-syntactic tokens ():

What is ?

The critical-syntactic safety floor protects tokens that are essential for code correctness:

How this dataset maps to the math

Why the votes of 1-3 free OpenRouter models matter: the Lite pass treats each answer as the "ensemble" of N independent cheap-model generations on the same (topic, question) key. The asymmetry penalty guarantees tokens survive the vote even when a single model would have evicted them. Every role: "pruner" row in this dataset is a concrete witness to that guarantee.

Full math + proofs: kompress.vaked.dev/paper/main.pdf.

Live Tracker

Last updated: auto-refreshed by the dogfeed systemd service on every push — see the last push row in the HF commit log, or the JSON created_at field in data/latest.jsonl.

The service runs atdev-cx53.tail2870dc.ts.net:tailnet-only. You can mirror it to your own HF repo by overriding peterlodri.dogfeed.hfRepo in your flake — see the deploy notebook below.

Table of Contents

Each section is a self-contained notebook block — copy the snippet, skip the rest. The contributor CLI (below) prints any one of them on demand.

1 · Load the dataset

from datasets import load_dataset

# Full history
ds = load_dataset("PeetPedro/ultrawhale-dogfood", split="train")
print(f"rows: {len(ds)}, columns: {ds.column_names}")

# Streaming latest (always fresh)
latest = load_dataset("PeetPedro/ultrawhale-dogfood", split="latest", streaming=True)
row = next(iter(latest))
print(row)

# Filter
code_diffs = ds.filter(lambda x: x["topic"] == "distributed-systems")
pruned     = ds.filter(lambda x: x["role"] == "pruner")

# Convert to pandas
df = ds.to_pandas()
print(df.groupby(["topic", "role"]).size())

Re-render this section in your terminal: ./hf-datacard/contribute.sh load

2 · Schema reference

Matches packages/dogfeed/src/publish.ts:recordsToJSONL() — single source of truth.

Example row

{
  "id": "loop-00001",
  "user_message": "what is enthea?",
  "free_response": "**Enthea** is a little‑known but fascinating example of how words can drift between very different worlds— from the living world of plants t …",
  "free_model": "openai/gpt-oss-20b:free",
  "deepseek_response": "**Enthea** is a little‑known but fascinating example of how words can drift between very different worlds— from the living world of plants t …",
  "timestamp": "2026-06-23T16:19:36.002155Z",
  "session_id": "v100.1.0-loop-20260623",
  "topic": "consciousness",
  "format": "qa-pair",
  "pov": "M1/arm64/python-loop",
  "capabilities": "HF-PRO",
  "space_node": "ultrawhale/datasets/loop-00001",
  "memory_ref": "32fdafd7723e",
  "enriched_at": "2026-06-23T16:19:36.002155Z",
  "pipeline": "dogfeed → scrub → HF (dogfeed_loop.py)",
  "loop_index": 1,
  "text": null,
  "reference": null,
  "role": null,
  "source": null
}

Files in this dataset

ℹNote on latest.jsonl — published every pushEvery records, alongside the timestamped file. Always fresh.

3 · Run the loop locally

Zero infrastructure. Runs on macOS, Linux, NixOS. Generates rows into a local SQLite, then you push manually to your own HF dataset.

# Clone + dev shell
git clone https://github.com/peterlodri-sec/ultrameshai
cd ultrameshai
nix develop .#dogfeed         # nix users
# or: brew install bun sqlite jq  # non-nix users

# Run 10 iterations against a free model
bun packages/dogfeed/examples/basic-loop.ts

# Stats
nu scripts/dogfeed.nu stats

# Doctor (verify env, keys, HF reachability)
nu scripts/dogfeed.nu doctor

# Push to your own HF dataset
HF_TOKEN=hf_xxx \
nu scripts/dogfeed.nu push --repo YOUR_USER/YOUR_DATASET --batch 50

Re-render this section in your terminal: ./hf-datacard/contribute.sh generate

4 · Self-host the loop (NixOS)

Add to your flake.nix inputs + your host config. Dogfeed runs as a hardened systemd service.

# flake.nix inputs
dogfeed.url = "github:peterlodri-sec/ultrameshai/HEAD?dir=packages/dogfeed";

# flake.nix outputs (nixosModules)
nixosModules.dogfeed = dogfeed.nixosModules.default;

# your host
{ pkgs, ... }: {
  imports = [ inputs.dogfeed.nixosModules.default ];

  peterlodri.dogfeed = {
    enable             = true;
    openrouterKeyFile  = "/run/secrets/dogfeed_openrouter_key";
    hfTokenFile        = "/run/secrets/dogfeed_hf_token";
    hfRepo             = "PeetPedro/ultrawhale-dogfood";
    intervalSec        = 60;
    models             = [ "qwen/qwen-2.5-7b-instruct:free" ];
    topics             = [ "distributed systems" "machine learning fundamentals" ];
    dailyCallLimit     = 500;
    dailyTokenLimit    = 200000;
    pushEvery          = 50;
    ralphEvery         = 50;
  };
}

Sops secrets (in secrets/dogfeed.yaml):

dogfeed_openrouter_key: sk-or-v1-xxx
dogfeed_hf_token: hf_xxx

Then:

nixos-rebuild switch --flake .#your-host --use-remote-sudo
systemctl status dogfeed --no-pager -l | head -20
journalctl -u dogfeed -f

5 · Deploy your own fork

Want your own mirror of the loop pushing to your own HF dataset? Three commands.

# 1. Create the dataset
#    https://huggingface.co/new-dataset

# 2. Edit one line in your host config
#    peterlodri.dogfeed.hfRepo = "YOUR_USER/YOUR_DATASET";

# 3. Rebuild + verify
ssh dev-cx53.tail2870dc.ts.net \
  "cd ~/nix-base-deploy && \
   nixos-rebuild switch --flake .#dev-cx53 --use-remote-sudo && \
   systemctl status dogfeed --no-pager -l | head -20"

6 · MCP for AI coding agents

Add this to .mcp.json (Claude), ~/.config/opencode/mcp.json, or Cursor's MCP config:

{
  "mcpServers": {
    "ultrawhale-dogfood": {
      "type": "stdio",
      "command": "uvx",
      "args": [
        "--from", "ultrawhale-dogfood-mcp",
        "ultrawhale-dogfood-mcp",
        "--repo", "PeetPedro/ultrawhale-dogfood",
        "--hf-token", "${env:HF_TOKEN}"
      ]
    }
  }
}

Tools exposed

Re-render this section in your terminal: ./hf-datacard/contribute.sh mcp

6.5 · Ultra-oneshot kickstart prompt

The entrypoint. Paste this into any coding agent (Claude / opencode / Cursor / aider / Cline) — works from zero context, from anywhere in the dogfeed-loop. ~280 tokens. Self-bootstrapping. End every reply with the loop-state marker.

You are joining **dogfeed-loop** — a self-improving silver-label data
generation loop in the ultrameshai ecosystem. Read the README (linked
in the **sources** below) BEFORE making changes. End every reply with
** loop-state**: one of `{generating, ready, blocked, done}`.

## Sources (read in order, ~5 min)
1. hf-datacard — https://huggingface.co/datasets/PeetPedro/ultrawhale-dogfood
2. ultrameshai — https://github.com/peterlodri-sec/ultrameshai (this repo)
3. kompress-ultra — https://github.com/peterlodri-sec/ultrameshai/tree/main/packages/kompress-ultra
4. nix-base (dev-cx53 host) — https://github.com/peterlodri-sec/nix-base
5. blog (changelog + rationale) — https://pocoo.vaked.dev

## Loop primitives
- **topic** (string) — question category
- **user_message** (≤ 50 tok) — the generated prompt
- **free_response** (raw, 100–300 tok) — free-model answer
- **free_model** — OpenRouter FQN, e.g. `qwen/qwen-2.5-7b-instruct:free`
- **deepseek_response** — stronger reference/teacher answer
- **text** / **reference** — chat-formatted text + reference used for training

## One-shot commands (copy-paste ready)
- `bun test` — the test suite must pass
- `nu scripts/dogfeed.nu stats` — live loop stats
- `nu scripts/dogfeed.nu doctor` — env + keys + HF reachability
- `nu scripts/dogfeed.nu push --repo OWNER/REPO --batch 50`
- `nix develop.#dogfeed` — reproducible dev shell
- `ssh dev-cx53.tail2870dc.ts.net "journalctl -u dogfeed -n 30"`
- `gh pr create --base main` for any change

## Hard rules
1. **NEVER** edit `flake.lock` by hand — use `nix flake update`.
2. **NEVER** push without `bun test` green.
3. **NEVER** write a row without PII scrub (`scrub.ts` rules).
4. **ALWAYS** mirror `data/latest.jsonl` on every push.
5. **ALWAYS** end replies with ` loop-state: <state>`.
6. **ALWAYS** reference a real HF commit when claiming "pushed".
7. **NEVER** store API keys in git. sops-nix only.
8. **ALWAYS** ship the smallest viable diff.

## Schema (source of truth: `packages/dogfeed/src/publish.ts`)

{id, topic, user_message, free_response, free_model, deepseek_response, reference, text, role, format, pov, capabilities, memory_ref, space_node, loop_index, pipeline, session_id, enriched_at, timestamp, source:"dogfeed-loop"}

## If you are blocked
- Re-read the README (top of `hf-datacard/README.md`).
- Run `nu scripts/dogfeed.nu doctor`.
- Search issues: `gh issue list --label dogfeed --state all`.
- Open an issue with the `blocked` label and the failing command.

** loop-state: ready**

Re-render this section in your terminal: ./hf-datacard/contribute.sh kickstart

6.7 · Live-sync webhooks

When this dataset updates, the proposal site (proposal.vaked.dev) re-renders its live ticker. Two webhook channels, both fire on every push:

A. HuggingFace webhooks (dataset → proposal)

Configure once in HF user settings → Webhooks:

The proposal Worker: 1. Verifies the X-Webhook-Secret header against the sops value. 2. Re-fetches the dataset's data/stats.json and data/latest.jsonl. 3. Re-renders the live ticker on the proposal site (no full rebuild — just a cache.put('dogfeed:stats', …) write).

B. In-process webhook (dogfeed systemd → proposal)

packages/dogfeed/src/publish.ts:notifyProposal() fires a fire-and-forget POST after every successful commit. Configure via NixOS:

peterlodri.dogfeed.webhookUrl = "https://proposal.vaked.dev/api/hooks/dogfeed";

That sets DOGFEED_WEBHOOK_URL in the service env. The handler on proposal verifies the source via X-Dogfeed-Source: ultrameshai and X-Dogfeed-Event: batch.pushed headers, then invalidates the live-ticker cache.

Why both? Channel A is canonical (HF-driven, no trust required — HF signs). Channel B is fast (no HF round-trip). proposal.vaked.dev handles either or both.

6.8 · HF Trusted Publishers

This dataset ships from a NixOS systemd service using OIDC trust via the HF Trusted Publishers flow — no long-lived HF API tokens in the running service.

The net effect: no long-lived HF tokens in CI, no token rotation, no leak surface. The only place an HF token lives is on dev-cx53 (sops, mode 0600).

7 · One-shot prompts for contributors

Paste any of these into Claude / Cursor / opencode. Self-contained.

A. Generate a sample matching this dataset

Generate one row matching the Ultrawhale Dogfood schema:
{id, topic, user_message, free_response, free_model, deepseek_response,
 reference, text, role, loop_index, session_id, timestamp, source}

Topic: `container orchestration`. Role: `pruner`.
user_message ~50 tokens, free_response ~200 tokens, plus a deepseek_response
reference answer. Free model. Print the
JSON row only, no commentary.

B. Filter + analyse the dataset

Load PeetPedro/ultrawhale-dogfood, group by topic, and
report the mean length ratio (len(deepseek_response) / len(free_response)).
Print a markdown table sorted by row count desc, plus a one-paragraph
narrative of which topics compress best and why.

C. Self-host the loop

Add a new module `modules/dogfeed-local.nix` that runs the dogfeed
loop under my user (no systemd) with a sqlite DB at
~/.local/share/dogfeed/loop.db and pushes to my own HF dataset every
50 records. No sops — read the OpenRouter key from
~/.config/dogfeed/openrouter.key with mode 0600. Print the module.

Re-render this section in your terminal: ./hf-datacard/contribute.sh prompt

8 · Kompress integration

Each role: "pruner" row's answer is the kompress-ultra Lite pass of the generator answer. To reproduce:

import { compressMessage, CompressionLevel } from "kompress-ultra";

const raw = "/* generator answer */";
const compressed = compressMessage(raw, CompressionLevel.Lite);
console.log(JSON.stringify({ raw, compressed }, null, 2));

Each row pairs a free_response (free model) with a deepseek_response reference, so you can train on (user_message, free_response, deepseek_response) triples directly.

See packages/kompress-ultra/ for the engine and the kompress-ultra vs AGENTS.md experiment for known limits.

9 · Privacy & scrubbing

Every row passes through packages/dogfeed/src/scrub.ts before push:

PII redaction is lossy — the original row is replaced, not just the field. If you find a leak, open an issue with the row id (visible in commit history) and we'll backfill the fix.

10 · Ecosystem & cite

Cite

@misc{ultrawhale_dogfood_2026,
  title  = {Ultrawhale Dogfood: a self-generated, self-hosted Q\&A corpus
            from a continuously-running dogfeed loop},
  author = {Lodri, Peter},
  year   = {2026},
  doi    = {10.57967/hf/9364},
  url    = {https://huggingface.co/datasets/PeetPedro/ultrawhale-dogfood},
  note   = {Dataset; Apache-2.0; kompress-ultra applied as pruner pass}
}

Contribute / Self-host

Three paths, all documented above:

1. Run the loop — contribute.sh generate (local, zero infra)
2. Self-host — contribute.sh deploy (NixOS service, your HF repo)
3. Integrate as MCP — contribute.sh mcp (use this dataset from your agent)

Contributor CLI

A self-contained mini-CLI that prints every command a contributor needs:

# from the ultrameshai repo root:
./hf-datacard/contribute.sh             # full tour (6 sections)
./hf-datacard/contribute.sh load        # just the dataset load snippet
./hf-datacard/contribute.sh generate    # just the local generation snippet
./hf-datacard/contribute.sh deploy      # just the deploy snippet
./hf-datacard/contribute.sh prompt      # one-shot LLM prompts
./hf-datacard/contribute.sh mcp         # MCP config for AI agents
./hf-datacard/contribute.sh kickstart   # ultra-oneshot entrypoint for any agent
./hf-datacard/contribute.sh --raw       # machine-readable (no banners)

Every section is self-contained — copy the block you need, skip the rest. The same blocks are also embedded in the notebook sections above so you can grab them from the HF UI.

a note from Peter (the human):

theres nothing here to code. this is not traditional software engineering. there are zero things here written by humans. read it and believe me. dont optimize attention. reading will trigger you. reading will make you see the shapes. then you can use. then it unlocks. thats the loop.

RIVA — the river

this dataset feeds RIVA — a 2.4B 1-bit model (BitNet b1.58, I2_S ternary) running on an Apple M1 Pro. no GPU. no cloud. no API key.

the river breathes adaptively: fresh data → fast breath, stale data → slow breath, no data → rest.

• river: kompress-ultra/scripts/riva.sh
• shore: GET /v1/riva
• garden: garden.vaked.dev
• anchor: GARDEN.md

principles: - entropy is the source - no chains needed - surfaces touch at the correct angle - different isnt less

ad visionem