looked-up

Looked Up: hourly Wikipedia attention across 30 languages

What the world pays attention to, hour by hour, across languages. This dataset holds the hourly pageviews of every Wikipedia article in the 30 most-read language editions. It is refreshed every hour from Wikimedia's public pageview dumps, with desktop and mobile views kept apart and every title mappable to a Wikidata item.

Built and maintained by github.com/Sbaiii/looked-up, where the code, the data model and all design decisions (ADRs) live.

Quick start (DuckDB)

import duckdb
con = duckdb.connect(); con.execute("INSTALL httpfs; LOAD httpfs;")
lake = "hf://datasets/Sbaiiiiii/looked-up/data/hourly/*/*/*.parquet"
con.sql(f"""SELECT title, sum(views_desktop + views_mobile) AS views FROM read_parquet('{lake}')
            WHERE lang = 'fr' AND ts_hour_start >= (now() AT TIME ZONE 'UTC') - INTERVAL 24 HOUR GROUP BY 1 ORDER BY 2 DESC LIMIT 20""").show()

One day is one file, so reading a single day is fastest: read_parquet('hf://datasets/Sbaiiiiii/looked-up/data/hourly/year=2026/month=10/day=06.parquet').

Layout

data/hourly/year=YYYY/month=MM/day=DD.parquet   all ingested hours of one UTC day
data/wikidata/sitelinks.parquet                 (lang, title) -> Wikidata QID
data/manifest.json                              every hour present, with row count, source and ingestion time
data/app/{today,stats}.json, data/app/days/*.json   small JSON for the web app (+ .json.gz twins)
data/live/bursts/*.jsonl, data/live/daily/*      daily snapshots of the live editing layer (counts only, no users)

Schema

data/hourly/…/day=DD.parquet, with rows sorted by (lang, title, ts_hour_start), zstd level 9:

data/wikidata/sitelinks.parquet has lang VARCHAR, title VARCHAR and qid INTEGER (597 means Q597). Join on (lang, title). About 98.6 % of retained views match a QID.

What is kept (retention)

A row is kept when the title is an article and views_desktop + views_mobile >= 5 in that hour. Excluded are the main page, namespace pages (Special:, Talk:, File:, User:, Category:, Template:, Help:, Portal:, Draft:, Wikipedia:, … in each wiki's own language, taken from its siteinfo) and the - title.

A missing row means fewer than 5 views that hour, not zero. Only user traffic is included: Wikimedia already removes spiders and traffic it classifies as automated.

Why desktop and mobile are separate columns

Humans read Wikipedia mostly on phones (≈ 56–90 % mobile, depending on the language). Bot traffic that passes Wikimedia's filters is almost all desktop. The mobile share is therefore a strong human-vs-bot signal. For example, fr:Cookie_(informatique) gets tens of thousands of views per hour at 0.6 % mobile, while the human articles around it are 60–93 % mobile. Sum the two columns for total attention, or compare them to filter automation.

Languages

The 30 Wikipedias with the most human article views over 1–7 Oct 2026. The ranking excludes namespace pages, main pages and titles with > 1,000 views and < 5 % mobile. config/languages.yml has the full ranking of 60.

Attention events (derived)

Every hour, new hours are also scored for attention events: one entity spiking in at least 2 languages within 6 hours, measured against its own 28-day baseline. Each event has a tier: noticed (≥ 2 languages), international (≥ 5) or planetary (≥ 20). The evaluated Phase 2 definition used ≥ 3 languages. The spike files, event files and data/latest.json (last 24 h, top 50 multi-language events plus a separate single_language_events list, labels in the 30 languages) are described in the data model. The definitions were pre-registered and evaluated honestly; the Phase 2 results report what the detector does and doesn't capture, and the Phase 2b results test it against deaths, earthquakes and matches. - Single-language events. An event whose lead language holds ≥ 95 % of its excess views has event_class = single_language. Such events are kept, not filtered: they mean "attention from one community", not "false". No topic is blocklisted.

The web app sbaiii.github.io/looked-up reads data/app/ directly from this dataset. The schema is described in the data model.

The live editing layer (what editors across the 30 Wikipedias are updating right now) is served by a Cloudflare Worker. This dataset keeps one daily snapshot of its bursts for research (ADR 0032). Only counts are kept, never user names.

Update frequency and provenance

• Hourly. A GitHub Actions job runs at :17 every hour (with an external trigger as backup) and ingests every published hour of the last 7 days that is still missing, up to 12 per run. Wikimedia publishes each hourly dump ≈ 2 h 15 min (median) after the hour ends, so the newest hour is usually 2–3 h old.
• Repository history is squashed monthly to keep storage bounded. Pin a file by its path and date, not by a commit hash: old revisions disappear after each squash. Current data is never affected.
• History is backfilled from Wikimedia's daily pageview_complete files (source = pageview_complete in the manifest). Live hours come from the hourly dumps (source = hourly_dump).
• Known caveat: for the same hour, the two sources agree on ≈ 99.999 % of rows. pageview_complete misses a few mobile views, mostly on pages that do not exist or are redirects, worth about 0.0003 % of views.

Source and license

Derived from the Wikimedia pageview dumps and pageview_complete, and from Wikidata's wb_items_per_site table. All are released under CC0 by the Wikimedia Foundation. This dataset is also CC0.

Titles are kept exactly as Wikimedia reports them: no redirect resolution and no moderation of article names.