german-newspaper-pages

German Newspaper Pages (Public Domain, 1671–1964)

7,000,060 newspaper page scans from 650 German newspapers, each shipped with the layout boxes and OCR transcription produced for this corpus, and — for 38 % of pages — the digitising library's own ALTO XML.

Every page here comes from an issue whose own rights statement is CC Public Domain Mark 1.0 or CC0 1.0. Sources were harvested from the Deutsche Digitale Bibliothek newspaper portal.

Contents of a sample

Shards are data/pages-NNNNN.tar, ordered chronologically; an issue is never split across two shards. Each page is one WebDataset sample, key <date>_<issuehash>_<pageid>:

import webdataset as wds

ds = wds.WebDataset("data/pages-{00000..02522}.tar").decode("pil")
for s in ds:
    img, meta = s["jpg"], s["json"]
    for r in meta["ocr"]["regions"]:
        x1, y1, x2, y2 = r["bbox"]      # pixels of THIS jpg
        print(r["label"], r["text"][:60])

Coordinate systems — read this before drawing boxes

• .json boxes are in the coordinates of the .jpg in the same sample. They were produced on the full-resolution scan and rescaled by the same factor the image was, so they need no adjustment.
• .alto.xml is verbatim from the library and keeps the ORIGINAL full-resolution coordinates. Its own <Page WIDTH=… HEIGHT=…> header states them, and they equal image.original_width/original_height in the JSON. To overlay ALTO on the shipped JPEG, multiply by image.scale_from_original.

.json schema

{
  "key": "1883-02-14_HR6PLRK…_p0001",
  "issue_id": "1883-02-14_HR6PLRK…", "page_id": "p0001", "page_number": 1,
  "date": "1883-02-14", "year": 1883,
  "zdb_id": "…", "paper_title": "Kölner Nachrichten", "place": "Köln",
  "language": "ger", "provider_ddb_id": "…",
  "license": "http://creativecommons.org/publicdomain/mark/1.0/",
  "ddb_url": "https://www.deutsche-digitale-bibliothek.de/newspaper/item/…?issuepage=1",
  "image": {"width": 1799, "height": 2400,
            "original_width": 4905, "original_height": 6543,
            "scale_from_original": 0.366769},
  "ocr": {
    "layout_model": "PP-DocLayout_plus-L", "text_model": "churro-3B",
    "bbox_space": "image",
    "regions": [
      {"order": 0, "label": "text", "bbox": [820.4, 694.5, 1163.5, 1023.2],
       "score": 0.93, "transcribed": true, "text": "…", "finish_reason": "stop"}
    ]
  },
  "alto": {"present": true, "coordinate_space": "original"}
}

Region labels: text, paragraph_title, image, header, table, doc_title, footer, number, figure_title, aside_text, footnote, seal, abstract, chart, formula, reference, content, and a few rarer ones.

How it was made

1. Harvest — issues fetched from the DDB API, one directory per issue, with the page images, the library ALTO when the API offered one, and a manifest recording the per-issue rights statement at download time.
2. Layout — PP-DocLayout_plus-L on the full-resolution scan; boxes merged so a region contained in another keeps the outer box.
3. Transcription — each region cropped and read by churro-3B (a VLM fine-tuned for historical print). Regions labelled as pictorial, and crops under 8 px, are boxed but not transcribed. The model's XML wrapper is stripped; one line of text per printed line.
4. Packaging — public-domain issues only, images resized to 2400 px long edge.

Limitations — please read

• Not full resolution. Originals are ~2× larger (median long edge 4,450 px). 2400 px is a working resolution for layout and document-understanding work; for fine Fraktur re-transcription, fetch the original from the ddb_url. The release would be 22 TB at original size.
• OCR is machine output, not ground truth. The transcription model loops or truncates on a minority of regions (~0.9 % of regions, but touching ~16 % of pages in a 10k-page audit). finish_reason is "length" where a region hit the token cap and "repetition" where the loop detector fired — filter on it if clean text matters.
• Layout boxes are detections. One printed table or article may fall into several boxes, and classified-ad columns are sometimes read as tables. On a hand-audited quality-filtered sample the table label was 97 % precise; other labels are not separately audited.
• aside_text is effectively never detected (recall ~0 in a small annotation study).
• ALTO is only on 38 % of pages, is the library's own OCR (usually much older and weaker than the transcription here), and its quality varies by institution. A crawler bug deleted a further ~2.3 M valid ALTO files (they began with a UTF-8 BOM and failed an is-xml check); the bug is fixed but those files were not re-fetched.
• The corpus is shaped by what is digitised and public domain, not by what was printed. Coverage is heavily west-German and Saxon; Berlin is almost absent before 1871; the 1700–1814 material is about two thirds Hamburg. Do not read page counts per year as publication volume. The handful of post-1950 pages are rights-statement outliers, not coverage.
• Titles drift. 650 ZDB ids carry 1,930 distinct title strings, because papers were renamed; join on zdb_id, not on paper_title.

Content warning

The corpus includes the German press of 1933–1945 (≈ 260,000 pages), which carries National Socialist propaganda, antisemitic agitation, denaturalisation lists and other material naming persecuted individuals. It also contains the everyday racism, colonial and antisemitic language of the 18th and 19th century press. It is published for historical research. Anyone training generative models on it should expect the material to be reproduced.

Licence

Every issue carries CC Public Domain Mark 1.0 (6,971,082 pages) or CC0 1.0 (28,978 pages), taken from that issue's own rights statement in the DDB, never from a newspaper-level default. The layout boxes and transcriptions added here are released under CC0 1.0.

Please credit the holding institution — provider_ddb_id and ddb_url identify it for each page.

Related

• ai-historian/german-newspaper-tables — 1,044,813 table crops cut from these same layout boxes.

Citation

@misc{german_newspaper_pages_2026,
  title  = {German Newspaper Pages (Public Domain, 1671--1964)},
  author = {Hufe, Lorenz},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/datasets/ai-historian/german-newspaper-pages}}
}