greek-cc

Greek Common Crawl

A FineWeb-style Greek-language text dataset extracted from Common Crawl, following the FineWeb-2 recipe adapted for Greek (ell_Grek).

Pipeline source: github.com/alexliap/greek-cc.

Crawl coverage starts at CC-MAIN-2024-22 rather than Common Crawl's earliest snapshots: this project picks up right where the FineWeb-2 dataset's own Greek (ell_Grek) subset leaves off (2013 through April 2024), so it extends FineWeb-2's Greek coverage forward instead of re-extracting and re-deduping ground FineWeb-2 already covers.

Dataset structure

Each processed crawl (e.g. CC-MAIN-2024-22) contributes two kinds of files, exposed as named configs (load_dataset("alexliap/greek-cc", name="CC-MAIN-2024-22"), etc. — see the viewer's dataset-config dropdown, same convention FineWeb uses):

• raw/<crawl>/chunk=<offset>/*.parquet (config raw, all crawls pooled together): per-chunk survivors, written as soon as each ~100k-row manifest chunk finishes extraction and filtering. These are pre-deduplication: near-duplicate detection only runs once a whole crawl is fully processed, so documents here may have near-duplicates elsewhere in the same crawl.
• <crawl>/*.parquet (one config per crawl, e.g. CC-MAIN-2024-22, plus a default config that pools every finished crawl's final output together): the final, whole-crawl output, written once every chunk is done. Near-duplicates have been resolved down to one surviving document per cluster (see below).

New crawls need their own config entry added to this card's configs once they finish — it isn't inferred automatically from the file layout.

Columns

* only present in the final <crawl>/*.parquet files, not in raw/.

Near-duplicates are removed. Documents that MinHash considers near-duplicates of each other share a minhash_cluster_id; only one representative per cluster survives into the final <crawl>/*.parquet files, and minhash_cluster_size tells you how many documents that cluster had, i.e. how many were dropped alongside the survivor. If you want the removed documents, anti-join raw/<crawl>/chunk=<offset>/ (pre-dedup) against <crawl>/ (post-dedup) on (warc_filename, warc_record_offset).

Loading

from datasets import load_dataset

# one crawl's final, deduplicated output
ds = load_dataset("alexliap/greek-cc", name="CC-MAIN-2024-22", split="train")

# every finished crawl's final output, pooled together
ds = load_dataset("alexliap/greek-cc", split="train")  # name="default"

# all crawls' pre-dedup raw chunks, pooled together
raw = load_dataset("alexliap/greek-cc", name="raw", split="train")

Each finished crawl is its own config (name=), matching how FineWeb exposes its own monthly dumps — see the config dropdown in the dataset viewer above for the full list.

How it was built

1. Manifest: Common Crawl's columnar index is scanned for Greek-flagged records, producing a Parquet manifest of (warc_filename, offset, length) per candidate record.
2. Fetch and extract: each manifest row is range-fetched from the crawl's WARC files on S3 and run through Trafilatura for boilerplate removal.
3. Filter, in order: - URL filter (blocklists) - Language ID (GlotLID), kept only if predicted language is Greek (ell) above a tuned confidence threshold - Gopher repetition filter - FineWeb quality filter - Gopher quality filter - all thresholds are FineWeb-2's own Greek (ell_Grek) config, used as-is rather than hand-tuned
4. Clean: FTFY (mojibake/encoding fixes), PII redaction, stray symbol line removal.
5. Deduplicate, per-snapshot rather than per-language-global. MinHash (14 buckets, 8 hashes per bucket, 5-grams, xxhash 64-bit, the same hashing scheme FineWeb-2 uses, run through datatrove's own signature/buckets/cluster/filter pipeline), but scoped to one crawl at a time instead of across every crawl. FineWeb-2 dedups Greek globally across its full 2013 to April 2024 span, then has to upsample surviving documents by cluster size to claw back the volume that global dedup throws away. The original FineWeb (v1) found the opposite worked better: per-snapshot dedup kept far more tokens (20T vs 4T in their ablation) and avoided the quality drop global dedup caused, where what survives skews toward ads and keyword-stuffed text. This project follows FineWeb v1's per-snapshot scope for that reason, and removes near-duplicates outright, matching both FineWeb v1 and FineWeb-2. Pre-deduplication data stays available under raw/, in case you want to redo deduplication globally the FineWeb-2 way, or reconstruct exactly which documents were dropped (see above).

Source and license

Licensed under ODC-By 1.0, matching FineWeb-2's ell_Grek. Built from Common Crawl WARC data, so Common Crawl's own Terms of Use also apply to the underlying web content.

References

• FineWeb-2 dataset, whose Greek (ell_Grek) subset this project picks up after
• Penedo et al., The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale (arXiv 2406.17557)
• Penedo et al., FineWeb2: One Pipeline to Scale Them All (arXiv 2506.20920)
• datatrove, the processing library both FineWeb papers and this pipeline are built on

Status

Crawls are processed and uploaded incrementally, oldest to newest.

• [x] CC-MAIN-2024-22
• [x] CC-MAIN-2024-26
• [x] CC-MAIN-2024-30
• [x] CC-MAIN-2024-33
• [ ] CC-MAIN-2024-38 (in progress)
• [ ] CC-MAIN-2024-42 (in progress)
• [ ] CC-MAIN-2024-46 (in progress)
• [ ] CC-MAIN-2024-51
• [ ] CC-MAIN-2025-05
• [ ] CC-MAIN-2025-08
• [ ] CC-MAIN-2025-13
• [ ] CC-MAIN-2025-18
• [ ] CC-MAIN-2025-21
• [ ] CC-MAIN-2025-26
• [ ] CC-MAIN-2025-30
• [ ] CC-MAIN-2025-33
• [ ] CC-MAIN-2025-38
• [ ] CC-MAIN-2025-43
• [ ] CC-MAIN-2025-47
• [ ] CC-MAIN-2025-51
• [ ] CC-MAIN-2026-04
• [ ] CC-MAIN-2026-08
• [ ] CC-MAIN-2026-12
• [ ] CC-MAIN-2026-17
• [ ] CC-MAIN-2026-21
• [ ] CC-MAIN-2026-25
• [ ] CC-MAIN-2026-30