FineWeb-HQ is a high-quality, model-filtered pretraining dataset derived as a subset of FineWeb. FineWeb-HQ was created by selecting the top 10% of FineWeb documents based on a deep learning classifier trained to identify structured and knowledge-rich samples. This classifier uses XLM-RoBERTa embeddings to score documents. To validate our approach, we pretrained 1B-parameter LLM models with a Llama-like architecture across multiple languages and scripts. The results showed improvements on standard English benchmarks, with our dataset outperforming its English counterparts DCLM and FineWeb-Edu. For its multilingual version, FineWeb2-HQ, evaluations on CMMLU (Chinese), MMLU (German), and MMLU…