tinyDTVi-dataset

TinyDTVi Pretraining Dataset

This is the official pretraining dataset for the TinyDTVi-500M causal language model. The corpus is aggressively filtered and deduplicated to maximize the representation of standard Vietnamese text for training resource-constrained models.

Dataset Composition

The initial raw dataset (~135GB) was curated by mixing three high-quality open-source Vietnamese corpora. The approximate mixture ratio is: - 50% CulturaX-Vi (Cleaned web crawl) - 30% VNews (Vietnamese News Corpus) - 20% Vietnamese Wikipedia (Encyclopedia)

After strict preprocessing and deduplication, the final dataset size is approximately 85GB (yielding roughly 18 billion unique tokens).

Data Processing Pipeline

To compress the vocabulary down to 50k tokens and focus strictly on standard Vietnamese semantics, we applied a highly aggressive filtering pipeline: 1. Unicode Normalization: Forced all text into Unicode Normalization Form C (NFC) to unify decomposed Vietnamese diacritics. 2. Artifact Removal: Stripped specific domain artifacts like Wikipedia citation brackets (e.g., [1], [citation needed]). 3. Regex-based Whitelisting: Retained only standard Vietnamese and English alphabetical characters, digits, basic punctuation, and fundamental mathematical operators. All foreign scripts (Chinese, Korean, Arabic, etc.), emojis, and complex markdown structures were discarded at the character level. 4. Whitespace Normalization: All newline characters (\n) and redundant spaces were collapsed into a single space to maintain a dense, continuous token stream for training efficiency. 5. Deduplication: Applied document-level exact-match deduplication (via global hashing) across all three data sources to prevent memorization.

Data Sources & Acknowledgements

This corpus is built upon the incredible open-source efforts of the following datasets. We strictly adhere to their respective licenses (mostly CC-BY and MIT) for non-commercial research purposes: - CulturaX: uonlp/CulturaX - VNews (BinhVQ News): vietgpt/binhvq_news_vi - Vietnamese Wikipedia: wikimedia/wikipedia (20231101.vi)

Limitations

Because this dataset has been stripped of newlines and structural formatting (like JSON, Markdown, or Code blocks), it is not suitable for training models on coding tasks or structured data generation. Trace amounts of PII (Personally Identifiable Information) may also remain due to the inclusion of news and web-crawled data.

Citation

If you use this dataset, please remember to cite the original authors of the constituent datasets:

CulturaX:

@article{nguyen2023culturax,
  title={CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages},
  author={Nguyen, Thuat and Chuang, Chien-Sheng and others},
  journal={arXiv preprint arXiv:2309.09400},
  year={2023}
}

VNews:

@misc{binhvq2024vnews,
  author = {BinhVQ},
  title = {Vietnamese News Corpus (binhvq_news_vi)},
  year = {2024},
  publisher = {Hugging Face},
  url = {https://huggingface.co/datasets/vietgpt/binhvq_news_vi}
}

Wikipedia:

@misc{wikimedia2023wikipedia,
  author = {Wikimedia Foundation},
  title = {Wikimedia Downloads},
  year = {2023},
  url = {https://dumps.wikimedia.org}
}