Pustaka adalah korpus teks terbuka berskala besar yang dikembangkan oleh MEA Ecosystem, dimulai dengan fokus penuh pada Bahasa Indonesia.
Rows54,311,748
Configurations1
Size26.1 GB
Licenseother
AccessPublicly accessible
Monthly Downloads22.1k
Dataset Card
Pustaka adalah korpus teks terbuka berskala besar yang dikembangkan oleh MEA Ecosystem, dimulai dengan fokus penuh pada Bahasa Indonesia. Corpus ini disusun dari beberapa sumber publik berkualitas — web crawl yang telah difilter, ensiklopedia, berita, forum, hingga lexicon bahasa gaul — lalu diproses ulang melalui pipeline pembersihan (deteksi bahasa, filter panjang dokumen, deteksi boilerplate/spam, dan deduplikasi) sebelum dirilis. Pustaka dibangun untuk digunakan siapa saja — baik untuk pretraining model bahasa, penelitian NLP, maupun eksperimen pribadi — dan dirilis secara terbuka di bawah Hugging Face. Tabel di atas mencakup corpus-bahasa-indonesia/ saja. Untuk corpus lain (kode…
Excerpt from the card by MEA Ecosystem, licensed other.
Structure
default 54,311,748 rows
| Split | Rows | Size |
|---|---|---|
| train | 54,311,748 | 5.0 GB |
textstring
Details
- Repository
- meaecosystem/pustaka
- Publisher
- MEA Ecosystem
- Task category
- Text generation
- Tags
- indonesian, text-corpus, pretraining
- Size category
- 10M<n<100M
- Languages
- id, en, ja, zh, ko
- Revision
- 98661a69edf51089f43ccf4b7855cfb4304e8130
- Last updated
- 2026-09-18
Files
109 files, 26.1 GB in total.
Data103 files · 26.1 GB
Tokenizer1 file · 10.5 MB
Documentation3 files · 87.9 KB
Other1 file · 23.3 KB
Repository1 file · 2.6 KB
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| _progress/ccnews_hashes.txt.gz | Data | 25.8 MB | 3782dcdbbfb0 |
| _progress/ccnews_meta.json | Data | 150 B | — |
| _progress/code_cpp_hashes.txt.gz | Data | 3.5 MB | 4eb7c7fb1b8d |
| _progress/code_cpp_meta.json | Data | 146 B | — |
| _progress/code_css_hashes.txt.gz | Data | 2.8 MB | a4da66cb5d01 |
| _progress/code_css_meta.json | Data | 146 B | — |
| _progress/code_go_hashes.txt.gz | Data | 8.0 MB | 5c16fe553dc3 |
| _progress/code_go_meta.json | Data | 145 B | — |
| _progress/code_html_hashes.txt.gz | Data | 3.8 MB | 38f6739e449e |
| _progress/code_html_meta.json | Data | 147 B | — |
| _progress/code_java_hashes.txt.gz | Data | 10.3 MB | 5b3cca208766 |
| _progress/code_java_meta.json | Data | 148 B | — |
| _progress/code_javascript_hashes.txt.gz | Data | 5.8 MB | 0565a6263206 |
| _progress/code_javascript_meta.json | Data | 153 B | — |
| _progress/code_python_hashes.txt.gz | Data | 7.2 MB | a842afb0b46c |
| _progress/code_python_meta.json | Data | 149 B | — |
| _progress/code_rust_hashes.txt.gz | Data | 971.2 KB | 4cde15181f36 |
| _progress/code_rust_meta.json | Data | 146 B | — |
| _progress/code_sql_hashes.txt.gz | Data | 1.8 MB | 26ae3c9227cd |
| _progress/code_sql_meta.json | Data | 145 B | — |
| _progress/code_typescript_hashes.txt.gz | Data | 3.5 MB | 25dd2242f322 |
| _progress/code_typescript_meta.json | Data | 153 B | — |
| _progress/culturax_hashes.txt.gz | Data | 220.3 MB | 092d77513f7d |
| _progress/culturax_meta.json | Data | 154 B | — |
| _progress/hukum_japan_hashes.txt.gz | Data | 86.5 KB | 95175dc49399 |
| _progress/hukum_japan_meta.json | Data | 125 B | — |
| _progress/hukum_us_hashes.txt.gz | Data | 337.4 KB | 76cd36d8adac |
| _progress/hukum_us_meta.json | Data | 123 B | — |
| _progress/indo4b_hashes.txt.gz | Data | 176.3 MB | b07374e3c48a |
| _progress/indo4b_meta.json | Data | 151 B | — |
| _progress/instruct_alpaca_id_hashes.txt.gz | Data | 449.2 KB | db4fa7da7143 |
| _progress/instruct_alpaca_id_meta.json | Data | 121 B | — |
| _progress/instruct_dolly_hashes.txt.gz | Data | 146.9 KB | 20551d5f0bbe |
| _progress/instruct_dolly_meta.json | Data | 117 B | — |
| _progress/instruct_oig_hashes.txt.gz | Data | 4.9 MB | 7ac0a5b73559 |
| _progress/instruct_oig_meta.json | Data | 116 B | — |
| _progress/kamus_alay_meta.json | Data | 160 B | — |
| _progress/kaskus_hashes.txt.gz | Data | 313.1 KB | 0b3d657ec5d0 |
| _progress/kaskus_meta.json | Data | 147 B | — |
| _progress/math_gsm8k_hashes.txt.gz | Data | 74.1 KB | d3e32b7e405f |
| _progress/math_gsm8k_meta.json | Data | 116 B | — |
| _progress/math_numinamath_hashes.txt.gz | Data | 8.4 MB | afb037d06d28 |
| _progress/math_numinamath_meta.json | Data | 123 B | — |
| _progress/native_id_hashes.txt.gz | Data | 11.2 MB | 2f3ed1dc87cc |
| _progress/native_id_meta.json | Data | 1.6 MB | — |
| _progress/wikipedia_hashes.txt.gz | Data | 5.1 MB | f9094a743851 |
| _progress/wikipedia_id_hashes.txt.gz | Data | 5.1 MB | 6dc90a3c9faa |
| _progress/wikipedia_id_meta.json | Data | 138 B | — |
| _progress/wikipedia_ja_hashes.txt.gz | Data | 12.7 MB | 726162145e7b |
| _progress/wikipedia_ja_meta.json | Data | 153 B | — |
| _progress/wikipedia_ko_hashes.txt.gz | Data | 4.4 MB | b713320d08ba |
| _progress/wikipedia_ko_meta.json | Data | 151 B | — |
| _progress/wikipedia_meta.json | Data | 151 B | — |
| _progress/wikipedia_zh_hashes.txt.gz | Data | 7.5 MB | 805e46d29e1c |
| _progress/wikipedia_zh_meta.json | Data | 151 B | — |
| corpus-bahasa-indonesia-native/_inprogress/native_id_shard_00005.inprogress.txt.gz | Data | 423.5 MB | 4b8bfbf7efd5 |
| corpus-bahasa-indonesia-native/native_id_shard_00001.txt.gz | Data | 685.7 MB | 5994ea1f5e84 |
| corpus-bahasa-indonesia-native/native_id_shard_00002.txt.gz | Data | 683.4 MB | a9c853b2ff34 |
| corpus-bahasa-indonesia-native/native_id_shard_00003.txt.gz | Data | 678.2 MB | 1ca8cc37c2c8 |
| corpus-bahasa-indonesia-native/native_id_shard_00004.txt.gz | Data | 678.3 MB | 9468a897b843 |
| corpus-bahasa-indonesia/ccnews/ccnews_shard_00001.txt.gz | Data | 1.8 GB | 07f371741641 |
| corpus-bahasa-indonesia/culturax/culturax_shard_00003.txt.gz | Data | 3.0 GB | 1c16e9c113e6 |
| corpus-bahasa-indonesia/culturax/culturax_shard_00004.txt.gz | Data | 3.0 GB | 96593555862a |
| corpus-bahasa-indonesia/culturax/culturax_shard_00006.txt.gz | Data | 1.3 GB | dab23551863c |
| corpus-bahasa-indonesia/indo4b/indo4b_shard_00001.txt.gz | Data | 676 B | 1578aec52ed4 |
| corpus-bahasa-indonesia/indo4b/indo4b_shard_00002.txt.gz | Data | 1.8 GB | 4ec045cc9cc5 |
| corpus-bahasa-indonesia/kamus_alay/kamus_alay.txt.gz | Data | 53.6 KB | eec3f5e72700 |
| corpus-bahasa-indonesia/kaskus/kaskus_shard_00001.txt.gz | Data | 16.5 KB | 18f7a0a671df |
| corpus-bahasa-indonesia/kaskus/kaskus_shard_00002.txt.gz | Data | 38.1 MB | f8c11d1e36bc |
| corpus-bahasa-indonesia/wikipedia/wikipedia_shard_00001.txt.gz | Data | 8.5 KB | 759b84e54fd6 |
| corpus-bahasa-indonesia/wikipedia/wikipedia_shard_00002.txt.gz | Data | 314.1 MB | c4a7af304d57 |
| corpus-code/cpp/code_cpp_shard_00001.txt.gz | Data | 358.4 MB | 033ab0373682 |
| corpus-code/cpp/code_cpp_shard_00002.txt.gz | Data | 416.6 MB | 944d2bd00b5e |
| corpus-code/css/code_css_shard_00001.txt.gz | Data | 7.9 KB | e6911e412643 |
| corpus-code/css/code_css_shard_00002.txt.gz | Data | 438.7 MB | 3d6f108a4b05 |
| corpus-code/go/code_go_shard_00001.txt.gz | Data | 1.4 GB | df6da89fae1d |
| corpus-code/html/code_html_shard_00001.txt.gz | Data | 630.4 MB | 1c7971c15f24 |
| corpus-code/html/code_html_shard_00002.txt.gz | Data | 142.0 MB | fc47e44d7eee |
| corpus-code/java/code_java_shard_00001.txt.gz | Data | 958.4 MB | 907b7be280af |
| corpus-code/java/code_java_shard_00002.txt.gz | Data | 277.2 MB | 679b9bd1828f |
| corpus-code/javascript/code_javascript_shard_00001.txt.gz | Data | 620.3 MB | 4fff9aca5cca |
| corpus-code/javascript/code_javascript_shard_00002.txt.gz | Data | 274.6 MB | f6d4f2a68418 |
| corpus-code/python/code_python_shard_00001.txt.gz | Data | 27.8 KB | 3c1b5b1074fe |
| corpus-code/python/code_python_shard_00002.txt.gz | Data | 580.9 MB | a14b094ae66a |
| corpus-code/python/code_python_shard_00003.txt.gz | Data | 680.1 MB | 7d94e2cbaef7 |
| corpus-code/rust/code_rust_shard_00001.txt.gz | Data | 250.9 MB | 76870ac563f4 |
| corpus-code/sql/code_sql_shard_00001.txt.gz | Data | 177.3 MB | 92f62f6ddc32 |
| corpus-code/typescript/code_typescript_shard_00001.txt.gz | Data | 198.7 MB | f82f78c70a83 |
| corpus-code/typescript/code_typescript_shard_00002.txt.gz | Data | 228.7 MB | ae8d5afd7f09 |
| corpus-hukum/japan/hukum_japan_shard_00001.jsonl.gz | Data | 53.9 KB | 968885e2ca1d |
| corpus-hukum/japan/hukum_japan_shard_00002.jsonl.gz | Data | 70.7 MB | 7977d377314b |
| corpus-hukum/us/hukum_us_shard_00001.jsonl.gz | Data | 12.8 KB | 6fd619669fae |
| corpus-hukum/us/hukum_us_shard_00002.jsonl.gz | Data | 37.0 MB | cd0a49d0a192 |
| corpus-instruct/alpaca_id/instruct_alpaca_id_shard_00001.jsonl.gz | Data | 11.0 MB | 7230f478c35b |
| corpus-instruct/dolly/instruct_dolly_shard_00001.jsonl.gz | Data | 5.6 KB | 9a5ca2c76aba |
| corpus-instruct/dolly/instruct_dolly_shard_00002.jsonl.gz | Data | 4.6 MB | 518fa6cd5a5f |
| corpus-instruct/oig/instruct_oig_shard_00001.jsonl.gz | Data | 218.5 MB | 97c349d1895a |
| corpus-math/gsm8k/math_gsm8k_shard_00001.jsonl.gz | Data | 2.4 KB | 63bf1224055f |
| corpus-math/gsm8k/math_gsm8k_shard_00002.jsonl.gz | Data | 1.4 MB | f6f897f5c492 |
| corpus-math/numinamath/math_numinamath_shard_00001.jsonl.gz | Data | 370.5 MB | 531214364708 |
| wikipedia/ja/wikipedia_ja_shard_00002.txt.gz | Data | 1.3 GB | 29beb724bfcb |
| wikipedia/ko/wikipedia_ko_shard_00001.txt.gz | Data | 443.7 MB | 53ee97d9d0aa |
| wikipedia/zh/wikipedia_zh_shard_00001.txt.gz | Data | 1.0 GB | d6a98eb3fd16 |
| LICENSE.md | Documentation | 4.6 KB | — |
| MEA-LICENSE.md | Documentation | 3.0 KB | — |
| README.md | Documentation | 80.3 KB | — |
| assets/pustaka_banner.svg | Other | 23.3 KB | — |
| .gitattributes | Repository | 2.6 KB | — |
| tokenizer/tokenizer.json | Tokenizer | 10.5 MB | 26125a50e1de |
License and Download
- License
- other
- Access
- No access gate
Download from MEA Ecosystem
Released by MEA Ecosystem through its official repository on Hugging Face. Read the license.