Pustaka adalah korpus teks terbuka berskala besar yang dikembangkan oleh MEA Ecosystem, dimulai dengan fokus penuh pada Bahasa Indonesia.
Rows—
Configurations—
Size35.4 GB
Licenseother
AccessPublicly accessible
Monthly Downloads95.4k
Dataset Card
Pustaka adalah korpus teks terbuka berskala besar yang dikembangkan oleh MEA Ecosystem, dimulai dengan fokus penuh pada Bahasa Indonesia. Corpus ini disusun dari beberapa sumber publik berkualitas — web crawl yang telah difilter, ensiklopedia, berita, forum, hingga lexicon bahasa gaul — lalu diproses ulang melalui pipeline pembersihan (deteksi bahasa, filter panjang dokumen, deteksi boilerplate/spam, dan deduplikasi) sebelum dirilis. Pustaka dibangun untuk digunakan siapa saja — baik untuk pretraining model bahasa, penelitian NLP, maupun eksperimen pribadi — dan dirilis secara terbuka di bawah Hugging Face. Tabel di atas mencakup corpus-bahasa-indonesia/ saja. Untuk corpus lain (kode…
Excerpt from the card by MEA Ecosystem, licensed other.
Details
- Repository
- meaecosystem/pustaka
- Publisher
- MEA Ecosystem
- Task category
- Text generation
- Tags
- indonesian, text-corpus, pretraining
- Size category
- 10M<n<100M
- Languages
- id, en, ja, zh, ko
- Revision
- 36dc37aa4662b0d9536c95920829644761ddbc69
- Last updated
- 2026-10-15
Files
123 files, 35.4 GB in total.
Data117 files · 35.4 GB
Tokenizer1 file · 10.5 MB
Documentation3 files · 87.9 KB
Other1 file · 23.3 KB
Repository1 file · 2.6 KB
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| _progress/ccnews_hashes.txt.gz | Data | 25.8 MB | 3782dcdbbfb0 |
| _progress/ccnews_meta.json | Data | 150 B | — |
| _progress/code_cpp_hashes.txt.gz | Data | 3.5 MB | 4eb7c7fb1b8d |
| _progress/code_cpp_meta.json | Data | 146 B | — |
| _progress/code_css_hashes.txt.gz | Data | 2.8 MB | a4da66cb5d01 |
| _progress/code_css_meta.json | Data | 146 B | — |
| _progress/code_go_hashes.txt.gz | Data | 8.0 MB | 5c16fe553dc3 |
| _progress/code_go_meta.json | Data | 145 B | — |
| _progress/code_html_hashes.txt.gz | Data | 3.8 MB | 38f6739e449e |
| _progress/code_html_meta.json | Data | 147 B | — |
| _progress/code_java_hashes.txt.gz | Data | 10.3 MB | 5b3cca208766 |
| _progress/code_java_meta.json | Data | 148 B | — |
| _progress/code_javascript_hashes.txt.gz | Data | 5.8 MB | 0565a6263206 |
| _progress/code_javascript_meta.json | Data | 153 B | — |
| _progress/code_python_hashes.txt.gz | Data | 7.2 MB | a842afb0b46c |
| _progress/code_python_meta.json | Data | 149 B | — |
| _progress/code_rust_hashes.txt.gz | Data | 971.2 KB | 4cde15181f36 |
| _progress/code_rust_meta.json | Data | 146 B | — |
| _progress/code_sql_hashes.txt.gz | Data | 1.8 MB | 26ae3c9227cd |
| _progress/code_sql_meta.json | Data | 145 B | — |
| _progress/code_typescript_hashes.txt.gz | Data | 3.5 MB | 25dd2242f322 |
| _progress/code_typescript_meta.json | Data | 153 B | — |
| _progress/culturax_hashes.txt.gz | Data | 220.3 MB | 092d77513f7d |
| _progress/culturax_meta.json | Data | 154 B | — |
| _progress/hukum_japan_hashes.txt.gz | Data | 86.5 KB | 95175dc49399 |
| _progress/hukum_japan_meta.json | Data | 125 B | — |
| _progress/hukum_us_hashes.txt.gz | Data | 337.4 KB | 76cd36d8adac |
| _progress/hukum_us_meta.json | Data | 123 B | — |
| _progress/indo4b_hashes.txt.gz | Data | 176.3 MB | b07374e3c48a |
| _progress/indo4b_meta.json | Data | 151 B | — |
| _progress/instruct_alpaca_id_hashes.txt.gz | Data | 449.2 KB | db4fa7da7143 |
| _progress/instruct_alpaca_id_meta.json | Data | 121 B | — |
| _progress/instruct_dolly_hashes.txt.gz | Data | 146.9 KB | 20551d5f0bbe |
| _progress/instruct_dolly_meta.json | Data | 117 B | — |
| _progress/instruct_oig_hashes.txt.gz | Data | 4.9 MB | 7ac0a5b73559 |
| _progress/instruct_oig_meta.json | Data | 116 B | — |
| _progress/kamus_alay_meta.json | Data | 160 B | — |
| _progress/kaskus_hashes.txt.gz | Data | 313.1 KB | 0b3d657ec5d0 |
| _progress/kaskus_meta.json | Data | 147 B | — |
| _progress/math_gsm8k_hashes.txt.gz | Data | 74.1 KB | d3e32b7e405f |
| _progress/math_gsm8k_meta.json | Data | 116 B | — |
| _progress/math_numinamath_hashes.txt.gz | Data | 8.4 MB | afb037d06d28 |
| _progress/math_numinamath_meta.json | Data | 123 B | — |
| _progress/native_id_hashes.txt.gz | Data | 44.6 MB | 97551300e688 |
| _progress/native_id_meta.json | Data | 6.8 MB | — |
| _progress/wikipedia_hashes.txt.gz | Data | 5.1 MB | f9094a743851 |
| _progress/wikipedia_id_hashes.txt.gz | Data | 5.1 MB | 6dc90a3c9faa |
| _progress/wikipedia_id_meta.json | Data | 138 B | — |
| _progress/wikipedia_ja_hashes.txt.gz | Data | 12.7 MB | 726162145e7b |
| _progress/wikipedia_ja_meta.json | Data | 153 B | — |
| _progress/wikipedia_ko_hashes.txt.gz | Data | 4.4 MB | b713320d08ba |
| _progress/wikipedia_ko_meta.json | Data | 151 B | — |
| _progress/wikipedia_meta.json | Data | 151 B | — |
| _progress/wikipedia_zh_hashes.txt.gz | Data | 7.5 MB | 805e46d29e1c |
| _progress/wikipedia_zh_meta.json | Data | 151 B | — |
| corpus-bahasa-indonesia-native/_inprogress/native_id_shard_00019.inprogress.txt.gz | Data | 182.7 MB | 9aaa77517b53 |
| corpus-bahasa-indonesia-native/native_id_shard_00001.txt.gz | Data | 685.7 MB | 5994ea1f5e84 |
| corpus-bahasa-indonesia-native/native_id_shard_00002.txt.gz | Data | 683.4 MB | a9c853b2ff34 |
| corpus-bahasa-indonesia-native/native_id_shard_00003.txt.gz | Data | 678.2 MB | 1ca8cc37c2c8 |
| corpus-bahasa-indonesia-native/native_id_shard_00004.txt.gz | Data | 678.3 MB | 9468a897b843 |
| corpus-bahasa-indonesia-native/native_id_shard_00005.txt.gz | Data | 671.1 MB | 77380bc9150d |
| corpus-bahasa-indonesia-native/native_id_shard_00006.txt.gz | Data | 684.2 MB | c57dc5997ccc |
| corpus-bahasa-indonesia-native/native_id_shard_00007.txt.gz | Data | 683.5 MB | deafe6b0f294 |
| corpus-bahasa-indonesia-native/native_id_shard_00008.txt.gz | Data | 691.6 MB | 9df9295c704c |
| corpus-bahasa-indonesia-native/native_id_shard_00009.txt.gz | Data | 680.3 MB | 2cde230d6f75 |
| corpus-bahasa-indonesia-native/native_id_shard_00010.txt.gz | Data | 675.2 MB | eb5845a61ed1 |
| corpus-bahasa-indonesia-native/native_id_shard_00011.txt.gz | Data | 674.1 MB | 84082494dfcb |
| corpus-bahasa-indonesia-native/native_id_shard_00012.txt.gz | Data | 684.8 MB | 0992addde6d2 |
| corpus-bahasa-indonesia-native/native_id_shard_00013.txt.gz | Data | 672.5 MB | ff5990e0d54b |
| corpus-bahasa-indonesia-native/native_id_shard_00014.txt.gz | Data | 672.0 MB | d2ab05ea9312 |
| corpus-bahasa-indonesia-native/native_id_shard_00015.txt.gz | Data | 682.0 MB | e74b2422d97a |
| corpus-bahasa-indonesia-native/native_id_shard_00016.txt.gz | Data | 680.3 MB | e00fe38cf65b |
| corpus-bahasa-indonesia-native/native_id_shard_00017.txt.gz | Data | 686.1 MB | c2eeae71b292 |
| corpus-bahasa-indonesia-native/native_id_shard_00018.txt.gz | Data | 695.2 MB | 306aaed81d5a |
| corpus-bahasa-indonesia/ccnews/ccnews_shard_00001.txt.gz | Data | 1.8 GB | 07f371741641 |
| corpus-bahasa-indonesia/culturax/culturax_shard_00003.txt.gz | Data | 3.0 GB | 1c16e9c113e6 |
| corpus-bahasa-indonesia/culturax/culturax_shard_00004.txt.gz | Data | 3.0 GB | 96593555862a |
| corpus-bahasa-indonesia/culturax/culturax_shard_00006.txt.gz | Data | 1.3 GB | dab23551863c |
| corpus-bahasa-indonesia/indo4b/indo4b_shard_00001.txt.gz | Data | 676 B | 1578aec52ed4 |
| corpus-bahasa-indonesia/indo4b/indo4b_shard_00002.txt.gz | Data | 1.8 GB | 4ec045cc9cc5 |
| corpus-bahasa-indonesia/kamus_alay/kamus_alay.txt.gz | Data | 53.6 KB | eec3f5e72700 |
| corpus-bahasa-indonesia/kaskus/kaskus_shard_00001.txt.gz | Data | 16.5 KB | 18f7a0a671df |
| corpus-bahasa-indonesia/kaskus/kaskus_shard_00002.txt.gz | Data | 38.1 MB | f8c11d1e36bc |
| corpus-bahasa-indonesia/wikipedia/wikipedia_shard_00001.txt.gz | Data | 8.5 KB | 759b84e54fd6 |
| corpus-bahasa-indonesia/wikipedia/wikipedia_shard_00002.txt.gz | Data | 314.1 MB | c4a7af304d57 |
| corpus-code/cpp/code_cpp_shard_00001.txt.gz | Data | 358.4 MB | 033ab0373682 |
| corpus-code/cpp/code_cpp_shard_00002.txt.gz | Data | 416.6 MB | 944d2bd00b5e |
| corpus-code/css/code_css_shard_00001.txt.gz | Data | 7.9 KB | e6911e412643 |
| corpus-code/css/code_css_shard_00002.txt.gz | Data | 438.7 MB | 3d6f108a4b05 |
| corpus-code/go/code_go_shard_00001.txt.gz | Data | 1.4 GB | df6da89fae1d |
| corpus-code/html/code_html_shard_00001.txt.gz | Data | 630.4 MB | 1c7971c15f24 |
| corpus-code/html/code_html_shard_00002.txt.gz | Data | 142.0 MB | fc47e44d7eee |
| corpus-code/java/code_java_shard_00001.txt.gz | Data | 958.4 MB | 907b7be280af |
| corpus-code/java/code_java_shard_00002.txt.gz | Data | 277.2 MB | 679b9bd1828f |
| corpus-code/javascript/code_javascript_shard_00001.txt.gz | Data | 620.3 MB | 4fff9aca5cca |
| corpus-code/javascript/code_javascript_shard_00002.txt.gz | Data | 274.6 MB | f6d4f2a68418 |
| corpus-code/python/code_python_shard_00001.txt.gz | Data | 27.8 KB | 3c1b5b1074fe |
| corpus-code/python/code_python_shard_00002.txt.gz | Data | 580.9 MB | a14b094ae66a |
| corpus-code/python/code_python_shard_00003.txt.gz | Data | 680.1 MB | 7d94e2cbaef7 |
| corpus-code/rust/code_rust_shard_00001.txt.gz | Data | 250.9 MB | 76870ac563f4 |
| corpus-code/sql/code_sql_shard_00001.txt.gz | Data | 177.3 MB | 92f62f6ddc32 |
| corpus-code/typescript/code_typescript_shard_00001.txt.gz | Data | 198.7 MB | f82f78c70a83 |
| corpus-code/typescript/code_typescript_shard_00002.txt.gz | Data | 228.7 MB | ae8d5afd7f09 |
| corpus-hukum/japan/hukum_japan_shard_00001.jsonl.gz | Data | 53.9 KB | 968885e2ca1d |
| corpus-hukum/japan/hukum_japan_shard_00002.jsonl.gz | Data | 70.7 MB | 7977d377314b |
| corpus-hukum/us/hukum_us_shard_00001.jsonl.gz | Data | 12.8 KB | 6fd619669fae |
| corpus-hukum/us/hukum_us_shard_00002.jsonl.gz | Data | 37.0 MB | cd0a49d0a192 |
| corpus-instruct/alpaca_id/instruct_alpaca_id_shard_00001.jsonl.gz | Data | 11.0 MB | 7230f478c35b |
| corpus-instruct/dolly/instruct_dolly_shard_00001.jsonl.gz | Data | 5.6 KB | 9a5ca2c76aba |
| corpus-instruct/dolly/instruct_dolly_shard_00002.jsonl.gz | Data | 4.6 MB | 518fa6cd5a5f |
| corpus-instruct/oig/instruct_oig_shard_00001.jsonl.gz | Data | 218.5 MB | 97c349d1895a |
| corpus-math/gsm8k/math_gsm8k_shard_00001.jsonl.gz | Data | 2.4 KB | 63bf1224055f |
| corpus-math/gsm8k/math_gsm8k_shard_00002.jsonl.gz | Data | 1.4 MB | f6f897f5c492 |
| corpus-math/numinamath/math_numinamath_shard_00001.jsonl.gz | Data | 370.5 MB | 531214364708 |
| wikipedia/ja/wikipedia_ja_shard_00002.txt.gz | Data | 1.3 GB | 29beb724bfcb |
| wikipedia/ko/wikipedia_ko_shard_00001.txt.gz | Data | 443.7 MB | 53ee97d9d0aa |
| wikipedia/zh/wikipedia_zh_shard_00001.txt.gz | Data | 1.0 GB | d6a98eb3fd16 |
| LICENSE.md | Documentation | 4.6 KB | — |
| MEA-LICENSE.md | Documentation | 3.0 KB | — |
| README.md | Documentation | 80.3 KB | — |
| assets/pustaka_banner.svg | Other | 23.3 KB | — |
| .gitattributes | Repository | 2.6 KB | — |
| tokenizer/tokenizer.json | Tokenizer | 10.5 MB | 26125a50e1de |
License and Download
- License
- other
- Access
- No access gate
Download from MEA Ecosystem
Released by MEA Ecosystem through its official repository on Hugging Face. Read the license.