A Japanese-language dataset for training and fine-tuning models on modern data engineering topics as of 2026.
Rows1,966
Configurations1
Size2.6 MB
Licensecc-by-4.0
AccessPublicly accessible
Monthly Downloads—
Dataset Card
By Nebou Rocket, published under cc-by-4.0, revision 19a96856bfc2.
Data Engineer 2026 (Japanese) Dataset
A Japanese-language dataset for training and fine-tuning models on modern data engineering topics as of 2026. It contains instruction-tuning (SFT) Q&A pairs and a pretraining corpus covering distributed processing, streaming, data lakehouse, orchestration, transformation, local analytics, modeling, quality and governance, cloud and data platforms, MLOps, and AI/agentic data workflows.
Dataset splits
| Split | Size | Description |
|---|---|---|
sft |
1,780 samples | Instruction-following Q&A pairs in Japanese (instruction / input / output). Covers definitions, comparisons, how-tos, design and best practices, debugging, interview-style questions, 2026 trends, outlines, and aspect-specific questions. |
pretrain |
186 samples | Japanese technical explanatory text (chunked articles and notes) about data engineering topics, intended for continued pretraining. |
Schema
SFT (sft)
| Column | Type | Description |
|---|---|---|
id |
str | Sample ID (sft_N) |
category |
str | High-level category (分散処理エンジン, ストリーミング基盤, DWH・SQL, テーブル格式, オーケストレーション, 変換, ローカル分析, モデリング, データ運用, クラウド基盤, AI・ML, 現場, トレンド, 移行, 設計, ...) |
topic |
str | Topic slug (e.g. spark, flink, kafka, iceberg, airflow, dbt, duckdb, polars, lakehouse, contracts, governance-ai, ...) |
qtype |
str | Question type (faq, definition, overview, design, howto, debug, trend, interview, compare, outline, aspect, variant) |
aspect |
str | Aspect (general, 全体像, 設計, パフォーマンス, コスト, 運用, セキュリティ, 移行, 2026年動向) |
difficulty |
str | intermediate (default) |
instruction |
str | Question or instruction in Japanese |
input |
str | Optional context (empty in this release) |
output |
str | Answer or explanation in Japanese (markdown) |
lang |
str | ja |
source |
str | curated+synthetic or synthetic |
Pretrain (pretrain)
| Column | Type | Description |
|---|---|---|
id |
str | Sample ID (pre_N) |
text |
str | Explanatory Japanese text (markdown-style article or FAQ block) |
topic |
str | Topic slug |
category |
str | Category |
lang |
str | ja |
source |
str | synthetic |
Coverage (key topics)
- Distributed engines: Apache Spark 4.x, Apache Flink 2.x
- Streaming: Apache Kafka 4.x (KRaft), streaming fundamentals, CDC (Debezium), streaming databases (RisingWave / Materialize)
- Lakehouse and table formats: Apache Iceberg, Delta Lake, Hudi, catalogs (Unity Catalog / Polaris / Gravitino / Nessie), file formats (Parquet / Arrow / Avro)
- Warehouses and SQL engines: BigQuery, Snowflake, Redshift, Trino, StarRocks, ClickHouse, SQL craft
- Orchestration and transformation: Apache Airflow 3.x (Task SDK), Dagster (SDA), dbt (Fusion / Core v2.0), SQLMesh
- Local and modern analytics: DuckDB 1.5 / 2.0, Polars, Python data stack 2026 (pandas 3.x, uv, ruff, marimo, pydantic)
- Data modeling and contracts: Kimball / Data Vault / OBT, data contracts (ODCS), data quality (Soda / Great Expectations / Deequ / Elementary), lineage (OpenLineage)
- Cloud and platform: AWS / GCP / Azure data platforms, Kubernetes data (Spark / Flink operators / Strimzi / Kueue)
- AI / ML and data for AI: AI/LLM data pipelines (embeddings / RAG / vector DB), MLOps data (feature store / lineage / versioning), governance for AI agents (MCP / text2SQL)
- Ops / FinOps / reliability, testing / CI, migration, design patterns
All content reflects 2026-era trends (e.g. Iceberg as the open table format, Spark Connect, Flink disaggregated state, Airflow 3, dbt Fusion, DuckDB / Wasm / Iceberg, streaming-first lakehouse, data contracts, agentic data workflows).
How to load
pip install datasets
from datasets import load_dataset
ds = load_dataset("LoNebula/data-engineer-2026-ja")
print(ds)
# ds["sft"], ds["pretrain"]
Notes and license
- This dataset is primarily synthetically generated from a curated Japanese knowledge base. It is intended for research, educational purposes, and model training or evaluation.
- While care was taken for factual accuracy (aligning to 2026 versions and trends), it may contain minor inaccuracies. Always verify against official documentation.
- License: CC BY 4.0. Retain attribution to
LoNebulaand a link to this dataset when redistributing or building derived work.
Acknowledgments
Generated with assistance from Kilo / Claude Code tooling. The dataset structure and schema are designed for instruction tuning (SFT) and continued pretraining in Japanese.
Structure
default 1,966 rows
| Split | Rows | Size |
|---|---|---|
| pretrain | 186 | 144.9 KB |
| sft | 1,780 | 709.4 KB |
idstringcategorystringtopicstringqtypestringaspectstringdifficultystringinstructionstringinputstringoutputstringlangstringsourcestringtextstring
Details
- Repository
- LoNebula/data-engineer-2026-ja
- Publisher
- Nebou Rocket
- Task category
- Question answering
- Tags
- data-engineering, japanese, instruction-tuning
- Size category
- 1K<n<10K
- Languages
- ja
- Revision
- 19a96856bfc2ba9c25d276647685f59e98ce6d50
- Last updated
- 2026-10-12
Files
29 files, 2.6 MB in total.
Data6 files · 2.1 MB
Documentation3 files · 9.9 KB
Other19 files · 473.5 KB
Repository1 file · 2.5 KB
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| data/hf-de2026-pretrain.jsonl | Data | 163.4 KB | — |
| data/hf-de2026-sft.jsonl | Data | 1.4 MB | — |
| data/pretrain-00000-of-00001.parquet | Data | 79.2 KB | 5807e85385d9 |
| data/pretrain.parquet | Data | 79.2 KB | 5807e85385d9 |
| data/sft-00000-of-00001.parquet | Data | 218.8 KB | 0e189ea81c10 |
| data/sft.parquet | Data | 218.8 KB | 0e189ea81c10 |
| README.md | Documentation | 5.1 KB | — |
| README_EN.md | Documentation | 4.3 KB | — |
| data_card.md | Documentation | 490 B | — |
| __pycache__/kb_a.cpython-314.pyc | Other | 41.1 KB | — |
| __pycache__/kb_all.cpython-314.pyc | Other | 514 B | — |
| __pycache__/kb_b.cpython-314.pyc | Other | 35.0 KB | — |
| __pycache__/kb_c.cpython-314.pyc | Other | 33.7 KB | — |
| __pycache__/kb_d.cpython-314.pyc | Other | 31.7 KB | — |
| __pycache__/topics_a.cpython-314.pyc | Other | 41.2 KB | — |
| __pycache__/topics_b.cpython-314.pyc | Other | 35.3 KB | — |
| __pycache__/topics_c.cpython-314.pyc | Other | 34.0 KB | — |
| __pycache__/topics_d.cpython-314.pyc | Other | 31.9 KB | — |
| __pycache__/topics_e.cpython-314.pyc | Other | 7.5 KB | — |
| build_dataset.py | Other | 4.6 KB | — |
| kb_all.py | Other | 294 B | — |
| topics_a.py | Other | 46.7 KB | — |
| topics_b.py | Other | 39.5 KB | — |
| topics_c.py | Other | 39.3 KB | — |
| topics_d.py | Other | 36.2 KB | — |
| topics_e.py | Other | 9.1 KB | — |
| train_and_upload.py | Other | 4.7 KB | — |
| upload_dataset.py | Other | 1.1 KB | — |
| .gitattributes | Repository | 2.5 KB | — |
License and Download
- License
- cc-by-4.0
- Access
- No access gate
Download from Nebou Rocket
Released by Nebou Rocket through its official repository on Hugging Face. Read the license.