oellm-32b-cooldown-base-eval-scores

Evaluation scores for the two OpenEuroLLM 32B cooldown (annealing) runs, every 2000 iterations from 100k to 120k, scored on the eng_base_main and deu_base_main suites of base-eval via eval-hive.

Weights live in openeurollm/oellm-32b, one branch per checkpoint (<run>_iter_<step>).

Both runs share global_batch_size 4096 × seq_length 4096 = 16,777,216 tokens/step, so tokens_trained = 16_777_216 × step (120000 → 2.013T). The two runs are on an identical step grid, so they can be compared point by point.

Columns

Same schema as openeurollm/prelude-base-eval-scores.

• model, step, tokens_trained, model_display_name, model_path
• task, metric, score, score_stderr, num_fewshot, language, n_samples
• task_type — benchmark (leaf task), task_group, or eval_suite (headline rollup)
• higher_is_better, task_formulation, parent_task, subtask_tree
• eval_date, result_source, eval_config, task_config

Headline numbers are the task_type == 'eval_suite' rows:

import pandas as pd
df = pd.read_parquet("scores.parquet")
suite = df[(df.task_type == "eval_suite") & (df.metric == "acc")]
print(suite.pivot_table(index=["model", "step"], columns="task", values="score"))

Evaluation setup

vLLM 0.21.0, bf16, max_model_len 4096, tensor-parallel 4 on one MareNostrum 5 accelerated node (4×H100) per checkpoint. A suite-level aggregate is only emitted once every applicable child task has a score, so partially-evaluated checkpoints carry leaf benchmark rows but no eval_suite rollup.