dwd-icon-global

Dataset Card for DWD ICON Global Forecast

This dataset is comprised of forecasts from the German Weather Service's (DWD) ICON-Global model from March 2023 to the present with all variables included. Each forecast runs up to 4 days into the future, and the model is ran 4 times per day. This data is an archive of the publicly available data at https://opendata.dwd.de/weather/nwp/, converted to Zarr format with Xarray. No other processing of the data is performed.

Dataset Details

• Curated by: Jacob Bieker, Sol Cotton, Open Climate Fix
• License: cc-by-4.0

Dataset Sources [optional]

• Raw files: https://opendata.dwd.de/weather/nwp/

Note: The raw files are deleted after 24 hours, and there is no long-term archive available publicly.

Uses

This data is intended for use in renewable energy forecasting, weather forecasting, and anything that can use high-quality weather forecasts over Europe.

Dataset Structure

The dataset is comprised of one Zarr file per forecast initialization time, and each forecast goes out between 48-96 hours. The files are located at data/year/month/day/YYYYMMDDHH.zarr.zip.

Dataset Creation

Curation Rationale

The DWD ICON Global model provides high-quality, high-resolution forecasts for Global weather that is also publicly available and free of charge. The model should generally outperform NOAA's GFS forecast model, and has a higher temporal and spatial resolution. The main downside of this model is that the files are only available for a short period publicly, so this dataset was setup to provide a public archive of the forecasts for use by researchers in many fields, but especially renewable energy forecasting and weather forecasting.

Source Data

The source data is the grib2 files from the DWD Open Data Server.

Data Collection and Processing

The data is collected every day, around 6-8 hours after forecast initialization time to ensure the forecast is finished running before the data is pulled. The grib2 files are opened with Xarray and collated into a single Xarray Dataset, with one data variable per ICON variable. Surface variables have "_s" appended to their names to differentiate them from multi-level variables. The Dataset is then written to Zarr using "ocf_blosc2" to encode and compress the variables. No scaling or changing of the variables values is performed. This does mean that the data is not in a regular lat/lon grid and is instead in the icosohedral grid of the model. To obtain a regular grid, the data will need to be regridded.

Who are the source data producers?

German Weather Service (DWD)

Recommendations

These files can be opened directly from HuggingFace, and streamed in with Xarray. HuggingFace is fairly slow though, so the recommended way would be to download the files you want and open them locally. In either case, to access the data you can do the following

import ocf_blosc2
import xarray as xr
data = xr.open_zarr("path/to/zarr/file")
print(data)

Dataset Card Contact

OCF Data Team: [email protected]