Summary and Setup

This lesson introduces interoperability in climate and atmospheric sciences as the ability to make research data usable across different tools, systems, and workflows with minimal dataset-specific intervention.

Climate and atmospheric research routinely combines heterogeneous and often large datasets produced by models, satellites, radar systems, sensors, and research infrastructures. Making these data available is therefore not enough: researchers and software must also be able to determine how the data are organised, understand what they mean, and access them through predictable technical mechanisms.

The course approaches this challenge through three complementary layers of interoperability: structural interoperability, which concerns how data are organised and represented; semantic interoperability, which concerns how scientific meaning is expressed and shared; and technical interoperability, which concerns how independent systems access and exchange data and metadata.

Using climate and atmospheric data as the practical context, learners work with community formats and conventions such as NetCDF and the CF Conventions, inspect and subset remote datasets through DAP/OPeNDAP, interact with repository metadata through Web APIs, and explore how Zarr and Kerchunk support selective and scalable access to large multidimensional datasets.

Together, these episodes show how interoperability depends on coordinated choices about data structure, scientific meaning, access mechanisms, storage layouts, and reproducibility rather than on any single format or technology.

Learning objectives

  • Assess a climate or atmospheric dataset in terms of structural, semantic, and technical interoperability and identify barriers to its reuse.

  • Analyse how the structure of a scientific dataset, particularly the NetCDF data model, enables software to identify and process its dimensions, variables, coordinates, attributes, and relationships.

  • Evaluate whether scientific variables are described with sufficient shared, machine-actionable meaning for reliable interpretation and comparison, using semantic resources.

  • Use DAP/OPeNDAP with Python to inspect and subset remote NetCDF data while distinguishing remote metadata access from data transfer.

  • Use a Web API to programmatically query and retrieve research data and metadata and explain how APIs support machine-to-machine interoperability.

  • Compare conventional NetCDF, Zarr, and Kerchunk-based access models to explain how cloud-native layouts support selective and scalable access to large multidimensional datasets.

  • Explain AI readiness as a task-dependent property of a data workflow by connecting structural, semantic, and technical interoperability with scalable access and reproducibility requirements.

Target audience

This lesson is intended for researchers in the climate and atmospheric sciences who handle multidimensional NetCDF datasets and intend to make their data and software more reusable by others. It is also intended for support staff that need capacity in those topics.

Ash’s challenge: combining climate data for rainfall and drizzle research

Ash is studying the spatial and temporal distribution of rainfall and drizzle in Europe. She wants to compare climate model output with satellite observations, urban sensor measurements, radar or aircraft observations, national meteorological datasets, and datasets deposited in research repositories.

At first, the data ecosystem looks rich. She can search across platforms such as Copernicus Climate Data Store, NASA EarthData, the KNMI Data Platform, and 4TU.ResearchData. Many datasets are open, downloadable, and described online. Some platforms provide climate model output, others provide satellite products, national weather observations, radar composites, or research datasets deposited by individual research groups.

At 4TU.ResearchData, Ash finds a dataset from the IRCTR Drizzle Radar (IDRA). IDRA is a high-resolution, polarimetric X-band radar developed by TU Delft and located at the Cabauw experimental site in the Netherlands. It is designed to observe low-reflectivity precipitation such as drizzle and light rain within a local observation radius. This makes it highly relevant for Ash’s research question, because drizzle is often difficult to capture consistently across different observation systems.

Callout

The historical measurements by IDRA are available online.

The problem is not simply finding data. The problem is making different datasets work together.

For rainfall and drizzle research, Ash may encounter precipitation data in many different forms. Some files are NetCDF, CSV, GeoTIFF, Excel, HDF5, GRIB, or Zarr. Some datasets can be accessed through APIs, OPeNDAP, THREDDS, WMS services, or cloud-native object storage, while others require manual download from a web interface.

Even when the data is available, it may not be immediately clear how to combine it. One dataset may describe precipitation_flux, another may use rainfall_rate, rain_intensity, precipitation_amount, RR, reflectivity, equivalent_reflectivity_factor, or DBZH. These names do not always represent the same physical quantity. Some describe rainfall accumulation over a time interval, some describe instantaneous rainfall rate, and others describe radar reflectivity, which is related to precipitation but is not the same as rainfall amount.

Units may also differ or be missing. Rainfall can be expressed in mm, mm h-1, kg m-2 s-1, or accumulated over 5 minutes, 1 hour, 1 day, or a model time step. Radar variables may use units such as dBZ, while coordinates may be stored inside the file, described in a separate document, exposed through an API response, or not documented clearly at all.

Spatial and temporal alignment adds another challenge. A satellite product may provide gridded observations over Europe. A climate model may provide daily or hourly output on a coarser grid. A national meteorological service may provide radar composites every 5 minutes. IDRA may provide local high-resolution radar measurements around Cabauw. Urban sensors may measure rainfall at specific locations. To compare these sources, Ash needs to understand not only the data values, but also their resolution, coordinate reference system, time coverage, processing level, uncertainty, provenance, and version.

Interoperability turns fragmented climate data into connected, reusable research workflows. Image created with AI
Interoperability turns fragmented climate data into connected, reusable research workflows. Image created with AI

To combine these datasets reliably, Ash needs to answer a sequence of questions:

  1. Can I find the right datasets? Are they described in APIs(Application Programming Interfaces) in a way that supports search by time, location, variable, version, and data type?

  2. Can I read the data structure? Are the files organized using community formats such as NetCDF, Zarr, GeoTIFF, or Parquet, with explicit dimensions, variables, coordinates, and attributes?

  3. Can I understand what the variables mean? Do the datasets use shared metadata conventions, controlled vocabularies, standard names, units, coordinate systems, and provenance information?

  4. Can I access the data programmatically? Can Ash use APIs, OPeNDAP , THREDDS, or other standard access mechanisms instead of downloading everything manually?

  5. Can I work with the data at scale? Can she subset remote files, read only the variables and time periods she needs, or use cloud-native layouts such as Zarr or Kerchunk for repeated analysis?

  6. Can I reproduce and automate the workflow? Are dataset versions, identifiers, metadata, and access routes stable enough for notebooks, dashboards, pipelines, or AI(Artifical Intelligence) workflows?

References and Glossary

For further reading and definitions of key terms introduced in this workshop, consult the Reference section.

Prerequisite

To follow this lesson, learners should already be able to have :

  • Working knowledge in Python (write and execute short scripts in Python)
  • Awareness of NetCDF format

1. Software Setup

We will use JupyterLab for live coding and exercises.

This course requires:

  • A Unix-like terminal
  • uv, a Python package and project manager
  • Python 3.11 or newer
  • Several Python libraries, defined in pyproject.toml
  • jq library - optional for API episode

Get Unix Shell Terminal

For practical exercises, we will often use a Unix-like terminal.

Install uv (Required)

We use uv instead of manually creating a virtual environment with venv and installing packages from requirements.txt.

Install uv using one of the options below.

Verify uv Installation

Open a terminal and run:

BASH 

uv --version

Expected output:

BASH 

uv 0.x.x

jq is a command-line tool for reading and formatting JSON output. It is useful when working with REST APIs.

Verify jq Installation

BASH 

jq --version

Expected output:

BASH 

jq-1.x

2. Project and Environment Setup

Create a course working directory somewhere convenient, for example your home directory, and then download the course environment file.

BASH 

mkdir  ~/Interoperability_climate_sciences
cd ~/Interoperability_climate_sciences
curl -o pyproject.toml \
https://raw.githubusercontent.com/4TUResearchData-Carpentries/interoperability-climate-sciences/main/learners/files/pyproject.toml

The pyproject.toml file defines the Python dependencies required for this course.

Verify that the file was downloaded successfully:

BASH 

ls

Expected output:

BASH 

pyproject.toml
Callout

If the download fails, open the download URL in a web browser and save the file as: pyproject.toml inside your Interoperability_climate_sciences folder.

Create and Synchronise the Environment

Run:

BASH 

uv sync

This command will:

  • create a local .venv folder if it does not exist;
  • install all packages listed in pyproject.toml;
  • create or update the uv.lock file.

Register the Environment in Jupyter

Register the course environment as a Jupyter kernel:

BASH 

uv run python -m ipykernel install --user --name nes-course-env --display-name "NES Course (Python)"

This makes the environment available inside JupyterLab as:

NES Course (Python)

Launch JupyterLab

Launch JupyterLab with:

BASH 

uv run jupyter lab

In JupyterLab, click on the button NES Course (Python) under Notebook.

Screenshot from JupyterLab Launcher with title Notebook and two Python icons underneath: one named 'Python 3 (ipykernel)' and the other named 'NES Course (Python)'

4. Final Setup Check

Before the workshop, make sure the following commands work:

BASH 

uv --version
uv sync
uv run jupyter lab
jq --version

If all commands work, you are ready for the course.

5. Troubleshooting

uv: command not found

Close and reopen your terminal.

Then try:

BASH 

uv --version

If it still fails, reinstall uv or check whether the installation folder was added to your PATH.

JupyterLab opens but the course kernel is missing

Run:

BASH 

uv run python -m ipykernel install --user --name nes-course-env --display-name "NES Course (Python)"

Then restart JupyterLab:

BASH 

uv run jupyter lab

A package import fails

Run:

BASH 

uv sync

Then verify again:

BASH 

uv run python -c "import xarray, netCDF4, pydap, zarr, kerchunk, fsspec; print('All good')"

You are not sure which Python is being used

Run:

BASH 

uv run python --version
uv run python -c "import sys; print(sys.executable)"

The executable path should point to the .venv folder inside your course directory.

Example:

.../Interoperability_climate_sciences/.venv/...

Optional Fallback: venv and requirements.txt

Use this fallback only if uv cannot be installed on your machine.

Caution

The recommended setup for this course is uv.

Use this section only if your institution blocks uv installation or if you cannot get uv working before the lesson.

Create a virtual environment:

BASH 

python -m venv nes-course-env

Activate it.

Download requirements.txt file and use it to install dependencies:

BASH 

curl -o requirements.txt \
https://raw.githubusercontent.com/4TUResearchData-Carpentries/interoperability-climate-sciences/main/learners/files/requirements.txt
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

Register the environment in Jupyter:

BASH 

python -m ipykernel install --user --name nes-course-env --display-name "NES Course (Python)"

Launch JupyterLab:

BASH 

jupyter lab