Introduction

Interoperability in Climate and Atmospheric sciences
FAIR NetCDF Data

Big picture

General definition of interoperability

• The lesson introduces interoperability as the ability to make research data usable across different tools, systems, and workflows with minimal dataset-specific intervention.

Practical context

• Climate and atmospheric research routinely combines heterogeneous and often large datasets produced by models, satellites, radar systems, sensors, and research infrastructures.

Challenges

• Making these data available is not enough. Researchers and software must be able to determine: how the data are organised, understand what they mean, and access them through predictable mechanisms.

The Problem

Ash case study

Hi! I am Ash, and I am studying the spatial and temporal distribution of rainfall and drizzle in Europe

Rainfall and drizzle

Types of precipitation: rain, drizzle (liquid), freezing rain, sleet (liquid/solid), snow, hail, graupel, ice crystals (solid)

Ash’s approach to study the rainfall and drizzle

She needs to integrate multiple data from different sources

• Climate models
• Satellite observations
• Urban sensor data
• Radar observations
• National meteorological datasets

Most probably each type of data comes in different formats, different ways of access and many of them use different variable names for the same physical quantity.

Ash’s data: IDRA radar

At 4TU.ResearchData, Ash finds a dataset from the IRCTR Drizzle Radar (IDRA).

• High-resolution, polarimetric X-band radar developed by TU Delft
• Located at the Cabauw experimental site, the Netherlands
• Designed to observe drizzle and light rain, which are hard to capture consistently across observation systems
• Data are published as NetCDF files

Image: TU Delft IDRA, http://ftp.tudelft.nl/TUDelft/irctr-rse/idra/index.html

IDRA is the IRCTR Drizzle Radar, installed on the meteorological tower at the Cabauw experimental site for atmospheric research (CESAR). It scans horizontally at a low elevation angle and is operated by TU Delft’s ATMOS group. Because drizzle and light precipitation have low reflectivity, they are difficult to capture consistently across observation systems, which makes the dataset highly relevant to Ash’s research question.

We will use these files throughout the session: in the hands-on, and in the optional demo where Ash compares a 2009 and a 2019 file.

Dimensions in the files: time_raw_data, sample_beat_signal, time_processed_data and range. Processed variables used later: equivalent_reflectivity_factor, differential_reflectivity, radial_velocity, spectrum_width, differential_phase.

Challenge

How can these datasets be combined together to be able to answer Ash’s scientific questions?

She wants to compare climate model output with satellite observations, urban sensor measurements, radar or aircraft observations, national meteorological datasets, and datasets deposited in research repositories.

At first, the data ecosystem looks rich. She can search across platforms such as Copernicus Climate Data Store, NASA EarthData, the KNMI Data Platform, and 4TU.ResearchData. Many datasets are open, downloadable, and described online. Some platforms provide climate model output, others provide satellite products, national weather observations, radar composites, or research datasets deposited by individual research groups.

Notion of interoperability

Interoperability is about enabling data to work with other data, software, services and research workflows as “easy” as possible.

“Easy” means with the fewest assumptions possible about how to integrate them, and also less manual intervention.

Interoperable datasets can connect to an ecosystem

Interoperable dataset

↑ it can be

Opened directly in a Python notebook

it can be →

Visualized in an online viewer or dashboard

it can be ↓

Accessed remotely through an API without downloading the file

← it can be

Combined with a second climate dataset

Three layers of interoperability

Structural/Syntactic
representation

Can software understand how the data are organized?

Semantic
meaning

Can another researcher or machine understand what the variables mean?

Technical
access

Can software retrieve or exchange the data through standard mechanisms?

Structural layer is also sometimes called syntactic layer.

Example: Ash wants to compare radar observations from 2009 and 2019

• Structural
  • Do the variables and dimensions match?
• Semantic
  • Given a variable with the same name, would that imply they represent the same physical quantity?
• Technical
  • Can both datasets be accessed programmatically?

Exercise: True or False

1. As long as data is open access, it is interoperable
2. Metadata standards help ensure interoperability
3. As long as the data is using an open standard format, it is interoperable.

What is coming in the workshop

• Morning session
  • Structural interoperability
  • Semantic interoperability

• Afternoon session
  • Technical interoperability: data access protocols
  • Technical interoperability: API
  • Cloud-native layouts
  • Interoperable infrastructure in the AI era