Technical interoperability I: Data access protocols

Interoperability in Climate and Atmospheric sciences
FAIR NetCDF Data

What is technical interoperability?

60 minutes: 13:00 -14:00

The Problem

How can one computer system access data held by another system?

How can a computer access scientific data stored on a remote system without necessarily downloading the entire file?

~2 min. One idea: the problem is getting the data to where the work happens.

1. “Ash faces another problem: How does she get at the data when they sit on someone else’s system?”
2. Read both questions aloud. Pause on “without necessarily downloading the entire file”.
3. “Technical interoperability is about answering these two questions.”

Definition

Technical Interoperability concerns the mechanisms that allow independent systems to communicate and exchange data through agreed technical interfaces and protocols

~1 min. One idea: agreed interfaces and protocols let independent systems exchange data.

Read the definition aloud. Stress interfaces and protocols: “the agreed way to ask, and the agreed rules for the exchange.”

Example: one variable, one period, one region

A climate dataset is stored in a research data repository. A researcher is interested in one variable, covering a particular period and a geographical region.

Consider two approaches:

1. Download the complete file and perform the selection locally
2. The remote infrastructure provides a mechanism to request only the required part

~4 min. One idea: often you need only a small part of a large dataset.

1. (Before any click) Read the scenario: one variable, one period, one region. Point at the image: “the part we need is a small slice of a big array.” Read the two approaches.

Which one is more suitable…?

2. (Click 1) Ask “Which one is more suitable?” Wait 10-15 seconds and take one or two answers. Don’t confirm yet: the answer is “it depends”.

It depends on the task

• Downloading the file is fine for small files, or when you need the complete dataset.

• Requesting only the part you need matters for large datasets and automated workflows.

3. (Click 2) “It depends on the task.” Small file, or you need everything: download. Large dataset or automated workflow: request only the part.

Concepts

~2 min. One idea: storage is not access.

1. “Where the data are stored, a repository or a server, is one thing. How a program can ask for them is another.”
2. “A data-access service sits in between, retrieving data from storage and delivering the requested subset.”
3. “So remote or cloud storage alone does not guarantee interoperability. An agreed access mechanism is also needed.”

~3 min. One idea: the protocol defines what a client may ask and how the server answers.

1. “NetCDF describes how data are structured. A protocol such as DAP describes how that structured data can be requested over a network.”
2. “The protocol defines the conversation between client and server: discover metadata, request a subset, and receive the selected data.”
3. “Because the requests and responses are standardized, the same client can work with any compatible server without custom code for each dataset.”

DAP and OPeNDAP

DAP and OPeNDAP

The problem

Important scientific datasets were spread across institutions, each with its own way of storing and accessing data.

~4 min. One idea: DAP started as a common way to reach data spread across institutions.

1. The problem: datasets spread across institutions, each with its own way of storing and accessing data.

The idea

A common way to access data, wherever it is stored.

2. The idea: a common way to access data, wherever it is stored.

“A URL equals a dataset.”

“A URL with constraints equals a subset.”

3. Read the two quotes: “A URL equals a dataset. A URL with constraints equals a subset.”
4. Origin in one line: it started in oceanography (a 1993 workshop; the first system, DODS, in 1995). Climate, atmospheric science and remote sensing had the same problem, so it evolved into the Data Access Protocol (DAP). OPeNDAP Inc. has maintained it as a nonprofit since 2000.

Who is who?

DAP	the protocol: how clients and servers talk
OPeNDAP	the project and software ecosystem behind DAP
Hyrax, THREDDS	data servers that implement DAP

~2 min. One idea: DAP is the protocol, OPeNDAP the project, Hyrax and THREDDS the servers.

1. (Before the click) Go through the three rows: “the protocol, the project behind it, the servers that implement it.”

Researchers just receive an endpoint URL. Data providers run the server.

2. (Click) “Researchers just receive an endpoint URL. Data providers run the server.” For researchers the distinction is usually invisible; the purpose stays the same: independent clients accessing distributed, structured data through one common protocol.

Hands-on

Let’s make use of DAP

We will access remote NetCDF datasets stored in 4TU.ResearchData using the DAP protocol

https://edu.nl/9fh7g

~35 min (the time that is left). One idea: reach remote data with DAP, without downloading the file.

1. “Open the link.” Wait until everyone has the page open before you start.
2. Remind them of the three ideas as they work: explore the dataset without downloading it, subset on the server, and what is transferred is only the part they asked for.

Wrap-up

Key takeaways

• Technical interoperability is about how systems agree to request and exchange data over a network
• Storage is not access: remote data still needs an agreed access mechanism
• DAP is the protocol. OPeNDAP is the project behind it. Hyrax and THREDDS are servers that implement it
• With DAP you can explore a remote dataset without downloading it, and subset on the server before any data is transferred
• A repository can be more than a place to store data: it can also make data accessible to machines, selectively

~3 min. One idea: Think about access, not only storage.

1. Read the five bullets quickly; they mirror the episode’s key points.
2. Main message for data stewards: when choosing infrastructure, ask not only “Can it store my NetCDF files?” but also “How will machines access the data after I publish them?”