Cloud Native Layouts

Interoperability in Climate and Atmospheric sciences
FAIR NetCDF Data

Problem

Is my data optimised for access even if it is stored in the cloud with a predictable format and following standard metadata conventions?

A bit of context

Climate datasets are often huge, but researchers usually need only a small part of the data for a particular analysis. A researcher might only need one variable, a short time period, one pressure level, or a small geographical region.

See Climate Data Store from Copernicus

Large data collections such as ERA5 can extend across terabytes or petabytes of data.

Large data collections such as ERA5 or CMIP6 can extend across terabytes or petabytes of data.

Researchers, however, rarely need an entire collection for a particular analysis. A researcher might only need one variable, a short time period, one pressure level, or a small geographical region.

But, how can that be achieved?

Cloud native does not just mean “in the cloud”

Cloud native is about the access model

1. Cloud native layouts support workflows in which software retrieves only selected parts of a dataset and can perform multiple reads in parallel.
2. The important idea is NOT only where the data are stored, but how the data are organised for access.

Lets take a step back on how the traditional file-based works

One file, one unit of access

Many traditional scientific workflows are organised around files. A researcher finds a file on a repository or server, downloads or opens it, selects the required observations, and then performs the analysis.

Cloud native layouts

Rationale: Break down each file into smaller independently accessible parts

NetCDF vs Zarr

NetCDF

NetCDF:
a primarily file-oriented representation

A NetCDF dataset is commonly packaged into a single binary file.

→

It gets uploaded to an object storage cloud infrastructure

→

• Remote access is possible
• Software can use HTTP byte-range requests to retrieve regions of the file.

But it is not a scalable model for workflows involving many repeated subsets, many parallel readers

Zarr : a cloud-oriented chunked representation

• Zarr divides arrays into smaller pieces called chunks.
• A chunk represents a region of a multidimensional array, can be read independently from the others
• The size of the chunk is a design choice that should match the expected access pattern.

If a Zarr object is stored in an object storage cloud infrastructure

• Remote access is possible.
• Each chunk can be retrieved independently
• Supports parallel access

A Zarr storage layout can expose those chunks as independently addressable parts of the storage system

So, then:

• Zarr should not be understood simply as a replacement for NetCDF
• NetCDF remains fundamental to climate and atmospheric sciences, large scientific archives, established tools, community conventions, repositories, and existing research workflows depend on it.
• Zarr becomes particularly relevant when multidimensional datasets need to be accessed repeatedly over a network
• Both can represent multidimensional scientific data. What changes is how that data is physically organised and retrieved.

Cloud optimized Geospatial data formats

From https://guide.cloudnativegeo.org/

Self-paced demo

Hands-on:
Creating Zarr-compatible access from NetCDF , with Kerchunk

Key takeaways

• Cloud-native data layouts address the problem of efficiently accessing small parts of very large remote datasets.
• Object storage manages independently addressable objects rather than providing the same access model as a traditional filesystem.
• A conventional NetCDF file can be stored in the cloud without becoming a cloud-native data layout.
• Zarr uses a chunk-oriented representation that maps naturally onto selective and parallel access in object storage.
• Cloud-native layouts affect structural and technical interoperability; semantic interoperability still depends on conventions such as CF.
• Kerchunk provides a Zarr-compatible reference view of existing NetCDF or HDF5 data without duplicating the underlying scientific data.
• OPeNDAP, Zarr, and Kerchunk provide different approaches to efficient remote scientific-data access.