ESG Security hooks are in NetCDF

This is a big boost for ESG and access control interoperability with environmental datasets. The security API for the Earth System Grid is now integrated into the NetCDF C libraries. The NetCDF data format is used widely for storing data across the earth sciences.   With these changes, a NetCDF client can make calls to OPeNDAP servers serving secured datasets.  Security settings are made via a standard configuration file so there is no need for any changes to the API.  This is important as many tools and high level language bindings such as Python use the C libraries.

Inner Workings

The NetCDF libraries already support network based data retrieval using OPeNDAP and implementations such as PyDAP and TDS support a HTTP redirect pattern to support authentication schemes like CAS.   For ESG, a security layer to in the form of server side middleware has been developed for PyDAP and TDS.   This builds on the redirect functionality but in addition, enables a client to use SSL client based authentication.  The redirect capability means data can still be served over an unencrypted channel an important consideration when serving large datasets.   SSL client based authentication fits well with Grid based tools used in ESG like MyProxy which enables a user to obtain a short term certificate based on their usual username/password credentials with their Identity Provider.

The NetCDF client authenticates by following a HTTP redirect response set by the server to redirect to a HTTPS based authentication endpoint.  The client passes a certificate in the subsequent SSL handshake so that the server can authenticate the request.  Once authenticated, a redirect response is sent back to the client in order for the client to return to the original URI invoked.   With the client authenticated a cookie is set and control can pass to the data serving application underlying the security middleware.  This sequence from a previous posting illustrates the steps:

Example

These settings in the users .dodsrc file set a client certificate and key (both contained in creds.pem in the example below) and a directory containing CA certificates - used to verify the identity of the server certificate:

CURL.VERBOSE=1
CURL.COOKIEJAR=.dods_cookies
CURL.SSL.VALIDATE=1
CURL.SSL.CERTIFICATE=/.../creds.pem
CURL.SSL.KEY=/.../creds.pem
CURL.SSL.CAPATH=/.../ca-certificates

Full example code here.

Special thanks to Dennis Heimbigner at Unidata and to my colleague Steve Crothers and others at RAL for their contributions on this.