Friday, 3 September 2010

ESG Security hooks are in NetCDF

This is a big boost for ESG and access control interoperability with environmental datasets. The security API for the Earth System Grid is now integrated into the NetCDF C libraries. The NetCDF data format is used widely for storing data across the earth sciences.   With these changes, a NetCDF client can make calls to OPeNDAP servers serving secured datasets.  Security settings are made via a standard configuration file so there is no need for any changes to the API.  This is important as many tools and high level language bindings such as Python use the C libraries.

Inner Workings

The NetCDF libraries already support network based data retrieval using OPeNDAP and implementations such as PyDAP and TDS support a HTTP redirect pattern to support authentication schemes like CAS.   For ESG, a security layer to in the form of server side middleware has been developed for PyDAP and TDS.   This builds on the redirect functionality but in addition, enables a client to use SSL client based authentication.  The redirect capability means data can still be served over an unencrypted channel an important consideration when serving large datasets.   SSL client based authentication fits well with Grid based tools used in ESG like MyProxy which enables a user to obtain a short term certificate based on their usual username/password credentials with their Identity Provider.

The NetCDF client authenticates by following a HTTP redirect response set by the server to redirect to a HTTPS based authentication endpoint.  The client passes a certificate in the subsequent SSL handshake so that the server can authenticate the request.  Once authenticated, a redirect response is sent back to the client in order for the client to return to the original URI invoked.   With the client authenticated a cookie is set and control can pass to the data serving application underlying the security middleware.  This sequence from a previous posting illustrates the steps:



Example

These settings in the users .dodsrc file set a client certificate and key (both contained in creds.pem in the example below) and a directory containing CA certificates - used to verify the identity of the server certificate:


CURL.VERBOSE=1
CURL.COOKIEJAR=.dods_cookies
CURL.SSL.VALIDATE=1
CURL.SSL.CERTIFICATE=/.../creds.pem
CURL.SSL.KEY=/.../creds.pem
CURL.SSL.CAPATH=/.../ca-certificates

Full example code here.

Special thanks to Dennis Heimbigner at Unidata and to my colleague Steve Crothers and others at RAL for their contributions on this.

Friday, 25 June 2010

Introspecting Identity with Yadis

Yadis is a powerful tool for service discovery. It answers the question, given an identity URI, what identity services can I invoke on it?  The Earth System Grid security architecture has OpenID as its core identity service but there are others: MyProxy from the Globus toolkit enables users to obtain a short lived X.509 certificate for SSL based authentication to secured resources.  In addition there is the SAML Attribute Service enabling a client to query attribute information about a given subject.   Locating these services is a problem and this was illustrated in a use case recently: a trusted service in the federation would like to notify a user by e-mail.  The service has their OpenID but it doesn't have access to the usual attributes available through the Attribute Exchange interface - they're not signed in.  How can I query their e-mail address to contact them?

OpenID (v2.0) already supports the Yadis protocol, and its uses is mandatory for ESG OpenID Providers.   If I HTTP GET a user's identity URI, I get a document containing the OpenID Provider endpoint:

Extending this I can include the corresponding MyProxy and the Attribute Service URIs:


<?xml version="1.0"; encoding="UTF-8"?>
<xrds:XRDS xmlns:xrds="xri://$xrds" xmlns="xri://$xrd*($v*2.0)">
  <XRD>
    <Service priority="0">
      <Type>http://specs.openid.net/auth/2.0/signon</Type>
      <Type>http://openid.net/signon/1.0</Type>
      <URI>https://openid.provider.somewhere.ac.uk</URI>
      <LocalID>https://somewhere.ac.uk/openid/PJKershaw</LocalID>
    </Service>
    <Service priority="1">
      <Type>urn:esg:security:myproxy-service</Type>
      <URI>socket://myproxy-server.somewhere.ac.uk:7512</URI>
      <LocalID>https://somewhere.ac.uk/openid/PJKershaw</LocalID>
    </Service>
    <Service priority="20">
      <Type>urn:esg:security:attribute-service</Type>
      <URI>https://attributeservice.somewhere.ac.uk</URI>
      <LocalID>https://somewhere.ac.uk/openid/PJKershaw</LocalID>
    </Service>
  </XRD>
</xrds:XRDS>

The priority attribute for each <Service/> element lets the client know which is the preferred service for authentication. The Attribute Service is of course not an authentication service but I'd argue it has a place here as it is an identity service. [17/03/11 - Note correction for this use case: each service element is in the single XRD element, not many XRD elements each with a single service element as I had before!]

Taking this a step further, I can combine it with MyProxy's provisioning functionality, the ability to provision a client with trusted CA certificates it needs to bootstrap trust.

Given an identity URI, I can:
  1. retrieve the Yadis document
  2. parse the MyProxy server endpoint
  3. retrieve the trust roots to bootstrap trust in the identity services for this identity URI.

Wednesday, 23 June 2010

Earth System Grid Security Architecture Blueprint

We now have a blueprint documented for the ESG security architecture:

http://esg-pcmdi.llnl.gov/esgf/esgf-security-interface-control-documents/esgf-security-interface-control-document-1.0/

The Interface Control Document describes all the interfaces that make up the federated security infrastructure.  This is the product of a lot of collaboration work and discussion.  Given the parellel Java and Python implementations this has been a true test of interoperability.

Thursday, 10 June 2010

Python SAML 2.0 Released

I've been working on a Python implementation of SAML 2.0 for some months now and recently pushed this to the Python repository PyPI.  SAML is an important tool in the Earth System Grid Security architecture that I've been working on and the NERC DataGrid security framework which predates it. It's used for attribute retrieval and authorisation decision queries.  The code focusses on these aspects of the specification.  This example illustrates an client attribute query to an Attribute Authority:


from ndg.saml.saml2.core import (AttributeQuery, SAMLVersion, Issuer, Subject,
                                 NameID, Attribute, XSStringAttributeValue)
from uuid import uuid4
from datetime import datetime

attributeQuery = AttributeQuery()
attributeQuery.version = SAMLVersion(SAMLVersion.VERSION_20)
attributeQuery.id = str(uuid4())
attributeQuery.issueInstant = datetime.utcnow()

attributeQuery.issuer = Issuer()
attributeQuery.issuer.format = Issuer.X509_SUBJECT
attributeQuery.issuer.value = '/O=NDG/OU=BADC/CN=PolicyInformationPoint'
                
attributeQuery.subject = Subject()  
attributeQuery.subject.nameID = NameID()
attributeQuery.subject.nameID.format = NameID.X509_SUBJECT
attributeQuery.subject.nameID.value = '/O=NDG/OU=BADC/CN=PhilipKershaw'

# special case handling for 'LastName' attribute
emailAddressAttribute = Attribute()
emailAddressAttribute.name = "urn:esg:email:address"
emailAddressAttribute.nameFormat = "%s#%s" % (
                                XSStringAttributeValue.TYPE_NAME.namespaceURI,
                                XSStringAttributeValue.TYPE_NAME.localPart)

emailAddress = XSStringAttributeValue()
emailAddress.value = 'pjk@somewhere.ac.uk'
emailAddressAttribute.attributeValues.append(emailAddress)

attributeQuery.attributes.append(emailAddressAttribute)

# Convert to ElementTree representation
from ndg.saml.xml.etree import AttributeQueryElementTree, prettyPrint

elem = AttributeQueryElementTree.toXML(attributeQuery)

# Serialise as string
xmlOut = prettyPrint(elem)
print(xmlOut)



The XML representation and serialisation is independent of the core code. The ndg.saml.xml.etree ElementTree representation comes with the rest of the code but it would be straightforward to add an adaptor for another Python XML package such as 4Suite-XML.

Here's the serialised output:


<samlp:attributequery id="1c15e748-0f74-41f1-848c-1fbdfeef2a06" issueinstant="2010-06-01T13:19:50.690263Z" version="2.0" xmlns:samlp="urn:oasis:names:tc:SAML:2.0:protocol">
    <saml:issuer format="urn:oasis:names:tc:SAML:1.1:nameid-format:x509SubjectName" xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion">/O=NDG/OU=BADC/CN=PolicyInformationPoint</saml:issuer>
    <saml:subject xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion">
        <saml:nameid format="urn:oasis:names:tc:SAML:1.1:nameid-format:x509SubjectName">/O=NDG/OU=BADC/CN=PhilipKershaw</saml:nameid>
    </saml:subject>
    <saml:attribute name="urn:esg:email:address" nameformat="http://www.w3.org/2001/XMLSchema#string" xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion">
        <saml:attributevalue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">pjk@somewhere.ac.uk</saml:attributevalue>
    </saml:attribute>
</samlp:attributequery>


More to follow - including client/server SOAP bindings - as I migrate the SAML specific code out of the NERC DataGrid security package.

Monday, 17 May 2010

Proxy Certificates and Delegation with OPeNDAP Services

I'm back from EGU in Vienna and one topic I presented on was a solution for securing OPeNDAP services which I've developed working with colleagues at STFC and members of the development team for Earth System Grid. Services configured with the security filter middleware can accept either PKI or OpenID based authentication. So far so good but one piece that's missing and that would greatly enhance the solution would be support for delegation.

Proxy certificates would be the obvious choice since this should easily piggyback on the existing PKI based authentication adopted but alternative technologies like OAuth need more exploration. Just how easy the proxy certificate solution is explored here but first why bother? - It would give immediate compatibility with Grid based authentication schemes and would solve the delegation problem. For example in an upcoming project here a portal is required which is effectively an experimental area for scientists to gather data from different sites and combine it with their own that they upload to it. The portal could query OPeNDAP services on behalf of the user for data retrieval. In the case of a secured OPeNDAP service, the user would need to delegate authority to the portal to use their credentials to access the OPeNDAP service on their behalf.

The sequence below shows how security filter middleware intercept requests to an OPeNDAP service and redirect unauthenticated clients to an Authentication Service running over SSL.  At this point if the client submits a certificate in the SSL handshake, the service uses this to authenticate the user.  If no certificate is provided, it defaults to OpenID based sign in.


This Python snippet shows how the client code can work:
>>> import urllib2
>>> from M2Crypto import m2urllib2, SSL
>>> ctx = SSL.Context('sslv3')
>>> ctx.load_cert_chain('./proxy.pem')
>>> opener = m2urllib2.build_opener(ctx)
>>> data = opener.open('http://localhost:8001/dap/sample.nc')
urllib2's redirect handler means that it will conveniently follow the HTTP redirects required in the sequence.  M2Crypto plugs into this to provide the SSL support.   The SSL context uses load_cert_chain to ensure that the complete certificate chain is loaded for the proxy certificate.  For simplicity here no peer certificate validation code has been set.

The server side filters shown in the diagram are implemented as Python WSGI middleware hosted under Apache with mod_wsgi.   The OPeNDAP service is PyDAP.   Apache provides the SSL functionality so that the Python SSLAuthentication filter has minimal work to do.   This scheme has been tested with client certificates obtained with a MyProxy CA service.  Short lived certificates obtained are issued direct from a CA hosted with MyProxy.   This presents no problems for Apache mod_ssl configuration.  Making a request with a proxy certificate however yields:
sslv3 alert certificate unknown
The OpenSSL code used by mod_ssl doesn't understand proxy certificates and key to the verification process, that it must verify a chain passed by the client of proxy certificate and the user certificate which issued the proxy certificate.   Fortunately from OpenSSL 0.9.8g there's a hack to allow support for proxy certificates, simply setting the environment variable to some value:
export OPENSSL_ALLOW_PROXY_CERTS=1
For Apache it means, setting this in the environment of the Apache start-up.  This NCSA howto is really helpful.  For my Ubuntu Apache configuration, there's an envvars file to which this setting can be added in /etc/apache2/.

The client verification settings are made in the configuration file:
SSLOptions +ExportCertData
SSLVerifyClient optional
SSLVerifyDepth  10
SSLCACertificatePath    /etc/grid-security/certificates/
Verification is optional so that OpenID based sign in can be invoked if no client certificate is submitted. ExportCertData enables the middleware to examine the certificate and use it to authenticate the client e.g. based on certificate Distinguished Name.

Adding proxy certificate support then has proved fairly painless for the Python/Apache server side implementation.  There's a parallel Java Tomcat implementation for ESG.  Let's hope this proves as straightforward.


Thursday, 29 April 2010

New Python MyProxyClient Release.

I've just released a new version of this Python based implementation of the client interface to the MyProxy
credential management service.  

In this release I've added a new method getTrustRoots to support the ability to download the CA certificates for a given MyProxy server (command=7 - see the protocol page).  I've also switched from M2Crypto to PyOpenSSL for the bindings to OpenSSL.  M2Crypto offers a broad range of the API but memory and installation issues have finally put me off.

Put is still not supported as unfortunately, the PyOpenSSL X.509 Extensions interface doesn't support the proxyCertInfo extension type needed for creating proxy certificates.

This simple example shows how to obtain credentials:

$ python 
Python 2.6.4 (r264:75706, Dec  7 2009, 18:45:15) 
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information. 
>>> from myproxy.client import MyProxyClient 
>>> clnt = MyProxyClient(hostname='myproxy.somewhere.ac.uk', caCertDir='/home/testuser/.globus/certificates')
>>> from getpass import getpass
>>> creds = clnt.logon('testuser', getpass())
Password:

The certificate chain and private key are returned in the "creds" tuple. The "caCertDir" option points to a directory containing the trust roots so that the client authenticate the peer.

Friday, 23 April 2010

EGU2010

The EGU General Assembly is fast approaching. I'm giving these two presentations:
  1. Applying the Earth System Grid Security System in a Heterogeneous Environment of Data Access Services
    Session: ESSI8 Service-Oriented Architecture solutions for Earth and Space Sciences
  2. A Flexible Component based Access Control Architecture for OPeNDAP Services
    Session: ESSI13 Scientific Gateways and Visualization