Proxy Certificates and Delegation with OPeNDAP Services

I'm back from EGU in Vienna and one topic I presented on was a solution for securing OPeNDAP services which I've developed working with colleagues at STFC and members of the development team for Earth System Grid. Services configured with the security filter middleware can accept either PKI or OpenID based authentication. So far so good but one piece that's missing and that would greatly enhance the solution would be support for delegation.

Proxy certificates would be the obvious choice since this should easily piggyback on the existing PKI based authentication adopted but alternative technologies like OAuth need more exploration. Just how easy the proxy certificate solution is explored here but first why bother? - It would give immediate compatibility with Grid based authentication schemes and would solve the delegation problem. For example in an upcoming project here a portal is required which is effectively an experimental area for scientists to gather data from different sites and combine it with their own that they upload to it. The portal could query OPeNDAP services on behalf of the user for data retrieval. In the case of a secured OPeNDAP service, the user would need to delegate authority to the portal to use their credentials to access the OPeNDAP service on their behalf.

The sequence below shows how security filter middleware intercept requests to an OPeNDAP service and redirect unauthenticated clients to an Authentication Service running over SSL.  At this point if the client submits a certificate in the SSL handshake, the service uses this to authenticate the user.  If no certificate is provided, it defaults to OpenID based sign in.

This Python snippet shows how the client code can work:

>>> import urllib2
>>> from M2Crypto import m2urllib2, SSL
>>> ctx = SSL.Context('sslv3')
>>> ctx.load_cert_chain('./proxy.pem')
>>> opener = m2urllib2.build_opener(ctx)
>>> data = opener.open('http://localhost:8001/dap/sample.nc')

urllib2's redirect handler means that it will conveniently follow the HTTP redirects required in the sequence.  M2Crypto plugs into this to provide the SSL support.   The SSL context uses load_cert_chain to ensure that the complete certificate chain is loaded for the proxy certificate.  For simplicity here no peer certificate validation code has been set.

The server side filters shown in the diagram are implemented as Python WSGI middleware hosted under Apache with mod_wsgi.   The OPeNDAP service is PyDAP.   Apache provides the SSL functionality so that the Python SSLAuthentication filter has minimal work to do.   This scheme has been tested with client certificates obtained with a MyProxy CA service.  Short lived certificates obtained are issued direct from a CA hosted with MyProxy.   This presents no problems for Apache mod_ssl configuration.  Making a request with a proxy certificate however yields:

sslv3 alert certificate unknown

The OpenSSL code used by mod_ssl doesn't understand proxy certificates and key to the verification process, that it must verify a chain passed by the client of proxy certificate and the user certificate which issued the proxy certificate.   Fortunately from OpenSSL 0.9.8g there's a hack to allow support for proxy certificates, simply setting the environment variable to some value:

export OPENSSL_ALLOW_PROXY_CERTS=1

For Apache it means, setting this in the environment of the Apache start-up.  This NCSA howto is really helpful.  For my Ubuntu Apache configuration, there's an envvars file to which this setting can be added in /etc/apache2/.

The client verification settings are made in the configuration file:

SSLOptions +ExportCertData
SSLVerifyClient optional
SSLVerifyDepth  10
SSLCACertificatePath    /etc/grid-security/certificates/

Verification is optional so that OpenID based sign in can be invoked if no client certificate is submitted. ExportCertData enables the middleware to examine the certificate and use it to authenticate the client e.g. based on certificate Distinguished Name.

Adding proxy certificate support then has proved fairly painless for the Python/Apache server side implementation.  There's a parallel Java Tomcat implementation for ESG.  Let's hope this proves as straightforward.

New Python MyProxyClient Release.

I've just released a new version of this Python based implementation of the client interface to the MyProxy
credential management service.  

In this release I've added a new method getTrustRoots to support the ability to download the CA certificates for a given MyProxy server (command=7 - see the protocol page).  I've also switched from M2Crypto to PyOpenSSL for the bindings to OpenSSL.  M2Crypto offers a broad range of the API but memory and installation issues have finally put me off.

Put is still not supported as unfortunately, the PyOpenSSL X.509 Extensions interface doesn't support the proxyCertInfo extension type needed for creating proxy certificates.

This simple example shows how to obtain credentials:

$ python 
Python 2.6.4 (r264:75706, Dec  7 2009, 18:45:15) 
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information. 
>>> from myproxy.client import MyProxyClient 
>>> clnt = MyProxyClient(hostname='myproxy.somewhere.ac.uk', caCertDir='/home/testuser/.globus/certificates')
>>> from getpass import getpass
>>> creds = clnt.logon('testuser', getpass())
Password:

The certificate chain and private key are returned in the "creds" tuple. The "caCertDir" option points to a directory containing the trust roots so that the client authenticate the peer.

EGU2010

The EGU General Assembly is fast approaching. I'm giving these two presentations:

1. Applying the Earth System Grid Security System in a Heterogeneous Environment of Data Access Services
   Session: ESSI8 Service-Oriented Architecture solutions for Earth and Space Sciences
• Abstract
2. A Flexible Component based Access Control Architecture for OPeNDAP Services
   Session: ESSI13 Scientific Gateways and Visualization
• Abstract

MyProxy logon via OpenID

The access control architecture for CMIP5 makes use of OpenID and the MyProxy Certificate Authority for single sign on. Recently work has been underway to provide filters to secure data server applications with both OpenID and SSL client based authentication front ends. In my case helped along considerably by the WSGI based architecture adopted.

An important requirement has been to enable the user community to access data via a browser but also via scripts such as wget and dedicated client software. The issue of delegation will have to be tackled at some point and one example scenario is Live Access Server with accessing TDS instances on behalf of a user. Also in the future perhaps workflows with OGC services. There has been a lot of work done in this area already.

Given we have PKI based authentication incoporated a 'classic' grid based solution with proxy certificates would be possible. Maybe OAuth might be another avenue to look into.

One possibility with OpenID, would be to exploit Attribute Exchange (AX) extension to perform an MyProxy logon over this interface. A Relying Party could obtain a user certificate returned across the AX interface from the user's OpenID Provider (OP). MyProxy logon normally involves,

1. creation of a key pair at the client
2. creation of a certificate request containing the public key
3. authentication of the client against a MyProxy Server
4. submission of certificate request to the server
5. certificate issued by the server and returned to the client

Once obtained the client can use the PKI credentials to authenticate against other services. In this case an OpenID Relying Party (RP) represents a service which wants to access other services on behalf of the user. It could perform steps 1 and 2 but then pass the certificate request to the Provider during the sign in process. Fortunately the AX spec has a store message feature which could make this possible.

When the user signs in at the Provider, they provide username/password as usual, but the Provider links with its own MyProxy Server and makes a logon client call passing username, password and the certificate request obtained from the RP. If all is well, MyProxy returns a new certificate and the Provider can then pass this back to the RP over the AX interface. The RP now has PKI credentials for the user delegated to it so that it can make calls to other services on the user's behalf. Of course this method only provides one level of delegation but that may be sufficient for many use cases.

The sequence diagram below illustrates the steps:

WSGI Architecture for NERC DataGrid Security

Over the past year I've been porting the Python based security system for NERC DataGrid to a WSGI based architecture. This is paying huge dividends in terms of the modularity of the code and ability to apply flexible deployment configurations especially when used together with Paste Deploy.

Key parts to the architecture are the authentication and authorisation handlers triggered from the respective 401 and 403 HTTP response codes together with a URI based access control policy. I've used the Python security middleware package AuthKit to help me put this together. One thing I've been meaning to do is to lay this out in a simple example. This first snippet gives an overview:

app = AuthorisationPolicyMiddleware(myApp)

app = MultiHandler(app)
app.add_method("checkerID", AuthenticationHandlerMiddleware)
app.add_checker("checkerID", AuthenticationHandlerMiddleware.trigger)

app = MultiHandler(app)
app.add_method("checkerID", AuthorisationHandlerMiddleware)
app.add_checker("checkerID", AuthorisationHandlerMiddleware.trigger)

from paste.httpserver import serve
from paste.deploy import loadapp

serve(app, host='0.0.0.0', port=9080)

The application to be protected is defined in a WSGI elsewhere. This is wrapped in a number of pieces of middleware chained together to form a pipeline to intercept requests to the application. On the last line it is served using Paste.

The first middleware component listed, AuthorisationPolicyMiddleware, checks the requested URI against a policy. If the user is not authorised, it sets a HTTP "403 Forbidden" response bypassing myApp.

Following this, there are two pieces of middleware making use of AuthKit's authkit.authenticate.multi.Multihandler. The MultiHandler accepts two key inputs: a checker function which determines the criteria for intercepting a request, and a method, a WSGI middleware to determine what action to take once an intercept has been made.

In the first case, a class method AuthenticationHandlerMiddleware.trigger has been defined to intercept HTTP "401 Unauthorized" status codes. The AuthenticationHandlerMiddleware itself determines the action taken:

class AuthenticationHandlerMiddleware(object):
    """Handler for HTTP 401 Unauthorized responses"""

    triggerStatus = "401 Unauthorized"

    def __init__(self, global_conf, **app_conf):
        pass

   def __call__(self, environ, start_response):
        log.info("AuthenticationHandlerMiddleware access denied response ...")
        response = "HTTP 401 Unauthorised response intercepted"
        start_response('200 OK', [('Content-type', 'text/plain'),
                                  ('Content-length', str(len(response)))])
        return [response]

    @classmethod
    def trigger(cls, environ, status, headers):
        if status == cls.triggerStatus:
          log.info("Authentication Trigger caught status [%s]",
                   cls.triggerStatus)
          return True
       else:
          return False

In the above, the middleware simply outputs a message but it effectively provides a hook to trigger a login or other authentication interface.

A second Multihandler is in place to handle HTTP "403 Forbidden" responses. This follows a similar pattern:

class AuthorisationHandlerMiddleware(object):
    """Handler for HTTP 403 Forbidden responses"""

    triggerStatus = "403 Forbidden"

    def __init__(self, global_conf, **app_conf):
       pass

    def __call__(self, environ, start_response):
       log.info("AuthorisationHandlerMiddleware access denied response ...")
       response = "HTTP 403 Forbidden response intercepted"
       start_response('200 OK', [('Content-type', 'text/plain'),
                                 ('Content-length', str(len(response)))])
       return [response]

    @classmethod
    def trigger(cls, environ, status, headers):
       if status == cls.triggerStatus:
           log.info("Authorisation Trigger caught status [%s]",
                    cls.triggerStatus)
           return True
       else:
           return False

The trigger method sets a True response to signal to the Multihandler to intercept the request and invoke AuthorizationMiddleware to deliver an access denied message.

This next snippet shows myApp effectively a test harness to demonstrate the middleware behaviour:

def myApp(environ, start_response):
    """Test application to be secured"""

    if environ['PATH_INFO'] == "/test_401":
       status = "401 Unauthorized"
       response = status

   elif environ['PATH_INFO'] == "/test_403":
       status = "403 Forbidden"
       response = status

   elif environ['PATH_INFO'] == "/secured":
       status = "200 OK"
       response = "Secured URI"

   else:
       status = "404 Not Found"
       response = status

   log.info("Application is setting [%s] response..." % status)
   start_response(status,
                  [('Content-type', 'text/plain'),
                   ('Content-length', str(len(response)))])

   return [response]

As set-up above,


1. http://localhost:9080/test_401 will trigger the authentication middleware and
   
2. http://localhost:9080/test_403 the authorisation middleware.
   
3. The last, http://localhost:9080/test_secured, demonstrates the access control policy implemented in AuthorisationPolicyMiddleware:

class AuthorisationPolicyMiddleware(object):
   """Apply a security policy based on the URI requested"""

   def __init__(self, app):
       self.securedURIs = ['/test_secured']
       self.app = app

   def __call__(self, environ, start_response):
       if environ['PATH_INFO'] in self.securedURIs:
           log.info("Path [%s] is restricted by the Authorisation policy" %
                    environ['PATH_INFO'])
           status = "403 Forbidden"
           response = status
           start_response(status, [('Content-type', 'text/plain'),
                                   ('Content-length', str(len(response)))])
           return [response]
       else:
           return self.app(environ, start_response)

The middleware has a policy consisting of a list of URIs to be secured in the securedURIs attribute. In practice this could link to a policy file, database link or some other interface. The __call__ method intercepts request URIs which match the policy and invokes a HTTP 403 response. This in turn brings into play the AuthorisationMiddleware handler triggering it to return an access denied response.

The complete example is in the NERC DataGrid SubVersion repository.

Python List Utility Classes

I've been adapting some Java code to Python recently and wanted some tighter control over list elements than the default list type. I've extended list with two custom classes. The first, TypedList restricts list elements to a given type or types e.g.

>>> t=TypedList(float)
>>> t+=[9]
Traceback (most recent call last):
File "", line 1, in 
File "", line 34, in __iadd__
TypeError: List items must be of type float

The existing array type gives similar capability but with this you can put in any type ...

class TypedList(list):
    """Extend list type to enabled only items of a given type.  Supports
    any type where the array type in the Standard Library is restricted to
    only limited set of primitive types
    """

    def __init__(self, elementType, *arg, **kw):
        """@type elementType: type/tuple
        @param elementType: object type or types which the list is allowed to
        contain.  If more than one type, pass as a tuple
        """
        self.__elementType = elementType
        super(TypedList, self).__init__(*arg, **kw)

    def _getElementType(self):
        return self.__elementType

    elementType = property(fget=_getElementType,
                           doc="The allowed type or types for list elements")

    def extend(self, iter):
        for i in iter:
            if not isinstance(i, self.__elementType):
                raise TypeError("List items must be of type %s" %
                                (self.__elementType,))

        return super(TypedList, self).extend(iter)

    def __iadd__(self, iter):
        for i in iter:
            if not isinstance(i, self.__elementType):
                raise TypeError("List items must be of type %s" %
                                (self.__elementType,))

        return super(TypedList, self).__iadd__(iter)

    def append(self, item):
        if not isinstance(item, self.__elementType):
            raise TypeError("List items must be of type %s" %
                            (self.__elementType,))

        return super(TypedList, self).append(item)

For the second class I wanted a way of avoiding the addition of duplicate elements to a list:

>>> u=UniqList()
>>> u.append('a')
>>> u
['a']
>>> u.append('a')
>>> u
['a']

It silently ignores the duplicate element. It would be straightforward to alter to raise an exception if this was the preferred behaviour. Here's the class:

class UniqList(list):
    """Extended version of list type to enable a list with unique items.
    If an item is added that is already present then it is silently omitted
    from the list
    """
    def extend(self, iter):
        return super(UniqList, self).extend([i for i in iter if i not in self])

    def __iadd__(self, iter):
        return super(UniqList, self).__iadd__([i for i in iter
                                               if i not in self])

    def append(self, item):
        for i in self:
            if i == item:
                return None

        return super(UniqList, self).append(item)

PyDev, PyLint and Refactoring

I've been using PyDev the Eclipse plugin for Python for some time now and it's certainly improving with each new release (I'm currently on 1.4.2). There are a couple of features I've been getting some benefit from recently.

With PyLint, I actually broke my Eclipse set-up for this a while ago but when I got it reinstated it's reminded me of how useful it is. One thing I hate about about interpreted languages is the huge scope for runtime errors. The PyLint plugin gives immediate feedback in the margin of the editor window with error information and hints. This is saving me a lot of time down the line. What's the metric for time spent fixing a bug immediately vs. at the other end of the scale trying to pick through code on a production deployment? :) Picking up references to undeclared variables is particularly useful. Especially in error blocks or places where test coverage might miss out. There's no doubt some messages can be annoying and getting a mark out of 10 for your code might not appeal to all. Fortunately, you can edit the settings for the Pylint command line that's executed and explicitly filter out warnings you don't like. The second feature I've revisited is the PyDev refactoring options. I have had a low expectation of these perhaps unfairly but there a couple of potential time savers. Take this simple class:

class MyTest(object):
    def doSomething(self):
        self.__a = None
        self.__b = None
        self.__c = []

Left like this PyLint will warn me that I haven't set up these attributes in an __init__ method. ;) PyDev refactoring to the rescue :) ... If I now right click in the editor and pick "Refactoring" -> "Generate Constructor using Fields ...". I'm taken through a series of steps in a dialog. I can get,

class MyTest(object):
    def __init__(self, a, b, c):
        self.__a = a
        self.__b = b
        self.__c = c

    def doSomething(self):
        self.__a = None
        self.__b = None
        self.__c = []

Not that exciting but could save some typing. More useful for me is the "Generate Properties..." to generate properties for new style classes together with their getters and setters:

class MyTest(object):
    def __init__(self, a, b, c):
        self.__a = a
        self.__b = b
        self.__c = c

    def getA(self):
        return self.__a

    def getB(self):
        return self.__b

    def getC(self):
        return self.__c

    def setA(self, value):
        self.__a = value

    def setB(self, value):
        self.__b = value

    def setC(self, value):
        self.__c = value

    def delA(self):
        del self.__a

    def delB(self):
        del self.__b

    def delC(self):
        del self.__c

    def doSomething(self):
        self.__a = None
        self.__b = None
        self.__c = []

    a = property(getA, setA, delA, "A's Docstring")

    b = property(getB, setB, delB, "B's Docstring")

    c = property(getC, setC, delC, "C's Docstring")

It could be improved on - there's no explicit keywords inputs for property() - but it could save some time with the boiler plate. :)