tag:blogger.com,1999:blog-80700626275213783612024-02-19T12:39:05.477+00:00Philip KershawPhilip Kershawhttp://www.blogger.com/profile/05141787349333619736noreply@blogger.comBlogger31125tag:blogger.com,1999:blog-8070062627521378361.post-84962266722637131772013-01-26T22:15:00.000+00:002013-01-26T22:16:12.281+00:00Helix Nebula, the Science Cloud - Results and User Engagement Meeting<style type="text/css">
<!--
@page { margin: 2cm }
P { margin-bottom: 0.21cm }
</style> <br />
<div style="margin-bottom: 0cm;">
A write up from the notes I took at this meeting last week at ESRIN. This is my first closer look at this project (<a href="http://helix-nebula.eu/">http://helix-nebula.eu/</a>) after hearing bits and piece along the way. It's interesting on a number of levels - application of science use cases to cloud, academic-industrial partnership, federating clouds, service management and the European landscape - policy, legal issues and politics.</div>
<div style="margin-bottom: 0cm;">
<br /></div>
<div style="margin-bottom: 0cm;">
Helix Nebula has big ambitions – with the aim
to make it the first port of call for cloud provision for the
research community by 2020: development as a multi-tenant open market
place, the creation of an eco-system to facilitate growth for both
science and industry. Science users are the
initial target with use cases but this is fully expected to expanded to a wider user
base. One member of the consortium noted the increased attendance at
meetings over the course of the project as an encouraging sign and
there has been a growth in membership for 20 to 34 members over the
past year. There are there large players from the science community: CERN, EMBL (European Molecular Biology Laboratory)
and ESA. Collaboration is seen as an essential element to meet the
goals drawing from the various stakeholders IT providers, SMEs,
Science and Space organisations and EC support and projects. Some of the commercial companies represented: Atos, CloudSigma, T-Systems, Terradue, Logica. EGI
representing cloud provision via the academic community.</div>
<div style="margin-bottom: 0cm;">
</div>
I've heard put before the argument for a collective approach as
the only means to develop something which is in any way comparable
with the establish big cloud providers. Here the case was put
forward, that drawing the collective resources of multiple providers
together can build scale. A new one to me also, the aspect of risk sharing. A federated approach has the potential to build something stronger since it has
the advantage of diversity over a single cloud provider with
comparable resource. There are also political aspects: EC
compliance re. privacy, policy and ethics and the ability to meet
these by hosting resources exclusively within the EU's boundaries.<br />
<br />
<h3>
Technical Architecture </h3>
<div style="margin-bottom: 0cm;">
The technical architecture has been developed by representatives from
the <i>supply</i>-side a Tech-Arch group. This was a little concerning but then I found
out that there are actually two, a complementary Serve-Arch group
so that the demand side is represented also. It was reassuring to hear that the
user interests are represented in this way but could there just have
been one group and how does it work in practice having the two? Hearing more about the requirements analysis, it sounded more holistic with user stories created around a set of actors identified from both supply and demand-side.</div>
<br />
A service-enabling framework, is the blue-print, a federated system
of cloud providers.
The so-called 'Blue-Box' (apparently so-called because when first
put forward the relevant Powerpoint slides had a default blue
scheme!). This is an interface between the many consumers and
providers harmonising the interfaces of the various cloud providers.
This was all starting to sound very familiar coming from my
experience with <a href="http://contrail-project.eu/">Contrail</a> and this was confirmed 'opening' the Blue-Box it revealed,
users at the 'north'-end an SOA to manage the various
federation-level cloud capabilities in the middle and then below this
at the 'south'-end APIs to the various cloud provider interfaces. This system is being constructed via a roadmap involving three
releases. This is building capability incrementally using reference
technologies as building blocks and identifying low hanging fruit
from which to prioritise work. The federation layer is then a thin one. I can see major issues to tackle - for example management of SLAs across multiple providers - but the approach at least starts with what is possible.<br />
<br />
I was encouraged to hear about this federated approach but coming
to the meeting my feeling was that it's surely not in the interests of
commercial cloud providers to harmonise their interfaces and make a
federation with their other competitors possible. There's further
inertia though: in creating a generic interface for a number of
different cloud providers, there's a danger that the federation layer
generalises or dilutes their individual capabilities. All cloud
providers are not made equal and each may have specific capabilities
which make them stand out and give them a particular market edge.
At worst, a cloud provider might be being shoe-horned into a system
that dilutes this or gives a vanilla or sub-optimal exposure of their
functionality through the federation layer to the users sitting at
the North-end.<br />
<br />
This brings me to the input from EGI. I would there is a greater potential to develop a federated system since its
strongly in the interest of the academic sector. Matteo Turilli
(OeRC) gave an overview of current developments. Private clouds are
being deployed within EGI with twenty providers from across Europe.
A cloud profile has been developed as a standard and a testbed
established aiming to build a federation across the private clouds
represented in the community. The heritage of a production Grid is providing an
infrastructure upon which to build. Solutions for federated identity management are already established with Shibboleth and GSI. I don't see support for such technology amongst commercial cloud providers. I suppose there is no incentive but it means that at the interface between federation layer and cloud provider, the best the federation can do is cache the user's credentials for each cloud provider and hope the user doesn't mind. To paraphrase from a social networking site, 'you can trust us with your password, we won't pass it on to anyone else'. ;)<br />
<br />
OCCI 1.1 has been adopted for
the cloud standard and various cloud solutions are in use: OpenNebula, OpenStack, WNoDes, Okeanos. IaaS is the initial service
model being offered augmented with various services including
capability for VM sharing. Demonstrations of the federated system
are carried out every six months with 3 completed to date. How does this fit in with the Helix Nebula Blue-box? A strategy
of a thin federation layer is being adopted to avoid the complex
problems that will inevitably arise with deeper integration. There
are problems of course. OCCI does not seem to be getting much
traction in the commercial community and the perception seemed to be
that it does not have the level of granularity in its interface to
communicate more detailed user needs. jclouds was mooted as a
possible alternative. It too exposes a REST-based interface.<br />
<br />
<ul></ul>
<h3>
Flagships</h3>
There were presentations from each of the three flagships: CERN, EMBL and ESA. I don't have space to go through these in depth here but the presentations are available on the <a href="http://www.helix-nebula.eu/">Helix Nebula site</a>. CERN's use case used results from the ATLAS experiment. Interesting that processing jobs had low i/o. Network bandwidth between providers and CERN was not an issue. ESA's case seemed to be an expansion of the SSEP (Supersites Exploitation Platform) looking at disaster response scenarios where processing needs to be very rapid. <a href="http://www.helix-nebula.eu/uploads/4.%20EMBL%20Flagship.pdf">EMBL's large scale genome analysis</a> was particularly impressive handling 100,000s of jobs auto-provisioned on 50 node clusters. GlusterFS was used for shared file system and <a href="http://star.mit.edu/cluster/">StarCluster</a> from MIT for processing. This manages provisioning of images and setting up of a cluster with capability for fluctuating workloads. It also integrates with Sun Grid Engine. EMBL also had a nice user interface.<br />
<br />
It was revealing hearing some feedback on the supplier-side perspective to the flagships. The technology stacks in use by the cloud providers included:<br />
<ul>
<li>StratusLab OpenNebula with KVM hypervisor<br />
</li>
<li>Zimory with VMware and vCloud<br />
</li>
<li>Abiquo with ESX and KVM hypervisors</li>
</ul>
Customers were presented with four different interfaces re-emphasising the need for federation layer. There's a need to find a means to share VMs between different vendor flavours. Other issues that came up: the differing accounting models between suppliers, the need for logic to handle workload assignment and scaling between clouds and perhaps most interesting: revenues were small for suppliers and the need to convince management and stakeholders of the longer term benefits and opportunities.<br />
<br />
For future work, the Tech-Arch carried out some analysis of candidate technologies picking out SlipStream (Open Source, European) and enStratus (proprietary, US) from a list which also included OpenNebula and BonFIRE. <br />
<br />
More Flagships are in the on the way so it will be interesting to follow:<br />
<ul>
<li><a href="http://www.helix-nebula.eu/uploads/PIC%20candidate%20flagship.pdf">PIC Neuroimaging Center on Cloud, PIC (Port d'Informació Científica)</a></li>
<li><a href="http://www.helix-nebula.eu/uploads/8.%20UNESCO%20Candidate%20Flagship.pdf">Ocean and Costal Information Supersite, UNESCO</a></li>
<li><a href="http://www.helix-nebula.eu/uploads/9.%20ecmwf%20candidate%20flagship.pdf">Weather Data Information Supersite, ECMWF</a></li>
</ul>
<br />
<br />Philip Kershawhttp://www.blogger.com/profile/05141787349333619736noreply@blogger.com0tag:blogger.com,1999:blog-8070062627521378361.post-37919254193822321222012-09-03T20:54:00.000+01:002012-09-03T20:54:52.979+01:00ndg_oauth for PyPIAfter a long silence this is a catch up on things OAuth-related. Firstly, a follow up to announce that ndg_oauth is now finally moved to PyPI for its first 'official' release. There are separate client and server packages. The latter containing the functionality for both authorisation and resource server:<br />
<br />
<a href="http://pypi.python.org/pypi/ndg-oauth-client/">http://pypi.python.org/pypi/ndg-oauth-client/</a><br />
<a href="http://pypi.python.org/pypi/ndg-oauth-server/">http://pypi.python.org/pypi/ndg-oauth-server/</a><br />
<br />
Before any more on that though I should really say something about the recent <a href="http://hueniverse.com/2012/07/oauth-2-0-and-the-road-to-hell/">news</a> with OAuth 2.0, something that colleagues nudged me about. It's disappointing but then maybe signs were there. When I first became familiar with the draft for 2.0, it looked like it would be easier to implement and it would meet a broader set of use cases. Two things appealed particularly: the simplification of the message flow and the use of transport layer instead of message-level security. Having had experience wrestling with different implementations of WS-Security to get them to interoperate and particularly XMLSec digital signature, it was relief to not to have to deal with that in this case. Also, transport layer security is established in Grid security and federated identity worlds and so is a known quantity.<br />
<br />
All the same, the reported problems with OAuth 1.0 and signatures seems a little surprising. Consistent canonicalisation of a HTTP header for OAuth 1.0 seems a much easier prospect vs. XML signature. It hardly seems such a difficult requirement to implement. Message level security also provides some protection for the redirects in use: securing at the transport layer will give me secure channels between user agent and authorisation server and user agent and OAuth client but the content can still be tampered in between these interactions at the user agent itself. <br />
<br />
Our use of OAuth has definitely been taking it beyond the simple web-based use cases that I've seen for 1.0. When I say 'our' I'm thinking of its application for Contrail and some interest shown by others in the academic research community. In our case, we can exploit version 2.0 to address the multi-hop delegation required to fit together the various actors in the layered architecture for Contrail or it could be applied to manage delegation across multiple organisations in a federated infrastructure like ESGF. These cases need support for non-browser based clients and for actions executed asynchronously, removed from direct user interactions. These are in effect enterprise use cases for the academic communities involved. <br />
<br />
As I looked over the draft specification with colleague Richard and he started an implementation it became clear that there was opening of possibilities to support many use cases but with it a concern that when it came to an implementation there were gaps to fill in and choices to be made about how best to implement. It was clear from that it would need careful profiling if it was to be of wider use in our communities but this is familiar ground in geospatial informatics and I'm sure elsewhere to. <br />
<br />
A consensus is badly needed but I think that's practicable and can be a process. By making an implementation, and working with other collaborators we are making the first steps towards standardising a version 2.0 profile for use with a Short-Lived Credential Service. This was the <a href="http://philipkershaw.blogspot.co.uk/2011/11/oauth-myproxy-mash-up.html">original motivating driver</a> - to provide a means for delegation of user X.509 certificates. So returning to the ndg_oauth implementation, it aims to fit this purpose but at the same time provide a baseline of more generic functionality that others can extend and build on. More to follow! ...<br />
<br />
<br />Philip Kershawhttp://www.blogger.com/profile/05141787349333619736noreply@blogger.com2tag:blogger.com,1999:blog-8070062627521378361.post-89166264238187914172012-04-24T19:35:00.001+01:002012-04-25T14:18:32.308+01:00OAuth Service Discovery<br />
<div style="margin-bottom: 0cm;">
This picking up from where I left off in any <a href="http://philipkershaw.blogspot.com/2011/11/oauth-myproxy-mash-up.html">earlier post about OAuth</a>. To recap, this
fronts a short-lived credential service with an OAuth 2.0 interface.
This is now being integrated into Contrail described in <a href="http://philipkershaw.blogspot.com/p/research.html">this paper</a>
just submitted following ISGC. There are few more design
considerations to give some thought to though for its wider deployment. As well as use for
managing delegation around entities in a cloud federation with
Contrail, we are applying this to <a href="http://ceda.ac.uk/">CEDA</a>'s applications and web
services. We have a full <a href="http://ndg-security.ceda.ac.uk/browser/trunk/ndg_oauth">Python implementation</a> of both client and
server side for OAuth 2.0 which I hope to push out to PyPI soon.
We're using this to secure our CEDA OGC Web Processing Service and
and Web Map Service. Here though, I want to focus on how we might
integrate with federated identity management system for the Earth System Grid Federation (ESGF). This
is foremost given our involvement with CMIP5.</div>
<div style="margin-bottom: 0cm;">
<br /></div>
<div style="margin-bottom: 0cm;">
It's firstly a question of <i>service
discovery</i>, something that need not be addressed for Contrail. The
Contrail system consists of federation layer which abstracts a set of
underlying cloud provider interfaces. Users can manage resources
from each of these collectively through the federation's interface.
The access control system manages identity at the federation level.
A user authenticates with a single federation IdP. When it comes to
the OAuth delegation interface, there are single OAuth Authorisation server and Resource server instances associated with the IdP.</div>
<div style="margin-bottom: 0cm;">
<br /></div>
<div style="margin-bottom: 0cm;">
Now consider, ESGF. There are many
IdPs integrated into the federation, so how does the OAuth infrastructure map to this? Imagine,
for example a Web Processing Service hosted at CEDA. A user invokes
a process. This process will access secured data from an OPeNDAP
service. The WPS will therefore need a delegated credential. When
the user requests the process then, the WPS triggers the OAuth flow
redirecting their browser to an OAuth Server but how does the WPS
know where that service is? A simple approach would be to configure
the WPS with an endpoint in its start up settings. This server could
be:</div>
<ol>
<li><div style="margin-bottom: 0cm;">
a global one for the whole
federation – clearly this won't scale given the deployment across many organisations</div>
</li>
<li><div style="margin-bottom: 0cm;">
Associated with the given Service
Provider – in this case CEDA</div>
</li>
<li><div style="margin-bottom: 0cm;">
Associated with the user's IdP</div>
</li>
</ol>
<div style="margin-bottom: 0cm;">
2. could work but now consider the
other steps in the flow: the user is prompted to authenticate with
the OAuth server and approve the delegation i.e. allow the WPS to get
a credential on their behalf. In a federated system, the user could
be from anywhere, they will need to sign in with their IdP. In this
case then, the user enters their OpenID at the OAuth Authorisation server.
Already, this is sounding complicated and potentially confusing for
the user: they've already been redirected away from the WPS to a
central OAuth Authorisation server at CEDA, now they will be redirected again to
their IdP probably somewhere else.</div>
<div style="margin-bottom: 0cm;">
<br /></div>
<div style="margin-bottom: 0cm;">
Also, consider the approval process.
Rather then having to approve the delegation every single time the user
invokes the WPS process, they would like the system to remember their approval
decision. The OAuth authorisation server could record a set of user approvals
associated with a profile stored their. However, if we manage approvals at
CEDA, this information will only be scoped within the bounds of
CEDA's services. If the user now goes to use services at another
service provider, they will need to record a fresh set of delegation
approvals. Not a smooth user experience.</div>
<div style="margin-bottom: 0cm;">
<br /></div>
<div style="margin-bottom: 0cm;">
Turning to 3., this could scale better.
All approval decisions would be centralised with the OAuth Authorisation server
associated with their IdP. However, there is now a service discovery
problem. Looking at the protocol flow again, the WPS does not know
the location of the user's OAuth Authorisation server endpoint. Given ESGF's already established use of OpenID,
an obvious solution is to <a href="http://philipkershaw.blogspot.com/2010/06/introspecting-identity-with-yadis.html">leverage Yadis</a>. During OpenID sign in, the Relying Party HTTP GETs the user's OpenID URL returning an XRDS document. This contains the service endpoint for the OpenID Provider. This has already been extended for ESGF to include the MyProxy and Attribute Service endpoints. I'm told that there's some debate in the community about choice of discovery technology with SWD (Simple Web Discovery) as an alternative. Given ESGF's legacy with XRD it makes logical sense to add in the address for the OAuth Authorisation server as an entry into the XRDS document returned. </div>
<pre class="wiki"><pre style="background-color: #eeeeee; border: 1px solid rgb(153, 153, 153); color: black; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>
<?xml version="1.0"; encoding="UTF-8"?>
<xrds:XRDS xmlns:xrds="xri://$xrds" xmlns="xri://$xrd*($v*2.0)">
<XRD>
<Service priority="0">
<Type>http://specs.openid.net/auth/2.0/signon</Type>
<Type>http://openid.net/signon/1.0</Type>
<URI>https://openid.provider.somewhere.ac.uk</URI>
<LocalID>https://somewhere.ac.uk/openid/PJKershaw</LocalID>
</Service>
<Service priority="1">
<Type>urn:esg:security:myproxy-service</Type>
<URI>socket://myproxy-server.somewhere.ac.uk:7512</URI>
<LocalID>https://somewhere.ac.uk/openid/PJKershaw</LocalID>
</Service>
<Service priority="10">
<Type>urn:esg:security:oauth-authorisation-server</Type>
<URI>https://oauth-authorisation-server.somewhere.ac.uk</URI>
<LocalID>https://somewhere.ac.uk/openid/PJKershaw</LocalID>
</Service>
<Service priority="20">
<Type>urn:esg:security:attribute-service</Type>
<URI>https://attributeservice.somewhere.ac.uk</URI>
<LocalID>https://somewhere.ac.uk/openid/PJKershaw</LocalID>
</Service>
</XRD>
</xrds:XRDS></code></pre>
</pre>
<div style="margin-bottom: 0cm;">
Bringing it together, the delegation flow would start with the Service Provider present an interface to enable the user to select their IdP. This could be by entering a full OpenID URL or preferably picking one from a list of trusted IdPs. The SP could then GET the URL and extract the OAuth Authorisation Server endpoint from the XRDS document returned. From there, the standard OAuth flow would proceed as before.</div>
<div style="margin-bottom: 0cm;">
</div>
<div style="margin-bottom: 0cm;">
<br /></div>Philip Kershawhttp://www.blogger.com/profile/05141787349333619736noreply@blogger.com0tag:blogger.com,1999:blog-8070062627521378361.post-89453337557395974772012-02-09T22:46:00.003+00:002012-02-09T22:46:54.464+00:00New Python HTTPS ClientI've added a new HTTPS client to the ndg_* list of packages in PyPI. <span style="font-family: "Courier New",Courier,monospace;">ndg_httpsclient</span> as it's become has been on a todo list for a long time. I've wanted to make use of all the features PyOpenSSL offers with convenience of use wrapping it in the standard <span style="font-family: "Courier New",Courier,monospace;">httplib</span> and <span style="font-family: "Courier New",Courier,monospace;">urllib2</span> interfaces. <br />
<br />
Here's a simple example using the <span style="font-family: "Courier New",Courier,monospace;">urllib2</span> interface. First create an SSL context to set verification of the peer:<br />
<div style="font-family: inherit;">
</div>
<pre style="background-color: #eeeeee; border: 1px solid rgb(153, 153, 153); color: black; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>>>> from OpenSSL import SSL
>>> ctx = SSL.Context(SSL.SSLv3_METHOD)
>>> verify_callback = lambda conn, x509, errnum, errdepth, preverify_ok: preverify_ok
>>> ctx.set_verify(SSL.VERIFY_PEER, verify_callback)
>>> ctx.load_verify_locations(None, './cacerts')</code>
</pre>
<div style="font-family: inherit;">
<br />
Create an opener adding in the context object and GET the URL. The custom build opener adds in a new<span style="font-family: inherit;"> </span> PyOpenSSL based <span style="font-family: "Courier New",Courier,monospace;">HTTPSContextHandler</span>.</div>
<div style="font-family: inherit;">
<br /></div>
<div style="font-family: "Courier New",Courier,monospace;">
</div>
<pre style="background-color: #eeeeee; border: 1px solid rgb(153, 153, 153); color: black; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>>>> from ndg.httpsclient.urllib2_build_opener import build_opener
>>> opener = build_opener(ssl_context=ctx)
>>> res = opener.open('https://localhost/')
>>> print res.read()</code>
</pre>
<br />
The above verify callback above is just a placeholder. For more a comprehensive implementation <span style="font-family: "Courier New",Courier,monospace;">ndg_httpsclient</span> includes a callback with support for checking of the peer FQDN against the <span style="font-family: "Courier New",Courier,monospace;">subjectAltName</span> in the certificate. If <span style="font-family: "Courier New",Courier,monospace;">subjectAltName</span> is absent, it defaults to an attempted match against the certificate subject <span style="font-family: "Courier New",Courier,monospace;">CommonName</span>.<br />
<br />
The callback is implemented as a class which is a callable. This means that you can instantiate it, configuring the required settings and then pass the resulting object direct to the context's <span style="font-family: "Courier New",Courier,monospace;">set_verify</span>:<br />
<br />
<pre style="background-color: #eeeeee; border: 1px solid rgb(153, 153, 153); color: black; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>>>> from ndg.httpsclient.ssl_peer_verification import ServerSSLCertVerification
>>> verify_callback = ServerSSLCertVerification(hostname='localhost')</code>
</pre>
<br />
To get the <span style="font-family: "Courier New",Courier,monospace;">subjectAltName</span> support I needed <span style="font-family: "Courier New",Courier,monospace;">pyasn1</span> with some help from this <a href="http://stackoverflow.com/questions/5519958/how-do-i-parse-subjectaltname-extension-data-using-pyasn1">query</a> to correctly parse the relevant certificate extension. So adding this into the context creation steps above:<br />
<br />
<pre style="background-color: #eeeeee; border: 1px solid rgb(153, 153, 153); color: black; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>>>> from OpenSSL import SSL
>>> ctx = SSL.Context(SSL.SSLv3_METHOD)
>>> verify_callback = </code><code>ServerSSLCertVerification(hostname='localhost')</code><code>
>>> ctx.set_verify(SSL.VERIFY_PEER, verify_callback)
>>> ctx.load_verify_locations(None, './cacerts')</code>
</pre>
<br />
The package will work without <span style="font-family: "Courier New",Courier,monospace;">pyasn1</span> but then you loose the <span style="font-family: "Courier New",Courier,monospace;">subjectAltName</span> support. Warning messages will flag this up. I can pass this context object to the <span style="font-family: "Courier New",Courier,monospace;">urllib2</span> style opener as before, or using the <span style="font-family: "Courier New",Courier,monospace;">httplib</span> interface:<br />
<br />
<pre style="background-color: #eeeeee; border: 1px solid rgb(153, 153, 153); color: black; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>>>> from ndg.httpsclient.https import HTTPSConnection
>>> conn = HTTPSConnection('localhost', port=4443, ssl_context=ctx)
>>> conn.connect()
>>> conn.request('GET', '/')
>>> resp = conn.getresponse()
>>> resp.read()</code>
</pre>
<br />
A big thank you to Richard for his help getting this package written and ready for use. Amongst other things he's added a suite of convenience wrapper functions and a command line script.Philip Kershawhttp://www.blogger.com/profile/05141787349333619736noreply@blogger.com0tag:blogger.com,1999:blog-8070062627521378361.post-58871795316527069692011-11-23T21:22:00.001+00:002011-12-16T22:47:18.721+00:00OAuth MyProxy Mash upThis is a revisit of OAuth from two perspectives, the first in that it was under consideration for the MashMyData project and the second, because this is looking at using OAuth with <a href="http://grid.ncsa.illinois.edu/myproxy/ca">MyProxyCA</a>, something already exploited with the <a href="http://www.cilogon.org/">CILogon</a> project.<br />
<br />
<a href="http://philipkershaw.blogspot.com/2010/12/mash-my-security-for-mashmydata.html">MashMyData</a> explored solutions for delegation in a secured workflow involving a portal, an OGC Web Processing Service and OPeNDAP. The aim was where possible, to re-use existing components a large part of which was the federated access control infrastructure developed for the Earth System Grid Federation. ESGF supports single sign-on with PKI short-lived credentials but we had no use case requiring user delegation. When it came to MashMyData the traditional solution with Proxy certificates won out. So why bother revisiting OAuth?<br />
<ul>
<li>OAuth 2.0 simplifies the protocol flow compared with OAuth 1.0. Where OAuth 1.0 used signed artifacts in the HTTP headers, with OAuth 2.0, message confidentiality and integrity can be taken care of with SSL. </li>
<li>Custom SSL verification code is required for consumers to correctly verify certificate chains involving proxy certificates. We can dispense with this as OAuth uses an independent mechanism for delegation.</li>
</ul>
There are a number of different ways you could apply OAuth. Here we're looking at MyProxyCA, a short-lived credential service (SLCS). Usually, a user would make a request to the service for a new credential (X.509 certificate) and authenticate with username/password. Applying OAuth, the user gives permission to another entity to retrieve a credential representing them on their behalf.<br />
<br />
There's a progression to this solution as follows:<br />
<ol>
<li>MyProxyCA alters the standard MyProxy to enable certificates to be dynamically issued from a CA configured with the service. User's input credentials (username/password) are authenticated against a PAM or SASL plugin. This could link to a user database or some other mechanism. MyProxyCA issues EECs.</li>
<li>Front MyProxyCA with a web service interface and you gain all the benefits - tools, middleware, support and widespread use of HTTP.</li>
<li>If you have a HTTP interface you can easily front it with OAuth </li>
</ol>
2) Has been developed for CILogon and there is also <a href="http://pypi.python.org/pypi/MyProxyWebService">MyProxyWebService</a>. This is how 3) could look:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_sAKbKEINoo2e_6ye_VnbGS0ZAHZrWIX96p9bcqrnWxtwUSPq6kgntGMdIofMGSsl2bkn7AWft992DUkadZOth5v_Ei7_U_5xyZmwNFRsWIp8AqcsPgLxNEYruyKCw_wLM67c0somIbvB/s1600/slcs-oauth2-delegation.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="562" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_sAKbKEINoo2e_6ye_VnbGS0ZAHZrWIX96p9bcqrnWxtwUSPq6kgntGMdIofMGSsl2bkn7AWft992DUkadZOth5v_Ei7_U_5xyZmwNFRsWIp8AqcsPgLxNEYruyKCw_wLM67c0somIbvB/s640/slcs-oauth2-delegation.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
The user agent could be a browser or command line client or some other HTTP rich client application. The OAuth 2.0 Authorisation Server is implemented in this case as middleware fronting the online CA. The user must authenticate with this so that the online CA can ensure that this user has the right to delegate to the client. I would envisage it also keeping a record of previously approved clients for the user so that approvals can be made without user intervention should they wish. I like this ability for the user to manage which clients may be delegated to offline of the process of brokering credentials.<br />
<br />
Once the client has obtained an authorisation grant, it generates a key pair and puts the public key in a certificate request to despatch to the online CA. The latter verifies the authorisation grant and returns an OAuth access token in the form of a new user certificate delegated to the client. The client can then use this in requests to other services where they need to act on behalf of the user. A big thank you to Jim Basney for his feedback in iterating towards this solution. <br />
<br />
There is the question remaining of the provenance of credentials: how do I distinguish this delegated certificate to one obtain directly by the user? With proxy certificates you at least have the chain of trust going back to the original user certificate issuer and the CN component of the certificate subject name added for each delegation step. For ESGF, we added in a SAML attribute assertion into the certificates extension for issued EECs. There's a profile for this with VOMS which we hope to adopt for <a href="http://contrail-project.eu/">Contrail</a> and the same idea has been used elsewhere. I'd like to be able to express provenance information in this same space. I would think it would likewise be as an attribute assertion but it needs some more thought. The certificate extension content could be extended with additional SAML to meet a couple of other objectives:<br />
<ul>
<li>add an authorisation decision statement to restrict the scope of resources that the issued certificate is allowed to access</li>
<li>add level of assurance information via a SAML Authentication Context.</li>
</ul>
<br />
<br />
<ul>
</ul>Philip Kershawhttp://www.blogger.com/profile/05141787349333619736noreply@blogger.com0tag:blogger.com,1999:blog-8070062627521378361.post-33574962385517169632011-10-20T21:49:00.000+01:002011-10-20T21:52:18.605+01:00Federated CloudsImagine the ability to seamlessly manage independent resources on multiple cloud providers with a single interface. There are some immediate benefits to consider: avoiding vendor lock-in, migration of a resource from one cloud to another, replication of data ...<br />
<br />
You might be excused for thinking it's a little ambitious but a colleague on the <a href="http://contrail-project.eu/">Contrail project</a> drew my attention to this article on <a href="http://www.forrester.com/rb/Research/cloud_broker_%26%238212%3B_new_business_model_paradigm/q/id/57809/t/2">Cloud Brokering</a>. As Lorenzo said, you don't have to pay for the full article to get the jist but it seems from a rudimentary search that there are a number of commercial products already ventured into this area:<br />
<br />
<a href="http://www.datacenterknowledge.com/archives/2009/07/27/cloud-brokers-the-next-big-opportunity/">http://www.datacenterknowledge.com/archives/2009/07/27/cloud-brokers-the-next-big-opportunity/</a><br />
<br />
Federating clouds is a core objective of Contrail, and from what I heard at the <a href="http://ec.europa.eu/information_society/events/ssai/ios2011/index_en.htm"><span id="goog_1042270175"></span>Internet of Services <span id="goog_1042270176"></span>meeting</a> I attended last month, there's plenty of research interest in this topic. Picking out some points raised in the discussions (with some of my own thoughts mixed in):<br />
<ul>
<li>the importance of federating clouds for the European model. Cloud infrastructures deployed in smaller member states can't match the resources available to the large US enterprises but if those smaller infrastructures are joined in a federation their resources can be pooled to make something greater.</li>
<li>Standards are essential for federated clouds to succeed (an obvious point really) but that existing standards such as OVF and OCCI provide incomplete coverage of what is needed across the spectrum of cloud architecture related concerns.</li>
<li>The problem of funding and continuity of work, general to many research efforts but cloud technology by its nature surely needs a long term strategy for it to flourish. </li>
<li>The need for longer term research goals with 10-15 year gestation, short-term goals will be absorbed by commercial companies. There's a danger of simply following rather than leading.</li>
</ul>
So on the last point then, it's all right to be ambitious ;) <br />
<br />
<br />Philip Kershawhttp://www.blogger.com/profile/05141787349333619736noreply@blogger.com0tag:blogger.com,1999:blog-8070062627521378361.post-83191912397607173392011-10-07T22:53:00.000+01:002011-10-07T22:53:06.770+01:00Federated Identity Workshop Coming up...This is a plug for a workshop on Federated Identity Management for Scientific Collaborations coming at RAL 2-3 November:<br />
<br />
<a href="http://indico.cern.ch/conferenceDisplay.py?ovw=True&confId=157486">http://indico.cern.ch/conferenceDisplay.py?ovw=True&confId=157486</a><br />
<br />
It's a follows the <a href="https://indico.cern.ch/conferenceDisplay.py?confId=129364">first</a> of it's kind held earlier this year held at CERN and brought together experts in the field and representatives from a range of different scientific communities to present the state of play for federated identity management in each of the various fields and draw together a roadmap for future development. See the <a href="https://indico.cern.ch/materialDisplay.py?materialId=minutes&confId=129364">minutes</a> for a full account.<br />
<br />
Picking out just a few themes that were of interest to me: inter-federation trust came up a number of times and the need for services to translate credentials from one domain to another. I read that as a healthy sign that a) various federated identity management systems have bed down and become established and b), that there is not a fight of competing security technologies for one to take over all, rather a facing up to realities of how can we make it work so that these co-exist along side each other.<br />
<br />
Credential translation brings in another two interesting issues: provenance and levels of assurance that actually also arose independently in some of the talks and discussions. If I have a credential that is as a result of a translation of another credential from a different domain how much information is transferred between the two, is it lossy are the identity concepts and various attributes semantically the same? The same issues arise perhaps to a lesser degree with delegation technologies. <br />
<br />
Levels of assurance is another issue that is surely going to crop up more and more as different authentication mechanisms are mixed together in systems: the same user can enter a federated system with different methods how do we ensure that they are assigned access rights accordingly. Some complicated issues to tackle but the fact that they can begin to be addressed shows the progress that has been made building on the foundations of established federated systems.<br />
<br />
<br />Philip Kershawhttp://www.blogger.com/profile/05141787349333619736noreply@blogger.com0tag:blogger.com,1999:blog-8070062627521378361.post-26083339060164624332011-08-19T22:57:00.005+01:002011-08-19T23:02:01.600+01:00Java SSL Whitelisting of Peer Certificate Subject NamesHelping a colleague just today has reminded me to finish this post drafted a long time ago. Last year I was dipping into the Java SSL libraries to write a short piece of code to make a call to a service running over HTTPS where mutual authentication is required. - The client authenticates the server based on the server's X.509 certificate passed in the SSL handshake but in addition, the client must pass a certificate to enable the server to authenticate it the client.<br />
<br />
By default, the Java SSL trust manager will trust peer certificates provided that they are issued by any of the CAs (Certificate Authorities) whose certificates appear in the default trust store for the JVM. It's possible to customise the trust manager to use a given trust store to give more fine grained control but what if we want to trust only a certain subset of certificates issued by a given CA or CAs? <br />
<br />
One way to achieve this is to whitelist based on the peer certificate DN or Distinguished Name. This is something that is straightforward to do on the server side with, for example, Apache using the <span style="font-family: "Courier New",Courier,monospace;">SSLRequire</span> directive. It's also a practice used in Grid computing authorisation middleware with ACLs (Access Control Lists). Rather than the protection of some server-side resource, the problem to solve in this case is a client invocation.<br />
<br />
Returning to the SSL API then, this can be achieved by implementing <span style="font-family: "Courier New",Courier,monospace;">javax.net.ssl.X509TrustManager </span>interface. The key method for client side checking of server certificates is <span style="font-family: "Courier New",Courier,monospace;">checkServerTrusted</span>. The relevant hooks can be set in here to check the peer certificate against a whitelist:<br />
<br />
<pre style="background-color: #eeeeee; border: 1px solid rgb(153, 153, 153); color: black; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><span class="Apple-style-span" style="font-family: inherit;"><code> public void checkServerTrusted(X509Certificate[] chain, String authType)
throws CertificateException {
// Default trust manager may throw a certificate exception
pkixTrustManager.checkServerTrusted(chain, authType);
// If chain is OK following previous check, then execute whitelisting
// of DN
X500Principal peerCertDN = null;
if (certificateDnWhiteList == null ||
certificateDnWhiteList.isEmpty())
return;
int basicConstraints = -1;
for (X509Certificate cert : chain) {
// Check for CA certificate first - ignore if this is the case
basicConstraints = cert.getBasicConstraints();
if (basicConstraints > -1)
continue;
peerCertDN = cert.getSubjectX500Principal();
for (X500Principal dn : certificateDnWhiteList)
if (peerCertDN.getName().equals(dn.getName()))
return;
throw new CertificateException("No match for peer certificate \"" +
peerCertDN + "\" against Certificate DN whitelist");
}
}
</code></span></pre><br />
<span style="font-family: "Courier New",Courier,monospace;">pkixTrustManager</span> is the default trust manager whilst <span style="font-family: "Courier New",Courier,monospace;">certificateDnWhiteList</span> is a list of accepted DNs as <span style="font-family: "Courier New",Courier,monospace;">X500Principal</span> types. These can be initialised in the classes' constructor from a properties file or some other input. The <span style="font-family: "Courier New",Courier,monospace;">pkixTrustManager.checkServerTrusted</span> call applies the default verification of the peer's certificate based on the CA certificates present in the client's trust store. If this succeeds, a loop then iterates over the certificate chain returned by the peer skipping any CA certificates*. Once the peer certificate is found, its DN is extracted and checked against the whitelist. If matched, it returns silently to the caller indicating all is OK. If no match is found, a <span style="font-family: "Courier New",Courier,monospace;">CertificateException</span> is thrown to indicate that the peer certificate is not in the accepted list of DNs. This could easily be extended to do more sophisticated matching for example using regular expressions.<br />
<br />
This <a href="http://java.sun.com/developer/technicalArticles/Security/secureinternet2/">technical article</a> provides some more background (scroll down a long way to the Trust Manager heading). The full source for the example above is available <a href="http://ndg-security.ceda.ac.uk/browser/trunk/EsgYadisParser/src/esg/security/DnWhitelistX509TrustMgr.java">here</a>.<br />
<br />
[* The peer can of course pass back not only its own certificate, but any intermediate CA certificates needed to complete the chain of trust to a root CA certificate held by the client.]Philip Kershawhttp://www.blogger.com/profile/05141787349333619736noreply@blogger.com0tag:blogger.com,1999:blog-8070062627521378361.post-54085897504322887922011-02-10T20:36:00.000+00:002011-02-10T20:36:33.977+00:00Proxy certificates and delegation with netCDF Beta releaseI've <a href="http://philipkershaw.blogspot.com/2010/09/esg-security-hooks-are-in-netcdf.html">written previously</a> about the extensions to the netCDF C API to enable SSL client based authentication. It's been great to see how something slotted in at the base of a software stack filters down to benefit all the dependents: colleagues have been testing <i>Ferret</i>, <i>ncview</i> and Python bindings built against the updated libraries and used to query ESG-secured OPeNDAP services. This links with another thread, the <a href="http://philipkershaw.blogspot.com/2010/12/mash-my-security-for-mashmydata.html">MashMyData</a> project extends this SSL client based authentication mechanism from EECs (End Entity Certificates) - the current currency for short lived credentials in ESGF (Earth System Grid Federation) - to RFC3820 proxy certificates. This is necessitated by the need for delegation in chain of operations in our use case: a chain linking a portal to an OGC Web Processing Service which itself calls an OPeNDAP service. So, on to trying out the netCDF C client with a proxy certificate:<br />
<ul><li>Get ESG-enabled netCDF - now in 4.1.2 beta2</li>
<li><a href="http://proj.badc.rl.ac.uk/ndg/browser/TI12-security/trunk/NDGSecurity/C/openDapPatch/">build simple client</a> against this version of the library</li>
<li>Get EEC and delegate (need Globus Toolkit for this example)</li>
</ul><div>So expanding the last step:<br />
<br />
</div>1) Get short lived EEC from home MyProxy server:<br />
<span class="Apple-style-span" style="color: #bebebe; font-family: 'Trebuchet MS',Trebuchet,sans-serif; font-size: 13px; line-height: 18px;"></span><br />
<pre style="background-color: #eeeeee; border: 1px solid rgb(153, 153, 153); color: black; font-family: 'Andale Mono','Lucida Console',Monaco,fixed,monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 550px;"><code>$ myproxy-logon -s <my idp's myproxy host address> -o user.pem -b</code></pre><br />
2) Delegate to obtain proxy certificate:<br />
<span class="Apple-style-span" style="color: #bebebe; font-family: 'Trebuchet MS',Trebuchet,sans-serif; font-size: 13px; line-height: 18px;"></span><br />
<pre style="background-color: #eeeeee; border: 1px solid rgb(153, 153, 153); color: black; font-family: 'Andale Mono','Lucida Console',Monaco,fixed,monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 550px;"><code>$ grid-proxy-init -cert user.pem -key user.pem -out ./credentials.pem -rfc</code></pre><br />
3) Update netCDF configuration to pick up credentials:<br />
<span class="Apple-style-span" style="color: #bebebe; font-family: 'Trebuchet MS',Trebuchet,sans-serif; font-size: 13px; line-height: 18px;"></span><br />
<pre style="background-color: #eeeeee; border: 1px solid rgb(153, 153, 153); color: black; font-family: 'Andale Mono','Lucida Console',Monaco,fixed,monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 550px;"><code>CURL.VERBOSE=1
CURL.COOKIEJAR=.dods_cookies
CURL.SSL.VALIDATE=1
CURL.SSL.CERTIFICATE=<path>/credentials.pem
CURL.SSL.KEY=<path>/credentials.pem
CURL.SSL.CAPATH=<home path>.globus/certificates</code></pre><br />
Calling the netCDF client makes the underlying <i>Curl</i> library invocation and correctly passes the certificate chain comprising proxy certificate and EEC that issued it (grid-proxy-init step). The OPeNDAP server and associated security middleware is <a href="http://philipkershaw.blogspot.com/2010/05/proxy-certificates-and-delegation-with.html">correctly configured</a> to accept proxy certificates. I get my data back :).Philip Kershawhttp://www.blogger.com/profile/05141787349333619736noreply@blogger.com0tag:blogger.com,1999:blog-8070062627521378361.post-75612980946737725282011-01-24T13:30:00.000+00:002011-01-24T13:30:40.363+00:00It's nice when it just worksLast week we deployed the full access control infrastructure with our TDS (<a href="http://www.unidata.ucar.edu/projects/THREDDS/">THREDDS</a> Data Server) part of the <i>Data Node</i> component we are hosting at the BADC as part of the Earth System Grid Federation (ESGF). What's been pleasing is that we have been able to mix independent implementations together and yet combine them easily in a working system. <br />
<br />
The ESGF in terms of software implementation is predominantly Java based but within the context of access control there is a parallel Python based 'NDG Security' implementation here. We now have TDS deployed too but hooked up to the same system. This follow-ups from a previous post on the <a href="http://philipkershaw.blogspot.com/2010/11/earth-system-grid-federated.html">authorisation infrastructure for ESG</a> where I showed PyDAP, a Python implementation of <a href="http://www.opendap.org/">OPeNDAP</a> deployed with our authorisation system. TDS is of course Java based and we run it within Tomcat fronted with a servlet based authorisation filter. The common interface to the authorisation system is via a SAML web service callout from the filter to an Authorisation Service. ESGF has a Java based Authorisation Service implementation but here we've deployed with a Python based one from NDG Security which shares the same interface. Plugging in the TDS to this was simply a question of making the connection settings and adding the additional rules needed in the XACML policy.<br />
<br />
So below, a user's NetCDF (could equally be a browser) client can talk to two apps PyDAP and TDS to make OPeNDAP queries. PyDAP is deployed with mod_wsgi / Apache. Each service is fronted by an authorisation filter (In practice, authentication filters too but omitted here for simplicity). The respective filters intercept requests and query the authorisation service to make an access control decision. The Authorisation Service is itself a Python app is also running under mod_wsgi/Apache. <br />
<br />
Within the Authorisation Service, a context handler translates the incoming SAML decision request query to XACML (yes, XACML could have been used instead between the filters and Authorisation Service) and feeds the request to the Policy Decision Point. The PDP has a XACML policy fed to it at start-up. When making an access decision, it can also query for additional attributes by requesting the context handler query the Policy Information Point. The PIP can query for federation wide attributes from an Attribute Service at <a href="http://www-pcmdi.llnl.gov/">PCMDI</a>. PCMDI have a key role administering access in the federation. The PDP makes its decision and a response is sent via the Context handler back to the filter fronting the respective app.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCHetuqOwyG-6cg3JLFHJPDzTirZ-1y2uci1q1Q3KytKo34QlCj_qOKj4uUB1D4dXV5FSC3Q_laBDuQ5MU-p0WhxhoTteu13F1PasmR7ZpINrgIPS6zY9uySOUercEVOoDbWu2PvG2lFv9/s1600/PyDAP%252BTDSAuthorisation.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCHetuqOwyG-6cg3JLFHJPDzTirZ-1y2uci1q1Q3KytKo34QlCj_qOKj4uUB1D4dXV5FSC3Q_laBDuQ5MU-p0WhxhoTteu13F1PasmR7ZpINrgIPS6zY9uySOUercEVOoDbWu2PvG2lFv9/s1600/PyDAP%252BTDSAuthorisation.png" /></a></div>Philip Kershawhttp://www.blogger.com/profile/05141787349333619736noreply@blogger.com0tag:blogger.com,1999:blog-8070062627521378361.post-56919484004732339522010-12-21T10:05:00.000+00:002010-12-21T10:05:48.377+00:00Only as strong as the weakest link in the chainA security system is only as strong as the weakest link in the chain to paraphrase a well known saying. There can be lots of links to think about in a federated security system. How do you ensure a thorough analysis and an acceptable baseline level of security? The <a href="http://csrc.nist.gov/publications/nistpubs/800-63/SP800-63V1_0_2.pdf">NIST levels of assurance framework</a> provides a valuable metric against which to make an assessment. <br />
<br />
One argument that could made for example is that we only need an authentication mechanism at some low level of assurance, say X (worse still - no analysis is made and there's an implicit assumption!). You develop your system to meet that requirement and deploy it accordingly but what happens when you find that your system needs to secure slightly more sensitive data? Your original analysis breaks down and your system is no longer fit for purpose. With a large federated system, that's potentially a lot of infrastructure to have to tear down and re-implement and deploy. Wouldn't it have been better to design the system so that the respective links in the chain could support something more secure? That way you could provide a degree of future proofing.<br />
<br />
Clearly there would be some cost benefit analysis to weigh up but maybe many of those chain links could be made more secure to encompass a broader spectrum of use cases. There's only so far you could take this approach. An alternative is to pass level of assurance information - SAML supports this concept and this interesting <a href="http://kantarainitiative.org/confluence/download/attachments/3408008/ntt-madsen-rsa-concordia.pdf?version=1&modificationDate=1240259827000">presentation</a> looks at the 'policy impedance' introduced where protocols that would not otherwise by combined together are joined - in this case OpenID and SAML. <br />
<br />
It seems to me though that the value of expressing an assurance level is poorly understood and the case is made, why bother with this when you can express this in your authorisation policy? - A given user authenticates via an insecure mechanism therefore don't give them access rights to dataset 'A'. That's not going to help though if your trying to apply an authorisation policy to sensitive data but your system supports an authentication scheme that is only very weakly secured. It doesn't matter what hoops you have to go through to register for the required authorisation, the authentication link in the chain has made that a pointless exercise.Philip Kershawhttp://www.blogger.com/profile/05141787349333619736noreply@blogger.com0tag:blogger.com,1999:blog-8070062627521378361.post-74686241604675570052010-12-10T17:20:00.002+00:002010-12-14T10:25:20.540+00:00Mash My Security for MashMyData<a href="https://code.google.com/p/mashmydata/">MashMyData</a> is underway. This proof of concept project is exploring the provision of an online environment for scientific users to upload their data and intercompare it with environmental datasets gathered from a variety of independent data services. These services secure access to datasets and so the management of authentication/authorisation credentials across potentially multiple hops between services provides a significant challenge. For a project where data mash up is the primary focus, this involves a fair quantity of security related mash up too.<br />
<br />
A recent code sprint with project partners has brought into sharp focus how we can address and implement our use case in a short space of time. There's much to tell of the implementation details but for now this more high level overview of the use case...<br />
<br />
A MashMyData Portal provides the user environment for data mash up. It supports OpenID based single sign on enabling authentication within the scope of the portal but the portal itself must broker access to other services on the user's behalf. An initial study investigated both OAuth and the classic Grid based solution of proxy certificates. I'm keen to explore OAuth more extensively but surprisingly for me at least, the latter was easier to realise within the scope of the code sprint. This is due in no small part to the fact that it was given something of a head start: MashMyData leverages the existing security system developed for Earth System Grid. In this, services support both OpenID and PKI based authentication. The latter fits nicely with the paradigm of individual short term user certificates used in the Grid world.<br />
<br />
At this point though it's worth taking a step back to look at how OpenID might fit in this scenario. Some considerable time was spent in the development of the ESG security architecture on this: you could argue the case for OpenID approach for authentication of the user by the portal at the secondary service. By it's nature though it's unsuited in a case where the client is not a browser especially when you consider that any given OpenID Provider can impose an number of steps in its user interface and still adhere to the interface with OpenID Relying Party. This makes it difficult to manage in our case here with a programmatic interface where there is no user interaction.<br />
<br />
Back to the PKI based approach then. Each ESG Identity Provider deploys a MyProxy service to enable users to obtain short term credentials but for the MashMyData portal, the user has already authenticated via OpenID. We don't wish them to have to sign in again via MyProxy. We can however, translate their signed in status and issue a short term certificate. This is something that has already been employed with projects like <a href="http://www.computer.org/portal/web/csdl/doi/10.1109/IAS.2009.163">SARoNGS</a> and with the system devised for <a href="http://middleware.internet2.edu/idtrust/2010/papers/02-basney-federated-teragrid.pdf">federated login to TeraGrid</a>. The diagram below shows how the MyProxy can be employed:<br />
<br />
<div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHw5_twtvg-O0uQ3oKrDdqzghKxZ2TwN4Xv4Z8YNkSWKkZCRcTOkykX21UaWN-pLeTu20uTGxbkqBIGCIJbkvRYh-LVyv-yjSEUQycOzC54V3nyGQHqP6c5rEwXO67LmorbJHMStH7eNlL/s1600/MyProxyAndProxyCertificateBasedDelegationSolution.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="474" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHw5_twtvg-O0uQ3oKrDdqzghKxZ2TwN4Xv4Z8YNkSWKkZCRcTOkykX21UaWN-pLeTu20uTGxbkqBIGCIJbkvRYh-LVyv-yjSEUQycOzC54V3nyGQHqP6c5rEwXO67LmorbJHMStH7eNlL/s640/MyProxyAndProxyCertificateBasedDelegationSolution.png" width="640" /></a></div><br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
The user signs in at the MashMyData portal and invokes CEDA's (Centre for Environmental Data Archival) Python based WPS (OGC Web Processing Service) implementation to execute a job. The WPS requires authentication so the portal calls the Credential Translation Service to obtain a short term certificate to represent the user and authenticate at the WPS. [I'm leaving authorisation out of this for simplicity. - Authorisation is enforced on all services]. The translation service is in fact a MyProxy service configured with a CA. For the purposes of the MashMyData demonstrator certain CEDA services have been configured to trust this CA. <br />
<br />
Usually in this mode, the MyProxyCA responds to MyProxy logon requests by authenticating based on the input username/password against a given PAM service module. The PAM might for example link to a user database. In this case however a custom PAM accepts the users OpenID and the MyProxy service 'authenticates' against this alone and returns an End Entity Certificate back to the portal. The portal can then use this in its request to the WPS. The obvious question here is, given such a permissive policy what is to stop anyone requesting as many credentials as they like?! However, only the portal needs access to this service, so access can be restricted to it alone.<br />
<br />
Next, the job at the WPS itself needs to retrieve data from CEDA's Python based OPeNDAP service, PyDAP. The portal pre-empts this step by priming a second MyProxy server with a delegated user credential which the WPS can then retrieve. This second MyProxy server is configured in an alternative more conventional mode for MyProxy in which it acts as a repository for short term credentials. The portal then, adds a new credential to this repository so that it can be available for the WPS or any other service which has been allocated retrieval rights. In this process - a put request on the part of the portal, makes the MyProxy server create a new key pair and return a certificate signing request in return. The Portal signs this using the user certificate previously obtained and the signed proxy is uploaded to the MyProxy server's repository.<br />
<br />
With this in place, the WPS can execute a MyProxy logon to obtain a proxy certificate for use to authenticate with the PyDAP service. In fact, any number of services can be configured in a chain. Some interesting remaining issues to consider:<br />
<ol><li>Services must be able to consume proxy certificates. This needs special configuration, something I've discussed <a href="http://philipkershaw.blogspot.com/2010/05/proxy-certificates-and-delegation-with.html">previously</a>. </li>
<li>The MyProxy delegation service has a static configuration determining which services are authorised to retrieve user proxy credentials. On this point and the previous an OAuth based solution might provide a better alternative plus you might throw away proxy certificates altogether which would remove an SSL configuration overhead.</li>
<li>How do services discover the MyProxy service endpoint in order to know where to get delegated credentials from? For the moment this is a static configuration but there could be a way of passing this information from the client. Another alternative could be to add this information to the OpenID Provider's Yadis document so that it can be discovered by invoking a HTTP GET on the user's OpenID URL. <a href="http://philipkershaw.blogspot.com/2010/06/introspecting-identity-with-yadis.html">Extending the Yadis document</a> has already been exploited with ESG but implicit is the ability for OpenID Providers to include this customisation. This would obviously break interoperability with vanilla OpenID Providers.</li>
</ol>The code sprint has provided a great demonstration of the flexibility of MyProxy and the ease with which something can be assembled quickly to meet the given use case.Philip Kershawhttp://www.blogger.com/profile/05141787349333619736noreply@blogger.com0tag:blogger.com,1999:blog-8070062627521378361.post-75576782516586695762010-11-10T17:20:00.000+00:002010-11-10T17:20:07.433+00:00Earth System Grid Federated Authorisation deployedAt the end of last week I deployed a <a href="http://pydap.org/">PyDAP</a> (Python implementation of OPeNDAP) instance with the latest version of the Python ESG security implementation. This brings into play here the last major pieces in the federated security solution for Earth System Grid Federation. I've talked <a href="http://philipkershaw.blogspot.com/2010/05/proxy-certificates-and-delegation-with.html">before</a> about authentication and support for both OpenID and PKI based (MyProxy) single sign on for ESG. This latest deployment completes the picture with the authorisation mechanism.<br />
<br />
The authorisation architecture follows the familiar PEP - PDP paradigm where in our case, a particular service is secured by a Policy Enforcement Point which itself refers authorisation decisions to a Policy Decision Point. The PDP runs as part of an independent authorisation service. This is shown below. A HTTP based client, in this case a NetCDF based OPeNDAP client makes a request to the PyDAP service. At the server, an authorisation filter (PEP) intercepts requests to the underlying PyDAP application. It makes authorisation decision queries over a SAML interface and enforces these allowing or preventing access to the requested URL. (Nb. authentication components are left out for simplicity. See this <a href="http://philipkershaw.blogspot.com/2010/05/proxy-certificates-and-delegation-with.html">posting</a> for details).<br />
<br />
<div class="separator" style="clear: both; text-align: center;"></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZWgkx7s0e0IY6MHHX7VbCzE-UWjhB4ktevlQAavOJ_yRevTCRv5I1jFn5R26n0U2cBRzzq-UCe8FYoV6sZvQdSoXkXhKTenrnwBgF3qndJvndz3MBve1dhJA3JvuWjNeAhGsuVkHW8eK1/s1600/PyDAPAuthorisation.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhZWgkx7s0e0IY6MHHX7VbCzE-UWjhB4ktevlQAavOJ_yRevTCRv5I1jFn5R26n0U2cBRzzq-UCe8FYoV6sZvQdSoXkXhKTenrnwBgF3qndJvndz3MBve1dhJA3JvuWjNeAhGsuVkHW8eK1/s640/PyDAPAuthorisation.png" width="590" /></a></div><br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
In the above, both PyDAP application and Authorisation service reside within the same organisation so where is the federated part of all this? Putting that aside for a moment, the standard SAML interface between authorisation filter and authorisation service mean I can interchange different applications and authorisation services and implementations. For example, on the left replace PyDAP with a THREDDS Data Server with ESG servlet authorisation filter, or keep the PyDAP service but secure it with the alternative Authorisation Service packaged with the Java ESG Gateway application.<br />
<br />
Returning to the question of the federated aspect, this comes into play with PCMDI's Attribute Service. PCMDI have an overall responsibility across the federation for management and allocation of <a href="http://cmip-pcmdi.llnl.gov/cmip5/">CMIP5</a> access roles (attributes) to registered users. Consider a request to the PyDAP service: the authorisation filter protecting PyDAP can <i>push</i> user attributes to the authorisation service but the authorisation service can also <i>pull</i> attributes from outside. If the policy determines that CMIP5 access is required, PCMDI's Attribute Service is queried to check that the user has the required CMIP5 attribute(s). These attributes are pulled into the request context for the PDP to process and make a decision. Following this architecture then has the consequence that PCMDI maintain control over the CMIP5 attribute. They can add or withdraw any users enrolement and this will immediately take affect across the federation.<br />
<br />
At this point the contents of the Authorisation Service component above needs some further explanation. To be compliant with ESG, the component must have the outward facing interfaces shown for SAML authorisation decision and attribute query. Within this however, I've chosen to follow a XACML based solution. The Context Handler receives SAML based authorisation decision queries and translates them to XACML based requests and passes these to the PDP. The PDP takes XACML policy file at start up. The policy expresses the restrictions needed for the different datasets served from the PyDAP service. The Policy Information Point serves the PDP with additional user attributes when required by querying Attribute Service(s). In this case it's PCMDI's. The Authorisation Service is written as a Python WSGI application as part of NERC DataGrid Security. The XACML components use the Python XACML implementation <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><a href="http://pypi.python.org/pypi/ndg-xacml">ndg_xacml</a></span>. I want to write more about this in a future post.<br />
<br />
Taking a step back then and looking from a user perspective, I can go to the PCMDI Gateway, sign in with an OpenID from any of the trusted IdPs in the federation, register for the CMIP5 access attribute and then potentially go to any site in the federation where CMIP5 data is served and access it.Philip Kershawhttp://www.blogger.com/profile/05141787349333619736noreply@blogger.com0tag:blogger.com,1999:blog-8070062627521378361.post-81622865174444490282010-09-03T10:42:00.002+01:002010-09-03T11:22:31.268+01:00ESG Security hooks are in NetCDFThis is a big boost for ESG and access control interoperability with environmental datasets. The security API for the <a href="http://www.earthsystemgrid.org/">Earth System Grid</a> is now integrated into the NetCDF C libraries. The <a href="http://www.unidata.ucar.edu/software/netcdf/">NetCDF</a> data format is used widely for storing data across the earth sciences. With these changes, a NetCDF client can make calls to OPeNDAP servers serving secured datasets. Security settings are made via a standard configuration file so there is no need for any changes to the API. This is important as many tools and high level language bindings such as Python use the C libraries.<br />
<br />
<h3>Inner Workings</h3>The NetCDF libraries already support network based data retrieval using OPeNDAP and implementations such as PyDAP and TDS support a HTTP redirect pattern to support authentication schemes like <a href="http://www.jasig.org/cas">CAS</a>. For ESG, a security layer to in the form of server side middleware has been developed for PyDAP and TDS. This builds on the redirect functionality but in addition, enables a client to use SSL client based authentication. The redirect capability means data can still be served over an unencrypted channel an important consideration when serving large datasets. SSL client based authentication fits well with Grid based tools used in ESG like MyProxy which enables a user to obtain a short term certificate based on their usual username/password credentials with their Identity Provider.<br />
<br />
The NetCDF client authenticates by following a HTTP redirect response set by the server to redirect to a HTTPS based authentication endpoint. The client passes a certificate in the subsequent SSL handshake so that the server can authenticate the request. Once authenticated, a redirect response is sent back to the client in order for the client to return to the original URI invoked. With the client authenticated a cookie is set and control can pass to the data serving application underlying the security middleware. This sequence from a <a href="http://philipkershaw.blogspot.com/2010/05/proxy-certificates-and-delegation-with.html">previous posting</a> illustrates the steps:<br />
<br />
<br />
<div style="font-family: inherit;"></div><div class="separator" style="clear: both; font-family: inherit; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzhYpzTFpVugRms1hiYip7iYgvmRtvz_DYJw9yLhuox3NpsXgREY4YaDecXupYx54xKNNeFbigIJCA4n_r5RA86z3LmKCNKA_PPHiNQGiIApamgXcUVuoDsBW-8Ng9Rc7JrGhOxnE5mFD7/s1600/OPeNDAPSSLAuthentication.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="221" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzhYpzTFpVugRms1hiYip7iYgvmRtvz_DYJw9yLhuox3NpsXgREY4YaDecXupYx54xKNNeFbigIJCA4n_r5RA86z3LmKCNKA_PPHiNQGiIApamgXcUVuoDsBW-8Ng9Rc7JrGhOxnE5mFD7/s400/OPeNDAPSSLAuthentication.png" width="400" /></a></div><div style="font-family: inherit;"></div><div style="font-family: inherit;"></div><br />
<h3>Example</h3>These settings in the users <span style="font-family: "Courier New",Courier,monospace;">.dodsrc</span> file set a client certificate and key (both contained in <span style="font-family: "Courier New",Courier,monospace;">creds.pem</span> in the example below) and a directory containing CA certificates - used to verify the identity of the server certificate:<br />
<br />
<pre style="background-color: #eeeeee; border: 1px solid rgb(153, 153, 153); color: black; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>
CURL.VERBOSE=1
CURL.COOKIEJAR=.dods_cookies
CURL.SSL.VALIDATE=1
CURL.SSL.CERTIFICATE=/.../creds.pem
CURL.SSL.KEY=/.../creds.pem
CURL.SSL.CAPATH=/.../ca-certificates
</code></pre><br />
Full example code <a href="http://proj.badc.rl.ac.uk/ndg/browser/TI12-security/trunk/NDGSecurity/C/openDapPatch/">here</a>.<br />
<br />
Special thanks to Dennis Heimbigner at Unidata and to my colleague Steve Crothers and others at RAL for their contributions on this.Philip Kershawhttp://www.blogger.com/profile/05141787349333619736noreply@blogger.com0tag:blogger.com,1999:blog-8070062627521378361.post-27531815860836217302010-06-25T14:18:00.009+01:002011-03-17T20:38:45.337+00:00Introspecting Identity with Yadis<a href="http://yadis.org/">Yadis</a> is a powerful tool for service discovery. It answers the question, given an identity URI, what identity services can I invoke on it? The <a href="http://philipkershaw.blogspot.com/2010/06/earth-system-grid-security-architecture.html">Earth System Grid security architecture</a> has OpenID as its core identity service but there are others: MyProxy from the Globus toolkit enables users to obtain a short lived X.509 certificate for SSL based authentication to secured resources. In addition there is the SAML Attribute Service enabling a client to query attribute information about a given subject. Locating these services is a problem and this was illustrated in a use case recently: a trusted service in the federation would like to notify a user by e-mail. The service has their OpenID but it doesn't have access to the usual attributes available through the Attribute Exchange interface - they're not signed in. How can I query their e-mail address to contact them?<br />
<br />
OpenID (v2.0) already supports the Yadis protocol, and its uses is mandatory for ESG OpenID Providers. If I HTTP GET a user's identity URI, I get a document containing the OpenID Provider endpoint:<br />
<br />
Extending this I can include the corresponding MyProxy and the Attribute Service URIs:<br />
<br />
<pre style="background-color: #eeeeee; border: 1px solid rgb(153, 153, 153); color: black; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>
<?xml version="1.0"; encoding="UTF-8"?>
<xrds:XRDS xmlns:xrds="xri://$xrds" xmlns="xri://$xrd*($v*2.0)">
<XRD>
<Service priority="0">
<Type>http://specs.openid.net/auth/2.0/signon</Type>
<Type>http://openid.net/signon/1.0</Type>
<URI>https://openid.provider.somewhere.ac.uk</URI>
<LocalID>https://somewhere.ac.uk/openid/PJKershaw</LocalID>
</Service>
<Service priority="1">
<Type>urn:esg:security:myproxy-service</Type>
<URI>socket://myproxy-server.somewhere.ac.uk:7512</URI>
<LocalID>https://somewhere.ac.uk/openid/PJKershaw</LocalID>
</Service>
<Service priority="20">
<Type>urn:esg:security:attribute-service</Type>
<URI>https://attributeservice.somewhere.ac.uk</URI>
<LocalID>https://somewhere.ac.uk/openid/PJKershaw</LocalID>
</Service>
</XRD>
</xrds:XRDS></code>
</pre><br />
The priority attribute for each <Service/> element lets the client know which is the preferred service for authentication. The Attribute Service is of course not an authentication service but I'd argue it has a place here as it is an identity service. [17/03/11 - <b>Note</b> correction for this use case: <i>each</i> service element is in the <i>single</i> XRD element, <b>not</b> many XRD elements each with a single service element as I had before!]<br />
<br />
Taking this a step further, I can combine it with <a href="http://grid.ncsa.illinois.edu/myproxy/trustroots/">MyProxy's provisioning</a> functionality, the ability to provision a client with trusted CA certificates it needs to bootstrap trust.<br />
<br />
Given an identity URI, I can:<br />
<ol><li>retrieve the Yadis document</li>
<li> parse the MyProxy server endpoint</li>
<li>retrieve the trust roots to bootstrap trust in the identity services for this identity URI.</li>
</ol>Philip Kershawhttp://www.blogger.com/profile/05141787349333619736noreply@blogger.com0tag:blogger.com,1999:blog-8070062627521378361.post-66553374257746453582010-06-23T10:20:00.000+01:002010-06-23T10:20:56.449+01:00Earth System Grid Security Architecture BlueprintWe now have a blueprint documented for the ESG security architecture:<br />
<br />
<a href="http://esg-pcmdi.llnl.gov/esgf/esgf-security-interface-control-documents/esgf-security-interface-control-document-1.0/">http://esg-pcmdi.llnl.gov/esgf/esgf-security-interface-control-documents/esgf-security-interface-control-document-1.0/</a><br />
<br />
The Interface Control Document describes all the interfaces that make up the federated security infrastructure. This is the product of a lot of collaboration work and discussion. Given the parellel Java and Python implementations this has been a true test of interoperability.Philip Kershawhttp://www.blogger.com/profile/05141787349333619736noreply@blogger.com0tag:blogger.com,1999:blog-8070062627521378361.post-37244940493781430722010-06-10T14:57:00.000+01:002010-06-10T14:57:51.454+01:00Python SAML 2.0 ReleasedI've been working on a Python implementation of SAML 2.0 for some months now and recently pushed this to the Python repository <a href="http://pypi.python.org/pypi/ndg-saml/0.4">PyPI</a>. SAML is an important tool in the <a href="http://www.earthsystemgrid.org/about/overview.htm">Earth System Grid</a> Security architecture that I've been working on and the NERC DataGrid security framework which predates it. It's used for attribute retrieval and authorisation decision queries. The code focusses on these aspects of the specification. This example illustrates an client attribute query to an Attribute Authority:<br />
<br />
<pre style="background-color: #eeeeee; border: 1px solid rgb(153, 153, 153); color: black; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>
from ndg.saml.saml2.core import (AttributeQuery, SAMLVersion, Issuer, Subject,
NameID, Attribute, XSStringAttributeValue)
from uuid import uuid4
from datetime import datetime
attributeQuery = AttributeQuery()
attributeQuery.version = SAMLVersion(SAMLVersion.VERSION_20)
attributeQuery.id = str(uuid4())
attributeQuery.issueInstant = datetime.utcnow()
attributeQuery.issuer = Issuer()
attributeQuery.issuer.format = Issuer.X509_SUBJECT
attributeQuery.issuer.value = '/O=NDG/OU=BADC/CN=PolicyInformationPoint'
attributeQuery.subject = Subject()
attributeQuery.subject.nameID = NameID()
attributeQuery.subject.nameID.format = NameID.X509_SUBJECT
attributeQuery.subject.nameID.value = '/O=NDG/OU=BADC/CN=PhilipKershaw'
# special case handling for 'LastName' attribute
emailAddressAttribute = Attribute()
emailAddressAttribute.name = "urn:esg:email:address"
emailAddressAttribute.nameFormat = "%s#%s" % (
XSStringAttributeValue.TYPE_NAME.namespaceURI,
XSStringAttributeValue.TYPE_NAME.localPart)
emailAddress = XSStringAttributeValue()
emailAddress.value = 'pjk@somewhere.ac.uk'
emailAddressAttribute.attributeValues.append(emailAddress)
attributeQuery.attributes.append(emailAddressAttribute)
# Convert to ElementTree representation
from ndg.saml.xml.etree import AttributeQueryElementTree, prettyPrint
elem = AttributeQueryElementTree.toXML(attributeQuery)
# Serialise as string
xmlOut = prettyPrint(elem)
print(xmlOut)
</code>
</pre><br />
The XML representation and serialisation is independent of the core code. The ndg.saml.xml.etree ElementTree representation comes with the rest of the code but it would be straightforward to add an adaptor for another Python XML package such as 4Suite-XML.<br />
<br />
Here's the serialised output:<br />
<br />
<pre style="background-color: #eeeeee; border: 1px solid rgb(153, 153, 153); color: black; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>
<samlp:attributequery id="1c15e748-0f74-41f1-848c-1fbdfeef2a06" issueinstant="2010-06-01T13:19:50.690263Z" version="2.0" xmlns:samlp="urn:oasis:names:tc:SAML:2.0:protocol">
<saml:issuer format="urn:oasis:names:tc:SAML:1.1:nameid-format:x509SubjectName" xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion">/O=NDG/OU=BADC/CN=PolicyInformationPoint</saml:issuer>
<saml:subject xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion">
<saml:nameid format="urn:oasis:names:tc:SAML:1.1:nameid-format:x509SubjectName">/O=NDG/OU=BADC/CN=PhilipKershaw</saml:nameid>
</saml:subject>
<saml:attribute name="urn:esg:email:address" nameformat="http://www.w3.org/2001/XMLSchema#string" xmlns:saml="urn:oasis:names:tc:SAML:2.0:assertion">
<saml:attributevalue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">pjk@somewhere.ac.uk</saml:attributevalue>
</saml:attribute>
</samlp:attributequery>
</code>
</pre><br />
More to follow - including client/server SOAP bindings - as I migrate the SAML specific code out of the NERC DataGrid security package.Philip Kershawhttp://www.blogger.com/profile/05141787349333619736noreply@blogger.com0tag:blogger.com,1999:blog-8070062627521378361.post-62969025191851863632010-05-17T10:43:00.001+01:002010-05-17T10:44:41.337+01:00Proxy Certificates and Delegation with OPeNDAP Services<div style="font-family: inherit;">I'm back from <a href="http://meetings.copernicus.org/egu2010/">EGU</a> in Vienna and <a href="http://philipkershaw.blogspot.com/2010/04/egu2010.html">one topic I presented on</a> was a solution for securing OPeNDAP services which I've developed working with colleagues at STFC and members of the development team for Earth System Grid. Services configured with the security filter middleware can accept either PKI or OpenID based authentication. So far so good but one piece that's missing and that would greatly enhance the solution would be support for delegation. </div><div style="font-family: inherit;"><br />
</div><div style="font-family: inherit;">Proxy certificates would be the obvious choice since this should easily piggyback on the existing PKI based authentication adopted but alternative technologies like OAuth need more exploration. Just how easy the proxy certificate solution is explored here but first why bother? - It would give immediate compatibility with Grid based authentication schemes and would solve the delegation problem. For example in an upcoming project here a portal is required which is effectively an experimental area for scientists to gather data from different sites and combine it with their own that they upload to it. The portal could query OPeNDAP services on behalf of the user for data retrieval. In the case of a secured OPeNDAP service, the user would need to delegate authority to the portal to use their credentials to access the OPeNDAP service on their behalf.</div><div style="font-family: inherit;"><br />
</div><div style="font-family: inherit;">The sequence below shows how security filter middleware intercept requests to an OPeNDAP service and redirect unauthenticated clients to an Authentication Service running over SSL. At this point if the client submits a certificate in the SSL handshake, the service uses this to authenticate the user. If no certificate is provided, it defaults to OpenID based sign in.<br />
<br />
</div><div style="font-family: inherit;"></div><div class="separator" style="clear: both; font-family: inherit; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzhYpzTFpVugRms1hiYip7iYgvmRtvz_DYJw9yLhuox3NpsXgREY4YaDecXupYx54xKNNeFbigIJCA4n_r5RA86z3LmKCNKA_PPHiNQGiIApamgXcUVuoDsBW-8Ng9Rc7JrGhOxnE5mFD7/s1600/OPeNDAPSSLAuthentication.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="221" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzhYpzTFpVugRms1hiYip7iYgvmRtvz_DYJw9yLhuox3NpsXgREY4YaDecXupYx54xKNNeFbigIJCA4n_r5RA86z3LmKCNKA_PPHiNQGiIApamgXcUVuoDsBW-8Ng9Rc7JrGhOxnE5mFD7/s400/OPeNDAPSSLAuthentication.png" width="400" /></a></div><div style="font-family: inherit;"></div><div style="font-family: inherit;"><br />
This Python snippet shows how the client code can work:<br />
<pre style="background-color: #eeeeee; border: 1px dashed rgb(153, 153, 153); color: black; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>>>> import urllib2
>>> from M2Crypto import m2urllib2, SSL
>>> ctx = SSL.Context('sslv3')
>>> ctx.load_cert_chain('./proxy.pem')
>>> opener = m2urllib2.build_opener(ctx)
>>> data = opener.open('http://localhost:8001/dap/sample.nc')</code></pre><div style="font-family: inherit;">urllib2's redirect handler means that it will conveniently follow the HTTP redirects required in the sequence. M2Crypto plugs into this to provide the SSL support. The SSL context uses load_cert_chain to ensure that the complete certificate chain is loaded for the proxy certificate. For simplicity here no peer certificate validation code has been set.<br />
<br />
</div>The server side filters shown in the diagram are implemented as Python WSGI middleware hosted under Apache with mod_wsgi. The OPeNDAP service is PyDAP. Apache provides the SSL functionality so that the Python SSLAuthentication filter has minimal work to do. This scheme has been tested with client certificates obtained with a MyProxy CA service. Short lived certificates obtained are issued direct from a CA hosted with MyProxy. This presents no problems for Apache mod_ssl configuration. Making a request with a proxy certificate however yields:</div><pre style="background-color: #eeeeee; border: 1px dashed rgb(153, 153, 153); color: black; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>sslv3 alert certificate unknown</code></pre><div style="font-family: inherit;">The OpenSSL code used by mod_ssl doesn't understand proxy certificates and key to the verification process, that it must verify a chain passed by the client of proxy certificate and the user certificate which issued the proxy certificate. Fortunately from OpenSSL 0.9.8g there's a hack to allow support for proxy certificates, simply setting the environment variable to some value:</div><pre style="background-color: #eeeeee; border: 1px dashed rgb(153, 153, 153); color: black; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>export OPENSSL_ALLOW_PROXY_CERTS=1</code></pre><div style="font-family: inherit;">For Apache it means, setting this in the environment of the Apache start-up. This <a href="http://security.ncsa.illinois.edu/research/grid-howtos/usefulopenssl.html">NCSA howto</a> is really helpful. For my Ubuntu Apache configuration, there's an envvars file to which this setting can be added in /etc/apache2/.<br />
<br />
The client verification settings are made in the configuration file:<br />
<pre style="background-color: #eeeeee; border: 1px dashed rgb(153, 153, 153); color: black; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>SSLOptions +ExportCertData
SSLVerifyClient optional
SSLVerifyDepth 10
SSLCACertificatePath /etc/grid-security/certificates/
</code></pre>Verification is optional so that OpenID based sign in can be invoked if no client certificate is submitted. ExportCertData enables the middleware to examine the certificate and use it to authenticate the client e.g. based on certificate Distinguished Name.<br />
<br />
Adding proxy certificate support then has proved fairly painless for the Python/Apache server side implementation. There's a parallel Java Tomcat implementation for ESG. Let's hope this proves as straightforward.<br />
<br />
</div><div style="font-family: inherit;"><br />
</div>Philip Kershawhttp://www.blogger.com/profile/05141787349333619736noreply@blogger.com2tag:blogger.com,1999:blog-8070062627521378361.post-12963745045831548482010-04-29T08:40:00.000+01:002010-04-29T08:40:45.596+01:00New Python MyProxyClient Release.I've just released a <a href="http://pypi.python.org/pypi/MyProxyClient/1.0.0">new version</a> of this Python based implementation of the client interface to the <a href="http://grid.ncsa.illinois.edu/myproxy/">MyProxy </a><br />
credential management service. <br />
<br />
In this release I've added a new method getTrustRoots to support the ability to download the CA certificates for a given MyProxy server (command=7 - see the <a href="http://grid.ncsa.illinois.edu/myproxy/protocol/">protocol page</a>). I've also switched from M2Crypto to PyOpenSSL for the bindings to OpenSSL. M2Crypto offers a broad range of the API but memory and installation issues have finally put me off.<br />
<br />
Put is still not supported as unfortunately, the PyOpenSSL X.509 Extensions interface doesn't support the <i>proxyCertInfo</i> extension type needed for creating proxy certificates.<br />
<br />
This simple example shows how to obtain credentials:<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed rgb(153, 153, 153); color: black; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>$ python
Python 2.6.4 (r264:75706, Dec 7 2009, 18:45:15)
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from myproxy.client import MyProxyClient
>>> clnt = MyProxyClient(hostname='myproxy.somewhere.ac.uk', caCertDir='/home/testuser/.globus/certificates')
>>> from getpass import getpass
>>> creds = clnt.logon('testuser', getpass())
Password:</code></pre><br />
The certificate chain and private key are returned in the "creds" tuple. The "caCertDir" option points to a directory containing the trust roots so that the client authenticate the peer.Philip Kershawhttp://www.blogger.com/profile/05141787349333619736noreply@blogger.com0tag:blogger.com,1999:blog-8070062627521378361.post-76641092214784912992010-04-23T14:34:00.000+01:002010-04-23T14:34:49.904+01:00EGU2010The <a href="http://meetings.copernicus.org/egu2010/home.html">EGU General Assembly</a> is fast approaching. I'm giving these two presentations:<br />
<ol><li><b>Applying the Earth System Grid Security System in a Heterogeneous Environment of Data Access Services</b> <br />
Session: ESSI8 Service-Oriented Architecture solutions for Earth and Space Sciences<strong style="font-weight: normal;"></strong></li>
<ul><li><a href="http://meetingorganizer.copernicus.org/EGU2010/EGU2010-11737-1.pdf">Abstract</a></li>
</ul><li><b>A Flexible Component based Access Control Architecture for OPeNDAP Services</b> <br />
Session: ESSI13 <strong style="font-weight: normal;">Scientific Gateways and Visualization</strong></li>
<ul><li><a href="http://meetingorganizer.copernicus.org/EGU2010/EGU2010-9289.pdf">Abstract</a></li>
</ul></ol><a href="http://meetingorganizer.copernicus.org/EGU2010/EGU2010-9289.pdf"><br />
</a>Philip Kershawhttp://www.blogger.com/profile/05141787349333619736noreply@blogger.com0tag:blogger.com,1999:blog-8070062627521378361.post-84415837534104462572010-01-22T09:29:00.007+00:002010-01-22T11:01:16.997+00:00MyProxy logon via OpenID<span style="font-family:trebuchet ms;">The access control architecture for </span><a style="font-family: trebuchet ms;" href="http://cmip-pcmdi.llnl.gov/cmip5/">CMIP5</a><span style="font-family:trebuchet ms;"> makes use of OpenID and </span><a style="font-family: trebuchet ms;" href="http://grid.ncsa.illinois.edu/myproxy/ca/"><span style="text-decoration: underline;">the MyProxy Certificate Authority</span></a><span style="font-family:trebuchet ms;"> for single sign on. Recently work has been underway to provide filters to secure data server applications with both OpenID and SSL client based authentication front ends. In my case helped along considerably by the </span><a style="font-family: trebuchet ms;" href="http://philipkershaw.blogspot.com/2009/08/wsgi-architecture-for-nerc-datagrid.html">WSGI based architecture adopted</a><span style="font-family:trebuchet ms;">.</span><br /><br /><span style="font-family:trebuchet ms;">An important requirement has been to enable the user community to access data via a browser but also via scripts such as wget and dedicated client software. The issue of delegation will have to be tackled at some point and one example scenario is Live Access Server with accessing TDS instances on behalf of a user. Also in the future perhaps workflows with OGC services. There has been a lot of work done in this area already.</span><br /><br /><span style="font-family:trebuchet ms;">Given we have PKI based authentication incoporated a 'classic' grid based solution with proxy certificates would be possible. Maybe OAuth might be another avenue to look into.</span><br /><br /><span style="font-family:trebuchet ms;">One possibility with OpenID, would be to exploit </span><a style="font-family: trebuchet ms;" href="http://openid.net/specs/openid-attribute-exchange-1_0.html">Attribute Exchange</a><span style="font-family:trebuchet ms;"> (AX) extension to perform an MyProxy logon over this interface. A Relying Party could obtain a user certificate returned across the AX interface from the user's OpenID Provider (OP). MyProxy logon normally involves,</span><br /><ol style="font-family: trebuchet ms;"><li>creation of a key pair at the client</li><li>creation of a certificate request containing the public key</li><li>authentication of the client against a MyProxy Server</li><li>submission of certificate request to the server</li><li>certificate issued by the server and returned to the client</li></ol><span style="font-family:trebuchet ms;">Once obtained the client can use the PKI credentials to authenticate against other services. In this case an OpenID Relying Party (RP) represents a service which wants to access other services on behalf of the user. It could perform steps 1 and 2 but then pass the certificate request to the Provider during the sign in process. Fortunately the AX spec has a </span><a style="font-family: trebuchet ms;" href="http://openid.net/specs/openid-attribute-exchange-1_0.html#store">store message</a><span style="font-family:trebuchet ms;"> feature which could make this possible.</span><br /><br /><span style="font-family:trebuchet ms;">When the user signs in at the Provider, they provide username/password as usual, but the Provider links with its own MyProxy Server and makes a logon client call passing username, password and the certificate request obtained from the RP. If all is well, MyProxy returns a new certificate and the Provider can then pass this back to the RP over the AX interface. The RP now has PKI credentials for the user delegated to it so that it can make calls to other services on the user's behalf. Of course this method only provides one level of delegation but that may be sufficient for many use cases.</span><br /><br /><span style="font-family:trebuchet ms;">The sequence diagram below illustrates the steps:<br /></span><br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnAvaLRkM_YzCE3lL1R1QvUEUyK-6ZgxnE68hxBPkqOCm7P2R9rJ2GgNKkO5y6lns1mNI_O2GUtXd64d58qt3UFLVHLvKuah_ipP4r5AjBNXzoXp0kYEl6UeS073cAe-x4A5Svx3AO0-AG/s1600-h/OpenIDwithMyProxy.png"><img style="margin: 0pt 10px 10px 0pt; float: left; cursor: pointer; width: 287px; height: 400px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnAvaLRkM_YzCE3lL1R1QvUEUyK-6ZgxnE68hxBPkqOCm7P2R9rJ2GgNKkO5y6lns1mNI_O2GUtXd64d58qt3UFLVHLvKuah_ipP4r5AjBNXzoXp0kYEl6UeS073cAe-x4A5Svx3AO0-AG/s400/OpenIDwithMyProxy.png" alt="" id="BLOGGER_PHOTO_ID_5429511116350264610" border="0" /></a>Philip Kershawhttp://www.blogger.com/profile/05141787349333619736noreply@blogger.com0tag:blogger.com,1999:blog-8070062627521378361.post-30099219688915237462009-08-06T10:17:00.000+01:002009-08-07T08:46:23.011+01:00WSGI Architecture for NERC DataGrid SecurityOver the past year I've been porting the <a href="http://www.python.org/">Python</a> based security system for <a href="http://ndg.nerc.ac.uk/">NERC DataGrid</a> to a <a href="http://www.wsgi.org/">WSGI</a> based architecture. This is paying huge dividends in terms of the modularity of the code and ability to apply flexible deployment configurations especially when used together with <a href="http://pythonpaste.org/deploy/">Paste Deploy</a>.<br /><br />Key parts to the architecture are the authentication and authorisation handlers triggered from the respective 401 and 403 HTTP response codes together with a URI based access control policy. I've used the Python security middleware package <a href="http://authkit.org/">AuthKit</a> to help me put this together. One thing I've been meaning to do is to lay this out in a simple example. This first snippet gives an overview:<br /><br /><pre style="border: 1px dashed rgb(153, 153, 153); padding: 5px; overflow: auto; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; color: rgb(0, 0, 0); background-color: rgb(238, 238, 238); font-size: 12px; line-height: 14px; width: 100%;"><code><br />app = AuthorisationPolicyMiddleware(myApp)<br /><br />app = MultiHandler(app)<br />app.add_method("checkerID", AuthenticationHandlerMiddleware)<br />app.add_checker("checkerID", AuthenticationHandlerMiddleware.trigger)<br /><br />app = MultiHandler(app)<br />app.add_method("checkerID", AuthorisationHandlerMiddleware)<br />app.add_checker("checkerID", AuthorisationHandlerMiddleware.trigger)<br /><br />from paste.httpserver import serve<br />from paste.deploy import loadapp<br /><br />serve(app, host='0.0.0.0', port=9080)<br /></code></pre><br />The application to be protected is defined in a WSGI elsewhere. This is wrapped in a number of pieces of middleware chained together to form a pipeline to intercept requests to the application. On the last line it is served using Paste.<br /><br />The first middleware component listed, <code>AuthorisationPolicyMiddleware</code>, checks the requested URI against a policy. If the user is not authorised, it sets a HTTP "403 Forbidden" response bypassing <code>myApp</code>.<br /><br />Following this, there are two pieces of middleware making use of AuthKit's <code>authkit.authenticate.multi.Multihandler</code>. The MultiHandler accepts two key inputs: a checker function which determines the criteria for intercepting a request, and a method, a WSGI middleware to determine what action to take once an intercept has been made.<br /><br />In the first case, a class method <code>AuthenticationHandlerMiddleware.trigger</code> has been defined to intercept HTTP "401 Unauthorized" status codes. The <code>AuthenticationHandlerMiddleware</code> itself determines the action taken:<br /><br /><pre style="border: 1px dashed rgb(153, 153, 153); padding: 5px; overflow: auto; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; color: rgb(0, 0, 0); background-color: rgb(238, 238, 238); font-size: 12px; line-height: 14px; width: 100%;"><code>class AuthenticationHandlerMiddleware(object):<br /></code><code> </code><code>"""Handler for HTTP 401 Unauthorized responses"""<br /><br /></code><code> </code><code>triggerStatus = "401 Unauthorized"<br /><br /></code><code> </code><code>def __init__(self, global_conf, **app_conf):<br /></code><code> </code><code> pass<br /><br /> def __call__(self, environ, start_response):<br /></code><code> </code><code> </code><code>log.info("AuthenticationHandlerMiddleware access denied response ...")<br /></code><code> </code><code> </code><code>response = "HTTP 401 Unauthorised response intercepted"<br /></code><code> </code><code> </code><code>start_response('200 OK', [('Content-type', 'text/plain'),<br /></code><code> </code><code> </code><code> </code><code> ('Content-length', str(len(response)))])<br /></code><code> </code><code> </code><code>return [response]<br /><br /></code><code> </code><code>@classmethod<br /></code><code> </code><code>def trigger(cls, environ, status, headers):<br /></code><code> </code><code> if status == cls.triggerStatus:<br /> log.info("Authentication Trigger caught status [%s]",<br /> cls.triggerStatus)<br /> return True<br /></code><code> </code><code>else:<br /> </code><code> </code><code>return False<br /><br /></code></pre><br />In the above, the middleware simply outputs a message but it effectively provides a hook to trigger a login or other authentication interface.<br /><br />A second Multihandler is in place to handle HTTP "403 Forbidden" responses. This follows a similar pattern:<br /><br /><pre style="border: 1px dashed rgb(153, 153, 153); padding: 5px; overflow: auto; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; color: rgb(0, 0, 0); background-color: rgb(238, 238, 238); font-size: 12px; line-height: 14px; width: 100%;"><code><br />class AuthorisationHandlerMiddleware(object):<br /></code><code> </code><code>"""Handler for HTTP 403 Forbidden responses"""<br /><br /></code><code> </code><code>triggerStatus = "403 Forbidden"<br /><br /></code><code> </code><code>def __init__(self, global_conf, **app_conf):<br /></code><code> </code><code>pass<br /><br /></code><code> </code><code>def __call__(self, environ, start_response):<br /></code><code> </code><code>log.info("AuthorisationHandlerMiddleware access denied response ...")<br /></code><code> </code><code>response = "HTTP 403 Forbidden response intercepted"<br /></code><code> </code><code>start_response('200 OK', [('Content-type', 'text/plain'),<br /> </code><code> </code><code> ('Content-length', str(len(response)))])<br /></code><code> </code><code> return [response]<br /><br /></code><code> </code><code>@classmethod<br /></code><code> </code><code>def trigger(cls, environ, status, headers):<br /></code><code> </code><code>if status == cls.triggerStatus:<br /> </code><code> </code><code>log.info("Authorisation Trigger caught status [%s]",<br /> </code><code> </code><code> cls.triggerStatus)<br /> </code><code> </code><code>return True<br /></code><code> </code><code>else:<br /> </code><code> </code><code>return False<br /><br /></code></pre><br />The <code>trigger</code> method sets a <code>True</code> response to signal to the Multihandler to intercept the request and invoke <code>AuthorizationMiddleware</code> to deliver an access denied message.<br /><br />This next snippet shows <code>myApp</code> effectively a test harness to demonstrate the middleware behaviour:<br /><br /><pre style="border: 1px dashed rgb(153, 153, 153); padding: 5px; overflow: auto; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; color: rgb(0, 0, 0); background-color: rgb(238, 238, 238); font-size: 12px; line-height: 14px; width: 100%;"><code>def myApp(environ, start_response):<br /></code><code> </code><code>"""Test application to be secured"""<br /><br /></code><code> </code><code>if environ['PATH_INFO'] == "/test_401":<br /> </code><code> </code><code>status = "401 Unauthorized"<br /> </code><code> </code><code> response = status<br /><br /> elif environ['PATH_INFO'] == "/test_403":<br /> </code><code> </code><code>status = "403 Forbidden"<br /> response = status<br /><br /> elif environ['PATH_INFO'] == "/secured":<br /> status = "200 OK"<br /> response = "Secured URI"<br /><br /> else:<br /> status = "404 Not Found"<br /> response = status<br /><br /> log.info("Application is setting [%s] response..." % status)<br /> start_response(status,<br /> [('Content-type', 'text/plain'),<br /> ('Content-length', str(len(response)))])<br /><br /> return [response]<br /><br /></code></pre><br />As set-up above, <code><br /></code><ol><li><code>http://localhost:9080/test_401</code> will trigger the authentication middleware and<br /></li><li><code>http://localhost:9080/test_403</code> the authorisation middleware.<br /></li><li>The last, <code>http://localhost:9080/test_secured</code>, demonstrates the access control policy implemented in <code>AuthorisationPolicyMiddleware</code>:</li></ol><pre style="border: 1px dashed rgb(153, 153, 153); padding: 5px; overflow: auto; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; color: rgb(0, 0, 0); background-color: rgb(238, 238, 238); font-size: 12px; line-height: 14px; width: 100%;"><code>class AuthorisationPolicyMiddleware(object):<br /> """Apply a security policy based on the URI requested"""<br /><br /> def __init__(self, app):<br /> self.securedURIs = ['/test_secured']<br /> self.app = app<br /><br /> def __call__(self, environ, start_response):<br /> if environ['PATH_INFO'] in self.securedURIs:<br /> log.info("Path [%s] is restricted by the Authorisation policy" %<br /> environ['PATH_INFO'])<br /> status = "403 Forbidden"<br /> response = status<br /> start_response(status, [('Content-type', 'text/plain'),<br /> ('Content-length', str(len(response)))])<br /> return [response]<br /> else:<br /> return self.app(environ, start_response)<br /><br /></code></pre><br />The middleware has a policy consisting of a list of URIs to be secured in the <code>securedURIs</code> attribute. In practice this could link to a policy file, database link or some other interface. The <code>__call__</code> method intercepts request URIs which match the policy and invokes a HTTP 403 response. This in turn brings into play the <code>AuthorisationMiddleware</code> handler triggering it to return an access denied response.<br /><br />The complete <a href="http://proj.badc.rl.ac.uk/ndg/browser/TI12-security/trunk/python/Tests/wsgiStack/test_multihandler.py">example</a> is in the NERC DataGrid SubVersion repository.Philip Kershawhttp://www.blogger.com/profile/05141787349333619736noreply@blogger.com0tag:blogger.com,1999:blog-8070062627521378361.post-20392392575237977072009-08-03T16:39:00.000+01:002009-08-04T16:08:36.407+01:00Python List Utility ClassesI've been adapting some Java code to Python recently and wanted some tighter control over list elements than the default list type. I've extended list with two custom classes. The first, TypedList restricts list elements to a given type or types e.g.<br /><br /><pre style="border: 1px dashed rgb(153, 153, 153); padding: 5px; overflow: auto; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; color: rgb(0, 0, 0); background-color: rgb(238, 238, 238); font-size: 12px; line-height: 14px; width: 100%;"><code>>>> t=TypedList(float)<br />>>> t+=[9]<br />Traceback (most recent call last):<br />File "<stdin>", line 1, in <module><br />File "<stdin>", line 34, in __iadd__<br />TypeError: List items must be of type float</stdin></module></stdin></code></pre>The existing <a href="http://docs.python.org/library/array.html?highlight=arraytype#module-array">array type</a> gives similar capability but with this you can put in any type ...<br /><br /><pre style="border: 1px dashed rgb(153, 153, 153); padding: 5px; overflow: auto; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; color: rgb(0, 0, 0); background-color: rgb(238, 238, 238); font-size: 12px; line-height: 14px; width: 100%;"><code><br />class TypedList(list):<br /> """Extend list type to enabled only items of a given type. Supports<br /> any type where the array type in the Standard Library is restricted to<br /> only limited set of primitive types<br /> """<br /><br /> def __init__(self, elementType, *arg, **kw):<br /> """@type elementType: type/tuple<br /> @param elementType: object type or types which the list is allowed to<br /> contain. If more than one type, pass as a tuple<br /> """<br /> self.__elementType = elementType<br /> super(TypedList, self).__init__(*arg, **kw)<br /><br /> def _getElementType(self):<br /> return self.__elementType<br /><br /> elementType = property(fget=_getElementType,<br /> doc="The allowed type or types for list elements")<br /><br /> def extend(self, iter):<br /> for i in iter:<br /> if not isinstance(i, self.__elementType):<br /> raise TypeError("List items must be of type %s" %<br /> (self.__elementType,))<br /><br /> return super(TypedList, self).extend(iter)<br /><br /> def __iadd__(self, iter):<br /> for i in iter:<br /> if not isinstance(i, self.__elementType):<br /> raise TypeError("List items must be of type %s" %<br /> (self.__elementType,))<br /><br /> return super(TypedList, self).__iadd__(iter)<br /><br /> def append(self, item):<br /> if not isinstance(item, self.__elementType):<br /> raise TypeError("List items must be of type %s" %<br /> (self.__elementType,))<br /><br /> return super(TypedList, self).append(item)<br /><br /></code></pre>For the second class I wanted a way of avoiding the addition of duplicate elements to a list:<br /><br /><pre style="border: 1px dashed rgb(153, 153, 153); padding: 5px; overflow: auto; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; color: rgb(0, 0, 0); background-color: rgb(238, 238, 238); font-size: 12px; line-height: 14px; width: 100%;"><code><br />>>> u=UniqList()<br />>>> u.append('a')<br />>>> u<br />['a']<br />>>> u.append('a')<br />>>> u<br />['a']<br /><br /></code></pre>It silently ignores the duplicate element. It would be straightforward to alter to raise an exception if this was the preferred behaviour. Here's the class:<br /><br /><pre style="border: 1px dashed rgb(153, 153, 153); padding: 5px; overflow: auto; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; color: rgb(0, 0, 0); background-color: rgb(238, 238, 238); font-size: 12px; line-height: 14px; width: 100%;"><code><br />class UniqList(list):<br /> """Extended version of list type to enable a list with unique items.<br /> If an item is added that is already present then it is silently omitted<br /> from the list<br /> """<br /> def extend(self, iter):<br /> return super(UniqList, self).extend([i for i in iter if i not in self])<br /><br /> def __iadd__(self, iter):<br /> return super(UniqList, self).__iadd__([i for i in iter<br /> if i not in self])<br /><br /> def append(self, item):<br /> for i in self:<br /> if i == item:<br /> return None<br /><br /> return super(UniqList, self).append(item)<br /><br /></code></pre>Philip Kershawhttp://www.blogger.com/profile/05141787349333619736noreply@blogger.com0tag:blogger.com,1999:blog-8070062627521378361.post-38554375076942619972009-07-23T15:52:00.000+01:002009-08-04T16:17:23.694+01:00PyDev, PyLint and RefactoringI've been using <a href="http://pydev.sourceforge.net/">PyDev</a> the Eclipse plugin for Python for some time now and it's certainly improving with each new release (I'm currently on 1.4.2). There are a couple of features I've been getting some benefit from recently.<br /><br />With <a href="http://pypi.python.org/pypi/pylint">PyLint</a>, I actually broke my Eclipse set-up for this a while ago but when I got it reinstated it's reminded me of how useful it is. One thing I hate about about interpreted languages is the huge scope for runtime errors. The PyLint plugin gives immediate feedback in the margin of the editor window with error information and hints. This is saving me a lot of time down the line. What's the metric for time spent fixing a bug immediately vs. at the other end of the scale trying to pick through code on a production deployment? :) Picking up references to undeclared variables is particularly useful. Especially in error blocks or places where test coverage might miss out. There's no doubt some messages can be annoying and getting a mark out of 10 for your code might not appeal to all. Fortunately, you can edit the settings for the Pylint command line that's executed and explicitly filter out warnings you don't like. The second feature I've revisited is the PyDev refactoring options. I have had a low expectation of these perhaps unfairly but there a couple of potential time savers. Take this simple class:<br /><pre style="border: 1px dashed rgb(153, 153, 153); padding: 5px; overflow: auto; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; color: rgb(0, 0, 0); background-color: rgb(238, 238, 238); font-size: 12px; line-height: 14px; width: 100%;"><code><br /><br />class MyTest(object):<br /> def doSomething(self):<br /> self.__a = None<br /> self.__b = None<br /> self.__c = []<br /></code></pre>Left like this PyLint will warn me that I haven't set up these attributes in an __init__ method. ;) PyDev refactoring to the rescue :) ... If I now right click in the editor and pick "Refactoring" -> "Generate Constructor using Fields ...". I'm taken through a series of steps in a dialog. I can get,<br /><pre style="border: 1px dashed rgb(153, 153, 153); padding: 5px; overflow: auto; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; color: rgb(0, 0, 0); background-color: rgb(238, 238, 238); font-size: 12px; line-height: 14px; width: 100%;"><code><br />class MyTest(object):<br /> def __init__(self, a, b, c):<br /> self.__a = a<br /> self.__b = b<br /> self.__c = c<br /><br /> def doSomething(self):<br /> self.__a = None<br /> self.__b = None<br /> self.__c = []<br /><br /></code></pre><br />Not that exciting but could save some typing. More useful for me is the "Generate Properties..." to generate properties for <a href="http://www.python.org/doc/newstyle/">new style classes</a> together with their getters and setters:<br /><br /><pre style="border: 1px dashed rgb(153, 153, 153); padding: 5px; overflow: auto; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; color: rgb(0, 0, 0); background-color: rgb(238, 238, 238); font-size: 12px; line-height: 14px; width: 100%;"><code><br />class MyTest(object):<br /> def __init__(self, a, b, c):<br /> self.__a = a<br /> self.__b = b<br /> self.__c = c<br /><br /> def getA(self):<br /> return self.__a<br /><br /> def getB(self):<br /> return self.__b<br /><br /> def getC(self):<br /> return self.__c<br /><br /> def setA(self, value):<br /> self.__a = value<br /><br /> def setB(self, value):<br /> self.__b = value<br /><br /> def setC(self, value):<br /> self.__c = value<br /><br /> def delA(self):<br /> del self.__a<br /><br /> def delB(self):<br /> del self.__b<br /><br /> def delC(self):<br /> del self.__c<br /><br /> def doSomething(self):<br /> self.__a = None<br /> self.__b = None<br /> self.__c = []<br /><br /> a = property(getA, setA, delA, "A's Docstring")<br /><br /> b = property(getB, setB, delB, "B's Docstring")<br /><br /> c = property(getC, setC, delC, "C's Docstring")<br /><br /></code></pre>It could be improved on - there's no explicit keywords inputs for property() - but it could save some time with the boiler plate. :)<br /><p></p>Philip Kershawhttp://www.blogger.com/profile/05141787349333619736noreply@blogger.com0tag:blogger.com,1999:blog-8070062627521378361.post-46057068293153971602008-04-02T10:29:00.009+01:002010-08-11T10:36:23.120+01:00Calling Python from PerlAfter a long silence this is definitely worth an entry - the Perl Inline module for calling code in various other languages from Perl... <br />
<br />
I've been looking into integrating NERC DataGrid Security (all written in Python) into the existing BADC web site Perl CGI code - quick dirty if necessary.<br />
<br />
The NDG Security Single Sign On service has a http interface so this should be straightforward - Perl on one side of the interface and Python the other. However the various SOAP interfaces with WS-Security added into the mix doesn't make it very attractive. I've not looked into Perl SOAP support - and this may be great - but if we are migrating all our Perl code to Python it would seem a wasted effort. <br />
Inline is really impressive. I had a python client call to the NDG Attribute Authority working in minutes - Perl calling Python calling a SOAP Service and returning the response back to Perl as a local variable:<br />
<br />
Perl <-> Python <-> SOAP + WS-Security <-> Python Attribute Authority Web service<br />
<br />
Installation was fairly straightforward on Ubuntu - <span style="font-family: "Courier New",Courier,monospace;">libinline-perl</span> package is available but <span style="font-family: "Courier New",Courier,monospace;">Inline-Python</span> required downloaded and install from <i>CPAN</i>.<br />
<br />
I was expecting all kinds of shared library link errors but all works perfectly. It's nice when things work simply easily and quickly for a change :)<br />
<br />
<pre style="background-color: #eeeeee; border: 1px solid rgb(153, 153, 153); color: black; font-family: Andale Mono,Lucida Console,Monaco,fixed,monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 5px; width: 100%;"><code>
#!/usr/bin/env perl
use Inline Python => <<'END';
def getAAHostsInfo():
from ndg.security.common.AttAuthority import AttAuthorityClient
import os
aaClnt = AttAuthorityClient(uri="http://localhost:5000/AttributeAuthority",
cfgFilePath='wssecurity.cfg')
allHostsInfo = aaClnt.getAllHostsInfo()
return "All hosts info = %s" % allHostsInfo
END
print getAAHostsInfo(), "\n";
</code></pre>Philip Kershawhttp://www.blogger.com/profile/05141787349333619736noreply@blogger.com0