Using Shibboleth as an SSO

University at Buffalo

Joel Murphy

jmurphy@buffalo.edu

overview

• Id mgmt @ UB• UB deployment Timeline• Deployment considerations• Certificate management• LDAP• Load Balancer• Web Farm and Service Providers• Webmail proposed configuration• Cosign• Shibboleth Identity Provider• Load Testing• Challenges

Id Mgmt @ UB

• “UBIT Name” branding– All University affiliated persons (faculty, staff,

students, etc.)– single username/password, many directories, many

applications– usernames and passwords and groups synced across

directories– One user namespace for campus– People retain same username after returning to Univ

• Unix groups primary mechanism for authorization (authz) and entitlements

• Applications responsible for authorization

Id Mgmt @ UB

• “Accounts Management” oracle database and application core to Id Mgmt– Home grown Perl, C, oracle SQL

• Downstream Directories– DCE – authentication, authorization, dir svcs– LDAP – dir svcs, authn via Kerberos, authz– Kerberos – authn, enables encrypted authenticated

communication channels– Active Directory

• X.509 Certificates

Password management

• password replication across directories

Long road to SSO• Pre-1997 – Unix passwords/NIS – just Unix services• 1997 - DCE – first success at consolidation of Id mgmt and password

database, multiple platforms• 1998 – DCE/DFS deployed as enterprise filesystem• 2003 – Kerberos/Active Directory/LDAP/Shibboleth/Cosign/Cisco CSS• Summer 2005 – production Oracle RAC database• Fall 2005 – First production Shibboleth Services – Course Management,

Download service• Fall 2005 - LDAP Campus Portal and administrative apps (was DCE)• Summer 2006 – Network Appliance deployment for DFS replacement for

enterprise filesystem (CIFS/NFS)• Summer 2006 – Business Continuity site deployment and machine move• Fall 2006 – Shibbolize authn portions of Campus web server• Fall 2006 – Shibbolize Campus Portal and apps (from LDAP)• Fall 2006 or Spring 2007 – Shibbolize Web Mail (IMP or @mail)

Id Mgmt/SSO Outliers

• Enterprise Active Directory “exception” accounts (eg. OU administrators) – not in Kerberos/LDAP

• Science and Engineering NIS passwd (migrating to enterprise Kerberos for auth)

• Various Active Directory Forests outside Enterprise AD

• Various applications with limited technical resources or need for Enterprise integration

Id Mgmt challenges

• Authorization not always done or works by “accident” (user not in /etc/passwd)

• Application owners don’t always know who should be authorized – typically request a group

• DCE still core for to IdM– Authoritative for groups and data for deactivated users– RPCs use heavily, replacing with SOAP/SSL

• CIO level requests– Better authorization for network access– Online password reset

• Requests for LDAP access for outsourced apps• Hard to get resources for not-so-low hanging fruit

Some Shibboleth/SSO and Web Farm Goals/Considerations

• Survive network split (multiple campuses)• Don’t preclude logoff• Ability Load balance commodity hardware• Detect and Isolate failures - Transparent failover

of Identity Providers and Service Providers (multiple machines, same configuration)

• Capacity/Responsiveness – Shibboleth is Enterprise SSO

• Commitments made to sell Shibboleth solution

How we got there

• Internal certificate signing service• LDAP environment for our attribute data (eduPerson,

UBEduPerson)• Cisco CSS (Content Switch) load balancer • Multi-purpose Oracle RAC database (“Real Application

cluster” - active/active)– Hitachi SAN

• Cosign SSO• Shibboleth• Lots of glue• Lots of help and contributions from UB and I2 community

Did we make it?• Survive network split (multiple campuses)

– No – multiple load balancers, but can’t migrate IPs across subnets, opted against DNS delegation

– Business continuity site not on one of the campuses (no people)• Don’t preclude logoff

– Supported in Cosign, limited Shib 2.0 support?• Detect and Isolate failures - Ability Load balance commodity hardware

– Large farm of Sun 280R Solaris servers for admin applications– Farm of 8 Dell 1750 Linux servers for shibboleth/cosign– 4 Sun 280R Solaris servers for LDAP– 4 Sun V120 Solaris servers for Kerberos

• Transparent failover of Identity Providers and Service Providers– Yes, except for failure mid-transaction

• Capacity/Responsiveness – Shibboleth is Enterprise SSO– Yes, our auth systems are very under capacity– Adding 2 additional LDAP servers (new/bad applications with unpredictable

performance profiles)

Physical Layout

Shibboleth IDP Setup• InCommon member• Apache 1.3/Tomcat connector/Tomcat• Handle Service authenticated by Cosign, trusted by Shib• Still at version 1.2 IDP (origin)• Hindsight: put IDP source tree in CVS• Web Farm setup requires Shib Handle configured as “CryptoHandleGenerator”

instead of shared memory• All Attributes currently resolved via LDAP (from Person objects)

– Non-web apps may have similar needs• Entitlements are currently mapped from group memberships and stored in LDAP• Once configured, IDP changes very infrequently• Attribute release is negotiated when configuring new targets• Meta-data for Federations (sites and trusts) changes periodically• We continue to deploy all configuration in one “war” file.• Log files are somewhat large and unwieldy• No deterministic policies, data custodians authorize attribute release

Application Web Farm/Shibboleth SP Setup

• Apache 1.3• Caching apache proxy in front of application apache• Shibboleth integrated into application apache• Apache proxy/cache runs SSL• Machines load balanced with Cisco CSS• Currently assume UB IdP for applications• Standard attribute release and acceptance• Multiple services (virtual hosts) on one machine

– Haven’t needed to adjust ARP for Virtual Hosts• One certificate and ARP shared by all web farm machines• Modified MySQL cache to be Shibboleth Oracle shar cache

– Sticky connections on CSS makes unnecessary– Was one source of early stability problems

Application Web Farm/Shibboleth SP Issues

• Canonical Service names – Forcing Canonical host names (UseCanonicalName) makes IDP configuration of

AssertionConsumerServiceURL simple.– Prevents dreaded error message from Handle Service (unknown ACS URL…)– Makes it very difficult for developers or sysadmins to poke a specific machine

(use hosts file)– Alternative – list every possible hostname/short name in meta-data?

• Web cache– Cookie based authentication doesn’t prevent caching of data where HTTP

BasicAuth does automatically– Need to set header in response of authenticated pages: “Cache-Control: private"

• Web proxy (URL rewrite)– We needed to set checkAddress=“false” in shibboleth XML– ShibURLScheme, ServerName and Port needed for apache

• Shibboleth needs to know how to write a “return address” for redirects• Multiple virtual hosts, odd port mapping schemes allow us to run proxy and server on

same hosts

Shibboleth SP Issues

• Provide a “UB” shibboleth distribution and instructions– Build out of date, XML config, metadata and instructions mostly

valid– Where to put certs– What startup files are needed– What should be monitored– What settings to tweak in a web farm environment

• We generally recommend rebuilding apache module for local apache– Unnecessary?

• New versions of shib from I2 come as packages and are significantly simpler than the past

Blackboard/Course Management

• Shibboleth just used for SSO (authentication)

• Nightly data processing feeds Blackboard system data from institutional systems– Creates users object in Blackboard– Manages course registrations

Download Service

• Shibboleth used as SSO

• Group membership used for authorization– Authorization based on licensing– Microsoft software download

Campus Portal

• Shibboleth to be used as SSO

• Current phase is to transition from LDAP auth (was DCE)

• Nightly processing generates data about people authorized portal in portal application

Webmail Proposal• Cyrus IMAP email• Would like to link Campus Portal/Webmail/Course mangement• Multiple possibilities

– Integrate with cosign directly and use Kerberized IMAP session– Shibbolize webmail and use secure fabricated password for auth– Trusted webmail– Pass Kerberos ticket via Shibboleth Attribute???– Upcoming Shibboleth solution for n-tier apps???

• Concerns/Issues– Desire to abstract out Cosign– Fabricated password must not be reproducible or usable via normal

IMAP connections– Fabricated password requires maintaining a password file– Kerberizing IMAP possibly more Webmail customization than tinkering

with password

Webmail Proposal

• Current working proposal is to use shib with fabricated passwords

Shibboleth Authorization• Our group management is inflexible (DCE still authoritative)• Working to remove DCE, would like Grouper (groups of groups?)• Business rules don’t always match single group membership• Needed to match sets of groups• Require membership in active and some staff group:

ShibRequireAll onrequire memberof ~ ^…staff$require memberof ~ ^active$

• Require membership in active or some staff group:ShibRequireAll offrequire memberof ~ ^…staff$require memberof ~ ^active$

• Shib1.3b adds XML access control, nested requirements– https://authdev.it.ohio-state.edu/twiki/bin/view/Shibboleth/XMLAccessControl

• Using mod_setenvif/mod_access does not work with Shibboleth environment variables

– Shibboleth module doesn’t have a chance to generate env variables before mod_access and order is Apache hard coded, not dependant on module order

Cosign Setup

• SSO just for Shibboleth Handle Service• Cosign supports logoff (hidden)• Cosign can pass Kerberos tickets and do

certificate based auth• SSO’s at the time of our deployment did not

support logon server redundancy.• Cosign writes state information to disk, extended

this to write state to Oracle.• Oracle transactions through an external server

(cookie_server)

Cosign Issues

• Custom code makes upgrading difficult– Backported fixes from newer cosigns

• Bugs in our oracle interface• Oracle RAC configured Active/Active, but updates not

synchronous– Needed max_commit_propagation_delay=0 on Oracle RAC

instances

• Newer cosign versions have async replication of state– Could loose some state in logon server failure

• Move state files from Oracle RAC to HA Netapp NFS share?

Cosign diagram

LDAP Environment• 4 load balanced Sun ONE 5.2 Directory Servers• One giant directory

– ou=People, dc=buffalo, dc=edu• posixAccount• Person• OrganizationalPerson• InetOrgPerson• eduPerson• UBEduPerson

– ou=Groups, dc=buffalo, dc=edu• posixGroup• groupOfUniqueNames

• Added institutional IDs and UBEPmemberOf to UBEduPerson• memberOf on Person makes shib Attribute release simple and fast• Future deployments should use eduMember schema

– http://middleware.internet2.edu/dir/docs/internet2-mace-dir-ldap-group-membership-200507.html

LDAP Environment (cont)• Hourly processing updates LDAP with changes from Id mgmt database

(Perl Net::LDAP)• Oracle layout

– One row per principal– One column per attribute, multi-valued attributes glued with a separator

• Goal to co-locate non-institutional data in directory– Available via LDAP or via Shibboleth– Allow administrators to manage source oracle table or write changes to LDAP

• All Access management policy at the root of the directory granting to groups, some based on attribute value (FERPA)

• Avoids confusion by avoiding object acls or deny acls• Multi-master replication – supports application writes with load balancing• No passwords – simple binds proxied to Kerberos with a modified version of

preAuth plugin by Mike Gettes– Forces simple binds over SSL

LDAP Issues

• Large groups don’t perform well, use memberOf• Can’t “browse” the directory – administrative limits

– Last modified time• DS 5.1 didn’t support multi-master replication• Introduced memory leak in Kerberos pre-auth plugin

when porting to DS 5.2• Replication of schemas created redundant definitions

(fixed?)• Replication failures with large directory and lots of

changes– Fixes for bugs 6242420 and 6283717 included in DS 5.2 patch 4– Turned multi-master back on to support LDAP applications

needing to write data

Internal Certificate Management

• Manages certs for test servers, shibboleth SP’s and VPN access

• Openssl on Id mgmt system• requests via email• Daily job for monitoring expirations• No audit trail and little metadata associated with

certs• Not particularly safe for multiple administrators• Tightening access/clarifying procedures• Plan on moving to OpenCA

Load Balancer Setup

• Began with Cisco Local Directors for email system load balancing (SMTP/IMAP/POP)

• Current - Two Cisco CSS 11503, active/standby• Load balanced services on private network behind load

balancers, using normal UB address space• Servers see IP address of Clients• NAT, health monitoring, sticky connections, load

balancing• Standard architecture for all UB load balanced services

(LDAP, Shibboleth, Cosign, web farm, etc.)• Ability to directly connect to a specific machine in a pool

Load Balancer Issues• Services see the IP of client connecting• Can easily roll-in/deploy and test changes with no downtime• Network management of private net• Everything on one subnet

– Allows migrating IPs between CSS– Extended subnet to Bus. Cont. site, but not other campus sites

• Danger of private net connecting to public• Load Balanced services talking to other load balanced services

– Must NAT both source and destination host• Sysadmin Management

– Too many services, anyone can take everything down– Shared serial console (HTTP disabled for security/auditing concerns)– Many ports on many machines– Secure web tool in development

• Monitoring services, removing from pool– CSS can monitor HTTP results for testing server health

• Notification for down servers (hooked into Big Brother)

Load Balancer example

Troubleshooting

• Check our Big Brother console for alerts

• Check Load balancer – what’s active?

• Application Service provider– Check access_log/error_log– Check Shibboleth shar.log and shire.log– Is shar (shibd) running?– Can we connect to Shib AA with openssl

s_client?

Troubleshooting (cont)

• Identity Provider– Are apache/cosignd running and accepting

connections?– Is cookie_server running?– Check all apache logs– Check tomcat bootstrap log – did shib start?– Check shib application logs

Load Testing

• Webload (Radview)

• JMeter

• Perl LWP

• SAR (Linux/Solaris)

• Infrastructure survives intense load testing

• Shibboleth initial logon “costs” lessen with longer sessions (perhaps less in some cases?)

Challenges

• Transitioning into production– Project team not all of support team

• Too much change at one time– Service maturity takes time

• Lots of layers– Lots of machines and services– Troubleshooting complex– Errors don’t go travel upstream well– Multiple support teams

• Help Desk support complex, especially for Fed apps