EGI-InSPIREAustin 21 October 2010 Nova, Swift Bexar 3 February 2011 Nova, Glance, Swift Cactus 15 April 2011 Nova, Glance, Swift ... { message queue message hub. May 22, 2014 20 EGI-InSPIRE

EGI-InSPIRE

OpenStack Hands-On

Alvaro Lopez Garcıa, Enol Fernandez del [email protected] National Resarch Council

May 22, 2014 1

EGI-InSPIRE RI-261323 www.egi.eu

Part 1: OpenStack projectPart 2: OCCI at OpenStackPart 3: VOMS in OpenStackPart 4: Hands on

May 22, 2014 2


Part I

OpenStack project

May 22, 2014 3


Outline

Introduction

OpenStack Architecture

OpenStack Components

OpenStack authenticationKeystone conceptsAuthentication process

May 22, 2014 4


What is OpenStackI’m not sponsored by OpenStack!

• Cloud middleware (an obvious one).

• Aims to manage and orchestrate compute, storage andnetwork resources.

• OpenStack is based on a global collaboration.

• Quite simple to implement and deploy.

• Feature rich, open to new features.

• Massively scalable (discrete pluggable components).

May 22, 2014 5


Who is behind OpenStack

• Initially founded by Rackspace and NASA.

• 2012: OpenStack Foundation: Independent body.

– Protect, empower and promote OpenStack.– Anyone can join: More than 7000 individual members, more

than 850 organizations.

• Code is Free Software: Apache License.

– More than 1000 contributors.– Code is peer-reviewed, discussed and tested (unit and

functional testing) before it is merged.– Anybody can contribute.

May 22, 2014 6


OpenStack Releases

Release Name Release Date Included ComponentsAustin 21 October 2010 Nova, SwiftBexar 3 February 2011 Nova, Glance, SwiftCactus 15 April 2011 Nova, Glance, SwiftDiablo 22 September 2011 Nova, Glance, SwiftEssex 5 April 2012 Nova, Glance, Swift, Horizon, Keystone

Folsom 27 September 2012Nova, Glance, Swift, Horizon, Keystone,Quantum, Cinder

Grizzly 4 April 2013Nova, Glance, Swift, Horizon, Keystone,Quantum, Cinder

Havana 17 October 2013Nova, Glance, Swift, Horizon, Keystone,Neutron, Cinder

Icehouse 17 April 2014Nova, Glance, Swift, Horizon, Keystone,Neutron, Cinder, Ceilometer, Heat

Juno October 2014Nova, Glance, Swift, Horizon, Keystone,Neutron, Cinder, Ceilometer, Heat, etc

May 22, 2014 7


Outline

Introduction




May 22, 2014 8


OpenStack Conceptual Architecture

May 22, 2014 9


OpenStack Logical Architecture

May 22, 2014 10



• Everything has an API.

• Almost everything can be scaled horizontally.

• Some components have evolved into separate projects (forexample cinder evolved from nova-volume).

– Normally they are forklifted so that the new component is justa drop-in replacement.

– One exception: Neutron (before Quantum) for nova-network.

May 22, 2014 11


Outline

Introduction




May 22, 2014 12



• Dashboard (horizon) provides a web front end to the otherOpenStack services.

• Compute (nova) provides virtual servers on demand.

• Network (neutron) provides virtual networking for Compute.

• Block Storage (cinder) provides storage volumes for Compute.

• Image (glance) stores the images and metada for nova.

• All the services authenticate with Identity (keystone).

• Telemetry (ceilometer) tracks the unsage of resources.

• Orchestration (heat) uses a special language to deploy andorchestrate several services on top of the cloud.

May 22, 2014 13


Compute (nova)

• Provides virtual servers on demand.• Several hypervisors supported: KVM, Xen, XenServer,

VMWare, etc.• Broken down in several sub-components:

– nova-api: OpenStack Compute API, EC2, OCCI (not natively).– nova-compute: spawns the instances– nova-scheduler: schedules the requests– nova-consoleauth: provides auth for VNC requests– nova-xvpcproxy: VNC proxy– nova-conductor: Makes queries to the database trough RPC.– nova-cert: Handles certificates– nova-volume (replaced with cinder): Block device storage– nova-network (replaced with neutron): Networking capabilities.– database: stores the state of the cloud (configured IPs, running

instances, available flavors, etc.)– message queue: hub for passing messages and RPC calls.

• Internal RPC API via message queues (RabbitMQ, 0MQ).May 22, 2014 14


Storage (swift)

• Provides BLOB storage.

• Several Components

– swift-proxy-server: accepts incoming requests via theOpenStack Object API

– Container servers manage a mapping of containers (i.e folders)within the object store service.

– Object servers manage actual objects (i.e. files) on the storagenodes.

– Periodic processes such as replication services, auditors,updaters and reapers.

May 22, 2014 15


Image (glance)

• Virtual images registration and catalog.

• Stores the virtual images in pluggable backends (disk, swift,Ceph, etc.)

• Subcomponents:

– glance-api: image discovery, retrieval, creation and storage.– glance-registry: storage and retrieval of metadata.– glance-cache, glance-reaper, glance-replication.– repository: actual storage for image files.– database.

May 22, 2014 16


Networking (neutron)

• Provides networking capabilities to the instances.

• Replacement for nova-network.

• Several components

– neutron-sever: accepts API requesta and routes to theappropriate plugin

– netron-plugins: Agents performing operations

I Plugging ports into the virtual switches.I Create neworks, IP addresses.I DHCP, L3 agent.

– message queue.

May 22, 2014 17


Dashboard (horizon)

• Provides access and management trough a web interface.

• Developed in Django.

• Extensible and modular.

• Uses the public APIs.

May 22, 2014 18


Identity (keystone)

• Provides authentication and authorization.

• Central authentication hub for all authentication related tasks.

• Provides several functionalities:

– User and project management.– Role (permissions) management.– Service catalog: provides the users with a catalog of services

they can access.

•

May 22, 2014 19


Block storage (cinder)

• Provides volumes (block storage) to the instances.

• Replacement for old nova-volume.

• Several components:

– cinder-api– cinder-scheduler schedules the requests into a cinder-volume– cinder-volume acts upon the requests, managing the actual

block device. Several backends: iSCSI + LVM, NetApp, etc.– message queue message hub.

May 22, 2014 20



May 22, 2014 21


Outline

Introduction




May 22, 2014 22


AuthN and AuthZ in OpenStack

• Authentication and Authorization is orchestrated around theidentity service Keystone.

• AuthN/Z is based on the following concepts:

User A representation of somebody or something using OpenStack.Tenant or group, project: Container to group or isolate users and

resources.Domain Administrative boundaries.

Role A set of rights and privileges, applied to a user.Service A OpenStack service (nova, glance, etc.)

Endpoint An URL from where a user can access a Service.Token A piece of text (arbitrary or not) used to access resources. A

token contains the set of roles for a user.

May 22, 2014 23



• A user is member of 1 or more tenants.

• A tenant (group, project) is part of 1 or more domains.

• A user may have specific roles within a tenant or globallywithin a Keystone domain.

• A token may be associated with a tenant or not:

– Unscoped tokens are not associated with a tenant. Used fordiscovery (available tenants, endpoints) and are onlyunderstood by Keystone.

– Scoped tokens are associated within a tenant and are requiredto interact with any other component.

• A token can be unsigned (UUID) or signed (PKI based).

May 22, 2014 24


Authentication diagram

May 22, 2014 25


How authentication works

• Authentication in Keystone is a 2 part mechanism.1. 1st phase: A user initiates authentication against Keystone

and a token is issued.2. 2nd phase: The token is used to authenticate against all the

other OpenStack services.

• All authenticated requests require a scoped token.• A token has a limited validity.

– Valid within only one tenant.– Fixed expiration time.

• The token is verified with each of the requests by all of theOpenStack components.

– UUID tokens are validated online: it requires a call back to theKeystone server.

– PKI tokens can be verified offline: CMS signed message.

• Role based authorization (RBAC).

May 22, 2014 26


Part II

OCCI at OpenStack

May 22, 2014 27


OCCI at OpenStack

• There is no official release of OCCI for OpenStack.

• OpenStack decided to only support their native API.

• The only exception is the EC2 Compatibility layer.

May 22, 2014 28


OCCI at OpenStack

• OCCI-OS interface started by Thijs Metsch from Intel andAndy Edmonds with community contributions.

• Code locations:

– Stackforge will become the official place (being migrated):https://github.com/stackforge/occi-os

– Thjis repository (development):https://github.com/tmetsch/occi-os

– My repo (development with additions):https://github.com/alvarolopez/occi-os/

• stable/<release name branches should contain stable codeto be deployed with the corresponding version.

• Usage documentationhttps://wiki.openstack.org/wiki/Occi

May 22, 2014 29


https://github.com/stackforge/occi-os

https://github.com/tmetsch/occi-os

https://github.com/alvarolopez/occi-os/

https://wiki.openstack.org/wiki/Occi

OCCI-OS Installation

• Install the code and dependencies. You should ensure that youare using the correct branch from the code.

$ pip install pyssf

$ git clone https://github.com/alvarolopez/occi-os/

$ cd occi-os

$ python setup.py install

• Currently, use the master branch for Havana.

May 22, 2014 30


OCCI-OS Configuration

• Add it to the nova’s api-paste.ini configuration:

[composite:occiapi]

use = egg:Paste#urlmap

/: occiapppipe

[pipeline:occiapppipe]

pipeline = authtoken keystonecontext occiapp

# with request body size limiting and rate limiting

# pipeline = sizelimit authtoken keystonecontext ratelimit occiapp

[app:occiapp]

use = egg:openstackocci-havana#occi_app

• Enable it in your nova.conf file:

enabled_apis=ec2,occiapi,osapi_compute,osapi_volume,metadata

• Restart nova-api and you’re done.

May 22, 2014 31


Use it!

• First get a keystone token:

$ curl --insecure -H \

"Content-type: application/json" -d ’{"auth": \

{"tenantName": "whatever", "passwordCredentials": \

{"username": "demo", "password": "secret"}}}’ \

https://keystone.example.org:5000/v2.0/tokens

(...)

"token": {

"expires": "2013-09-20T14:34:54Z",

"id": "ae6259e89fc8434a8d7122e1f9fdc0f0",

"issued_at": "2013-09-19T14:34:54.827264",

(...)

• Grab the token ID:

$ export KID=ae6259e89fc8434a8d7122e1f9fdc0f0

May 22, 2014 32


Use it!

• See what you can provision:

$ curl -v -H ’Content-Type: text/occi’ -H ’X-Auth-Token: ’$KID \

-X GET http://cloudapi.example.org:8787/-/

May 22, 2014 33


Use it!

• Spawn a virtual machine:

$ curl -v -X POST http://cloudapi.example.org:8787/compute/ \

-H ’Category: compute; scheme="http://schemas.ogf.org/occi/infrastructure#"; class="kind"’ \

-H ’Content-Type: text/occi’ \

-H ’X-Auth-Token: ’$KID \

-H ’Category: m1-tiny; scheme="http://schemas.openstack.org/template/resource#"; class="mixin"’ \

-H ’Category: 18d99a06-c3e5-4157-a0e3-37ec34bdfc24; scheme="http://schemas.openstack.org/template/os#"; class="mixin"’ \

-H ’Category: public_key; scheme="http://schemas.openstack.org/instance/credentials#"; class="mixin"’

May 22, 2014 34


Outline

Wrap up

May 22, 2014 35


Wrap up

• Code locations:

– Stackforge will become the official place (being migrated):https://github.com/stackforge/occi-os

– Thjis repository (development):https://github.com/tmetsch/occi-os

– My repo (development with additions):https://github.com/alvarolopez/occi-os/

• stable/<release name> branches should contain stablecode to be deployed with the corresponding version.

• Usage documentationhttps://wiki.openstack.org/wiki/Occi

May 22, 2014 36


https://github.com/stackforge/occi-os

https://github.com/tmetsch/occi-os

https://github.com/alvarolopez/occi-os/

https://wiki.openstack.org/wiki/Occi

Part III

VOMS in OpenStack

May 22, 2014 37


Outline

Virtual Organizations

VOMS in OpenStackKeystone AuthenticationKeystone VOMS module for v2 AuthVOMS module for Keystone v3 Auth

Wrap up

May 22, 2014 38



• Collaboration context, not bounded to a particularorganization.

• Group of individuals anr/or institutions emerged from a set ofresource sharing rules.

• A user belonging to a VO can access a given set of resources.

– Using the same set of credentials.– Across different resource providers (or even infrastructures).– Different roles and groups inside the VO to model different

access rules.

• A provider contributes with its resources (e.g. computing,storage)

– Fine-grained control over what it is shared and not.– VO management (user creation, revokal) is leveraged to the

VO administrators.

May 22, 2014 39


Virtual OrganizationsVOMS

• VOMS is the acronym for Virtual Organization MembershipService.

• Attribute authority which serves as central repository for VOuser authorization information.

• Initially based on X.509 proxy certificates, now with SAMLsupport.

• De-facto tool used in the Grid world for authentication andauthorization.

• Assigns roles and grouop to VO users.

• Emits signed assertions, so resource providers can trust them.

May 22, 2014 40


VOMS Authentication in the Cloud

Apply VOs to the Cloud using VOMS-based authentication.

• VOMS is the standard tool in the Grid. The infrastructure isalready in place (PKI, VOMS servers, portals, etc.)

• User communities are familiar with it.– No extra credentials for users– No extra effort for managers.– No transition effort.

• Resource providers are familiar with it.– No extra effort for configuration.

• Grid tools can be easily adapted to interact with cloudtestbeds

• Integrated (or possible integration) with other operationaltools.

• Extensible (for example it is possible to move towards SAML).• Possible integration with Shibboleth (remove the certificate

management burden).May 22, 2014 41


Outline



Wrap up

May 22, 2014 42



May 22, 2014 43



• Authentication and Authorization is orchestrated around theidentity service Keystone.

• AuthN/Z is based on the following concepts:

User A representation of somebody or something using OpenStack.Tenant or group, project: Container to group or isolate users and

resources.Domain Administrative boundaries.

Role A set of rights and privileges, applied to a user.Service A OpenStack service (nova, glance, etc.)

Endpoint An URL from where a user can access a Service.Token A piece of text (arbitrary or not) used to access resources. A

token contains the set of roles for a user.

May 22, 2014 44



• A user is member of 1 or more tenants.

• A tenant (group, project) is part of 1 or more domains.

• A user may have specific roles within a tenant or globallywithin a Keystone domain.

• A token may be associated with a tenant or not:

– Unscoped tokens are not associated with a tenant. Used fordiscovery (available tenants, endpoints) and are onlyunderstood by Keystone.

– Scoped tokens are associated within a tenant and are requiredto interact with any other component.

• A token can be unsigned (UUID) or signed (PKI based).

May 22, 2014 45


Authentication diagram

May 22, 2014 46


How authentication works

• Authentication in Keystone is a 2 part mechanism.1. 1st phase: A user initiates authentication against Keystone

and a token is issued.2. 2nd phase: The token is used to authenticate against all the

other OpenStack services.

• All authenticated requests require a scoped token.• A token has a limited validity.

– Valid within only one tenant.– Fixed expiration time.

• The token is verified with each of the requests by all of theOpenStack components.

– UUID tokens are validated online: it requires a call back to theKeystone server.

– PKI tokens can be verified offline: CMS signed message.

• Role based authorization (RBAC).

May 22, 2014 47


Keystone VOMS middleware

Deployment.

• Keystone is a WSGI application.

• Therefore Keystone can be deployed behind Apache (or otherHTTP server).

• The HTTP server verifies the X.509 proxy: validity, CA, CRLs.

VOMS module.

• WSGI middleware filter in the Keystone pipeline.

• Add-on to the Keystone server, no need for patch ormodification.

• The VOMS proxy is authenticated upstream (in the HTTPserver).

• The VO info is extracted from the VOMS proxy and ismapped to a Keystone user, tenant and domain.

May 22, 2014 48


VO support in OpenStack

User VOMS server HTTPD server Keystone WSGI VOMS WSGI Middleware Identity Backend Token Backend

voms-proxy-init

VOMS proxy

req token via SSL conn (using proxy)

verify proxy

request token

check voms

extract VOMS info

map VO/Group to tenant

create user in tenant

ifif user does not exist

ifif VO is authorized

user, tenant

request token

token

credentials

credentials

May 22, 2014 49


OpenStack Client module

• Pluggable authentication mechanism has been contributed tothe mainline from version 2.13.0.

• VOMS auth module available for novaclient.

$ git clone https://github.com/IFCA/voms-auth-system-openstack

$ pip install voms-auth-system-openstack

$ voms-proxy-init -voms VONAME -rfc

$ nova --os-auth-system voms --x509-user-proxy /tmp/proxy credentials

May 22, 2014 50


VOMS in Keystone v3

• V2 will be deprecated after Icehouse.

• Development is ongoing.

• Federation inside Keystone from Icehouse: mapping notanymore static, it can be done trough the Federation API.

May 22, 2014 51


Outline



Wrap up

May 22, 2014 52


Wrap up

• Keystone VOMS module:https://ifca.github.io/keystone-voms

– Documentation:https://Keystone-voms.readthedocs.org/en/latest/

• Client authentication plugin:https://github.com/IFCA/voms-auth-system-openstack

May 22, 2014 53


https://ifca.github.io/keystone-voms

https://Keystone-voms.readthedocs.org/en/latest/

https://github.com/IFCA/voms-auth-system-openstack

Part IV

Hands on

May 22, 2014 54


Outline

Preparation

Using OpenStackAuthentication

Booting a machine

VOMS installation

May 22, 2014 55


Preliminars

• We will NOT be using standard Keystone authentication.

• We will be using VOMS authentication, trough the nova CLI.

– There’s a problem with the temporary certificates, so we haveto use plain standard nova authentication.

• We will be using IFCA site.

• Prerequisites: git, curl, voms clients, virtualenv.

May 22, 2014 56


nova CLI installation

• Create a virtualenv (nothing will be installed in your machine).

$ mkdir /tmp/tutorial

$ virtualenv /tmp/tutorial/VENV

$ source /tpm/tutorial/VENV/bin/activate

• Install the nova clients.

(VENV) $ cd /tmp/tutorial

(VENV) $ git clone https://github.com/openstack/python-novaclient

(VENV) $ pip install python-novaclient

May 22, 2014 57


nova VOMS plugin installation

(VENV) $ git clone https://github.com/IFCA/voms-auth-system-openstack

(VENV) $ pip install voms-auth-system-openstack

(VENV) $ nova help

May 22, 2014 58


Outline

Preparation


Booting a machine

VOMS installation

May 22, 2014 59


Setting up authentication

• The nova endpoint is not used directly.

• Instead, the keystone endpoint is used.

• The client receives a catalog of the services and selects theendpoint.

May 22, 2014 60


Setting up authentication

(VENV) $ cat > novarc << EOF

#!/bin/bash

export OS_AUTH_URL=https://keystone.ifca.es:5000/v2.0

#export OS_TENANT_ID=

export OS_TENANT_NAME=VO:demo.fedcloud.egi.eu

export OS_USERNAME=<username>

export OS_PASSWORD=<password>

EOF

(VENV) $ source novarc

(VENV) $ nova credentials

(VENV) $ nova endpoints

(VENV) $ nova list

May 22, 2014 61


Outline

Preparation


Booting a machine

VOMS installation

May 22, 2014 62


Booting an instance

To boot an instance, the following is needed

• Image to use.

• Flavor for the instance (i.e. how many vcpus, memory, disk).

• Keypair to be injected (no root password should be allowed).

• Afterwards, we can request a public (floating) IP.

May 22, 2014 63


Keypair creation

In order to connect to a node, an SSH keypair is needed.(VENV) $ nova keypair-add <name> > privkey.pem

(VENV) $ nova keypair-list

May 22, 2014 64


Listing the images and flavors

(VENV) $ nova image-list

(VENV) $ nova flavor-list

May 22, 2014 65


Launching a machine

(VENV) $ nova boot --flavor m1.small --key-name <key> \

--image 07c98683-8ccd-4001-80fd-3a8b83596a26 \

<server-name>

(VENV) $ nova flavor-list

May 22, 2014 66


Public IP

• Machines are spawned with a private IP.

• A public IP may be requested and assigned to a runningmachine

(VENV) $ nova floating-ip-create

(VENV) $ nova add-floating-ip <server> <ip>

May 22, 2014 67


Outline

Preparation


Booting a machine

VOMS installation

May 22, 2014 68


Keystone installation

• We will install a keystone machine, and configure it toauthenticate using VOMS.

$ ssh -i <pubkey> ubuntu@<ip>

$ sudo aptitude update

$ sudo apt-get install python-software-properties

$ sudo add-apt-repository cloud-archive:havana

$ sudo aptitude update

$ sudo aptitude install keystone

$ sudo vi /etc/keystone/keystone.conf

$ sudo service keystone restart

May 22, 2014 69


Keystone installation

Creating users, tenants and roles:$ export OS_SERVICE_TOKEN=ADMIN_TOKEN

$ export OS_SERVICE_ENDPOINT=http://controller:35357/v2.0

$ keystone user-create --name=admin --pass=ADMIN_PASS --email=ADMIN_EMAIL

$ keystone role-create --name=admin

$ keystone tenant-create --name=admin --description="Admin Tenant"

$ keystone user-role-add --user=admin --tenant=admin --role=admin

$ keystone user-role-add --user=admin --role=_member_ --tenant=admin

$ keystone user-create --name=demo --pass=DEMO_PASS --email=DEMO_EMAIL

$ keystone tenant-create --name=fedcloud --description="Fedcloud tenant"

$ keystone user-role-add --user=demo --role=_member_ --tenant=fedcloud

May 22, 2014 70


Installing and configuring the VOMS service

• We will follow the documentation on:https://keystone-voms.readthedocs.org/en/latest/

May 22, 2014 71


https://keystone-voms.readthedocs.org/en/latest/

This is the end

[email protected]

May 22, 2014 72


[email protected]

Documents

EGI-InSPIREAustin 21 October 2010 Nova, Swift Bexar 3 February 2011 Nova, Glance, Swift Cactus 15 April 2011 Nova, Glance, Swift ... { message queue message hub. May 22, 2014 20 EGI-InSPIRE