OpenStack CE - user.xmission.com

© 2012 IBM Corporation

OpenStack CE Technology Review & Demo Egan Ford IBM Distinguished Engineer [email protected]

© 2012 IBM Corporation 2

PPT’s and Videos: http://xmission.com/~egan/cloud/!


Agenda

•  IBM SmartCloud and OpenStack •  Cloud Taxonomy •  Some OpenStack Public Use Cases •  What is OpenStack •  OpenStack Resources •  IBM Resources/Solutions for OpenStack Available Today •  OpenStack (Video) Demo


Evolve existing infrastructure

to Cloud

Accelerate adoption with

expert integrated systems

Immediate access to a managed

platform

Common Open Standards-based Cloud Management Services

Open architectures enable real innovation through interoperability


IBM embraces & invests in open source to foster innovation

Cloud Computing

Application Servers

Service Orientation

Service Oriented Architecture

Systems of Interaction

Social Business

Open Cloud Architecture

June 1998: IBM enters into an engineering agreement with The Apache Group for development of the open-source Apache HTTP server software eventually becoming the leader of the new Application Server market

September 1999: IBM capitalizes on an untapped market trend and begins participating in the community development of Linux with a $60M annual investment

November 2001: IBM rallies 150 influential vendors and the development community around a new tools environment with a $40 Million software donation disrupting the leadership of the software development ecosystem

September 2012: IBM orchestrates the launch of The OpenStack Foundation boasting $10 million in funding and 5,600 members changing the dynamics of the Cloud ecosystem


An open cloud architecture is emerging…built on open technologies

Hardware

OAuth

TOSCA

OSLC

cloudfoundry.org


OpenStack is a global collaboration of developers & cloud computing technologists working to produce an ubiquitous Infrastructure as a Service (IaaS) open source cloud computing platform for public & private clouds.

The OpenStack Foundation – IBM is a driving force in it’s success

Platinum Sponsors APR 2012

Sep 2013 150

Contributors 2600 Individuals

1021 Contributors 11,800+ Individuals

Exponential growth in 1+ YR Gold Sponsors

Software Defined

Environment


IBM is #2 in contributions Cloud Computing

Reference Architecture (CCRA)

Cloud Standards Customer Council

(CSCC)

LinkedData, OSLC

TOSCA

IBM’s ecosystem approach to an Open Cloud Architecture

OpenStack Open Source

Reference

IBM is working to accelerate OpenStack Foundation success

Because an open interoperable Cloud is critical for flexible cloud deployment and customer success…

IBM has 12 core contributors 12

IBMers working on OpenStack – from formation of the Foundation to Code Quality & New Function 370+

IBMers have signed the CLA

2

90

© 2011 IBM Corporation IBM CONFIDENTIAL

IBM contributions to OpenStack are wide-ranging, contributing to OpenStack success and delivering real value

OpenStack Compute Provision and manage large networks of virtual machines"•  Platform integration (CCS)"•  High Availability enhancements (RES)"•  Resource optimization (RES)"•  Live upgrade contributions (LTC)"•  Enablement for P & Z Systems, DB2 (CCS, SOS)"•  ESXi support (SOS w/ VMWare)"•  VM group enablement in scheduler (RES)"•  CPU allocation for vCPUs (RES)"

OpenStack Object Store Create petabytes of secure, reliable storage using standard HW"• Block & object storage enablement for IBM capability (RES)"• Nova blueprints (LTC)"• Cinder local storage & local instance clone (CCS)"• Efficient clone image in Cinder SVC driver for cFlex (RES)"• Nova & Cinder storage blueprints (CCS, SOS)"• Storwise/SVC driver update – support iSCSI CHAP auth (SOS, RES)"• Wsgi application interface enabling external web server (RES)"• Swift / Keystone interface for Keystone v3 API (RES)"

OpenStack NetworkingCreate petabytes of secure, reliable storage using standard HW"•  Support for key emerging networking standards (RES)"•  Quantum blueprints & migration from Nova (LTC)"•  FibreChannel support (RES)"

OpenStack Dashboard Enables administrators and users to access & provision cloud-based resources through a self-service portal."•  Globalization and crowd-sourced translation integration (SOS)"•  Cross hypervisor testing and validation (CCS)"

OpenStack Shared ServicesLibraries that provide image management, authentication & security across all OpenStack projects"•  Security & authentication enhancements (CCS, SOS)"•  Image activation for OVF (CCS)"•  Guest level metric collection (CCS)"•  APIs: Enablement for key emerging standards (SOS)"•  Membership services enhancements (CCS, RES)"•  Glance: multiple image locations (CCS)"

General OpenStack contributions"• Drive IBM value-add capability from SCP (CCS)"• Community facing contributions – bug fixing, community building & promotion (LTC, SOS)"• QA items (LTC)"

CCS – Common Cloud Stack (STG & SWG) LTC – Linux Technology Center (STG) SOS – Standards & Open Source (SWG) RES – Research

Part

icip

atio

n K

ey

© 2011 IBM Corporation IBM CONFIDENTIAL

IBM SmartCloud solutions are moving to an OpenStack-based infrastructure layer, enabling smooth migration and upgrade

Related Standards & Organizations

SmartCloud Orchestrator Orchestrate Services across multiple environments and domains

CIMI & OVF

TOSCA

CCRA OSLC

§  Simple 3 tier structure, with increased Client Value at each tier §  Using open, common, standards based architecture

providing choice, flexibility, interoperability, portability §  Clean upgrade paths with progression to fully integrated and

factory optimized PureApplication System §  Significant customer benefits above and beyond base

OpenStack

SmartCloud Provisioning Automate Optimized

Workloads

SmartCloud Entry Automate IT Delivery

SmartCloud Provisioning Automate Optimized

Workloads

SmartCloud Entry Automate IT Delivery

Customer integrated hardware

PureFlex System PureApplication

System

Automate Optimized Workloads

Key

Common Cloud Stack

Factory Integrated

Bundle Option

PureApp Server


Cloud Taxonomy

Source: http://it20.info/2012/02/the-cloud-magic-rectangle-tm/


Cloud Value Proposition and Positioning



How You (Provider) Build These Clouds



What You (Consumer) Get with These Clouds:



Policy-based Clouds and Design-for-fail Clouds are purpose optimized Infrastructure Management solutions

Policy-based Clouds

•  Purpose optimized for longer-lived virtual machines managed by Server Administrator

•  Centralizes enterprise server virtualization administration tasks

•  High degree of flexibility designed to accommodate virtualization all workloads

•  Significant focus on managing availability and QoS for long-lived workloads with level of isolation

•  Characteristics derived from exploiting enterprise class hardware

•  Legacy applications

Design-for-fail Clouds

•  Purpose optimized for shorter-term virtual machines managed via end-user or automated process

•  Decentralized control, embraces eventual consistency, focus on making “good enough” decisions

•  High degree of standardization •  Significant focus on ensuring availability

of control plane •  Characteristics driven by software •  New applications


Some OpenStack Public Use Cases

•  Internap •  http://www.internap.com/press-release/internap-announces-world%E2%80%99s-first-

commercially-available-openstack-cloud-compute-service/ •  Rackspace Cloud Servers, Powered by OpenStack

•  http://www.rackspace.com/blog/rackspace-cloud-servers-powered-by-openstack-beta/ •  Deutsche Telekom

•  http://www.telekom.com/media/media-kits/104982 •  AT&T

•  http://arstechnica.com/business/news/2012/01/att-joins-openstack-as-it-launches-cloud-for-developers.ars

•  MercadoLibre •  http://openstack.org/user-stories/mercadolibre-inc/mercadolibre-s-bid-for-cloud-

automation/ •  NeCTAR

•  http://nectar.org.au/ •  San Diego Supercomputing Center

•  http://openstack.org/user-stories/sdsc/


OpenStack design tenets focus on delivering essential infrastructure on an available, scalable, elastic control plane

Sources: http://www.openstack.org/downloads/openstack-compute-datasheet.pdf http://wiki.openstack.org/BasicDesignTenets

Basic Design Tenets

1) Scalability and elasticity are our main goals

2) Any feature that limits our main goals must be optional

3) Everything should be asynchronous. If you can't do something asynchronously, see #2

4) All required components must be horizontally scalable

5) Always use shared nothing architecture (SN) or sharding. If you can't Share nothing/shard, see #2

6) Distribute everything. Especially logic. Move logic to where state naturally exists.

7) Accept eventual consistency and use it where it is appropriate.

8) Test everything. We require tests with submitted code. (We will help you if you need it)

OpenStack Leadership's vision statement

“essential Infrastructure, support platform”


OpenStack

Source: http://ken.pepple.info/openstack/2012/09/25/openstack-folsom-architecture/


OpenStack is comprised of seven core projects that form a complete IaaS solution

Compute (Nova) Storage (Cinder) Network (Quantum) Provision and manage virtual resources Dashboard (Horizon) Self-service portal Image (Glance) Catalog and manage server images Identity (Keystone) Unified authentication, integrates with existing systems Object Storage (Swift) petabytes of secure, reliable object storage

IaaS

Source: http://ken.pepple.info/openstack/2012/09/25/openstack-folsom-architecture/

IaaS


Compute delivers a fully featured, redundant, and scalable cloud computing platform

Architecture

Sources: http://ken.pepple.info/openstack/2012/09/25/openstack-folsom-architecture/ http://openstack.org/projects/compute/

Key Capabilities:

•  Manage virtualized server resources •  CPU/Memory/Disk/Network Interfaces

•  API with rate limiting and authentication

•  Distributed and asynchronous architecture •  Massively scalable and highly available system

•  Live guest migration •  Move running guests between physical hosts

•  Live VM management (Instance) •  Run, reboot, suspend, resize, terminate instances

•  Security Groups

•  Role Based Access Control (RBAC) •  Ensure security by user, role and project

•  Projects & Quotas

•  VNC Proxy through web browser


Compute management stack control plane is built on queue and database

Key Capabilities: •  Responsible for providing communications hub and

managing data persistence

•  RabbitMQ is default queue, MySQL DB •  Documented HA methods •  ZeroMQ implementation available to decentralize

queue

•  Single “cell” (1 Queue, 1 Database) typically scales from 500 – 1000 physical machines

•  Cells can be rolled up to support larger deployments

•  Communications route through queue •  API requests are validated and placed on queue •  Workers listen to queues based on role or role +

hostname •  Responses are dispatched back through queue


nova-compute manages individual hypervisors and compute nodes

Key Capabilities: •  Responsible for managing all interactions with individual

endpoints providing compute resource, e.g. -- Attach iSCSI volume to phsyical host, map to guest as additional HDD

•  Implementations direct to native hypervisor APIs –  Avoids abstraction layers that bring least common

denomination support –  Enables easier exploitation of hypervisor

differentiators

•  Service instance runs on every physical compute node, helps to minimize failure domain

•  Support for security groups that define firewall rules

•  Support for –  KVM –  LXC –  VMware ESX/ESXi (4.1 update 1) –  Xen (XenServer 5.5, Xen Cloud Platform) –  Hyper V


nova-scheduler allocates virtual resources to physical hardware

Key Capabilities: •  Determines which physical hardware to allocate to a

virtual resource

•  Default scheduler uses a series of filters to reduce set of applicable hosts and uses costing functions to provide Weight

•  Not a focus point for OpenStack –  Default implementation finds first fit –  Shorter the workload lifespan, less critical the

placement decision

•  If default does not work, often deployers have specific requirements and develop custom


nova-api supports multiple API implementations and is the entry point into the cloud

Key Capabilities: •  APIs supported

–  OpenStack Compute API (REST-based) –  Similar to RackSpace APIs

–  EC2 API (subset) –  Can be excluded

–  Admin API (nova-manage) •  Robust extensions mechanism to add new capabilities


Network automates management of networks and attachments (network connectivity as a service)

Key Capabilities:

•  Responsible for managing networks, ports, and attachments on infrastructure for virtual resources

•  Create/delete tenant-specific L2 networks

•  L3 support (Floating IPs, DHCP, routing)

•  Moving to L4 and above in Grizzly

•  Attach / Detach host to network

•  Similar to dynamic VLAN support

•  Support for •  Open vSwitch •  OpenFlow (NEC & Floodlight controllers) •  Cisco Nexus •  Niciria

Architecture


Cinder manages block-based storage, enables persistent storage

Key Capabilities: •  Responsible for managing lifecycle of volumes and

exposing for attachment

•  Structure is a copy of Compute (Nova), sharing same characteristics and structure in API server, scheduler, etc.

•  Enables additional attached persistent block storage to virtual machines

•  Support for booting virtual machines from nova-volume backed storage

•  Allows multiple volumes to be attached per virtual machine

•  Supports following –  ISCSI –  RADOS block devices (e.g. Ceph distributed file

system) –  Sheepdog –  Zadara

Architecture


Identity service offers unified, project-wide identity, token, service catalog, and policy service designed to integrate with existing systems

Key Capabilities: •  Identity service provides auth credential validation and

data about Users, Tenants and Roles

•  Token service validates and manages tokens used to authenticate requests after initial credential verification

•  Catalog service provides an endpoint registry used for endpoint discovery.

•  Policy service provides a rule-based authorization engine and the associated rule management interface.

•  Each service configured to serve data from pluggable backend

–  Key-Value, SQL, PAM, LDAP, PAM, Templates

•  REST-based APIs


Image service provides basic discovery, registration, and delivery services for virtual disk images

Key Capabilities: •  Think Image Registry, not Image Repository

•  REST-based APIs

•  Query for information on public and private disk images

•  Register new disk images

•  Disk images can be stored in and delivered from a variety

of stores (e.g. SoNFS, Swift)

•  Supported formats –  Raw –  Machine (a.k.a. AMI) –  VHD (Hyper-V) –  VDI (VirtualBox) –  qcow2 (Qemu/KVM) –  VMDK (VMWare) –  OVF (VMWare, others) References

http://openstack.org/projects/image-service/


Dashboard enables administrators and users to access and provision cloud-based resources through a self-service portal

Key Capabilities: •  Thin wrapper over APIs, no local state

•  Registration pattern for applications to hook into

•  Ships with three central dashboards, a “User

Dashboard”, a “System Dashboard”, and a “Settings

•  Out-of-the-box support for all core OpenStack projects •  Nova, Glace, Switch, Quantum

•  Anyone can add a new component as a “first-class

citizen”. •  Follow design and style guide.

•  Visual and interaction paradigms are maintained

throughout.

•  Console Access References http://horizon.openstack.org/intro.html


OpenStack Resources

•  Forums •  http://forums.openstack.org/

•  Wiki •  http://wiki.openstack.org/

•  Documentation •  http://docs.openstack.org/

•  Mailing Lists •  http://wiki.openstack.org/MailingLists

•  OpenStack Project Management •  https://launchpad.net/openstack

•  Blogs •  http://planet.openstack.org

•  Real-time chat room •  #openstack and #openstack-dev on irc://freenode.net (443 users currently logged in)

•  Rackspace Reference Architectures •  http://www.referencearchitecture.org/

•  Easy Install •  http://www.hastexo.com/resources/docs/installing-openstack-grizzly-20131-ubuntu-1204-precise-

pangolin


IBM Resources/Solutions for OpenStack CE Available Today

•  developerWorks •  Google: openstack IBM developerworks

•  xCAT (FOSS) for 0-day deployment

•  xCAT OpenStack Paper (CATStack) •  Automated qcow2 image creation for Glance •  HW control •  Bare-metal discovery and bring up

• Firmware, Base OS, etc…

•  IBM Intelligent Cluster Solutions (see Matt Ziegler's PPT) •  Preconfigured Switches •  Rack and stacked and ready to go •  Lab Services for 0-day


IBM Resources/Solutions for OpenStack Available Today

•  All IBM System Software and Tools can coexist with OpenStack. •  Director, ASU, lflash, etc…

•  SoNAS for shared file (NFS, SMB)

•  XIV, v7000 for block storage (Cinder) •  iDPX for scale-out Nova Compute and Swift

•  BNT switches for OpenFlow and Quantum

•  GPFS for iSCSI/block (Cinder) or file.


OpenStack Demo Setup

10.0.9.10! 10.0.9.11! 10.0.9.12! 10.0.9.13! 10.0.9.X!

172.20.249.10! 172.20.249.11! 172.20.249.12! 172.20.249.13! 172.20.249.X!

os-essex0! os-essex1! os-essex2! os-essex3! os-essexX!

Control Nodes Compute Nodes

Private Networks: eth0: 172.20.249/24 vm: 172.20.250/24!

Public Networks: eth1: 10.0.9.0/25 vm: 10.0.9.128/25 !

compute!network!

compute!network!

compute!network!

compute!network!scheduler!volume!console!glance!api!

compute!network!scheduler!volume!console!glance!api!

Scale Out HA Active/Passive

VM

VM

VM

VM

VM Firewall


PPT’s and Videos: http://xmission.com/~egan/cloud/!

Documents

OpenStack CE - user.xmission.com