Intel and OpenStack: Contributions and

Deployment

Das Kamhout, Principal Engineer, Intel ITDr. Malini Bhandaru, Open Source Technology Center, Intel SSG

OpenStack Summit, Hong Kong, Nov’13

Helping Fuel Innovation—and Opportunities

#2 Linux Contributor

improving performance, stability & efficiency

Across the Stack

contributions span every layer of the stack

Red Hat

11.1%

Intel SUSE IBM

9.3%

4.9%4.2%

Proven Components

building blocks simplify development, reduce costs and speed time-to-market

QTLinux Kernel

KVMCairo

OfonoConnman

Clutter

0% 20% 40% 60% 80% 100%

Code Contributions to Open Source Projects

Intel is single largest contributor to these projects

Intel in Open Source

Project Contributor

X.org

GNU

Webkit

JQuery

Eclipse

OpenStack

Yocto Project

Hadoop

3,000

2,500

2,000

1,500

1,000

500

0

KVM

Th

rou

gh

pu

t

MC-DP WSM-EP SNB-EP WSM-EX

SPCEvirt_sc2010* Performance

3x improvement o

ver 3 years

Hig

her

i s B

ett

er

01.org

kernel.org

2

Intel Enables OpenStack Cloud Deployments

Contributions

Intel® IT Open Cloud

Intel® Cloud Builders

• Across OpenStack projects • Open Source Tools• Top contributor to Grizzly and Havana releases1

• Optimizations, validation, and patches

• Intel IT Open Cloud with OpenStack• Delivering Consumable Services• Single Control Plane for all Infrastructure

• Collection of best practices• Intel IT Open Cloud Reference Arch • Share best practices with IT and CSPs• http://www.intel.com/cloudbuilders

1Source: www.stackalytics.com 3

Stress on Datacenter Operations

1: Source: Intel IT internal estimate; 2: 3: IDC’s Digital Universe Study, sponsored by EMC, December 2012; 4: IDC Server Virtualization and The Cloud 2012

Network2-3 weeks to provision new services1

Storage40% data growth CAGR, 90% unstructured3

ServerAverage utilization <50% despite virtualization4

New Challenges are coming….

4

The Intel SDI Vision

1: Source: Intel IT internal estimate

Datacenter Today Software-defined Infrastructure

Time to Provision New Service: Minutes1Time to Provision New Service: Months1

Idea for service

IT scopes needs

Balanceuser demands

Idea forservice

Servicerunning

Manually configuredevices

Set up service components,

assemble software

Servicerunning

Software components assembled

Private

Public

Self service catalog & services

orchestration

Automated composition of resources

5

Self-provisioning, automated orchestration, composable resource pools

Open Data Center Alliance Cloud Adoption Roadmap

Year 1 Year 2 Year 3 Year 4 Year 5

End User

App Dev

App Owner

IT Ops

Federated, Interoperable,

and Open Cloud

Simple SaaS

Enterprise Legacy Apps

Compute, Storage, and

Network

Simple Compute IaaS

Simple SaaS

Enterprise Legacy Apps

Cloud Aware Apps

Complex Compute IaaS

Simple Compute IaaS

Compute, Storage, and

Network

Complex SaaS Hybrid SaaS

Full Private IaaS

Hybrid IaaS

Cloud Aware Apps

Legacy Apps

Private PaaS Hybrid PaaS

Cloud Aware Apps

Legacy Apps

Consumers

Lega

cy A

pplic

ation

s on

ded

icat

ed

Infr

astr

uctu

reSt

art

6

Intel IT Quick History

Design Grid since 1990’s60k servers across 60+

datacentersCloud’s Uncle

Enterprise Private Cloud 201013k VMs across 10 datacenters

75% of Enterprise Server Requests

80% virtualized

Open Source Private Cloud 2012

1.5k VMs across 2 datacentersRunning cloud-aware and

some traditional apps

OpenStack

Silicon Design

Validation Labs

Enterprise Hosting

Existing Infrastructure New Infrastructure

OpenStack - Intel IT Convergence Platform

Top Challenges & Technical Responses

Security & Compliance

Unit Cost Reduction

Business Uptime

• Trusted Compute Pools• Geo-tagging• Key Management• Enhanced Platform Awareness (crypto processing)

• Intelligent storage allocation in Cinder• Multiple publisher support in ceilometer• Erasure code in Icehouse release• COSbench performance measurement tool• Erasure Code (storage cost)• Enhanced Platform Awareness (PCIe Accelerators etc.)• Intelligent workload & storage scheduling

• Live Migration, Rack-level redundancies• Intel® Virtualization Technology with FlexMigration 9

Intel Contributions* to OpenStack

*Note: A mixture of features that are completed, in development or in Planning

Compute Networking Storage• Enhanced Platform Awareness

• CPU Feature Detection• PCIe SR-IOV Accelerators• OVF Meta-Data Import

• Trusted Compute Pools• With Geo Tagging

•Key Management• Intelligent Workload

Scheduling (Metrics)

• Intel® DPDK vSwitch• VPN-as-a-Service with

Intel® QuickAssist Acceleration

• Advanced Services in VMs

• Filter Scheduler• Erasure Code• Object Storage

Policies

User Interface (Horizon)

Object Store (Swift)

Image Store (Glance)

Compute (Nova) Block Storage (Cinder)

Network Services (Neutron)

Key Service (Barbican)

Trusted Compute Pools(Extended with Geo Tagging)

OVF Meta-Data Import

Intel® DPDK vSwitch

Enhanced Platform AwarenessErasure

Code

Expose Enhancements

Filter Scheduler

Monitoring/Metering (Ceilometer)

Object Storage Policy

Key Encryption & Management

Advanced Services in VMsIntelligent Workload Scheduling

Metrics

10

VPN-as-a-Service (with Intel® QuickAssist Technology)

Trusted Compute Pools (TCP)

Enhance visibility, control and compliance

TCP Solution - Platform Trust - new attribute for Management- Intel® TXT initiates Measured Boot

- basis for Platform Trust- Open Attestation (OAT) SDK – Remote Attestation

Mechanism https://github.com/OpenAttestation/OpenAttestation

- TCP-aware scheduler controls placement & migration of workloads in trusted pools

1source: McCann “what’s holding the cloud back?” cloud security global IT survey, sponsored by Intel, May 2012No computer system can provide absolute security under all conditions.  Intel® Trusted Execution Technology (Intel® TXT) requires a computer system with Intel® Virtualization Technology, an Intel TXT-enabled processor, chipset, BIOS, Authenticated Code Modules and an Intel TXT-compatible measured launched environment (MLE). The MLE could consist of a virtual machine monitor, an OS or an application.  In addition, Intel TXT requires the system to contain a TPM v1.2, as defined by the Trusted Computing Group and specific software for some uses. For more information, see here

TCP is enabled in OpenStack (Folsom release)

11

12

Trusted Compute Pools with Geo-Tagging

• OpenStack* Enhancements • Secure mechanism for Provisioning geo certificates• Dashboard – display VM/storage geo• Nova flavor extra spec – geo• Enhanced TCP scheduler filter • Geo Attestation Service (OAT +)• Geo-tagged Storage

• Volumes• Objects

Work in progress - Provide feedback, use cases

Use geo-location descriptor stored in TPM on Trusted Servers to control workload placement & migration

Cloud ServiceProvider Portal

Trust AttestationOAT/MTW

Key MgtService

Keys

CSP-Image Server

(Glance)

Host + VMM

OAT

MH: OVF Plug-in

DOM0

TXT + TPM

1

2

34

6

5

7

8

9

Customer Data Center

MH Client

Cloud Service ProviderData Center

Encrypted VM Image

Launch request(from anywhere)

Encryption Key (enveloped)

Policy

Encrypted VM Image

Launch command

Request Encryption Key (AIK, KeyID)

Request Host Trust Attestation

Encrypted VM SymKey

Response Trust Status, BindPubKey

MH ClientMH Client

Concept: Trusted Compute Pools (TCP) – VM ProtectionTenant-Controlled, Hardware-Assisted VM Protection in the Cloud

Concept Demo in Citrix Booth

Key ManagementEase Security Adoption, new use cases, compliance• Server-side encryption• Data-at-rest security

• Random high quality keys • Secure Key Storage • Controlled key access via Keystone • High availability • Pluggable backend – HSM, TPM• Barbican Key Manager:

https://github.com/cloudkeep/barbican

Intel technologies: Intel® Secure Key, Intel® AES-NI

Prototype in Havana, incubate in Icehouse14

Filter Scheduler (Cinder)Volume Service 1

Volume Service 2

Volume Service 3

Volume Service 4

Volume Service 5

Volume Service 1

Volume Service 2

Volume Service 3

Volume Service 4

Volume Service 5

Weight = 25

Weight = 20

Weight = 41

Volume Service 2

Volume Service 4

Volume Service 5

Filters Weighers

Winner!

• AvailabilityZoneFilter

• CapabilitiesFilter

• JsonFilter• CapacityFilter• RetryFilter

• CapacityWeigher• AllocatedVolumesWeigher• AllocatedSpaceWeigher

Example Use Case: Differentiated Service with Different Storage Back-ends

• CSP: 3 different storage systems, offers 4 levels of volume services

• Volume service criteria dictates which storage system can be used

• Filter scheduler allows CSP to name storage services and allocate correct volume

15 15

16

Data Collection for Efficiency:Intelligent Workload SchedulingEnhanced usage statistics allow advanced scheduling decisions

• Pluggable metric data collecting framework

• Compute (Nova) - New filters/ weighers for utilization-basedscheduling

Metering in Havana release, scheduling in future release

17

Enhanced Platform Awareness

Allows OpenStack* to have a greater awareness of the capabilities of the hardware platforms

• Expose CPU & platform features to OpenStack Nova scheduler

• Use ComputeCapabilities filter to select hosts with required features

Intel® AES-NI or PCI Express acceleratorsfor security and I/O workloads

Upto 10x encryption & 8x decryption performanceimprovement observed 1

Intel® AES-NI = Intel® Advanced Encryption Standard New InstructionsSee http://www.oracle.com/us/corporate/press/173758

Some features in Havana, more in future releases

ProcessorUnencrypted

Data

ABCDEFGHIJKLMNOPQRSTUVW

Faster Encryptions

Faster Decryptions

Data In MotionEncrypted

Data

#@$%&%@#&%@#$@&%$@#$@%&&

SDN & NFV: Driving Architectural Transformation

To This:Networking within VMsStandard x86 COTS HW

Open SDN standard solutions

From This:Traditional networking topology

Monolithic vertical integrated boxTEM proprietary solutions

VM: Firewall

VM:VPN

VM: IDS/IPS

SDN/NFV

Firewall VPN IDS/IPS

IA CPU ChipsetAcceleration

SwitchSilicon

NICSilicon

Wind RiverLinux + Apps

TEM/OEMProprietary OS

ASIC, DSP, FPGA, ASSP

18

19

Intel® DPDK Accelerated Open vSwitch In Neutron

Open vSwitch ML2 Driver/Agent in Development

Neutron API APIExtensions

Neutron-ML2-PluginDB

External Controller

vSwitch

VMVMVMVM

L2 Agent

DPDK vSwitch

VMVMVMVM

DPDK vSwitchL2 Agent

DPDK vSwitch Mechanism Driver

Intel DPDK vSwitch

10x

Unleashing Intel® DPDK vSwitch Performance in Neutron

20

Capacity Tier (Storage)

Access Tier (Concurrency)

OpenStack* Swift With Erasure Code

Load Balancer

Proxy

StorageStorageStorage

StorageStorageStorage

StorageStorageStorage

StorageStorageStorage

StorageStorageStorage

Proxy Proxy

Zone 1 Zone 2 Zone 3 Zone 4 Zone 5

Clients

RESTful API, Similar to S3

Download

Frag 1

Frag 2

Frag 3

Frag 4

Frag N

Decoder

Upload

Encoder

Obj A Obj A

• New Storage Policy capability• Applications control policy• EC can be inline or offline

• Supports multiple policies at the same time via container tag

• EC flexibility via plug-in

AuthService

Detailed Tutorial at: https://intel.activeevents.com/sf13/connect/sessionDetail.ww?SESSION_ID=1180&tclass=popupCommunity Collaboration: https://intel.activeevents.com/sf13/connect/sessionDetail.ww?SESSION_ID=1180&tclass=popup

Lowers TCO with increased storage efficiency

Intel actively contributing to OpenStackDelivering interoperable, federated, efficient and secure Open Cloud solutions

Security & Compliance

Unit Cost Reduction

Business Uptime

• Trusted Compute Pools• Geo-tagging• Key Management• Enhanced Platform Awareness (crypto processing)

• Intelligent storage allocation in Cinder• Multiple publisher support in ceilometer• Erasure code in Icehouse release• COSbench performance measurement tool• Erasure Code (storage cost)• Enhanced Platform Awareness (PCIe Accelerators etc.)• Intelligent workload & storage scheduling

• Live Migration, Rack-level redundancies• Intel® Virtualization Technology with FlexMigration 21

Q&A

23

Legal Disclaimers:

 INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS.  NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT.  EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT.

A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death.  SHOULD YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL INDEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE ATTORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCONTRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS.

Intel may make changes to specifications and product descriptions at any time, without notice.  Designers must not rely on the absence or characteristics of any features or instructions marked "reserved" or "undefined".  Intel reserves these for future definition and shall have no responsibility whatsoever for conflicts or incompatibilities arising from future changes to them.  The information here is subject to change without notice.  Do not finalize a design with this information.

The products described in this document may contain design defects or errors known as errata which may cause the product to deviate from published specifications.  Current characterized errata are available on request.

Intel product plans in this presentation do not constitute Intel plan of record product roadmaps. Please contact your Intel representative to obtain Intel's current plan of record product roadmaps.

Intel processor numbers are not a measure of performance. Processor numbers differentiate features within each processor family, not across different processor families. Go to: http://www.intel.com/products/processor_number.

Contact your local Intel sales office or your distributor to obtain the latest specifications and before placing your product order.

Copies of documents which have an order number and are referenced in this document, or other Intel literature, may be obtained by calling 1-800-548-4725, or go to:  http://www.intel.com/design/literature.htm

Code names featured are used internally within Intel to identify products that are in development and not yet publicly announced for release.  Customers, licensees and other third parties are not authorized by Intel to use code names in advertising, promotion or marketing of any product or services and any such use of Intel's internal code names is at the sole risk of the user

Intel,  and the Intel logo are trademarks of Intel Corporation in the United States and other countries. 

*Other names and brands may be claimed as the property of others.

Copyright ©2013 Intel Corporation.

Legal Disclaimers and Notices

Intel Trademark Notice: Celeron, Intel, Intel logo, Intel Core, Intel® Core™ i7, Intel® Core™ i5, Intel® Core™ i3, Intel® Atom™ Intel Inside, Intel Inside logo, Intel. Leap ahead., Intel. Leap ahead. logo, Intel NetBurst, Intel SpeedStep, Intel XScale, Itanium, Pentium, Pentium Inside, VTune, Xeon, and Xeon Inside are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.Non-Intel Trademark Notice: *Other names and brands may be claimed as the property of others.General Performance Disclaimer/"Your Mileage May Vary"/Benchmark: Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors.  Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions.  Any change to any of those factors may cause the results to vary.  You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products.Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel® products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance. Buyers should consult other sources of information to evaluate the performance of systems or components they are considering purchasing. For more information on performance tests and on the performance of Intel products, visit http://www.intel.com/performance/resources/limits.htm or call (U.S.) 1-800-628-8686 or 1-916-356-3104.Estimated Results Benchmark Disclaimer: Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance.Pre-release Notice: This document contains information on products in the design phase of development.Processor Numbering Notice: Intel processor numbers are not a measure of performance.  Processor numbers differentiate features within each processor family, not across different processor families:  Go to:  http://www.intel.com/products/processor_number Roadmap Notice: All products, computer systems, dates and figures specified are preliminary based on current expectations, and are subject to change without notice.Excerpted Product Roadmap Notice: Intel product plans in this presentation do not constitute Intel plan of record product roadmaps.  Please contact your Intel representative to obtain Intel's current plan of record product roadmaps.Intel® AES-New Instructions (Intel® AES-NI): Intel® AES-NI requires a computer system with an AES-NI enabled processor, as well as non-Intel software to execute the instructions in the correct sequence.  AES-NI is available on select Intel® processors.  For availability, consult your reseller or system manufacturer.  For more information, see http://software.intel.com/en-us/articles/intel-advanced-encryption-standard-instructions-aes-ni/ Enhanced Intel SpeedStep® Technology : See the Processor Spec Finder at http://ark.intel.com or contact your Intel representative for more information.Intel® Hyper-Threading Technology (Intel® HT Technology): Available on select Intel® Core™ processors.  Requires an Intel® HT Technology-enabled system.  Consult your PC manufacturer.  Performance will vary depending on the specific hardware and software used.  For more information including details on which processors support HT Technology, visit http://www.intel.com/info/hyperthreading.Intel® 64 architecture: Requires a system with a 64-bit enabled processor, chipset, BIOS and software.  Performance will vary depending on the specific hardware and software you use.  Consult your PC manufacturer for more information.  For more information, visit http://www.intel.com/info/em64t Intel® Turbo Boost Technology: Requires a system with Intel® Turbo Boost Technology.  Intel Turbo Boost Technology and Intel Turbo Boost Technology 2.0 are only available on select Intel® processors.  Consult your PC manufacturer.  Performance varies depending on hardware, software, and system configuration.  For more information, visit http://www.intel.com/go/turbo

24

6Months

6Months

Infr

astr

uctu

reAs

a S

ervi

ce

Compute Storage Network 12-18 MonthsPh

ysic

alIn

fras

truc

ture

IaaS

Compute(Nova*)

Block Storage (Cinder*)

Object Storage(Swift*)

Network(Neutron*)

Dashboard (Horizon*)

OS Images(Glance*)

Open-Source (OpenStack*)

Manageability

3Months

Mon

itorin

gAs

a S

ervi

ce

Watcher(Nagios*, Shinken*, Heat*)

Decider(Heat)

Collector(Hadoop*)

Actor(Puppet*, Cfengine*)

Open-Source Foundation

Inte

rfac

es GUI(Graphical User Interface)

API(Application Programming Interface)

ReleaseCadence

App

Platf

orm

Se

rvic

es PaaS

Analytics Messaging Data Web

3Months

Intel IT Open Cloud Components

25

26

Benefits of Enhanced Platform Awareness

Enabler for Enhanced Cloud Efficiency & Deploying SDN/NFV WorkloadsSome features enabled in Havana, more coming in future releases

Intel® QuickAssist Accelerator Intel® Data Plane Development Kit

Intel® AES New Instructions Intel® Advanced Vector Extensions 2 (AVX2)

Intel® Secure Key

Source: http://lwn.net

0

2

4

6

8

10

12

14 IntelRed HatSUSEIBMTILinaro (ARM)

Kernel Releases

Con

trib

uti

on

by P

erc

en

tag

e

Linux Kernel Contributions

Summary: Key Intel Contributions into OpenStackContribution Project Release CommentsTrusted Filter Nova Folsom Place VMs in Trusted Compute PoolsTrusted Filter UI Horizon Folsom GUI interface for Trusted Compute Pool managementFilter Scheduler Cinder Grizzly Intelligent storage allocationMultiple Publisher Support

Ceilometer Havana Pipeline manager; pipelines of collectors, transformers, publishers

Open Attestation SDK To Open Source Remote Attestation service for Trusted Compute PoolsCOSBench To Open Source Object store benchmarking toolEnhanced Platform Awareness

Havana + future Leverages advanced CPU and PCIe device features for increased performance

Key Manager Icehouse+ Makes data protection more readily available via server side encryption with key management

Erasure Code Icehouse Augments tri-replication algorithm in Swift enabling application selection of alternate storage policies

28

Re-architect the Datacenter

1: Source: Intel IT internal estimate

Datacenter Today Software-defined Infrastructure

Time to Provision New Service: Minutes1Time to Provision New Service: Months1

Idea for service

IT scopes needs

Balanceuser demands

Idea forservice

Servicerunning

Manually configuredevices

Set up service components,

assemble software

Servicerunning

Software components assembled

Private

Public

Self service catalog & services

orchestration

Automated composition of resources

29

The Intel SDI Vision

Automated provisioning

Orchestrated placement

Composable Resource Pools

30