Are We Done Yet ? Testing Your OpenStack Deployment

Accelerating the adoption of Cloud Computing

Are We Done Yet ? Testing Your OpenStack Deployment

OpenStack Summit, Paris, France November 4, 2014

© Solinea, Inc. 2

Who Am I ?

Ken Pepple is the Chief Technology Officer of Solinea Prior to founding Solinea, he led the introduction of Internap's OpenStack-based public cloud services while serving as their Director of Cloud Development Author of O'Reilly "Deploying OpenStack” book OpenStack code contributor since Bexar release

© Solinea, Inc.

!

Who we are

3

Solinea is the leading software and services company that delivers integrated OpenStack™ solutions, enabling enterprise adoption of Cloud Computing

Cloud is the only domain we focus on, with vertical industry and horizontal solutions specialization

Purpose-built for cloud

Track record of success architecting, building and operating production clouds – private and public – world-wide

Proven Delivery Success

We understand cloud adoption challenges of global enterprise companies

Enterprise IT Experience

Pre-configured management and monitoring platform for OpenStack™ and other Open Infrastructure components

Differentiating IP

Built the first OpenStack production clouds and contributors to the platform since its inception

OpenStack™ Experience

© Solinea, Inc.

Agenda !  Problem statement !  Types of testing !  Testing tools !  Future improvements

4

© Solinea, Inc.

Problem !  Many components, even more connections !  No two clouds are identical

–  Hypervisor choices, network choices, storage choices

!  Many rarely used code paths (backups ?) !  Operational and integration concerns !  Tools for operators !  Requires skills across all the services

5

© Solinea, Inc.

In a nutshell … distributed systems

OpenStack Object Store

OpenStack Image Service OpenStack Compute

OpenStackDashboard

OpenStack Identity Service

OpenStack Compute API /

Admin API

keystone(service & admin APIs)

nova-api(OS, EC2, Metadata, Admin)

nova-consoleauth

nova-cert/objectstore

nova-consolenova-*proxy

VNC/Spice/RDP

OpenStack Object API

http://www.solinea.comOpenStack Data Processing

sahara-api

sahara-engine saharadatabase

Queue sahara-conductor

OpenStack Data Processing API

(8386)

OpenStack Identity

API

OpenStack Data Processing API

Queue

nova-compute

nova-scheduler

novadatabase

OpenStackCompute API

(8774, 8775, 8776)

OpenStack Image API

Horizon

OpenStack Image API

(9292)

identity backend

swift-proxy

objectcontaineraccount

objectstore

accountDB

containerDB


HTTP(S)

OpenStackObject API

OpenStack Identity API

OpenStack Identity

API

OpenStack Identity

API

OpenStackImage API

OpenStack Identity

API

OpenStack Image API

catalog backendtoken backend

OpenStack Identity

API(5000)

hypervisor

libvirt, XenAPI, etc.

HTTP(S)

Amazon Web Services

EC2 API

Internet / Enterprise Network

OpenStack Network Service

glance-api

glancedatabase

OpenStack Block Storage

OpenStack Block Storage API

cinder-api

cinder-volume

neutron-server

neutron plugin(s)

OpenStack Identity

API

cinder-scheduler

cinderdatabase

OpenStack Network API

(9696)

networkprovider

OpenStack Block

StorageAPI


policy backend

Queue


neutrondatabase

neutron agent(s)

nova-conductor

memcached

OpenStack Identity

API

⁃ OpenStack Command Line Tools (nova-client, swift-client, etc.)⁃ Cloud Management Tools (CloudForms, Enstratius, etc.)⁃ GUI tools (Web Browsers, Cyberduck, iPhone client, etc.)

volume provider

cinder-backup


OpenStack Identity

API

Queue

OpenStack Block Storage API

OpenStack Orchestration

heat-api

heat-engine

heatdatabase

Queue

cloudwatch-api

OpenStack Orchestration API

OpenStack Identity

API

OpenStack Compute API

OpenStack Bock Storage API


glance-registry

OpenStack Metering Service

ceilometer-api

ceilometerdatabase

ceilometer-collector

OpenStack Database

trove-api

trove-taskmgr

trovedatabase

trove-conductor

OpenStack Database API

(8779)

OpenStack Block Storage APIOpenStack Orchestration API

OpenStack Compute API

Database(mysql, redis, mongo,

cassandra, etc.)

Queue

trove-guestagent

OpenStack Identity

API

6

© Solinea, Inc.

What Should We Be Testing? !  Functionality

–  User scenarios –  Not code quality … unless running trunk

!  Non-functional requirements –  Scalability –  Performance –  Security

!  Operations –  High availability and failover

7

© Solinea, Inc.

How Should We Test ? !  Automated

–  Repeatability –  Multiple checks

!  Continual !  Ops not specialists

–  Should need coders or QA people –  Testing should be an easy command

!  Integrated into operational procedures –  “Let me send a canary in …”

8

© Solinea, Inc.

Types of Testing !  Unit testing !  Integration !  User Acceptance Tests !  Operational

9

© Solinea, Inc.

Functional Areas !  Networking !  Storage !  Compute !  User functional process !  OUTSIDE applications …

–  Billing/metering, user signup, audit/compliance …

10

© Solinea, Inc.

OpenStack Testing Options !  Tempest !  Rally !  Manual

11

© Solinea, Inc.

Tempest !  OpenStack testing project !  Runs hundreds of tests

–  Unit –  Scenario –  Stress –  API –  CLI

!  Complex to configure !  Not portable to every cloud deployment configuration !  Requires significant investment in time

12

© Solinea, Inc.

An OpenStack project to create a framework for validating, performance testing and benchmarking OpenStack at scale with Tempest https://wiki.openstack.org/wiki/Rally

13

Rally

© Solinea, Inc.

Rally Scenario $ rally task start summit/create-and-list-users.json================================================================================Task 57bc6d22-76c2-4450-b834-daa48c41d6e0 is started--------------------------------------------------------------------------------

================================================================================Task 57bc6d22-76c2-4450-b834-daa48c41d6e0 is finished.--------------------------------------------------------------------------------

test scenario KeystoneBasic.create_and_list_usersargs position 0args values:{u'args': {u'name_length': 10}, u'runner': {u'concurrency': 10, u'times': 100, u'type': u'constant'}}+----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+| action | min (sec) | avg (sec) | max (sec) | 90 percentile | 95 percentile | success | count |+----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+| keystone.create_user | 1.767 | 3.478 | 5.944 | 4.565 | 4.694 | 100.0% | 100 || keystone.list_users | 0.077 | 0.464 | 1.516 | 1.01 | 1.143 | 100.0% | 100 || total | 1.982 | 3.942 | 6.218 | 4.956 | 5.214 | 100.0% | 100 |+----------------------+-----------+-----------+-----------+---------------+---------------+---------+-------+Whole scenario time without context preparation: 40.6584420204

14

© Solinea, Inc.

Rally Database !  Rally stores all scenario/task runs !  Can be used to trend and benchmark results $ rally task list+--------------------------------------+----------------------------+----------+--------+-----+| uuid | created_at | status | failed | tag |+--------------------------------------+----------------------------+----------+--------+-----+| 57bc6d22-76c2-4450-b834-daa48c41d6e0 | 2014-11-05 12:10:38.235211 | finished | False | || 12d34393-0520-4830-9516-e7404935c0e9 | 2014-11-05 12:32:39.632810 | finished | False | || 3b160652-ed3b-4e2b-b09c-14b6d900e18e | 2014-11-05 12:40:21.544483 | running | False | |+--------------------------------------+----------------------------+----------+--------+-----+

15

© Solinea, Inc.

Rally Scenarios { "KeystoneBasic.create_delete_user": [ { "args": { "name_length": 10 }, "runner": { "type": "constant", "times": 100, "concurrency": 10 } } ]}

16

Method

Customization

Concurrency

© Solinea, Inc.

Rally Scenarios !  Authenticate !  Ceilometer !  Cinder !  Designate !  Glance !  Heat !  Keystone

!  Neutron !  Nova !  quotas !  Requests !  Sahara !  Tempest !  VM !  Zaqar

17

© Solinea, Inc.

Rally Visualization

18

RALLY DEMO

19

© Solinea, Inc.

Operational HA Tests !  Kill process / hung process !  Lost interface !  Lost machine !  Lost rack

20

© Solinea, Inc.

Continual Testing !  Testing should be part of monitoring and

management !  “Order Canary”

–  Chao monkey may not be appropriate but “happy path” test should be

–  Define rally scenarios or custom scripts to feed monitoring tools

!  Subset of tests that affect business critical functions

21

© Solinea, Inc. 22

Putting it all together

OpenStack Cloud Rally [os-admin]

OS API commands Monitoring [goldstone]

© Solinea, Inc.

Best Practices !  ABT (Always be testing) !  Manual testing " operational procedure

–  Failover regularly as part of normal operations –  HA is an event, it is SOP

!  Testing is part of the deployment, monitoring and upgrade process

23

© Solinea, Inc.

Goldstone Log Monitoring

© Solinea, Inc.

Future !  /info APIs for all services

–  Describe deployment beyond services

!  Simplified DSL to configure tempest !  Monitoring product incorporate testing

25

Ken Pepple, CTO [email protected]

http://www.solinea.com

Thank You