Building FOSS Clouds

Sebastien Goasguen, Kris BuytaertLinuxCon EU, Nov 6th

Barcelona, Spain

Slides at: slideshare.net/sebastiengoasguen

Speakers…• Joe stayed home to deal with the release,

• Kris Buytaert stepped in:

Tutorial Outline

• 10:10 – 10:45 FOSS Clouds and CloudStack intro

• 10:45 – 11:00 DevCloud demo

• 11:00 – 11:50 FOSS Cloud Automation, DevOps configuration and monitoring tools

Define:tutorial

• A tutorial is a method of transferring knowledge and may be used as a part of a learning process. More interactive and specific than a book or a lecture…<snip>

• Therefore:

DevCloud

• A CloudStack sandbox– Pass the USB keys around and get the Vbox appliance

devcloud.ova file

• Import the appliance in VirtualBox– Play with CloudStack– http://wiki.cloudstack.org/display/COMM/DevCloud– https://cwiki.apache.org/confluence/display/

CLOUDSTACK/CloudStack+devcloud+environment+setup

Objectives

• Learn the basic architecture of a cloud• Learn a few new tools• Explore CloudStack via its sandbox• Discover some concepts of automation• Get home with some ideas to try out• Mileage may vary

NIST Definition of Cloud

•"cloud computing is a model for enabling ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction.“ – NIST

NIST Service/Deployment Models•Service Models:• IaaS (e.g EC2, rackspace)•PaaS (e.g GAE, Azure)• SaaS (e.g Gapps)

•Deployment model:•Private cloud•Public cloud•Hybrid cloud (Private than can use Public)

NIST Key Characteristics of Cloud•Elasticity• Expands and Shrinks as needed• E.g Elastic Load Balancing, Elastic Map Reduce

•Measured Service•Utility Pricing. You pay for what you use• E.g $0.08 for one hour of small instance

•On-demand• You get it when you ask for it• E.g One api call and you get your machine, ~no

queue

“Guidelines”

• Virtualization and automation are key enablers of the cloud characteristics

• Build your infrastructure to enable on-demand elastic provisioning of –servers, storage, network –

• Setup a PaaS that suits you and your applications

• Then move to continuous delivery of applications …

OSS SaaS…and more

OSS PaaS…

OSS IaaS…

IaaS challenges

• Not out of the box by itself.• Need a farm of hypervisors – Xen, KVM, Vmware…

• Need storage– For image catalogue– For volume/snapshot management

• Need flexible network that can be configured on-demand– VLANS, no VLANS, existing net infra, SDN…

Let’s make Red Hat happy

Announcing Apache CloudStack 4.0

Coming at 12:00 EST

• Apache CloudStack 4.0• Check out the testing procedure:– https://cwiki.apache.org/CLOUDSTACK/

cloudstack-40-test-procedure.html

• Or watch my amazing screencast:– http://vimeo.com/52150218

A bit of History• Original company VMOPs (2008)– Founded by Sheng Liang former lead dev on JVM

• Open source (GPLv3) as CloudStack• Acquired by Citrix (July 2011)• Relicensed under ASL v2 April 3, 2012• Accepted as Apache Incubating Project April

16, 2012• First Apache (ACS 4.0) release expected now

Apache Software Foundation

Apache Process

• 100% community driven• New ideas, decisions only taken on mailing lists.

Votes taken by community• Project led by Project Management Committee

(PMC):– http://www.apache.org/dev/pmc.html– http://incubator.apache.org/guides/committer.html

• Non committers get invited as committers:– http://community.apache.org/newcommitter.html

Apache Processes

ContributionsExamples:

Sungard: Announced that 6 developers were joining the Apache projectSchuberg Philis: Big contribution in building/packaging and Nicira supportGo Daddy: Early proto of Maven buildingCaringo: Support for own object storeBasho: Devcloud packaging

A Very Flexible IaaS PlatformCompute Hypervisor

Storage Block & Object

Network Network & Network Services

Primary Storage Secondary Storage

http://www.slideshare.net/cloudstack/cloudstack-architecture

Architecture / Language

• Java application• Tomcat6, Axis2, Maven build + ant– Ant going away in 4.1

• Collaboration Conference, Nov 30th -Dec 2nd – http://collab12.cloudstack.org/

Build and Run in 4.1

• git clone https://git-wip-us.apache.org/repos/asf/incubator-cloudstack.git

• mvn clean• mvn install• mvn –P developer –pl developer –Ddeploydb

• mvn –pl :cloud-client-ui jetty:run

Cloud Interactions

CloudStackCloudStack

Cloud user{API client (Fog/etc)}

End User UI

End User UI

Admin UI

Admin UI

MySQLMySQL

CloudStackCloudStackClustered

CloudStackManagement

Server

ClusteredCloudStack

ManagementServer

Domain Admin

UI

Domain Admin

UI

CS Admin & End-user API

Cloud user{ec2 API client }

ec2 API

Monitoring CS API vSphere ClusterPrimaryStorage

vcentervcenter

Cluster Mgmt

XS ClusterPrimaryStorage

vCenter API

XAPI

KVM ClusterPrimaryStorageJSON

OVM Cluster PrimaryStorage

XenApi

NetConf

Nitro APIJuniper SRX

Netscaler

Console Proxy VMConsole

Proxy VMConsole Proxy VMConsole

Proxy VM

JSON

Cloud user

HTTPSAjax Console

Ajax Console

VNC

Sec. StorageVM

Sec. StorageVM

NFS Server

NFSSec. Storage

VMSec. Storage

VM

HTTP (Template Download)

HTTP (Template Copy)

HTTP (Swift)

NFS

Router VMRouter VMRouter VMRouter VM

Router VMRouter VM

JSON

{Proxied} SSH

http://www.slideshare.net/cloudstack/cloudstack-architecture

TerminologyZone: Availability zone, aka Regions. Could be worldwide. Different data centersPods: Racks or aisles in a data centerClusters: Group of machines with a common type of HypervisorHost: A Single serverPrimary Storage: Shared storage across a clusterSecondary Storage: Shared storage in a single Zone

Storage• Primary Storage:

– Anything that can be mounted on the node of a cluster. – Cluster LVM…iSCSI…– Holds disk images of running VMs– Support for CEPH with KVM hypervisors

• Secondary Storage:– Available across the zone– Holds snapshots and templates (image repo)– Can use Openstack swift or any object store (Gluster FS…)– New support for Caringo

• Can use NFS for both to start• Storage Abstraction refactoring underway

Networking

• Extremely flexible to:– Provide isolation with VLANs– Provide isolation at L3 with shared L2 (scalability)– Support hardware devices that exposes API– Deployed on existing networking infrastructure– Support new networking paradigm (SDN)• Support for Nicira Virtual P• Extensive use of Open VSwitch

A customizable GUIAJAX + API

A very extensive API

API• Not really REST• A set of methods available over http(s)• Unauthenticated on integration port• Authenticated on 8080 using Access and Secret Key• Python/Ruby clients available• Internal Marvin client• Cloudmonkey CLI• Other clouds client support the API

Enabling EC2 and S3

• Via the GUI

• Via API call on integration API port 8096http://localhost:8096/client/api?command=updateConfiguration&name=enable.s3.api&value=true

http://localhost:8096/client/api?command=updateConfiguration&name=enable.ec2.api&value=true

DevCloud

• Several use cases:– Try CloudStack in an isolated sandbox. Runs within

the appliance– Develop CloudStack on own machine, build locally

and deploy new version in DevCloud (Build and test)

– Develop and Run locally, use DevCloud as Xen hosts

DevCloud: self-containedCloudStack runs in the appliance

Testing “4.0” code in DevCloud

• Deploy new CloudStack code in self-contained DevCloud:

• mvn –P deps• ant rdeploy • ant rdeploydb – Wipes database of mgt server, you will need to

reconfigure the “data center”

• ant rdebug

DevCloud: as HostRun CloudStack on local machineUse DevCloud to setup hosts

DevCloud to test 4.1 branch

• mvn -P developer clean• mvn -P developer install• mvn -P developer -pl developer –Ddeploydb• mvn -P developer -pl tools/devcloud –Ddeploydb

• mvn -pl :cloud-client-ui jetty:run

Configure infrastructure:• mvn -P developer,deploysvr -pl tools/devcloud -Ddeploysvr

Testing Framework –for the PyUG • Marvin is a Python

based framework to run tests against a CloudStack install

• Could be used as a simulator of a datacenter

• Used to configure an infrastructure on a mgt server

[environment]

dns=10.147.28.6

mshost=10.147.39.69

mysql.host=10.147.39.69

[cloudstack]

private.gateway=10.147.40.1

private.pod.startip=10.147.41.121

private.pod.endip=10.147.41.160

private.netmask=255.255.254.0

public.gateway=10.147.40.1

public.vlan.startip=10.147.41.162

public.vlan.endip=10.147.41.200

public.netmask=255.255.254.0

hypervisor=XenServer

host=10.147.40.10

host.password=password

#storage pools

primary.pool=nfs://10.147.28.7:/export/home/automation/sadhu/primary

secondary.pool=nfs://10.147.28.6:/export/home/automation/sadhu/secondary

Demo time

• http://vimeo.com/52150218

Info• Apache incubator project• http://www.cloudstack.org• http://incubator.apache.org/cloudstack/• #cloudstack on irc.freenode.net• @cloudstack on Twitter• http://www.slideshare.net/cloudstack• http://cloudstack.org/discuss/mailing-lists.html

Welcoming contributions and feedback, Join the fun !