Upload
hathu
View
237
Download
0
Embed Size (px)
Citation preview
Challenges and lessons learned with Openstack deployments and MySQL
Sandro Mazzio+a Sr Director Product Management IaaS
Who I am
o I am not a DB expert or a DBA o I work in Product Management since 15+ years o I work at eNovance since 2 months o I started with Openstack in July 2011 o I’ve been launching HP Cloud Services DBaaS offering
San Francisco
Montréal
Paris
Bangalore
eNovance loca>ons and figures
People
Customers
Growth in Revenue
Founded in
Value Engineering
Softw
are
Software Architecture : from centralized apps to distributed services Software Engineering : from Waterfall to Agile
Infra
stru
ctur
e
IT Infrastructure : from Mainframe to Cloud Cloud Infrastructure Engineering : from Ops to DevOps
Journey to the cloud
Client / Server Vertical Scaling
Hardware is key Centralized
Services / API Horizontal Scaling
Software is key Distributed
Expensive Hw Vertical Scaling
Monolithic Centralized
Cheap Hw Horizontal Scaling
Distributed
Market & opportunity Large
Enterprises Software for Private Cloud: $ 10 bn by 2017
Telcos xSP Software for Public Cloud : $ 8 bn by 2017
Telecom Equipment Vendors NFV : $ 2bn by 2017
Not Scalable Expensive Not Cloud
Rich Set of Features No Versioning
Shared Infrastructure
Not Scalable Expensive Not Cloud
Containers Not Scalable Not Cloud
Migrate from Physical Appliance To Network Functions Virtualization
eNovance Cloud Solu/ons
Seamless Build & Delivery of OpenStack Cloud Infrastructures
Build a Cloud Deliver your Private or Public Cloud on Premise with a real Cloud Approach :
• No Versioning • Pay as you Grow • AWS Features Match • Enterprise Grade SLA • 24/7 Support
Use a Cloud
Deliver Cloud-Ready applications on top of existing Cloud infrastructures : • Elastic & Scalable • Continuous Delivery • Pay as you Grow • Enterprise Grade SLA • 24/7 Support
What we do ?
Openstack is not cloud (Subbu Allamaraju eBay, July 25, 2013)
All key requirements for producRon: • By Enterprise for Private Cloud • By Service Providers for Public Cloud
“infra onboarding, boostrapping, remediaRon, config management, patching, packaging, upgrades, high availability, monitoring, metrics, user support, capacity forecasRng and management, billing or chargeback, reclamaRon, security, firewalls, DNS, integraRon with other internal infrastructure and tools, and on and on and on”.
Openstack Upstream does not address HA
Openstack HA Guide
h+p://docs.openstack.org/high-‐availability-‐guide/content/
index.html
ApplicaRon on top Openstack OS Infra. Plane:
Openstack API Message Queues DB
VirtualizaRon Layer …
eNovance Openstack Reference Architecture V1.0
Ø Management cluster full highly available Ø OpenStack API, schedulers Ø RabbitMQ, Database, Cache Ø Network services
Ø Compute nodes with KVM and Open-‐vSwitch with GRE tunnels as overlay
Ø Swia cluster Ø Ceph cluster, used as backend for Nova,
Glance and Cinder
eNovance Openstack product
• Contains all core projects of Openstack
• Supports HA in a Single DC mode • Based on • eDeploy (eNovance Bare Metal Deployment Opensource)
• Puppet as Soaware ConfiguraRon Management
What about DB in our product?
DB Requirements: FuncRonal & Performance requirements: • 100 % upRme • Performance DegradaRon is ok
Business Requirements:
Opensource
Galera cluster for MySQL
eNovance Galera for MySQL usage: results
Experience from ProducRon Environment: 1 cluster of 3 Management Nodes
• No Service InterrupRon • Performance DegradaRon are OK • Galera performed correctly
On the topic of Mul> Data Center
West 1
East
EMEA
ASIA
Goal is to address:
• Disaster Recovery • Local Affinity
Not all services are ready for Mul>-‐DC
• Openstack has a concept of AZ, Host AggregaRon = good in a single DC . Not good If your DC are separated by a WAN…
• Nova Cells is good for distribuRon (but nobody else is taken care)… • MulRple Region – Keystone in every region or one overlay
Galera 3 Galera SynchronizaRon over
WAN
Using Heat to orchestrate MulRple Region
Keystone
Our experience working at large customers is that:
• Token storage is a nightmare and • and do not scale well on large scale
deployment
• Key Value Store (memcache, redis, …) -‐> MySQL -‐> memcache (but we have a bug)… euh…
Ceilometer, we use mongoDB
But Requires MongoDB DBA, Runbook, skills,
…
Because we need scalibility and because of the nature of informaRon to store
Openstack is not cloud (Subbu Allamaraju eBay, July 25, 2013)
All key requirements for producRon: • By Enterprise for Private Cloud • By Service Providers for Public Cloud
“infra onboarding, boostrapping, remediaRon, config management, patching, packaging, upgrades, high availability, monitoring, metrics, user support, capacity forecasRng and management, billing or chargeback, reclamaRon, security, firewalls, DNS, integraRon with other internal infrastructure and tools, and on and on and on”.
Why we need update – upgrade –patching… ?
User
CRM
Billing
Portal
seongs changes
new release
bug fixes driver firmware update
IntegraRon mess
What is our vision
Deliver Openstack and app on top of it using Openstack process and tools ConRnuous IntegraRon/Delivery
Leverage the Openstack process
How Openstack handle • Contribu/on of 400 developers / month? • Developers from 250 companies? • 10+ integrated projects?
and yet release a new version • every 6 months • with 400 new features • without major inconsistencies?
… and tools
Everything in
• Process descrip/on and code • Infrastructure used to handle the process • Code to deploy the infrastructure
100 Rmes be+er than documentaRon of a manual process that is never up to date...
Reproducible environments
• Dynamic Provision test env when needed by Jenkins
• Allow dev to generate a test env for debug at will
• Store validated releases and customer specific env (in Openstack Swi^)
Benefits
Improve Efficiency
• Nothing is local, everything is shared • Control access rights • Contribute model • No need for a full release team • Factorize hardware needs
Reproducible Env
• Dynamically provision test environments when needed by Jenkins
• Allow developers to generate a test environment for debugging at will
• Store validated releases and customer specific environments (in Swi^)
What we are describing is Devops…
• Validate code changes and mulRple deployments environments at once • Accommodate more devs working on the same projects • Limit wait Rmes ⇒ Industrializes development and...
Conclusion
• Openstack is not easy. • It is ok at small scale in a single DC mode. • eNovance • has a product to address Single DC HA mode deployment • is applying Openstack CD/CI to Openstack deployment itself • is working on the MulR DC mode