Running Operations for Carbon and Stratos

Chamith KumarageAssociate Technical Lead & Manager, InfrastructureWSO2 Inc.

Agenda● Automated deployment

● Deployment orchestration

● Maintenance

● Monitoring and alerting

● Backups and DR

● Scaling into the cloud

● High availability for services

Automated deployment● Self serving model

● Responsible from the ground up

● Templated configuration

● Enforce checklists and best practices

● Enables orchestration

● Powered by Puppet

“Let there be an AppServer!”

http://www.erie-flooring.com/automation.jpg

Not anymore..

Deployment orchestration● Based on Ruby Facts

● Agents publish facts to a broker service

● Orchestrator search for servers based on facts

● Plugins to enable remote management

● Integrates well with Puppet

● Powered by MCollective

Maintenance● Patch management and distribution

● Configuration management

● Centralized management

● Live service migrations

● Remote maintenance

“Reducing deployment complexity is better than managing complex deployments”

http://www.nimmot.net/base/worksamples/bad-timing-maintenance-page1.jpg

404s, less often

Monitoring and alerting● Behavioral monitoring

● JVM monitoring

● System resource monitoring

● Powerd by Ganglia, Nagios and Cacti

● Realtime and historical data analysis

● Email, SMS and phone-call alerts based on the severity

“If it's not monitored, it's not in production”

“In operations, you got to have metrics. When in doubt, make graphs.”

Backups and DR● Routine backups and restoration drills

● Replicated artifact repositories

● Multi-site replication

● Automated recovery tools

● Documentation

“One cannot take past successes as a guarantee against future failures”

Scaling into the cloud● Powered by WSO2 Elastic Loadbalancer (ELB)

● Scale up early, scale down slowly

● Based on Jclouds API

● Now: Amazon EC2 and OpenStack Nova support

● Soon: VCloud and Rackspace support

High availibility for services● Service aware loadbalancing

● Multiple service clusters in different sites

● Set min_app_instances to suite your need

● Linux HA

“He who fails to plan is planning to fail”

Q&A

Thank You!

chamith@wso2.com @gnuchami