Upload
botchagalupe
View
2.536
Download
0
Embed Size (px)
DESCRIPTION
Citation preview
Operations is a Strategic Weapon
1Friday, October 7, 2011
DTO Solutions
2Friday, October 7, 2011
We do process improvement and automated infrastructure for companies who build and operate revenue producing services... e-commerce, financial services, gaming
Are you an ...
aaS3Friday, October 7, 2011
-Are you an ass?
-What kind of ass are you? Golly gee.. really.. who gives a shit...
-Ironically all this “cloud” discussions focus around the wrong letter.
----------------------------------------
-What does amazon and netflix get that barnes and noble and blockbuster doesn’t ?
_aaS (circle the "S")
-Over the next 45 minutes I am going to try and convince you that companies that fcous on the behind Devops are the best
solution to address the "S" in your aaS
Are you an ...
[spi]aaS3Friday, October 7, 2011
-Are you an ass?
-What kind of ass are you? Golly gee.. really.. who gives a shit...
-Ironically all this “cloud” discussions focus around the wrong letter.
----------------------------------------
-What does amazon and netflix get that barnes and noble and blockbuster doesn’t ?
_aaS (circle the "S")
-Over the next 45 minutes I am going to try and convince you that companies that fcous on the behind Devops are the best
solution to address the "S" in your aaS
Are you an ...
[spi]aaS3Friday, October 7, 2011
-Are you an ass?
-What kind of ass are you? Golly gee.. really.. who gives a shit...
-Ironically all this “cloud” discussions focus around the wrong letter.
----------------------------------------
-What does amazon and netflix get that barnes and noble and blockbuster doesn’t ?
_aaS (circle the "S")
-Over the next 45 minutes I am going to try and convince you that companies that fcous on the behind Devops are the best
solution to address the "S" in your aaS
Are you an ...
[spi]aaS3Friday, October 7, 2011
-Are you an ass?
-What kind of ass are you? Golly gee.. really.. who gives a shit...
-Ironically all this “cloud” discussions focus around the wrong letter.
----------------------------------------
-What does amazon and netflix get that barnes and noble and blockbuster doesn’t ?
_aaS (circle the "S")
-Over the next 45 minutes I am going to try and convince you that companies that fcous on the behind Devops are the best
solution to address the "S" in your aaS
Are you an ...
[spi]aaS3Friday, October 7, 2011
-Are you an ass?
-What kind of ass are you? Golly gee.. really.. who gives a shit...
-Ironically all this “cloud” discussions focus around the wrong letter.
----------------------------------------
-What does amazon and netflix get that barnes and noble and blockbuster doesn’t ?
_aaS (circle the "S")
-Over the next 45 minutes I am going to try and convince you that companies that fcous on the behind Devops are the best
solution to address the "S" in your aaS
Are you an ...
[spi]aaS3Friday, October 7, 2011
-Are you an ass?
-What kind of ass are you? Golly gee.. really.. who gives a shit...
-Ironically all this “cloud” discussions focus around the wrong letter.
----------------------------------------
-What does amazon and netflix get that barnes and noble and blockbuster doesn’t ?
_aaS (circle the "S")
-Over the next 45 minutes I am going to try and convince you that companies that fcous on the behind Devops are the best
solution to address the "S" in your aaS
Are you an ...
[spi]aaS3Friday, October 7, 2011
-Are you an ass?
-What kind of ass are you? Golly gee.. really.. who gives a shit...
-Ironically all this “cloud” discussions focus around the wrong letter.
----------------------------------------
-What does amazon and netflix get that barnes and noble and blockbuster doesn’t ?
_aaS (circle the "S")
-Over the next 45 minutes I am going to try and convince you that companies that fcous on the behind Devops are the best
solution to address the "S" in your aaS
Are you an ...
[spi]aaS3Friday, October 7, 2011
-Are you an ass?
-What kind of ass are you? Golly gee.. really.. who gives a shit...
-Ironically all this “cloud” discussions focus around the wrong letter.
----------------------------------------
-What does amazon and netflix get that barnes and noble and blockbuster doesn’t ?
_aaS (circle the "S")
-Over the next 45 minutes I am going to try and convince you that companies that fcous on the behind Devops are the best
solution to address the "S" in your aaS
4Friday, October 7, 2011
-How many ppl are familiar with the metaphor used in softw dev called “TD”?-In softw dev you typally have two choices get it done quick and take hit of future issues or spend to time on a cleaner design, but will take longer to put in place.-Quick typically p with a technical debt, which is similar to a financial debt.interest payments, which come in the form of the extra effort that we have to do in future development because of the quick and dirty design choice.--Most of you are probably familiar with “Software” TD-I am going to tell you a story about “Infrastructure TD”
5Friday, October 7, 2011
-How many ppl know who these guys are?-The dudes who invented Facebook.. rght?-However, this was not there first venture. -Green vs Red Widgits compnay story. -After many arguments the both decide to split up. Cameron to make green widgets and Tyler decides to make red widgets. -They both go to their father each ask for a 1m dollars and the father asks how much money are you going to make. They both say 10 million. -Father calculates the ROR to be 900% and gives em each 1m. 10-1 = 9/1 = 900%-However one of them lied.... The red widgets only return 233%10-1-2 = 7/3 = 233%
Green vs Red Widgits
1 Million2 Million10 Million Profit10 - 3 = 7/3 233% ROR
1 MillionNo TD10 Million Profit10 - 1 = 9/1 900% ROR
6Friday, October 7, 2011
Technical Debt
Vicious Cycle
ToxicOperations Terminal
It Gets Worse...
7Friday, October 7, 2011
- It gets even worse. Isreal Ghat of the Cutter Group call the vicious cycle of TD.
- You wind up fixing a lot of things that you didn’t fix in the first place..
- this pulls more resources from delivery good service and compound effect is that you are spending more and more
resources that you should have gotten right the firs time..
- but ever worse the effect of customer satisfaction starts loosing more business and the “V” cycle is out of control.
- TD->VTD->Toxic operations->Terminal
- I call this are you running a business or building a business?
- Toxic operations. Amertrade/etrade story
Tale of Two Startups
8Friday, October 7, 2011
- Jesse Robbins my ex boss and CEO of Opscode/Chef did a great post on O’rielly rdar a few years ago called the Tale of
Two Startups.
- The chart looked like this first 4 weeks.
- First chart legacy (I call it the non devops startup/project)
- Second Chart is the (secret sauce startup ... #devops)
- I played around with this using “R” to be cool and I came up with a=140% ROR and 700% ROR
The Meat to Math Ratio
http://radar.oreilly.com/2011/08/meat-to-math-ratio.html
9Friday, October 7, 2011
- Alistar Croll has a great post called the meat to math ratio.
- Amazon had $12.95B in Q410 revenues and 33,700 employees, revenue per employee of $384,273.
- For Barnes & Noble: Barnes & Noble: $1.91B in Q410 revenues, and 35,000 employees, meaning a revenue per employee
of $54,571.
- Netflix: $444M in Q409 revenues, and 1,000 employees, meaning a revenue-per-employee of $444,000.
- Blockbuster: $400M in Q409 revenues. The company peaked at 60,000 employees i
- Dropbox: In Q211 Dropbox had $25M in revenues, and 74 employees, for a revenue per employee of $338K.
When do they call?
10Friday, October 7, 2011
We started asking ourselves some questions about our clients and our work for them
When do they call?
A. First signs of issues brewing
B. Initial negative impact felt by some
C. Heads are on fire
10Friday, October 7, 2011
We started asking ourselves some questions about our clients and our work for them
When do they call?
A. First signs of issues brewing
B. Initial negative impact felt by some
C. Heads are on fire65%
30%
10%
10Friday, October 7, 2011
We started asking ourselves some questions about our clients and our work for them
Why did they wait?
#1 Answer:
11Friday, October 7, 2011
Aside from human nature to procrastinate?translation... “you and or the problem you are telling me about isn’t worth the money”
Why did they wait?
#1 Answer:
“Couldn’t get budget approval or business support”
11Friday, October 7, 2011
Aside from human nature to procrastinate?translation... “you and or the problem you are telling me about isn’t worth the money”
Operations has a perception problem
12Friday, October 7, 2011
Operations has a perception problem
Necessary Cost
Business View
12Friday, October 7, 2011
Operations has a perception problem
Necessary Cost
Business View
vs
Strategic Weapon
Ops View
12Friday, October 7, 2011
First Law of Business Spending
Things that cost you money
Things that make you money
13Friday, October 7, 2011
Make no mistake about it... someone is spending money on you... and they are asking themselves... do I cut it, outsource it, or spend more on it.
Change the Perception of Ops
Necessary Cost
vs
Strategic Weapon
14Friday, October 7, 2011
Operations becomes a strategic weapon
15Friday, October 7, 2011
If we are going to make the business case that operations is a strategic weapon... and we need to do it in language that they understand.Business care about innovation (ideas that resonnate with customers) and getting the best return on their money... Let’s look at the first point... continuously increasing velocity of innovation
Operations becomes a strategic weapon
When you are continuously...
15Friday, October 7, 2011
If we are going to make the business case that operations is a strategic weapon... and we need to do it in language that they understand.Business care about innovation (ideas that resonnate with customers) and getting the best return on their money... Let’s look at the first point... continuously increasing velocity of innovation
Operations becomes a strategic weapon
1. Increasing velocity of innovation
When you are continuously...
15Friday, October 7, 2011
If we are going to make the business case that operations is a strategic weapon... and we need to do it in language that they understand.Business care about innovation (ideas that resonnate with customers) and getting the best return on their money... Let’s look at the first point... continuously increasing velocity of innovation
Operations becomes a strategic weapon
1. Increasing velocity of innovation
2. Increasing return on investment
When you are continuously...
15Friday, October 7, 2011
If we are going to make the business case that operations is a strategic weapon... and we need to do it in language that they understand.Business care about innovation (ideas that resonnate with customers) and getting the best return on their money... Let’s look at the first point... continuously increasing velocity of innovation
Ah-ha!
Result
16Friday, October 7, 2011
Core process of any business
Ah-ha!Ka-ching!
17Friday, October 7, 2011
Now of course you hope that result is one that resonates with customers
Ah-ha!
18Friday, October 7, 2011
But for many reasons it more often than not goes the other way
Day 0 Day n
Velocity of Innovation...
Ah-ha!Ka-ching!
19Friday, October 7, 2011
That cycle time... getting from idea to result is one of the most critical metrics for both a startup and an established service. And the bulk of this time is spent in the application lifecycle across dev, QA, and operations.
Companies were able to achieve somewhat defensible positions based on technology...
20Friday, October 7, 2011
Companies were able to achieve somewhat defensible positions based on technology... then came the web
21Friday, October 7, 2011
Then along came this thing called the web and screwed it all up... now your customers are coming to you through a standard interface --the browser-- over standard published protocols. You competition is only a few keystrokes away. The even applies in newer distribution channels like mobile apps and their app stores.
How do we compete now?
22Friday, October 7, 2011
So how do we compete in this commoditized world? 1. Scale... scale of users, scale of data, scale of compute power2. Velocity of innovation... how fast can you react to and execute on new market forces or opportunities.
Scaling is the most straight forward problem to solve. It’s a known problem with a lot of known solutions to borrow from. Hire smart architects... let them follow best practices... throw a big pile of cash at it and there aren’t many scaling problems you can’t solve. Obviously that’s somewhat of a simplification... but I’ll let someone else’s presentation get into the details
1. Scale
How do we compete now?
22Friday, October 7, 2011
So how do we compete in this commoditized world? 1. Scale... scale of users, scale of data, scale of compute power2. Velocity of innovation... how fast can you react to and execute on new market forces or opportunities.
Scaling is the most straight forward problem to solve. It’s a known problem with a lot of known solutions to borrow from. Hire smart architects... let them follow best practices... throw a big pile of cash at it and there aren’t many scaling problems you can’t solve. Obviously that’s somewhat of a simplification... but I’ll let someone else’s presentation get into the details
1. Scale
2. Velocity of Innovation
How do we compete now?
22Friday, October 7, 2011
So how do we compete in this commoditized world? 1. Scale... scale of users, scale of data, scale of compute power2. Velocity of innovation... how fast can you react to and execute on new market forces or opportunities.
Scaling is the most straight forward problem to solve. It’s a known problem with a lot of known solutions to borrow from. Hire smart architects... let them follow best practices... throw a big pile of cash at it and there aren’t many scaling problems you can’t solve. Obviously that’s somewhat of a simplification... but I’ll let someone else’s presentation get into the details
1. Scale
2. Velocity of Innovation
How do we compete now?
22Friday, October 7, 2011
So how do we compete in this commoditized world? 1. Scale... scale of users, scale of data, scale of compute power2. Velocity of innovation... how fast can you react to and execute on new market forces or opportunities.
Scaling is the most straight forward problem to solve. It’s a known problem with a lot of known solutions to borrow from. Hire smart architects... let them follow best practices... throw a big pile of cash at it and there aren’t many scaling problems you can’t solve. Obviously that’s somewhat of a simplification... but I’ll let someone else’s presentation get into the details
Ah-ha!Ka-ching!
Ah-ha!
Innovation is really a numbers game...
23Friday, October 7, 2011
The global innovation success rate across all geographies and industries
4%*
94%*
*Study by Doblin Innovation Consultants
Ah-ha!Ka-ching!
Ah-ha!
Innovation is really a numbers game...
23Friday, October 7, 2011
The global innovation success rate across all geographies and industries
4%*
94%*
*Study by Doblin Innovation Consultants
Ah-ha!Ka-ching!
Ah-ha!
Innovation is really a numbers game...
23Friday, October 7, 2011
The global innovation success rate across all geographies and industries
Ah-ha!
Ah-ha!
Result
Result
Ah-ha!
Result
Ah-ha!
Result
Ah-ha!
Result
How to win a numbers game...
Company A
Company B
24Friday, October 7, 2011
Put it this way... if in the time it takes you to get through one cycle, your competitor can get through 4... who do you think will be more competitive??
Ah-ha!
Ah-ha!
Result
Result
Ah-ha!
Result
Ah-ha!
Result
Ah-ha!
Result
How to win a numbers game...
Company A
Company B
25Friday, October 7, 2011
This is why the notion of a “burn rate” is a meaningless statistic. Say you have enough cash in the bank to pay for x number of months. That doesn’t tell you anything... do you have 10 shots at making customers happy or just 1? Think about it like a carnival game. If you have ten balls to throw at the target and someone else only has 1... who has a better probability of success?
How fast can ops move?
• Production deployment every 11.6 seconds (weekday)
• 1,079 deployments in one hour (record)
• ~0.001% of deployments actually cause an outage
http://news.ycombinator.com/item?id=2971521
26Friday, October 7, 2011
Jon Jenkins at VelocityOps is amazon’s strategic weapon... their books aren’t better, their storage isn’t better, their hypervisor isn’t better.Everyone copies them but their just keep rolling out features and lowering costs faster than their competitors can do either
Business is already thinking about this...
Customer Development Model
27Friday, October 7, 2011
But can the biz move fast enough? They already are wanting to do soCustomer Development
Operations becomes a strategic weapon
28Friday, October 7, 2011
Operations becomes a strategic weapon
When you are continuously...
28Friday, October 7, 2011
Operations becomes a strategic weapon
1. Increasing velocity of innovation
When you are continuously...
28Friday, October 7, 2011
Operations becomes a strategic weapon
1. Increasing velocity of innovation
2. Improving return on investment
When you are continuously...
28Friday, October 7, 2011
29Friday, October 7, 2011
•I got into a Twttier argument that went like this... • See “Cloud Gone Wrong”
apache/php
memcachedmysql
Rightscale
AWSs3
Businessbusiness
ideas !
s3putssh
Dev
Clouds Gone Wild
30Friday, October 7, 2011
-So letʼs start the story in the begining... Biz guy got a great idea for a service.
-They used a classic web2.0 app architecture: apache/php, memcached & mysql. Development done on a single server-Production ran on EC2 and they used rightscale server templates -Release done by pushing code and assets to s3 buckets and then running a parallel SSH scripts to distribute them-This approach seemed to work... They got up to a few hundred nodes pretty fast.. business was cooking
apache/phpmemcached
mysql
Rightscale
AWS s3
Rightscale
AWSs3
Rightscale
AWS s3
yum
puppet
s3put
ssh
3rd party,
&
middleware
apache/phpmemcached
mysql
apache/phpmemcached
mysql
Business
Business3
s3put
ssh
Business2
31Friday, October 7, 2011
-First service was such a huge success they decided to launch other sevices-So they ”Copy and pasted” the whole architecture and lifecycle to launch the new businesses-Each new group pushed assets to s3, scripted the distribution, and hacked the rightscripts and templates-Things were obviously getting more complicated, so they did what they were supposed to do and added centralized tooling like puppet and yum-They thought they were doing things the cloud way and that all would be fine
“As-is”
app devs
middlewaredevs
operations
systemeng
EC2
servertemplates
yum repo
RS deploymentRS
deployment
puppet
S3
code,content
code
TARTAR
TARTAR
platform
TARRPM
TARRPM
servertemplates
EXE
AUTO
TOOLS
Provision-time builds
CONTROL
restarts
deploys
reconfigs
PROVISION RELEASE
32Friday, October 7, 2011
First we got the team on the whiteboard to map out the “as is” picture. This is a, believe it or not, a simplified version of that.Some of the highlights...-First youʼll notice that different groups had their own path to production... different methods of control, provisioning, and release. -Each group and role seemed to have a different way to editing or storing config-There were differing ways of packaging software... sometimes it might be a .tar.gz other times it might be an RPM.-Shockingly... Some things were even being built directly on production servers.-There was no authoritative source of information is maintained about nodes, application topology, software versions, etc.... -Everything was being stored in S3 buckets... which is great because itʼs so easy to use... but itʼs unversioned and people would just upload whatever was newest for them. But that stuff was never really tested in unison.. so old stuff wasnʼt working with new stuff... and its was unclear what was different or why it was different.-We can go on... but you should get the point that they had the right cloud, the right tools, and lots of smart people, but it all got dangerously out of control very quickly
Other:
-Changes hit all customers at once => Puppet configs in unversioned S3 buckets-Buggy node classification causing provisioning problems = > complex/long node classifier script-”Dead boxes” after provisioning => rightscript/puppet ordering problems-”my box got clobbered!” => puppet, is it supposed to be on or off?-new environment setup was taking longer and longer => from days to weeks because of “fooled by false horizons”-”is the system ready yet?” => nobody knows what “ready” means-Scripts contained a list of role-to-node lists to put things in the right places- scripts crap out on nodes taken out of commission-”software works differently” => rightscale driven compile/installs
Technical
33Friday, October 7, 2011
pup
pet
SVN
active
directory
splunk
instance create
resource model
users packages
new node Node a
commands
right
scale
run
deck
hud
son
yum
repo
nag
ios
packages
sys cfg
eventslog data
agent cfg
CONTROL PROVISION RELEASE
Everything starts here
Infrastructure Deveopment Life Cycle
34Friday, October 7, 2011
I wonʼt go into too much detail about the tooling that was put into place to support all of this but here are some highlights..-took a loosely coupled toolchain approach... using mostly open source tools -This became their standard stack of “operations middleware”. Of course, we are all used to the notion of application middleware... but to an online service, the management infrastructure is just as much a part of the service as the application itself.-This operations middleware stack is a first class citizen along with application stack and it all goes through the same SDLC... everything is versioned, built, deployed, and packaged via the same process -Once in place, this middleware provides a single path for releasing, provisioning, and controlling anything that goes into an environment.
Other:Management infrastructure based on “swap-able” sets of integrated tools- Organized into three rough categories: Control, Provisioning, Monitoring - Control tools support routine and ad hoc procedures executed as commands/scripts - Provisioning tools support package delivery and post install customization - Monitoring tools actively check health and collect log data
When you think of middleware, you think of where your app code works.. but in the service world, you have an operations middleware that is just as important. All the provisioning and management stuff... itʼs just as important. Itʼs one and the same.
Solve the information problem... where is the system of record? easy in cloud to get basic node data... that comes from the compute service... but what about everything else you need to manage your infrastructure?
Key integrations:- SVN drives everything in the tool chain!- Rundeck synchronizes to RS; must be connected to compute service to know what nodes are provisioned and ready- All packages come through yum
Infrastructure SDLC
95% 5%
Necessary Liability Strategic Weapon
35Friday, October 7, 2011
subscribe in
iTunes
DevOps Cafe
36Friday, October 7, 2011
I also do a podcast with the famous cloud and IT management guru john willis.Interview based series where we talk to all kinds of movers and shakers across the development and operation spectrum.
Between DTO and doing the devops cafe content I get to talk to a lot of companies and see what’s working and what isn’t working.
@botchagalupe
Let’s Talk....
37Friday, October 7, 2011
We do this stuff all day long for a lot of large and cutting edge clients... and we love talking about DevOps so drop me a line anytime if you want to talk
@botchagalupe dev2ops.org
Let’s Talk....
37Friday, October 7, 2011
We do this stuff all day long for a lot of large and cutting edge clients... and we love talking about DevOps so drop me a line anytime if you want to talk
Extra Slides
38Friday, October 7, 2011
CAMS
39Friday, October 7, 2011
First get you mind around what you are looking for... Simple framework for categorizing DevOps problems and solutions.
CultureAutomationMeasurementSharing
40Friday, October 7, 2011