Exploits in

Management*

Robert BarnesGeneral Manger, Microsoft.Com Core

Microsoft*No lab animals were harmed in the making of this story – some were used…

1

Chapters

• Background • Path to Management• Management Challenges

2

Background

• Rubber bands…• Be Prepared…• To Be or Not to Be…• Outliers…

3

• Microsoft.com• Staffing

• 150 FTE, 300 vendor• Redmond, Beijing, Hyderabad

• What we build• www.microsoft.com• www.microsoft.com/downloads• Content Publishing and analytics• Redemption

• Infrastructure and Application Footprint• 3 Internet data centers and 5 CDN partnerships• 2+ gb/sec website; 100-300+ gb/sec downloads• 1.3 – 1.7 billion hits/day from 57+ million unique IPs• 128 Web Servers• 10 SQL Servers• Reach: #11 US, #8 Worldwide domain with 303M UU/month**• Availability: MSCOM Platform at 99.93% availability as measured by Keynote

** Data Source: WebTrends FY10Q2

My Day Job

4

In the Beginning• In the early 1980s, relational databases were new• Most applications used Flat files, ISAM files, or

Navigational Databases (Codasysl)• Relational was what people wanted….– What did they need?

• The Ice Box– Getting paid to play….

• Key lessons: – Scientific method critical to results– Feedback systems critical – you get what you measure– Outsiders perspective - Ignorance can be helpful

5

Scale-out vs. Scale-up

• The power of parallelism• The challenge of parallelism– Scale-up begins to look like scale-out as you add

processors– Scale-out with commodity hardware is economically

compelling– Every large successful web site uses scale-out – it is

practical, but hard– Operations complexity is the key barrier to scale-out

• Simple demo

6

DebitCredit History 1987: 256 tps Benchmark

• 14 M$ computer (Tandem)• A dozen people• False floor, 2 rooms of machines

Simulate 25,600 clients

A 32 node processor array

A 40 GB disk array (80 drives)

Hardware expertsAdmin expert

Performance expert

OS expert

Network expert Auditor

Manager

DB expert7

DebitCredit History 1988: DB2 / CICS Mainframe 65 tps

• IBM 4391 • Simulated network of 800 clients

2 x 3725 network controllers

Refrigerator-sizedCPU

16 GB disk farm4 x 8 x .5GB

• 2m$ computer• Staff of 6 to do benchmark

8

DebitCredit History1997: 10 years later 1 Person and 1 box = 1400 tps

• 1 Breadbox ~ 5x 1987 machine room• 23 GB is hand-held• One person does all the work• Cost/tps is 20,000x less

1 micro dollar per transaction4x200 Mhz cpu1/2 GB DRAM12 x 4GB disk

Hardware expert

3 x7 x 4GB disk arrays

OS expertNet expertDB expertApp expert

9

DebitCredit • Database is a hypothetical Bank– 1.6 Billion Accounts– 160,000 tellers, – 16,000 branches,– 30 day history file

• Transaction is – Debit or Credit an account • (and update teller, branch and history)

– 15% of transactions are distributed• non-local to teller/branch so involve multiple nodes

10

8 Seconds of Terror

11

Going Over to the Dark Side

• After years of IC and technical manager roles, I decided to find out what I had been doing wrong

• Learning the language of business was valuable

• Leadership and organizational change were my favorite courses– Playing in the desert

12

Challenge: Driving Cross-Company Initiatives

• Common Engineering Criteria– Standardizing capabilities across server products– Driving new capabilities across server products

• Lessons Learned– Driving organizational change across divisions requires

top-down and bottoms-up efforts– Participation contributes towards support– Transparency in the process – Decision owner(s) and process must be clear– Exceptions process must be published and consistent

13

Challenge: New Org Manager, then Disaster Strikes

• Took on new role in December, 2007• Release review in November as observer– I told them they my have performance problems….– I told them they needed to improve engineering

• Jan. 8, 2008, a blogger post criticizes key executives in the company about Partner

• It turns out, May 2007 and Nov. 2007 releases had been flawed

• I had to fix the problem , learn the organization, and lead the organization at the same time

14

New Manager Disaster

• Quickly brought communication under control• Established “follow-the-sun” monitoring and

management • Drove hard changes in the organization

because we had to• People wanted to be led• People accepted change• People saw the value in change

15

Challenge: Re-org• Microsoft.Com reorganized into IT from product

groups in March, 2009• May, 2010: I become leader– Due to org-depth and SPOC, leadership changes

required– Friction between disciplines– IT resentment– Need for improved engineering practices– Managing performance to a curve

• “Engineering Culture” – a failed attempt

16

Engineering Culture

• Attempted to solve friction issue by asking Dev and Test disciplines to become “fungible”

• In new team (Beijing), part of org structure• In existing, not part of structure– People did not want to be led– People did not accept change– People did not see the value in change– Poll results tanked

17

The Future of Management

• Why to people become managers?– Seat at the table– Career growth– Wanting to be in charge– Someone has to do it

• Feedback systems are critical• Management innovation can only happen when

integrated with feedback systems and aligned with organization culture – context matters

18