Pstrong Cybera 29 Sept 2008

®

Paul StrongDistinguished Research Scientist, eBay Research Labs

eBay –Examples Of NGDC Technologies At Work

Cybera Summit 2008

©2008, eBay Inc.

®

Copyright Notice

© 2008 eBay Inc. All rights reserved.• No part of these materials may be reproduced or

transmitted in any form, by any means (electronic, photocopying, recording, or otherwise) without the prior permission of eBay Inc.

• eBay and the eBay logo are registered trademarks of eBay Inc.

• PayPal and the PayPal logo are registered trademarks of PayPal, Inc.

• Other trademarks and brands are the property of their respective owners.

• Please do not take our picture or record the class/session without asking permission.

©2008, eBay Inc.

®

What Is A Next Generation Data Center?

• Is it the platform for SOA?• Is it a Grid?• Does it leverage Virtualization technologies?• Is it full of blades?• Is it Greener?• Is it more efficient?• Is it more automated?

©2008, eBay Inc.

®

• Is it the platform for SOA?• Is it a Grid?• Does it leverage Virtualization technologies?• Is it full of blades?• Is it Greener?• Is it more efficient?• Is it more automated?

What Is A Next Generation Data Center?

©2008, eBay Inc.

®

NGDC Is…

More Than A Collection Of TechnologiesMore Than A Collection Of Technologies• Runs Business Processes driven by SLAs• Is a Value Center, rather than a Cost Center

– A new dialog between IT and the Business

• Enables– Internal utilities for core business functions/processes– External utilities for business process elements– Businesses based on business process mash-ups– Opportunities for new platforms

©2008, eBay Inc.

®

Cloud Computing

©2008, eBay Inc.

®

Cloud Computing Is…

• A new way about thinking of business…– Business = Codified core differentiating value

• Outsourcing everything that is not core– Physical infrastructure →

Clouds, Platform as a Service (PaaS)– Logical infrastructure (non-core business software) →

Software as a Service (SaaS)

• About being flexible– Temporary outsourcing…

• Not a new set of technologies– A new context for applying a set of evolving technologies that

are now more mature or capable

©2008, eBay Inc.

®

Cloud Computing Is About…

©2008, eBay Inc.

®

NGDC Is…

More Than A Collection Of TechnologiesMore Than A Collection Of Technologies• Runs Business Processes driven by SLAs• Is a Value Center, rather than a Cost Center

– A new dialog between IT and the Business

• Enables– Internal utilities clouds for core business functions/processes– External utilities clouds (PaaS, SaaS) for infrastructure and business

process elements– Businesses based on business process mash-ups– Opportunities for new platforms

©2008, eBay Inc.

®

NGDC, Friends & Relations

U

©2008, eBay Inc.

®

Managing NGDCs…

Is not about managing servers, operating systems, disks etc.

©2008, eBay Inc.

®

©2008, eBay Inc.

®

Why eBay Is A Useful Example

New Challenges

Extreme Engineering

Technology trickle down/transfer

Everyday useEveryday useThe Bleeding EdgeThe Bleeding Edge

©2008, eBay Inc.

®

• A car sells every 56 seconds

• A pair of shoes sell every 7 seconds

• A cell phones sells every 7 seconds

eBay – The 30 Second Introduction!

On an average day on eBay…

eBay users trade about $1,989 worth of goods on the site every second

1.3m people make all or part of their living selling on*

*ACNielsen International Research, June 2006

©2008, eBay Inc.

®

eBay’s Drivers

• Extreme Scale– 241m Registers Users, 112m+ Items, 7m+ New Items Per Day

• Extreme Growth– Near exponential growth in listings for most of history – 13

years

• Extreme Agility– Roll code to the site every 2 weeks

• Constant, predictable presence– Must be 24x7x365

• Efficiency

Failure To Keep Up Is Not An Option!Failure To Keep Up Is Not An Option!

©2008, eBay Inc.

®

eBay – The Simple Diagram

©2008, eBay Inc.

®

eBay Example #1Making The Database Scale

• Second Database for failover• CGI pools, Listings, Pages, and Search continued to scale horizontally

However …

By November 1999, the database servers approached their limits of physical growth.

Web ServerWeb Server

OSOS“Listings”


OSOS“Pages”

ApacheApacheCOTS SearchCOTS Search

UNIXUNIX“Search”

S/W Load Balancer S/W Load Balancer S/W Load Balancer S/W Load Balancer


C++C++OSOS

“CGIn”

RDBMSRDBMSUNIXUNIX

bear.ebay.com

RDBMSRDBMSUNIXUNIX

bull.ebay.com19991999

©2008, eBay Inc.

®

eBay Example #1Making The Database Scale

• Database "split" technology.• Logically partition database into separate instances.• Horizontal scalability through 2000, but not beyond.


OSOS“Listings”


OSOS“Pages”

ApacheApacheCOTS SearchCOTS Search

UNIXUNIX“Search”

S/W Load Balancer S/W Load Balancer S/W Load Balancer S/W Load Balancer


C++C++OSOS

“CGIn”

RDBMSRDBMSUNIXUNIX

bear.ebay.com

RDBMSRDBMSUNIXUNIX

bull.ebay.com

RDBMSRDBMSUNIXUNIX

chard.ebay.com

RDBMSRDBMSUNIXUNIX

cab/bongo.ebay.com20002000

©2008, eBay Inc.

®

eBay Example #1Virtualizing the Database

• Separate Application notion of a database from physical implementation• Databases may be combined and separated with no code changes• Reduce cost of creating multiple environments (Dev, QA, …)• Application can continue to function without non-critical data (markdown)

Attributes Catalogs RulesCATY1…N User Account Feedback Misc API Scratch

Application ServersApplication Servers

DB 1 DB 2 DB 3

©2008, eBay Inc.

®

November, 1999November, 1999

eBay Example #1Virtualizing & Scaling the Database

©2008, eBay Inc.

®

December, 2002SAN


©2008, eBay Inc.

®


• Scales Out– 241 million registered users– 112 million plus Items– 7 million plus new items per day– 50 billion SQL transactions per day– 600+ production database instances (inc replicas)– 100+ clusters

• Cheaper– Smaller, potentially commodity, servers

• Highly Resilient– 2-4 copies of everything– Minimized impact of outage to [relatively] small sub-set of data

• Flexible/Agile– Easy to change – database, schemas, partitioning etc.– Minimal impact on architecture or code

©2008, eBay Inc.

®

eBay Example #2Scaling The Application

©2008, eBay Inc.

®


• Partition code into functional areas– Application is specific to a single area (Buying, Selling etc.)– Domain contains common business logic across applications

• Restrict inter-dependencies– Applications depend on Domains, not on other applications– No dependencies among shared domains

User ApplicationUser Application

User DomainUser Domain

Selling ApplicationSelling Application

Selling DomainSelling Domain

Buying ApplicationBuying Application

Buying DomainBuying Domain

Billing ApplicationBilling Application

Billing DomainBilling Domain

Search ApplicationSearch Application

Search DomainSearch Domain

Personalization DomainPersonalization Domain Shared Billing DomainShared Billing DomainUser Validation DomainUser Validation Domain

Shared Buying DomainShared Buying Domain myEBay DomainmyEBay Domain Shared Search DomainShared Search Domain

Core DomainCore Domain API DomainAPI Domain

Lookup DomainLookup Domain

SharedDomains

Applications

©2008, eBay Inc.

®

eBay Example #2Scaling The Application• Segment functions into separate application pools

– Minimizes/isolates DB dependencies– Allows for parallel development, deployment and monitoring

Load Balancer

Load Balancer

Load Balancer

Load Balancer

ViewItem Poolhttp://cgiX.ebay.com...

SYI Poolhttp://cgiY.ebay.com...

Web Servers

App Servers

WebWeb WebWeb

Load Balancer

Load Balancer

ASAS ASAS ASAS

WebWeb WebWeb

Load Balancer

Load Balancer

ASAS ASAS ASAS

Load Balancer

Load Balancer

User Acct Caty1 Caty20+

©2008, eBay Inc.

®


• Everything behaves as loosely coupled services• Minimize inter-dependencies• Infrastructure is like a giant FPGA

– Potential to re-program by re-routing traffic

• Scales Up And Out (Choice)– Scale out means scaled throughput and resilience– 16000+ concurrent instances– 8000+ servers (mainly blades)

• Efficiency– Run traffic from different time zones on the same server but

different instances

©2008, eBay Inc.

®

Consequences

• Scale Out– Pro – Scale, throughput, resilience, use commodity products– Con – More to manage – complexity, relationships

• Virtualization– Pro – Flexibility– Con – more relationships to manage

• Commodity– Interchangeable, choice, no lock in, lower unit cost

©2008, eBay Inc.

®

The Big Problem

# Relationships

# Relationships

# Components# Components

Management complexity scales with thisManagement complexity scales with this

©2008, eBay Inc.

®

Understanding Relationships

Service A is Service A is composedcomposed ofofPersistence SubPersistence Sub--Service BService B

Business Logic SubBusiness Logic Sub--Service CService CPresentation SubPresentation Sub--Service DService D A

B C DD

©2008, eBay Inc.

®


A

B C DD

Business Logic SubBusiness Logic Sub--Service C is Service C is composedcomposed ofofA Load Balancing ServiceA Load Balancing ServiceSeveral Application InstancesSeveral Application Instances

AppApp AppApp LBSLBS

©2008, eBay Inc.

®


A

C

The Application Instances are The Application Instances are hostedhosted onon


B DD

Operating System InstancesOperating System Instances

OSOS OSOS LBLB

The Load Balancing Service is The Load Balancing Service is hostedhosted ononA Load Balancer Operating SystemA Load Balancer Operating System

©2008, eBay Inc.

®


OSOS OSOS LBLB

A

C


B DD

SvrSvr SvrSvr LBLB

VSVS

The Operating System Instances are The Operating System Instances are hostedhosted ononServers or Virtual Servers, which are in turn hosted on serversServers or Virtual Servers, which are in turn hosted on servers

The Load Balancer OS is The Load Balancer OS is hostedhosted ononA Physical Load BalancerA Physical Load Balancer

©2008, eBay Inc.

®

Interaction/Traffic RelationshipsStarting To Look Complicated!

©2008, eBay Inc.

®

Relationships Are Everything!

• Everything is interconnected• Changing one thing causes ripples• How you connect things together determines

business functionality and business value• Agility is the ability to change these relationships

dynamically (easier with loosely coupled services)

©2008, eBay Inc.

®

Relationships Are Everything!

• Virtualization is about standardizing relationships and interposing/isolating one end from the other

• Understanding these relationships allows you to– Tie business processes to the infrastructure they run on– Map value to cost – Understand and manage traffic flow– Understand and manage provisioning etc.

• It’s all about managing relationships, not things!

©2008, eBay Inc.

®

Data Center Evolution

• Non-Deterministic• Out of band changes• Hosting other people’s code

• Emergent Behavior• People and other software using things in unintended ways

• SO-Aaaaaaaghhhhh• More relationships

©2008, eBay Inc.

®

Conclusions

• NGDC is not just about technology that enables greater scaling, flexibility, resilience etc.

• NGDC has to be about changing the nature of the data center and its relationship to the business

• The challenge is how to understand and manage relationships, not just things!– Relationships inside the datacenter– Relationships to external services (SaaS et al)– Relationships managed on external infrastructure (Clouds,

PaaS)– Dynamically created and destroyed relationships as services

come and go!

®

Thank YouPaul Strong

[email protected]@ebay.comDistinguished Research Scientist

eBay Research Labs,eBay Inc.