CS 672 – Introduction to Performance Evaluation and ...menasce/cs672/slides/CS672-intro.pdf · CS 672 – Introduction to Performance Evaluation and Capacity Planning Dr. Daniel

1

1

CS 672 – Introduction to Performance Evaluation and

Capacity Planning

Dr. Daniel A. Menascéhttp://www.cs.gmu.edu/faculty/menasce.html

Department of Computer ScienceGeorge Mason University

1999−2000 D. A. Menascé. All Rights Reserved.

2

Examples of Performance Problems

• AOL• Market crash of October 1997: investors

lost millions of dollars!• Holiday season of 1998: e-commerce sites

were crowded!• Online trading surge:

http://www.nytimes.com/aponline/f/AP-Internet-Trading-Snarls.html


2

3

AOL Demand Increase and Capacity Planning Problem

Avg . Sess ion Dura t i on (m in )

0

5

10

15

20

25

30

35

D e c -95

Sep -96

Jan-97

A v g .S e s s i o nDuration( m i n )

P o l y .(Avg .S e s s i o nDuration(min))

Dai ly Sessions (Mi l l ions)

0

2

4

6

8

10

12

Dec-95 Sep-96 J a n - 9 7

DailySess ions(mill ions)

Linear (Dai lySess ions(mil l ions))

Sessions(millions)

Linear (DailySessions(millions))

Peak No. Users

-

50,000

100,000

150,000

200,000

250,000

300,000

Dec-95

Sep-96

Jan-97

PeakSimultaneous Users

Linear (PeakSimultaneous Users)


4

Performance Problems Threaten E-commerce Success

• “... nascent attempt to reclaim its role as a leading source of information faltered Tuesday when its Web site crashed after being flooded by visitors. ... the overwhelming response caught the company unprepared.”

latimes.com October 20, 1999.

1999−2000 Menascé. All Rights Reserved.

3

5

• The site went down around 4 p.m. PT yesterday while experiencing record traffic, according to a company spokeswoman. The site posted this message: "Please check back soon. Due to the high number of visitors currently using our site, we're unable to start your session at this time."

... and other e-commerce sites have experienced similar intermittent outages as the volume of visitors flooding their sites steadily increases.

CNET News.com October 22, 1999, 11:30 a.m. PT

Performance Problems Threaten E-commerce Success


6

E-commerce Growth(source: Giga Information Group)

94,960

207,572

349,242

438,879

43,926

72,730

97,529

21,227

0

50,000

100,000

150,000

200,000

250,000

300,000

350,000

400,000

450,000

500,000

1999 2000 2001 2002

Ann

ual V

olum

e (in

$m

illio

n)

Business to Business Business to Consumer


4

7

Top e-tailers of 2001(*)

Rank Site

Unique Audience (thousands)

1 amazon.com 31,4872 Columbia House 8,6833 ToysRUs 7,5644 Barnes & Noble 6,2125 apple.com 6,1346 bestbuy.com 5,9937 dell.com 5,7138 walmart.com 5,6529 hp.com 5,223

10 sears.com 4,91911 target.com 4,68912 JCPenney.com 4,65413 cdnow.com 4,27514 hallmark.com 3,97415 compaq.com 3,657

(*) excludes travel sites,financial services, auctions,fnline subscriptions andsoftware OEM.

Source: Nielsen/NetRatings.

2003 Menascé. All Rights Reserved.

8

Nielsen/NetRatings: 2001 Holiday E-commerce Index (*)

Category Week ending Week ending Percent28-Oct 2-Dec Change

Toys & Games 2,761,504 6,156,039 122.90%Consumer Electronics 2,928,326 5,779,183 97.40%Home & Garden 868,266 1,582,941 82.30%Shopping Aggregators 20,055,366 36,286,128 80.90%Apparel 3,469,005 5,785,191 66.80%Virtual Dept. Stores 21,748,016 35,057,201 61.20%Value-Oriented Sites 6,773,770 9,137,098 34.90%Specialty Gifts 594,747 797,077 34.00%Computer Hardware 9,631,755 11,398,165 18.30%Books/Music/Video 4,931,930 5,071,701 2.80%Total 73,762,686 117,050,723 58.70%


(*) 5 representatives in each area. Number of visits.

5

9

Business in the Internet Age(Business Week, June 22, 1998)

Type of Business 1997 2001 (forecast)Business to Business 8.000 183.000Travel 0.654 7.400Financial Services 1.200 5.000PC Hardware & Software 0.863 3.800Entertainment 0.298 2.700Ticket Event Sales 0.079 2.000Books & Music 0.156 1.100Apparel & Footware 0.092 0.514Total 11.342 205.514


10

How often do you use the Web for shopping for personal reasons?

GVU's 10 th WWW User Survey (October 1998)

3%

27%

31%

17%

9%

13%

0%

5%

10%

15%

20%

25%

30%

35%

Never <1/mo 1-2/mo 3-5/mo 6-9/mo >10/mo


6

11

Caution Signs Along the Road(Gross & Sager, Business Week, June 22, 1998, p. 166.)

There will be jolts and delays alongthe way for electronic commerce:congestion is the most

obvious challenge.


12

Dissatisfying Experiences in E-commerce GVU's 10th WWW User Survey (October 1998)

4%

23%

14%13%

18%

22%

7%

0%

5%

10%

15%

20%

25%

Hasn't

Happe

ned

Confu

sing S

ite

Poor

page

desig

n

Obnox

ious S

ite

Downlo

ad to

o slow

Couldn

't find

Other R

easo

ns


7

13

What are people saying about Web performance…

• “Tripod’s Web site is our business. If it’s not fast and reliable, there goes our business.”,

Don Zereski, Tripod’s vice-president of Technology (Internet World)

• “Computer shuts down Amazon.com book sales. The site went down at about 10 a.m. and stayed out of service until 10 p.m.”

The Seattle Times, 01/08/98


14


• “Sites have been concentrating on the right content. Now, more of them -- specially e-commerce sites -- realize that performance is crucial in attracting and retaining online customers.”

Gene Shklar, Keynote, The New York Times, 8/8/98


8

15


• “Capacity is King.”Mike Krupit, Vice President of Technology, CDnow,

06/01/98

• “Being able to manage hit storms on commerce sites requires more than just buying more plumbing.”

Harry Fenik, vice president of technology, Zona Research, LANTimes, 6/22/98


16

What are people saying about Internet performance…

• “The capacity crunch is real and will continue for quite some time.”

Mike O’Dell, Chief Scientist, UUNET


9

17

Impacts of Bad Performance

• Bad performance: response time above 8 seconds (eight-second rule).

• $43.5 billion lost each year in e-commerce due to bad performance (Zona Research, April 1999).

• Holiday Season of 1998: over 1/3 of customers gave up due to slowness, 44% turned to conventional stores, 14% moved to another site.


18

Performance Problems in E-commerce tend to get worse!

1999 Menascé, Almeida, Fonseca & Mendes. All Rights Reserved.

• Proliferation of mobile devices• Easier to use interfaces (VUI, novel

browsing paradigms)• Increasing load placed by software agents• Impacts of authentication and payment

protocols (e.g., SSL, TLS, and SET) on e-commerce site performance.

10

19

Important Questions

• What is performance? What are important performance metrics?

• What is capacity?• What is capacity planning?• How important are capacity planning and

performance evaluation?


20

Virtual Car Dealer Example

Internet

clients

Webserver

Database:• vehicle description• price• picture

virtual car dealership


11

21

Virtual Car Dealer Example• Web server request types:

– retrieve document and images– search the DB according to make, model, price

range, and distance of dealer from buyer.– purchase request.

• Critical request: search• 5% of searches generate a car sale• Average sale generates $18,000 in revenues.


22

Virtual Car Dealer Example• Service Level Considerations:

Response time(in sec)

Outcome

4 < r ≤ 6 60% lost

r > 6 95% lost


12

23

Virtual Car Dealer Example• Management questions:

– will the Web server support the load increase while preserving the response time below 4 sec?

– if not, at which point will its capacity be saturated?

– how much money could be lost daily if the Web server saturates when the load increases?


24

Virtual Car Dealer Results

2003 D. A. Menascé. All Rights Reserved.

Current Current+10% Current+20% Current+30%Searches/day 92,448 101,693 110,938 120,182Response time (sec) 2.9 3.8 5.7 11.3Sales lost (%) 0 0 60 95Sales per day 4,622 5,085 2,219 300Daily revenue (in $1,000) 1,664 1,831 1,997 2,163Potential daily revenue (in $1,000) 1,664 1,831 799 108Lost daily revenue (in $1,000) 1,198 2,055

13

25

Important Concepts in Example• Workload characterization?

• Workload Growth Forecast?

• Performance metrics?

• Desired Service Levels?


26


– retrieve, search, and buy transactions– critical transaction: search– arrival rate: 92,448 transactions/day


14

27

Important Concepts in Example• Workload growth forecast?

– 10%, 20%, 30% increase in the workload intensity.

– no change expected in workload nature.


28

Important Concepts in Example• Performance metrics of interest?

– response time for search transactions.– How can be the response time measured?


15

29


– Response time for search transactions.– How can the response time be measured?

• at the server. What are the components?

• at the browser. Then, Internet and client LAN delays are part of response time.

• Measuring end-user response time for e-commerce sites: www.keynote.com


30

Important Concepts in Example• Service levels?

– response time for search transactions < 4 sec.


16

31

Intranet Performance Example• Major airplane manufacturer with 60,000

employees is implementing an intranet to support:– corporate training,– help desk support,– dissemination of internal corporate news, and– handling of personnel forms and memos.


32

Intranet Performance Example• Help desk application:

– dedicated Web server with a FAQs DB about common hardware/software problems and solutions,

– submission of problem descriptions via forms,– requests for status on previous claims.


17

33

Intranet Performance Example• 10% of employees submit requests to the help

desk application every day• 70% of these requests fall in the 10:00AM

12:00PM period and from 2:00PM to 4:00PM.• arrival rate =

(60,000 * 0.1 * 0.7) / (4 * 3,600) = 0.29 request/sec.


34

Intranet Performance Example• The OS on all clients will be changed:

– likely to create big surge in number of requests to the help desk application.

• Management question:– how will the help desk application response time

vary with the arrival rate?


18

35

Intranet Performance Result

0

10

20

30

40

50

60

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

Arrival Rate (tps)

Res

po

nse

Tim

e (s

ec) Connections start to

be refused.

current load.


36


• Workload Growth Forecast?

• Performance metrics?

• Desired Service Levels?


19

37


– help desk application: searches to a FAQs DB, submission of problems via forms, requests for status on previous claims

– critical transaction: help desk support requests– arrival rate: 0.29 tps during peak period.


38

Important Concepts in Example• Workload growth forecast?

– surge in requests to help desk due to new OS installed on client machines.


20

39


– response time for help desk requests.– connection rejection probability


40

Important Concepts in Example• Service levels?

– response time for search transactions < 12 sec.


21

41

Capacity Planning Input and Output Variables

Capacity Planning

WorkloadEvolution

SystemParameters

DesiredService Levels

SaturationPoints

Cost-effectiveAlternatives


42


Capacity Planning

WorkloadEvolution

SystemParameters


SaturationPoints


•intensity•nature


22

43


Capacity Planning

WorkloadEvolution

SystemParameters


SaturationPoints



• processors, disks, networks• max. no. connections


44


Capacity Planning

WorkloadEvolution

SystemParameters


SaturationPoints




• RT <= 2 sec• tput > 10 tps

when are service levels violated? 1999−2000 D. A. Menascé. All Rights Reserved.

23

45


Capacity Planning

WorkloadEvolution

SystemParameters


SaturationPoints




• RT <= 2 sec• tput > 10 tps

when are service levels violated? e.g., tps/$ 1999−2000 D. A. Menascé. All Rights Reserved.

46

Capacity Planning Definition

Capacity Planning is the process of predicting when the service levels will be violated as a function of the workload evolution, as well as the determination of the most cost-effective way of delaying system saturation.


24

47

Capacity Planning Concept

0

1

23

4

5

67

8

9

50 60 70 80 100

no. of client workstations

responsetime(sec)

servicelevel


48

Capacity Planning Concept

0

1

23

4

5

67

8

9

50 60 70 80 100

no. of client workstations

responsetime(sec)

servicelevel

cpuIO

network


25

49

Typical Capacity Planning Questions

• Situation: migrating from a mainframe based to a C/S system.

• Questions:– how many clients will the new system support

with acceptable response time?– How many servers and how should they be

configured to handle the load?– Should I use a two-tier or a three-tier

architecture?


50

Two-tier C/S architecture

GUIAppl.OS OS

DBMSExec SQL request

Result of SQL request

DBclient DB server


26

51

Three-tier C/S architecture

GUI +validation

OS OS

DBMS

Exec SQLrequest

Result of SQL request

DB

OS

clientApplication

ServerDB Server

Appl.

Exec function

Function result


52


• Situation: migrating from a mainframe based to a C/S system.

• Questions:– What should be the configuration of the

application servers?– Should the DB be replicated and how?– Which DB replication strategy should be used?


27

53


• Situation: redesigning an e-commerce Web site and adding a lot of multimedia content.

• Questions:– Will the bandwidth of the link to the ISP

support more multimedia content?– Will the IO bandwidth be enough to provide

adequate response time to multimedia requests?– What is the most appropriate configuration for

the Web site?


54


• Situation: redesigning an e-commerce Web site and adding a lot of multimedia content.

• Questions:– Should I use many small boxes or a few large

boxes to support the load?– What type of load balancing scheme should be

used?


28

55

Importance of Performance Evaluation and Capacity Planning

• Risk of financial losses• External image of the company• User dissatisfaction• Productivity decrease• Procurement cycle and budgetary

constraints.


56

Common Mistakes in Capacity Planning

• Performance varies linearly with the workload!

• Just throw more iron and the problem is solved.

• Incorrect data gathering procedures: garbage in garbage out.


Documents

CS 672 – Introduction to Performance Evaluation and ...menasce/cs672/slides/CS672-intro.pdf · CS 672 – Introduction to Performance Evaluation and Capacity Planning Dr. Daniel