Using Standard Industry Benchmarks

Chapter 7 CSE807

Reasons For Using Benchmarks

• Real Workload impractical to set up Costly, hard to measure Unknown for most part

• Benchmarks Set of well-defined representative

programs to be run on different systems and networks to compare performance

Nature of Benchmarks

• Measurable and Repeatable

• Used for Monitoring and Diagnostics

• Used for Capacity Planning

• Can be confusing if customer workload does not match with the benchmark

• Best for comparing systems

• Complementary source of information in capacity studies

Nature of Benchmarks (cont.)• Users want fast Response Time and high

availability• Managers ask

– How many transactions/minute can the system handle

– How many requests/minute can the web site service

– What is the system’s operational cost– What is the initial hardware & installation cost– How to determine standard measure of system

performance

Performance Measures• Server Measures

– CPU speed– MIPS - millions of instructions per second

• RISC - Reduced Instruction Set Computer• CISC - Complex Instructions Set Computer

• Web Servers Measures– Inclusive of Clients and Server – Network related effects– Throughput– Latency

Benchmarks Hierarchy

Synthetic - Basic Operations, very limited utility– Dhrystone -Speed for fixed point computations– Whetstone -Speed for floating point

computations

Toy Benchmarks– Small programs implementing classical puzzles– Does not help in predicting performance for

any real workload

Benchmarks Hierarchy (cont.)

• Kernels– Livermore Loops and Linpack– Mainly for CPU performance– Not used for performance perceived by users

• Real Programs– SPEC and TPC– C compiler, UNIX utilities, debit & credit bank transactions etc.– Used to obtain most accurate picture of the system

performance perceived by the user

Avoiding Pitfalls

• Understand the benchmark environment– Processor specification, memory, I/O

subsystem, network and software(OS, DB)

• Compare your system with benchmark– Similarity and dissimilarity in the environment

• Representativeness of workload– Similarity and dissimilarity of the workload

Properties of a good benchmark

• Relevant - it must provide meaningful performance measure within a specific problem domain

• Understandable - results should be simple and easy to understand

• Scaleable - must be applicable to wide range of systems (costs, performance)

• Acceptable - should present unbiased results that are recognized by users and vendors

Component Level Benchmarks

• CPU - SPECxx– CINT Compute-intensive Integer performance– CFP Floating point performance– Designed for performance of computer

processor, memory architecture and compiler

– SPECratio is Ratio of Reference time to run time

– SPECint is the geometric mean of eight normalized ratios

Component Level (cont.)

• File Servers - synthetic benchmark that model a workload of input mix to file server– LADDIS - measure NFS server performance– measured at various load levels– generates throughput and average response time– parameters can be modified and adjusted to get

a workload representing a user environment– 50 msec is an arbitrary reference point

System Level Benchmarks

• Transaction Processing Systems• Measure the CPU, I/O subsystem, the

network, database, compilers and the OS• TPC - Transaction Processing Council

– TPC-B measures database transactions– TPC-A also measures network performance– TPC-C measure order-entry applications– TPC-D for decision support systems

System Level Benchmarks(cont.)

• Web Servers - simulate web browsers (assume no transmission errors)

• Webstone - simulates server and client processes, spawns a predefined number of clients for HTTP requests to server– Results include Throughput and Latency– Little’s Load Factor (LLF) gives degree of

concurrency– Avg. # of connections equal to Connection rate

times Avg. residence time

System Level Benchmarks(cont.)

• SPECweb - uses logs from popular web servers– workload parameters are fixed unlike LADDIS– result of SPECweb is the server’s maximum

throughput– Response time is for server only, does not

include network delays