Upload
gabrielle-kelley
View
222
Download
1
Tags:
Embed Size (px)
Citation preview
TPC BenchmarksTPC Benchmarks
Charles LevineMicrosoft
[email protected] by Jim GrayGray @ Microsoft.com
March 1997
OutlineOutline
Introduction History of TPC TPC-A and TPC-B TPC-C TPC-D TPC Futures
Benchmarks: What and WhyBenchmarks: What and Why
What is a benchmark? Domain specific
No single metric possible The more general the benchmark, the less useful it is for
anything in particular. A benchmark is a distillation of the essential attributes of a
workload
Benchmarks: What and WhyBenchmarks: What and Why
Desirable attributes Relevant meaningful within the target domain
Understandable Good metric(s) linear, orthogonal, monotonic
Scaleable applicable to a broad spectrum of hardware/architecture
Coverage does not oversimplify the typical environment
Acceptance Vendors and Users embrace it
Portable Not limited to one hardware/software vendor/technology
Benefits and LiabilitiesBenefits and Liabilities
Good benchmarks Define the playing field Accelerate progress
Engineers do a great job once objective is measurable and repeatable
Set the performance agenda Measure release-to-release progress Set goals (e.g., 10,000 tpmC, < 100 $/tpmC) Something managers can understand (!)
Benchmark abuse Benchmarketing Benchmark wars
more $ on ads than development
Benchmarks have a LifetimeBenchmarks have a Lifetime
Good benchmarks drive industry and technology forward. At some point, all reasonable advances have been made. Benchmarks can become counter productive by encouraging
artificial optimizations. So, even good benchmarks become obsolete over time.
OutlineOutline
Introduction History of TPC TPC-A and TPC-B TPC-C TPC-D TPC Futures
What is the TPC?What is the TPC?
TPC = Transaction Processing Performance Council Founded in Aug/88 by Omri Serlin and 8 vendors. Membership of 40-45 for last several years
Everybody who’s anybody in software & hardware
De facto industry standards body for OLTP performance
Administered by:Shanley Public Relations ph: (408) 295-8894777 N. First St., Suite 600 fax: (408) 295-9768San Jose, CA 95112-6311 email: [email protected]
Most TPC specs, info, results on web page: www.tpc.org
TPC database (unofficial): www.microsoft.com/sql/tpc/
News: Omri Serlin’s FT Systems News (monthly magazine)
Two Seminal Events Leading to TPCTwo Seminal Events Leading to TPC
Anon, et al, “A Measure of Transaction Processing Power”, Datamation, April fools day, 1985. Anon, Et Al = Jim Gray (Dr. E. A. Anon) and 24 of his closest
friends Sort: 1M 100 byte records Mini-batch: copy 1000 records DebitCredit: simple ATM style transaction
Tandem TopGun Benchmark DebitCredit 212 tps on NonStop SQL in 1987 (!) Audited by Tom Sawyer of Codd and Date (A first) Full Disclosure of all aspects of tests (A first) Started the ET1/TP1 Benchmark wars of ’87-’89
1987: 256 tps Benchmark
14 M$ computer (Tandem) A dozen people False floor, 2 rooms of machines
Simulate 25,600 clients
A 32 node processor array
A 40 GB disk array (80 drives)
OS expert
Network expert
DB expert
Performance expert
Hardware experts
Admin expert
Auditor
Manager
1988: DB2 + CICS Mainframe 65 tps
IBM 4391 Simulated network of 800 clients 2m$ computer Staff of 6 to do benchmark
2 x 3725 network controllers
16 GB disk farm4 x 8 x .5GB
Refrigerator-sizedCPU
1997: 10 years later1 Person and 1 box = 1250 tps
1 Breadbox ~ 5x 1987 machine room 23 GB is hand-held One person does all the work Cost/tps is 1,000x less
25 micro dollars per transaction
4x200 Mhz cpu1/2 GB DRAM12 x 4GB disk
Hardware expertOS expertNet expertDB expertApp expert
3 x7 x 4GB disk arrays
What Happened?
Moore’s law: Things get 4x better every 3 years (applies to computers, storage, and networks)
New Economics: Commodityclass price/mips software $/mips k$/yearmainframe 10,000 100 minicomputer 100 10microcomputer 10 1
GUI: Human - computer tradeoffoptimize for people, not computers
mainframeminimicro
time
pric
e
TPC MilestonesTPC Milestones
1989: TPC-A ~ industry standard for Debit Credit 1990: TPC-B ~ database only version of TPC-A 1992: TPC-C ~ more representative, balanced OLTP 1994: TPC requires all results must be audited 1995: TPC-D ~ complex decision support (query) 1995: TPC-A/B declared obsolete by TPC Non-starters:
TPC-E ~ “Enterprise” for the mainframers TPC-S ~ “Server” component of TPC-C Both failed during final approval in 1996
TPC vs. SPECTPC vs. SPEC
SPEC (System Performance Evaluation Cooperative) SPECMarks
SPEC ships code Unix centric CPU centric
TPC ships specifications Ecumenical Database/System/TP centric Price/Performance
The TPC and SPEC happily coexist There is plenty of room for both
OutlineOutline
Introduction History of TPC TPC-A and TPC-B TPC-C TPC-D TPC Futures
TPC-A OverviewTPC-A Overview
Transaction is simple bank account debit/credit Database scales with throughput Transaction submitted from terminal
Read 100 bytes including Aid, Tid, Bid, Delta from terminal (see Clause 1.3)BEGIN TRANSACTION
Update Account where Account_ID = Aid:Read Account_Balance from AccountSet Account_Balance = Account_Balance + Delta
Write Account_Balance to AccountWrite to History:
Aid, Tid, Bid, Delta, Time_stampUpdate Teller where Teller_ID = Tid:
Set Teller_Balance = Teller_Balance + DeltaWrite Teller_Balance to Teller
Update Branch where Branch_ID = Bid:Set Branch_Balance = Branch_Balance + DeltaWrite Branch_Balance to Branch
COMMIT TRANSACTIONWrite 200 bytes including Aid, Tid, Bid, Delta, Account_Balance to terminal
TPC-A TransactionTPC-A Transaction
TPC-A Database SchemaTPC-A Database Schema
LegendLegend
Table NameTable Name<cardinality><cardinality>
one-to-manyone-to-manyrelationshiprelationship
BranchBranchBB
AccountAccountB*100KB*100K
100K100K
HistoryHistoryB*2.6MB*2.6M
TellerTellerB*10B*101010
10 Terminals per Branch row10 Terminals per Branch row10 second cycle time per terminal10 second cycle time per terminal1 transaction/second per Branch row1 transaction/second per Branch row
TPC-A TransactionTPC-A Transaction
Workload is vertically aligned with Branch Makes scaling easy But not very realistic
15% of accounts non-local Produces cross database activity
What’s good about TPC-A? Easy to understand Easy to measured Stresses high transaction rate, lots of physical IO
What’s bad about TPC-A? Too simplistic! Lends itself to unrealistic optimizations
TPC-A Design RationaleTPC-A Design Rationale
Branch & Teller in cache, hotspot on branch
Account too big to cache requires disk access
History sequential insert hotspot at end 90-day capacity ensures reasonable ratio of disk to cpu
RTE SUTRTE SUT
RTE - Remote Terminal Emulator Emulates real user behavior
Submits txns to SUT, measures RT Transaction rate includes think time Many, many users (10 x tpsA)
SUT - System Under Test All components except for terminal
Model of system:
T
T
T - C Network*
CL
IE
NT
C - S Network*
SUTRTE
Response Time Measured Here
Host System(s)
S - S Network*
SER
VER
TPC-A MetricTPC-A Metric
tpsA = transactions per second, average rate over 15+ minute interval, at which 90% of txns get <= 2 second RT
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Response time (seconds)
Nu
mb
er
of
Tra
ns
acti
on
s
Average Response Time
90th Percentile Response Time
TPC-A PriceTPC-A Price
Price 5 year Cost of Ownership:
hardware, software, maintenance
Does not include development, comm lines, operators, power, cooling, etc.
Strict pricing model one of TPC’s big contributions List prices System must be orderable & commercially available Committed ship date
Differences between TPC-A and TPC-BDifferences between TPC-A and TPC-B
TPC-B is database only portion of TPC-A No terminals No think times
TPC-B reduces history capacity to 30 days Less disk in priced configuration
TPC-B was easier to configure and run, BUT Even though TPC-B was more popular with vendors,
it did not have much credibility with customers.
TPC LoopholesTPC Loopholes
Pricing Package pricing Price does not include cost of five star wizards needed to get optimal
performance, so performance is not what a customer could get.
Client/Server Offload presentation services to cheap clients, but report performance
of server
Benchmark specials Discrete transactions Custom transaction monitors Hand coded presentation services
TPC-A/B LegacyTPC-A/B Legacy
First results in 1990: 38.2 tpsA, 29.2K$/tpsA (HP) Last results in 1994: 3700 tpsA, 4.8 K$/tpsA (DEC) WOW! 100x on performance & 6x on price in 5 years !! TPC cut its teeth on TPC-A/B; became functioning, representative
body Learned a lot of lessons:
If benchmark is not meaningful, it doesn’t matter how many numbers or how easy to run (TPC-B).
How to resolve ambiguities in spec How to police compliance Rules of engagement
TPC-A Established OLTP Playing FieldTPC-A Established OLTP Playing Field
TPC-A criticized for being irrelevant, unrepresentative, misleading
But, truth is that TPC-A drove performance, drove price/performance, and forced everyone to clean up their products to be competitive.
Trend forced industry toward one price/performance, regardless of size.
Became means to achieve legitimacy in OLTP for some.
OutlineOutline
Introduction History of TPC TPC-A and TPC-B TPC-C TPC-D TPC Futures
TPC-C OverviewTPC-C Overview
Moderately complex OLTP The result of 2+ years of development by the TPC Application models a wholesale supplier managing orders. Order-entry provides a conceptual model for the benchmark;
underlying components are typical of any OLTP system. Workload consists of five transaction types. Users and database scale linearly with throughput. Spec defines full-screen end-user interface. Metrics are new-order txn rate (tpmC) and price/performance
($/tpmC) Specification was approved July 23, 1992.
TPC-C’s Five TransactionsTPC-C’s Five Transactions
OLTP transactions: New-order: enter a new order from a customer Payment: update customer balance to reflect a payment Delivery: deliver orders (done as a batch transaction) Order-status: retrieve status of customer’s most recent order Stock-level: monitor warehouse inventory
Transactions operate against a database of nine tables. Transactions do update, insert, delete, and abort;
primary and secondary key access. Response time requirement:
90% of each type of transaction must have a response time 5 seconds, except (queued mini-batch) stock-level which is 20 seconds.
TPC-C Database SchemaTPC-C Database Schema
WarehouseWarehouseWW
LegendLegend
Table NameTable Name<cardinality><cardinality>
one-to-manyone-to-manyrelationshiprelationship
secondary indexsecondary index
DistrictDistrictW*10W*10
1010
CustomerCustomerW*30KW*30K
3K3K
HistoryHistoryW*30K+W*30K+
1+1+
ItemItem100K (fixed)100K (fixed)
StockStockW*100KW*100K100K100K WW
OrderOrderW*30K+W*30K+1+1+
Order-LineOrder-LineW*300K+W*300K+
10-1510-15
New-OrderNew-OrderW*5KW*5K0-10-1
22
TPC-C WorkflowTPC-C Workflow
11
Select txn from menu:Select txn from menu:1. New-Order 1. New-Order 45%45%2. Payment 2. Payment 43%43%3. Order-Status3. Order-Status 4%4%4. Delivery 4. Delivery 4%4%5. Stock-Level 5. Stock-Level 4%4%
Input screenInput screen
Output screenOutput screen
Measure menu Response TimeMeasure menu Response Time
Measure txn Response TimeMeasure txn Response Time
Keying time
Think time
33
Go back to 1Go back to 1
Cycle Time DecompositionCycle Time Decomposition(typical values, in seconds,(typical values, in seconds, for weighted average txn)for weighted average txn)
Menu = 0.3Menu = 0.3
Keying = 9.6Keying = 9.6
Txn RT = 2.1Txn RT = 2.1
Think = 11.4Think = 11.4
Average cycle time = 23.4Average cycle time = 23.4
Data SkewData Skew
NURand - Non Uniform Random NURand(A,x,y) =
(((random(0,A) | random(x,y)) + C) % (y-x+1)) + x Customer Last Name: NURand(255, 0, 999) Customer ID: NURand(1023, 1, 3000) Item ID: NURand(8191, 1, 100000)
bitwise OR of two random values skews distribution toward values with more bits on
75% chance that a given bit is one (1 - ½ * ½) data skew repeats with period “A” (first param of NURand())
NURand DistributionNURand Distribution
TPC-C NURand function: frequency vs 0...255
Record Identitiy [0..255]
Rel
ativ
e F
requ
ency
of
Acc
ess
to T
his
Rec
ord
0
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
0.1
0
10
20
30
40
50
60
70
80
90
10
0
110
12
0
13
0
14
0
15
0
16
0
17
0
18
0
19
0
20
0
21
0
22
0
23
0
24
0
25
0
cumulativedistribution
ACID TestsACID Tests
TPC-C requires transactions be ACID. Tests included to demonstrate ACID properties met. Atomicity
Verify that all changes within a transaction commit or abort.
Consistency Isolation
ANSI Repeatable reads for all but Stock-Level transactions. Committed reads for Stock-Level.
Durability Must demonstrate recovery from
Loss of power Loss of memory Loss of media (e.g., disk crash)
TransparencyTransparency
TPC-C requires that all data partitioning be fully transparent to the application code. (See TPC-C Clause 1.6) Both horizontal and vertical partitioning is allowed All partitioning must be hidden from the application
Most DB do single-node horizontal partitioning. Much harder: multiple-node transparency.
For example, in a two-node cluster:
Any DML operation must beAny DML operation must beable to operate against the able to operate against the entire database, regardless of entire database, regardless of physical location.physical location.
Warehouses:Warehouses:1-1001-100 101-200101-200
Node ANode Aselect * select * from warehousefrom warehousewhere W_ID = 150where W_ID = 150
Node BNode Bselect * select * from warehousefrom warehousewhere W_ID = 77where W_ID = 77
Transparency (cont.)Transparency (cont.)
How does transparency affect TPC-C? Payment txn: 15% of Customer table records are non-local
to the home warehouse. New-order txn: 1% of Stock table records are non-local to
the home warehouse.
In a cluster, cross warehouse traffic cross node traffic 2 phase commit, distributed lock management, or both.
For example, with distributed txns:Number of nodesNumber of nodes % Network Txns% Network Txns
11 0022 5.55.533 7.37.3nn10.910.9
TPC-C Rules of ThumbTPC-C Rules of Thumb
1.2 tpmC per User/terminal (maximum) 10 terminals per warehouse (fixed) 65-70 MB/tpmC priced disk capacity (minimum) ~ 0.5 physical IOs/sec/tpmC (typical) 300-700 KB main memory/tpmC So use rules of thumb to size 10,000 tpmC system:
How many terminals? How many warehouses? How much memory? How much disk capacity? How many spindles?
Response TimeResponse Timemeasured heremeasured here
Typical TPC-C Configuration (Conceptual)Typical TPC-C Configuration (Conceptual)
DatabaseDatabaseServerServer
......
ClientClient
C/SLAN
Term.LAN
Presentation ServicesPresentation Services Database FunctionsDatabase FunctionsEmulated User LoadEmulated User Load
Har
dwar
eH
ardw
are
RTERTE, e.g.:, e.g.:EmpowerEmpowerpreVuepreVueLoadRunnerLoadRunnerS
oftw
are
Sof
twar
e TPC-C application +TPC-C application +Txn Monitor and/orTxn Monitor and/ordatabase RPC librarydatabase RPC librarye.g., Tuxedo, ODBCe.g., Tuxedo, ODBC
TPC-C application TPC-C application (stored procedures) + (stored procedures) + Database engine +Database engine +Txn MonitorTxn Monitore.g., SQL Server, Tuxedoe.g., SQL Server, Tuxedo
Driver SystemDriver System
Competitive TPC-C Configuration TodayCompetitive TPC-C Configuration Today
7,128 tpmC; $89/tpmC; 5-yr COO= 569 K$ 2 GB memory, 85x9-GB disks (733 GB total) 6500 users
Demo of SQL Server + Web interfaceDemo of SQL Server + Web interface
User interface implemented w/ Web browser via HTML
Client to Server via ODBC SQL Server database engine All in one nifty little box!
TPC-C Current ResultsTPC-C Current Results
Best Performance is 30,390 tpmC @ $305/tpmC (Oracle/DEC)
Best Price/Perf. is 6,712 tpmC @ $65/tpmC (MS SQL/DEC/Intel) graphs show
high price of UNIX diseconomy of UNIX scaleup
tpmC vs $/tpmC(ignores Oracle 30ktpmC on 4x12 Alpha)
0
50
100
150
200
250
300
350
400
0 5000 10000 15000 20000
tpmC
$/tp
mC
DB2
Informix
MS SQL Server
Oracle
Sybase
tpmC & Price Performance(only "best" data shown for each vendor)
0
50
100
150
200
250
300
350
400
0 5000 10000 15000 20000
tpmC
$/t
pm
C
DB2
Informix
MS SQL Server
Oracle
Sybase
Compare SMP PerformanceCompare SMP Performance
tpmC vs CPS
0
5,000
10,000
15,000
20,000
0 5 10 15 20
CPUs
tpm
C
SUN Scaleability
0
2,000
4,000
6,000
8,000
10,000
12,000
14,000
16,000
18,000
20,000
0 5 10 15 20
cpus
tpm
C
SUN Scaleability
SQL Server
TPC-C SummaryTPC-C Summary
Balanced, representative OLTP mix Five transaction types Database intensive; substantial IO and cache load Scaleable workload Complex data: data attributes, size, skew
Requires Transparency and ACID Full screen presentation services De facto standard for OLTP performance
OutlineOutline
Introduction History of TPC TPC-A and TPC-B TPC-C TPC-D TPC Futures
TPC-D OverviewTPC-D Overview
Complex Decision Support workload The result of 5 years of development by the TPC Benchmark models ad hoc queries
extract database with concurrent updates multi-user environment
Workload consists of 17 queries and 2 update streams SQL as written in spec
Database load time must be reported Database is quantized into fixed sizes Metrics are Power (QppD), Throughput (QthD), and Price/Performance
($/QphD) Specification was approved April 5, 1995.
TPC-D SchemaTPC-D Schema
CustomerCustomerSF*150KSF*150K
LineItemLineItemSF*6000KSF*6000K
OrderOrderSF*1500KSF*1500K
SupplierSupplierSF*10KSF*10K
NationNation2525
RegionRegion55
PartSuppPartSuppSF*800KSF*800K
PartPartSF*200KSF*200K
Time2557
Legend:Legend:• • Arrows point in the direction of one-to-many relationships.Arrows point in the direction of one-to-many relationships.• • The value below each table name is its cardinality. SF is the Scale The value below each table name is its cardinality. SF is the Scale Factor.Factor.• • The Time table is optional. So far, not used by anyone.The Time table is optional. So far, not used by anyone.
TPC-D Database Scaling and LoadTPC-D Database Scaling and Load
Database size is determined from fixed Scale Factors (SF): 1, 10, 30, 100, 300, 1000, 3000 (note that 3 is missing, not a typo) These correspond to the nominal database size in GB.
(I.e., SF 10 is approx. 10 GB, not including indexes and temp tables.) Indices and temporary tables can significantly increase the total disk capacity.
(3-5x is typical)
Database is generated by DBGEN DBGEN is a C program which is part of the TPC-D spec. Use of DBGEN is strongly recommended. TPC-D database contents must be exact.
Database Load time must be reported Includes time to create indexes and update statistics. Not included in primary metrics.
TPC-D Query SetTPC-D Query Set
17 queries written in SQL92 to implement business questions. Queries are pseudo ad hoc:
Substitution parameters are replaced with constants by QGEN QGEN replaces substitution parameters with random values No host variables No static SQL
Queries cannot be modified -- “SQL as written” There are some minor exceptions. All variants must be approved in advance by the TPC
TPC-D Update StreamsTPC-D Update Streams
Update 0.1% of data per query stream About as long as a medium sized TPC-D query
Implementation of updates is left to sponsor, except: ACID properties must be maintained Update Function 1 (UF1)
Insert new rows into ORDER and LINEITEM tables equal to 0.1% of table size
Update Function 2 (UF2) Delete rows from ORDER and LINEITEM tables
equal to 0.1% of table size
TPC-D ExecutionTPC-D Execution
Power Test Queries submitted in a single stream (i.e., no concurrency) Sequence:
Throughput Test Multiple concurrent query streams Single update stream Sequence:
CacheCache FlushFlush
QueryQuerySet 0Set 0(optional)(optional)
UF1UF1QueryQuerySet 0Set 0 UF2UF2
Timed SequenceTimed SequenceWarm-up, untimedWarm-up, untimed
Query Set 1Query Set 1Query Set 2Query Set 2
Query Set NQuery Set N
UF1 UF2 UF1 UF2 UF1 UF2UF1 UF2 UF1 UF2 UF1 UF2Updates:Updates:
.... ..
TPC-D MetricsTPC-D Metrics
Power Metric (QppD) Geometric queries per hour times SF
Throughput (QthD) Linear queries per hour times SF
QppD Size SF
QI i UI jj
j
i
i@
( , ) ( , )
3600
0 0191
2
1
17
where
QI(i,0) Timing Interval for Query i, stream 0
UI(j,0) Timing Interval for Update j, stream 0
SF Scale Factor
QthD SizeS
SFTS@
17
3600
where:
S number of query streams
T elapsed time of test (in seconds)S
TPC-D Metrics (cont.)TPC-D Metrics (cont.)
Composite Query-Per-Hour Rating (QphD) The Power and Throughput metrics are combined to get
the composite queries per hour.
Reported metrics are: Power: QppD@Size Throughput: QthD@Size Price/Performance: $/QphD@Size
Comparability: Results within a size category (SF) are comparable. Comparisons among different size databases are strongly discouraged.
QphD Size QppD Size QthD Size@ @ @
TPC-D Current ResultsTPC-D Current Results
0
2,000
4,000
6,000
8,000
10,000
12,000
14,000
0 200 400 600 800 1000 1200 1400 1600 1800 2000
Performance
Pri
ce/P
erfo
rman
ce
IBM/DB2
NCR/Teradata
Pyramid/Oracle
Sun/Oracle
Sun/Oracle
TPC-D 300 GB Results
TPC-D 100 GB Results
$0
$2,000
$4,000
$6,000
$8,000
$10,000
$12,000
$14,000
0 200 400 600 800 1000 1200 1400 1600
HP/Oracle
NCR/Teradata
Sun/Oracle
Tandem/NonStop SQL
Digital/Oracle (9/96)
Digital/Oracle (11/96)
IBM/DB2
Example TPC-D ResultsExample TPC-D Results
Want to learn more about TPC-D?Want to learn more about TPC-D?
TPC-D Training Video Six hour video by the folks who wrote the spec. Explains, in detail, all major aspects of the benchmark.
Available from the TPC:Shanley Public Relations ph: (408) 295-8894777 N. First St., Suite 600 fax: (408) 295-9768San Jose, CA 95112-6311 email: [email protected]
OutlineOutline
Introduction History of TPC TPC-A and TPC-B TPC-C TPC-D TPC Futures
TPC Future DirectionTPC Future Direction
TPC-Web The TPC is just starting a Web benchmark effort. TPC’s focus will be on database and transaction
characteristics. The interesting components are:
Web Server
WebServer
Appl.
FileSystem
DBMS Server
SQLEngine Stored
Procs
Database
TCP/IP
Browser Browser
...
Rules of ThumbRules of Thumb
Answer Set for TPC-C rules of Thumb (slide 38)a 10 ktpmC system
» » 8340 terminals ( = 5000 / 1.2)8340 terminals ( = 5000 / 1.2)
» 834 warehouses ( = 8340 / 10)» 834 warehouses ( = 8340 / 10)
» 3GB-7GB DRAM of memory (10,000 * [3KB..7KB])» 3GB-7GB DRAM of memory (10,000 * [3KB..7KB])
» 650 GB disk space = 10,000 * 65» 650 GB disk space = 10,000 * 65
» # Spindles depends on MB capacity vs. physical IO. » # Spindles depends on MB capacity vs. physical IO. Capacity: 650 / 4GB = 162 spindlesCapacity: 650 / 4GB = 162 spindlesIO: 10,000*.5 / 140 = 31 IO/sec IO: 10,000*.5 / 140 = 31 IO/sec (OK!)(OK!)
but 9GB or 23GB disks would be but 9GB or 23GB disks would be TOO HOT!TOO HOT!
Reference MaterialReference Material
Jim Gray, The Benchmark Handbook for Database and Transaction Processing Systems, Morgan Kaufmann, San Mateo, CA, 1991.
Raj Jain, The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling, John Wiley & Sons, New York, 1991.
William Highleyman, Performance Analysis of Transaction Processing Systems, Prentice Hall, Englewood Cliffs, NJ, 1988.
TPC Web site: www.tpc.org Microsoft db site: www.microsoft.com/sql/tpc/ IDEAS web site: www.ideasinternational.com