SQLCAT: Designing scalable and complex Analysis Services cubes

SQLCAT: Designing scalable and complex Analysis Service CubesLessons learned from the Field

Denny Lee, Thomas KejserMicrosoft SQL Customer Advisory Team

BIA499A • SQLCAT: Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field

Are you sitting in front of the correct Denny? They look very much alike.

Designing scalable and complex Analysis Service Cubes: Lessons learned from the Field – Room 615-617

Getting SQL Service Broker Up and Running – Room 609


SQL Server Customer Advisory Team (SQLCAT)

• Works on the largest, most complex SQL projects worldwide• MySpace - 4.4 million concurrent users at peak time, 8 billion friend

relationships, 34 billion e-mails, 1 PetaByte store, scale-out using SSB and SOA http://www.microsoft.com/casestudies/Case_Study_Detail.aspx?casestudyid=4000004532

• Bwin – Most popular European online gaming site – 30000 database trans / second, motto: “Failure is not an option”; 100 TB total storage http://www.microsoft.com/casestudies/Case_Study_Detail.aspx?casestudyid=4000004138http://www.microsoft.com/casestudies/Case_Study_Detail.aspx?casestudyid=4000001470

• Korea Telecom - Largest telco in Korea serves 26 million customers; 3 TB Data Warehousehttp://www.microsoft.com/casestudies/Case_Study_Detail.aspx?casestudyid=4000001993

• Drives product requirements back into SQL Server from our customers and ISVs

• Shares deep technical content with SQL Server community• SQLCAT.com & http://blogs.msdn.com/mssqlisv

http://www.microsoft.com/casestudies/Case_Study_Detail.aspx?casestudyid=4000004532




http://www.sqlcat.com/

http://blogs.msdn.com/mssqlisv


SQL Server Design Win Program• Target the most challenging and innovative SQL Server

applications

• 10+ TB DW, 3k/tran/s OLTP, Large 500GB+ Cubes, Competitive migrations, Complex deployments, Server Consolidation (1000+)

• Invest in large scale, referenceable SQL Server projects across the world

• Provide SQLCAT technical & project experience• Conduct architecture and design reviews covering performance,

operation, scalability and availability• Offer use of HW lab in Redmond with direct access to SQL Server

development team

• Work with Marketing Team Developing PR


Upgrade Reliability Assurance

Regular status updates within one month of completing each playback test cycle

Complimentary testing for qualified customer against every new version of SQL Server

Opportunity to request improvements directly from the product group for potential upgrade issues

Preparing for a smooth transition to the next version of SQL Server ...

Status Reports Customer Workload

Microsoft Playback Lab

Two-way NDA and Participation Agreement to ensure security and

privacy

For more information, contact [email protected]


SQLCAT and SQL CSS Invite You…

ROOM 611

• To the SQL Server Clinic where the most experienced SQL Server experts in the world will be waiting to talk with you.

• Bring your toughest Questions / Challenges to the experts who have seen it all• Architect and Design your future applications with experts who have done it before

with some of the largest, most complex systems in the world• Or just stop in to say hello!

Session Objectives and Takeaways

Session Objective(s): • Provide the primary themes that revolve around

successful enterprise Analysis Services implementations• Show the themes in relation to how SQLCAT customers

address these issues

Themes

• Get that hardware balance right (yes, we have to talk about IOPS)

• Repeat after me: partitioning, partitioning, partitioning!

• Multi-user query concurrency– how to handle it• Keeping it simple• Locking – how it works – and how to work around it• What? Did you say ROLAP?

SQLCAT Case Studies

Generalized Learnings from the Field


The Importance of I/O

• Enterprise = lots of data = lots of threads on the storage engine

• Disk Partition Alignment Best Practices• Scale Out Using Read-Only

• 64K vs. 32K block size• Scale Out Query performance

• Key Point:• Need to understand System Engineering concepts and

principles

http://sqlcat.com/whitepapers/archive/2009/05/11/disk-partition-alignment-best-practices-for-sql-server.aspx


The Importance of I/ODisk Partition Alignment

Disk Partition Alignment Best Practices

6 Disks - Not Aligned6 Disks - Aligned

8 Disks - Not Aligned8 Disks - Aligned

0

5

10

15

20

25

30

35

40

45

50

Latency (ms)

Duration (s)

31

21 24

16

48.03

38.7138.65

31.45

Latency (ms) Duration (s)

*Lower is better

http://sqlcat.com/whitepapers/archive/2009/05/11/disk-partition-alignment-best-practices-for-sql-server.aspx


The Importance of I/OSSAS I/O pattern

Typically Low block sizes on scans – 32KIOPS are wide and shallow (lots of threads, short queue)Latency is key, low latency = less context switchingTypical NAND device speedup: 5-10x on SE heavy queries

0 32 64 96 128 160 192 224 2560

50000

100000

150000

200000

250000

0

1

2

3

4

5

8K IOPS on FusionIO drive

IOPSLatency (ms)

Queue Depth (total outstanding kept constant at 256)


Repeat after me: partitioning, partitioning, partitioning!

• To get performance on multi-core boxes,• Partition, partition, partition

• Query each partition, results in separate threads• Distinct count

• Non-overlapping, continuous • The importance of I/O is directly related to

storage engine queries


Repeat after me: partitioning,… Distinct Count performance

0:00:00

0:00:15

0:00:30

0:00:45

0:01:00

0:01:15

0:01:30

0:01:45

0:02:00

0:02:15

0:02:30

0:02:45

0:03:00

0:03:15

0:03:300

2

4

6

8

10

12

Physical Disk > Avg Disk Queue Length

Time x User (4) Time (23)Single UserTime x User (10) Time x Browser (4)

• Six different partitioning schemes• Server has 4 processors• Lower / Shorter Duration Avg

Disk Queue Length

Type Ranking

Time x User (4) 2.13

Time 3.72

Single 4.64

User 2.25

Time x User (10) 2.97

Time x Browser 5.37


Multi-user query concurrencyGeneral rules

• Heap/Virtual Memory config settings• MemoryHeapType = 2• HeapTypeForObjects = 0

• Long running queries block processing• It’s just time for a separation…

• Querying and processing servers• Also importance of scale-out querying


Multi-user query concurrencyScale Out Analysis Services: Read Only

AS Processing ServerAS

Que

ry La

yer

AS database

AS Query 1

AS Query 2

AS Query 3

AS Query 4

AS database

Fast Disk (e.g. SAN)

SE Query

Clients

Clients

Clients

NLB

SEQ

SynchronizeUser Queries

User Queries

SE Query

SE Query

SE Query

1

23

4


Multi-user query concurrencyQuery performance

One Two Three Four0

50

100

150

200

250

300

238.5634875

150.3148394

101.9169981

FE Heavy (80 cc users)

One Two Three Four0

20

40

60

80

100

120

88.177996003996

25.1254835164835

102.022311407161

28.3542539550375

SE Heavy

50 CC Users

60 CC Users


Keeping it simple!

Careful with calculation scripts• Don’t: One cube with calculation script thousands

of lines long• Cell-by-cell mode WILL kill you• Example from bank: speedup 1000x by rewriting

cell-by-cell to block styleWhen it’s large, careful with large features• Keep Parent/Child small (tens of thousands)• Keep M-M small (low millions max)


What? Did you say ROLAP?!!

-67158236

-2085251220

-458579892

-903461230

482449446

-1958890532

1101641660

-954727192

663180516

-808192897

2023856459

0 100000 200000 300000 400000 500000 600000 700000

MOLAP

ROLAP1757523554

-420451054

-1602608104

721132422

-889102219

990819614'

0

200000

400000

600000

800000

1000000

1200000

MOLAP

ROLAP

Case Study: AdCenter

How does AdCenter handle two 3.2TB (yes TB) cubes?


The Importance of I/O

EMC DMX V-Max to handle the I/O• V-Max are dedicated to the application• Hundreds of disks and spindles devoted to this process• Short-stroking the disks to ensure fastest I/O• Works closely with EMC directly (present at EMC World

regularly)• Testing with EMC EFDs (Enterprise Flash Drives)

• Systems Engineering team dedicated to DW process• Work closely with various vendors (EMC, HDS, etc.)

• References• Accelerating Microsoft adCenter

with Microsoft SQL Server 2008 Analysis Services.• REAL PRACTICES: Performance Scaling Microsoft adCenter with

Microsoft SQL Server 2008 Analysis Services on EMC VMAX

http://technet.microsoft.com/en-us/library/ee410017.aspx





It’s all about I/OView of adCenter Production

Storage Area Network

OLAP Processing Server

Windows Server 2003 x64 SP2SQL Server Enterprise Edition

32 GB RAM, 8 Xeon procs (16 cores)

Staging Data Warehouse

Windows Server 2003 x64 SP2SQL Server Integration Services

Network Load Balancing

Data Feeds

HBA BHBA A

Windows Server 2003 x64 SP2SQL Server Analysis Services

64GB RAM, 8 Xeon procs (16 cores)

OLAP Standby Server

SAN Fabric A

SAN Fabric B

HBA BHBA A HBA BHBA A

Host Bus Adapters: 400 MB/sec each

HBA BHBA A HBA BHBA A HBA BHBA A

adCenter Production Environment

Windows Server 2003 x64 SP2SQL Server Analysis Services

64GB RAM, 8 Xeon procs (16 cores)

OLAP LUNStandby OLAP LUN

19200 Max Reads9600 Max Writes

DW LUN

180 300GB 10K Drives

RAID 119200 Max Reads9600 Max Writes

180 300GB 10K Drives

RAID 12560 Max Reads2560 Max Writes

32 300GB 10K Drives

RAID 1


Repeat after me: partitioning, …

Facts Database

1 Partition per Day31 Partitions, 1 Month of Data

ALTER PARTITION FUNCTION PerDay () SPLIT RANGE(CAST(CONVERT(varchar, GETDATE()+1, 112) AS int))

1 2 3 4 5 6 7 8 9 10 11

12 13 14 15 16 17 18 19 20 21 22 23

0

0 11 17 1 14 18 2 5 22 3 12 21

4 13 19 6 15 23 7 209 8 10 16

...WHERE [date] = CAST(CONVERT(varchar, GETDATE(), 112) AS int) AND [hour] IN (0, 11, 17)

8 Evenly Distributed Partitions per Day3120 Partitions, 13 Months of Data

8 Parallel Partition Processing Jobs

Current Day Partition Set

Current Day Partition

Cube

Case Study: E&D

Immense amounts of Xbox Live service usage, purchase, and achievements data


Partitioning Strategy Evenly distributed, continuous, and non-overlapping

Rollups

June 2008

Mar 2010

...

Usage

-9223372036854775808 TO -8070015239897208167

-8070015239897208166 TO -6917692104582506052

8069877141170462663 TO 9223372036854775807

13 more...

• Monthly partitions• Data further subdivided into 16

partitions each month (1 per core)• Split based on SHA1 hash of userid

• Provides even distribution and non-overlapping data spread


The Importance of I/OThe effect of design and hardware

Day Week Month Quarter 7 months0

500

1000

1500

2000

2500

3000

Usage Cube Distinct Count Performance

Amount of Data

Run

Tme

(sec

onds

)

select [Measures].[Online Unique Users] on 0,[Date].[Date].[Date] on 1from Usagewhere [Date].[Calendar Month].[Calendar Month].&[2009]&[1]


Multi-User Query ConcurrencyUsing SSD to allow each server to handle more simultaneous queries

Run 6, One at a time Run 6, All at once Run 12, All at once0

500

1000

1500

2000

2500

3000

Distinct Count Parallel Querying - Total Runtime

Dev HDDDev SSDSQLCat SSD

Access Pattern

Runti

me

(sec

onds

)

Case Study: Yahoo!

Yes, I said 12TB cube


Oracle 10gCDF

Ad Optimization Platform Architecture

SSAS Cube Builder

NAS

SSAS Query Servers

HW NLB

Users

Partition 1

Partition 2

Partition N

Partition 1

Partition 2

Partition N

1.2TB/dayFile1

File2

FileN

50GB/hr

12TB cube

Case Study – Large Investment Banks


Real time Cube

Locking in Analysis Services – The Truth you have to deal with

Process Commit

Long Running Query

Perceived Runtime from User

TimeOut

New Perceived Runtime


Relational locking – same story, different engineImportant limitations in real time part of cube

• SWITCH/MERGE must take SCH-M lock (but held only a few ms)

• Can cause blocking queues

• If bulking into partitioned table – may cause a sort operations (use small batch sizes)

• Alternative Solution: partition real time part into a seperate table. Keep stale data in partitioned tabled.

Table2000

2001

2002LCK_S

LCK_SCH_M LCK_S

LCK_SLCK_S

LCK_S

LCK_S

LCK_S

LCK_S

LCK_S

LCK_S

Long Running Query


MOLAP Switching In Action

Basic idea:• Use MOLAP for historical data• Process latest MOLAP partitions oftenTypical Latencies in minutesConcerns:• How to partition cube for fast, frequent MOLAP

processing• Handling the processing lock when you need to

update the data

High Level Cube PartitioningPartitions by BOTH time and region

Full processing can be done on ”active” partitions

• ”Roll with the daylight”

• Isolates changes and reloads

Requirement: Must be able to fully recalculate risk

2010-01-01

NY London Bombay Tokyo

2010-01-02


2010-01-03


Cube FlippingRecall: Server level lock required to process

Alternative Solution:

• Two copies of the cube, taking turns processing

• ”Flip” between them

Two ways to flip:

• Use ASLB from CodePlex

• Write small Excel Plug-in

Excel Plug-in may be used when user holds session state

• Session state automatically refreshes when connectionstring is changed

Process

Query

(Query)

(Process)

http://www.google.com/imgres?imgurl=http://www.westpenngbc.com/excel_icon1.png&imgrefurl=http://www.westpenngbc.com/&usg=__LDr-jZp6MiBzhS82Ldm2dWZ0o7U=&h=256&w=256&sz=77&hl=da&start=17&sig2=oSIioBqxPfs5iFr4tMIgLQ&um=1&itbs=1&tbnid=Sofnz-1BkEnRQM:&tbnh=111&tbnw=111&prev=/images?q=excel&um=1&hl=da&sa=N&tbs=isch:1&ei=hZLnS6vtFoPmswPPraCYBA


ROLAP + MOLAP in Action

Basic ideas:

• Use MOLAP for historical/stale data

• Use ROLAP for frequently updated data

Typical latencies achievable are in seconds

Concerns/Challenges:

• Must maintain coherent ROLAP cache

• Use Real Time OLAP=true in connectionstring

• When MOLAP changes – must have minimum impact on others

• Consider archiving strategy

• Locking inside SQL Server must be handled (concurrent load). Consider RCSI


Real time Cube

Active Book

MOLAP(Market

Snapshot)

ROLAP

Active Book

MOLAP(Market

Snapshot)

ROLAP

Trader

Active Book

MOLAP(Market

Snapshot)

ROLAP

Active Book

MOLAP(Market

Snapshot)

ROLAP

x 25Trader

| SQL Server Customer Advisory Team

Real Time risk runs (RCSI mode)

Real time Cube

Active Book_X

MOLAP(Market

Snapshot)

ROLAP

Market Snapshots (MOLAP Processing)

ROLAP_Book_X

Snapshot_Xx 25

x 25Trickle feed ETL

| SQL Server Customer Advisory Team

Real Time risk runs (RCSI mode)

Market Snapshots (MOLAP Processing)

ROLAP_Book_X

Snapshot_Xx 25

x 25

History

Day1

Empty

Day2

Day3

6 months

…

Merge +

Agg

SWITCH

History Cube

Process


Two Real Time Cube methods - TradeoffsComponent MOLAP Switching ROLAP+MOLAP

Relation Tuning Low Must get right

AS Locking Must handle Not a big issues

Cache Usage Good Poor

Relational Concurrency N/A RCSI

Data Storage Best Compression ROLAP typically 2x MOLAP

Aggregate management Profiler + UBO Manual

Latency Minutes Seconds

Case Study: Sandvik FinanceDesigning scalable and complex Analysis Service Cubes


Challenge: Financial Consolidation

Need: Writeback with 5 second latency

Need: 100% Uptime during month closing

Need: 300 truly concurrent users

Need: Dynamic scale as company grows


Considerations

All about the hardware: Cubes a small, they will fit in memory if we get enough (dont use corporate SANMulti User: • Locking could be an issue, but all queries are

small (in control of the front end)• Processing Online may work.. But see later• Global company – what about latency?KISS: Review calculation script – CHECK!Security for auditing purpose



Latency of the link

Observations:• XMLA is flexible – but it is ”bloated” on bandwidth• Excel tends to be chatty clientResults: User will not NOT like the latency added by the speed of light• The ms required really adds up

Solution: Citrix farms with remote users. Gives great response times and good user experience


Partitioning for Real Time Setup

Financial Data

Historial Records

Current Month

Real Time

Historial Records(MOLAP / monthly)

Current Month(MOLAP / 10 min)

Current Month(ROLAP / ProActive)

Historial Records(MOLAP / monthly)

Current Month(MOLAP / 10 min)

Current Month(ROLAP / ProActive)

Read

Read

Writeback


Enryption techniques and overhead

Securing all lines and data:

• Crypt the XMLA traffic with SSL

• EFS on MOLAP data

• TDE on SQL Server data

• SQL Server TDS encryption

Overhead:

• Less than 10% on query performance

• Dimension processing times doubles!

• Fact Processing, only a few %


ROLAP – yes ROLAP!

No way to bring in data to MOLAP fast enough

• No – ProActive caching will not help.

• There is never a silent interval long enough

Solution: ROLAP partition with recent data

• Keep ROLAP partition small. Don’t want to touch the I/O system

• Avoids locking issue during processing

Keep the ROLAP small..

Technique:• Gradually shift data from ROLAP to MOLAP

”mid day” partition using ProActive cache

• Steps (every 10 min)

1. INSERT...SELECT from ROLAP partitions to MOLAP source table

2. Lock Cube

3. Process MOLAP (fast, only intra day data)

4. Modify View on ROLAP to exclude data

5. Unlock Cube

6. Delete Data (delay this, to avoid holding lock)

• Necessary assumption: All queries are short running


Summarizing (1/2)

You HAVE to get the design right if you want to scale

Partitioning is absolute crucial

• Partition for processing speed

• Partition for data latency (real time vs. Stale)

• Partition for archival of old data

• Secondary concern to above: partition for query speed

Hardware really matters for large cubes

• NAND devices are match made in heaven.

• Tests show two core Nehalem CPU can often support hundreds of users

• Carefully balance IOPS vs. Memory, consider hot portion of cube


Summarizing (2/2)

Locking is your enemy

• Consider processing server with sync vs. Individual servers processing

• Consider how to flip

• For super low latency, ROLAP is the way

Multi User Settings

• Just run them as per perf guide

• Consider adjusting CoordinatorBalancingFactor

Keep it Simple

• Remember the premise: AS is a ”send me any query” database.

• You really have to get this right at scale


Call to Action!Feedback really works – a story and “announcement”

Rules:

• Min 100GB cube and/or 100 concurrent users

• Max one pain / month

• Prefix mail subject: “Cube Feedback:”

Feedback, crucial questions:

• Who are you? (customer name / company profile)

• Measure/dimension size, users, cube project

• What are you trying to achieve?

• What does it cost you not to get there / lost opportunity?

My email: [email protected]

mailto:[email protected]


Complete the Evaluation Form to Win!

Win a Dell Mini Netbook – every day – just for handing in your completed form. Each session evaluation form represents a chance to win.

Pick up your evaluation form:• In each presentation room• At the PASS Booth near registration

Drop off your completed form:• Near the exit of each presentation room• At the PASS Booth near registration

Sponsored by Dell

for attending this session and the 2010 PASS Summit in Seattle

Technology

SQLCAT: Designing scalable and complex Analysis Services cubes