Table Partitioning in SQL Server: A Magic Solution for Better Performance? (Pragmatic Works)

Table Partitioning in SQL ServerA Magic Solution for Better Performance?

Cathrine Wilhelmsen

@cathrinew

cathrinewilhelmsen.netData Warehouse Architect

Business Intelligence Developer

You?New to table partitioning

"Everything is slow"

Today?Basic Introduction

What, Why & How

A Magic Solution for Better Performance?

Spoiler Alert!

A Magic Solution for Better Performance?

Implementing table partitioning is not a trivial task

and can actually cause worse performance...

A Magic Solution for Better Performance?

...but don't worry, I made plenty of mistakes

so you can avoid them

Enterprise Edition only

What?Partition Key

Partition Function

Partition SchemeWhy?

Backup & Restore

Maintenance

Load & ArchiveHow?

Partition Elimination

Switch, Split & Merge

Sliding Windows

Table Partitioning Basics

What is a partitioned table?

2012-01-01 ... ...

2012-12-31 ... ...

2013-01-01 ... ...

2013-12-31 ... ...

2014-01-01 ... ...

2014-12-31 ... ...

2015-01-01 ... ...

2015-12-31 ... ...

Data is physically stored in groups of rows called partitions

Each partition can be accessed and maintained separately

Partitioning is not visible to users, it behaves like one logical table

Partition Key

2012-01-01 ... ...

2012-12-31 ... ...

2013-01-01 ... ...

2013-12-31 ... ...

2014-01-01 ... ...

2014-12-31 ... ...

2015-01-01 ... ...

2015-12-31 ... ...

Data is partitioned based on a single column, the Partition Key

The Partition Key should always be used as a filter in queries

This ensures Partition Elimination:only relevant partitions are accessed

Partition Function

2012-01-01 ... ...

2012-12-31 ... ...

2013-01-01 ... ...

2013-12-31 ... ...

2014-01-01 ... ...

2014-12-31 ... ...

2015-01-01 ... ...

2015-12-31 ... ...

The Partition Function defines how to partition the data

It specifies boundary values, the points between two partitions

It specifies if the boundary value belongs to its left (upper) partition or its right (lower) partition

Partition Function: Range Left and Range Right

2012-01-01 ... ...

2012-12-31 ... ...

2013-01-01 ... ...

2013-12-31 ... ...

2014-01-01 ... ...

2014-12-31 ... ...

2015-01-01 ... ...

2015-12-31 ... ...

2012-01-01 ... ...

2012-12-31 ... ...

2013-01-01 ... ...

2013-12-31 ... ...

2014-01-01 ... ...

2014-12-31 ... ...

2015-01-01 ... ...

2015-12-31 ... ...

Partition Function: Range Left and Range Right

Range Left means the boundary value is the last value in the left partition

CREATE PARTITION FUNCTIONpfLeft (INT) AS RANGE LEFTFOR VALUES (20,30,40);

Range Right means the boundary value is the first value in the right partition

CREATE PARTITION FUNCTIONpfRight (INT) AS RANGE RIGHT

FOR VALUES (20,30,40);

...20 21-30 41...31-40 ...19 20-39 40...30-39

20 30 40 20 30 40

Partition Scheme

2012-01-01 ... ...

2012-12-31 ... ...

2013-01-01 ... ...

2013-12-31 ... ...

2014-01-01 ... ...

2014-12-31 ... ...

2015-01-01 ... ...

2015-12-31 ... ...

The Partition Scheme maps logical partitions to physical filegroups

Filegroups?

Data files on one or more disks

Backed up and restored individually

Can be Read-Only

Map all partitions to one filegroup

FILEGROUP

2012-01-01 ... ...

2012-12-31 ... ...

2013-01-01 ... ...

2013-12-31 ... ...

2014-01-01 ... ...

2014-12-31 ... ...

2015-01-01 ... ...

2015-12-31 ... ...

Map partitions to separate filegroups

FILEGROUP1(Read-Only)

FILEGROUP2(Read-Only)

2012-01-01 ... ...

2012-12-31 ... ...

2013-01-01 ... ...

2013-12-31 ... ...

2014-01-01 ... ...

2014-12-31 ... ...

2015-01-01 ... ...

2015-12-31 ... ...

FILEGROUP3

FILEGROUP4

How are partitions mapped to filegroups?

...20 21-30 41...31-40 ...19 20-39 40...30-39

20 30 40 20 30 40

The partition function specified the boundary values and partitions:

How are partitions mapped to filegroups?

CREATE PARTITION SCHEMEpsLeft AS PARTITION pfLeftTO (FG1, FG2, FG3, FG4);

CREATE PARTITION SCHEMEpsRight AS PARTITION pfRight

TO (FG1, FG2, FG3, FG4);

...20 21-30 41...31-40 ...19 20-39 40...30-39

20 30 40 20 30 40

The partition scheme uses the partition function...

How are partitions mapped to filegroups?

CREATE PARTITION SCHEMEpsLeft AS PARTITION pfLeftTO (FG1, FG2, FG3, FG4);

CREATE PARTITION SCHEMEpsRight AS PARTITION pfRight

TO (FG1, FG2, FG3, FG4);

...20 21-30 41...31-40 ...19 20-39 40...30-39

20 30 40 20 30 40

FG1 FG2 FG4FG3 FG1 FG2 FG4FG3

...to map each partition to filegroups:

Filegroups

Partition Scheme

Partitioned Table

2012-01-01 ... ...

2012-12-31 ... ...

2013-01-01 ... ...

2013-12-31 ... ...

2014-01-01 ... ...

2014-12-31 ... ...

2015-01-01 ... ...

2015-12-31 ... ...

A partitioned table is created on a partition scheme instead of directly on a filegroup

The partition scheme uses the partition key to store rows in the correct partition and filegroup based on the definition specified in the partition function

Table Partitioning Basics

Why Table Partitioning?

Partition Elimination

SELECT COUNT(*) FROM TableWHERE Year = 2012;

SELECT COUNT(*) FROM Table;

Partition Elimination

SELECT COUNT(*) FROM TableWHERE Year = 2012;

SELECT COUNT(*) FROM Table;

Backup & Restore Partitions

Filegroups can be backed up and restored individually

If each partition is mapped to a separate filegroup, partitions with the most critical data can be restored first

Backup & Restore Partitions

Filegroups can be read-only

If each partition is mapped to a separate filegroup, partitions with historical, unchanging data can be excluded from regular backups

Index Maintenance per Partition

Rebuild and reorganize indexes per partition

Rebuild indexes online per partition was introduced in SQL Server 2014

Set data compression per partition

ALTER INDEX IndexNameON TableNameREBUILD PARTITION = 2WITH (ONLINE = ON);

Statistics Maintenance per Partition

Update statistics on specific partitions instead of scanning and updating the whole table

UPDATE STATISTICSTableName (StatisticsName)WITH RESAMPLEON PARTITIONS (3,5);

CREATE STATISTICSStatisticsName ONTableName (ColumnName)WITH INCREMENTAL = ON;

Incremental Statistics was introduced in SQL Server 2014

Load & Archive: Partition Switching

Partitions can be switched between tables, called switching in or switching out

Partition switching is a metadata operation that updates the location of the data, no data is physically moved

Extremely fast compared to inserting into or deleting from a large table

2013-01-01 ... ...

2013-12-31 ... ...

2014-01-01 ... ...

2014-12-31 ... ...

2015-01-01 ... ...

2015-12-31 ... ...

2012-01-01 ... ...

2012-12-31 ... ...

ALTER TABLE Table1 SWITCH PARTITION 5

TO Table2 PARTITION 5;

SELECT$PARTITION.pf(2014);

Load & Archive: Switch out

2012-01-01 ... ...

2012-12-31 ... ...

2013-01-01 ... ...

2013-12-31 ... ...

2014-01-01 ... ...

2014-12-31 ... ...

2015-01-01 ... ...

2015-12-31 ... ...

Called switch out when you move data out of a table (archive)

Load & Archive: Switch out

2013-01-01 ... ...

2013-12-31 ... ...

2014-01-01 ... ...

2014-12-31 ... ...

2015-01-01 ... ...

2015-12-31 ... ...

2012-01-01 ... ...

2012-12-31 ... ...

Called switch out when you move data out of a table (archive)

Load & Archive: Switch in

2013-01-01 ... ...

2013-12-31 ... ...

2014-01-01 ... ...

2014-12-31 ... ...

2015-01-01 ... ...

2015-12-31 ... ...

2016-01-01 ... ...

2016-12-31 ... ...

Called switch in when you move data into a table (load)

Load & Archive: Switch in

2013-01-01 ... ...

2013-12-31 ... ...

2014-01-01 ... ...

2014-12-31 ... ...

2015-01-01 ... ...

2015-12-31 ... ...

2016-01-01 ... ...

2016-12-31 ... ...

Called switch in when you move data into a table (load)

Sliding Windows

The Sliding Windows technique automates data loading, data archiving and partition management

It keeps a certain number of partitions by continuously switching out the oldest data and switching in new data

2013-01-01 ... ...

2013-12-31 ... ...

2014-01-01 ... ...

2014-12-31 ... ...

2012-01-01 ... ...

2012-12-31 ... ...

2015-01-01 ... ...

2015-12-31 ... ...

Sliding Windows: Split & Merge

2012-01-01 ... ...

2012-12-31 ... ...

2013-01-01 ... ...

2013-12-31 ... ...

2014-01-01 ... ...

2014-12-31 ... ...

2015-01-01 ... ...

2015-12-31 ... ...

Partitions are not actually added or removed, they are split or merged

Be careful!Splitting and merging partitionscan cause data movement!!

Sliding Windows: Split & Merge

2012-01-01 ... ...

2012-12-31 ... ...

2013-01-01 ... ...

2013-12-31 ... ...

2014-01-01 ... ...

2014-12-31 ... ...

2015-01-01 ... ...

2015-12-31 ... ...

Split one partition in two by adding a new boundary value

ALTER PARTITION FUNCTION pf ()

SPLIT RANGE ('2013-06-01');

Data movement will occur if there is data on both sides of the new boundary value!

Sliding Windows: Split & Merge

2012-01-01 ... ...

2012-12-31 ... ...

2013-01-01 ... ...

2013-12-31 ... ...

2014-01-01 ... ...

2014-12-31 ... ...

2015-01-01 ... ...

2015-12-31 ... ...

Merge two partitions to one by removing a boundary value

ALTER PARTITION FUNCTION pf ()

MERGE RANGE ('2014-01-01');

Data movement will occur if there is data on both sides of the old boundary value!

Sliding Windows Steps

1. Add new filegroup and file

2. Create switch out (archive) table

3. Create switch in (load) table and load it with new data

4. Split to add new partition

5. Switch in new partition

6. Switch out old partition

7. Merge to remove old partition

8. Delete switch out and switch in tables

9. Delete old file and filegroup

Sliding Windows Demo

Alternative Sliding Windows Steps

1. Add new filegroup and file

2. Create switch out (archive) table

3. Switch out old partition

4. Merge to remove old partition

5. Create switch in (load) table and load it with new data

6. Split to add new partition

7. Switch in new partition

8. Delete switch out and switch in tables

9. Delete old file and filegroup

Why Table Partitioning?

Case Study: What Went Wrong?

Background: Financial Data Warehouse

Periodic Snapshot Fact TablesDaily, Weekly and Monthly Balances

SQL Server, DB2, Oracle, .csv, .txtLoaded at different times during the day

1 2 3 4 5 6 7

Pension Insurance Bank Fund

First version

Daily fact table:Keep 1 day per source

1 2

Pension

6 7

Bank Fund

Monthly fact table: Keep 36 months per source

3 4 5

Insurance

Second version

Daily fact table:Keep 1 day per source

1 2

Pension

Monthly / Weekly fact table:Keep 36 months for some sourcesKeep 106 weeks for some sources

3 4 5

Insurance

!

6 7

Bank Fund

Third version

Daily fact table:Keep 1 day per source

1 2

Pension

Monthly / Weekly / Daily fact table:Keep 36 months for some sourcesKeep 106 weeks for some sources

Keep 7 days for some sources

3 4 5 6 7

Insurance Bank Fund

!!

"Everything is slow"

"Let's try partitioning"

Daily fact table partitioning

1 ... ...

2 ... ...

3 ... ...

4 ... ...

5 ... ...

6 ... ...

7 ... ...

Partition Key: source number

One partition per source

Helped with deadlock issues

Easy to switch data in and out

Monthly / Weekly / Daily fact table partitioning

12011 ... ...

12012 ... ...

12013 ... ...

12014 ... ...

19990 ... ...

19991 ... ...

19992 ... ...

Partition Key: Source number + Period type

Period type:YYYY for monthly data9991 for weekly data9992 for daily data

Helped with deadlock issues, but…

Monthly / Weekly / Daily fact table partitioning

12011 ... ...

12012 ... ...

12013 ... ...

12014 ... ...

19990 ... ...

19991 ... ...

19992 ... ...

Partition key was difficult to remember and was not used in queries

Data had to be inserted into and deleted from partitions with existing data, instead of switching partitions in and out

What went wrong?

"The usual suspects": Conflicting priorities, changing requirements and high pressure to deliver on time

Fact tables were expanded, not changed, when new sources and requirements were implemented

Partitioning was implemented on fact tables with multiple grains, instead of correcting the fact tables first

What did we learn?

test, test, test, test

References and Resources: Table Partitioning

Brent Ozar Unlimited: SQL Server Table Partitioning Resourcesbrentozar.com/sql/table-partitioning-resources/

Bradley Ball: Partitioning in SQL Server 2012pragmaticworks.com/Training/FreeTraining/ViewWebinar/WebinarID/541

References and Resources: Further Reading

Partial Backups in SQL Server 2014msdn.microsoft.com/en-us/library/ms191539.aspx

Piecemeal Restores in SQL Server 2014msdn.microsoft.com/en-us/library/ms177425.aspx

Benjamin Nevarez: SQL Server 2014 Incremental Statisticsbenjaminnevarez.com/2015/02/2014-incremental-statistics/

Thank you!

@cathrinew

cathrinewilhelmsen.net

no.linkedin.com/in/cathrinewilhelmsen

contact@cathrinewilhelmsen.net