Embed Size (px)
Citation preview
1
Advanced Topics Using Microsoft SQL Server 2005 Integration
Services
Allan Mitchell – SQLBits – Oct 2007
2
Introduction• Loading SSAS partitions directly• Using an SSIS Package as a data source• Data Mining in your Packages• Understanding Performance• Q and A
3
Who am I?• SQL Server MVP
• Co author on Wrox book on SSIS
• www.SQLDTS.com, www.SQLIS.com
• Consultancy company – Konesans (www.konesans.com)
4
What I will not cover
• Finding SSIS
• Creating packages
• General package design
5
Loading SSAS Directly
• Dimension And Partition Processing– Do not need to load the underlying DB– Quicker ROI– Don’t fully process the Partition/Dimension
outside of the ETL process !
6
DEMO
7
Using SSIS as a data source
• SSRS– Complex logic
(Microsoft.SqlServer.Dts.DtsClient)
• Windows App– Populate a web page perhaps
• Custom Component– Take the output of one package and use it in another
8
DEMO
9
Data Mining in SSIS
• Transforms– Data Mining Query
• Insurance Company perhaps– Term Extraction
• Textual Extraction (DT_WSTR or DT_NTEXT) – Term lookup
• Textual Lookup (DT_WSTR or DT_NTEXT)
• Destinations– Data Mining Model training
10
DEMO
11
Performance Tuning
• Data Correlation
•Use a merge Join over lookup – why?
•Remember sorted inputs – Do on source preferably.
•Avoid delay between 2 sources hitting.
12
Performance Tuning
• Buffer Strategy
•DefaultMaxBufferRows (10000)
•DefaultMaxBufferSize (10 MB)
•MaxBufferSize and MinBufferSize (100MB and 64KB)
•Avoid many small buffers.
•Turn on logging of BufferSizeTuning event
13
Parallel Load (Try 1)
14
Parallel Load (Try 2)
15
Parallel Load (Try 3)
16
The Raw File• Only SSIS can read and write• Only Adapter that does not use a
Connection Manager• High Speed• Source and Destination• No datatype conversions required• Eases back pressure
17
Demo
• The Raw File and How To Read
18
Access Modes and OLEDB Source Adapter
• Do not use Table or View as Access Mode
• Use SQL Command instead
• Openrowset (Overhead + bad exec plan)
• Sp_prepare (much better)
• (MS say 17* improvement)
