Upload
sabina-hawkins
View
214
Download
1
Tags:
Embed Size (px)
Citation preview
Connectivity Options for Data Integration with Microsoft SQL Server Integration Services (SSIS)
Cedric LabuschagneSolutions ArchitectBI-LateralBIN301
ERP
Thinking Bigger About Business Intelligence
Majority of the content for decision making are in multiple heterogeneous sources (e.g. ERP systems, databases, flat files)
Need to integrate & consolidate data from all these sources, and make quality data available to users
Connectivity is the key to address these challenges
FlatFile
ERP Systems
Excel
XML
Databases
What is new in SSIS 2008?
Improved and new task componentsNew Lookup Capabilities (caching enhanced)Data Profiling Task (data quality checking)C# Scripting. VSA replaced by VSTAScalable Data Flow (thread pooling)
Enhanced and new connectivity supportADO.NET Source and DestinationNew connectors for Oracle, Teradata and SAP BI
SSIS Connectivity ComponentsSSIS Package
Data Flow Task
FTP FLATFILE ADO.NETOLEDB
FTP ExecuteSQL Task FlatFileLookup ADO.NET
ConnectionManagerTask Data Source/
DestinationDataTransformation
Note: Execute SQL Task can also make use of ADO.Net, ODBC Connection Managers
Component InteractionSSIS
Data Storage
AcquireConnection()
Connection Object
Write() Read()
Taskor
Component
Connection Manager
Data Flow Connectivity OptionsData Flow Task
OLEDBConnector
ADO.NETConnector
CustomConnector
OLEDBProvider
NativeADO.NETAdapter
ADO.NETODBC Data Source
SpecificInterface/Protocol
Data Source Specific Interface/ Protocol
ODBC
Data Storage
Data Access APIs and SSIS
"When developing in BIDS, 32-bit data providers are always used at design time, if 64-bit providers will be used in production packages, the equivalent 32-bit provider must be available at design time."
API Pros Cons
OLEDB • Comprehensive support with SSIS • 64-bit OLE DB providers less prevalent.
ADO.NET• Good support with SSIS• Direct support with ADO.Net Source & ADO.Net Destination
• Somewhat limited availability of out of the box ADO.NET providers
ODBC • ODBC drivers for possible data sources widely available
• Requires the use of a bridge - introduces performance and reliability issues
Custom-API • Designed for performance of native utilities
•Limited availability for most databases • Inconsistent support with SSIS - varies according to architecture
SSIS Enterprise Data Connectivity Versions Suggested Connector SSIS Source &
DestinationSSIS Connection Manager
SQL Server All SQL Data Access Components ADO.NETOLEDB
ADO.NETOLEDB
Oracle 9iR2, 10, 11 Microsoft Connector for Oracle Oracle MSORA
Teradata V2R6.0/1/2, 12 Microsoft Connector for Teradata Teradata MSTERA
SAP BI 7 Microsoft Connector for SAP BI SAP BI SAPBI
DB2 8.1, 9.x Microsoft OLEDB Provider for DB2 OLEDB OLEDB
SAP R/3 4.6c, 4.7, 5, 6 BizTalk Adapter 3.0 for mySAP ADO.NET ADO.NET
Siebel 7.5, 7.7, 7.8, 8 BizTalk Adapter 3.0 for Siebel ADO.NET ADO.NET
Connectivity CategoriesDatabase Systems (RDBMS)• Microsoft SQL Server • Oracle • Sybase • IBM DB2 • Teradata • MS Access
Queue Systems & Protocols• MSMQ (Message Queue Task) • (s)FTP (FTP Task) ** • HTTP/HTTPs (Web Service Task) • IBM MQ Series* • Tibco Rendezveus*
Unstructured ,Semi-structured Data• Microsoft Excel, CSV • Text • XML • EDI (Electronic Data Interchange)
Application Systems • Siebel • Microsoft Excel • Hyperion (Essbase, Planning,
Financial Management)• SAP R/3 & SAPBI
* not supported out-of-the-box, however, custom components can do this.** sFTP is not supported out-of-the-box
Microsoft Connector for SAP BI
Provider Microsoft
Version SAP NetWeaver BI Version 7
Pre-requisites SQL Server Enterprise LicenseSAP librfc32 librarySAP OpenHub license
Platform x86, x64, ia64
Microsoft Connector for SAP BIData Flow Task
SAP BIConnection Mgr
Remote Function Calls (RFC’s) *
SAP BI
SAP BIData Source
SAP BIData Destination
OpenHub BAPI
*standard programming interface used by SAP
SQL Server
Application Scenario 1Extracting data from SAP BI into SQL Server
SAP BI
OHSDestination
SSIS
SSIS Package
OLE dBDestination
SAP BISource
Ope
n H
ub
Info Cube
ProcessChainTriggered
SAP BIConnection
Manager
Microsoft Connector for SAP BI Extracting data from SAP BI to SQL Server
1. SAP BI Configuration1. Set up the RFC Destination2. Configure and create the Open Hub Destination3. Create the Data Transfer Process (DTP) and transformation4. Configure the process chain
2. SSIS Package1. Setup the SAP BI Source2. Setup the connection manager for SAP BI3. Setup the Destination (OLE DB, SQL Destination)4. Construct Workflow
3. Run the SSIS package
SAP BI
Stag
ing
BAPI
Application Scenario 2Loading non-SAP data into SAP BI
SSIS
SSIS Package
SAP BIConnection
Manager
OLE DBSource
SAP BIDestinationSQL
Server XML
FlatFile
InfoSource/InfoPackage
Microsoft Connector for Oracle
Provider AttunitySupported by Microsoft
Version Oracle 9iR2, 10, and 11
Pre-requisites SQL Server Enterprise License*Oracle client components
Platform SSIS: x86, x64, ia64
* 32 bit @design time, 64 bit @ runtime)
Microsoft Connector for OracleData Flow Task
OracleConnection Mgr
OCI
Oracle dB
OracleData Source
OCIArray Binding
OracleData Destination
OCIDirect Path
OCI
Performance of Oracle Connectors
Load Extract
27680
37362
1646
29756
Microsoft Connector for Oracle (by Attunity)Microsoft OLE DB Provider for Oracle
Tota
l Thr
ough
put (
Row
s/Se
c)
x16 x 1.25
Performance of Microsoft Connector for Oracle
Load Load - 10K Extract Extract - 10K
2901933747 36722 39466
26413
50158
8425491895
Mixed Types String Only
Tota
l Thr
ough
put (
Row
s/Se
c)
Microsoft Connector for OracleCustom Properties
Component Access Mode Property (Default)Oracle Source Table Name Batch Size (100)
Prefetch Count (0)SQLOracle Destination Table Name Batch Size (100)
Transaction SizeNo Logging (False)
Table Name – Fast Load Parallel (False)No Logging (False)Buffer Size (64KB)
Microsoft Connector for Teradata
Provider AttunitySupported by Microsoft
Version Teradata V2R6.0/1/2, 12
Pre-requisites SQL Server Enterprise LicenseTeradata ODBC driverTPT 12 for high-performance
Platform x86, x64
Microsoft Connector for TeradataData Flow Task
TeradataConnection Mgr
TPT12
Teradata DW
TeradataData Source
TeradataData Destination
TPTFastLoad
TPTTPUMP
TPTFast
Export
Microsoft Connector for TeradataCustom Properties
Component Access Mode Property (Default)Teradata Source Table Name – TPT Export BlockSize (64330)
DataEncryption (False)Max Sessions (1)Min Sessions (1)TenacityHours(4)TenacitySleep(6)
SQL Command – TPT Export
Teradata Destination Table Name – TPT Stream BlockSize (64330)DataEncryption(False)MaxErrors (0)MaxSessions (1)MinSessions(1)TenacityHours(4)TenacitySleep(6)
Table Name – TPT Load
Recap
SSIS connectivity Provides a robust connectivity architectureProvides an extensible connectivity architectureProvides rich connectivity options to enterprise data via :
Built-in connectorsAdditional connectors (available from Microsoft downloads)*Third party connectors
Rich ecosystem for SSIS connectors
*[codeplex or custom dev]
www.microsoft.com/teched
International Content & Community
http://microsoft.com/technet
Resources for IT Professionals
http://microsoft.com/msdn
Resources for Developers
www.microsoft.com/learning
Microsoft Certification & Training Resources
Resources Tech·Ed Africa 2009 sessions will be made available for download the week after the event from: www.tech-ed.co.za
SSIS Community ResourcesSSIS Community wiki
http://wiki.sqlis.com/default.aspxSSIS Codeplex
http://sqlsrvintegrationsrv.codeplex.com/SSIS Junkies :
http://blogs.conchango.com/jamiethomson/Attunity Support Forums :
http://www.attunity.com/forums/microsoft-ssis-oracle-connectorhttp://www.attunity.com/forums/micorosft-ssis-teradata-connector
Additional Community ResourcesTechNet Community for IT Professionalshttp://technet.microsoft.com/en-us/sqlserver/bb671048.aspxDeveloper Center http://msdn.microsoft.com/en-us/sqlserver/bb671064.aspxSQL Server 2008 Learning Portalhttp://www.microsoft.com/learning/sql/2008/default.mspx
Additional Resources
External ResourcesSSIS ETL Record – Loaded 1TB in 30 Minutes! http://msdn.microsoft.com/en-us/library/dd537533.aspxConnectivity Options for Microsoft SQL Server 2008 Integration Serviceshttp://msdn.microsoft.com/en-us/library/dd299429.aspxMicrosoft Connectors for Oracle and Teradata by Attunity http://www.microsoft.com/downloads/details.aspx?FamilyId=D9CB21FE-32E9-4D34-A381-6F9231D84F1EUsing SQL Server 2008 Integration Services with SAP BI 7.0
http://msdn.microsoft.com/en-us/library/dd299430.aspx
SQL Server 2008 Business Value Calculator: www.moresqlserver.com
• Team Forum: http://social.msdn.microsoft.com/forums/en-US/sqlintegrationservices/threads/
© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS,
IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.