Click here to load reader

SQL Server Integration Services (SSIS) Step by Step Tutorial

  • View
    1

  • Download
    0

Embed Size (px)

Text of SQL Server Integration Services (SSIS) Step by Step Tutorial

SQL Server Integration Services (SSIS) – Step by Step TutorialA SSIS eBook from Karthikeyan Anbarasan, www.f5Debug.net
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 1
I dedicate this eBook to my Parents and my Wife, who make it all worthwhile.
— Karthikeyan Anbarasan(Karthik) www.f5debug.net
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 2
ABOUT THE AUTHOR
Karthikeyan Anbarasan (Karthik) has more than 5 years’ experience on Microsoft
Technologies (ASP.Net, C#.net, VB.Net, ADO.Net, Ajax, SQL Server, SSIS, SSRS, SSAS,
BizTalk Server, IBM MQ Server, WCF, WPF and some tools like Infragisitcs, Sync
fusion, etc..)
He is the founder of www.f5debug.net & Azuretutorials.in also he has written over
180 articles on many topics including SSIS, SQL Azure and Microsoft .Net. He
currently holds the MVP (Most Valuable Professional) Award from Mindcracker and
Dotnetfunda Online Community sites.
He works for a Multinational Company in Chennai as an Analyst where his primary
role starts with Design, Development, Testing, Production support and collaboration
with onsite team on various activities.
Karthik in his free time used to play cricket and watch Action Movies. Below is the list
of the Certification completed by Karthik.
Certifications:
Microsoft Certified Technology Specialist (Win and Web)
Microsoft Certified Technology Specialist (BizTalk Server 2006 R2) w w w .f5 de bu g. ne t
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 3
ACKNOWLEDGEMENT
I would like to express my heartful thanks to Mahesh Chand (Founder of Mindcracker
Networks) and Pinal Dave (Founder of Blogs.SQLAuthority.com), for constant
motivation in publishing this first eBook of mine.
Thanks to Bharath Radhekrishna.Chennu for compiling different articles of mine to
this eBook and to Sheo Narayan for his design inputs.
I should also mention about my website www.f5debug.net, which has always inspired
me to write more on .NET and related technologies.
A lot of thanks to my wife Janani, for all her support and encouragement. Without
her it would have been impossible in accomplishing this.
DISCLAIMER
The publisher and the author make no representations or warranties with respect to
the accuracy or completeness of the contents of this eBook. The strategies contained
herein may not be suitable for every situation. Neither the publisher nor the author
shall be liable for damages arising here from. Further, readers should be aware that
Internet Web sites listed in this work may have changed or disappeared between
when this work was written and when it is read.
w w w .f5 de bu g. ne t
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 4
CONTENTS AT A GLANCE
Chapter 3 Export Data using Wizard
Chapter 4 Import Data using Wizard
Chapter 5 Building and Executing a Package
Chapter 6 Options to execute a package
Chapter 7 Options to deploy a package
Chapter 8 Scripting in SSIS Packages
Chapter 9 Breakpoints in SSIS Packages
Chapter 10 Check Points in SSIS Packages
Chapter 11 Send Mail in SSIS Packages
Chapter 12 For Loop task
Chapter 13 Backup Database task and Send Mail
Chapter 14 Folder Structure
Chapter 17 Create/Delete a table in SQL
Chapter 18 Bulk Insert task
Chapter 19 ActiveX Script task container
Chapter 20 Executing package from Stored Procedure
Chapter 21 FTP Task Operations in SSIS Package
w w w .f5 de bu g. ne t
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 5
Chapter 22 Receive File using FTP Task in SSIS Package
Chapter 23 Send File using FTP Task in SSIS Package
Chapter 24 Delete Remote File using FTP Task in SSIS Package
Chapter 25 Delete local file using FTP Task in SSIS Package
Chapter 26 Delete remote folder using FTP Task in SSIS Package
Chapter 27 Delete local folder using FTP Task in SSIS Package
Chapter 28 Create remote folder using FTP Task in SSIS Package
Chapter 29 Create local folder using FTP Task in SSIS Package
Chapter 30 Data Flow Transformation
Chapter 31 Aggregate (Average) Transformation Control
Chapter 32 Aggregate (Group By) Transformation Control
Chapter 33 Aggregate (SUM) Transformation Control
Chapter 34 Aggregate (COUNT) Transformation Control
Chapter 35 Aggregate (COUNT DISTINCT) Transformation Control
Chapter 36 Aggregate (MAXIMUM) Transformation Control
Chapter 37 Aggregate (MINIMUM) Transformation Control
Chapter 38 Audit Transformation Control
Chapter 39 Character Map (Upper to Lower) Transformation
Chapter 40 Character Map (Lower to Upper) Transformation
Chapter 41 Copy Column Transformation
Chapter 42 Data Conversion Transformation
Chapter 43 Derived Column Transformations
Chapter 44 Export Column Transformation
w w w .f5 de bu g. ne t
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 6
Chapter 48 Lookup Transformation
Chapter 50 Merge Transformation
Chapter 52 Merge Join Transformation
Chapter 53 Multi Cast Transformation
Chapter 54 Transformation Categorized
Chapter 55 Connection Managers
Chapter 56 Data Viewers
Chapter 58 Data Viewers (Scatter Plot)
Chapter 59 Data Viewers (Column Chart)
Chapter 60 OLE DB Command Task
Chapter 61 Percentage Sampling Transformation (Selected Output)
Chapter 62 Percentage Sampling Transformation (Un-Selected Output)
Chapter 63 Percentage Sampling Transformation
Chapter 64 Row Sampling (Selected Output) Transformation
Chapter 65 Row Sampling (Un-Selected Output) Transformation
Chapter 66 Row Sampling Transformation
w w w .f5 de bu g. ne t
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 7
Introduction
In this chapter we will see what a SQL Server Integration Services (SSIS) is; a basic
on what SSIS is used for, how to create a SSIS Package and how to debug the same.
SSIS and DTS Overview
SSIS is an ETL tool (Extract, Transform and Load) which is very much needed for
Data warehousing applications. Also SSIS is used to perform the operations like
loading data based on the need, performing different transformations on the data
like doing calculations (Sum, Average, etc.) and to define workflow of the process
flow and perform some tasks on the day to day activities.
Prior to SSIS, Data Transformation Services (DTS) in SQL Server 2000 performs the
tasks with limited features. With the introduction of SSIS in SQL Server 2005 many
new features can be used. To develop your SSIS package you need to have SQL
Server Business Intelligence Development Studio installed, which will be available
as client tool when installing SQL Server Management Studio (SSMS).
SSMS and BIDS
SSMS provides different options to develop your SSIS package starting with Import
and Export wizard with which we can copy the data from one server to another or
from one data source to another. With these wizards we can create a structure on
how the data flow should happen and make a package and deploy it based on our
need to execute in any environment.
w w w .f5 de bu g. ne t
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 8
Business Intelligence Development Studio (BIDS) is a tool which can be used to
develop the SSIS packages. BIDS is available with SQL Server as an interface which
provides the developers to work on the work flow of the process that can be made
step by step. Once the BIDS is installed with the SQL Server installation we can
locate it and start our process as shown in the steps below.
Steps
We will take an example of importing data from a text file to the SQL Server
database using SSIS. Let us see the step by step process of how to achieve this task
using SSIS.
Step 1 – Go to Start Programs Microsoft SQL Server 2005 SQL Server
Business Intelligence Development Studio as shown in the below figure.
w w w .f5 de bu g. ne t
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 9
It will open the BIDS as shown in the screen below. This will be similar to the Visual
Studio IDE where we normally do the startup projects based on our requirements.
Step 2 – Once the BID studio is open, now we need to create a solution based on our
requirement. Since we are going to start with the integration services just move on to
File New Project or Ctrl + Shift + N It will open a pop up where we need to select
Integration Services Project and give the project name as shown in the screen below.
w w w .f5 de bu g. ne t
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 10
After creating this new project we can see that by default a new SSIS package is added
(Package.dtsx). When you create an Integration Services Project; you can right click on
it and rename it. Now the designer will show the empty workflow tabs as shown
below.
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 11
Step 3 – Since our task is to load the text file into the database, we need to create a
new data source. Right click on the connection manager pane and select “New Flat File
Connection” as shown in the screen below.
Step 4 – It will open a window as shown in the screen below. We need to fill in details
like the path of the text file and Format of the text file. If the first row of the text file
contains the columns, then we need to select the “Column names in the first data
row” check box as shown in the screen below.
w w w .f5 de bu g. ne t
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 12
Step 5 – You can see the columns which we are trying to import from the text file by
going to the columns tab or the preview tab in the screen below. In the preview tab
we can see the data as per our requirement. For example, say a huge amount of data
is available in the source with 1 million records. In the preview tab it shows only 100
records at a time. To see the records from 500 to 600 you need to specify “Data rows
to skip” = 500 in the preview tab. So we can see the records from 501 to 600. Click on
the ok button to complete this task.
w w w .f5 de bu g. ne t
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 13
Step 6 – Now we are done with defining the source section. We need to follow
similar steps for defining the destination section. Now right click on the connection
manager pane and select “New OLEDB Connection”. It will pop up a window to get
the connection details as shown below. Fill in all the connection details and test
the connection.
After clicking on Ok we will see the connection details in the connection manager
as shown in the screen below.
w w w .f5 de bu g. ne t
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 14
Step 7 – Now we need to create the data flow task in order to meet our
requirement to import the data from the text file into the database. SSIS uses the
work flow tasks in order to process the request as step by step process. Most of
our tasks (from coding to packaging) will be completed here. So in the tool box at
the left pane Click on the “Data Flow Task” item, and drag it to the screen. You’ll
see an icon as shown in the screen below. Double-click on that icon to open it.
w w w .f5 de bu g. ne t
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 15
Step 8 – Double click on the Data Flow Task, it will redirect to the Data Flow tab
where we need to have our logic for data flow as per our requirement. Since our
requirement is to import the text file, drag and drop the “Flat File Source” from
the tool box to the data flow tab screen. Again double click on the Flat File Source
task it will pop up a window which has the connection that we had set up at the
initial stage as shown in the screen below.
w w w .f5 de bu g. ne t
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 16
The “Retain null values from the source as null values in the data flow” is disabled
by default, which means that null values in the source defined will be treated as
empty string. You can click on the Preview button to confirm your structure.
Step 9 – Now we need to set the destination flow i.e. till now we read the data and
now we need to insert it to the database as per our requirement. Go to the tool
box and click on the data flow destination section and drag and drop the “OLE DB
Destination”. We need to map the source and the destination in order to tell SSIS
how the mapping works. So click on the Flat file Source and drag the green arrow
to the Ole DB destination as shown in the screen below.
w w w .f5 de bu g. ne t
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 17
Step 10 – Double click on the Destination task to configure it. Notice that this also
allows you to keep null values. In our example, we are going to create a new table.
But if you have a table already created, you can select it from the table drop-down
menu available. A table has been created in the database already for this example
with the structure below. This table name can be seen in the list as shown in the
screen below.
Step 11 -- We need to map the columns from the Source (Flat File Schema) to the
destination (DB Schema) as shown in the screen below.
w w w .f5 de bu g. ne t
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 18
Step 12 – Finally once everything is configured, click on Ok button. You should see
a screen similar to the one below.
w w w .f5 de bu g. ne t
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 19
Step 13 – Press F5 to start the execution of the SSIS package. We should see each
step turn green as the data is read and written. And you can see the data that has
been imported from the Text file to the Database.
Conclusion
In this chapter we have seen the basics of the SQL Server Integration Services with
the step by step process on how to create a basic integration services package
used in our day to day activity.
w w w .f5 de bu g. ne t
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 20
Introduction
In this chapter we will see how data can be transferred from SQL Server DB to
Excel sheet. This task is quite easy to perform and almost used in every SSIS
package that will be created.
Steps
Step 1 and Step 2 – Refer to the first chapter on the steps by step process to open
BIDS and select the correct project template to start developing the Integration
package.
Step 3 - Since our task is to transform the data from SQL Server DB to Excel sheet,
add a Data Flow task in the control task tab of package.dtsx as shown in the screen
below and double click on the task which will redirect to the Data Flow tab.
w w w .f5 de bu g. ne t
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 21
Step 4 - In Data Flow tab add an OLE DB Source task and configure it to the database where we need to perform the transformations and also select the table as shown in the screen below. In order to follow the steps on how to configure the SQL DB configuration check my previous chapter as stated above.
w w w .f5 de bu g. ne t
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 22
.
Step 5 - Now add the destination source (Excel Data source) as shown in the screen below and make the configuration. To do the configuration, first drag the green arrow from SQL data source to the Excel destination tasks and then double click on the excel destination task. It will open the screen below for configuration.
w w w .f5 de bu g. ne t
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 23
Step 6 - Do the mapping of the columns from both the source and the destination tasks as shown in the screen below.
w w w .f5 de bu g. ne t
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 24
Step 7 - Once everything is configured (Source and destination) press F5 to execute
the task and you can find the result at the path where we specified the excel sheet.
Check the download section to see the excel sheet which is created with this
project.
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 25
Conclusion
In this chapter we have seen the step by step process on how to do the transformation
of data from SQL Server database to the Excel sheet.
w w w .f5 de bu g. ne t
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 26
Introduction
In this chapter we will see how to export the data from SQL server using the wizard
which is provided with the SSMS (SQL server Management Studio). Using SSMS we can
perform many tasks like copying data from one server to another or from one data
source to another in variety of formats.
Here our task is to transform data from SQL server to Excel using the Wizard.
Steps
Let us see the step by step process on how to Export data using the inbuilt wizard with
the SQL Server Business Intelligence Studio
Step 1: Go to Programs Microsoft SQL Server 2005 SQL Server Management
Studio and connect to the list of server databases with which we have to perform the
task, as shown in the screen below.
w w w .f5 de bu g. ne t
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 27
Step 2: Once you locate the Database where we need to perform the transformation,
right click on the database, go to Tasks and select Export Data. It will open a welcome
screen. Click Next and move to the Data source tab.
w w w .f5 de bu g. ne t
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 28
Step 3 – ‘Choose a Data source tab’ helps to confirm the source of the data
transformation selected initially. Once the required configurations are selected, click
on next and it will ensure that select the destination source is selected.
Step 4: The destination data source window will open in which the destination needs
to be specified (In our example, the excel sheet). Select Microsoft Excel from the drop
down and provide the path in which it should save the excel sheet.
w w w .f5 de bu g. ne t
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 29
Step 5 - Now we need to specify the table from which we need to transform the data
or we can write our own query, based on which the data needs to be transformed. In
this example we will specify the table, so mark that option and click on next as shown
in the screen below.
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 30
Step 6 - Once we click on next button it will show the list of tables to be selected.
Select the table on which we need to do the transformation and click on preview to
verify the output as shown in the screen below and click on Next button.
w w w .f5 de bu g. ne t
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 31
Step 7: Once we are done with the source and destination it will ask to save and
execute the package. Click Next and Finish to complete the transformation as shown in
the screen below.
Sample Chapters
© Karthikeyan Anbarasan, www.f5Debug.net 32
Step 8: Once we are done it will show the process on the how…