Deep Inside the Microsoft SQL Server Integration Services Server

Matt MassonMatthew RocheMicrosoft Corporation

DBI405

Best Practices

SQL Server 2012 Integration Services Server Best Practitioners

Matt MassonMatthew RocheMicrosoft Corporation

DBI405

Not actually a Session AgendaProject and Package DeploymentProject deployment model – what, how, and whyDeploying single packages – can it be done? Should it be done?

Logging and MonitoringSSIS Server built-in capabilitiesUsing the SSIS Server API3rd party and community options

Troubleshooting and DebuggingArchitectureTools and techniques

SSIS Server Lightning Review

• The SSIS Server is a set of components that which include a SQL Server user database (SSISDB), an execution host process (ISServerExec.exe) and the tools and APIs to manage and control them.

• The SSIS Server is the target of deployment for SSIS projects when the Project Deployment Mode is used.

• The SSIS Server (and the Project Deployment Mode) is an optional part of SSIS in SQL Server 2012, but key development capabilities (such as parameters and project-level shared connection managers) require its use.

What is the SSIS Server?

SSIS CatalogConfiguration

Set values for parameters

Central connection manager configuration

Advanced property override functionality

Security

Encryption of projects and parameter values

Row-level security to control access to packages

Management

Interactive package execution and SQL Agent integration

Dashboard and built in reports for troubleshooting

Diving into the SSIS Server

SQ

L S

erv

er

Insta

nce SSIS Server

SSIS Catalog

Deploy Manage Security Validate Execute

IS Objects

Security

Operation logs

State

Execution Process

Execution Control

Components

PowerShell SSMS Deployment Wizard ApplicationOM:

ManageOM:

ManageOM:

ManageOM:

ProjectOM:

ManageOM:

Project

SSIS Server Architecture

Runtime

The SSIS Catalog: SSISDB

SSISDB

• Provides a set of stored procedures and views for managing, configuring, executing and monitoring SSIS packages deployed to the SSIS Catalog

• The Catalog schema is designed for customer use, and is a supported API• Contains views, stored procedures, and functions

• The Internal schema is neither public nor supported• Contains base tables and lower-level objects

• All projects and sensitive values are encrypted • Protected by a pair of certificate and symmetric

key• Utilizes the built-in SQL Server encryption

infrastructure• http://

blogs.msdn.com/b/mattm/archive/2012/03/23/ssis-catalog-backup-and-restore.aspx

Important: Backup the database master key!

Whatever you dousing SSMS can be scriptedeither using T-SQL or PowerShell

Tables

Views

SSISDB

Stored Procedures

Stored Procedures

SQLCLR Assembly

ISServerExec.exe

Microsoft.SqlServer.IntegrationServices.Server. Shared.dll

Functions

• 42 store procedures• 25 views• 3 functions• 1 trigger• 1 SQL CLR assembly

Public objects:

Internal CatalogTriggers

SSISDB Overview

Client SQL Server instance

Entry point: T-SQL sp

InvokedCLRsp

Return success/failThrow if errors occur

External ProcessISServerExec

Client SQL Server instance

InvokedCLR SP

Return success/failThrow if errors

occur

Create process by impersonating

caller of the stored proc

Either Asynchronous or Synchronous

SSISDB: Usage Patterns

T-SQL sp (entry point)T-SQL sp invokes managed sp

T-SQL sp (entry point)T-SQL sp invokes managed spManaged sp creates external process by impersonating caller of sp

Client SQL Server instance

T-SQL sp

Return success/failThrow if errors occur

T-SQL spCASE A CASE B CASE C

Entry point: T-SQL sp

SSISDB - Protecting Sensitive DataGuiding Design Principle: No sensitive data should be stored in plain text in the server

Service Master Key

Database Master key

Project CertificateEnvironment Certificate

Execution Certificate

Project Symmetric key

Environment Symmetric key

Execution Symmetric key

Project Parameter

values

Environment values

Execution Parameter

values

Supported 'TRIPLE_DES_3KEY', 'AES_128', 'AES_192', 'AES_256'T-SQL functions: EncryptByKey; DecryptByKeyNote: We do not support sensitive data with length > 8000

SET @key_name = 'MS_Enckey_Proj_'+CONVERT(varchar,@project_id) SET @certificate_name = 'MS_Cert_Proj_'+CONVERT(varchar,@project_id)OPEN SYMMETRIC KEY key_name DECRYPTION BY CERTIFICATE certificate_nameSELECT parameter_name, DECRYPTBYKEY([sensitive_parameter_value]) FROM internal.[object_parameter_values]WHERE [project_id] = @project_idCLOSE SYMMETRIC KEY key_name

Note: The Catalog Views will automatically decrypt the values for you.

Projects and Deployment

Oh, the good old days……said nobody, ever.

Project and Package DeploymentPackage Deployment – The Bad Old DaysDesign time vs. deployment time – what is a project?Deployment locations and their design implications (production pain prevention predicated on palliative planning prior to package placement!)Does anyone remember the poison apple?

Project Deployment – The Glorious FutureDesign time and deployment time – what a project is!An ispac for your headacheA consistent experience between design time, deployment and run time

Projects & Parameters

Groups of packages (anywhere) Projects

SalesHR

HR DW project

SAP migration project

Configurations

<xml>x\y\MyTask\Server = “TestServer”

x y

Parameters

HR DW project

ServerName is String BatchNumber is Int32

CLR Cryptography

• When a project is deployed, the project .ispac file is stored as binary data in SSISDB

• The .ispac binary is also encrypted on the server, using SQLCLR and System.Security.Cryptography

SSISDB - Protecting Project Data

Service Master Key

Database Master key

Project Certificate

Project Symmetric key

key

internal. catalog_encryption_keys

TripleDESCryptoServiceProvider

AesCryptoServiceProvider(192)

AesCryptoServiceProvider(128)

AesCryptoServiceProvider(256)

Project binary

Project Deployment and the SSIS ServerWhat is an ispac file, anyway?The output of the project build processThe complete contents of the SSIS project: packages, parameters, and connections, oh my!The component that gets deployed when you deploy an SSIS project

What happens when I deploy a project?What DOESN’T happen?!But seriously folks… …let’s take a look at a demo!

Demo: Deploying an SSIS Project

Project Deployment and the SSIS ServerWhat is an ispac file, anyway?The complete contents of the SSIS project: packages, parameters, and connections, oh my!OpenXML (zip) document

What happens when I deploy a project?

Is There a Happy Medium?What about single-package deployment?

EXECUTE AS CALLEROPEN SYMMETRIC KEYWAITFOR DELAY

[internal].[encrypt_binarydata][internal].[deploy_project_internal][internal].[append_packages]

[internal].[projects][internal].[object_versions][internal].[packages]

Branching and Deployment

Development

Release

Integration

Multiple code branches.Regular integration from Development -> Integration -> Release.

All deployments come from a branch build.Never deploy “one off” changes.Deployment can be automated.

Test

Production

Server Execution and Logging

Oh, the good old days……wept everyone, always.

SSISDB and ISServerExec

SSISDB

• ISServerExec.exe – External host for SSIS package operations (deploy, validate and execute)

• ISServer Assembly• UNSAFE Assembly granted to

##MS_SQLEnableSystemAssemblyLoadingUser##

• Created from Microsoft.SqlServer.IntegrationServices.Server.dll

• SQLCLR stored procedures used for• Deploy, validate, and execute require impersonation when

starting external process• IPC communication with ISServerExecISServerExec

Processes

ISServerExec Overview

SSISDB

Logging EventsWrites back to SSISDB events that are produced during package execution

ISServerExec

SSIS Events

SqlConnection

Named Pipes

Named Pipe Server

IPC between ISServerExec and Stored Procedures• CLR stored procedure sends command to ISServerExec

Examples• Get me all the per-instance performance counters• Stop Operation• Create execution dump

• ISServerExec • Performs the operation• Sends back information via named pipes

SSIS Server Package Execution

SSISDB

EXEC [SSISDB].[catalog].[create_execution]…EXEC [SSISDB].[catalog].[set_execution_parameter_value] @execution_id, …

EXEC [SSISDB].[catalog].[set_execution_parameter_value] @execution_id, …

EXEC [SSISDB].[catalog].[start_execution] @execution_id

ISServerExec

Named Pipe Server

SSIS Events

catalog.executablescatalog.executable_statistics

Updated when the OnPostExecute eventfor each component if fired

Review: SSIS Server Execution Architecture

OM/Engine

ISServerExec

Named Pipe

IDTSEvents

ISServerExec.exe

Events Listener

ADO.Net

SqlServr.exe

Tables

Components

Views

Reports

TVFs

SSISDB

Log Provider

IDTSLogging

CreateProcessAsUser

API

Monitoring

• Logging is automatically performed by the serverNo specific design patterns must be followed by the package developerData is stored in the SSISDB catalog, and is available for reporting and analysis

• NoneLogging is turned off. Only the package execution status is logged.

• Basic(Default) All events are logged, except custom and diagnostic events.

• PerformanceOnly performance statistics, and OnError and OnWarning events, are logged.Enables use of Execution Performance report and catalog.execution_component_phases view

• VerboseAll events are logged, including custom and diagnostic events, including the DiagnosticEx event.Enables use of catalog.execution_data_statistics view.

SSIS Package Execution & Logging Levels

Built-In Reporting and LoggingBuilt-In Reports

Validate Pre Execute ProcessInput ProcessInput Post Execute

SELECT package_name, task_name, subcomponent_name, SUM(DATEDIFF(ms,start_time,end_time)) as active_time,DATEDIFF(ms,min(start_time),max(end_time)) as total_timeFROM catalog.execution_component_phasesWHERE execution_id = 1841GROUP BY package_name, task_name, subcomponent_name, execution_pathORDER BY package_name, task_name, subcomponent_name, execution_path 

Component Timing & Row Counts

Custom Reporting and LoggingAll Catalog logging exposed in ViewsSome features (like real-time perf counters) only available while the package is running

Common pattern in previous versionsCapture events using Event Handlers or custom logging frameworkYou can link custom logging with Catalog logging with $User::ServerExecutionID system variable

Community Reporting Toolshttp://ssisreportingpack.codeplex.com and sp_ssiscataloghttp://www.mattmasson.com/2013/04/monitoring-ssis-package-executions/ (many links!)

3rd Party Commercial Reporting ToolsPragmatic Works - BI xPress Auditing Framework Wizard

Troubleshooting

Oh, the good old days……wept Matthew, over drinks.

• New in SSIS 2012 (for Verbose Logging Level)

• Captures diagnostic information whenever an Execute Package Task executes a child package

DiagnosticEx Event

Flattening the DiagnosticEx Event Data

WITH DiagnosticExTable(EventMessageID, EventName, MessageSourceName, XmlData)AS ( SELECT event_message_id,event_name,message_source_name,cast( message as xml) FROM catalog.event_messages m WHERE m.operation_id = 16 AND m.event_name = 'DiagnosticEx') SELECT EventMessageID,Eventname,MessageSourceName, parameter.value('declare namespace DTS=''www.microsoft.com/SqlServer/Dts''; (@DTS:ObjectName)[1]','nvarchar(260)') as ParameterName,parameter.value( 'declare namespace DTS=''www.microsoft.com/SqlServer/Dts''; (DTS:Property/text())[1]', 'nvarchar(256)') as ParameterValue FROM DiagnosticExTable CROSS APPLY XmlData.nodes('declare namespace DTS=''ww.microsoft.com/SqlServer/Dts''; (/DTS:ParameterValues/DTS:PackageParameter)') as PackageParameter(parameter)

Per-Instance Performance CountersThe SSISDB T-SQL API includes a function to return performance counters for SSIS package executions

SSISDB

SELECT *FROM catalog.dm_execution_performance_counters(<your execution ID value>)

ISServerExec

SSIS Events

Named Pipe Server

What are the values for the Perf Counters?

Perf Details

Results

ISServerExec

Creating Execution Dumps• Stored procedure to cause a running SSIS package to pause and create a

dump file• Dump file stored in …\Program Files\Microsoft SQL Server\110\Shared\

ErrorDumps• Similar to dtutil.exe /Dump

EXEC catalog.create_execution_dump @execution_id = 88

SSISDB

SSIS Events

Named Pipe Server

Create execution Dump DumpFile

Pause running packageDumpResume running package

Data Taps – Data Viewers on the Server

Data Tap Files

-- Create the data tap on a data flow path in the packageexec catalog.create_execution …

exec catalog.add_data_tap @execution_id, '\Package\DFT Load Dim Vendor','Paths[SRC DimDCVendor.OLE DB Source

Output]', 'DCVendorOutput.csv'

exec catalog.start_execution @execution_id …

• Data Taps are essentially server-side data viewers

• Created using one of two stored procedures• [catalog].[add_data_tap]: only for parent packages• [catalog].[add_data_tap_by_guid]: for both parent and child packages

• Data Taps create CSV outputs• Tap files are put under %DTS%\DataDumps folder• Tap files contain all data that passes through the specified data flow path

• Records are created in [catalog].[operation_messages]

• When a data tap file is created• If the specified data flow path is invalid• If the specified data flow task is never executed• If the data tap file creation fails

Creating Data Taps

Closing

Project Deployment ModelParameters. Reusable connection managers. Automatic logging, monitoring and reporting. Relative references for child execution. A real deployment utility. Parameters!

Remote ExecutionISServerExec.exe runs on the server where the package is deployed, not on the client where the execution is initiated. Boo yah!

T-SQL APIScript it from within SSMS. Save it to .SQL script files. Edit and customize. Store and version control with other system artifacts. Execute from any SQL-aware client. Smile, sit back, and enjoy a cold beverage.

SSIS Catalog DataExecution and operation data automatically logged. Built in reports. Community reports and stored procedures for ease of access. Opportunity for PowerPivot and Power View models and cross-catalog consolidation. Let’s overload the term “metadata” once and for all, because…

Matthew’s Favorite Bits of the SSIS Server

Related contentBreakout SessionsDBI-B210 BI Power Hour: Wednesday 1:30 PM, New Orleans Theater C

Related Certification ExamExam 70-463 - Implementing a Data Warehouse with Microsoft SQL Server 2012

Find Us Later At...The Data Platform booth in the Expo Hall

Track Resources

@sqlserver

mvaMicrosoft Virtual Academy

SQL Server Website

Get Certified!

Hands-On Labs

Download Data Explorer

Download Geoflow

Windows Azure

msdn

Resources for Developers

http://microsoft.com/msdn

Learning

Microsoft Certification & Training Resources

www.microsoft.com/learning

TechNet

Resources

Sessions on Demand

http://channel9.msdn.com/Events/TechEd

Resources for IT Professionals

http://microsoft.com/technet

Complete an evaluation on CommNet and enter to win!

Evaluate this session

Scan this QR code to evaluate this session and be automatically entered in a drawing to win a prize

© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Appendix

IPC between SSISDB and ISServerExec• Named Pipe: ISServerExec_{ExecutionGuid}

• Stopping execution needs, to issue a command to ISServerExec• Creating dynamic dump needs to issue a command and get file name back• Querying performance data needs to request a list of key-value pair

SSISDB(1) Named pipe

client

ISServerExec.exe

(2) Named pipe server

(3) pipe

(5) Stop executionCreate dump

Query perf data

Perf data

(4) Listen to the command(6) Perform actionYou can use

pipelist.exe to check the named pipe used

SSIS Package Execution Lifecycle

Created (1)

Pending(5)

Running(2)

Stopping(8)

Canceled (3)

Success(7)

Completed(9)

Failed(4)

Unexpected

Termination / Crash

(6)

catalog.start_execution

catalog.create_execution catalog.stop_operation

ISServerExec Unexpected Crash

• Asymmetric Key• MS_SQLEnableSystemAssemblyKey

• Logins• ##MS_SQLEnableSystemAssemblyLoadingUser##• ##MS_SSISServerCleanupJobLogin##

• SQL Server Agent Jobs• SSIS Server Maintenance Job – Cleans up execution log data

outside retention window and project versions beyond configured limit

• master Stored Procedure• dbo.sp_ssis_startup – Redirects to [SSISDB].[catalog].[startup]• Cleans up orphaned operation status for unexpected shutdown

SSIS Server Objects: What’s not in SSISDB?

SSISDB

• To debug ISServerExec.exe• Create a key named “ISServer” under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SQL Server\110\SSIS\WaitOnStartup\ISServer• add a DWORD value with name "*" or the project id.• ISServerExec will pause at startup so you can attach a debugger

Debugging Server Package Execution

SSIS Server Security

SSIS Server Security Overview v1

• Provides row-level security for SSIS Securables (Folder, Project, Environment, Operations/Executions)

• Members of ssis_admin or sysadmin access all securables

• Views• catalog.explicit_object_permissions

Permissions explicitly assigned to the userhttp://msdn.microsoft.com/en-us/library/ff878037

• catalog.effective_object_permissions Effective permissions for the current principal for all objectshttp://msdn.microsoft.com/en-us/library/ff878149.aspx

SSIS Server Security Overview v2

SSISDB Securables and Permissions

Securable

Permissions

Read Modify Execute Manage Permission

Create New

Read Objects

Modify Objects

Execute Objects

Manage Objects

Permission

Folder ⦁ ⦁   ⦁ ⦁ ⦁ ⦁ ⦁ ⦁Project ⦁ ⦁ ⦁ ⦁          

Environment ⦁ ⦁   ⦁          

Operation / Execution ⦁ ⦁   ⦁          

Troubleshooting Permissions• Check explicit (raw) permission entries from view [catalog].[explicit_object_permission]• Effective (valid) permission is a computation result from explicit permission items:

[catalog].[effective_object_permissions]• The server does not check permissions if a user is sysadmin/ ssis_admin

• Each view presents a filtered rowset from a base table• Each user can see only the rows for which he has

READ permission• sysadmin / ssis_admin can see all rows in all views

• Based on best practices• Implementation based on pattern documented in

“Implementing Row- and Cell-Level Security in Classified Databases” white paper

• http://technet.microsoft.com/en-us/library/cc966395.aspx

• Troubleshooting Row-Level Security• If a user can’t see some object

1. Logon as ssis_admin to see if it’s in base table2. Get the user sid from sys.database_principals3. Query view

catalog.effective_object_permissions to make sure the sid has READ permission on the record

SSISDB Views - Row-level Security