SQL Server Integration ServicesMonitoring Deep DiveDAVIDE MAURIINFO@DAVIDEMAURI.IT - @MAURIDB HTTP://SPEAKERSCORE.COM/D46B

About Me Microsoft Data Platform MVP since 2006 Works with SQL Server from 6.5, on BI from 2003 Specialized in Data Solution Architecture, Database Design, Performance Tuning, High-Performance Data Warehousing, BI, Big Data

Very strong developer background Loves community!

◦ President of UGISS (Italian SQL Server UG) for 11 Years◦ AppDev VG Leader since 2017

Regular Speaker @ SQL Server events Director SW & Cloud @ Sensoria: http://www.sensoriafitness.com/ E-mail: info@davidemauri.it -Twitter: @mauridb Blog: http://sqlblog.com/blogs/davide_mauri/default.aspx

Agenda Deep Dive into SSISDB Log-Related Catalog Logging Levels Building a Dashboard

◦ Native Solutions◦ 3rd Community Party Solutions◦ My Solutions:

◦ HTML 5◦ Power BI

Notice to SSIS Men Everything said here works for SQL Server 2012 and over and applies to the Project Deployment Model *only*

If you’re sill using the Package Deployment model…

…change it! It’s OLD!◦ And even O.L.D.: Omitted Log Data

SSISDB – SSIS Catalog Contains *everything* needed to run, manage and monitor SSIS Packages◦ (only when using the “Project Deployment Model”)

“internal” schema := private API/Objects, for MS use only

“catalog” schema := public, supported, API/Objects

SSIS Server Package Execution Each time a Package is executed:

◦ Create Package Execution “context”◦ Set Execution Parameters Values◦ Execute the Package

SSIS Server Package Execution Logging

Logs and Execution Statistics are written to several tables◦ catalog.executable◦ catalog.executable_statistic◦ catalog.event_messages

◦ catalog.operations, catalog.operation_messages

The execution_id binds everything together

Named “operation_id” sometimes because event_messages is an extension of operation_messages

SSIS Server Package Logging Levels

Logging is automatically performed by the server◦ None: Logging is turned off. Only the package execution status is logged.◦ Basic: (Default) All events are logged, except custom and diagnostic events.◦ RuntimeLineage: Collects the data required to track lineage information in the

data flow. (2016 only)◦ Performance: Only performance statistics, and OnError and OnWarning

events, are logged.◦ Verbose: All events are logged, including custom and diagnostic events,

including the DiagnosticEx event.

Of course, logging level can be changed at execution-time, no special design patterns needed

Reference Solution Used for Tests

Master, Child, GrandChild case Master and Child have objects with same name Master and Child have loops Master and Child have parallel executions Package and Projects connection managers have the same name Package and Project Parameters Scripts (Dts.Events.FireInformation) logging via Script Task DataFlows

DemoTHE REFERENCE SOLUTION

Before Execution Where data is stored in the SSISDB?

Especially interesting the parameter and connection managers configured and default values◦ They will be useful to have the complete execution context when doing log

analysis

Demo

Package Execution

Created (1) Pending(5)

Running(2)

Stopping(8) Canceled (3)

Success(7)

Completed(9)

Failed(4)

Unexpected Termination /

Crash (6)

catalog.start_execution

catalog.create_execution catalog.stop_operation

ISServerExec Unexpected Crash

Logging “None” Really means “Minimal”

No logged events, but still some data is logged in some “basic” log tables :◦ catalog.executions◦ catalog.execution_property_override_values◦ catalog.execution_parameter_values◦ catalog.executables◦ catalog.executable_statistics

◦ Log-Like table

Logging “Basic” Default Option

Data Available in “basic” log tables and◦ catalog.event_messages

Logged Events:◦ OnError, OnInformation, OnPostExecute, OnPostValidate, OnPreExecute,

OnPreValidate, OnWarning

No specific information on related to DataFlow

Logging “Performance” Data Available in “basic” log tables and

◦ catalog.event_messages

Logged Events:◦ OnError, OnWarning

Additional specific information on component “phases”◦ catalog.execution_component_phases

No specific information on related to DataFlow

Logging “RuntimeLineage” “Collects the data required to track lineage information in the data flow. You can parse this lineage information to map the lineage relationship between tasks. ISVs and developers can build custom lineage mapping tools with this information.”◦ No much more information…◦ Available in SSIS 2016

Logging “Verbose” Like Basic and Perfomance together PLUS DataFlow details

◦ DataFlow is logged in detail to catalog.execution_data_statistics

Logged Events◦ All!◦ Even Diagnostic and Custom log messages (Dts.Events.FireCustom)

Logging in event_messages is something like 100X more verbose◦ Hence the name

Logging “Custom” Available only in SSIS 2016

You can decide your own logging level…like◦ Performance +◦ OnInformation +◦ Data Flow

Demo

Something to keep in mind Dataflow logs some interesting data as a text message only

◦ Lookup Duplicate Rows warning◦ Memory Warning

◦ Memory Allocation◦ Low Virtual Memory

◦ Rows sent to each destination

Verbose is just to verbose, but we still want some DataFlow logging, so we can dig into the text into event_messages table

Demo

Building a Dashboard - Native

• Natively Available• Basic User Experience• “It just works”• Not really a dashboard

3Rd Party Community Project SSIS Reporting Pack (by Jamie Thomson)

◦ No more maintained (AFAIK)◦ Provides more than reporting

◦ usp_ssiscatalog

http://ssisreportingpack.codeplex.com/

HTML5 Based on

◦ HTML 5 / CSS 3◦ Bootstrap◦ jQuery, Morris.Js, SmartMenu◦ Python + Flask

Code on GitHub◦ https://github.com/yorek/ssis-queries ◦ https://github.com/yorek/ssis-dashboard

Working Demo:◦ http://ssis-dashboard.azurewebsites.net/

HTML5

HTML5 An forked version has been created by @niphlod here

◦ https://github.com/niphlod/ssis_dash

Still based on Python◦ But moved from Flask To Web2Py◦ Supports security, authentication and authorization◦ Can monitor more than one SSISDB instance◦ …much more

Very cool!

PowerBI Using PowerBI Desktop it is really easy to create a nice SSIS Dashboard

If you don’t have problems sending data to the Azure cloud, you can also move the Dashboard to the PowerBI Service◦ And use SSIS or Azure Data Factory or PowerBI Data Gateway to keep data in

Azure updated

One last note Due to how security is applied in SSIS, if you’re not sysadmin or ssis_admin queries can be *really* *really* slow◦ Each row is under custom row security, using certificates◦ *A lot of certificates* needs to be open and used: One for each Execution!!!!◦ SQL Server 2012 SP2 fix this by allowing you to do it on project-basis instead

of Execition-basis◦ https://support.microsoft.com/en-us/kb/2972285

A *unsupported* workaround is to fix the [internal].[current_user_object_permissions] procedure, avoiding the usage of the scalar function [get_principal_id_by_sid], use a JOIN instead

Thanks!QUESTIONS?

Demos and Tools available on GitHubHTTPS://GITHUB.COM/YOREK/SQLSAT613HTTPS://GITHUB.COM/YOREK/SSIS-DASHBOARDHTTPS://GITHUB.COM/YOREK/SSIS-QUERIES