Level Up Your Biml: Best Practices and Coding Techniques (NTK 2016)

Level Up Your Biml:Best Practices and Coding Techniques

Cathrine Wilhelmsen

Session Description

You already know how to use Biml to build a staging environment in an hour, so

let's dive straight into some of the more advanced features of Biml.

Attend this session for an overview of Biml best practices and coding techniques.

Learn how to centralize and reuse code with include files and the CallBimlScript

method. Make your code easier to read and write by utilizing LINQ (Language-

Integrated Queries). Share code between files by using Annotations and

ObjectTags. And finally, if standard Biml is not enough to solve your problems,

you can create your own C# helper classes and extension methods to implement

custom logic.

Start improving your code today and level up your Biml in no time!

Cathrine Wilhelmsen

@cathrinew

cathrinewilhelmsen.net

Data Warehouse Architect

Business Intelligence Developer

You…

Know basic Biml and BimlScript

Completed BimlScript.com lessons

Have created a staging environment

…?

Today…

Code Management

Practical Biml Programming

C# Classes and Methods

… :)

Quick Recap of

Basic Biml

What is Biml?

Business Intelligence Markup Language

Easy to read and write XML language

Describes business intelligence objects:

• Databases, Schemas, Tables, Views, Columns

• SSIS Packages

• SSAS Cubes

• Metadata

What do you need?

…or you can use the new Biml tools

How does it work?

dem

o t

ime!

Let's generate

some packages!

Ok, so we can go from Biml to SSIS…

…can we go from SSIS to Biml?

Yes! :)

dem

o t

ime!

Let's reverse-engineer

some packages!

The magic is in the BimlScript!

Extend Biml with C# or VB code blocks

Import database structure and metadata

Loop over tables and columns

Expressions replace static values

BimlScript allows you to control and manipulate Biml code

BimlScript Code Nuggets

<# … #> Control Nuggets (Control logic)

<#= … #> Text Nuggets (Returns string)

<#@ … #> Directives (Compiler instructions)

<#+ … #> Class Nuggets (Create C# classes)

How does it work?

Yes, but how does it work?

Yes, but how does it actually work?

<Biml xmlns="http://schemas.varigence.com/biml.xsd"><Packages>

<# foreach (var table in RootNode.Tables) { #><Package Name="Load_<#=table.Name#>"></Package>

<# } #></Packages>

</Biml> <Biml xmlns="http://schemas.varigence.com/biml.xsd"><Packages>

<Package Name="Load_Customer"/><Package Name="Load_Product"/><Package Name="Load_Sales"/>

</Packages></Biml>

Biml vs. BimlScript

Automate, control and

manipulate Biml with C#

Flat XML"Just text"

dem

o t

ime!

Let's generate

a lot of packages!

Code

Management

Don't Repeat Yourself

Move common code to separate files

Centralize and reuse in many projects

Update code once for all projects

1. Include files

2. CallBimlScript with Parameters

3. Tiered Biml files

BimlExpress vs. BimlOnline / BimlStudio

"Black Box"

Only SSIS packages visible

Visual Editors

All in-memory objects visible

Include Files

Include common code in multiple files and projects

Can include many file types: .biml .txt .sql .cs

Use the include directive

<#@ include file="CommonCode.biml" #>The directive will be replaced by the included file

Include pulls code from the included file into the main file

Works like an automated Copy & Paste

Include Files

Include Files

Include Files

CallBimlScript with Parameters

Works like a parameterized include

File to be called (callee) specifies input parameters it accepts

<#@ property name="Parameter" type="String" #>File that calls (caller) passes input parameters

<#=CallBimlScript("CommonCode.biml", Parameter)#>

CallBimlScript pushes parameters from the caller to the callee, and the

callee returns code






Tiered Biml Files

Split Biml code in multiple files and use the template directive:

<#@ template tier="1" #>

Create objects in-memory from lowest to highest tier to:

• Solve logical dependencies

• Simulate manual workflows

In-memory objects are added to the RootNode

Higher tiers can get objects added to RootNode in lower tiers

What is this RootNode?

The RootNode contains all in-memory objects:

• Connections, Databases, Schemas, Tables

• Projects, Packages

• Annotations, Metadata

Query the RootNode to loop over collections:<# foreach (var table in RootNode.Tables) { #>

Query the RootNode to get specific objects:<#=RootNode.Tables["Product"].Schema#>

Inside the Black Box: Tiered Biml Files

<#@ template tier="1" #><Connections>...</Connections>

<#@ template tier="2" #><Packages>...</Packages>

<#@ template tier="3" #><Package>...</Package>






































How do you use Tiered Biml files?

1. Create Biml files with specified tiers

2. Select all the tiered Biml files

3. Right-click and click Generate SSIS Packages

1

2

3

dem

o t

ime!

How does this

actually work?

Debugging Biml

Debugging Biml

BimlExpress is a "black box":• You can only see the generated SSIS packages

• It is not possible to see the compiled Biml first

Add a high-tier helper file to save compiled, flat Biml to file• Check Biml For Errors to save flat Biml without generating packages

SaveFlatBimlToFile.biml

Add the helper file to your project…

<#@ template tier="999" #><# System.IO.File.WriteAllText(

@"C:\Biml\FlatBiml.xml", RootNode.GetBiml()

); #>


…with a high tier so it is executed as the last step



); #>


It creates a file…



); #>


…at the specified path…



); #>


…with all the Biml for all the object in RootNode



); #>

How do you use this helper file?

1. Create the helper file

2. Select all the Biml files and the helper file

3. Right-click and click Check Biml For Errors

1

2

3

dem

o t

ime!

How is this

helper file used?

Annotations and

ObjectTags

Annotations and ObjectTags

Biml Annotations != SSIS Annotations

Annotations are string/string Key/Value pairs

ObjectTags are string/object Key/Value pairs

Use Annotations and ObjectTags to pass code

between Biml files

Annotations

Create annotations:<OleDbConnection Name="Destination" ConnectionString="…"><Annotations><Annotation Tag="Schema">AW2014</Annotation>

</Annotations></OleDbConnection>

Use annotations:<# var destinationSchema =

RootNode.OleDbConnections["Destination"].GetTag("Schema"); #>

ObjectTags

Create ObjectTags:<# RootNode.OleDbConnections["Destination"].ObjectTag["TableFilter"] = new List<string> {"Product","ProductSubcategory","ProductCategory"};

#>

Use ObjectTags:<#var TableFilter = (List<string>)RootNode.OleDbConnections["Destination"].ObjectTag["TableFilter"];

#>

LINQ

LINQ (Language-Integrated Query)

One language to query:

SQL Server Databases

XML Documents

Datasets

Collections

Two ways to write queries:

SQL-like Syntax

Extension Methods

LINQ Extension Methods

..and many, many more!

Sort

OrderBy, ThenBy

Filter

Where, OfType

Group

GroupBy

Aggregate

Count, Sum

Check Collections

All, Any, Contains

Get Elements

First, Last, ElementAt

Project Collections

Select, SelectMany

LINQ Extension Methods

var numConnections = RootNode.Connections.Count()

foreach (var table in RootNode.Tables.Where(…))

if (RootNode.Packages.Any(…))

LINQ and Lambda expressions

Use lambda expressions to filter or specify values:

.Where(table => table.Schema.Name == "Production").OrderBy(table => table.Name)


For each element in the collection…



…evaluate a criteria or get a value:


LINQ: Filter collections

Where()Returns the filtered collection with all elements that meet the criteria

RootNode.Tables.Where(t => t.Schema.Name == "Production")

OfType()Returns the filtered collection with all elements of the specified type

RootNode.Connections.OfType<AstExcelOleDbConnectionNode>()

LINQ: Sort collections

OrderBy()Returns the collection sorted by key…

RootNode.Tables.OrderBy(t => t.Name)

ThenBy()…then sorted by secondary key

RootNode.Tables.OrderBy(t => t.Schema.Name).ThenBy(t => t.Name)


OrderByDescending()Returns the collection sorted by key…

RootNode.Tables.OrderByDescending(t => t.Name)

ThenByDescending()…then sorted by secondary key

RootNode.Tables.OrderBy(t => t.Schema.Name).ThenByDescending(t => t.Name)


Reverse()Returns the collection sorted in reverse order

RootNode.Tables.Reverse()

LINQ: Group collections

GroupBy()Returns a collection of key-value pairs where each value is a new collection

RootNode.Tables.GroupBy(t => t.Schema.Name)

LINQ: Aggregate collections

Count()Returns the number of elements in the collection

RootNode.Tables.Count()RootNode.Tables.Count(t => t.Schema.Name == "Production")


Sum()Returns the sum of the (numeric) values in the collection

RootNode.Tables.Sum(t => t.Columns.Count)

Average()Returns the average value of the (numeric) values in the collection

RootNode.Tables.Average(t => t.Columns.Count)


Min()Returns the minimum value of the (numeric) values in the collection

RootNode.Tables.Min(t => t.Columns.Count)

Max()Returns the maximum value of the (numeric) values in the collection

RootNode.Tables.Max(t => t.Columns.Count)

LINQ: Check collections

All()Returns true if all elements in the collection meet the criteria

RootNode.Databases.All(d => d.Name.StartsWith("A"))

Any()Returns true if any element in the collection meets the criteria

RootNode.Databases.Any(d => d.Name.Contains("DW"))

LINQ: Check collections

Contains()Returns true if collection contains element

RootNode.Databases.Contains(AdventureWorks2014)

LINQ: Get elements

First()Returns the first element in the collection (that meets the criteria)

RootNode.Tables.First()RootNode.Tables.First(t => t.Schema.Name == "Production")

FirstOrDefault()Returns the first element in the collection or default value (that meets the criteria)

RootNode.Tables.FirstOrDefault()RootNode.Tables.FirstOrDefault(t => t.Schema.Name == "Production")

LINQ: Get elements

Last()Returns the last element in the collection (that meets the criteria)

RootNode.Tables.Last()RootNode.Tables.Last(t => t.Schema.Name == "Production")

LastOrDefault()Returns the last element in the collection or default value (that meets the criteria)

RootNode.Tables.LastOrDefault()RootNode.Tables.LastOrDefault(t => t.Schema.Name == "Production")

LINQ: Get elements

ElementAt()Returns the element in the collection at the specified index

RootNode.Tables.ElementAt(42)

ElementAtOrDefault()Returns the element in the collection or default value at the specified index

RootNode.Tables.ElementAtOrDefault(42)

LINQ: Project collections

Select()Creates a new collection from one collection

A list of table names:

RootNode.Tables.Select(t => t.Name)

A list of table and schema names:

RootNode.Tables.Select(t => new {t.Name, t.Schema.Name})

LINQ: Project collections

SelectMany()Creates a new collection from many collections and merges the collections

A list of all columns from all tables:

RootNode.Tables.SelectMany(t => t.Columns)

dem

o t

ime!

How is LINQ used

in Biml projects?

C#

Classes and

Methods

C# Classes and Methods

BimlScript and LINQ not enough?

Need to reuse C# code?

Create your own classes and methods!

C# Classes and Methods: From this…

public static class HelperClass {public static bool AnnotationTagExists(AstNode node, string tag) {if (node.GetTag(tag) != "") {return true;

} else {return false;

}}

}

C# Classes and Methods: …to this

public static class HelperClass {public static bool AnnotationTagExists(AstNode node, string tag) {return (node.GetTag(tag) != "") ? true : false;

}} * For bools you can just use:

return (node.GetTag(tag) != "");

But in this example we'll use the verbose, SSIS-like syntax

because it can be reused with other data types, like…

C# Classes and Methods: …or this

public static class HelperClass {public static string AnnotationTagExists(AstNode node, string tag) {return (node.GetTag(tag) != "") ? "Yes" : "No";

}}

Where do you put your code?

Inline code nuggets

Included Biml files with code nuggets

Reference code files

C# Classes and Methods: Inline

<Biml xmlns="http://schemas.varigence.com/biml.xsd"><# foreach (var table in RootNode.Tables) { #> <# if (HelperClass.AnnotationTagExists(table, "SourceSchema")) { #>...

<# } #><# } #>

</Biml>

<#+public static class HelperClass {public static bool AnnotationTagExists(AstNode node, string tag) {

return (node.GetTag(tag) != "") ? true : false;}

}#>

C# Classes and Methods: Included Files

<#@ include file="HelperClass.biml" #>


<# } #><# } #>

</Biml>

<#+public static class HelperClass {public static bool AnnotationTagExists(AstNode node, string tag) {


}#>

C# Classes and Methods: Code Files

<#@ code file="HelperClass.cs" #>


<# } #><# } #>

</Biml>public static class HelperClass {public static bool AnnotationTagExists(AstNode node, string tag) {


}

C#

Extension

Methods

Extension Methods

"Make it look like the method belongs to

an object instead of a helper class"

Extension Methods: From this…



<# } #><# } #>

</Biml>public static class HelperClass {public static bool AnnotationTagExists(AstNode node, string tag) {


}

Extension Methods: …to this



<# } #><# } #>

</Biml>public static class HelperClass {public static bool AnnotationTagExists(this AstNode node, string tag) {


}

Extension Methods: …to this


<Biml xmlns="http://schemas.varigence.com/biml.xsd"><# foreach (var table in RootNode.Tables) { #> <# if (table.AnnotationTagExists("SourceSchema")) { #>...

<# } #><# } #>



}

Extension Methods: …to this :)


<Biml xmlns="http://schemas.varigence.com/biml.xsd"><# foreach (var table in RootNode.Tables.Where(t =>

t.AnnotationTagExists("SourceSchema")) { #>...

<# } #><# } #>



}

Questions?

Get things done

Start small

Start simple

Start with ugly code

Keep going

Expand

Improve

Deliver often

Izpolnite anketo!

Vam je bilo predavanje všeč?

Ste se naučili kaj novega?

Vaše mnenje nam veliko pomeni!

Da bo NT konferenca prihodnje leto še boljša, vas

prosimo, da izpolnite anketo o zadovoljstvu, ki jo

najdete v svojem NTK spletnem profilu.

@cathrinew

cathrinewilhelmsen.net

linkedin.com/in/cathrinewilhelmsen

[email protected]

slideshare.net/cathrinewilhelmsen

Biml resources and references:

cathrinewilhelmsen.net/biml

Data & Analytics

Level Up Your Biml: Best Practices and Coding Techniques (NTK 2016)