LINQ Internals - STLDODN

LINQ Internals

Keith Dahlbyhttp://solutionizing.net/@dahlbyk

http://solutionizing.net/


http://twitter.com/dahlbyk

Who am I?

Iowa Native Iowa State University Cedar Rapids SharePoint Inetium Language Geek

Agenda

Terminology Enabling Technologies Query Expressions & Translation Standard Query Operators LINQ to Objects LINQ to IQueryable Beyond

Terminology

Language-INtegrated Query Query Expressions

o C# 3.0 & VB 9 Featureo Syntactic Sugar on Method Calls

LINQ Providero Standard Query Operatorso *LINQ / LINQ to *

Enabling Technologies

Anonymous Typeso Type Inference

Extension Methods Lambda Expressions

o Expression Trees

Anonymous Types

C# VB

var entry = new { Title = "Dear Diary", DateTime.Now};

Console.WriteLine("{0:d}: {1}", entry.Now, entry.Title);

Dim entry = New With { _ .Title = "Dear Diary", _ DateTime.Now _}

Console.WriteLine("{0:d}: {1}", _ entry.Now, entry.Title)

var and Dim types are inferred!

Extension Methods

Must be defined in static class (VB Module) Appear in IntelliSense if namespace in using/Imports

First argument serves as instance variableImports System.Runtime.CompilerServices

Module StringExtensions <Extension()> _ Public Sub Print(ByVal aString As String) Console.WriteLine(aString) End SubEnd Module

Dim hello = "Hello from StringExtensions"hello.Print()

Extension Methods

C# supports this keywordstatic void ForEach<T>(this IEnumerable<T> source, Action<T> action){ foreach (T o in source) action(o);} Generic Delegatesdelegate void Action();delegate void Action<T>(T arg);delegate void Action<T1, T2>(T1 arg1, T2 arg2);delegate TResult Func<TResult>();delegate TResult Func<T, TResult>(T arg);delegate TResult Func<T1, T2, TResult>(T1 arg1, T2 arg2);

Lambda Expressions

Evolution of C# Delegatesdelegate string MyDelegate(int i);

static string MyMethod(int i) { … }MyDelegate del1 = new MyDelegate(MyMethod); // C# 1.0MyDelegate del2 = delegate(int i) { … }; // C# 2.0MyDelegate del3 = MyMethod; // C# 2.0MyDelegate del4 = (int i) => { … }; // C# 3.0Func<int, string> del5 = i => { … }; // C# 3.0

New in VB 9.0Dim del4 As MyDelegate = Function(i As Integer) …

Expression Trees

System.Linq.Expressions Abstract Syntax Tree representing code Assignable from lambda expression:Expression<Func<int, int>> inc = (a => a + 1); Compiles into:ParameterExpression CS$a;Expression<Func<int, int>> inc = Expression.Lambda<Func<int, int>>( Expression.Add( CS$a = Expression.Parameter(typeof(int), "a"), Expression.Constant(1, typeof(int))), new ParameterExpression[] { CS$a });

Query Expressions

Provide query keywords in C# & VBvar names = from s in Students where s.Age % 2 == 0 orderby s.Name descending select s.Name;

Translated into method calls:IEnumerable<string> names = Students.Where(s => s.Age % 2 == 0) .OrderByDescending(s => s.Name) .Select(s => s.Name);

Query Translation

Part of language specificationo Section 7.15.2 of C# 3.0 Language Specificationo Section 11.21 of Visual Basic 9.0 Language Specification

C# translation is rule-based:o “A query expression is processed by repeatedly applying

the following translations until no further reductions are possible.”

VB translation is sequential:o “Query operator translation occurs in the order in which

the query operators occur in the expression.”

http://download.microsoft.com/download/3/8/8/388e7205-bc10-4226-b2a8-75351c669b09/CSharp%20Language%20Specification.doc

http://www.microsoft.com/downloads/details.aspx?FamilyID=39DE1DD0-F775-40BF-A191-09F5A95EF500&displaylang=en

C# Query Translation

Select and GroupBy clauses with continuationso from … into x … from x in ( from … ) …

Explicit range variable typeso from T x in e from x in ( e ) . Cast < T > ( )o join T x in e on k1 equals k2 join x in ( e ) . Cast < T > ( ) on

k1 equals k2

Degenerate query expressionso from x in e select x ( e ) . Select ( x => x )

From, let, where, join and orderby clauseso from, let and join are complicatedo from x in e where f … from x in ( e ) . Where ( x => f ) …o from x in e orderby k1 , k2 , … , kn … from x in ( e ) . OrderBy ( x =>

k1 ) . ThenBy ( x => k2 ) . … . ThenBy ( x =>

kn ) …

Select clauseso from x in e select v ( e ) . Select ( x => v )

GroupBy clauseso from x in e group x by k ( e ) . GroupBy ( x => k )o from x in e group v by k ( e ) . GroupBy ( x => k , x =>

v )

Demo: Queries and Translation

Standard Query Operators: 51 Total

Aggregate Average Count LongCount

Max Min Sum Concat

AsEnumerable Cast ToArray ToDictionary

AsQueryable OfType ToList ToLookup

ElementAt First Last Single

ElementAtOrDefault

FirstOrDefault LastOrDefault SingleOrDefault

SequenceEqual GroupBy Join GroupJoin

DefaultIfEmpty Empty Repeat Range

Skip SkipWhile Take TakeWhile

Distinct Select SelectMany Where

Except Any All Contains

Intersect OrderBy ThenBy Reverse

Union OrderByDesc ThenByDesc

Standard Query Operators: 42 “Real” Operators


Max Min Sum Concat




ElementAtOrDefault









Standard Query Operators: 24 in Visual Basic 9


Max Min Sum Concat




ElementAtOrDefault









Standard Query Operators: 11 in C# 3


Max Min Sum Concat




ElementAtOrDefault









Standard Query Operators

Aggregation Operations Concatenation Operations Converting Data Types Element Operations Equality Operations Generation Operations Grouping Data Join Operations Partitioning Data Projection Operations Quantifier Operations Filtering Data Set Operations Sorting Data

Aggregations Operations


Max Min Sum Concat




ElementAtOrDefault









Concatenation Operations


Max Min Sum Concat




ElementAtOrDefault









Converting Data Types


Max Min Sum Concat




ElementAtOrDefault









Element Operations


Max Min Sum Concat




ElementAtOrDefault









Equality Operations


Max Min Sum Concat




ElementAtOrDefault









Generation Operations


Max Min Sum Concat




ElementAtOrDefault









Grouping Data


Max Min Sum Concat




ElementAtOrDefault









Join Operations


Max Min Sum Concat




ElementAtOrDefault









Partitioning Data


Max Min Sum Concat




ElementAtOrDefault









Projection Operations


Max Min Sum Concat




ElementAtOrDefault









Quantifier Operations


Max Min Sum Concat




ElementAtOrDefault









Filtering Data


Max Min Sum Concat




ElementAtOrDefault









Set Operations


Max Min Sum Concat




ElementAtOrDefault









Sorting Data


Max Min Sum Concat




ElementAtOrDefault









Closer Look: Join & GroupJoin

This query expression:from x1 in e1

join x2 in e2 on k1 equals k2 …

Translates into:from * in ( e1 ) . Join( e2 , x1 => k1 , x2 => k2 , (x1 , x2) => new { x1 , x2 })…

Join parameters:o this outer – The first sequence to join.o inner – The sequence to join to the first sequence.o outerKeySelector – Select key from first sequence. o innerKeySelector – Select key from second sequence.o resultSelector – Select result from two matching elements.

Closer Look: Join & GroupJoin

Use default equality comparero EqualityComparer<TKey>.Default

Composite keyso Type with good Equals() & GetHashcode()o Anonymous types good choice – required by some providers

Custom IEqualityComparer<TKey>o No query syntaxo Not alone

Operators With Comparer Overload


Max Min Sum Concat




ElementAtOrDefault









Closer Look: Join, Group Join and Left Outer Join

Join (no into)o “Correlates the elements of two sequences based on matching

keys.”o Func<TOuter, TInner, TResult> resultSelector

Group Join (with into)o “Correlates the elements of two sequences based on equality

of keys and groups the results.”o Hierarchical: One on left to many on righto Func<TOuter, IEnumerable<TInner>, TResult>

resultSelector Left Out Join

o Use DefaultIfEmpty() on Group Join result:from x1 in e1

join x2 in e2 on k1 equals k2 into jfrom xj in j . DefaultIfEmpty()…

Closer Look: SelectMany

Used to translate multiple from clauseso “Projects each element of a sequence to an IEnumerable<T>

and flattens the resulting sequences into one sequence.” This query expression:from x1 in e1

from x2 in e2

… Translates into:from * in ( e1 ) . SelectMany(

x1 => e2 , ( x1 , x2 ) => new { x1 , x2 } )… What is * ?

o Transparent Identifier = Intermediate Step

LINQ to Objects

Implementation of all Standard Query Operators System.Linq.Enumerable

o Extension methods on IEnumerable<T> Use C# Iterators Extensively

o yield return LINQBridge – .NET 2.0

o LINQ to Objectso Generic delegates (Action, Func)o ExtensionAttributeo http://code.google.com/p/linqbridge/

http://code.google.com/p/linqbridge/

Manner of Execution

Immediateo Performed at the point where the query is declared.o All operators returning a non-enumerable result.

Deferredo Performed at the point where result is used.

• foreach• Immediate Operator

o Result depends on data source at point of use Deferred Streaming

o Consumes source data on demand Deferred Non-Streaming

o Caches all source data to perform operation

Immediate Execution


Max Min Sum Concat




ElementAtOrDefault









Deferred Streaming Execution


Max Min Sum Concat




ElementAtOrDefault









Deferred Non-Streaming Execution


Max Min Sum Concat




ElementAtOrDefault









LINQ to * via Objects

LINQ to XMLo System.Xml.Linqo All selectors return IEnumerable<>

LINQ to DataSeto System.Data.DataSetExtensionso EnumerableRowCollection<DataRow>

AsEnumerable(this DataTable source) o EnumerableRowCollectionExtensions provides some

operators LINQ to Legacy

o Take advantage of Cast<T>() and OfType<T>()o Query expression support, just like foreach:

from MyType obj in MyArrayList…

Tour of LINQ to XML and DataSet Internals

LINQ to IQueryable

Expression Trees from Standard Query Operators System.Linq.Queryable

o Extension methods on IQueryable<T>

public interface IQueryable : IEnumerable{ Type ElementType { get; } Expression Expression { get; } IQueryProvider Provider { get; }}

public interface IQueryable<T> : IEnumerable<T>, IQueryable, IEnumerable{ }

IQueryProvider

“Only” responsibility is processing expression trees

public interface IQueryProvider{ IQueryable CreateQuery(Expression expression); IQueryable<TElement> CreateQuery<TElement>(Expression expression);

object Execute(Expression expression); TResult Execute<TResult>(Expression expression);}

Queryable Operators

public static TSource First<TSource>( this IQueryable<TSource> source){ return source.Provider.Execute<TSource>( Expression.Call(null, ((MethodInfo) MethodBase.GetCurrentMethod()) .MakeGenericMethod(new Type[]

{ typeof(TSource) }), new Expression[] { source.Expression } ) );}

Tour of LINQ to SQL Internals

Beyond?

IEnumerable and IQueryable are not special Methods can come from anywhere Parameter types are flexible

o As long as the compiler can infer everything…o Who ever said LINQ predicates need to be Boolean-value

d?

New provider modelso LINQ to Eventso Lazy LINQ (coming soon)

http://community.bartdesmet.net/blogs/bart/archive/2008/09/14/who-ever-said-linq-predicates-need-to-be-boolean-valued.aspx

http://community.bartdesmet.net/blogs/bart/archive/2008/09/14/who-ever-said-linq-predicates-need-to-be-boolean-valued.aspx

Resources

C# in Depth by Jon Skeet (Manning) Bart De Smet – Microsoft Language Geek

o http://community.bartdesmet.net/blogs/bart/

Meo http://solutionizing.net/o keith@ ---^o @dahlbyk

http://community.bartdesmet.net/blogs/bart/


http://twitter.com/dahlbyk

Technology

LINQ Internals - STLDODN