INTEGRATION DAY 2015
Sam Vanhoutte
Azure Event Hubs, Stream Analytics & Power BI
Nice to meet youSam VANHOUTTECTO
7 year - BizTalk V-TSP1st year - Integration MVP
[email protected]+32 474 849 993@SamVanhoutte
be.linkedin.com/in/
samvanhoutte/
> 60 Active integration customers
International Focus - HQ in BEFocused on integration solutions
2000 Belgium2004 France2013 Portugal
60 employees > 50 consultants BizTalk certifiede-news + SoMe
2012 & 2013
Partner of the Year
Award FinalistApplication Integration
The Internet of Things
3
The Internet of Everything
M2M communicationSpecial purpose
devicesSmart things
The IoT value chain
Ingestor (broker)
Collection Presentation and action
Event producers Transformation Long-term storage
Event Hubs
Storage adapters
Stream processingCloud gateways
(web APIs)
Field gateways
Applications
Legacy IOT (custom protocols)
Devices
IP-capable devices(Windows/Linux)
Low-power devices (RTOS)
Search and query
Data analytics (Excel)
Web/thick client dashboards
Service bus
Azure DBs
Azure storage
HDInsight
Stream Analytics
Devices to take actionMachine Learning
Demo scenario
6
Event Hubs Stream Analytics Power BI
Ingest
Speeding tickets
Traffic jam detection
Power BI tools
7
• Power BI designer (when no office license)
• PowerQuery >> Get data
• PowerPivot >> Prepare data
• PowerView >> Present data
Event Hubsan Azure Service Bus service
(collection & ingestion)
Azure Service BusAzure Service Bus
Relay Request – Response services
Queue High-Reliability Async MessagingSessions, Scheduled Delivery, etc.
Topic Queues with rules-based pub/sub
Notification Hub Mobile push notification servicesMillions of notification targets
Event Hub partitioned event ingestion
Event Hub – IOT at Scale
Event Sources Cloud Services
Storage & Analytics
Custom Code & 3rd Party Services
Web/Mobile User Interfaces
Integration Services
Event Hub
- Hyper Scale -- Fully Managed -- Interoperable -
- Secure -- Cost Effective -
Introducing Azure Event Hubs
Azure Event Hub
Up to 32 partitions via portal, more on
request
Parti
tions
Event Producers
> 1M Producers> 1GB/sec Aggregate Throughput
Direct
PartitionKeyHash
Throughput Units:• 1 ≤ TUs ≤ Partition Count• TU: 1 MB/s writes, 2 MB/s reads
AMQP 1.0Credit-based flow controlClient-side cursorsOffset by Id or Timestamp
Consumer Group(s)
Receivers
Event Processor Host
IEventProcessor
Event Hubsan Azure Service Bus service
Producers(sending data to Event
Hubs)
15
Publishers
Event Producers
Direct
PartitionKeyHash
Parti
tions
Very many publishersShort-lived, low throughput: HTTPSLong-lived, high throughput: AMQP
Long lived AMQP connections are billable, HTTPS requests are not; AMQP connection allowance included in tier
Publish to …PartitionId
DirectPartitionKey
PartitionKey selecting PartitionIdPublisher Policy (<eh>/publishers/<name>)
<name> overriding PartitionKey
16
Publishers
Event Producers
Direct
PartitionKeyHash
Parti
tions
Very many publishersShort-lived, low throughput: HTTPSLong-lived, high throughput: AMQP
Long lived AMQP connections are billable, HTTPS requests are not; AMQP connection allowance included in tier
Publish to …PartitionId
DirectPartitionKey
PartitionKey selecting PartitionIdPublisher Policy (<eh>/publishers/<name>)
<name> overriding PartitionKey
Demo:• Create Event Hubs• Send events to Event Hub
17
Securing publishers
➔ SAS Policies are defined in the portal➔ Not linked to publishers and limited in numbers
➔ Create Signatures per device➔ Have a timespan➔ Are unique to a publisher
➔ Revoking or blacklisting publishers➔ Manage Rights needed for this
18
Securing publishers
➔ SAS Policies are defined in the portal➔ Not linked to publishers and limited in numbers
➔ Create Signatures per device➔ Have a timespan➔ Are unique to a publisher
➔ Revoking or blacklisting publishers➔ Manage Rights needed for this
Demo:• Generating SAS Signatures• Using Signatures by publishers• Revoking publishers
Reminder ! Sam, increase your instances
Event Hubsan Azure Service Bus service
Consumers(reading data to Event
Hubs)
Like subscriptions
Consumer groupsReceivers read from a consumer groups
Consumer Group(s)
Checkpointing per CG
Create receivers per
partitionMaximum 20
CGConsumer
Grp
{Default}
Like subscriptions
ConsumersPartitioned consumer model (not competing consumers!)
Each at their own pace
PartitionEvent Log
ID, Time, [Data]
ID, Time, [Data]
ID, Time, [Data]
ID, Time, [Data]
ID, Time, [Data]
Time
ID
22
EventProcessorHostOut of the box Lease management
Nuget package
Managed cursors by
client
Uses storage for state and
CPIEventProcess
or
23
EventProcessorHostOut of the box Lease management
Nuget package
Managed cursors by
client
Uses storage for state and
CPIEventProcess
or
Demo:• Implementing EventProcessorHost• Dashboarding demo
24
Securing consumers
➔ Using SAS key / values or ACS with SBAZTool➔ Full SAS support for consumer groups will
come
Event Hubsan Azure Service Bus service
Pricing & Tips + tricks
26
Pricing
Basic: Up to 100 connections, no extensionStandard: 1000 connections incl.
Price (US Dollars)
Throughput Unit Hour (Basic) 0.015 TU per hour
Throughput Unit Hour (Standard) 0.03 TU per hour
Ingress Events 0.028 per 1,000,000 events
Cost Brokered Connections (1k-100k) 0.00004 connection/hour
Cost Brokered Connections (100k-500k) 0.00003 connection/hour
Cost Brokered Connections (500k+) 0.00002 connection/hour
Storage Overage >TUs*84GB local-redundant Azure storage charge-through
27
Tips & tricks
Multiple hubs per namespace
Reuse senders & factories
Use consumer groups
Azure Stream Analytics complex event
processing.
Introduction(set up a basic ASA job)
Scenarios
Smart grid CRM alerting sales with customer scenario
Data and identity protection services
Real-time fraud detection
Click-stream analysis Real-time financial portfolio alerts
Connected car scenarioReal-time financial sales tracking
30
Real time analytics
Millions events / sec
Continuous stream of
data CorrelateFast time to
value
High availability SQL Syntax
Easy test & debug in
portal PowerBI !
.999% SQL
31
End to end Architecture overview
Data Source
Collect Process ConsumeDeliver
Event Inputs- Event Hub- Azure Blob
Transform- Temporal joins- Filter- Aggregates- Projections- Windows- Etc.
Enrich
Correlate
Outputs- SQL Azure- Azure Blobs- Event Hub- Table storage- PowerBI
☁
BI Dashboards
Predictive Analytics
AzureStorage
• Temporal Semantics
• Guaranteed delivery
• Guaranteed up time
Azure Stream Analytics
Reference Data- Azure Blob
32
End to end Architecture overview
Data Source
Collect Process ConsumeDeliver
Event Inputs- Event Hub- Azure Blob
Transform- Temporal joins- Filter- Aggregates- Projections- Windows- Etc.
Enrich
Correlate
Outputs- SQL Azure- Azure Blobs- Event Hub
☁
BI Dashboards
Predictive Analytics
AzureStorage
• Temporal Semantics
• Guaranteed delivery
• Guaranteed up time
Azure Stream Analytics
Reference Data- Azure Blob
Demo:• Set up Stream Analytics• The first job
Azure Stream Analytics complex event
processing.
Query Syntax(SQL like a pro)
Functions & supported types
Aggregate functionsCount, Min, Max, Avg, Sum
Scalar functionsCast
Date and time: Datename, Datepart, Day, Month, Year, Datediff, Dateadd
String: Len, Concat, Charindex, Substring, Patindex
Types
Type Descriptionbigint Integers in the range -2^63 (-9,223,372,036,854,775,808) to 2^63-1 (9,223,372,036,854,775,807).
float Floating point numbers in the range - 1.79E+308 to -2.23E-308, 0, and 2.23E-308 to 1.79E+308.
nvarchar(max) Text values, comprised of Unicode characters. Note: A value other than max is not supported.
datetime Defines a date that is combined with a time of day with fractional seconds that is based on a 24-hour clock and relative to UTC (time zone offset 0).
35
Tumbling windowsGroup events, based on timestamping
Tumbling window
Aggregate per time interval
Hopping window
Schedule overlapping windows
Sliding window
Windows constant re-evaluated
36
End to end Architecture overview
Data Source
Collect Process ConsumeDeliver
Event Inputs- Event Hub- Azure Blob
Transform- Temporal joins- Filter- Aggregates- Projections- Windows- Etc.
Enrich
Correlate
Outputs- SQL Azure- Azure Blobs- Event Hub
☁
BI Dashboards
Predictive Analytics
AzureStorage
• Temporal Semantics
• Guaranteed delivery
• Guaranteed up time
Azure Stream Analytics
Reference Data- Azure Blob
Important:• For PowerBI you need
organizational account
37
End to end Architecture overview
Data Source
Collect Process ConsumeDeliver
Event Inputs- Event Hub- Azure Blob
Transform- Temporal joins- Filter- Aggregates- Projections- Windows- Etc.
Enrich
Correlate
Outputs- SQL Azure- Azure Blobs- Event Hub
☁
BI Dashboards
Predictive Analytics
AzureStorage
• Temporal Semantics
• Guaranteed delivery
• Guaranteed up time
Azure Stream Analytics
Reference Data- Azure Blob
Demo:• Traffic management (speed tickets)• Traffic management (average speed)• Traffic management (suspected cars)
38
Tips & tricks
Use Timestamp by
Chose your windowing logic
Consumer groups
Don’t delete blobs while job
runs