Embed Size (px)
Citation preview
#SDF16
Building High-scalable Enterprise Solutions with Microsoft Azure and PowerBIThomas Claudius Huber – Principal Consultant@thomasclaudiush
#SDF16
Thomas Claudius Huber
Principal Consultant
– Microsoft Most Valuable Professional (MVP)
Focus on C#, TypeScript, Angular and Azure
– Part of Cloud-Unit @Trivadis
Book- and Pluralsight-Author
– See also www.thomasclaudiushuber.com
#SDF16
Agenda
The Big Picture
Azure Event Hubs
Stream Analytics and Power BI
#SDF16
The Big Picture
#SDF16
The big picture
#SDF16
Agenda
The big picture
Azure Event Hubs
Stream Analytics and Power BI
#SDF16
Azure Event Hubs
#SDF16
Azure Event Hubs
Scales up to millions of events
Great approach for telemetry data
Great approach for sensor-data
#SDF16
#SDF16
Azure Event Hubs: Partitions
Event stream is segmented by partitions for scale
– Scaling for event consumers
– Scaling for event submitters
You can have between 2 and 32 partitions
Azure Support can enable up to 1024 or even more «under special conditions»
#SDF16
Azure Event Hubs: Throughput Units
Throughput units (TUs) are important for billing. 1 Unit means
– Write maximum 1MB/sec and 1000 messages per second
– Read maximum 2MB/sec
Number of partitions has to be equal or greater than number of TUs
TUs are applied on namespace level
– You can have up to 10 event hubs in a namespace
– Namespace is used for the URI, for example https://<yourNamespace>.servicebus.windows.net
#SDF16
Azure Event Hubs: Consumer Groups
Views on the Event Stream
– Similar to topic subscriptions that are also a «view» on an event stream
– Organizational anchor for checkpointing
$Default consumer group is always there
– Up to 20 named consumer groups are supported
Limitations
– 5 concurrent reading connections per consumer group
• It’s recommended to use a separate consumer group for each job
#SDF16
Azure Event Hub vs. Azure IoT Hub
Quote: «It is not uncommon to use both IoT Hub and Event Hubs in the same solution»Official docmentation: https://azure.microsoft.com/en-us/documentation/articles/iot-hub-compare-event-hubs/
Event Hub IoT Hub
Communication Device to cloud Device to cloud and cloud to device
Security Event hub wide shared access policy Per device identity
Scale 5000 simultaneous connections Millions of simultaneous connections
SDKsSupported on .NET and C, also has AMQP and HTTP (REST) interfaces
Device SDKs for several platforms.
#SDF16
But why not always using IoT Hub?
IoT Hub scales on devices
– a lot of concurrent connections
– but is expensive for a few thousand devices and a lot events
Event Hub scales on events
– but is limited to 5000 concurrent connections
#SDF16
So what about pricing?
Event Hub
CHF 0.0253 per million events
1 Throughput Unit = CHF 0.0135/hr (~CHF 10/month)
– 1 Throughput Unit = 1MB / 1000 Messages Ingress, 2MB read
#SDF16
Creating an Event Hub: Step 1
Go to the https://portal.azure.com
Select «New Internet of Things Event Hubs»
Enter Namespace-Name and Resource Group and click «Create»
#SDF16
Creating an Event Hub: Step 2
So far you’ve created a namespace
– the namespace can have up to 10 event hubs
When you navigate to the namespace (via «Resources»), you can add a new event hub
#SDF16
Demo: Create an Event Hub
#SDF16
Connect to the Event Hub
To push events to an event hub, you have several options:
– From a .NET application you can use the NuGet-packageWindowsAzure.ServiceBus and the EventHubClient-class
• Samples: https://github.com/thomasclaudiushuber/EventHub.Clients
– You can also use a REST-API that is available automatically for each event hub
• https://github.com/thomasclaudiushuber/EventHub.RestClientGenerator
You need to setup a Shared Access policy for your event hub
To access the event hub, you need the URI and the shared access key
– both is part of a Connection String that you find in the Shared Access policy
#SDF16
Pushing Events to the Event Hub from a .NET App
Use the NuGet-package WindowsAzure.Service
– It has an EventHubClient-class
Events are JSON-data items
#SDF16
Consuming Events from the Event Hub in a .NET App
Same EventHubClient-class
Grab the consumerGroup and create the receivers for the partitionIds
#SDF16
Demo: Access the Event Hub
#SDF16
Agenda
The big picture
Azure Event Hubs
Stream Analytics and Power BI
#SDF16
The big picture
#SDF16
What is Stream Analytics
it’s a real-time event processing engine
Use a SQL-variant to query the event stream
High scalable
#SDF16
Stream Analytics: Inputs and Outputs
#SDF16
Stream Analytics: Windowed Aggregates
There are different windowed aggregates, one is the Tumbling Window
select DeviceName,max(value),count(*)into [SDFPBOutput]from [SDFEventHub99] Timestamp by Readtime group by devicename, TumblingWindow( Duration(second, 5), Offset(millisecond, -1))
#SDF16
Demo: Setting up Stream Analytics
#SDF16
Power BI as an Output
BI-tool to visualize your data
– Access it via https://www.powerbi.com
Stream Analytics can directly push its outupt to Power BI
In Power BI you can work with the Stream Analytics Data
#SDF16
Build a Live Dashboard in Power BI
#SDF16
Agenda
The big picture
Azure Event Hubs
Stream Analytics and Power BI
#SDF16
The big picture
