Azure IOT: EVENT HUB, STREAM

ANALYTICS & POWER BI

By Chourouk Hjaiej

February 2016

IOT with Azure Services Chourouk HJAIEJ

2

Table of content

Who should read this book?

Introduction

Azure event hubs, Stream Analytics & Power BI

Understanding the solution

Solution Overview with Practical Cases

Setting Things Up

Azure Event Hub

Azure Stream Analytics

Power BI

Examples of dashboard

Conclusion

Keywords

References

IOT with Azure Services Chourouk HJAIEJ

3

Who should read this book?

This paper is a practical look at IOT and different services in Azure which help to solve these problems.

It assumes the reader has a basic understanding of C#.

It is written for both developers and operations professionals who want to improve their team’s or

organization’s ability to deliver robust, highly available applications successfully. One of the biggest

challenges IOT projects face is how to reliably and securely connect devices to the solution backend.

Introduction

The Internet of Things (IOT) starts with your things: the things that matter most to your business. It`s

the Internet of your Things, it means that you are able to connect your device to a solution without

any difficulties, find and rely on a comprehensive set of technologies to connect to and analyze data,

or even build new intelligent devices.

One of the biggest challenges IOT projects face is how to reliably and securely connect devices to the

solution backend. In this paper, we will demonstrate how Azure is able to ensure these functionalities.

At WPC 2014, Microsoft announced the preview of Azure Event Hubs which will enable the collection

of event streams at high throughput which makes it perfect for IOT, data processing, scenarios. In this

paper, I'll start with a simple example to give you a first hands-on experience with Event Hubs.

We'll be building a simple example that publishes events to an Event Hub which can be processed by

one or more worker processed. I'm not fond of these examples, but in this case this could really be

useful. Think of all those applications deployed on hundreds or even thousands of machines. Some of

those applications are probably deployed on laptops, POS devices, … that don't have a direct link with

the main office, so being related to SQL Server is probably not an option. Wouldn't it be useful if all

these applications could send their information (performance…) to a single 'hub' instead of local files

or the event log, even if they're not in the main office? And what if we could then process all those

events for reporting, performance, …? These are the scenarios that are made possible by Event Hubs.

IOT raises important questions which we will address in this paper within the context of a set of

technologies. The most important one is Microsoft Azure, named by Gartner as a leading cloud

platform (2015 Magic Quadrant).

This paper will discuss the topic which is the real-time branch of the Lambda architecture and we will

implement the following components:

An Event Hub

A Stream Analytics with Power BI as an output.

We will use a program in C # to simulate sending information in real time.

Regards,

Chourouk HJAIEJ, DX Technical Evangelist Intern

IOT with Azure Services Chourouk HJAIEJ

4

Azure event hubs, Stream Analytics & Power BI

Understanding the solution

A complete IOT solution is made of several parts. First of all, we need to ingest all the events and data

sent from devices and it’s a such big problem because in the Internet of Things, we have to think at

“scale” with hundreds, thousands, million and billions of devices. In this scenario, we have to

use an ingestion system that is able to scale and handle incoming data very quickly without slowing

down the entire process.

After acquiring data, the ingestion system must provide them to the business engine of our solution

that could have a path for analyzing data as a stream in real time. This path could expose

these informations to the end user that can monitor the devices and what they are “saying” from the

real world. The same information is very useful as input for a machine learning system that can help

us for predictive analysis to understand how data could evolve in the future based on current incoming

values and take preventive actions.

Solution Overview with Practical Cases

The previous section described how to manage these informations with the Microsoft Azure by

using its services.

This section illustrates those concepts in a real world scenario and demonstrates the usefulness of

Microsoft Azure services.

Among these services are event hub which include ….. and stream analytics.

The aim here is to show how the combination of an event hub and Stream analytics provides an

efficient way to manage the huge number of events.

The figure below shows the scenario which we will use in this part.

IOT with Azure Services Chourouk HJAIEJ

5

The scenario we will use in this post builds upon previous posts that I have written so far:

We are simulating potentially millions of devices sending telemetry data. The data is sent to an Event

Hub, and processed in parallel by 3 different processes:

Instead of chaining processes together, making one process wait on the completion of a previous

process, we can scale our solution by having many independent processors work with the data in

parallel. This is incredibly powerful.

We will focus in this paper on using Stream Analytics to process streaming event data from Azure

Event Hubs at scale. We will show some pieces of code, but this code sample will show all the project.

You can build a Universal App with service bus (event hub) on Azure using AMQP. This sample explain

it very well.

Event

Source

Use Azure Event Hub with Visual Studio

Device Simulator for Event Hubs

Collect Use Azure Event Hubs with Visual Studio

Filter /

Analyze /

Aggregate

Scaling Azure Event Hubs Processing

Use the Azure Stream Analytics

Dashboard Use Power BI in order to have significant results

IOT with Azure Services Chourouk HJAIEJ

6

Setting Things Up

Before we start using Stream Analytics, we will need an Event Hub, something that sends data to the

Event Hub. Creating the Event Hub is very easy using the Azure Management Portal. The figure below

describes the steps to create an event hub.

Azure Event Hub

Provide a name and region and a namespace. Provide a partition count and message retention period.

IOT with Azure Services Chourouk HJAIEJ

7

As you can see, this is the list of event hubs created by me and this is the last one.

I will create “DevicePolicy” :

You can see the primary and the secondary key of this shared access policy:

Our Event hub is ready. We will now create and configure a service which will consume post events in

our Event Hub. This service is Azure Stream Analytics.

Azure Stream Analytics

From the Azure portal, click the "+" button at the bottom of the window. Select "Data Services",

"Stream Analytics," "Quick Create". Fill in the different fields and click "Create Stream Analytics job".

IOT with Azure Services Chourouk HJAIEJ

8

After a few seconds, my Azure Stream Analytics service will be created.

Setting the Azure Service Stream Analytics

Now that our Azure Stream Analytics service was created, we'll set it to receive information from our

Event Hub (then possibly turn), then send them to a Power BI portal.

Click the name of your Azure Stream Analytics ("streamanalyticsjob" in this example). Then click

"INPUTS". Click on "ADD AN INPUT".

We will select "Events Hub" (Event Hub).

IOT with Azure Services Chourouk HJAIEJ

9

In the Alias Entry field, enter the name you want to give your input in the Azure Stream Analytics.

For this demonstration, we'll select "Use Event Hub from current subscription."

IOT with Azure Services Chourouk HJAIEJ

10

Finally, we will choose the serialization settings. We have the choice between "Avro", "CSV" or "JSON".

We will select JSON because the program which I will use to send the information in the JSON format.

You will have an inputs for this Stream Analytics.

Setting a query

Now that we have defined an entry, we will define a query to retrieve the data. This query can be a

simple "select" or a more complex query to transform data or perform operations on time windows

(to average every 5 seconds).

Click on "Query" to define the data processing request. Note that the language used is very similar to

T-SQL. For this example, I will realize just a select all the fields that are present in the entrance to my

service:

IOT with Azure Services Chourouk HJAIEJ

11

SELECT * FROM Inputs

Click "Save" and confirm registration.

Setting an output

We will now set the output of our service to a Power BI portal.

IOT with Azure Services Chourouk HJAIEJ

12

Set in the "Add a Microsoft Power output BI" the name of the database and the table name. If the table

already exists Power BI side, it will be replaced with the new one. Confirm your changes.

You must obtain a new outlet for your Azure Stream Analytics service.

IOT with Azure Services Chourouk HJAIEJ

13

It remains for us to start the service. At the bottom of the page, click the Start button.

Simulating data injection

Now, we have to interest in how it sends data.

To do this, I used a small program in C # that can simulate sending data, but above all it shows how

to connect to an Event Hub precisely to send the data.

First, in my "Program.cs" file, don`t forget to replace the values with that of your Event Hub.

When running the program, here is an example of the data sent to the Event Hub:

IOT with Azure Services Chourouk HJAIEJ

14

Power BI

Now log in to the portal via the BI Power http://app.powerbi.com address.

In the left pane, in the "DataSets", a new data source, "DeviceDataset" has just appeared. It is

recognizable with the small green asterisk.

IOT with Azure Services Chourouk HJAIEJ

15

Examples of Dashboard

IOT with Azure Services Chourouk HJAIEJ

16

Azure Stream Analytics Monitoring

The Azure Stream Analytics also provides a monitoring dashboard as well as the possibility to refer to

the events related to the service logs. This can be very useful to understand what happens (especially

when it does not work).

In your Azure Stream Analytics service, click "Dashboard" to see the events related to your service (you

can also click on "Monitor").

In the dashboard, you default service supervisory common indicators and access to the operations log.

Click on the link "operations logs" to open the Services management window.

IOT with Azure Services Chourouk HJAIEJ

17

Keywords:

Azure Event Hub, Azure Stream Analytics, Scalability, Power BI, Dashboard, Cloud, Query, Azure,

References

Azure

> What is Microsoft Azure?

https://azure.microsoft.com/en-us/overview/what-is-azure/

> Microsoft – the only vendor named a leader in Gartner Magic Quadrants for IaaS,

Application PaaS and Cloud Storage

https://azure.microsoft.com/en-us/blog/microsoft-the-only-vendor-named-a-leader-in-gartner-

magic-quadrants-for-iaas-application-paas-cloud-storage-and-hybrid/

> Azure Preview Portal

https://azure.microsoft.com/en-us/features/azure-portal/

Azure Event Hub

> What is Azure Event hub?

https://azure.microsoft.com/en-gb/documentation/articles/event-hubs-overview/

Azure Stream Analytics

> What is Azure Stream analytics?

https://azure.microsoft.com/en-us/services/stream-analytics/

Power BI

> What is Power BI?

https://powerbi.microsoft.com/fr/

Azure IOT vs Amazon aws IOT

> What is the differences between Azure IOT and Amazon aws IOT?

https://paolopatierno.wordpress.com/2015/10/13/an-iot-platforms-match-microsoft-azure-iot-vs-

amazon-aws-iot/

IOT with Azure Services Chourouk HJAIEJ

18

Microsoft Azure