IoT Communication Design - cool.ntu.edu.tw

IoT Communication DesignLecturer: Cheng-Jen Liu

Outline● Introdcution of REST

● Introduction of gRPC

● Introduction of MQTT

● Homework Description (5%)

Communication Pattern (1) -- Client Server Paradigm

● Simple communication but heavy overhead

● Stateless communication

● General used in web technologies

Communication Pattern (2) -- Peep-To-Peer Paradigm

● Flexible communication model (Peer can be either client/server)

● Stateful communication

● Hard to scalable

● Remote function call pattern

Communication Pattern (3) -- Publish Subscribe Paradigm

● Decouple communication parties with central message broker

● Scalable Desgin

● Flexible Communication

How Youtube Works (Discussion) ??● Web Service

○ Client Server Model

● Video Processing○ Peer-to-Peer Model

○ Pub-Sub Model

● Notification System○ Pub-Sub Model

Q & AIs there any specific design/communication pattern fit all the scenarios ?

There is no one-solution-fit-all approach that can fit all the scenarios.

Usually, a project will combine lots of different communication patterns

to realize the application goal.

Introduction of RESTREST (Rest API) is a common application interface used for softwares to export

their services to other softwares. It is not a protocol but an architectural style.

Generally, developers use REST to create website and export CRUD operations

to manipulate resource on the website.

Let’s build a simple web service with Django and RESTWe are going to use a python web backend framework, called Django, to

realize a simple web service that allows clients to send GET requests and get

echoing response.

Anatomy of Django Framework

Web Server (e.g. NGINX)

● Security Check● Authentication● Preprocessing

Route requests to associated function

Backend Logics (functions)

Clone Django Tutorial ProjectType cmd `git clone https://github.com/johnnylord/django-rest-tutorial.git`.

Follow the instructions in the README file to run the project.

How to check Django project (1) -- settings.py

Installed or defines apps

Rest API Framework Related Library

tutorial app containing backend logics

How to check Django project (2) -- urls.py

mysite/urls.py

tutorial/urls.py

Target URL: http://localhost:8000/rest/tutorial

Start parsing

Reque

st will

be ro

ute to

here

How to check Django project (3) -- views.py

tutorial/views.pyHandling GET request from client

Runtime Snapshots (Server Logging)

GET Request from client

Runtime Snapshots (Client Request)

Server Response Header

Server Response Body

Q & A● How does Django route the client request to the assoicate view function ?

You need to define routing rules with regular experssions in the

urls.py file and specify associated view functions to the defined

rules.

● How to send file to the Django backend through REST API ?

Ask Google !!!

Introduction of gRPCgRPC is a language agnostic, high-performance Remote Procedure Call (RPC)

framework. The main benefits of gRPC are: Modern, high-performance,

lightweight RPC framework. Contract-first API development, using Protocol

Buffers by default, allowing for language agnostic implementations

What is Remote Procedure Call ?Remote Procedure Call (RPC) is a mechanism/concept that allow a process

running locally to be able to call functions provided by a remote process

running on another machine.

What is Serialization ?Serialization is a process that convert object into a stream of bytes that can be

used to send between two running processes through network communication.

What is Protobuf ?Protocol buffers (Protobuf) are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data.

● It is a schema-based serialization method

● It has its own compiler to compile schema code to language-sepcific code

Install the Protobuf CompilerVisit the installation webpage https://grpc.io/docs/protoc-installation/

Let’s build a fibonacci service with gRPC and ProtobufType cmd `git clone https://github.com/johnnylord/gRPC-with-protobuf.git` and follow the README instruction.

Protobuf & gRPC Schema File

service/fib.proto

gRPC Service interface

Request Data structure

Response Data structure

Compile the schema fileAfter the compilation (`$ make`), the generated python wrapper of fibonacci grpc modules.

build/service/fib_pb2_grpc.py build/service/fib_pb2.py

gRPC Servicer ImplementationThe generated gRPC Servicer is just an

abstract class, you need to define the

functionality by yourself.Interface needs to be implemented.

server.py

gRPC Stub(client) to use remote service Typically, we use `stub` to indicate the access point of the remote service.

client.py

Remote Procedure Call

Instantiate request

Establish connection

Q & A● What’s the difference between gRPC and JsonRPC ?

gRPC uses protobuf and JsonRPC uses json. The underlying

serialization method is different.

● What’s the advantage of schema-based serialization method ?

Schema-based serialization can compile the object into smaller

bytestreams. It doesn’t need to include field information in the

payload.

What is MQTT ?MQTT (Message Queuing Telemetry Transport) is a publish/subscribe messaging protocol that works on top of the TCP/IP protocol. The first version of the protocol was developed by Andy Stanford-Clark of IBM and Arlen Nipper of Cirrus Link in 1999.

AWS IoT Core MQTTMQTT is widely used in IoT Application. AWS even have dedicated services based on MQTT messaging protocol.

IoT Application System

Eclispse Mosquitto MQTT BrokerEclispse Mosquitto is an open source project that implements the MQTT protocol and can be used as the message broker.

You can consider this as message broker.

Possible IoT System Design with MQTT Broker (1)

HEAT

TEMP

MOIST

Data

Possible IoT System Design with MQTT Broker (2)

HEAT

HEAT

HEAT

HEAT

Let’s build a logging system with Eclipse MosquittoType cmd `git clone https://github.com/johnnylord/eclipse-mosquitto.git` and

follow the README instruction.

MQTT CPU Publisher

publisher.py

Connect to MQTT Broker

Get system CPU usage

Publish CPU usage to MQTT broker

MQTT Subscriber

subscriber.py

Connect to MQTT Broker

Subscribe topics of interest

Callback function when recive message from MQTT

broker

Homework (5%): Mixture of Communication PatternYou have to implement a system with a simple fibonacci caculator capability and a logging feature as following:

Server RequirementThe server is a backend web server implemented with the python Django

Framework. It must exports two REST Interfaces:

1. [POST] - /rest/fibonacci/

Client can launch POST request to ask for the result of fibonacci at N

order. The body content of POST is like { “order”: 10 }

2. [GET] - /rest/logs/

Client can launc GET request and get a list of history fibonacci requests

sent before. The result will be something like { “history”: [ 10, 5, …, 1 ] }

Fibonacci RequirementYou have to implement a fibonacci caculator service with gRPC interface. The

caculator will accept an integer number from the server, and return the

caculated result to the server. Following is the formula of fibonacci function:

Logging Requirement:The logging module will save the sent fibonacci order from the client request

through the subscribed topic on the message broker.

Whenever the server asks for the history information of fibonacci requests,

the logging module will respond the server with all the history of fibonacci

requests containing the order values.

Handover rules (Deadline: 11/17)Please open a new repo on github, and finish the homework requirements. Fill your repo’s url into https://docs.google.com/spreadsheets/d/1HHp0L4ObcaZAn29K8CozpH3JXVHJLZG7j3EPqhUTBSg/edit?usp=sharing

1. You need to provide README to tell TA how to run your code2. A recorded video demonstrates the workflow of how the system (1)

process the fibonacci request, and (2) how the logging module logs all the history request at runtime.

(YOU NEED TO OPEN SEVREAL TERMINALS, AND SHOW THE SYSTEM WORKS IN RUNTIME)