Eyes of things

1

This project has received funding from the European Union’s Horizon 2020 research and innovation

programme under Grant Agreement No 643924

Eyes of Things

2

Overview

• Horizon 2020, EU Framework Programme for

Research and Innovation

• 2014-2020

3

Overview

H2020 vs FP7

• More emphasis on the societal impact of the

research versus deepening of the knowledge

• More financial support for SMEs

• Higher total budget from 50 billion for FP7 to

around 80 billion for H2020

4

Overview

Eyes of Things, H2020 Innovation Action

• Topic: Smart cyber-physical systems

• Innovation-Type Action

R&I Action: “Novel solutions to old applications”

Innovation Action: “Novel applications with old

solutions”

• Started Jan 1st, 2015. Budget, 4M€

5

‘Eyes of Things’ Consortium

6

‘Eyes of Things’ Consortium

PARTNER TYPE ROLE/EXPERTISE

AWAIBA SME CAMERA

MOVIDIUS SME PROCESSOR

DFKI RESEARCH CENTER COMPUTER VISION

VISILAB UNIVERSITY COMPUTER VISION

THALES MULTINATIONAL SECURITY, VIDEO SURVEILLANCE

NVISO SME FACIAL ANALYSIS

FLUXGUIDE SME INTERACTIVE MUSEUM GUIDES

EVERCAM SME CLOUD-BASED APIs FOR CAMERAS

7

Motivation and Context

Vision is our richest sensor

One of the most complex tasks for both humans and

machines

Proven success in ‘machine vision’ context, i.e. factoryautomation, inspection,…

8


CV is going out-of-factory …• Microsoft Kinect

• Google Project Tango

• Google Goggles

• Google Glass

• Microsoft Hololens

• Facebook’s face recognition

• …

9


Lots of new developments• Drones• ADAS• Multispectral, 3D, …• Mobile imaging• Deep learning• …

10


“An Expanding Set of Applications Will Drive 183

Million Shipments of Computer Vision-Enabled Hardware

Annually by 2019”

Computer Vision Technologies and Markets, Tractica Report Q2 2015

11


• To date, most ‘out-of-the-factory’ applications involving hardware developed by big companies

• Many innovative ‘out-of-the-factory’ CV applications developed for smartphones…

12


smartphones are cheap, easy-to-use

and powerful !

NUDITY DETECTED

14


SAINET Project withSpain’s INDRA (second European IT services company)

14

15


•SAINET conclusion: new-platform needed

•Google Glass? No•Too expensive

•Too little battery time

•Don’t want screen

15

16


•SAINET conclusion: new-platform needed

•Others would agree… case of AIPOLY App

16

17


AIPOLY: App for the blind that uses CV and ML to

recognize what’s in a photo

Uses cloud processing

Feedback takes 5-20 secs

Currently in beta

18


“It’s not real-time. It’s not something you can

strap to your ear with a camera and it tells you

exactly what’s in front of you —yet. But there is

no reason why we should think that it won’t be

in the upcoming years”

Alberto Rizzoli, AIPOLY’s co-founder

19


• Most ‘out-of-the-factory’ applications involvinghardware developed by big companies

• Some innovative CV applications developed forsmartphones:

• Affordable

• Camera

• Connectivity

• Easy to program

• Easy to use

• But not appropriate for many applications

20


What are the challenges?

21


What are the challenges?

Size Cost Flexibility

22

The idea

Build a generic vision system that can be used standalone but also

embedded in more complex artifacts

BOM < $15

23

The idea

Optimize power consumption, size and cost

Demonstrate that it can be effectively used to develop

vision-based products

(4 demonstrators)

24

Similar devices?

Smart cameras

Raspberry PI, Arduino…

25

Similar devices?

•IoT devices:•WaRPboard, Ambarella, Intel EDISON, Ingenic

Newton2, Nixie…

26

Similar devices?

•Most of these systems process data from scalar sensors

•Some can be attached to cameras, but not do CV

•NONE has been designed from bottom-up with vision in mind !

27

Concept

Connect to EoT device from smartphone, tablet, PC

Upload application to the EoT device

Configure and interact with the application

28

Concept

Video streaming

Storage and processing in the cloud

29

Project organisation

WP1: Project management

WP5: Communication, Dissemination & Exploitation

Perceptual computing

Cloud computing

Augmented reality

Surveillance

WP4: Integration & Demonstration

T4.1: Demonstrator 1




WP2: Platform

HardwareWP3: Platform

Software

30

Project organisation

Roughly 1st half of project to develop platform

Then 4 companies will use it to develop 4 demonstrators

31

Demonstrator 1: Surveillance

• Peephole camera, allowing the owner to be alerted on his Smartphone with pictures and qualified events: movements, suspicious activities, etc.

• Leader: THALES

32

Demonstrator 2: Augmented reality

• Museum audio tour with automatic recognition of paintings in order to provide additional audio information

• Leader: Fluxguide

33

Demonstrator 3: Cloud computing

•Application of ‘Vision as a Service’ (VaaS)• Ex: “lifelogging” camera

•Leader: EVERCAM

34

Demonstrator 4: Perceptual computing

•Doll detecting and recognizing child’s emotion in order to react accordingly (with audio feedback)

•Leader: nViso

35

Main hardware elements

Movidius Myriad2

Optimum performance for CV vs power consumption, sizeand cost

36


Awaiba’s NanEyecamera

•World’s smallestdigital camera

•Power consumption< 10mW

•Disposable

37


Texas Instruments CC3100

•20x17 mm

•Low-cost, low-power

• Internet-on-a-chip: integrates all protocols

•Supports Station and Access Point modes

38

Prototypes

First prototype:

•Myriad2 development board•Daughterboard to expose

SPI•Level Shifter board•WiFi development board

39

Prototypes

First prototype

(another view)

40

Prototypes

Second prototype:

• Expected September 2015

• Interface to NanEye camera• Glue logic in FPGA

EoT DevBoardEoT DevBoard

WiFi Connector

Power

NanEye

JTAG Debug

41

Software

CV functions

MvCv

Middleware

RTOS

42

Software, CV functions

MvCv + …

• Rotation-invariant face detector

• Sparse optical flow (LK point tracking)

• Colour histogram matching

• Keypoint matching

• CNN runtime

43

Software, Middleware

• Wi-Fi data transfer

• Camera interface

• Video streaming

• Power management

• API middleware interpreter

• Middleware API Desktop and Android

• SD card management

• Audio output

• Input buttons/dipswitches

44




• Video streaming





• Audio output


45


•Wi-Fi data transfer

•For device configuration, application metadata, etc

•Using TCP/IP stack of Texas Instruments’ CC3100

•Common Internet protocols such as HTTP are notappropriate in IoT

•HTTP is text-oriented, not suitable for embedded

46


• In EoT we selected MQTT

• Open lightweight publish/subscribe protocol

• Efficient 1-to-n communication mechanism

47


Typical MQTT scenario

Problems:

1. The broker is in the

cloud (security risk)

2. It has to be leased

48


Our proposal: MQTT broker in the embedded device

Pros:

• Don’t need leased broker

• No security risk (unless you want to take it)

Cons:

• Broker is complex software

49


Mosquitto: The most popular (desktop) MQTT broker

3MB RAM

Our implementation for EoT device: Pulga (flea in Spanish)

512KB RAM (can be less)

50


51




• Video streaming





• Audio output


52


API middleware interpreter (for EoT device)

Middleware API (for Desktop and Android)

The API is a set of basic control functions forthe device

The API interpreter in EoT loads on boot(depending on a dipswitch)

53


The API:

• Get snapshot

• Work with SD card (save, load, list …)

• Upload and save application for device

• Run application

• Connect to existing WiFi network

•…

54




• Video streaming





• Audio output


55


• Minimal RTSP server implemented for EoT device

• Used by IP cameras

• Streaming only when requested

56

Software

First EoT streaming, July 2015

57

Ethics

EoT received very good (technical) reviews

BUT

Ethics screening raised someissues…

58

Ethics

“The proper provisions related to the involvement of children

must be in place”

59

Ethics

“Lack of awareness of the impact of the use of surveillance tools and combining

technologies (IoT, cloud, big data) and the risks of possible abuses, as lack of control over the data, stigmatization, profiling”

60

Ethics

“The problem of wearable computing, as in life-logging cameras, must also be addressed;

People are not aware that they are being recorded and they do not have the choice for

consent or opt-out”

61

Ethics

EoT Ethics Board

Prof. Dr. Petra Ahrweiler, Director of the

European Academy of Technology and

Innovation Assessment

Prof. Dr. José-Luis Piñar, Head of

the Google Chair on Privacy, Society

and Innovation

62

Conclusion

• Foreseen for September 2016

EoT open platform

• Foreseen end of 2017

Demonstrators

63

Follow us

• 9th International Conference on Distributed Smart Cameras, Seville, Spain, September

• First IEEE Workshop on Security and Privacy in the Cloud, Florence, Italy, September

• ICT 2015, Lisbon, Portugal, October

• 5th Int.Conf on Internet of Things, Seoul, South Korea, October

• Embedded Technology Conference, Yokohama, Japan, November

http://eyesofthings.eu/

64http://eyesofthings.eu/

65


Sample EoT workflow 1:

1. EoT creates Access Point

2. EoT Runs API interpreter and waits forcommands

3. User Connects to EoT device from Smartphone, Tablet..

66




2. EoT runs application previously stored in device

3. User connects to the EoT to configure and/ormonitor application

67




2. EoT Runs API interpreter and waits forcommands

3. I Connect to EoT device from my Smartphone, Tablet, PC

4. Issue command to connect to existing WiFi