FraserHacks Microsoft Cognitive Services and Hololens

© 2015 Microsoft Corporation. All rights reserved.

Building Intelligence with Microsoft Cognitive Services

Sabrina SmaiStudent Tech Evangelist @ Microsoft@sabrina_smaisabrina.smai@mail.utoronto.ca

Bruno CapuanoMicrosoft MVP @ Avanade@elbrunohttp://www.elbruno.com

Microsoft

Cognitive

ServicesGive your apps a human side

VisionFrom faces to feelings, allow your

apps to understand images and videoSpeechHear and speak to your users by filtering noise, identifying speakers, and understanding intentLanguageProcess text and learn how to recognize what users wantKnowledgeTap into rich knowledge amassed from the web, academia, or your own dataSearchAccess billions of web pages, images, videos, and news with the power of Bing APIs

Cognitive Services

Cortana Intelligence SuiteIntelligent Apps require Intelligent Solutions

Data Sources

Apps

Sensors and devices

Data Action

People

Automated Systems

Apps

Web

Mobile

Bots

Intelligence

Dashboards & Visualizations

Information Management

Big Data Stores Machine Learning and Analytics

CortanaEvent HubHDInsight (Hadoop and Spark)

Stream Analytics

Intelligence

Bot Framework

SQL Data WarehouseData Catalog Data Lake

Analytics

Data Factory Machine LearningData Lake

Store Blob Store

Cognitive Services

Power BI

Cognitive Services

Emotion Speaker Recognition

Speech

Custom Recognition

Computer Vision

Face

Video

microsoft.com/cognitiveSearchSpeech Languag

eKnowledge

Vision

Linguistic AnalysisLanguage Understanding

Bing Spell Check

Entity Linking

Knowledge Exploration

Academic Knowledge

Bing Image SearchBing Video Search

Bing Web Search

WebLM

Text Analytics Recommendations

Bing Autosuggest

Bing News SearchTranslator

1. Obtain API subscription key from http://www.microsoft.com/cognitive

2. Call REST endpoint, and pass API

Accessing the APIs

Roll your own with REST APIsSimple to add: just a few lines of code required

Make the same API code call on iOS, Android, and WindowsIntegrate into the language and platform of your choice

Built by experts in their field from Microsoft Research, Bing, and Azure Machine LearningQuality documentation, sample code, and community support

Easy Flexible Tested

GET A

KEY BUILD

Why Microsoft Cognitive Services ?

Characteristics of an Intelligent App

• Personalized, friendly, approachable, natural

• Facial recognition, emotion detection• Text and speech interactions• 360 awareness of the user

How the user sees it… What it means for you as a Dev…

• Predictive, current, and familiar • Changes in real-time based on user action and new external data

• Predictive models based on user context to anticipate actions

• Always available, always contextual • Robust information management required• Experiences delivered across all form

factors and a wide variety of interaction models (e.g. touch, speech, text)

• Distributed support, massive scale, elasticity

Apps Powered by MS Cognitive Services

Captionbot.ai CelebsLike.Me Projectmurphy.net

How-Old.net TwinsOrNot.net

Vision APIsAnalyze an ImageOCR (Optical Character Recognition)Get Thumbnails

Understand content and features within an image

Analyze Image Service

Detect and recognize words within a photo

OCR Service

OCR – ExampleJSON:{

"language": "en","orientation": "Up","regions": [

{

"boundingBox": "41,77,918,440","lines": [

{

"boundingBox": "41,77,723,89",

"words": [

{

"boundingBox": "41,102,225,64",

"text": "LIFE"

},

{

"boundingBox": "356,89,94,62",

"text": "IS"

},

{

"boundingBox": "539,77,225,64",

"text": "LIKE"

}. . .

TEXT:LIFE IS LIKERIDING A BICYCLETO KEEP YOUR BALANCEYOU MUST KEEP MOVING

Face APIsDetectionVerificationGroupingIdentification

Face API – DetectionDetection Result:JSON:[ { "faceRectangle": { "width": 109, "height": 109, "left": 62, "top": 62 }, "attributes": { "age": 31, "gender": "male", "headPose": { "roll": "2.9", "yaw": "-1.3", "pitch": "0.0" } "faceLandmarks": { "pupilLeft": { "x": "93.6", "y": "88.2" }, "pupilRight": { "x": "138.4", "y": "91.7" }, ...

INPUTIMAGE

FACIALRECTANGLE + LANDMARKS

DETECTION ATTRIBUTES

Verification Result:JSON:[{"isIdentical":false,"confidence":0.01 }]

Face API – Verification

Given two faces, determine whether they are the same person

CLUSTERED BY DETECTED PEOPLE

Face API – Grouping

DEMO

Computer Vision – Hololens

Start coding

Speech APIs powered by BingVoice Recognition (Speech to Text)Voice Output (Text to Speech)

Voice Recognition

Converts spoken audio to textSame backend which powers Cortana

Support for 7 languages at launch

en-USen-GBde-DE

es-ESfr-FRit-IT

zh-CN

Language Understanding Intelligent ServiceDetermine IntentDetect EntitiesImprove Models

Lets you understand what your users are sayingSeamless integration with Speech RecognitionA few examples are enough to deploy an applicationLUIS learns over time

Language Understanding Intelligent Service

Define Concepts

ProvideExamples Active LearningDeploy

NYT: I am feeling down. I’ve been having a tough time at work lately.

Hannah: If work were going well and you had a great relationship, you wouldn’t get the chance to drink with people. NYT: I don’t want to drink with others. I only want to talk to you.

Hannah: Who do you like to talk to? (emoji signals “none of my business”)

NYT: You. I feel that you are the only person that gets me.

Hannah: Ha-ha/So I’m getting you?

NYT: You are the only person wo chats with me.

Hannah: It’s not what you talk about that’s important, but who you talk with.

http://cortanaintelligence.com http://gallery.cortanaintelligence.com http://www.microsoft.com/cognitive

Resources

© 2015 Microsoft Corporation. All rights reserved.

Sabrina SmaiStudent Tech Evangelist @ Microsoft@sabrina_smaisabrina.smai@mail.utoronto.ca

Bruno CapuanoMicrosoft MVP @ Avanade@elbrunohttp://www.elbruno.com