A Semantics-based Approach to Machine Perception

A SEMANTICS-BASED APPROACHTO MACHINE PERCEPTION

Cory Andrew Henson

August 27, 2013

1

Committee: Amit Sheth (advisor)

Krishnaprasad Thirunarayan John Gallagher

Payam Barnaghi Satya Sahoo

Ph.D. Dissertation Defense

Wright State UniversityDept. of Computer Science and Engineering

2

3

Thesis

Machine perception can be formalized using semantic web technologies in order to derive abstractions from sensor data using background knowledge on the Web, and efficiently executed on resource-constrained devices.

4

3 primary issues to be addressed

Annotation of sensor data

SemanticSensor

Web

SemanticPerception

Intelligence

at the Edge

Interpretation of sensor data

Efficient execution onresource-constrained devices1 2 3

5

lives in

has pet

is ahas petPerson Animal

Concrete Facts Resource Description Framework

Semantic Web(according to Farside)

General Knowledge Web Ontology Language

“Now! – That should clear up a few things around here!”

is a

Government

Media

Publications

Life SciencesGeographic

Semantic Web

6

7http://www.opengeospatial.org/projects/groups/sensorwebdwg

Semantic Sensor Web

8

Sensor systems are too often stovepiped

9

With freedom comes responsibility1. discovery, access, and

search2. integration and

interpretation

We want to set this data free

10

OGC Sensor Web Enablement (SWE)

11

With freedom comes responsibility1. discovery, access, and

search2. integration and

interpretation

We want to set this data free

12

RDF OWL

How are machines supposed to integrate and interpret sensor data?

Semantic Sensor Networks (SSN)

13

W3C Semantic Sensor Network Ontology

Lefort, L., Henson, C., Taylor, K., Barnaghi, P., Compton, M., Corcho, O., Garcia-Castro, R., Graybeal, J., Herzog, A., Janowicz, K., Neuhaus, H., Nikolov, A., and Page, K.: Semantic Sensor Network XG Final Report, W3C Incubator Group Report (2011).

14

W3C Semantic Sensor Network Ontology

Lefort, L., Henson, C., Taylor, K., Barnaghi, P., Compton, M., Corcho, O., Garcia-Castro, R., Graybeal, J., Herzog, A., Janowicz, K., Neuhaus, H., Nikolov, A., and Page, K.: Semantic Sensor Network XG Final Report, W3C Incubator Group Report (2011).

15

W3C Semantic Sensor Network Ontology

Lefort, L., Henson, C., Taylor, K., Barnaghi, P., Compton, M., Corcho, O., Garcia-Castro, R., Graybeal, J., Herzog, A., Janowicz, K., Neuhaus, H., Nikolov, A., and Page, K.: Semantic Sensor Network XG Final Report, W3C Incubator Group Report (2011).

16

Semantic Annotation of SWE

Lefort, L., Henson, C., Taylor, K., Barnaghi, P., Compton, M., Corcho, O., Garcia-Castro, R., Graybeal, J., Herzog, A., Janowicz, K., Neuhaus, H., Nikolov, A., and Page, K.: Semantic Sensor Network XG Final Report, W3C Incubator Group Report (2011).

Semantic Sensor Observation Service (SemSOS)

Cory Henson, Josh Pschorr,Amit Sheth, Krishnaprasad Thirunarayan, SemSOS: Semantic Sensor Observation Service, In Proceedings of the 2009 International Symposium on Collaborative Technologies and Systems (CTS 2009), Baltimore, MD, May 18-22, 2009. 17

Semantic Sensor Observation Service (SemSOS)

Joshua Pschorr, Cory Henson, Harshal Patni, and Amit P. Sheth. Sensor Discovery on Linked Data. Kno.e.sis Center Technical Report 2010.

18

19

3 primary issues to be addressed

Annotation of sensor data

SemanticSensor

Web

SemanticPerception

Intelligence

at the Edge

Interpretation of sensor data

Efficient execution onresource-constrained devices1 2 3

20

Semantic Perception

• The role of perception is to transform raw sensory data into a meaningful and correct representation of the external world.

• The systematic automation of this ability is the focus of machine perception.

• For correct interpretation of this representation, we need a formal account of how this is done.

21

What can we learn from cognitive models of

perception?

People are good atmaking sense

of sensory input

22

Perception is an active, cyclical process of exploration and interpretation.

The perception cycle is driven by prior knowledge, in order to generate and test hypotheses.

Some observations are more informative than others (in order to effectively test hypotheses*).

Ulric Neisser

Richard GregoryKenneth Norwich

1970’s 1980’s 1990’s

Cognitive Models of Perception

* Applies to machine perception within Intellego, NOT human perception.

23

ObservedPropertie

s

PerceivedFeatures

Background knowledge

Explanation

Ontology of Perception

Focus

An Ontological Approach to Focusing Attention and Enhancing Machine Perception on the Web (Applied Ontology, 2011)

Example: Medical diagnosis as perception

24

Proactive,Preventative

Healthcare

25

26

The Patient of the Future

MIT Technology Review, 2012

http://www.technologyreview.com/featuredstory/426968/the-patient-of-the-future/

Digital Doctor

Let’s provide people with the tools needed to monitor and

manage their own health

http://worldofdtcmarketing.com/mobile-health-apps-a-new-opportunity-for-healthcare-marketers/mobile-healthcare-marketing-trends/ 27

Medical/healthcare expert systems have been around for a long time

28

1. Ubiquitous Sensing 2. Always-on Computing3. Knowledge on the

Web

3 recent developments have changed the technological landscape …

29

30

Making sense of sensor data with

DATAsensor observations

KNOWLEDGEsituation awareness useful

for decision making

Primary challenge is to bridge the gap between data and knowledge

31

SSNOntology

2 Interpreted data(deductive)[in OWL] e.g., threshold

1 Annotated Data[in RDF]e.g., label

0 Raw Data[in TEXT]e.g., number

Levels of Abstraction

3 Interpreted data (abductive)[in OWL]e.g., diagnosis

Intellego

“150”

Systolic blood pressure of 150 mmHg

ElevatedBlood

Pressure

Hyperthyroidism

less

use

ful …

…

mor

e us

eful

……

32

33

ObservedPropertie

s

PerceivedFeatures

Background knowledgeon the Web

Low-level observed properties suggest explanatory hypotheses through abduction

Explanation

Focus

Ontology of Perception

An Ontological Approach to Focusing Attention and Enhancing Machine Perception on the Web (Applied Ontology, 2011)

34

Semantics of Explanation

Abduction – or, inference to the best EXPLANATION

Task• Given background knowledge of the environment (SIGMA), and• given a set of sensor observation data (RHO),• find a consistent explanation of the situation (DELTA)

Backgroundknowledge Features

(objects/events)in the world

Sensor observation

data

35

Semantics of Explanation

Background knowledge is represented as a causal network between features (objects or events) in the world and the sensor observations they give rise to.

36

Semantics of Explanation

Finding the sweet spot between abduction and OWL

• Simulation of Parsimonious Covering Theory in OWL-DL (using the single-feature assumption*)

* An explanation must be a single feature which

accounts forall observed properties

Semantic Perception: Converting Sensory Observations to Abstractions (IEEE Internet Computing, 2012)

Theorem: Given a PCT problem P and its translation o(P) into OWL, Δ = {e} is a PCT explanation if and only if ExplanatoryFeature(e) is deduced by an OWL-DL reasoner — that is, if and only if o(P) ⊧ ExplanatoryFeature(e).

37

Finding the Sweet Spot

minimizeexplanations

degrade gracefullyw/ incomplete info

decidable

web reasoning

Abductive Logic (e.g., PCT)high complexity

Deductive Logic (e.g., OWL)(relatively) low complexity

Explanation(in Intellego)

38

Explanatory Feature: A feature is explanatory w.r.t. a set of observed properties if it causes each property in the set.

ExplanatoryFeature ≡ isPropertyOf∃ —.{p1} … isPropertyOf⊓ ⊓ ∃ —.{pn}

elevated blood pressure

clammy skin

palpitations

Hypertension

Hyperthyroidism

Pulmonary Edema

Observed Property Explanatory Feature

Semantics of Explanation

39

ObservedPropertie

s

PerceivedFeatures

Background knowledgeon the Web

Hypotheses imply the informational value of future observations through deduction

Explanation

Focus

Ontology of Perception

An Ontological Approach to Focusing Attention and Enhancing Machine Perception on the Web (Applied Ontology, 2011)

40

Universe of observable properties

Semantics of Focus

To predict which future observations have informational value, find those observable properties that can discriminate between the set of hypotheses.

ExpectedProperties

Not-applicableProperties

Discriminating

Properties

41

Expected Property: A property is expected w.r.t. a set of features if it is caused by each feature in the set.

ExpectedProperty ≡ isPropertyOf.{f∃ 1} … isPropertyOf.{f⊓ ⊓ ∃ n}

elevated blood pressure

clammy skin

palpitations

Hypertension

Hyperthyroidism

Pulmonary Edema

Expected Property Explanatory Feature

Semantics of Focus

42

Not Applicable Property: A property is not-applicable w.r.t. a set of features if it is not caused by any feature in the set.

NotApplicableProperty ≡ ¬∃isPropertyOf.{f1} … ¬⊓ ⊓ ∃isPropertyOf.{fn}

elevated blood pressure

clammy skin

palpitations

Hypertension

Hyperthyroidism

Pulmonary Edema

Not Applicable Property Explanatory Feature

Semantics of Focus

43

Discriminating Property: A property is discriminating w.r.t. a set of features if it is neither expected nor not-applicable.

DiscriminatingProperty ≡ ¬ExpectedProperty ¬NotApplicableProperty⊓

elevated blood pressure

clammy skin

palpitations

Hypertension

Hyperthyroidism

Pulmonary Edema

Discriminating Property

Explanatory Feature

Semantics of Focus

44

Off-the-shelf OWL-DL reasoners are too resource intensive in terms of both memory and time

• Runs out of resources with background knowledge >> 20 nodes

• Asymptotic complexity: O(n3)

O(n3) < x < O(n4)

Semantic perception on resource-constrained devices

45

3 primary issues to be addressed

Annotation of sensor data

SemanticSensor

Web

SemanticPerception

Intelligence

at the Edge

Interpretation of sensor data

Efficient execution onresource-constrained devices1 2 3

Internet of Things

46http://www.idgconnect.com/blog-abstract/900/the-internet-things-breaking-down-barriers-connected-world

47http://share.cisco.com/internet-of-things.html

48

Basis Watch• Heart Rate Monitor• Accelerometer• Skin Temperature• Galvanic Skin Response

Homo Digitus

49

How do we make sense of this data … and do it efficiently and at scale?

50

Approach 1: Send all sensor observations to the cloud for processing

Approach 2: downscale semantic processing so that each device is capable of machine perception

Intelligence at the Edge

51

Use bit vector encodings and their operations to encode background knowledge and execute perceptual inference

Efficient execution of semantic perception

0110001110001110110011100101011001110101

OWL-DL

An Efficient Bit Vector Approach to Semantics-based Machine Perception in Resource-Constrained Devices (ISWC, 2012)

52

lift

lower

Translate background knowledge, observations, and explanations between Semantic Web and bit vector representation

Lifting and lowering of knowledge

53

Efficient execution of semantic perception

Bit vector algorithms are provably equivalent to the OWL inference (i.e., semantics preserving)

Intuition: discover and dismiss those features that cannot explain the set of observed properties.

Intuition: discover and assemble those properties that discriminate between the explanatory features

54

bp 1

cs 0

pa 1

HN HM PE

bp 1 1 1

cs 0 1 0

pa 1 1 0

HN HM PE

1 1 1

HN HM PE

1 1 1

1 1 0

AND =>

AND

1 1 1

ObservedProperty Prior Knowledge

PreviousExplanatory Feature

CurrentExplanatory Feature

=>

INTUITION: The strategy employed relies on the use of the bit vector AND operation to discover and dismiss those features that cannot explain the set of observed properties.

Explanation: efficient algorithm

55

bp 1

cs 0

pa 1

HN HM PE

bp 0 1 1

cs 0 1 0

pa 1 1 0

HN HM PE

1 1 0

HN HM PE

0 1 0AND => 0 1 0

ObservedProperty Prior Knowledge

PreviousExplanatory Feature

CurrentExplanatory Feature

=

INTUITION: The strategy employed relies on the use of the bit vector AND operation to discover and assemble those properties that discriminate between the explanatory features

bp 0

cs 0

pa 0

Discriminating

Property

1 1 0

… expected?

0 0 0… not-applicable?ZERO Bit Vector

0 1 0

=

=> FALSE

=> FALSE

1

Is the propertydiscriminating?

Focus: efficient algorithm

56

O(n3) < x < O(n4) O(n)

Evaluation on a mobile device

Efficiency Improvement

• Problem size increased from 10’s to 1000’s of nodes• Time reduced from minutes to milliseconds• Complexity growth reduced from polynomial to

linear

57

Technical contributions in a nutshell

1. Semantic Sensor Web: Developed technologies for the semantic annotation of sensor data on the Web

- Semantic Sensor Web (IEEE Internet Computing, 2008) – 276 citations (as of Aug. 2013)- SemSOS: Semantic Sensor Observation Service (International Symposium on Collaborative Technologies and

Systems, 2009)- Semantic Sensor Network XG Final Report (W3C Incubator Group Report, 2011)

2. Semantic Perception: Designed a declarative specification of perception, capable of utilizing an off-the-shelf OWL-DL reasoner

- An Ontological Approach to Focusing Attention and Enhancing Machine Perception on the Web (Applied Ontology Journal, 2011)

- Semantic Perception: Converting Sensory Observations to Abstractions (IEEE Internet Computing: Special Issue on Context-Aware Computing, 2012) – most downloaded paid paper in IEEE-IC 2012

3. Intelligence at the Edge: Implemented efficient algorithms for executing perceptual inference on resource-constrained devices

- An Efficient Bit Vector Approach to Semantics-based Machine Perception in Resource-Constrained Devices (International Semantic Web Conference, 2012)

58

W3C Reports1. Semantic Sensor Network XG Final Report (W3C Incubator Group Report, 2011)

Journal Publications2. Physical-Cyber-Social Computing: An Early 21st Century Approach (IEEE Intelligent Systems, 2013)3. Semantic Perception: Converting Sensory Observations to Abstractions (IEEE Internet Computing, 2012)4. Semantics for the Internet of Things: Early Progress and Back to the Future (International Journal on Semantic Web and Information Systems, 2012)5. The SSN ontology of the W3C semantic sensor network incubator group (Web Semantics: Science, Services and Agents on the World Wide Web, 2012)6. An Ontological Approach to Focusing Attention and Enhancing Machine Perception on the Web (Applied Ontology, 2011)7. Semantic Sensor Web (IEEE Internet Computing, 2008)

Conference Publications8. An Efficient Bit Vector Approach to Semantics-based Machine Perception in Resource-Constrained Devices (International Semantic Web Conference,

2012)9. Computing Perception from Sensor Data (IEEE Sensors Conference, 2012)10. SemSOS: Semantic Sensor Observation Service (International Symposium on Collaborative Technologies and Systems, 2009)11. Situation Awareness via Abductive Reasoning for Semantic Sensor Data: A Preliminary Report (International Symposium on Collaborative

Technologies and Systems, 2009).

Workshop Publications12. SECURE: Semantics Empowered Rescue Environment (International Workshop on Semantic Sensor Networks, 2011) 13. Representation of Parsimonious Covering Theory in OWL-DL (International Workshop on OWL: Experiences and Directions, 2011)14. Provenance Aware Linked Sensor Data (Workshop on Trust and Privacy on the Social and Semantic Web, 2010)15. Linked Sensor Data (International Symposium on Collaborative Technologies and Systems, 2010)16. A Survey of the Semantic Specification of Sensors (International Workshop on Semantic Sensor Networks, 2009)17. An Ontological Representation of Time Series Observations on the Semantic Sensor Web (International Workshop on the Semantic Sensor Web)18. Video on the Semantic Sensor Web (W3C Video on the Web Workshop, 2007)

Relevant Publications

59

Application

Proactive, preventative healthcare

Heart disease is a critical issue

~815,000 (2011)

http://millionhearts.hhs.gov/abouthds/cost-consequences.html 60

Acute Decompensated Heart Failure (ADHF)

• Affects nearly 6 million people (in

U.S.)

• 555,000 new cases are diagnosed

each year

61U.S. Department of Health & Human Services. (2011). Hospital Compare. http://www.hospitalcompare.hhs.gov (Accessed on February 19, 2012).

• 4.8 million hospitalizations per year

• 50% are readmitted within 6

months

• 25% are readmitted within 30 days

• 70% due to worsening conditions

• Costing $17 billion per year

ADHF hospital readmission rates are too high

62U.S. Department of Health & Human Services. (2011). Hospital Compare. http://www.hospitalcompare.hhs.gov (Accessed on February 19, 2012).

Congress has incentivized hospitals to lower readmission rates

63U.S. Department of Health & Human Services. (2011). Hospital Compare. http://www.hospitalcompare.hhs.gov (Accessed on February 19, 2012).

Current state-of-the-art

64

Score (0: Not at all, 1: A little, 2: A great deal, 3: Extremely)

Heart Failure Somatic Awareness Scale (HFSAS)

Current state-of-the-art

65

kHealth – knowledge-enabled healthcare

Approach: • Use semantic perception inference• with data from cardio-related sensors• and curated medical background knowledge on

the Web

1. to monitor and abstract health conditions2. to ask the patient contextually relevant

questions

66

Cardiology Background Knowledge

• Symptoms: 284

• Disorders: 173

• Causal Relations: 1944

Unified Medical Language System

Causal Network

67

kHealth Kit

Weight Scale

Heart Rate Monitor

Blood PressureMonitor

68

Sensors

Android Device (w/ kHealth App)

Total cost: < $500

69

Explanation in kHealth

• Abnormal heart rate• High blood pressure

• Panic Disorder• Hypoglycemia• Hyperthyroidism• Heart Attack• Septic Shock

Observed Property Explanatory Feature

via Bluetooth

70

Focus in kHealth

Are you feeling lightheaded?

Are you have trouble taking deep breaths?

yes

yes

• Abnormal heart rate• High blood pressure• Lightheaded• Trouble breathing

• Panic Disorder• Hypoglycemia• Hyperthyroidism• Heart Attack• Septic Shock

Contextually dependent questioning based on prior observations(from 284 possible questions)

Observed Property Explanatory Feature

71

Evaluation of kHealth

Evaluate the ability to discriminate between sets of potential disorders using:

1. HFSAS/WANDA’s restricted set of observable symptoms (12)

2. kHealth’s more comprehensive set of observable symptoms (284)

72

Evaluation of kHealth

Evaluation Metrics:1. Efficiency: How many observations (or questions) required to

minimize the set of explanations?2. Specificity: How specific is the resulting minimum set of

explanations (i.e., how many explanatory disorders in the set)?

Explanatory Disorders

(computed by Intellego)

Actual Disorder(extracted from EMR)

Possible Disorders(derived from cardiology

KB)

73

Evaluation of kHealth – Early Results

HFSAS/WANDAEfficiency: ~7.45 (# questions asked)Specificity: ~11.95* (# minimum

explanations)

* Converged to 1 explanation 20% of the time

• 496 EMRs• ~3.2 diagnosed

disorders per EMR

• 173 possible disorders

The approach utilized by kHealth is

more efficient and more specificthan HFSAS/WANDA.

kHealthEfficiency: ~7.28 (# questions asked)Specificity: 1 (# minimum explanations)

Explanatory Disorders

(computed by Intellego)

Actual Disorder(extracted from EMR)

Possible Disorders(derived from cardiology

KB)

Pre-clinical usability trial

Dr. William Abraham, M.D.Director of Cardiovascular

Medicine

74

Sensingand

Perception

HealthCare

Academic Standards

Org.

Industry Government

ResearchCollaborators

by Research Topicand Organization Type

75

Special Thanks to AFRL and DAGSI

76

AFRL/DAGSI Research Topic SN08-8: Architectures for Secure Semantic Sensor Networks for

Multi-Layered Sensing

77

Semantic Sensor Web Team

78

A SEMANTICS-BASED APPROACHTO MACHINE PERCEPTION

Cory Andrew Henson

August 27, 2013

79

Thank you.

For additional information visit: http://knoesis.org/researchers/cory