Fabio Massimo Zanzotto and Danilo CroceUniversity of Rome “Tor Vergata”Roma, Italy

Reading what Machines ‘Think’

F.M.Zanzotto

University of Rome “Tor Vergata”

Prelude

Brain Activation

Pattern Recognizer

chair

Tom Mitchell, Invited Talk at the Association for Computational Linguistics Conference 2007

Mitchell, T.M., Shinkareva, S.V., Carlson, A., Chang, K.M., Malave, V.L., Mason, R.A., Just, M.A.: Predicting human brain activity associated with the meanings of nouns. Science 320(5880) (May 2008) 1191–1195

QuestionThis is a fascinating research problem. Can we find a more controlled setting where we can test if this is possible?

F.M.Zanzotto

University of Rome “Tor Vergata”

Idea

Cognitive physical object

Cognitive task

Observedimage

Observing a chair

Sorting a vector

BrainComputational

Machine

F.M.Zanzotto

University of Rome “Tor Vergata”

Why investigating the computer side is relevant?• Foundational perspective

– Computers are becoming extremely complex. They are fastly approaching the complexity of human brain

– Computers are controlled machines: their behavior and thier internal organization is known

– Then, computers offer a way to estimate if the claim on the brain side is reachable: if we can read what machines think, we can hope to read what brains think.

Motivation

F.M.Zanzotto

University of Rome “Tor Vergata”

Why investigating the computer side is relevant?• Applicative perspective

Can we develop technologies that “read the computer mind”? This predictive model can have a wide variety of applications, e.g., detecting malicious software, detecting the intentions of hostile computers by looking at their activation patterns.

Motivation

F.M.Zanzotto

University of Rome “Tor Vergata”

• Investigating the computer side: Long term research program

• Sketching the overall observation activity• Virtual Observation of Processes• Experimental Investigation

In the rest of the talk

F.M.Zanzotto

University of Rome “Tor Vergata”

Long-term research program

101010100101110111100001001010101001011101110101101001010101100000111010101010101111101010101110101101111110001101010100101110111100001001010101001011101110101101001010101100000111010101010101111101010101110101101111110000101010111010110111111000010101011101011011111100001010101110101101111110001…

011101111000010101010101

011101111000010101010101111101010101110101101111110001..

.

011101111000010101010101111101010101110101101111

110001...

Physical Memory Chip Physical Memory Dump Virtual Memory Dump(organized in processes)

Process Memory Dump

Physical device activation image

capturer

Virtual activation image

capturer

Virtual activation image

capturer

Virtual activation image

capturer

F.M.Zanzotto

University of Rome “Tor Vergata”

Sorting a vector

Sketching the overall observation activity

Brain Activation

Pattern Recognizer

chair

Sorting a vector

Process Activation

Pattern Recognizer

Building images from processes

Defining feature spaces for images

Observed Phenomena: Processes

F.M.Zanzotto

University of Rome “Tor Vergata”

Observed Phenomena: Processes

F.M.Zanzotto

University of Rome “Tor Vergata”

Given a cognitive activity, the procedure for extracting images from this activity is then the following:– running the process p representing the cognitive

activity c– stopping the process at given states or at given time

intervals – dumping the memory associated with the process M(p)– given a fixed height image and the memory dump, read

incrementally bytes of the memory dump and fill the associated RGB pixel with the read values I(M(p))

Building activation images from processes

F.M.Zanzotto

University of Rome “Tor Vergata”

Process memory in a given time interval

Activation image of the process in a given time interval

where

is the RGB pixel definition of the image

Building activation images from processes

][)( 0 mbbpM

kvv

k

pp

pppMI

,0,

,00,0

))((

][ 23133 h·j)(ih·j)(ih·j)(ii,j b bb p

F.M.Zanzotto

University of Rome “Tor Vergata”

Process: Vector Sorter Initial State

Process: Vector Sorter Final State

Building activation images from processesSmoothing

(more similar to real chip observation)

F.M.Zanzotto

University of Rome “Tor Vergata”

We used three major classes of features• Chromatic feaures

– Capture the color properties of the image determining, an n-dimensional vector representation of the 2D chromaticity histograms

• texture (OP - OGD) features– emphasize the background properties and their

composition.• transformation features (OGD)

Defining feature spaces for images

F.M.Zanzotto

University of Rome “Tor Vergata”

• Experimental Set-up– Collection of activation images– Used Machine Learning algorithms

• Experimental Results

Experimental Evaluation

F.M.Zanzotto

University of Rome “Tor Vergata”

• 3 different “cognitive tasks” (algorithms) – sorting, comparing two strings, visiting a binary tree

• 3 different programming languages – c, java, php

• for each pair algorithm-programming language– 20 different randomly generated input data– 3 snapshots (beginning, middle, end)

Experimental Set-up

F.M.Zanzotto

University of Rome “Tor Vergata”

We randomly splited the final set (540 images) in:• Training: 270 images Testing: 270 imagesTwo classification tasks:• Determining the programming language (3 classes) (lang) • Determining the cognitive task (3 classes) (algo) Used Machine learning Models:• Decision Tree Learners (DecTree)• Naive Bayes

Experimental Set-up

F.M.Zanzotto

University of Rome “Tor Vergata”

Results

Classification accuracy

F.M.Zanzotto

University of Rome “Tor Vergata”

The parallelism between computer and brain/mind is not new in general– Cognitive psychology– Cognitive sciences

We looked this parallelism from an other perspective

Conclusion

F.M.Zanzotto

University of Rome “Tor Vergata”

Future Work

101010100101110111100001001010101001011101110101101001010101100000111010101010101111101010101110101101111110001101010100101110111100001001010101001011101110101101001010101100000111010101010101111101010101110101101111110000101010111010110111111000010101011101011011111100001010101110101101111110001…

011101111000010101010101

011101111000010101010101111101010101110101101111110001..

.

011101111000010101010101111101010101110101101111

110001...

Physical Memory Chip Physical Memory Dump Virtual Memory Dump(organized in processes)

Process Memory Dump

Physical device activation image

capturer

Virtual activation image

capturer

Virtual activation image

capturer

Virtual activation image

capturer

F.M.Zanzotto

University of Rome “Tor Vergata”

Thank you for the attention!