- Home
- Data & Analytics
*Presentation v2*

prev

next

out of 38

View

105Category

## Data & Analytics

Embed Size (px)

Human Algorithmic Stabilityand Human Rademacher Complexity

European Symposium on Artificial Neural Networks,Computational Intelligence and Machine Learning

Mehrnoosh Vahdat, Luca Oneto, Alessandro Ghio,Davide Anguita, Mathias Funk and Matthias Rauterberg

University of Genoa

DIBRISSmartLab

April 22-24, 2014luca.oneto@unige.it - www.lucaoneto.com

http://www.unige.ithttp://www.unige.ithttp://www.dibris.unige.ithttp://www.dibris.unige.ithttp://www.smartlab.wshttp://www.smartlab.wsmailto:luca.oneto@unige.ithttps://www.lucaoneto.com

Outline

Goal of Our Work

Introduction

Machine LearningRademacher ComplexityAlgorithmic Stability

Human LearningRademacher ComplexityAlgorithmic Stability

Experimental Design

Results

Conclusions

Outline

Goal of Our Work

Introduction

Machine LearningRademacher ComplexityAlgorithmic Stability

Human LearningRademacher ComplexityAlgorithmic Stability

Experimental Design

Results

Conclusions

Goal of Our Work

In Machine Learning, the learning process of an algorithm given a set ofevidences is studied via complexity measures.

Algorithmic Stability Human Stability

Outline

Goal of Our Work

Introduction

Machine LearningRademacher ComplexityAlgorithmic Stability

Human LearningRademacher ComplexityAlgorithmic Stability

Experimental Design

Results

Conclusions

Introduction

Exploring the way humans learn is a major interest in Learning Analytics andEducational Data Mining1

Recent works in cognitive psychology highlight how cross-fertilization betweenMachine Learning and Human Learning can also be widened to the extents ofhow people tackle new problems and extract knowledge from observations 2

Measuring the ability of a human to capture information rather than simplymemorizing is a way to to optimize Human Learning

In a recent work is proposed the application of Rademacher Complexity approachto estimate human capability of extracting knowledge (Human RademacherComplexity) 3

As an alternative, we propose to exploit Algorithmic Stability in the HumanLearning framework to compute the Human Algorithmic Stability 4

Human Rademacher Complexity and Human Algorithmic Stability, is measured byanalyzing the way a group of students learns tasks of different difficulties

Results show that using Human Algorithmic Stability leads to beneficial outcomesin terms of value of the performed analysis

1Z. Papamitsiou and A. Economides. Learning analytics and educational data mining in practice: A systematicliterature review of empirical evidence. Educational Technology & Society, 17(4):49-64, 2014.

2N. Chater and P. Vitanyi. Simplicity: A unifying principle in cognitive science? Trends in cognitive sciences,7(1):19-22, 2003.

3X. Zhu, B. R. Gibson, and T. T. Rogers. Human rademacher complexity. In Neural Information ProcessingSystems, pages 2322-2330, 2009.

4O. Bousquet and A. Elisseeff. Stability and generalization. The Journal of Machine Learning Research,2:499-526, 2002.

Outline

Goal of Our Work

Introduction

Machine LearningRademacher ComplexityAlgorithmic Stability

Human LearningRademacher ComplexityAlgorithmic Stability

Experimental Design

Results

Conclusions

Machine Learning

Machine Learning studies the ability of an algorithm to capture information rather thansimply memorize it.5

Let X and Y {1} be, respectively, an input and an output space. We consider msets Dm = {S1n , . . . ,Smn } of n labeled i.i.d. data S

jn : {Z

j1, . . . ,Z

jn} (j = 1, ...,m), where

Z ji{1,...,n} = (Xji ,Y

ji ), where X

ji X and Y

ji Y, sampled from an unknown

distribution . A learning algorithm A is used to train a model f : X Y F based onS jn. The accuracy of the selected model is measured with reference to `(, ) (the hardloss function) which counts the number of misclassified examples.

Several approaches6 in the last decades dealt with the development of measures toassess the generalization ability of learning algorithms, in order to minimize risks ofoverfitting.

5V. N. Vapnik. Statistical learning theory. Wiley-Interscience, 1998.6O. Bousquet, S. Boucheron, and G. Lugosi. Introduction to statistical learning theory. Advanced Lectures on

Machine Learning. Springer Berlin Heidelberg, 2004

Outline

Goal of Our Work

Introduction

Machine LearningRademacher ComplexityAlgorithmic Stability

Human LearningRademacher ComplexityAlgorithmic Stability

Experimental Design

Results

Conclusions

Rademacher Complexity7

Rademacher Complexity (RC) is anow-classic measure:

Rn(F) =1m

mj=1

[1 inf

fF

2n

ni=1

`(f , (X ji , ji ))

]

ji (with i {1, . . . , n} and j {1, . . . ,m})

are 1 valued random variable for whichP[ji = +1] = P[

ji = 1] = 1/2.

7P. L. Bartlett and S. Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results.The Journal of Machine Learning Research, 3:463-482, 2003.

Outline

Goal of Our Work

Introduction

Machine LearningRademacher ComplexityAlgorithmic Stability

Human LearningRademacher ComplexityAlgorithmic Stability

Experimental Design

Results

Conclusions

Algorithmic Stability8,9

In order to measure Algorithmic Stability,we define the sets S j\in = S

jn \ {Z

ji }, where

the i-th sample is removed. AlgorithmicStability is computed as

Hn (A) =1m

mj=1

|`(AS jn,Z jn+1) `(AS j\in

,Z jn+1)|.

8O. Bousquet and A. Elisseeff. Stability and generalization. The Journal of Machine Learning Research,2:499-526, 2002.

9L. Oneto, A. Ghio, S. Ridella, and D. Anguita. Fully empirical and data-dependent stability-based bounds. IEEETransactions on Cybernetics, 10.1109/TCYB.2014.2361857 :in-press, 2014.

Outline

Goal of Our Work

Introduction

Machine LearningRademacher ComplexityAlgorithmic Stability

Human LearningRademacher ComplexityAlgorithmic Stability

Experimental Design

Results

Conclusions

Human Learning

Inquiry-guided HL focuses on contexts where learners are meant to discoverknowledge rather than passively memorizing10,11: the instruction begins with a set ofobservations to interpret, and the learner tries to analyze the data or solve the problemby the help of the guiding principles12, resulting in a more effective educationalapproach13.

Measuring the ability of a human to capture information rather than simply memorizingis thus key to optimize HL.

10A. Kruse and R. Pongsajapan. Student-centered learning analytics. In CNDLS Thought Papers, 2012.11V. S. Lee. What is inquiry-guided learning? New directions for teaching and learning, 2012(129):5-14, 2012.12V. S. Lee. The power of inquiry as a way of learning. Innovative Higher Education, 36(3):149-160, 2011.13T. DeJong, S. Sotiriou, and D. Gillet. Innovations in stem education: the go-lab federation of online labs. Smart

Learning Environments, 1(1):1-16, 2014.

Outline

Goal of Our Work

Introduction

Machine LearningRademacher ComplexityAlgorithmic Stability

Human LearningRademacher ComplexityAlgorithmic Stability

Experimental Design

Results

Conclusions

Rademacher Complexity

A previous work14 depicts an experiment targeted towards identifying the HumanRademacher Complexity, which is defined as a measure of the capability of ahuman to learn noise, thus avoiding to overfit data.

Unfortunately Rademacher Complexity has many disadvantages: We need to fix, a priori the class of function (which for a Human it may not even

exist) We have to make the hypothesis that every Human performs always at his best

(always find the best possible solution that is able to find) We have to discard the labels (different tasks with different difficulty levels, in a

fixed domain, results in the same Human Rademacher Complexity)

14X. Zhu, B. R. Gibson, and T. T. Rogers. Human rademacher complexity. In Neural Information ProcessingSystems, pages 2322-2330, 2009.

Outline

Goal of Our Work

Introduction

Machine LearningRademacher ComplexityAlgorithmic Stability

Human LearningRademacher ComplexityAlgorithmic Stability

Experimental Design

Results

Conclusions

Algorithmic Stability

We designed and carried out a new experiment, with the goal of estimating the averageHuman Rademacher Complexity and Human Algorithmic Stability for a class ofstudents: the latter is defined as a measure of the capability of a learner tounderstand a phenomenon, even when he is given slightly perturbedobservations.

In order to measure the Human Algorithmic Stabilitywe do not need to make any hypothesis.

Outline

Goal of Our Work

Introduction

Machine LearningRademacher ComplexityAlgorithmic Stability

Human LearningRademacher ComplexityAlgorithmic Stability

Experimental Design

Results

Conclusions

Experimental Design (I/V)

Shape Problem

Two problems: Shape Simple: (Thin) vs (Fat) Shape Difficult: (Very Thin + Very Fat) vs (Middle)

Experimental Design (II/V)

Word Problem: Wisconsin Perceptual Attribute Ratings Database15

Word Simple:(Long) vs (Short) pet cake cafe honeymoon harmonica handshake

Word Difficult:(Positive Emotion) vs (Negative Emotion) rape killer funeral fun laughter joy

15D

Recommended

View more >