Presentation v2

Embed Size (px)

Text of Presentation v2

  • Human Algorithmic Stabilityand Human Rademacher Complexity

    European Symposium on Artificial Neural Networks,Computational Intelligence and Machine Learning

    Mehrnoosh Vahdat, Luca Oneto, Alessandro Ghio,Davide Anguita, Mathias Funk and Matthias Rauterberg

    University of Genoa

    DIBRISSmartLab

    April 22-24, 2014luca.oneto@unige.it - www.lucaoneto.com

    http://www.unige.ithttp://www.unige.ithttp://www.dibris.unige.ithttp://www.dibris.unige.ithttp://www.smartlab.wshttp://www.smartlab.wsmailto:luca.oneto@unige.ithttps://www.lucaoneto.com

  • Outline

    Goal of Our Work

    Introduction

    Machine LearningRademacher ComplexityAlgorithmic Stability

    Human LearningRademacher ComplexityAlgorithmic Stability

    Experimental Design

    Results

    Conclusions

  • Outline

    Goal of Our Work

    Introduction

    Machine LearningRademacher ComplexityAlgorithmic Stability

    Human LearningRademacher ComplexityAlgorithmic Stability

    Experimental Design

    Results

    Conclusions

  • Goal of Our Work

    In Machine Learning, the learning process of an algorithm given a set ofevidences is studied via complexity measures.

    Algorithmic Stability Human Stability

  • Outline

    Goal of Our Work

    Introduction

    Machine LearningRademacher ComplexityAlgorithmic Stability

    Human LearningRademacher ComplexityAlgorithmic Stability

    Experimental Design

    Results

    Conclusions

  • Introduction

    Exploring the way humans learn is a major interest in Learning Analytics andEducational Data Mining1

    Recent works in cognitive psychology highlight how cross-fertilization betweenMachine Learning and Human Learning can also be widened to the extents ofhow people tackle new problems and extract knowledge from observations 2

    Measuring the ability of a human to capture information rather than simplymemorizing is a way to to optimize Human Learning

    In a recent work is proposed the application of Rademacher Complexity approachto estimate human capability of extracting knowledge (Human RademacherComplexity) 3

    As an alternative, we propose to exploit Algorithmic Stability in the HumanLearning framework to compute the Human Algorithmic Stability 4

    Human Rademacher Complexity and Human Algorithmic Stability, is measured byanalyzing the way a group of students learns tasks of different difficulties

    Results show that using Human Algorithmic Stability leads to beneficial outcomesin terms of value of the performed analysis

    1Z. Papamitsiou and A. Economides. Learning analytics and educational data mining in practice: A systematicliterature review of empirical evidence. Educational Technology & Society, 17(4):49-64, 2014.

    2N. Chater and P. Vitanyi. Simplicity: A unifying principle in cognitive science? Trends in cognitive sciences,7(1):19-22, 2003.

    3X. Zhu, B. R. Gibson, and T. T. Rogers. Human rademacher complexity. In Neural Information ProcessingSystems, pages 2322-2330, 2009.

    4O. Bousquet and A. Elisseeff. Stability and generalization. The Journal of Machine Learning Research,2:499-526, 2002.

  • Outline

    Goal of Our Work

    Introduction

    Machine LearningRademacher ComplexityAlgorithmic Stability

    Human LearningRademacher ComplexityAlgorithmic Stability

    Experimental Design

    Results

    Conclusions

  • Machine Learning

    Machine Learning studies the ability of an algorithm to capture information rather thansimply memorize it.5

    Let X and Y {1} be, respectively, an input and an output space. We consider msets Dm = {S1n , . . . ,Smn } of n labeled i.i.d. data S

    jn : {Z

    j1, . . . ,Z

    jn} (j = 1, ...,m), where

    Z ji{1,...,n} = (Xji ,Y

    ji ), where X

    ji X and Y

    ji Y, sampled from an unknown

    distribution . A learning algorithm A is used to train a model f : X Y F based onS jn. The accuracy of the selected model is measured with reference to `(, ) (the hardloss function) which counts the number of misclassified examples.

    Several approaches6 in the last decades dealt with the development of measures toassess the generalization ability of learning algorithms, in order to minimize risks ofoverfitting.

    5V. N. Vapnik. Statistical learning theory. Wiley-Interscience, 1998.6O. Bousquet, S. Boucheron, and G. Lugosi. Introduction to statistical learning theory. Advanced Lectures on

    Machine Learning. Springer Berlin Heidelberg, 2004

  • Outline

    Goal of Our Work

    Introduction

    Machine LearningRademacher ComplexityAlgorithmic Stability

    Human LearningRademacher ComplexityAlgorithmic Stability

    Experimental Design

    Results

    Conclusions

  • Rademacher Complexity7

    Rademacher Complexity (RC) is anow-classic measure:

    Rn(F) =1m

    mj=1

    [1 inf

    fF

    2n

    ni=1

    `(f , (X ji , ji ))

    ]

    ji (with i {1, . . . , n} and j {1, . . . ,m})

    are 1 valued random variable for whichP[ji = +1] = P[

    ji = 1] = 1/2.

    7P. L. Bartlett and S. Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results.The Journal of Machine Learning Research, 3:463-482, 2003.

  • Outline

    Goal of Our Work

    Introduction

    Machine LearningRademacher ComplexityAlgorithmic Stability

    Human LearningRademacher ComplexityAlgorithmic Stability

    Experimental Design

    Results

    Conclusions

  • Algorithmic Stability8,9

    In order to measure Algorithmic Stability,we define the sets S j\in = S

    jn \ {Z

    ji }, where

    the i-th sample is removed. AlgorithmicStability is computed as

    Hn (A) =1m

    mj=1

    |`(AS jn,Z jn+1) `(AS j\in

    ,Z jn+1)|.

    8O. Bousquet and A. Elisseeff. Stability and generalization. The Journal of Machine Learning Research,2:499-526, 2002.

    9L. Oneto, A. Ghio, S. Ridella, and D. Anguita. Fully empirical and data-dependent stability-based bounds. IEEETransactions on Cybernetics, 10.1109/TCYB.2014.2361857 :in-press, 2014.

  • Outline

    Goal of Our Work

    Introduction

    Machine LearningRademacher ComplexityAlgorithmic Stability

    Human LearningRademacher ComplexityAlgorithmic Stability

    Experimental Design

    Results

    Conclusions

  • Human Learning

    Inquiry-guided HL focuses on contexts where learners are meant to discoverknowledge rather than passively memorizing10,11: the instruction begins with a set ofobservations to interpret, and the learner tries to analyze the data or solve the problemby the help of the guiding principles12, resulting in a more effective educationalapproach13.

    Measuring the ability of a human to capture information rather than simply memorizingis thus key to optimize HL.

    10A. Kruse and R. Pongsajapan. Student-centered learning analytics. In CNDLS Thought Papers, 2012.11V. S. Lee. What is inquiry-guided learning? New directions for teaching and learning, 2012(129):5-14, 2012.12V. S. Lee. The power of inquiry as a way of learning. Innovative Higher Education, 36(3):149-160, 2011.13T. DeJong, S. Sotiriou, and D. Gillet. Innovations in stem education: the go-lab federation of online labs. Smart

    Learning Environments, 1(1):1-16, 2014.

  • Outline

    Goal of Our Work

    Introduction

    Machine LearningRademacher ComplexityAlgorithmic Stability

    Human LearningRademacher ComplexityAlgorithmic Stability

    Experimental Design

    Results

    Conclusions

  • Rademacher Complexity

    A previous work14 depicts an experiment targeted towards identifying the HumanRademacher Complexity, which is defined as a measure of the capability of ahuman to learn noise, thus avoiding to overfit data.

    Unfortunately Rademacher Complexity has many disadvantages: We need to fix, a priori the class of function (which for a Human it may not even

    exist) We have to make the hypothesis that every Human performs always at his best

    (always find the best possible solution that is able to find) We have to discard the labels (different tasks with different difficulty levels, in a

    fixed domain, results in the same Human Rademacher Complexity)

    14X. Zhu, B. R. Gibson, and T. T. Rogers. Human rademacher complexity. In Neural Information ProcessingSystems, pages 2322-2330, 2009.

  • Outline

    Goal of Our Work

    Introduction

    Machine LearningRademacher ComplexityAlgorithmic Stability

    Human LearningRademacher ComplexityAlgorithmic Stability

    Experimental Design

    Results

    Conclusions

  • Algorithmic Stability

    We designed and carried out a new experiment, with the goal of estimating the averageHuman Rademacher Complexity and Human Algorithmic Stability for a class ofstudents: the latter is defined as a measure of the capability of a learner tounderstand a phenomenon, even when he is given slightly perturbedobservations.

    In order to measure the Human Algorithmic Stabilitywe do not need to make any hypothesis.

  • Outline

    Goal of Our Work

    Introduction

    Machine LearningRademacher ComplexityAlgorithmic Stability

    Human LearningRademacher ComplexityAlgorithmic Stability

    Experimental Design

    Results

    Conclusions

  • Experimental Design (I/V)

    Shape Problem

    Two problems: Shape Simple: (Thin) vs (Fat) Shape Difficult: (Very Thin + Very Fat) vs (Middle)

  • Experimental Design (II/V)

    Word Problem: Wisconsin Perceptual Attribute Ratings Database15

    Word Simple:(Long) vs (Short) pet cake cafe honeymoon harmonica handshake

    Word Difficult:(Positive Emotion) vs (Negative Emotion) rape killer funeral fun laughter joy

    15D