Introduction to reinforcement learning - Cornell University › info4940 › documents ›...

Introductionto

reinforcementlearning

Pantelis P.Analytis

Introduction

classical andoperantconditioning

Modelinghumanlearning

Ideas forsemesterprojects

Introduction to reinforcement learning

Pantelis P. Analytis

March 12, 2018

1 / 27

Introductionto

Pantelis P.Analytis

Introduction

1 Introduction

2 classical and operant conditioning

3 Modeling human learning

4 Ideas for semester projects

2 / 27

Introductionto

Pantelis P.Analytis

Introduction

What’s reinforcement learning?

3 / 27

Introductionto

Pantelis P.Analytis

Introduction

What’s reinforcement learning?

4 / 27

Introductionto

Pantelis P.Analytis

Introduction

classical conditioning

Conditioned stimulus (e.g. a sound) , unconditionedstimulus (e.g. the taste of food), unconditioned response(unlearned behavior such as salivation).

5 / 27

Introductionto

Pantelis P.Analytis

Introduction

Behaviorism in psychology

Psychology was under the grip of behaviorism from the20s to the 60s.

Focus on expressed behavior rather than on psychologicalprocesses.

6 / 27

Introductionto

Pantelis P.Analytis

Introduction

The Rescola-Wanger model

∆V n+1X = αXβ(λ− Vtot)

V n+1X = V n

X + ∆V n+1X

∆VX is the change in the strength, on a single trial, of theassociation between the CS labelled ”X” and the US

α is the salience of X (bounded by 0 and 1)

β is the rate parameter for the US (bounded by 0 and 1),sometimes called its association value

λ is the maximum conditioning possible for the US

VX is the current associative strength of X

Vtot is the total associative strength of all stimuli present,that is, X plus any others

7 / 27

Introductionto

Pantelis P.Analytis

Introduction

The Rescola-Wanger model: predictions

The model captures acquisition and extinction ofassociations through a process of surprise. First model toincorporate several cues.

Importantly, the model captures interactions between cues.One cue may block the association of another with theUS. Extinction might not occur if an inhibitor is there.

Over time the model converges to optimal least squareweights.

Examples: Blocking, overshadowing and weakening ofstimuli.

8 / 27

Introductionto

Pantelis P.Analytis

Introduction

The first learning experiments

Thorndike studied the time that animals took to escapefrom his illustrious box.

9 / 27

Introductionto

Pantelis P.Analytis

Introduction

Thorndike’s law of effect

Thorndike s law of effect: Of several responses made to thesame situation, those which are accompanied or closelyfollowed by satisfaction to the animal will, other things beingequal, be more firmly connected with the situation, so that,when it recurs, they will be more likely to recur; those whichare accompanied or closely followed by discomfort to theanimal will, other things being equal, have their connectionswith that situation weakened, so that, when it recurs, they willbe less likely to occur. The greater the satisfaction ordiscomfort, the greater the strengthening or weakening of thebond (Thorndike, 1911, p.244).

10 / 27

Introductionto

Pantelis P.Analytis

Introduction

Learned helplessness

The organisms learn that it is impossible to escape, andeven when the hindrance is removed they do not attemptto escape.

11 / 27

Introductionto

Pantelis P.Analytis

Introduction

The first learning experiments

Operant conditioning can be described as a process thatattempts to modify behavior through the use of positiveand negative reinforcement. Through operantconditioning, an individual makes an association between aparticular behavior and a consequence (Skinner, 1938).

12 / 27

Introductionto

Pantelis P.Analytis

Introduction

Tolman’s cognitive maps

3 groups of rats, running in a maze for 17 days.

one group got a reward, the second got no reward, thethird got a reward on the 11th day.

13 / 27

Introductionto

Pantelis P.Analytis

Introduction

Implicit learning

The group that was rewarded only on the 11th dayimproved rapidly and surpassed in terms of performancethe group that was rewarded from the beginning.

14 / 27

Introductionto

Pantelis P.Analytis

Introduction

There are two strategies to solve RL problems. Organismcan memorize rewards or construct a contingency map andplan ahead behavior.

15 / 27

Introductionto

Pantelis P.Analytis

Introduction

The Iowa gambling task (Bachara et al. 1997)

Participants are presented 4 decks on the computer andthey are told that each deck will reward them or penalizethem.100 trials in total, unbeknownst to the participants. Theparticipants started with $ 2000 and are asked tomaximize their returns.

16 / 27

Introductionto

Pantelis P.Analytis

Introduction

The Iowa gambling task

Participants are presented 4 decks on the computer andthey are told that each deck will reward them or penalizethem.Deck’s A and B bring higher bring higher immediaterewards, but have negative expected value, while C and Dhave lower immediate rewards but positive expected value.

17 / 27

Introductionto

Pantelis P.Analytis

Introduction

Modeling human learning: expectation

The delta rule is a popular model-free learning rule:

Ej(t) = Ej(t − 1) + δj(t)η[Rj(t)− Ej(t − 1)],

where δj(t) is an indicator variable, being 1 if alternative jwas chosen on trial t, and 0 otherwise. We opted for asimple fixed learning rate, η ≥ 0.

18 / 27

Introductionto

Pantelis P.Analytis

Introduction

Modeling human learning: expectation

The decay rule is another popular model-free learning rule,according to which expected values of the unchosenalternatives decay towards 0 (e.g. Erev and Roth, 1998):

Ej(t) = ηEj(t − 1) + δj(t)Rj(t),

with decay parameter 0 ≤ η ≤ 1.

19 / 27

Introductionto

Pantelis P.Analytis

Introduction

Modeling human learning: choice rules

ε-greedy rule

P(C (t) = j) =

{(1− ε)/Kmax if Ej(t) > Ek(t), ∀k 6= j

ε/(K − Kmax) otherwise

where K is the number of arms and Kmax is the number ofarms with the same maximum value.

Softmax

P(C (t) = j) =exp(θEj(t))∑K

k=1 exp(θEk(t))

20 / 27

Introductionto

Pantelis P.Analytis

Introduction

The Iowa gambling task: behavioral results

Participants are presented 4 decks on the computer andthey are told that each deck will reward them or penalizethem.

Deck’s A and B bring higher bring higher immediaterewards, but have negative expected value, while C and Dhave lower immediate rewards but positive expected value.

21 / 27

Introductionto

Pantelis P.Analytis

Introduction

The Iowa gambling task: simulating models

The models were fitted on human data using maximumlikelihood estimation.

22 / 27

Introductionto

Pantelis P.Analytis

Introduction

Prediction competitions

23 / 27

Introductionto

Pantelis P.Analytis

Introduction

Replicating well known findings

24 / 27

Introductionto

Pantelis P.Analytis

Introduction

Studying widely used websites

Can you develop a model of likes and comments onInstagram or Twitter?How does attention interact with liking in websites likeFacebook?

25 / 27

Introductionto

Pantelis P.Analytis

Introduction

Using big data from KDD competitions

KDD regularly organizes competitions. Data from pastevents are available online.

26 / 27

Introductionto

Pantelis P.Analytis

Introduction

Dataset repositories

27 / 27

Introduction to reinforcement learning - Cornell University › info4940 › documents ›...

Documents

Reinforcement Learning Das Reinforcement Learning-Problem Alexander Schmid

Gabriel Ahlfeldt, Pantelis Koutroumpis, Tommaso Valletti ...eprints.lse.ac.uk/65339/7/Ahlfeldt_Speed 2.0 evaluating access_2017… · Gabriel Ahlfeldt, Pantelis Koutroumpis, Tommaso

Pantelis P. Analytis March 11, 2018 decision making Pantelis P. Analytis Introduction One-shot time allocation decisions Dynamic e ort allocation Extensions of the framework Atkinson

Finding Unsupervised Learning - Cornell UniversityUnsupervised Learning Pantelis P. Analytis Introduction Finding structure in graphs Clustering analysis Dimensionality reduction Elements

Reinforcement Learning - Multi-Agent Reinforcement

analytis work space manager

Reinforcement Learning Introduction Passive Reinforcement Learning Temporal Difference Learning Active Reinforcement Learning Applications Summary

DATACTIF SoNetA. BIG DATA ANALYTIS

The exploration-exploitation trade-offexploitation trade-o Pantelis Pipergias Analytis Exploration-exploitation problems The multi-armed bandit framework Strategies Contextual bandits

for high REINFORCEMENT SYSTEMS - nevoga.com€¦ · 2 REINFORCEMENT SYSTEMS REINFORCEMENT SYSTEMS REINFORCEMENT SYSTEM PLEXUS®, PYRAPLEX®, FTW The reinforcement system PLEXUS®,

Pantelis Bassakos. G.E.M. Anscombe: La description de l' action intentionnelle

Reinforcement Learning and Deep Reinforcement Learningcse.ucdenver.edu/.../Class-22-Reinforcement-learning-DL.pdf · 2018. 11. 28. · Outlines 1 Principles of Reinforcement Learning

Continuous hoops for transverse reinforcement of ... · Continuous hoops for transverse reinforcement of ... transverse reinforcement details for the ... Fig. 1 Transverse reinforcement

Automatic Control of State Space Systems - Pantelis Sopasakis

E3M-Lab – ICCS Paroussos Leonidas, Karkatsoulis Panagiotis, Vrontisi Zoi, Capros Pantelis WIOD in GEM-E3

ζωγραφοσ, παντελής Pantelis zografos

Pantelis Kammas and Vassilis Sarantides Originally

MASTER OF SIENE IN USINESS ANALYTIS GUIDELINES

Eick: Reinforcement Learning. Reinforcement Learning Introduction Passive Reinforcement Learning Temporal Difference Learning Active Reinforcement Learning

Inverse Reinforcement Learning CS885 Reinforcement