View
227
Download
4
Tags:
Embed Size (px)
Citation preview
Announcements
• Check class website: http://graphics.cs.cmu.edu/courses/P2P/
• Pick presentations slots for Feb 9th and Feb 14th.
• One CVG and one P slot per person (not on same day)
• Email slot preference to [email protected]
• Have to meet instructors 2x before presentation!
Overview
•Basics of designing a perceptual experiment
•Types of experiments
•Analysis of experiments
•IRB
Basics
•Independent variable
•This is what varies
•Can have several levels (i.e. values)
•An experiment may have many indep. vars.
•Dependent variable
•This is what is measured
An example • How well do people recognize objects (as
compared to a favorite CV algorithm)?
Images from the Visual Object Classes Challenge 2010
Independent variable?Dependent variable?
Class participation!
Class participation!
An example
• How well do people recognize objects?
Images from the Visual Object Classes Challenge 2010
Independent variable Object class (20 levels)Dependent variable Accuracy, Reaction time (RT)
Understanding your variables• Discrete or continuous? (e.g., object class is
discrete)
• Measurement scale
- Nominal: there is a difference (e.g., car vs. bus)
- Ordinal: <,=,> make sense (e.g., rank in VOC2010 challenge)
- Interval: size and sign of difference makes sense (e.g., Fahrenheit scale)
- Ratio: all the above and having a 0 makes sense (e.g., percentage correct)
Confounds • Ideally, stimuli only differ in the levels of the
independent variable.
Images from the Visual Object Classes Challenge 2010
Confounding variables Color, shape, size of object, contrast of images, background of objects, location in image, lighting, etc.
Confounds • Ideally, stimuli only differ in the levels of the
independent variable.
Images from the Visual Object Classes Challenge 2010
In practice, impossible to eliminate confounding variables.
A well-designed study minimizes as many confounds as possible.
Dealing with confounds
• Control for confounding factors by:
• Removing confounding variables in stimuli
• Balancing the presence of confounding variables by adding control conditions*
• Separating participants into groups and comparing their responses i.e., control group
Condition = any manipulation of independent variables.
Controls
Images from the Visual Object Classes Challenge 2010Hypothesis: Size of object in image is enough to distinguish the 20 object classes.
Confounding (or control) variable = Visual size of objects
Option 1: Resize and re-crop all the images, never allow size to be an issue.
Hypothesis: Size of object in image is enough to distinguish the 20 object classes.
Confounding (or control) variable = Visual size of objects
Option II: Create second set of images with resized objects, test if size is an issue. Question: Control condition (same
subjects) or control group (different subjects)?
Controls
Images from the Visual Object Classes Challenge 2010
Experimental Designs Between-subjects: Different subjects in different conditions. Minimize learning effects, more subjects
Within-subjects: Same subjects in all conditions. More power, fewer subjects
Mixed design: Some conditions within-subjects and some between-subjects.
Balanced designsEven after adding control conditions and/or groups, we have to worry about order effects
Seeing bikes before buses
Seeing dogs before buses
Balanced designsEven after adding control conditions and/or groups, we have to worry about order effects
Seeing bikes before buses
Seeing dogs before buses
Does it matter? Priming can occur.
Balanced designsEven after adding control conditions and/or groups, we have to worry about order effects
Seeing bikes before buses
Seeing dogs before buses
Solution:
RandomizationCounterbalancing
Balanced designsUse randomization when lots of subjects or each subject sees the same conditions many times.
Use counterbalancing otherwise. E.g., 3 conditions (Dog, Bike, Bus)
Slide content: Aude Oliva, MIT OCW
Dog
Bike
Bus
Order 1 Order 2 Order 3Bus
Dog
Bike
Bike
Bus
Dog
S 1,4,7,10...S 2,5,8,11...S 3,6,9,12...
Overview
•Basics of designing a perceptual experiment
•Types of experiments
•Analysis of experiments
•IRB
Thresholds
Absolute: At which visual size can you detect that there is a bike?
Difference: How much detail is needed to recreate the original shape?
Measuring ThresholdsMethod of Constant Stimuli: Present stimuli at several levels, some below and some above threshold, measure proportion detected.
Method of Limits: Start at one level, if detected decrease else increase level. Converge to an estimate of threshold.
Method of Adjustment: Let subjects adjust the level until they can just detect the stimulus.
Method of Constant Stimuli+ Easy to conduct and analyze
- Need to have an idea of threshold beforehand- Takes longer, more trials
Psychometric function
Image source: http://www.ncbi.nlm.nih.gov/books/NBK11513/Slide content: Lynee Werner, University of Washington
Method of Limits+ Fewer trials+ Don’t need to estimate threshold beforehand- Noise prone
Staircase procedure
Image source: http://www.ncbi.nlm.nih.gov/books/NBK11513/Slide content: Lynee Werner, University of Washington
Method of Adjustment+ Intuitive for subject- Can be unreliable
Image source: http://psychology.wikia.com/wiki/File:Method_of_Adjustment.png
Slide content: Lynee Werner, University of Washington
Overview
•Basics of designing a perceptual experiment
•Types of experiments
•Analysis of experiments
•IRB
Back to detectionWhat about response bias?
Image source: http://www.walker.co.uk/walkerdam/getimage.aspx?id=9781406314403-1&size=webuse
Slide content: Lynee Werner, University of Washington
Subjects might be more prone to saying yes (or no).
Signal detection theory
Slide content: Lynee Werner, University of Washington
Subject said yes Subject said no
Stimulus present Hit Miss
Stimulus absentFalse Alarm
Correct Rejection
‣ d’ is computed from Hits and False Alarms‣ Measure of sensitivity‣ Bias-free
2-AFC design
Image source: http://www.ratracetrap.com/wp-content/uploads/2009/09/Fork-in-the-road-300x237.png
Slide content: Lynee Werner, University of Washington
‣ Instead of asking for yes/no response on each trial, force subject to choose from two options
‣ Another way of removing response bias (interval bias can remain)
How do you know subjects aren’t guessing?
(Bad subjects.)
Image source: http://academic.kellogg.edu/mckayg/buad112/web/pres/coin%20flip.jpg
Need to calculate chance performance for every task
Subjects might have been sleeping, distracted, perverse, or
simply unable to do your task because it is humanly impossible.
Is performance better than chance?
Statistics to the rescue. VOC2010 Challenge example:
20 object classes implies chance = 1/20 = 5%
Between-subjects design, Group ‘No Size Control’ (Performance = 80%, N=10) Group ‘Size Control’ (Performance = 75%, N=10)
✓ Use independent one-sample t-tests to compare performance in both groups to chance✓ Choose significance threshold (usually p = 0.05)✓ Divide significance threshold by number of tests (Bonferroni correction), here 0.05/2 = 0.025✓ Quote the t-statistic and p-value if less than corrected threshold as showing statistical significance.
Is performance better in one condition than another?
Again, statistics to the rescue. VOC2010 Challenge example:
20 object classes implies chance = 1/20 = 5%
Between-subjects design, Group ‘No Size Control’ (Performance = 80%, N=10) Group ‘Size Control’ (Performance = 75%, N=10)
✓ Use independent two-sample t-test to compare performance in two groups to each other✓ Choose significance threshold (usually p = 0.05, if combining with two previous tests then 0.05/3 = 0.0167)✓ Quote the t-statistic and p-value if less than threshold as showing statistical significance.
Choosing the right tests is VERY important
Otherwise you don’t know if you are measuring noise or a real effect.
For example, for a within-subjects design, you would use a paired samples t-test.
Get hold of a good statistics book and package and understand precisely what you are doing.
I recommend using SPSS and material from: http://www.statisticshell.com
Overview
•Basics of designing a perceptual experiment
•Types of experiments
•Analysis of experiments
•IRB
IRB and other legalities
You are dealing with human subjects, this means youneed to be very ethical and careful.
CMU has a Regulatory Compliance Office: http://www.cmu.edu/osp/regulatory-compliance/human-subjects.html
You are encouraged to take the online human subjects training: https://www.citiprogram.org/Default.asp?
Image source: http://www.icts.uiowa.edu/drupal/sites/all/themes/icts/custom/images/news/warning.jpg
IRB and other legalities
For this class:
If you will not use any perceptual data you gather ever again, you don’t need to write an IRB protocol and get approval.
If there is even a tiniest chance, the perceptual data you gather will be show up anywhere, you NEED to write an IRB protocol and get it approved in time. Come talk to me about this.
Image source: http://www.icts.uiowa.edu/drupal/sites/all/themes/icts/custom/images/news/warning.jpg