Today:

• Assignment 2 misconceptions• Our process• True Experiments: Single-Factor Design• Assignment 3 Q&A• Mid-term: format, coverage

Assignment 2 problems• Still issues with IV/DV identification• Not enough detail to really answer the questions• Not fully operationalizing the concept of the DV into

the actual data that will be measured• Discrete/continuous (for the measurement)• Scaling types• What are descriptive statistics• Which descriptive statistics are appropriate for the

different types of data • Estimation of reliability/validity of measurement (not

the experiment)

Where we are in the process

• Have defined broad research question and identified hypotheses or very refined research questions that must be evaluated to answer the question (Ass 1)

• Have identified the variable that need to be manipulated/studied in order to evaluate the hypotheses and have thought about what kind of data will need to be measured (Ass 2)

Assignment 3• Start thinking about threats to validity – what will you have

to justify in order to convince readers that you have a valid study that addresses the research question– Note: as of yet, you don’t have a study designed. But you

should be thinking about possible study designs that might be appropriate

• Ass. 2 was descriptive stats (individual variables) – in Ass. 3, you need to think about which comparisons you need to make– Different measurements, single variable?

• Satisfaction before/after treatment?– Correlations between different variables?

• Satisfaction as it relates to a participant’s age?– Focus on describing the data – what tables/graphs would you

use to show the relationships

Next:

• Assignment 4: Hypothesis testing– Starting to think about statistical evaluation of the

hypotheses:• Stating alternate and null hypotheses• Is each hypothesis one tailed or two tailed?• If there was a type I error, what would be the

conclusion statement based on the error? What should it be if you knew it was an error?• If you had a type II error, what would be the conclusion

statement based on the error? What should it be if you knew it was a error?

Next…

• Assignment 5: Control– By now, you should be able to focus in on your

study design– For each hypothesis, identify:• Where you will get the data• How you will rule out alternate explanations for the

results (give an alternate & show why it’s not viable)• The limitations to your study

Ongoing action items (1)

• Look at related work:– What papers will you cite when writing your

research paper – Papers to motivate the problem– Papers that have used similar methodologies to

answer similar types of questions– Papers that establish baselines in your area– Papers about methodologies– Papers that you should compare your results with

(if you actually end up running the study)

Ongoing action items (2)

• Think about what kind of study is needed to be able to answer the research questions/hypothesis that you have identified– How many factors are you looking at?– How much control do you need?– What are your biggest validity concerns that need

to be addressed?– What is the usual approach in your discipline?– …

The research paper

• Full details on Tuesday• Proposing a study to evaluate your research

questions• The assignments are the setup to get you

thinking about the details• In the paper, you will be writing them formally• In the presentation, you will be presenting them• Your ability to discuss alternatives and justify

your chosen approach will be key to a good mark

Mid-term

• 2:35-4:05, Friday, October 28th

• 90 minutes (will aim to have most of you done in 60 minutes)

• At 4:20 we will have in-class discussion of the mid-term answers– note: this is your chance to discuss what would be

a good answer and I have not budgeted time to do this on an individual basis, so stick around. After this session, it should be self-evident why you receive the marks you get on your mid-term

Mid-term

• 2:35-4:05, Friday, October 28th

• 90 minutes (will aim to have most of you done in 60 minutes)

• At 4:20 we will have in-class discussion of the mid-term answers– note: this is your chance to discuss what would be

a good answer and I have not budgeted time to do this on an individual basis, so stick around. After this session, it should be self-evident why you receive the marks you get on your mid-term

What will be covered

• Concepts covered in class (including the material from today)

• Concepts covered in tutorials (up to the material covered on Monday, October 17)

• Anything is fair game – if we talked about it, you may be evaluated on it

Question types

• Multiple choice or T/F with explanation• Short answer– Define/contrast/explain/describe concepts

covered

• Analysis (apply your knowledge)– Given a research question or a study description,

answer questions about the study design, validity, data analysis, etc.

True Experiments: Single-Factor Design

Research Methods & StatisticsOct 21, 2011

True Experiment

• Experimental control• Control as many potential threats to validity

as possible• Random assignment of participants/data to

conditions• Could be within-subjects or between-subjects

Control

• True experiment = complete control over the subject assignment to conditions and the presentation of conditions to subjects– Control over the who, what, when, where, how

• Control of the who => random assignment to conditions– Only by chance can other variables be confounded

with IV• Control of the what/when/where/how => control

over the way the experiment is conducted

Quasi-Experiment

• When you can’t achieve complete control– Lock of complete control over conditions– Subjects for different conditions come from

potentially non-random pre-existing groups

Terminology

• Factors: IVs of an experiment• Level: particular value of an IV• Condition: a group or treatment (technique)– e.g., Condition 1: old system, Condition 2: new

system• Treatment: a condition of an experiment• Subject: participant (can also think more

broadly of data sets that are ‘subjected’ to a treatment)

Factors to Treatments• At least 1 Factor (IV) has to vary to have an

experiment– Effect of screen size and input technique on

performance (speed, accuracy)• An IV must always have at least 2 levels• Condition refers to a particular way that subjects

are treated– Between subject: experimental conditions are the

same as the groups– Within subjects: only 1 group, that experiences every

condition (can be many conditions in an experiment)

Experimental Design: spot the flaw

• One-Group Post-Test-Only Design– Group of subjects are given a treatment– Then tested on the dependent variables

• What’s the problem?

Experimental Design: spot the flaw

• Post-Test-Only, non-equivalent control groups– Non-random allocation of subjects into groups– One group is given the treatment, one doesn’t

receive it (different levels to each group)– Post-test: measure the DV

• What’s the problem?

Experimental Design: spot the flaw

• One-Group Pre-Test-Post-Test Design– Single group (within subjects)– Pre-test: measure the DVs– Give the treatment– Post-test: Re-measure the DVs

• What’s the problem?

Good Experimental Design

• Two-Group, Post-Test Design• Two conditions• Two groups:– Between subjects: random allocation

• Treatment• Post-test: measure the DV• What’s really important?

Within-subjects

• Similar to the one-group pre-test-post-test design• It solves the individual differences issues• But raises other problems:– Need to look at the impact of experiencing the two

conditions– Will they get tired? Gain practice? Learn what is

expected?

• Need to control for order and sequence effects?

Order Effects

• Changes in performance resulting from (ordinal) position in which a condition appears in an experiment (always first?)

• Arises from warm-up, learning, fatigue, etc.• Effect can be averaged and removed if all

possible orders are presented in the experiment and there has been random assignment to orders

Sequence effects

• Changes in performance resulting from interactions among conditions (e.g., if done first, condition 1 has an impact on performance in condition 2)

• Effects viewed may not be main effects of the IV, but interaction effects

• Can be controlled by arranging each condition to follow every other condition equally often

Counterbalancing• Controlling order and sequence effects by arranging

subjects to experience the various conditions (levels of the IV) in different orders

• Self-directed learning: investigate the different counterbalancing methods– Randomization– Block Randomization– Reverse counter-balancing– Latin squares and Greco squares (when you can’t fully

counterbalance)– http://www.experiment-resources.com/counterbalanced-mea

sures-design.html

Discussion

• Pros and cons of within versus between subjects designs

Assignment 3: Q&A• For five of the threats to validity covered in class, give an

example of a possible threat to validity for your research question. Note: as you have not yet designed your experiment, this exercise is to identify possible threats under a possible experimental design. You can use different possible experimental designs for each threat.

• What comparisons should you make to explore the answers to your research question? That is, which of the variables you identified in Assignment 2 should be analyzed individually and which variables should be compared against each other (which do you think might have interactions)? For each variable, specify what tabular and graphical methods you would use to describe the data.