Sept17 college 2

Methoden en Technieken

Ning Ding

September 14 2010

Overview Review of the last course

Research Problem Variables and Hypotheses

Today’s course Instrumentation Validity and Reliability Internal Validity

Structure of the course Requirement:

read the book Online exercises

130zmhzfx

Review of the last course Chapter 1 The Nature of Research

Why do we need a research in education?Waarom hebben we onderzoek naar het onderwijs nodig? Beschrijven, voorspellen, uitleggen

Describe, Predict, ExplainDescribe, Predict, Explain

Mixed method = quantitative + qualitativeMixed method = quantitative + qualitative


Why do we need a research in education? Several types of researchVerschillende soorten onderzoek

Overview of the research process

Experimental research

Correlational research

Causal-comparative research

Survey

Ethnographic research

Historical research

Action research

Compare; Single over time

Relationship

Cause, consequence

Characteristics

Everyday experience

Past

Active involvement


Why do we need a research in education? Several types of research Overview of the research processOverzicht over het onderzoeksproces

Review of the last course Chapter 2 The Research Problem

Characteristics of good research problem Feasible Clear Significant Ethical

Review of the last course Chapter 2 The Research Problem

Characteristics of good research problem Feasible Clear Significant Ethical

Is computer-supported learning an effective learning method?Is computer-supported learning an effective learning method?

Is computer-supported learning an effective learning method for secondary school students?Is computer-supported learning an effective learning method for secondary school students?

Is computer-supported learning an effective learning method for secondeary school students in physics problem-solving?Is computer-supported learning an effective learning method for secondeary school students in physics problem-solving?

Is computer-ondersteund leren een effectieve leermethode voor middelbare school leerlingen voor het oplossen van natuurkundeproblemen in vergelijking met face-to-face leren?

Is computer-ondersteund leren een effectieve leermethode voor middelbare school leerlingen voor het oplossen van natuurkundeproblemen in vergelijking met face-to-face leren?

Review of the last course Chapter 3 Variables and Hypotheses

Quantitative vs Categorical Variables Vary in degree / do not vary in degree Verschillen in gradatie/ verschillen niet in gradatie

Independent vs Dependent Variables Study its effect / presumed to be affected Het effect ervan bestuderen/ verondersteld

beinvloed te worden Moderator Variables Exaneous Variables


Quantitative vs Categorical Variables Vary in degree / do not vary in degree

Independent vs Dependent Variables Study its effect / presumed to be affected

Moderator Variables Exaneous Variables


Quantitative vs Categorical Variables Vary in degree / do not vary in degree

Independent vs Dependent Variables Study its effect / presumed to be affected

Moderator Variables

Exaneous Variables

Are secondary school students’ physics performances influenced by their gender?Are secondary school students’ physics performances influenced by their gender?

Worden de leerprestaties van de middelbare school leerlingen beinvloed door het geslacht? Worden de leerprestaties van de middelbare school leerlingen beinvloed door het geslacht?

As above, the relationship is moderated by SESAs above, the relationship is moderated by SES

Homework, teacher, etc. Homework, teacher, etc.


Advantages vs Disadvantages of stating hypotheses

Voordelen en Nadelen van het uitgaan van hypothesis Think deeply / bias Prediction / unnecessary for some research types

Significant Hypotheses Directional vs Nondirectional Hypotheses

Richtinggevende vs. Niet richtinggevende


Advantages vs Disadvantages of stating hypotheses Think deeply / bias Prediction / unnecessary for some research types

Significant Hypotheses Directional vs Nondirectional Hypotheses

In secondary school physics problem-solving, the learning performances of students learning via computers are different in comparison with that of students learning in face-to-face learning.

In secondary school physics problem-solving, the learning performances of students learning via computers are different in comparison with that of students learning in face-to-face learning.

Voor het oplossen van natuurkundenproblemen op de middelbare school, is de leerprestatie van de leerlingen die computers gebruiken beter dan die van leerlingen die leren met behulp van face-to-face leren.

Voor het oplossen van natuurkundenproblemen op de middelbare school, is de leerprestatie van de leerlingen die computers gebruiken beter dan die van leerlingen die leren met behulp van face-to-face leren.

Today’s course Chapter 7 Instrumentation

Chapter 8 Validity and Reliability

Chapter 9 Internal Validity

Chapter 7 Instrumentation Data Data Collection

Where will the data be collected?Waar kunnen wij de data verzamelen

When will the data be collected?Wanneer kunnen we de data verzamelen

How often are the data to be collected?Hoe vaak worden de data verzameld

Who is to collect the data?Wie verzamelt de data?

Validity, Reliability, Objectivity, Usability

Chapter 7 Instrumentation

Validity: is an important consideration in the choice of an instrument to be used in a research investigation

Reliability: is another important consideration, since researchers want consistent results from instrumentation

Objectivity: refers to the absence of subjective judgments

Belangrijk als je een instrument kiest voor een onderzoek

Belangrijk als je een betrouwbare instrument wilt

Er is geen subjectieve oordeel.

Chapter 7 Instrumentation Usability: how easy the instrument will actually be to use.

Is het gebruiksvriendelijk?

How long will it take to administer?

Hoe lang duurt het uitvoeren?

Are the directions clear?

Zijn de aanwijzingen duidelijk?

How easy is it to score?

Is het gemakkelijk te scoren?

Do equivalent forms exist? Is er een gelijkwaardige vorm?

Have any problems been reported by others who used it? Hebben anderen die het gebruikt hebben problemen gemeld?

Chapter 7 InstrumentationWho provides you the information? Wie biedt de informatie?

the researcher (observation)De onderzoeker (observatie)the subject (test, questionnaire, log book)De subjectthe informant (teacher, parents, interviewee)De informant

ExampleA researcher in an elementary school administers a weekly maths test that requires students to solve maths problems correctly.

A researchers asks teachers to use a rating scale to rate each of their students on their phonic reading skills.

SubjectSubject

InformantInformant

Chapter 7 Instrumentation Who provides you the information? Wie biedt de informatie?

the researcher (observation)De onderzoeker (observatie) the subject (test, questionnaire, log book)De subject the informant (teacher, parents, interviewee)De informant

Where can you get the instrument?Waar kan ik het instrument krijgen? b.v. Eric database

Chapter 7 Instrumentation Who provides you the information? Wie biedt de informatie?

the researcher (observation)De onderzoeker (observatie) the subject (test, questionnaire, log book)De subject the informant (teacher, parents, interviewee)De informant

Where can you get the instrument?Waar kan ik hetinstrument krijgen? b.v. Eric database

Voorbeeld van Data-Collection Instruments Rating scales, interview list, observation form, flow chart,

behavior checklist, anecdotal information, time-and motion log

Chapter 7 Instrumentation Rating scales interview list observation form flow chart behavior checklist anecdotal information time-and motion log

Instructions: For each of the behaviors listedbelow, circle the appropriate number, using the following key: 5 = Excellent, 4 = Above Average, 3 = Average, 2 = Below Average,1 = Poor.

A. Explains course material clearly.1 2 3 4 5

B. Establishes rapport with students.1 2 3 4 5

C. Asks high-level questions.1 2 3 4 5

D. Varies class activities.1 2 3 4 5

Instructions: For each of the behaviors listedbelow, circle the appropriate number, using the following key: 5 = Excellent, 4 = Above Average, 3 = Average, 2 = Below Average,1 = Poor.

A. Explains course material clearly.1 2 3 4 5

B. Establishes rapport with students.1 2 3 4 5

C. Asks high-level questions.1 2 3 4 5

D. Varies class activities.1 2 3 4 5 Researcher

Researcher

Chapter 7 Instrumentation Rating scales

interview list observation form flow chart behavior checklist anecdotal information time-and motion log

Researcher

Researcher

Chapter 7 Instrumentation Rating scales interview list

observation form flow chart behavior checklist anecdotal information time-and motion log

Researcher

Researcher

Chapter 7 Instrumentation Rating scales interview list observation form flow chart

behavior

checklist anecdotal information time-and motion log

Researcher

Researcher

Chapter 7 Instrumentation Rating scales interview list observation form flow chart behavior checklist anecdotal information

time-and motion log

Researcher

Researcher

Chapter 7 Instrumentation Examples of Data-Collection Instruments

The researcher himself Rating scales, interview list, observation form,

flow chart, behavior checklist, anecdotal information, time-and motion log

The subject himself Questionnaire, behavior checklist, attitude scale,

performance, competence test, projective test, socio-gram


The research himself Rating scales, interview list, observation form, flow chart, behavior

checklist, anecdotal information, time-and motion log

The subject himself

Questionnaire, behavior checklist, attitude scale, performance, competence test, projective test, socio-gram

Teachers’ unions should be abolished.

Strongly Strongly

agree Agree Undecided Disagree disagree

(5) (4) (3) (2) (1)

Teachers’ unions should be abolished.

Strongly Strongly

agree Agree Undecided Disagree disagree

(5) (4) (3) (2) (1)

Subject

Subject


The subject himself


Subject

Subject


The subject himself


Subject

Subject

Chapter 7 Instrumentation Item-format

True-false Matching Interpretative exercises Multiple Choice Short-answer items Essay questions

selectioselectionn

supplsupplyy

Chapter 7 InstrumentationUnobtrusive Measures To eliminate the reactive effect, data collection

procedure that involve no intrusion into the naturally occurring course of events.

Chapter 7 Instrumentation Types of Scores

Raw Derived scores: percentile ranks, standard

scores

Norm-referenced vs Criterion-referenced instruments Norm: score compared with a norm group Criterion: score compared with a goal or target

that each learner achieves

Chapter 7 Instrumentation Norm-referenced vs Criterion-referenced instruments

Norm: score compared with a norm group Criterion: score compared with a goal or target that each learner

achieves

He solved at least 75% of the problems and scored above 90 percent of all the students in the class. He solved at least 75% of the problems and scored above 90 percent of all the students in the class.

Criterion-referenced

Norm-referenced

Chapter 7 Instrumentation Measurement Scales

Nominal Ordinal Interval Ratio



MeasurementScale Characteristics

Nominal Groups and labels data only;reports frequencies or percentages.

Ordinal Ranks data; uses numbers only to indicate ranking.

Interval Assumes that equal differences between scores really mean equal differences in the variable used.

Ratio All of the above, plus true zero point.

4. Levels of Measurement

Nominal: Nominal: Ordinal: Ordinal:

No logical order Ranked or ordered



Chapter 8 Validity and Reliability Basic concepts

Validity: appropriate, meaningful, correct, useful

Relibility: the consistency of one item/instrument to another

Objectivity:

Chapter 8 Validity and Reliability Validity

Do the results of the assessment provide useful information about the topic or variable being measured?

Content-related evidence of validity

Criterion-related evidence of validity

Construct-related evidence of validity


Content-related evidence of validity Content and format of the instrument



Criterion-related evidence of validity Relationship between scores obtained using the

instrument and scores obtained from other instruments predictive / concurrent



Criterion-related evidence of validity Relationship between scores obtained using the instrument and scores obtained predictive /

concurrent

Construct-related evidence of validity Psychological construct being measured by the instrument

Chapter 8 Validity and Reliability

Reliability Validity

Instrument A

Instrument A

Instrument B

Instrument B

Chapter 8 Validity and Reliability Reliability

Instrument A

Instrument A

Instrument A

Instrument A

Week1 Week2

Can be reliable, but not validIf unreliable, must not be valid

Validity


Errors of Measurement Reliability Coefficient

Test-retest Equivalent-form method Internal-consistency method

Split-half procedure Kuder-Richardson Approaches Cronbach’s α


Can be reliable, but not valid If unreliable, must not be valid

Errors of Measurement Reliability Coefficient Methods to overcome the errors Some rules of thumb of Reliability Coefficient

>.80 Attitude scale: >.60 Apitude test: >.80 High stakes: >.95

Chapter 9 Internal Validity Internal Validity

Observed differences on the dependent variable are directly related to the independent variable

Threats to Internal Validity Subject Characteristics Mortality: loss of subjects Location: Instrumentation: Instrument decay, data collector

characteristics or bias



Threats to Internal Validity Subject Characteristics Mortality: loss of subjects

Location: Instrumentation: Instrument decay, data collector characteristics or bias



Threats to Internal Validity Subject Characteristics Mortality: loss of subjects Location: Instrumentation:

Instrument decay data collector characteristics data collector bias



Threats to Internal Validity Subject Characteristics Mortality: loss of subjects Location: Instrumentation:

Instrument decay

data collector characteristics

data collector bias

Chapter 9 Internal Validity Threats to Internal Validity

Subject Characteristics Mortality: loss of subjects Location: Instrumentation: Instrument decay, data collector

characteristics or bias Testing: e.g. pretest History: Maturation: aging or experiences of the subjects change Attitudes: Hawthorne effect; John Henry Regression: laag hoog, hoog laag Implementation:


Testing: e.g. pretest History: Maturation: aging or experiences of the subjects change Attitudes: Hawthorne effect; John Henry Regression: laag hoog, hoog laag Implementation:


Testing: e.g. pretest History: Maturation: aging or experiences of the subjects change

Attitudes: Hawthorne effect; John HenryHawthorne effect

Hawthorne effect

John Henry effect

John Henry effect

Chapter 9 Internal Validity How to minimize these threats

Today’s course Instrumentation

Data Data collection

Validity and Reliability

Internal Relibitliy

Education

Sept17 college 2