1. Methoden en Technieken Ning Ding September 19 2011

2. Overview

Review of the last course

• Research Problem

• Variables and Hypotheses

Todays course

• Instrumentation

• Validity and Reliability

• Internal Validity

3. Structure of the course

Requirement:

• read the book

• Online exercises

130

zmhzfx

4. Review of the last course

Chapter 1 The Nature of Research

• Why do we need a research in education?

• Waarom hebben we onderzoek naar het onderwijs nodig?

• Beschrijven, voorspellen, uitleggen

Describe, Predict, Explain Mixed method = quantitative + qualitative 5. Review of the last course

Chapter 1 The Nature of Research

• Why do we need a research in education?

• Several types of research

• Verschillende soorten onderzoek

• Overview of the research process

Experimental research Correlational research Causal-comparative research Survey Ethnographic research Historical research Action research Compare; Single over time RelationshipCause, consequence CharacteristicsEveryday experience PastActive involvement 6. Review of the last course

Chapter 1 The Nature of Research

• Why do we need a research in education?

• Several types of research

• Overview of the research process

• Overzicht over het onderzoeksproces

7. Review of the last course

Chapter 2 The Research Problem

• Characteristics of good research problem

• • Feasible

• • Clear

• • Significant

• • Ethical

8. Review of the last course

Chapter 2 The Research Problem

• Characteristics of good research problem

• • Feasible

• • Clear

• • Significant

• • Ethical

Is computer-supported learning an effective learning method? Is computer-supported learning an effective learning method for secondary school students? Is computer-supported learning an effective learning method for secondeary school students in physics problem-solving? Is computer-ondersteund leren een effectieve leermethode voor middelbare school leerlingen voor het oplossen van natuurkundeproblemen in vergelijking met face-to-face leren? 9. Review of the last course

Chapter 3 Variables and Hypotheses

• Quantitative vs Categorical Variables

• • Vary in degree / do not vary in degree

• • Verschillen in gradatie/ verschillen niet in gradatie

• Independent vs Dependent Variables

• • Study its effect / presumed to be affected

• • Het effect ervan bestuderen/ verondersteld beinvloed te worden

• Moderator Variables

• Exaneous Variables

10. Review of the last course

Chapter 3 Variables and Hypotheses

• Quantitative vs Categorical Variables

• • Vary in degree / do not vary in degree

• Independent vs Dependent Variables

• • Study its effect / presumed to be affected

• Moderator Variables

• Exaneous Variables

11. Review of the last course

Chapter 3 Variables and Hypotheses

• Quantitative vs Categorical Variables

• • Vary in degree / do not vary in degree

• Independent vs Dependent Variables

• • Study its effect / presumed to be affected

• Moderator Variables

• Exaneous Variables

Are secondary school students physicsperformancesinfluenced by theirgender ? Worden deleerprestatiesvan de middelbare school leerlingen beinvloed door hetgeslacht ?As above, the relationship is moderated bySES Homework, teacher, etc. 12. Review of the last course

Chapter 3 Variables and Hypotheses

• Advantages vs Disadvantages of stating hypotheses

• Voordelen en Nadelen van het uitgaan van hypothesis

• • Think deeply / bias

• • Prediction / unnecessary for some research types

• Significant Hypotheses

• Directional vs Nondirectional Hypotheses

• • Richtinggevende vs. Niet richtinggevende

13. Review of the last course

Chapter 3 Variables and Hypotheses

• Advantages vs Disadvantages of stating hypotheses

• • Think deeply / bias

• • Prediction / unnecessary for some research types

• Significant Hypotheses

• Directional vs Nondirectional Hypotheses

In secondary school physics problem-solving, the learning performances of students learning via computersare differentin comparison with that of students learning in face-to-face learning.Voor het oplossen van natuurkundenproblemen op de middelbare school, is de leerprestatie van de leerlingen die computers gebruikenbeter dandie van leerlingen die leren met behulp van face-to-face leren. 14. Todays course

Chapter 7 Instrumentation

Chapter 8 Validity and Reliability

Chapter 9 Internal Validity

15. Chapter 7 Instrumentation

7.1 Data

7.2 Data Collection

• 7.2.1 Where will the data be collected?

• Waar kunnen wij de data verzamelen

• 7.2.2 When will the data be collected?

• Wanneer kunnen we de data verzamelen

• 7.2.3 How often are the data to be collected?

• Hoe vaak worden de data verzameld

• 7.2.4 Who is to collect the data?

• Wie verzamelt de data?

7.3 Validity, Reliability, Objectivity, Usability

16. Chapter 7 Instrumentation 7.3.1 Validity: is an important consideration in the choice of an instrument to be used in a research investigation 7.3.2 Reliability: is another important consideration, since researchers want consistent results from instrumentation 7.3.3 Objectivity: refers to the absence of subjective judgments Belangrijk als je een instrument kiest voor een onderzoek Belangrijk als je een betrouwbare instrument wilt Er is geen subjectieve oordeel. 17. Chapter 7 Instrumentation

7.3.4 Usability: how easy the instrument will actually be to use.

• • Is het gebruiksvriendelijk?

• • How long will it take to administer?

• • • Hoe lang duurt het uitvoeren?

• • Are the directions clear?

• • • Zijn de aanwijzingen duidelijk?

• • How easy is it to score?

• • • Is het gemakkelijk te scoren?

• • Do equivalent forms exist?

• • Is er een gelijkwaardigevorm?

• • Have any problems been reported by others who used it?

• • Hebben anderen die het gebruikt hebben problemen gemeld?

18. Chapter 7 Instrumentation

7.4 Who provides you the information?

Wie biedt de informatie?

• the researcher (observation) De onderzoeker (observatie)

• the subject (test, questionnaire, log book) De subject

• the informant (teacher, parents, interviewee) De informant

Example

A researcher in an elementary school administers a weekly maths test that requires students to solve maths problems correctly.

A researchers asks teachers to use a rating scale to rate each of their students on their phonic reading skills.Subject Informant 19. Chapter 7 Instrumentation

Who provides you the information?

Wie biedt de informatie?

• the researcher (observation) De onderzoeker (observatie)

• the subject (test, questionnaire, log book) De subject

• the informant (teacher, parents, interviewee) De informant

7.5 Where can you get the instrument?

• Waar kan ik het instrument krijgen?

• b.v. Eric database

20. Chapter 7 Instrumentation

Who provides you the information?

Wie biedt de informatie?

• the researcher (observation) De onderzoeker (observatie)

• the subject (test, questionnaire, log book) De subject

• the informant (teacher, parents, interviewee) De informant

Where can you get the instrument?

• Waar kan ik hetinstrument krijgen?

• b.v. Eric database

7.6 Voorbeeld van Data-Collection Instruments

• Rating scales, interview list, observation form, flow chart, behavior checklist, anecdotal information, time-and motion log

21. Chapter 7 Instrumentation

• Rating scales

• interview list

• observation form

• flow chart

• behavior checklist

• anecdotal information

• time-and motion log

Instructions: For each of the behaviors listed below, circle the appropriate number, usingthe following key: 5 = Excellent, 4 = AboveAverage, 3 = Average, 2 = Below Average, 1 = Poor. A. Explains course material clearly. 1 2 3 4 5 B. Establishes rapport with students. 1 2 3 4 5 C. Asks high-level questions. 1 2 3 4 5 D. Varies class activities. 1 2 3 4 5 Researcher 22. Chapter 7 Instrumentation

• Rating scales

• interview list

• observation form

• flow chart

• behavior checklist

• anecdotal information

• time-and motion log

Researcher 23. Chapter 7 Instrumentation

• Rating scales

• interview list

• observation form

• flow chart

• behavior checklist

• anecdotal information

• time-and motion log

Researcher 24. Chapter 7 Instrumentation

• Rating scales

• interview list

• observation form

• flow chart

• behavior

• checklist

• anecdotal information

• time-and motion log

Researcher 25. Chapter 7 Instrumentation

• Rating scales

• interview list

• observation form

• flow chart

• behavior checklist

• anecdotal information

• time-and motion log

Researcher 26. Chapter 7 Instrumentation

7.7 Examples of Data-Collection Instruments

• 7.7.1 The researcher himself

• • Rating scales, interview list, observation form, flow chart, behaviour checklist, anecdotal information, time-and motion log

• 7.7.2 The subject himself

• • Questionnaire, behaviour checklist, attitude scale, performance, competence test, projective test, socio-gram

27. Chapter 7 Instrumentation

7.7.3 Examples of Data-Collection Instruments

• The research himself

• • Rating scales, interview list, observation form, flow chart, behavior checklist, anecdotal information, time-and motion log

• The subject himself

• • Questionnaire,behavior checklist, attitude scale, performance, competence test, projective test, socio-gram

Teachers unions should be abolished. Strongly Strongly agree Agree Undecided Disagree disagree (5) (4) (3) (2) (1) Subject 28. Chapter 7 Instrumentation

Examples of Data-Collection Instruments

• The subject himself

• • Questionnaire,behavior checklist , attitude scale, performance, competence test, projective test, socio-gram

Subject 29. Chapter 7 Instrumentation

Examples of Data-Collection Instruments

• The subject himself

• • Questionnaire, behavior checklist, attitude scale,performance , competence test, projective test, socio-gram

Subject 30. Chapter 7 Instrumentation

7.8 Item-format

• True-false

• Matching

• Interpretative exercises

• Multiple Choice

• Short-answer items

• Essay questions

selection supply 31. Chapter 7 Instrumentation

• 7.9 Unobtrusive Measures

• To eliminate the reactive effect,data collection procedure that involveno intrusioninto the naturally occurring course of events.

32. Chapter 7 Instrumentation

7.10 Types of Scores

• Raw

• Derived scores: percentile ranks, standard scores

7.11 Norm-referenced vs Criterion-referenced instruments

• Norm: score compared with a norm group

• Criterion: score compared with a goal or target that each learner achieves

33. Chapter 7 Instrumentation

Norm-referenced vs Criterion-referenced instruments

• Norm: score compared with a norm group

• Criterion: score compared with a goal or target that each learner achieves

Hesolved at least 75% of the problemsandscored above 90 percent of all the students in the class .Criterion-referenced Norm-referenced 34. Chapter 7 Instrumentation

7.12 Measurement Scales

• Nominal

• Ordinal

• Interval

• Ratio

35. Chapter 7 Instrumentation

Measurement Scales

• Nominal

• Ordinal

• Interval

• Ratio

Measurement Scale Characteristics Nominal Groups and labels data only; reports frequencies or percentages. Ordinal Ranks data; uses numbers only toindicate ranking. Interval Assumes that equal differences betweenscores really mean equal differences inthe variable used. Ratio All of the above, plus true zero point. 36. Nominal:Ordinal:No logical order Ranked or ordered Chapter 7 Instrumentation 37. Chapter 7 Instrumentation

Measurement Scales

• Nominal

• Ordinal

• Interval

• Ratio

38. Chapter 8 Validity and Reliability

8.1 Basic concepts

• Validity: appropriate, meaningful, correct, useful

• Reliability: the consistency of one item/instrument to another

• Objectivity:

39. Chapter 8 Validity and Reliability

8.2 Validity

Do the results of the assessment provide useful information about the topic or variable being measured?

• 8.2.1 Content-related evidence of validity

• 8.2.2 Criterion-related evidence of validity

• 8.2.3 Construct-related evidence of validity

40. Chapter 8 Validity and Reliability

8.2 Validity

• 8.2.1 Content-related evidence of validity

• • Content and format of the instrument

41. Chapter 8 Validity and Reliability

8.2 Validity

• Content-related evidence of validity

• • Content and format of the instrument

• 8.2.2 Criterion-related evidence of validity

• • Relationship between scores obtained using the instrument and scores obtained from other instruments predictive /concurrent

42. Chapter 8 Validity and Reliability

8.2 Validity

• Content-related evidence of validity

• • Content and format of the instrument

• Criterion-related evidence of validity

• • Relationship between scores obtained using the instrument and scores obtained predictive /concurrent

• 8.2.3 Construct-related evidence of validity

• • Psychological construct being measured by the instrument

43. Chapter 8 Validity and Reliability

8.3 Reliability

Validity Instrument A Instrument B 44. Chapter 8 Validity and Reliability

8.3 Reliability

Instrument A Instrument A Week1 Week2

• Can be reliable, but not valid

• If unreliable, must not be valid

Validity 45. Chapter 8 Validity and Reliability

8.3 Reliability

• 8.3.1 Errors of Measurement

• 8.3.2 Reliability Coefficient

• • • Test-retest

• • • Equivalent-form method

• • • Internal-consistency method

• • • • Split-half procedure

• • • • Kuder-Richardson Approaches

• • • • Cronbachs

46. Chapter 8 Validity and Reliability

8.3 Reliability

• Can be reliable, but not valid

• If unreliable, must not be valid

• 8.3.1 Errors of Measurement

• 8.3.2 Reliability Coefficient

• 8.3.3 Some rules of thumb of Reliability Coefficient

• • • >.80

• • • Attitude scale: >.60

• • • Apitude test: >.80

• • • High stakes: >.95

47. Chapter 9 Internal Validity

9.1 Internal Validity

• Observed differences on the dependent variable are directly related to the independent variable

9.2 Threats to Internal Validity

• Subject Characteristics

• Mortality: loss of subjects

• Location:

• Instrumentation: Instrument decay, data collector characteristics or bias

48. Chapter 9 Internal Validity

Internal Validity

• Observed differences on the dependent variable are directly related to the independent variable

Threats to Internal Validity

• Subject Characteristics

• Mortality: loss of subjects

• Location:

• Instrumentation: Instrument decay, data collector characteristics or bias

49. Chapter 9 Internal Validity

9.1 Internal Validity

• Observed differences on the dependent variable are directly related to the independent variable

9.2 Threats to Internal Validity

• Subject Characteristics

• Mortality: loss of subjects

• Location:

• Instrumentation:

• • Instrument decay

• • data collector characteristics

• • data collector bias

50. Chapter 9 Internal Validity

Internal Validity

• Observed differences on the dependent variable are directly related to the independent variable

9.2 Threats to Internal Validity

• Subject Characteristics

• Mortality: loss of subjects

• Location:

• Instrumentation:

• • Instrument decay

• • data collector characteristics

• • data collector bias

51. Chapter 9 Internal Validity

9.2 Threats to Internal Validity

• Subject Characteristics

• Mortality: loss of subjects

• Location:

• Instrumentation: Instrument decay, data collector characteristics or bias

• Testing: e.g. pretest

• History:

• Maturation: aging or experiences of the subjects change

• Attitudes: Hawthorne effect; John Henry

• Regression: laag hoog, hoog laag

• Implementation:

52. Chapter 9 Internal Validity

9.2 Threats to Internal Validity

• Testing: e.g. pretest

• History:

• Maturation: aging or experiences of the subjects change

• Attitudes: Hawthorne effect; John Henry

• Regression: laag hoog, hoog laag

• Implementation:

53. Chapter 9 Internal Validity

9.2 Threats to Internal Validity

• Testing: e.g. pretest

• History:

• Maturation: aging or experiences of the subjects change

• Attitudes: Hawthorne effect; John Henry

Hawthorne effect John Henry effect 54. Chapter 9 Internal Validity

9.3 How to minimize these threats

55. Todays course

7 Instrumentation

• Data

• Data collection

8 Validity and Reliability

9 Internal Reliability