Embed Size (px)
Citation preview
The Big-Fish-Little-Pond Effect in Computer-
Based Math Assessments –
Do Computers Really Matter?
Annual Conference of the Association for Educational Assessment – Europe Tallinn, 07. November 2014
Fazilat Siddiq & Ronny Scherer
Computer-Based
Assessments
(CBA)
• Construct validity
• Discrepancies in achievement
• Interplay between achievement
and factors related to student
characteristics
(Chua, 2012; Celik & Yesilyurt, 2013; Deutsch, Herrmann, Frese, & Sandholzer, 2012;
Sainz & Eccles, 2012)
Background
Paper-and-Pencil
Assessments
(PPA)
2
Big-Fish-Little-Pond
Effect (BFLPE)
• Domains and groups
• Countries and cultures
• Students‘ learning behavior
and attitudes towards a
subject
• Construct validity
• Discrepancies in
achievement
• Interplay between CBA
achievement and factors
related to student
characteristics?
CBA-PPA
(Marsh, 1987; Marsh, Byrne, & Shavelson, 1988; Marsh et al., 2008; Marsh, Abdelfattah, et al., 2014;
Seaton, Marsh, & Craven, 2010; Seaton et al., 2010)
Background
“…claims a negative relation between school-average ability and students' individual self-concept” Herbert W. Marsh.
3
• Domains and groups
• Countries and cultures
• Students‘ learning behavior,
personality, and attitudes
towards a subject
BFLPE
• Construct validity
• Discrepancies in
achievement
• Interplay between CBA
achievement and factors
related to studentl
characteristics?
CBA-PPA
• New assessment situations
• New sources of social
comparisons
• BFLPE robust against type
of assessment?
Research gap
(Dai, Rinn &Tan ,2013; Chu & Chu, 2010; Kim, Kil, & Shin, 2014; Terzis & Economides, 2011)
4
Research Questions
1. Existence of the BFLPE in paper-and-pencil
and computer-based assessments of
mathematical literacy?
2. Moderation of the BFLPE by ICT- and
personality-related constructs?
5
Materials & Method
• Norwegian PISA 2012 data
• N = 4,686 students (48.9% female)
• Students' mean age = 15.8 years
• Multilevel structural equation modeling
• PPA and CBA tests for mathematical literacy
• Background questionnaire
6
Measures
Academic self-concept in mathematics e.g., “I learn mathematics quickly”.
Mathematical literacy • formulating
• employing concepts, facts,
procedures and reasoning
• interpreting, applying, and evaluating
Personality-related constructs • math self-efficacy
• math anxiety
ICT-related constructs • ICT devices at home
• use of ICT in school and in math lessons 7
Examples of
PPA & CBA
8
Research Question 1,
BFLPE
Aggregated
Math literacy
Individual
Math literacy
Model fit: CFI=.983, TLI=.977, RMSEA=.040
* p < .001
School level
Student level
Individual
Self-Concept
9
BFLPE = -.27*
ES2 = -.33
PPA:
Research Question 1,
BFLPE
Aggregated
Math literacy
Individual
Math literacy
BFLPE = -.32*
ES2 = -.40
Model fit: CFI=.984, TLI=.979, RMSEA=.039
* p < .001
School level
Student level
Individual
Self-Concept
CBA:
10
BFLPE = -.27*
ES2 = -.33
PPA:
Research Question 2,
Moderation
Aggregated
Math literacy
Individual
Math literacy
* p < .001
School level
Student level
Individual
Self-Concept
Math anxiety:
PPA: B = -.09*
CBA: B = -.07*
Cross-level
interaction:
Effects of moderator (student level), aggregated math literacy
(school level), and cross-level interaction significant
11
Research Question 2,
Moderation
Aggregated
Math literacy
Individual
Math literacy
* p < .001
School level
Student level
Individual
Self-Concept
Math self-efficacy
No moderator effects for PPA and CBA present
ICT availability
ICT use in school
and math lessons
12
Summary
• BFLPE exists for both assessment types
(PPA and CBA)
• BFLPE in CBA comparable to that in PPA
• Math anxiety moderates the BFLPE
• Math self-efficacy and ICT-related
constructs do not moderate the BFLPE
13
Implications
• Students' social comparisons with their
schoolmates as an important frame of
reference of self-concept – regardless of the
type of assessment
• Special attention for anxious students in
classrooms needed
• Robustness of the BFLPE: math literacy more
important than the type of assessment
14
Future directions
• Generalizability across further domains
(reading, science, …)
• Potential cognitive moderator: ICT literacy
rather than ICT use/availability
• Generalizability of findings across a larger
set of countries
15
17
PPA – CBA Construct validity
IRT
18
Moderator Item Example Coding # Items
Math self-efficacy How confident do you feel about having to do the following
mathematics tasks? Solving an equation like 3x + 5 = 17
0 = Not at all confident,
3 = Very confident
8 .88
Math anxiety Thinking about mathematics: To what extent do agree with
the following statements? I get nervous doing mathematics
problems.
0 = Strongly disagree,
3 = Strongly agree
5 .87
ICT availability at
home
Are any of these devices available for you to use at home?
Desktop computer, Internet connection
0 = No, 1 = Yes, but I don’t
use it, 2 = Yes, and I use it
11 .54
ICT use at school How often do you use a computer for the following activities
at school? Browsing the Internet for schoolwork
0 = Never or hardly ever,
4 = Every day
9 .83
ICT use in math lessons Within the last month, has a computer ever been used for the
following purposes in your mathematics lessons?
Drawing the graph of a function (such as y = 4x + 6)
0 = No, 1 = Yes, but only the
teacher demonstrated this,
2 = Yes, students did this
7 .82
19
Research Question 1
The BFLPE across assessment types
20
Reference
Please cite this presentation as follows:
Siddiq, F., & Scherer, R. (2014, November). The Big-
Fish-Little-Pond Effect in Computer-Based Math
Assessments – Do Computers Really Matter? Paper
presented at the Annual Conference of the Association
for Educational Assessment – Europe, Tallinn, Estonia.
21
