Embed Size (px)
DESCRIPTION
Presented at the CyberSTEM conference, Berekeley, CA, 2011
Citation preview
Join the online discussion backchannel for this session at:
www.cyberlearningSTEM.org/sessions
Cyberlearning Tools for STEM Education Conference 2011
SCable Real-timeIndividual Behavior andLearningData
SCRIBL DataJustin ReichHarvard Graduate School of EducationCyberlearning Tools for STEM conference3/8/[email protected]
0 50 100 150 200 250 3000
10
20
30
40
50
60
70
80
90
100
MrBoyersClass.Pbworks.comPage Saves by Day
Page Saves
Days
Scatter Plot of Wiki Page Saves by Day (n=1,799)
ALLPS
0
10
20
30
day
0 100 200 300 400 500 600 700 800 900
Moving average of wiki development measured in page saves
SCable Real-timeIndividual Behavior andLearningData
SCRIBL Data
11
1,000K
10K
1,000
100
10
1
100K
# of Cases
Days Weeks Months Years
Design Research
Interviews
Surveys
Simulations
Semantic Analysis
Content Analysis
Discursive Analysis
Observational Research
Duration of data collection and capture
Time/Scale Web 2.0 Research State Space Modeling
Usage Statistics
Seconds
Biometric Analysis
School Leaders
Teachers
Policy Makers
Research with SCRIBL Data
• We can study SCRIBL data with microscopes and telescopes– Plan for interdisciplinarity– New methods are needed
• Operationalize time• Don’t invent new platforms, meter widely
adopted platforms
Learner Analytic ApproachesNetflix vs.Pandora
0 20000000 40000000 60000000 80000000 1000000000
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
All PBWorks Wikis
Es
tim
ate
d S
urv
iva
l P
rob
ab
ilit
y
(231) (463) (694) (926) (1157)Time in seconds (days)
Estimated Lifetimes
Seconds Days
25% 250 <150% 123,613 1.475% 5,282,874 61.1
PBworks wiki lifetimes in seconds/days (n=179,851)
Learner Analytic ApproachesNetflix vs.Pandora
Title I Eligible
Non-Title I Eligible
Does wiki persistence differ in Title I and non-Title I schools? (n=259)
Wiki Opportunities for Students to Develop 21st Century Skills
• Expert thinking:– Do students use academic content knowledge in wiki activities?– Do students reflect on the process/product?
• Collaboration:– Do students concatenate text on pages?– Do they substantively edit each others work and co-create
pages?
• New Media Literacy:– Do students use formatting?– Do they hyperlink?– Do they embed multimedia?
Wiki Quality Instrument25 QuestionsScale of 1-25
Do wikis provide opportunities for students to develop 21st century skills?
Are great wikis born or made?
0 50 100 150 200 250 300 350 400 4500
5
10
15
20
25
Days
Wik
i Qu
alit
y S
co
re
Do wikis created in high SES schools provide more opportunities for 21st century skill development?
0 50 100 150 200 250 300 350 400 4500
5
10
15
20
25
High SESLow SES
Days
Do wikis created in different subject areas provide different levels of opportunity to develop 21st C skills
0 50 100 150 200 250 300 350 400 4500
5
10
15
20
25
Social StudiesEnglishScienceComputer ScienceMath
Days
Wik
i Q
ua
lity
Sc
ore
SCable Real-timeIndividual Behavior andLearningData
SCRIBL Data
Educators routinely try to gather information about their students’ learning on the basis of what students do in class. But for any question posed in the classroom, only a few students respond. Educators’ insight into what the remaining students do and do not understand is informed only by selected students’ facial expressions of interest, boredom, or puzzlement.
To solve this problem, a number of groups are exploring the use of various technologies to “instrument” the classroom in an attempt to find out what students are thinking.
Classroom Wiki Research Questions
• How do we make them good?– What best practices, attitudes and resources
produce wiki learning environments that promote and nurture 21st century skills?
• Do only certain kids get the good ones?– What is the distribution of high quality wikis across
schools serving different student populations?
25
Open Education Resources• Twin Hopes for OER
– (Excellence) Teachers will use free, online tools and resources to create student-centered learning environments that prepare students for 21st century life
– (Equity) Since these materials are free, poor students will disproportionately benefit.
Brown, J. S., & Adler, R. P. (2008). Minds on fire: Open education, the long tail, and learning 2.0. Educause Review, 43(1), 16-32.
Bonk, C. J. (2009). The world is open : How web technology is revolutionizing education (1st ed.). San Francisco, Calif.: Jossey-Bass.
Series1
Poor students
Narrowing gaps or a rising tide?
Series1
Poor students
Non-poor students Non-poor students
21st C
entu
ry S
kill
s
Remapping the Digital Divide
• How can we design a study to examine whether or not poor students disproportionately benefit from the availability of free online tools?– Theoretical Framework– Operationalizing the theoretical framework– Research Design
Dimensions of the Digital Divide
1st Digital Divide: Access• Schools
– 3.8 Students/Computer in schools with <35% students eligible for FRPL
– 4.0 students/computer in schools with >75% student eligible for FRPL
• Anywhere– 86 % of students living in
households making <$30K use the internet
– 97% of students living in households making >$70K use the internet
2nd Digital Divide: Usage• Attewell (2003): “[There exists a]
real possibility that computing for already-disadvantaged children may be dominated by games at home and unsupervised drill-and-practice or games at school, while affluent children enjoy educationally richer fare with more adult involvement”
• Jenkins (2007) Participation Gap
Attewell, P. (2003). Beyond the digital divide. In P. Attewell, & N. Seel (Eds.), Disadvantaged teens and computer technologies (pp. 15-34). Munster, Germany: Waxmann.
Jenkins, H.; Clinton K., Purushotma R., Robison A. and Weigel M.(2007), Confronting the challenges of participatory culture: Media education for the 21st century. Chicago, Il.: MacArthur Foundation.
How can we operationalize usage?Usage as persistence• Wiki lifetime: number of days of activity of
a wiki community– Birth: Creation of wiki subdomain (e.g.
ReichWorldHistory.pbworks.com)– Death: Final wiki edit
• After a 90 day observational period
30
How can we operationalize usage?Usage as opportunities to develop 21st
century skills• Participation• Expert Thinking• Complex Communication • New Media Literacy
31
What are our research questions?
RQ #1) Persistence: Are wikis created in schools serving affluent students used for greater lengths of time than wikis created in poor schools?
RQ#2) Participation: Do wikis created in schools serving affluent students provide more opportunities for students to develop 21st century skills
33
Which wikis are in my sample?• Dataset
– All179,853 publicly-viewable education-related wikis started on the PBworks platform between June 2005 and August of 2008.
– Does not include “private” wikis (~70,000)
• Sample– Randomly sampled 1,799 wikis (1%)– Coded to identify 259 U.S. based, K-12 wikis from
specific public schools• Detailed usage statistics provided by PBworks.com• Demographic school level data from the Common Core of Data
(National Center for Education Statistics, 2007-2008) 34
What are our data analytic strategies?
RQ#1) Are wikis created in non-poor schools used for greater lengths of time than wikis created in poor schools?
• Estimate survival functions of wiki groups by Title I status using Kaplan-Meier estimation; use Wilcoxon’s test to test for differences
where
S(ti) is the estimated survival probability in any of t time periods, which are delineated by instances of wiki death
ni is the number of wikis still active at the beginning of time period ti;
di is the number of wikis that become inert during time period ti.
RQ#2) Do wikis created in non-poor schools exhibit more evidence of collaboration and student
involvement than wikis created in poor schools?
Estimate wiki quality trajectories using the multilevel model for change.
0 20000000 40000000 60000000 80000000 1000000000
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
All PBWorks Wikis
K-12 Wikis
Su
rviv
al
Pro
ba
bil
ity
(231) (463) (694) (926) (1157)Time in seconds (days)
Estimated Lifetimes
Seconds Days
All PBworks25% 250 <150% 123,613 1.475% 5,282,874 61.1
K-12 Wikis25% 2,721 <150% 763,195 8.875% 12,590,074 145.7
How long do K-12 wikis persist? (n=411)
What subjects are wikis used for? (n=411)
Education
Classics
ESL
Business
Health/PE
Modern FL
Contained Elementary
Art
Library
Math
Computer Science/ Technology
Science
Social Studies
English / Language Arts
0 20 40 60 80 100 120 140
2
4
5
6
8
10
20
22
26
45
60
61
70
120
Does subject area predict persistence? (n=411)
Subject
50% Lifetime
75% Lifetime
Computer Science 28 198
English/Language Arts 27 198
Science 18 152
Social Studies 6 56
Math 4 33No Subject 1 8
What Grade Levels are K-12 wikis used in? (n=411)
Unknown
Higher Ed
9--12
6--8
K-5
0 20 40 60 80 100 120 140 160 180 200
83
8
180
118
109
Does grade level predict persistence? (n=411)
• No
Do wikis from non-poor schools persist longer than wikis from poor schools? (n=259)
44
Non-poor schools (n=146)
Poor schools (n=110)
Days
Kaplan Meier Survival Estimates of Wikis from Poor and Non-Poor Schools
Summary statistics of wiki lifetimes in poor and non-poor schools (n=259)
• Day 1 Mortality:– Wikis from poor schools: 40%– Wikis from non-poor schools: ~20%
• Median Lifetime:– Wikis from poor schools: 7 days– Wikis from non-poor schools: 58 days
• 25% Lifetime:– Wikis from poor schools: 73 days– Wikis from non-poor schools: 259 days
45
RQ#2) Do wikis created in non-poor schools exhibit more evidence of collaboration and student
involvement than wikis created in poor schools?
Title I eligible(n=110)
Non-Title I eligible(n=146)
Concatenation 12 24Copyediting 10 15Co-construction 7 7Commenting 16 22Collaborative Sum= 0 82 (75%) 110 (75%)Collaborative Sum= 1 17 (15%) 13 (9%)Collaborative Sum= 2 7 (6%) 17 (12%)Collaborative Sum= 3 2 (2%) 3 (2%)Collaborative Sum= 4 2 (2%) 3 (2%)2 Goodness of fit test (2=4.2, df=4, p=.38)Student Involvement 28 (25%) 50 (34%)2 Goodness of fit test (2=2.28, df=1, p=.13)Student Involvement and at least 1 Collaborative Behavior
16 (15%) 20 (14%)
2 Goodness of fit test (2=0.04, df=1, p=.85)
Discussion• Participatory behavior is rare across all wikis,
but both student involvement and collaboration can be found in wikis from both poor and non-poor schools
• Wikis from non-poor schools persist longer than wikis from poor schools.
• The Open Education Resources strategy of promoting free online tools and resources may, counter-intuitively, expand the second digital divide—in the absence of targeted interventions.
48
How can we begin to explain these patterns?
• What might explain our empirical findings?– What obstacles do poor schools have in using wikis?
• Qualitative Research– Interviews with 50+ wiki-using teachers, many drawn
at random from our PBworks samples.– 35+ focus groups with students– Classroom observations in 12 schools in MA, CT, ME,
NH, CA, GA, VA
What obstacles exist for wiki use in poor schools?
• Differences in resources? • Differences in school culture?
Differences in networked technology resources among public school districts
Low Poverty Districts (<10%)
Medium Poverty Districts (11-20%)
High Poverty Districts (>20%)
Provide teachers with their own server space for posting their own Web pages or class materials
(Elementary Secondary) 90%
92%81%84%
74%74%
Provide students with electronic storage space on a server
76%92%
60%85%
50%72%
Provide students with online access to the library catalogue
82%92%
69%82%
66%72%
Provide students online access to databases (for library resources)
71%79%
58%67%
53%57%
Employ an individual responsible for education technology leadership
(Full TimePart Time
None)
60%26%13%
48%35%17%
47%33%20%
SOURCE: U.S. Department of Education, National Center for Education Statistics, Fast Response Survey System (FRSS), “Educational Technology in Public School Districts, Fall 2008,” FRSS 93, 2008.
Fear, anxiety and worry about student exposure to the Internet
Barbara: There was - the superintendent said, "Do want you want but I never want us on the frontpage with some bad headline" [chuckle]. I thought those are kind of interesting. That's why he told our technology committee. Our main goal is to not make the frontpage with the bad headline attached.
Jim: And as far as I know but I don't want to be the person to be in the newspaper for... Look at what this kid was doing on the internet in a school. I don't want that
Framework for Intervention:Cycle of Experiment and Experience
Experiment
Review (Experience)Plan
Fear - Growth+
Institutional Capacity+
The atmosphere at our school in general is to cautiously open up to the possibilities. So, we’re not being pushed to use technology. If we find our way to it, and if we find obstacles that are there and we need things to change to get access to certain things, the administration generally will make that happen. But this, I think in terms of the environment, security is more important than openness in general and sometimes that leads us into a few obstacles….
We haven't gotten to the point where we're making a lot of the stuff public yet. Is that possible down the road? I think with a little bit of experience, with a little bit of, we improve the comfort level then, we can start to say, “Let's make a blog that invites other people throughout the world. Whoever wants to come visit and check it out and be part of the conversation; let's figure how to do that.” I don't think I should... I'm not really in that place that right now. I'm not sure what the response would be if I did. I think we all sort of need to build that comfort level piece first. But I also see that it could be pretty awesome to move in that direction.
Dear Justin,
I am in the library right now and sat down to do some work on the Wikispace to get it ready for next year. However, upon sitting down, I discovered that over the summer this website has been blocked by the City. I spoke to the librarian about appealing blocked content, but he says that they are not unblocking any sites at this time.
Consequently, it does not look like we are going to be doing this project this year. I will be trying to put together something else for this unit, but at this point, I don't know what we will be doing and doubt it will be appropriate for your project because it will not be on wikispaces.com.
I am very sorry; it was quite a surprise to me.
Wikis created in non-poor schools persist
longer than wikis created in poor schools
Inequities in wiki usage suggest that Web 2.0 tools may exacerbate the 2nd digital divide
Framework for Intervention: Promoting a Cycle of
Experiment and Experience
Experiment
Review (Experience)Plan
Fear - Growth+
Institutional Capacity+Series1
Poor students
Non-poor students
21st C
entu
ry S
kill
s
What Next?1. Develop an instrument to measure Wiki
Quality
2. Correlate wiki quality profiles with teacher attitudes, practices, and resources
3. Develop computational tools to automate those analyses at scale
4. ?
5. Profit
Acknowledgements• Hewlett Foundation Open Education
Resources for grant support• PBworks for data support• Hunter Gehlbach, Stone Wiske, Laura
Schifter, Anna Savaadra, and other readers of this paper.
• Benjamin Mako Hill for coming up the river to offer his thoughts and critique! 58
