JCDL 2008

Exploring Educational Standard Alignment:

In Search of ‘Relevance’

René Reitsma*, Byron Marshall*Michael Dalton*, Martha Cyr

*Oregon State UniversityWorcester Polytechnic InstituteWorcester Polytechnic Institute

JCDL 2008

Exploring Educational Standard Alignment:In Search of ‘Relevance’

• Problem: aligning DL learning objects with educational standards

– Need & tantalizing promise

– National Science Digital Library (NSDL) efforts & accomplishments

– Early results show low Inter-Rater Reliability (IRR)

• Hypothesis: low IRR is partially a methodological artifact

• Proposal: multifactor concept of ‘alignment’

• Experiment: 10-factor alignment model

– High IRR

– Four factor regression model (R=.75) of ‘overall’ alignment

Aligning DL Learning Objects with Educational Standards

• Expanding DL learning resource base; e.g.,– National Science Digital Library (NSDL): 928 collections– K-12: TeachEngineering.org, TeachersDomain.org, Engineering is Elementary, etc.– NSF-GK-12 program (ongoing).

• ≈84,500 math, science & technology standards (changing frequently)

JCDL 2008

Curriculum Standard Alignment Efforts

• NSDL leadership:– Jes&Co:

• Achievement Standards Network (ASN)

– Center for Natural Language Processing (CNLP):

• Curriculum Alignment Tool (CAT)

• Standard Alignment Tool (SAT)

– WGBH Teachers’ Domain: standard alignment & lexicon

• Others:

– Academic Benchmarks

– AAAS/NSDL Strandmap server

– Etc.

JCDL 2008

NSDL-based Curriculum Alignment Services JCDL 2008

How Good are these Alignments?

JCDL 2008

Low Inter-Rater Reliability (IRR)

• Devaul, H., Diekema, A.R., Ostwald, J. (2007)

• Bar-Ilan, J. Keenoy, K., Yaari, E., Levene, M. (2007)

“There is no average user, and even if the users have the same basic knowledge of a topic, they evaluate information in their own context…”

• Hypothesis: Low IRR is partially a methodological artifact

–Alignment is a multifactor, multidimensional concept

–Learning objects may align with certain dimensions but not with others

–Levins, R, Lewontin, R.C. (1980): “Abstraction becomes destructive when the abstract becomes reified… so that the abstract descriptions are taken for descriptions of the actual objects”

JCDL 2008

Dimensions of Alignment

• Seracevic, T. (2007): ‘Relevance: A Review of the Literature and a Framework for Thinking on the Notion in Information Science. Part II: Nature and Manifestations of Relevance’

JCDL 2008

“Clues” Our(!) Mapping to Educational DLs

Content Topics & concepts

Object Cost, learning object type, formatting

Validity Trustworthiness

Use/Situational match Grade level, institutional requirements; e.g., testing procedures, professional development

Cognitive match Teacher qualifications, pedagogy

Belief /Affective match Emotional response

Hypotheses

• H: One-dimensional alignment/relevance IRR is partially a methodological artifact.

• H-1: At least some dimensional IRRs will be high(er)

• H-2: Dimensional IRR will vary

• H-3: ‘Overall alignment/relevance’ IRR will be low, even when asked in the context of dimensional relevance testing.

JCDL 2008

Experiment

‘Clue’ Alignment Statement

Affective match R-1 Appeal The document contains materials that are motivational or stimulating (interesting, appealing or engaging) for students

Content R-2 Concepts The document includes concepts, keywords, terms and definitions from the standard

Content R-3 Background The document provides interesting and important background material related to the standard

Object R-4 Grade level The grade level of this material is appropriate for this task or else I can easily adapt the materials in this document to my grade level

Situational match R-5 Nontextuals I can use (a) nontextual component(s) ; e.g., figures, tables, images, videos or graphics

Situational match R-6 Examples I can use the real-world examples provided in the document in class.

Situational match R-7 Hands-on I can use one or more of the hands-on , active engineering activities

Situational match R-8 Attachments I can use some of the attachments; e.g., score sheets, rubrics, test questions, etc.

Situational match R-9 References I can use references or Internet links to relevant materials elsewhere

R-10 overall relevance Overall, I consider this document relevant for this teaching assignment

JCDL 2008

Experiment Cont.’d

• 14 Subjects all familiar with the TeachEngineering system

• Two teaching tasks:

– “As a third grade Massachusetts teacher you are assigned to teach material related to the standard “Relate earthquakes, volcanic activity, mountain building, and tectonic uplift to plate movements.” You have two hours of class time to spend on instruction.”

• Judge the alignment of three curricular objects (R-1 – R-10, six-point Likert scale)

JCDL 2008

IRR-1: both subjects score on the same side of the scale; i.e., both score either ‘strongly agree,’ ‘agree,’ or ‘somewhat agree,’ or both score ‘somewhat disagree,’ ‘disagree,’ or ‘strongly disagree.’

IRR-2: same as IRR-1 except that answers may not differ with more than one scale point.

IRR-3: both subjects answer the question identically

Results• 91 IRR comparisons × 10 alignment dimensions × 6 alignments

• H-1: IRRs are relatively high; IRR-1 (binary): 64%-95%

• H-2: IRR variability

• H-3: Overall relevance (R-10) among the weaker ones

JCDL 2008

How about Overall Relevance (R-10)?

r R-1 R-2 R-3 R-4 R-5 R-6 R-7 R-8 R-9 R-10

R-1Appeal

1.0

R-2Concepts

.16 1.0

R-3Background

.36 .63 1.0

R-4Grade

.27 .20 .31 1.0

R-5Nontextuals

.09 .19 .20 .18 1.0

R-6Examples

.21 .26 .27 .30 .42 1.0

R-7Hands-on

.34 .31 .40 .59 .24 .40 1.0

R-8Attachments

-.02 .17 .21 .36 .35 .42 .39 1.0

R-9References

.06 .23 .20 .30 .25 .24 .45 .30 1.0

R-10Overall

.31 .76 .66 .50 .28 .35 .54 .35 .34 1.0

JCDL 2008

MLR Model of Overall Relevance (R-10)

JCDL 2008

R2 = .75

• ‘Overall alignment/relevance’ is meaningful as a complex variable.

• Some high IRR alignment dimensions do not contribute to overall alignment.

β Std. Error T-value p

Intercept -.272 .259 -1.048 .298

R-2 Concepts .567 .083 6.794 < .01

R-3 Background .173 .084 2.047 .044

R-4 Grade level .322 .090 3.576 < .01

R-7 Hands on .194 .093 2.082 .041

Conclusion

JCDL 2008

• K-12 educational DL content is expanding; educational standard alignment is needed.

• Innovative and promising resources are available but reported IRR of assessment of alignments is low.

• Propose that ‘Alignment’ is a complex concept:

– Recognize alignment dimensions

– Experiment suggests that dimension-specific IRR will be (much) higher

– ‘Overall’ alignment has a very specific interpretation.

• What do we need:

– Continued assessment and IRR collection

– Collections making their assessment data available.

– Alignment methods that can assimilate ‘evidence’ from the multiple dimensions that comprise ‘alignment’ of a learning resource with a teaching standard.