Embed Size (px)
Citation preview
Utilizing Cross-Campus Collaboration to Improve
the Assessment of Information Literacy
Development of the Information Literacy Test for Seniors (ILT-S)
Nisa Bakkalbasi, Jason Kopp, Jonathan Paulo, Donna Sundre, Stefanie Warlick, Sara Williams
Information Literacy
• Information literacy requires individuals to “recognize when information is needed and have the ability to locate, evaluate, and use effectively the needed information”
• Information literacy has become a crucial life and job skill
Information Literacy Standards
• Information Literacy Standards for higher education were developed by the Association of College and Research Libraries
– Standard 1: The information literate student determines the nature and extent of the information needed.
• Does the student know what information is needed?
– Standard 2: The information literate student accesses needed information effectively and efficiently.
• Can the student find the information?
Information Literacy Standards
– Standard 3: The information literate student evaluates information and its sources critically and incorporates selected information into his or her knowledge base and value system.
• Can the student evaluate the information?
– Standard 4: The information literate student, individually or as a member of a group, uses information effectively to accomplish a specific purpose.
• Can the student use the information to create a product?
– Standard 5: The information literate student understands many of the economic, legal, and social issues surrounding the use of information and accesses and use information ethically and legally.
• Can the student access and use information ethically and legally?
Development of the ILT-S
• Several majors expressed a desire to measure information literacy for graduating seniors
• An Information Literacy Test (ILT) was already developed. However, there are several potential issues
– Although the test claims to be applicable to all levels of undergraduate and graduate students, results suggest it may be too easy for seniors
– Some aspects of information literacy at the senior level may be major-specific knowledge
Development of the ILT-S
• The development of the ILT-S was undertaken, with several goals in mind
1) Aimed at graduating seniors
2) Measuring underlying information-literacy latent constructs without being overly dependent on current technology and resources
3) ILT-S will be composed of a core set of items, as well as items specific to each major
• Social Work
• Health Studies
• Psychology
Model for Collaboration
• The development of the ILT-S necessitated a cross-campus collaboration with three different groups, each playing a key role
– Liaison librarians
– Professors in the majors
– Assessment and measurement practitioners
Model for Collaboration
• Liaison librarians
– Provided knowledge of information literacy and research skills as a whole
– Were more familiar with the ACRL standards
– Four librarians served as the chief item writers for the core information literacy test
– Additionally, the liaison librarian for each major assisted with development of the major-specific subtests
Model for Collaboration
• Assessment and measurement practitioners
– Provided knowledge of test development best practices
– Assisted the librarians and professors in the majors in content development
– Analyzed psychometric properties of tests based on pilot data, and provided recommendations
– Coordinated development between the libraries and the majors
Development of the ILT-S Core
• A&M practitioners worked with four librarians to write the initial 55 multiple choice items
– Standards 1, 2, 3, & 5
– Standard 4 excluded - involved producing a “product”
– Multiple measurement practitioners confirmed that items conformed to best practices
– Liaisons from the majors provided feedback on the items, which was integrated prior to piloting
Development of the ILT-S Core
• Item-writing divided among librarians
– Everyone contributed items to each Standard
• at Performance Indicator level, i.e. 1.2, 3.3, etc.
– Revision process – keep/edit/delete items
– Everyone mapped all items
– Revision process – review mapping matrix
– Strongest correlations to Standards kept
– New items developed to address gaps
Model for Collaboration
• Professors in the majors
– Provided knowledge of instruction in the majors
– Knew the common difficulties students had when conducting research in the major
– Professors served as the chief item writers for the major-specific subtests
• Psychology
• Health Studies
• Social Work
Major-Specific Subtest Development
• Coverage of both major-specific information literacy objectives and ACRL standards
• Health Studies
– 34 items
• Social Work
– 20 items
• Psychology
– Different approach – creative items with items nested within Psychology research scenarios – still in development
Fall 2011 Pilot
• Administered ILT-S pilot to 390 students spread across class levels
– 298 Health Studies
– 92 Social Work
• Included non-seniors as well
– Increase sample size
– Some majors wanted to show growth as well as competency at the senior level
– Group heterogeneity
• What if all seniors answered the items correctly?
– 3 Freshmen, 138 Sophomores, 110 Juniors, 136 Seniors, 3 Post-Seniors
• Instruments administered through Qualtrics
Core Pilot Results
-0.1 0 0.1 0.2 0.3
1
3
6
10
11
12
15
21
23
26
29
32
35
41
47
48
54
Standard 1 (KR-20 = .393)
-0.1 0 0.1 0.2 0.3
4
7
12
13
17
20
35
43
45
46
53
Standard 2 (KR-20 = .253)
Core Pilot Results
-0.4 -0.2 0 0.2 0.4
3
13
16
19
22
25
27
30
31
36
37
38
44
49
50
51
52
53
54
55
Standard 3 (KR-20 = .503)
0 0.1 0.2 0.3
2
5
8
9
10
14
18
24
28
34
39
40
42
Standard 5 (KR-20 = .342)
Core Pilot Results
• Overall KR-20 = .668
– Need higher reliability
• Correlations between standards approximately equal to 1 after applying unreliability correction
– Suggests the standards are not distinct
• Moderate correlation (r = .31) between year and score
– Suggests some increase in score over time
• Revised and removed items based on distractor analysis – current 51 item pool
– Continue revision until KR-20 > .80, with ~ 35 items
Sample Distractor Analysis
36. When attempting to identify bias within an information source, you would consider all of the following EXCEPT A) reputation of the authors B) credentials of the authors C) date the source was published D) purpose of the source
Sample Distractor Analysis
Item 22
10.516.67
71.03
11.79
% C
hoos
ing
Res
pons
e
05
101520253035404550556065707580859095
100
Option
A B C* D
Average T
otal Score
0.00
10.00
20.00
30.0027.10
22.85
27.9926.85
22. Which source would be most helpful when beginning research on the topic of volcanoes? A) Hamilton, J. (2010). The Lure of Volcanoes. History Today, 60(7), 34-41. B) Zirulnick, A. (2010, October 26). World's most active volcanoes. Christian Science Monitor, p. N.PAG. C) Sigurdsson, H. (2000). Encyclopedia of volcanoes [electronic resource] / [editor-in-chief, Haraldur Sigurdsson ; associate editors, Bruce F. Houghton ... et al. ; forword by Robert D. Ballard]. D) Schipper, S., & Mattox, S. (2010). Using Google Earth to Study the Basic Characteristics of Volcanoes. Science Scope, 34(3), 28-37.
Major-Specific Results
-0.4 -0.2 0 0.2 0.4 0.6
1
3
5
7
9
11
13
15
17
19
21
23
25
27
29
31
33
Health Studies (KR-20 = .723)
-0.2 -0.1 0 0.1 0.2 0.3 0.4
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
Social Work (KR-20 = .434)
Major-Specific Results
• Both majors need revision
• Moderate correlations between year and score
– Health Studies: .479
– Social Work: .225
• Used distractor analyses to modify items
– Health studies: 36 items
– Social work still revising
– Continue revision until KR-20 > .80, with ~ 20 items
Conclusion
• Working collaboratively allowed us to design both core and major-specific measures that
– aligned with testing best practice
– aligned with the ACRL standards and what we know about information literacy
– was informed by major instruction
• Further collaboration will be needed to continue pushing development forward
• Future work will expand to other majors
– Biotechnology
