665 Session6-assessment-workshop-s13

Assessment Tools for Assessment Tools for Information Competency Information Competency

Dr. Diane NahlDr. Diane NahlLIS 665 Teaching Information Technology LiteracyLIS 665 Teaching Information Technology LiteracyLibrary and Information Science Program Library and Information Science Program University of Hawaii University of Hawaii Spring 2013Spring 2013

Assessment PhilosophyAssessment Philosophy1.1. Assessment is a Assessment is a continuous feedback process continuous feedback process

aimed at understanding and aimed at understanding and improving student improving student learninglearning..

1.1. Assessment is designed to Assessment is designed to enable measuring enable measuring student learning outcomes (SLOs) student learning outcomes (SLOs) to to demonstrate and provide evidence of learningdemonstrate and provide evidence of learning..

2.2. Assessment is a Assessment is a strategic planning tool strategic planning tool that that enables reporting of enables reporting of progress on short-term and progress on short-term and long-term goals and objectiveslong-term goals and objectives..

2

Nahl Spring 2013 LIS 665 Teaching Information Technology Literacy Literacy

The Assessment MandateThe Assessment Mandate

WASC has identified WASC has identified information literacy information literacy as as both a learning ability and a competency: both a learning ability and a competency:

““These programs also ensure the These programs also ensure the development of core learning abilities and development of core learning abilities and competencies including, but not limited to, competencies including, but not limited to, college-level written and oral communication; college-level written and oral communication; college-level quantitative skills; college-level quantitative skills; information information literacyliteracy; ; and the habit of critical analysis and the habit of critical analysis of of data and argument.data and argument.””

[Italics added] Western Association of Schools & Colleges[Italics added] Western Association of Schools & Colleges

3


The Assessment MandateThe Assessment Mandate

Through its accreditation process, WASC asks Through its accreditation process, WASC asks each of its member institutions of higher each of its member institutions of higher education to explain education to explain how its library contributes how its library contributes to the learning processto the learning process, specifically to , specifically to developing studentdeveloping student information literacy skillsinformation literacy skills, , in addition to providing information services.in addition to providing information services. Western Association of Schools & CollegesWestern Association of Schools & Colleges

4


Foundations Requirement: UHM Foundations Requirement: UHM Information Literacy HallmarkInformation Literacy Hallmark

To satisfy the To satisfy the Written CommunicationWritten Communication requirement, a course will:requirement, a course will:Help students develop Help students develop information literacy information literacy by by teaching search strategies, critical evaluation teaching search strategies, critical evaluation of information and sources, and effective of information and sources, and effective selection of information for specific purposes selection of information for specific purposes and audiences; teach appropriate ways to and audiences; teach appropriate ways to incorporate such information, acknowledge incorporate such information, acknowledge sources and provide citations. (2002)sources and provide citations. (2002)

5


Information Literacy as a Information Literacy as a Student Learning Outcome (SLO)Student Learning Outcome (SLO)

Outcomes measure what a student learned in Outcomes measure what a student learned in terms of terms of skills acquired skills acquired (e.g., distinctions (e.g., distinctions between Boolean operators, difference between between Boolean operators, difference between scholarly and popular information sources, scholarly and popular information sources, applying Web source evaluation criteria, etc.) or applying Web source evaluation criteria, etc.) or in terms of in terms of attitudes acquired or reinforcedattitudes acquired or reinforced (preferences and interests, values in information (preferences and interests, values in information seeking and use, self-efficacy beliefs, ethical seeking and use, self-efficacy beliefs, ethical uses of information, etc.).uses of information, etc.).

6


The Threefold Self in The Threefold Self in Information Skills AssessmentInformation Skills Assessment

• Attitude (affective skills)Attitude (affective skills)o Values, Perceptions, Preferences, PerseverenceValues, Perceptions, Preferences, Perseverenceo Coping skills, Self-efficacy, Emotional Intelligence Coping skills, Self-efficacy, Emotional Intelligence

(EI) (EI) • Knowledge (cognitive skills)Knowledge (cognitive skills)

o Understanding content and processesUnderstanding content and processeso Critical thinking and information evaluation skillsCritical thinking and information evaluation skills

• Performance (A-C-S skills)Performance (A-C-S skills)o Applying EI and knowledge in information tasksApplying EI and knowledge in information taskso Kinesthetic learning activities, noticing skillsKinesthetic learning activities, noticing skillso Embodied learningEmbodied learning

7


Threefold BiologicalThreefold BiologicalMental SystemMental System

Sensorimotor System Identifying, recognizing, noticing, ignoring, verbalizing,

executing, performing, writing, doing Cognitive System

Interpreting, appraising, planning, comparing, contrasting differentiating, recalling, defining, critically evaluating, problem-solving

Affective System Coping, evaluating, value-attaching, prioritizing, persevering,

regulating, intending, engaging

18


Assessment PhasesAssessment Phases

• Preliminary: Prior to teachingPreliminary: Prior to teachingo Needs assessment, baseline pre-tests and Needs assessment, baseline pre-tests and

questionnaires before instructionquestionnaires before instruction

• Formative: Concurrent with teachingFormative: Concurrent with teachingo During a process or sessionDuring a process or session

• Summative: Recollection after teachingSummative: Recollection after teachingo Immediately after a session, specific later Immediately after a session, specific later

time interval, or much later (longitudinal)time interval, or much later (longitudinal)

9


Formative AssessmentFormative Assessment

• Concurrent and situated in contextConcurrent and situated in context• Process-orientedProcess-oriented• Activity-basedActivity-based• Performance assessmentPerformance assessment• Knowledge assessmentKnowledge assessment• Attitude and perception assessmentAttitude and perception assessment• Authentic assessmentAuthentic assessment

10


Formative Assessment TypesFormative Assessment Types

• Worksheets (or other products) for exercisesWorksheets (or other products) for exercises• Real-time Surveys and PollsReal-time Surveys and Polls• Real-time Rating Scales (Evaluation) Real-time Rating Scales (Evaluation) • Structured Self-ReportsStructured Self-Reports• Real-time Open-ended FeedbackReal-time Open-ended Feedback• Email, Text Chat, Threaded DiscussionEmail, Text Chat, Threaded Discussion• Real-time RubricsReal-time Rubrics• Minute writing, Muddiest Point, Free Write, Writing Minute writing, Muddiest Point, Free Write, Writing

to Learnto Learn• Creative products (posters, stories, personal Creative products (posters, stories, personal

accounts)accounts)

11


Summative AssessmentSummative Assessment

• End-point assessmentEnd-point assessment• Retrospective reflectionRetrospective reflection• Recall-based encodingRecall-based encoding• Performance assessmentPerformance assessment• Knowledge assessmentKnowledge assessment• Attitude and perception assessment Attitude and perception assessment

(session or course evaluation)(session or course evaluation)• Authentic assessmentAuthentic assessment• Creative products (posters, stories, Creative products (posters, stories,

personal accounts)personal accounts)

12


Summative Assessment TypesSummative Assessment Types

• Knowledge Tests & ExercisesKnowledge Tests & Exercises• Performance Tests & ExercisesPerformance Tests & Exercises• SurveysSurveys• InterviewsInterviews• Focus Groups Focus Groups • Post-Rating Scales (Evaluation)Post-Rating Scales (Evaluation)• Content Analysis of Open-ended QuestionsContent Analysis of Open-ended Questions• RubricsRubrics

13


Knowledge Tests & ExercisesKnowledge Tests & Exercises

• MatchingMatching• Multiple ChoiceMultiple Choice• Checklists, identifying appropriate or relevant Checklists, identifying appropriate or relevant

elementselements• Fill-in number, concept words or phraseFill-in number, concept words or phrase• Short answer, a sentence or twoShort answer, a sentence or two• Concept maps Concept maps • Classification, ordering of steps or codesClassification, ordering of steps or codes• Criterion-referenced ranking or prioritizingCriterion-referenced ranking or prioritizing• EssayEssay

14


Performance Tests & ExercisesPerformance Tests & Exercises• Hands-on activitiesHands-on activities

o Worksheets filled-in during activityWorksheets filled-in during activityo Screen capture evidence of an activityScreen capture evidence of an activityo Structured self-report forms during activityStructured self-report forms during activityo Research and search logs & journalsResearch and search logs & journals

• Collaborative exercisesCollaborative exerciseso Peer LearningPeer Learningo Problem-Based Learning, Inquiry-Based Learning, Problem-Based Learning, Inquiry-Based Learning,

Jigsaw, etc.Jigsaw, etc.• Portfolios and PresentationsPortfolios and Presentations• Reflection ExercisesReflection Exercises• Research AssignmentsResearch Assignments

15


Authentic AssessmentAuthentic Assessment Authentic Assessment, e.g., concept mapping, Authentic Assessment, e.g., concept mapping,

minute writing, portfolios, logs, journals, minute writing, portfolios, logs, journals, creative products (e.g., poster, story, personal creative products (e.g., poster, story, personal accounts), rubrics, etc.accounts), rubrics, etc.

Use model student responses to build descriptions Use model student responses to build descriptions and criteria (e.g., LILO rubrics; BID)and criteria (e.g., LILO rubrics; BID)

Degree to which a skill has been masteredDegree to which a skill has been mastered Description of the elements and levels of Description of the elements and levels of

performance from low to moderate to highperformance from low to moderate to high Establishes criteria enabling students to understand Establishes criteria enabling students to understand

how to improvehow to improve

16


Composing Assessment QuestionsComposing Assessment Questions

Avoid bias, leading questions Avoid yes/no responses Include open-ended responses Accounted ratings

Please explain your number; Please state why you chose that number

Accounted knowledge Please explain your answer; Please state why you

gave that answer

Nahl Spring 2013 LIS 665 Teaching Information Technology Literacy Literacy17

Outcome Measures & Outcome Measures & Evaluation QuestionsEvaluation Questions All ACS Outcomes must be measured to provide evidence

of degree of student learning, and they must be scored (% correct or ideal rating) and include an Answer Key for the instructor

Identify the specific Outcome Measures you will use for each PI and set of ACS Outcomes

Worksheet(s) Pre-Post quizzes Poll data Observation data (classroom or online meetings) Rating scales and open-ended questions Rubrics

Include a Session & Instructor(s) Evaluation Form (like course evaluation)


Mapping Assessment ItemsMapping Assessment Items to Learning Outcomes to Learning Outcomes

Cognitive Outcome: Students will accurately estimate the time it takes to do the library research for an assignment.

Assessment Item:

1. Once you are in the online catalog, how long do you think it will take to find the call number for the book Tongue Fu?

A. A few seconds.B. Less than 2 minutes.C. More than 2 minutes.• Please state why you think it will take that amount of time:

See examples in LIS 665 student work on Nahl site


Mapping Assessment ItemsMapping Assessment Items to Learning Outcomes to Learning Outcomes

Affective Outcome: Students are willing to allocate sufficient time to the information research activity.

Assessment Item:1.How willing are you to allot the following amounts of time to library research for a 10 page paper?

a. 1 hour Gladly Only if I have to Not at allb. 5 hours Gladly Only if I have to Not at allc. 10 hours Gladly Only if I have to Not at all

See examples in LIS 665 student work on Nahl site


Exercise: Peer-Review of Exercise: Peer-Review of Active Learning ExercisesActive Learning Exercises• Teams 1 & 3, Teams 2 & 3; & Teams 4 & 5 meet to Teams 1 & 3, Teams 2 & 3; & Teams 4 & 5 meet to

usability test and to audit alignment of active usability test and to audit alignment of active learning exercises, SPIOs and ACS outcomeslearning exercises, SPIOs and ACS outcomes. .

o First do an exercise created by one Team, after which do an First do an exercise created by one Team, after which do an exercise for the other Team.exercise for the other Team.

o Map the hands-on exercise to the Team’s stated outcomes. Map the hands-on exercise to the Team’s stated outcomes. o Identify the form of evidence it will produce to demonstrate Identify the form of evidence it will produce to demonstrate

students have learned new knowledge and skills, or completed students have learned new knowledge and skills, or completed an assignment to meet specific ACS Outcomes.an assignment to meet specific ACS Outcomes.

o Determine whether certain ACS Outcomes are not mapped to Determine whether certain ACS Outcomes are not mapped to an assessment measure (exercise, test, scale, etc.).an assessment measure (exercise, test, scale, etc.).

o Include criteria for passing each Outcome and the form of Include criteria for passing each Outcome and the form of assessment to be usedassessment to be used

• Revise exercises based on testing and reviews.Revise exercises based on testing and reviews.

21


Instructional SequenceInstructional Sequence

Linear document showing the detailed order and schedule for a session

Each activity is marked with time limits Some activities are marked to skip or shorten

if needed (classroom settings) Each activity indicates who is teaching,

observing, facilitating, time keeping, troubleshooting, etc. (classroom settings)

Lists all equipment, materials and items needed in the session and who is bringing what (classroom settings)

Includes procedures for capturing data (paper, online, observed)


Next WeekNext Week

• Ch 6; Burkhardt et al. Ch 6; Burkhardt et al. • Final Draft of Instruction Unit Assessment Final Draft of Instruction Unit Assessment

documents and forms (Hands-on Exercises, documents and forms (Hands-on Exercises, Ratings, Polls, and Session Evaluation Ratings, Polls, and Session Evaluation items, etc.)items, etc.)

• Draft of Instructional Sequence with time in Draft of Instructional Sequence with time in minutes each portion takes to completeminutes each portion takes to complete

23


Education

665 Session6-assessment-workshop-s13