Assessment of a Team-Based Embedded Librarianship Model

Leo Lo, Research & Development LibrarianJason Coleman, Undergraduate & Community

Services Librarian

http://www.slideshare.net/JasonColeman1/

Time Tracking Instrument

Information Literacy is Served Jason Leo Total

Posts Made 33 13 46

Posts Read 902 856 1758

E-mails answered

1 1 2

Consultations 1 0 1

IM Chats 10 0 10

Online helps created

Libguide, 3 assignment tips documents

2 assignment tips documents

Libguide, 5 assignment tips documents

Cooking & Prep Time Jason Leo Total

Time Spent during class

656 minutes 201 minutes 857 minutes

Time Spent in preparation

420 minutes 240 minutes 660 minutes

Total 1076 minutes 441 minutes 1517 minutes

Consumption Aug. 2011 Jan. 2012 TotalStudents 53 52 105Students who posted questions to AAL thread

8 7 13

Total posts on AAL thread

56 30 86

Total AAL posts read 1184 951 2135

Avg # AAL posts read/student

22 18 20

Libguide n/a 407 page hits 407 page hitsIM chats 56 10 66

Was it Nutritious?

1. Our impact on Pre-test to Post-test improvement

2. Our impact on Overall grade3. Feedback

Pre-Test Question 1

Find an estimate of the current population of Beijing, China. What is the estimate?

a. Where did you find this estimate?

b. Do you think it is accurate? Why or why not?

Pre-Test Question 2

Find a current street map of Beijing, China.

a. Where did you find this map?

b. What process did you use to find it?

Pre-Test Question 4

How many tourists visited Beijing in 2010?

a. Where did you find this figure?

b. Do you think it is accurate? Why or why not?

Pre-Test Question 5

Find and cite (APA or MLA) a credible article on one of the ethnic minorities in China. Also, in two to three sentences, justify the credibility/authoritativeness of your source.

Post-Test Question 1

Find an estimate of the current population of Buenos Aires, Argentina. What is the estimate?

a. Where did you find this estimate?

b. Do you think it is accurate? Why or why not?

Post-Test Question 2

Find a current street map of Buenos Aires, Argentina.

a. Where did you find this map?

b. What process did you use to find it?

Post-Test Question 4

How many tourists visited Buenos Aires, Argentina in 2010?

a. Where did you find this figure?

b. Do you think it is accurate? Why or why not?

Post-Test Question 5

Find and cite (APA or MLA) a credible article on the status of women in Argentina. Also, in two to three sentences, justify the credibility/authoritativeness of your source.

Pre-Test to Post-Test Study

Sample:• Treatment group (n=10)

Average # of posts to AAL thread: 2.5Average # of posts to all threads: 13.9

• Matched control group (n=10)Average # of posts to AAL thread: 0Average # of posts to all threads: 13.9

Pre-Test to Post-Test Scoring Rubric

ACRL Standard Beginning Proficient Advanced

1. Determine the Extent of the Information Needed (examine question 2)

The research query is unfocused or unclear. Some or all of the words in the query are unrelated to the information need.

The research query is focused and clear. Most of the words in the query are related to the topic.

The research query is focused, clear, and complete. All of the words in the query are related to the topic.

Instructions: For each ACRL Standard circle Beginning, Proficient, or Advanced. If you are not able to make an assessment, write n/a

Pre-Test to Post-Test Scoring Rubric

ACRL Standard Beginning Proficient Advanced

2. Access the Needed Information Effectively and Efficiently (examine questions 1, 2, and 4)

Student uses only a single broad research tool (e.g., Google) and examines only the first page of results.

Student uses a single subject specific research tool and/or multiple broad research tools.

Student uses multiple subject specific research tools.

Pre-Test to Post-Test Scoring Rubric3. ACRL Standard

Beginning Proficient Advanced

Evaluate Information and its Sources Critically(examine question 5)

No mention is made of criteria that might be used to judge information quality.

Student mentions one or two of the following criteria: accuracy, authority, credibility, relevance, timeliness.

Student mentions three or more of the following criteria: accuracy, authority, credibility, relevance, timeliness.

On a scale of 1 (very poor) to 9 (very good), how would you rate the student’s information research skills. Base your rating on an overall assessment of their test answers.

Pre-Test to Post-Test Study

Raters:• Four, two scored all pre-tests and two scored all post-tests.•Blind and unaware of study design

Data Collection

Pre-Test to Post-Test Improvement

Stand 1 pre Stand 1 post Change One-way Anova

treatment 1.85 2.43 +.58 F= .124

control 2.0 2.69 +.69 p=0.73

Pre-Test to Post-Test Improvement

Stand 2 pre Stand 2 post Change One-way Anova

treatment 1.4 2.05 +.65 F= .649

control 1.3 2.15 +.85 p=0.431

Pre-Test to Post-Test Improvement

Stand 3 pre Stand 3 post Change One-way Anova

treatment 2.05 2 -.05 F= 3.032

control 1.85 2.2 +.35 p=0.1

Pre-Test to Post-Test Improvement

rating pre rating post Change One-way Anova

treatment 5.45 5.65 +.2 F= .19

control 4.9 5.4 +.5 p=0.668

Overall Grade related to Posting on AAL Thread

Avg. Overall Grade (out of 500) One-way Anova

treatment 493 F= 3.35

control 477 p=0.08

Pre-Test to Post-Test ImprovementAAL Posts Read Total Posts Read

Std. 1 Aug. 2011 0 -0.4 (p=0.19)

Std. 1 Jan. 2012 0.7 (p=0.03) 0.59 (p=0.06)

Std. 2 Aug. 2011 0.41 (p=0.12) -0.16 (p=0.33)

Std. 2 Jan. 2012 0.24 (p=0.25) 0.11 (p=0.38)

Std. 3 Aug. 2011 -0.29 (p=0.21) -0.64 (p=0.02)

Std. 3 Jan. 2012 -0.32 (p=0.18) -0.09 (p=0.4)

Rating Aug. 2011 0.08 (p=0.41) -0.26 (p=0.23)

Rating Jan. 2012 0.46 (p=0.09) 0.17 (p=0.32)

Overall Grades - correlationsAAL Posts Read Total Posts Read

Aug. 2011 0.52 (p=0.06) 0.297 (p=0.2)

Jan. 2012 0.03 (p=0.47) 0.009 (p=0.49)

Feedback• I found the requirement to chat with a librarian to be very

beneficial in finding new ways to locate information on my location. Overall I enjoyed the class and the variations of assignments that were required of us because they weren't all the same thing which forced us to develop as students. I feel I am leaving this class a much more competent researcher. – (post in reflections on the class thread, Aug. 2011)

Feedback• Your response does help and I will definitely want to look at

Google Scholar and EconLit as they are unfamiliar sources to me. I did feel better about using the data when Tray indicated that the WTTC was cited in this weeks reading. After taking this class and having the opportunity to learn about research sources on the Message Board, I am a much more cautious consumer of information on the internet. Thanks for your response, it is appreciated. – (post on main message board, aug. 2011)

Feedback• Oh my, this first assignment required some thoughtful

research. I assumed it would be multiple choices but it was almost a pleasant surprise. I learned a lot...FUN FUN FUN!! (post in Pre-test thread, Aug. 2011)

Feedback• I have a question about citing for a video. I found one on YouTube and

have really been using the link you gave me on Purdue Owl. Is the "Recorded Films or Movies" section correct for this type of citation?

PS I just want to say that both of you guys have really helped me a lot in answering questions and giving valuable and important information for this class. I have used not only the answers you have given me, but answers that you have given my fellow classmates. Thanks to you I may just make it thru this class!(post in AAL thread, Aug. 2011)

Feedback• I did not know about “Ask a Librarian” feature until I took

the library survey for this class. It is really helpful. I utilized this feature two nights now, I love it. (post in AAL Thread, Aug. 2011)

Student Learning Outcomes

Student Learning Outcomes

Critical Thinking• “Students will demonstrate the ability to access and

interpret information”

Communication• “Students will demonstrate the ability to communicate

clearly and effectively.”

Academic & Professional Integrity• “Students will demonstrate awareness and

understanding of the ethical standards of their academic discipline and/or profession”

Summary of Results• Students who posted to a librarian moderated performed better on class

grades. • The experience of posting to a librarian moderated thread had no impact on our

measure of information literacy skills.• In one class there was a positive relationship between number of AAL posts

read and class grades. In the other class there was a positive relationship between the number of AAL posts read and pre to post test improvement on standard 2 and overall rating.

• Students improved on our measures of information literacy. We do not know why, exactly.

• Some students seemed to enjoy the requirement to do research.

Next Steps• Be Draconian but helpful• Gather time of exposure as well as number of exposures• Increase incentive to use our services• Improve measures of learning• Perform experiments