The Evolution of Instruction Assessment - the UGA Perspective

Diana Hartle

Deb Raftus

Amy Watts

What came before

Half sheet handout to BIOL1107 (and other) classeswww.libs.uga.edu/ref/evalform.pdf

SurveyMonkey for Academic Enhancement classes

A dream born, a committee formed

Charge from the University Librarian to assess instruction program. More than attendance numbers were desired.

Committee formed with reference and other instruction librarians.

Our Quiz

Who will be assessed?

Students in introductory level classes across the curriculum.

How do we give the quiz?

Rejected ideas Internet based quiz services

Expense Reliability

Paper quizzes Collecting and correlating data Disseminating data

WebCT is the winner!

Availability Technical and administrative support Student familiarity with interface Data collection possibilities

What to ask?

Learning outcomes were defined by: ACRL standards for Information Literacy Local concerns

Skills / knowledge to be assessed

Research strategies Moving from a question to a search strategy

Narrowing/broadening topic Identifying keywords, synonyms Truncation and wildcards Boolean operators

Choosing which database/resource to search Revising search based on results

Skills / knowledge to be assessed

Locating full text of a journal article using a: Direct full-text link in a database Link to another database (FindIt@UGA) Specialized database (Electronic Journal Locator) Library OPAC for print

Using citation information and OPAC record to identify and locate specific item needed

Revising the quiz after Year 1

Merged Science/Humanities into one quiz Added questions for evaluative comments Did away with sequential elements to quiz

(“Based on answer to question 2…”) Added graphics to some questions Changed wording on some questions

Results

Snapshot:

1st semester (Fall 2005)

Humanities quiz: 199 students, Mean score 74.7%

Science quiz: 437 students, Mean score 83.5%

Results from Quiz 2, Fall 2006

Graded questions

Question Mean Discrimination

#1 94.3% 0.48

#2 81.6% 0.56

#3 92.6% 0.52

#4 91.5% 0.48

#5 43.3% -0.37

#6 71.7% 0.48

#7 95.7% 0.51

Results from Quiz 2, Spring 07

Graded Questions

Question Mean Discrimination

#1 92.8% 0.67

#2 84.3% 0.61

#3 92.1% 0.68

#4 91.0% 0.71

#5 54.2% 0.54

#6 76.6% 0.58

#7 94.4% 0.65

Results – evaluative questions

Weeding responses – determining what counts as a “valid” response

Coding responses – applying a standardized set of response possibilities to “valid” responses

Crosscheck the coding If needed, combine categories

What did they learn…

0.00%

5.00%

10.00%

15.00%

20.00%

25.00%

Truncation /Wildcards

Booleanoperators

Forming /inputting /

refiningsearches

GIL GALILEO(including

findingpassword)

Identifying /Choosing /

Using SpecificDatabases

Unclear /vague

response

Selected responses to question #9

I learned so much it was crazy.I didn't know the best way to use GALILEO. Now I

know how to use it quickly and efficiently in order to get the results I want.

I found out how to use Galileo, which I will now use a lot because of its credibility.

I also did not know the availability of help by the librarians.

That it is more important how you search than what you search.

Being more specific with searches greatly affects how much time you have to put into a search.

Still confused about…

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

Researchstrategies

Locating fulltext

Confused,other

Need morepractice

Unclear/vagueresponse

Praise forlibrarian

Not confused

Selected responses to #10

Why we weren't told about this earlier. Ask me when I'm doing it on my own. I feel like I have it now, but that

doesn't mean I can do it when I don't have someone leading me through it.

I am overwhelmed with the system and am not sure how to find the article that I need.

Not so much as confused, but a little overwhelmed. I like the fact that the instructor used an example to lead us through the search. I just think that it will take personal practice to really understand how to use the library.

I think it would be helpful to have us work on something outside of class immediately to see if we have any questions and make sure we can do it on our own.

I am not really confused about the search engines but overwhelmed because of the amount of information given to me in a little amount of time.

Why I had to sit through an hour lecture . . . when only 30 minutes was necessary.

There must be 50 ways to spell truncation

tranctuating

truction

truncationg

trunkation

trumcations

trunkating

troncation

trunk

trucation

asterix trick

asterick

asterik

asteric

astricks

asteriks

astricts

astericks

asterics

We don’t know whether to laugh or to cry…

That we actually have librarians!Well, first off, I finally learned where the library was.Nothing; this guy made it perfectly clear. He is amazing. He

should get a raise and take over President Adams job.Why would there be a physical subscription to a journal but not an

electronic when generally speaking the electronic is or should be cheaper...

Nothing! It was brilliant..............BRILLIANT!Why everyone thinks Galileo is so great. I never have needed to

use it.All the articles I need are available any time online through the

UGA Libraries homepage. I reviewed library skills I learned in 9th grade.

Future of quiz

Possible additional questions Year of student Have they had instruction session before?

Data Analysis Frequency of administration Migrate to new version of classroom management

software

Thanks for coming!

Don’t forget to take a bookmark.

Your presenters may be contacted at:

ugaref@gmail.com