EPIC Online Publishing Use and Costs

Evaluation Program

Overview

• Overview of Evaluation Program

• Research update

Purpose of EPIC Evaluation Program

• To investigate how online electronic resources affect different aspects of the scholarly communication process.

a) general perspective

b) through lens of EPIC projects

How Does the Shift to Electronic Resources Affect the Following Groups?

• Publishers

• Administrators/Information Technology

• Librarians

• Faculty

• Students

Research Approach

• Interviews

• Focus Groups

• Surveys

• Weblog analysis

Info gathering Survey development

Research Projects to Date• Librarian Focus Group

• Librarian Online Survey

• Faculty Interviews

• Faculty Online Survey

• Student Interviews

• Student Online Survey• http://epic.columbia.edu/eval/

Librarian Online Survey

Methodology• a 15 minute online survey of librarians was conducted in Winter, 2002

• e-mail invitations were sent to librarians at higher education institutions across the country

• All were librarians at institutions that are either CIAO or Earthscape subscribers, or have been identified as being potential subscribers

• 1007 respondents (response rate: 32%)

Objectives

• Learn what librarians perceive as the benefits and disadvantages of electronic resources.

• Gain insights into how electronic resources affect the role of the librarian.

• Gain insights into how electronic resources affect the role of print resources.

Conclusions from Librarian Survey

Electronic Resources:

• Allow libraries to serve patrons more effectively

• Put a strain on some libraries

• Do not replace print resources

Faculty Survey

Methods• a 25 minute online survey of faculty and graduate students was conducted in Winter 2003

• e-mail invitations were sent to scholars at higher education institutions across the country

• scholars were involved in the disciplines of International Affairs or Earth Sciences

• 845 respondents (response rate 14%)

Objectives

• Gain insights into how academics are using electronic resources in their academic work

• Gain insights into the perceived benefits and disadvantages of electronic resources

• Learn how academics perceive electronic resources as affecting undergraduate students’ scholarship

• Examine differences based on stage of career, type of discipline, and type of institution

Respondent Profile

Gender & AgeN=845

9.5%

21.3%

25.9%

28.0%

13.5%

27.9%

72.1%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

61+

51-60

41-50

31-40

21-30

Female

MaleGender

Age

Position & Type of InstitutionN=845

11.8%

20.4%

67.7%

19.8%

5.9%

3.8%

18.8%

21.2%

30.5%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Baccalaureate College

Master's College

Research University

Graduate Student/Post-doctoralFellow

AdjunctProfessor/Instructor/Lecturer/Other

Research position in an academicinstitution

Assistant Professor

Associate Professor

Full Professor

Level ofAppointment

Type ofInstitution

Academic Responsibilities and Field of Study N=845

2.2%

32.5%

40.7%

19.3%

5.2%

36.4%

63.6%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Solely as a teacher

More as a teacher than as aresearcher

About equally as a researcherand as a teacher

More as a researcher than asa teacher

Solely as a researcher

International Affairs

Earth/Environmental ScienceField of Study

Academic Responsibiliti

Electronic Resources and Academics’ Work Habits

Use of and Proficiency with Electronic ResourcesN=845

2.2%

29.3%

48.3%

20.1%

0.2%

0.6%

7.6%

29.3%

62.2%

0.0%

0.0%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Not at all proficient (1)

Not very proficient

Somewhat proficient

Very Proficient

Extremely proficient (5)

Never

Less than once a month

Once a month

A few times a month

A few times a week

Every day

Proficiency with electronic resources

Mean3.86

Use of electronic resources

Electronic Resources’ Affect on Work HabitsN=845

9.7%

46.0%

47.8%

18.7%

31.2%

34.0%

38.7%

14.9%

16.7%

8.4%

8.6%

27.9%

20.9%

6.6%

34.1%

20.7%

3.9%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

I have a tendency to go nofurther than electronicresources in search for

information

I sometimes find that I settlefor materials that are

available online rather thanmake a trip to the library toget what I really would like

I use the physical libraryless than I would if

electronic resources werenot available to me

I am increasinglydependent on electronic

resources

Strongly agree (5) Somewhat agree Neither agree nor disagree Somewhat disagree Strongly disagree (1) Does not apply

MEAN

4.27

4.13

2.88

2.30

Work Habits and The LibraryN=845

17.5%

22.8%

59.6%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

I do most of my library work inside the physicallibrary

I do most of my library work outside the physicallibrary because I can adjust my information needs tosuit my preference for working from my home/office

I do most of my work outside the physical librarybecause I can readily access the information I need

remotely

Perceived Benefits and Disadvantages

of UsingElectronic Resources

Benefits of Electronic Resources--ConvenienceN=845

44.7%

51.5%

57.2%

36.9%

26.3%

30.4%

9.0%

9.7%

7.1%

7.8%0.1

%

3.9%

5.3%

1.3%

3.9%

1.4%

3.3%

0.1%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

More information withless effort

Possible for me towork from home orother location offcampus/work site

Save time

Strongly agree (5) Somewhat agree Neither agree nor disagree Somewhat disagree Strongly disagree (1) Does not apply

MEAN

4.38

4.20

4.16

Slide 1 of 2

Benefits of Electronic Resources—Improved Availability N=845

10.2%

29.6%

33.0%

33.6%

41.2%

21.7%

44.3%

41.5%

34.0%

43.1%

19.9%

17.8%

14.8%

18.1%

10.3%

31.6%

4.1%

6.0%

15.1%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Greater access to olderscholarly information

Items become availablequicker electronically than

in print

Some materials are morereadily available

electronically than in print

Increased access topictures/graphics

Greater access to currentscholarly information

Strongly agree (5) Somewhat agree Neither agree nor disagree Somewhat disagree Strongly disagree (1) Does not apply MEAN 4.21

3.97

4.03

4.01

2.80

Slide 2 of 2

Disadvantages of Electronic ResourcesN=845

11.5%

14.8%

18.7%

29.5%

37.3%

37.4%

41.8%

22.0%

17.9%

20.1%

20.7%

20.9%

19.6%

18.5%

13.7%

16.3%

12.8%

9.0%

4.4%

10.9%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

The amount of informationavailable can be

overwhelming for me

I have difficulty judging thequality of content

Too much information isavailable

Increases the need toseparate out the "reliable"

information from the"unreliable" information.

Strongly agree (5) Somewhat agree Neither agree nor disagree Somewhat disagree Strongly disagree (1) Does not apply

MEAN

3.57

3.31

3.15

2.98

Benefits and Disadvantages by DisciplineN=845

3.22

3.95

4.10

4.32

3.46

4.18

3.70

4.49

3 3.2 3.4 3.6 3.8 4 4.2 4.4 4.6 4.8 5

Too much information is available

Some materials are more readily availableelectronically than in print

Increased access to pictures/graphics

Save time

International Affairs

Earth/Environmental Science

All of the differences in means by discipline on this chart are statistically significant.

Benefits and Disadvantages by SeniorityN=845

3.00

3.43

3.94

4.32

4.50

3.24

3.65

4.10

4.03

4.29

2.50 3.00 3.50 4.00 4.50 5.00

I have difficulty judging the quality of content

Increases the need to separate out the "reliable"information from the "unreliable" information.

Some materials are more readily availableelectronically

More information with less effort

Save time

Associate/FullProfessor

GraduateStudent/PostDoctoralFellow/AssistantProfessor

All of the differences in means by seniority on this chart are statistically significant.

Teaching with Electronic Resources

Use of Electronic Resources for TeachingN=419

8.6%

27.4%

35.3%

57.0%

65.2%

67.3%

72.1%

79.0%

85.7%

90.0%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

I teach extension courses online

I use online materials in place of textbooks

I use online classroom programs (e.g. Blackboard)

I created a course website for one or more of my courses

I place course materials online (e.g. reserve readings)

I go online to research lectures that I will be covering in class

I go online to get images that help demonstrate ideas

I go online to get current events information to help demonstrateideas

I retrieve data online for classroom use

I direct students to primary or supplementary materials found online

Use of Electronic Resources for Teaching by DisciplineN=419

30.9%

82.4%

72.9%

85.9%

42.7%

54.8%

89.2%

96.8%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

I use online classroomprograms

I go online to get imagesthat help demonstrate

ideas

I go online to get currentevents information to help

demonstrate ideas

I direct students to primaryor supplementary

materials found online

International Affairs

Earth/EnvironmentalScience

All of the differences in means by discipline on this chart are statistically significant.

Electronic Resources & Students—BenefitsN=419

10.3%

43.4%

62.3%

26.5%

39.6%

32.5%

23.4%

9.8%

27.0%

4.5%

9.5%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Students have increasedaccess to older

information because ofelectronic resources

Students are now able todo projects they wouldn'thave been able to in the

past because some typesof information are morereadily available online

Students have increasedaccess to current

information because ofelectronic resources

Strongly agree (5) Somewhat agree Neither agree nor disagree Somewhat disagree Strongly disagree (1) Does not apply

MEAN

4.57

4.24

3.01

Slide 1 of 2

Electronic Resources & Students—ProblemsN=419

28.2%

50.8%

51.3%

58.0%

64.0%

37.7%

33.4%

33.4%

33.2%

28.6%

16.9%

8.4%

4.8%0.5%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Students spend less time thinking or deliberating overmaterial because the whole research process is sped up

by electronic resources

Students' use of electronic resources has resulted in themoften not learning how to use the physical library

Electronic resources facilitate plagiarism among students

Students have a tendency to go no further than electronicresources in searching for information

Students have difficulty evaluating the quality of contentof some online information

Strongly agree (5) Somewhat agree Neither agree nor disagree Somewhat disagree Strongly disagree (1) Does not apply

MEAN

4.59

4.47

4.38

4.32

3.82

Slide 2 of 2

Class Time Spent Instructing Students How to Evaluate Online Information

N=419

64.5%

77.7%

74.0%

69.5%

73.7%

64.9%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Earth/EnvironmentalScience

International Affairs

Research Universities

Master'sColleges/Universities

Baccalaureate Colleges

Overall

Availability of Courses Instructing Students How to Evaluate Electronic Resources

N=419

32.5%

6.7%

50.8%

6.4%

3.6%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

I don't know/Not sure

Other

There is no formal mechanism set up to trainstudents how to evaluate online information

My institution has an optional course

My institution has a required course

The Effect of Electronic Resources on Overall Quality of Students’ Learning Experience

N=419

1.0%

5.3%

9.5%

61.3%

22.9%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Very negatively (1)

Somewhat negatively

Neither positively nornegatively

Somewhat positively

Very positively (5)

MEAN4.00

Satisfaction with Electronic Resources

Importance of and Satisfaction with ER

2.63

3.14

3.17

2.87

2.89

3.02

2.85

3.50

3.66

3.67

3.65

3.66

3.70

3.75

2.00 2.50 3.00 3.50 4.00

Availability of older/archived information

Currency of information

Ability to print materials

Ease of navigation

Search capabilities of database

Ability to download materials

Full text available

Importance

Satisfaction

Importance of and Satisfaction with ER

2.88

2.86

3.03

2.66

3.19

3.06

2.91

3.01

3.07

3.27

3.30

3.42

2.00 2.50 3.00 3.50 4.00

No required login

Availability of high qualityimages/graphics

Text is in preservedformat

The ability to tell if aresource on the site is

peer reviewed

Reputation of provider

Speed of site

ImportanceSatisfaction

Important and Satisfied

• Ability to download• Ability to print• Currency of info• Speed of sites• Reputation of

providers• Text in PDF or other

standard format

• Full text available• Search capabilities• Ease of Navigation• Availability of

older/archived information

• Ability to tell if peer reviewed

Important and Not Satisfied

Other suggestions• Reduce price

• Standardize navigation and search procedures across databases

• Synthesize databases so that they are accessible through a single portal

• Faster turn-around time/More frequent updates

• Clearly identify or label sources

Conclusions• Electronic resources have become the main tool for scholars’ information gathering.

• Electronic resources are seen as providing convenience more so than increased availability of information.

• The main problem with electronic resources is having to separate out reliable from unreliable information.

• Most faculty believe students have an enhanced learning experience because of the information available through electronic resources.

• Future efforts could focus on providing increased access to older and full text material, and improving search and navigation capabilities.

Christina Norman, Ph.D.Research Director, EPICColumbia University211 International Affairs Building420 West 118th Street, Mail Code 3301New York, NY 10027 cn2005@columbia.edu

EPIC evaluation web site:

http://epic.columbia.edu/eval/