Is there a statistically significant relationship between library resource access and student attainment at The Open University, a distance learning institution

The Open University Library Data Project

Is there a statistically significant relationship between library resource accesses and student attainment at the Open University, a distance learning institution?

Anne Gambles, Project Manager

The Northern Collaboration Conference 2016-09

Open University Facts and Figures1.89 million people worldwide have studied with the OU

2

173,899UK students

>21,000people with

disabilities studied with the OU in

2014/15.

76%of OU students

work full or part-time during

their studies

8353Overseas students

30%of new OU

undergraduates are under 25

21% of OU UK

undergraduates live in the 25% most deprived

areas

3

85%surveyed in the 2016

National Student Survey agree library

resources and services are good

enough for their needs

1,295,001 million

downloads from Open Research Online. The

Open University's open access repository of research publications

51kUnique visitors to

the OU Library website per month

Award winning

Digital and information literacy tools for

students

1stUK HE to provide a 24/7 webchat

serviceResource richness

600+ databases & collections,

134k+ electronic journals and over

200k ebooks

Open University Library Facts and Figures

Outline

Project

Background

Library Data

Strategy

Library Data

Project overview

Library Data

Progress and findings

4

5

Early stepsExploring the potential of library data 2010-

• RISE Jisc-funded project in 2010 investigating the potential of using library activity data from EZproxy to create an amazon-like recommender service for a discovery system. Part of the Jisc Activity Data programme

• Working with Jisc Library Analytics and Metrics Project (LAMP) and the Jisc Learning Analytics library community panel

• Keeping up-to-date with work on projects in other HEIs exploring the relationship between library use and student success

6

BackgroundExploring the potential of library data 2013-2014

Needham, Gill; Parker, Jo; Nurse, Richard; Scantlebury, Non and Dick, Sam (2013).

Can an excellent distance learning library service support student retention and how can we find out?

Open Learning, 28(2) pp. 135–140. http://dx.doi.org/10.1080/02680513.2013.847364

Article in Open Learning 2013 and

Poster at Design4Learning conference 2014

http://dx.doi.org/10.1080/02680513.2013.847364

7

BackgroundExploring the potential of library data 2014-2015

• Working with institutional Learning Analytics project• Providing library data to module teams• Menu of potential ‘data-driven’ interventions include library

activities

Why

University of Huddersfield 2013‘There is a statistically significant relationship between library resource use and level of degree result’

University of Minnesota 2014‘Students who utilize academic libraries within their first year have

higher cumulative grade point averages and retention, on average, than their peers who did not use library services.’

Library Analytics

“Using data about student engagement with library services and content to help institutions and students understand and improve library services to learners.”

Other UK HEIs report a correlation between student success /

retention and library use – is this the case at the OU?

8

9

Strategy

Project overview

Strategy Evidence impact Integration Skills Infrastructure Business as usual

Data on library use by students not

integrated into institutional data

spaces

No strong evidence base of relationship between

library use and retention/achievement at

OU

Library Services lack data analysis

skills

Challenges

Focus areas

10

11

How can library data benefit students, faculties and the University as a whole?

Analysing OU Library

data alongside OU data

will help to:

•Better understand student success factors; it may also help to improve the accuracy of student success predictions•Provide evidence and understand the impact of students’ engagement with independent study•Inform improvements to library services and trigger interventions for disengaged students

12

Intended outcomes

We can demonstrate that library engagement enhances student achievement and retention

Library engagement data is available for use across the OU and routinely being used where key decisions are made about learning

Library Staff are confident with using data to show how library use contributes to student success and retention

Faculty can be confident that library engagement improves student achievement and retention

Faculty can access real-time library data to identify student engagement

Faculty can be confident that library staff can use data to support retention & achievement

Contribute to improved student success: increased

retention and completion

Progress and findings

13

Skills (library staff)Readiness assessment

14

Strongly Disagree NeutralDisagree Agree Strongly

Agree


Agree


Agree


Agree


Agree

I am confident that I know what analytics data the library has and I

know how to access it

I am confident that I know what analytics data the university has

and I know how to access it

I am confident in my ability to use simple analytics tools

I am confident in my ability to analyse data

I am confident in my ability to use data to support decision making


AgreeOVERALL

High

Feb 2016 SDH

Confidence levelLow

Overall confidence in finding and using data is low.Library staff will need more help to find and use library and university data.Once data tools are in place then they will need help to build skills in analysing and using data.

Online library resource accessesData on online library resource accesses is taken from EZproxy and OpenAthens systems

15

Online library resource accesses

16

• Starting point URL only

• Data files include students who are studying more than one module

• Data files include data from students who didn’t complete their module

Level 1 Level 2 Level 3 PG0

5

10

15

20

25

30

35

40

7.23

13.22

23.25

33.44

Online library resource access by level (all faculties, 2014 presentations)

Mean per student

AVER

AGE

NU

MBE

R O

F AC

CESS

ES O

F LI

BRAR

Y RE

SOU

RCES

PER

STU

DEN

T (M

EAN

)

17

Online library accesses and student achievement

Deferral

Fail

Pass

Distinction

0.0 2.0 4.0 6.0 8.0 10.0 12.0 14.0 16.0 18.0 20.0 22.0

n = 40

n = 2070

n = 6434

n = 1029

Mean number of library resources used per student - 3 level 1 modules (Arts, Law, Technology)

2014-15 presentations

Number of library resource accesses per student (mean)Library Data project 2015/16

n=11,501n=11,406

Pilot study:

18

Pilot StudyLibrary use and Age

i Under 25 ii 26-35 iii 36-45 iv 46-55 v 56 and over0.00

2.00

4.00

6.00

8.00

10.00

12.00

14.00

n = 3382

n = 3805

n = 2000

n = 1324

n = 990

Age groups - average use of online library resources - 3 level 1 modules (Arts, Law, Technology) 2014-15

AV

ER

AG

E N

UM

BE

R O

F L

IBR

AR

Y R

ES

OU

RC

ES

A

CC

ES

SE

D

19

Pilot StudyLibrary use and gender

Female Male0.00

1.00

2.00

3.00

4.00

5.00

6.00

7.00

n = 6157

n = 5344

Average use of online library resources by gender - 3 level 1 modules (Arts, Law,

Technology) 2014-15

AV

ER

AG

E N

UM

BE

R O

F L

IBR

AR

Y R

ES

OU

RC

ES

A

CC

ES

SE

D

20

Pilot Study Library use and level of educational qualification on entry

1 No Formal Qual-ifications

2 Less than A Levels

3 A Levels or equivalent

4 HE Qualification 5 PG Qualification 6 Not known0.00

2.00

4.00

6.00

8.00

10.00

12.00

14.00

16.00

n = 291 n = 3888 n = 4839

n = 2085

n = 370

n = 28

Average use of online library resources by previous education - 3 level 1 modules (Arts, Law, Technology) 2014-15

AV

ER

AG

E N

UM

BE

R O

F L

IBR

AR

Y R

ES

OU

RC

ES

AC

CE

SS

ED

Library Data Project 2016

n=11,501

21

Online library resource accesses and student achievementWider study:

Fail

Pass

Distinction

0.00 5.00 10.00 15.00 20.00 25.00 30.00

n = 46300

n = 176268

n = 29041

Mean library resource accesses per student - 2014 and 2015 presentations (all modules)

Mean resource accesses per student

n = 315,460

22


23


The Wollongong methodology

24

University of Wollongong Library

Cox, B. L. and Jantti, M. (2012) Capturing business intelligence required for targeted marketing, demonstrating value, and driving process improvement, Library & Information Science Research. 34, PP308-316 doi:10.1016/j.lisr.2012.06.002

• Looked at whether there is a relationship between library resource accesses and student success

• Aimed to demonstrate Library impact• The approach and methodology was tested with OU data with some

small adjustments.• UoW used time accessing resources (from the full ezproxy logfiles),

OU has used count of accesses ezproxy and openathens for 2014 students

• UoW used Weighted Average Marks – OU used Overall Continuous Assessment Score (OCAS)

http://dx.doi.org.libezproxy.open.ac.uk/10.1016/j.lisr.2012.06.002

Replicating the Wollongong study

25


26

The average continuous assessment score for students who did not access OU Library online resources is 64. The average continuous assessment score for students that accessed one resource during the year was 68.

There is a non-linear correlation between average resource accesses and average student Score (R squared = 0.54)


27

Findings so far

●The percentage of students accessing library resources and the number of library resources they access on average increases the higher the level of study

●There is quite a wide variance between different faculties, levels and modules in access to library resources

●There seems to be a similar relationship between library access and student success at the Open University, despite students’ mainly online library access

●Early indications are that the relationship is statistically significant

Key indications from our early studies are:

28

29

What’s next?

Questions?Project Blog: www.open.ac.uk/blogs/LibraryData/

30

http://www.open.ac.uk/blogs/LibraryData/

Education

Is there a statistically significant relationship between library resource access and student attainment at The Open University, a distance learning institution