Moving into the mainstream: Researching the institutional introduction of EVS, from the realm of the enthusiasts to supporting the later adopters

www.herts.ac.uk

Amanda Jefferies School of Computer Science

Marija CubricBusiness School

Contents

Background

Literature review

Methodology

Preliminary findingsFocus on staff views on technology adoption and use of EVS

Conclusions

Background

Major institutional investment by the university in 2010-11 (Assessment and Feedback project) • 3,845 students form 8 academic schools issued with a personal handset from Jan 2011

JISC EEVS project (stand B) , alongside iTEAM Strand A , started in September 2011• EEVS = Evaluating Electronic Voting Systems for Enhancing Student Experience

EEVS Objectives• To provide an up-to-date view of the student and staff experiences of EVS • To identify a set of critical success factors for introducing and maintaining the use of EVS in

support of an institutional assessment and feedback strategy.• Focus on large-scale deployment and differences across multiple disciplines.

Literature Review: Use of EVS in HE

Survey of literature on the introduction of EVS in HE

• 300+ papers on published between 1998 and 2011• A wide variety of writers :

– Brown, Davis, Draper, d’Inverno, Kennedy, Nichol, Oliver, White et al – Our own early adopters :- Lorimer, Hilliard, Thornton and Willis (University of

Hertfordshire)

Most authors reported on local use in classes: • to engage students, • to encourage interactivity, • to support constructivist approaches to teaching and learning in line with e.g. Chickering and

Gamson’s guidelines.

Lack of literature on institutional deployment, with the exception of:• Twetten et al (2007) Successful Clicker Standardisation, Educause Quarterly 4

– Experience from EVS standardisation in four North American universities

Methodology: Data

Exploratory pragmatic approach:• Gaining understanding of the meanings from various stakeholders • Focus on practical applied research, integrating different perspectives to help interpreting data

(Sounders et al 2009)

Mixed-method data collection:• Qualitative data: students’ blogs, staff interviews, staff reports • Quantitative data: students’ questionnaire, staff questionnaire

Time horizon: Mainly cross-sectional (a ‘snapshot’ taken in 2011/12) with some elements of longitudinal study (students’ blogs over a period of 2-4 weeks).

Sampling: • Self-selection (voluntary student participants for questionnaires)• Self-selection & purposive (voluntary student participants for blogs; further selection made to enable

balanced representation across different schools)• Purposive (heterogeneous): staff involved with or leading EVS adoption in different Schools

Methodology: Students’ survey design

Survey launched in February 2012

Includes 27 questions spread across 9 sections

Total number of responses N= 590 (across 11 different Schools)

Response rate: 14.4%. (based on 4093 EVS handsets registered in June 2012 )

Section Source

Background Information

Standard demographic questions

Usage (Venkatesh et al, 2003 Ease of use Bangor et al , 2008Impact on Learning Draper & Brown 2002Performance Expectancy Venkatesh et al, 2003

Engagement JISC Evaluation guidelinesSatisfaction JISC Evaluation guidelinesOperational procedures and funding iTEAM questions

Free text comments

Methodology: Staff survey design

Survey launched in June 2012

Includes 18 questions spread across 5 sections

Total number of responses N= 88 (across 13 different Schools)

Response rate: 5.9%. (based on 1500 academic staff registered on staffq list in June 2012)

Section Source

Background Information Standard demographic questions

Technology adoption (Venkatesh et al, 2003)

Usage (Venkatesh et al, 2003) + iTEAM questions

Effect on teaching practice JISC Evaluation guidelines

Impact on Learning Draper & Brown 2002Operational procedures and funding

iTEAM questions

Free text comments

Methodology: Students’ blogs: Participants

Student blogs were completed in a variety of ways according to individual preferences : • Word Files (46), online blog entries (14), Short video clips (11), Audio files (1) and• submitted via a restricted access group area on StudyNet (UH MLE)

65 students registered for the web/ written blog of whom 27 were chosen to ensure a spread of gender and programme. One failed to complete their blog

7 Academic Schools: Business (6), Computer Science (7), Humanities (6), Law (4), Education (1), Sports management (1), Mathematics (1)

Equal male and female numbers applied but ratios varied according to subjects 15M:11F

All the bloggers had previously used EVS since at least January 2011

The reflection period varied between 2 and 4 weeks in the period Nov 11- Feb 12

Methodology: Student blogs:Original and Emerging Categories and Themes

All Categories Sub-categories (Themes) Total

Pedagogy

Examples of use ,Interesting lectures, Subject suitability, Subject-specific EVS use, Improvements in use, Potential use, Learning

292Formative use, Inadequate use, Misaligned expectations, Variety of assessment, Edutainment and more …

Usage Previous experience with EVS ,Use by modules and levels ,Frequency of use 78

TechnologyType of handset ,Technical issues ,EVS functions ,

99Changing channels

Institut’l benefits Attendance ,Institutional motivation 47

Expectations Initial thoughts ,Institutional use of data 37

Effort Expectancy Ease of use, Accessibility 40

Facilitating conditions Training ,Handsets distributionTechnical competence of tutors, Tutor's experience with EVS use/question writing 56

Institutional deployment

Handset ownership32

Handset cost, Inconsistent application, Too ambitious implementation

Social aspects Anonymity 28

Student experience Satisfaction and Dissatisfaction 21

Personal emotions Anxiety, and extra pressure, privacy 7

Methodology: Staff interviews: Participants and Developing Categories and Themes Original

CategoriesSub-categories (Themes)

Technology Adoption

Drivers for change, Institutional Support for technology adoption, Positive factors of EVSNegative factors of technology adoptionNegative factors of EVS usageEmbedding EVS successfully

Personal Competence

Training & Support at institutional levelSupport at local levelChampions for EVS Benefits of Training

Changing styles of Pedagogy

Use of MCQs for learning and teachingDeveloping Question BanksBenefits of EVS for Student support

Semi-structured interviews with12 staff members incl:• Local School

Champions (2)• Associate Heads

and Heads of School (3)

• Lecturers (6)• The iTEAM Project

Director (1)

Preliminary Findings : Summary of students’ views (survey and blogs) Usability or ease of use of the EVS was rated very highly by students

Very positive impact on students’ learning and satisfaction:• Responding to questions makes me think more about the

course material (80%+)• EVS provides them with an immediate check of their

understanding (80%+)• Viewing responses give them an idea of how they are doing in

relation to the rest of the class (80%+)

Less positive impact on perceived performance and engagement :• use of EVS will increase their chances of getting a good grade

(49.9%)• Spend more time discussing the subject with colleagues

(47.5%)• Thinks about the subject outside of the classroom (34%)• Ask more questions in classes that use EVS (21%)

Time given for answering questions is not always sufficient

Formative use of EVS has been seen as adding a value to students’ learning experience, however summative use has in some cases created “an unfortunate tension between the administrators, the tutors and the students” (MC12)

All students find the EVS very easy to use, with little or no training; however, accessibility might be an issue for students with sight or dexterity impairments. Majority agree that EVS is a great tool for answering QUESTIONS, but inadequate as a TEST tool

Preliminary Findings : Staff Survey

There is an overall (UH-wide) highly positive attitude to technology (EVS) acceptance with 83% of surveyed staff intending to use EVS in the next 6-12 months.

• A more positive attitude in schools from Social Sciences and Arts subject area than in other subject groups

There is an overall (UH-wide) high agreement on handing out EVS for free upon registration and disagreement that the use of EVS has led to immediate operational efficiencies

Preliminary Findings : Staff interviewsThe Head of School

‘Our biggest challenge was getting staff on board with it. I would think when I started, 25% of staff were very enthusiastic and 75% were not, now we’re probably running at 50/50. Everybody who’s used EVS and got it to work comes back and says it’s great and they’re going to use it again… the technology has got to be easy so that people can get in the swing of it’ Head of School interview June 2012

Preliminary Findings : Staff interviewsThe Lecturers

‘Once it’s embedded in the workload for staff then that should be greatly reduced in terms of marking, writing assignments, it’s really refreshing the bank of multiple choice questions [they’ve created.]’

…it was so obviously simple to generate questions and I think that has always been the main barrier in people’s minds, that because it’s something new and that it’s a technology that it has to be difficult as well

Conclusions

• High student enthusiasm for using EVS in class especially for formative and low-stakes summative assessments

• Plenty of top-down support and leadership and local support is important to embed the technology

• Academics using EVS are not yet seeing all the expected efficiencies in time saved that they had hoped for.

• BUT 93% of those responding to the survey and using it for the first time in 2011 said they plan to use it in the next 6 -12 months

• Patterns of previous technology adoption at UH are mirrored –– The example of the MLE saw early student enthusiasm but

academics took more time to feel fully at ease with the change in practice and pedagogy.

References• Bangor, A., Kortum, P., & Miller, J.A. (2008). The System Usability Scale (SUS): An Empirical Evaluation, International Journal of Human-Computer

Interaction, 24(6).

• Chickering and Gamson http://www.uis.edu/liberalstudies/students/documents/sevenprinciples.pdf

• D’Inverno, R., Davis, H., & White S. (2003), Using a personal response system for promoting student interaction, Teaching Mathematics and its Applications, Vol. 22 (No. 4): 163-169.

• Draper, S. W., & Brown, M. I.(2002).Use of the PRS handsets at Glasgow University, Interim Evaluation Report: March 2002 http://www.psy.gla.ac.uk/~steve/evs/interim.html accessed April 2012

• Draper, S. W., & Brown, M. I. (2004). Increasing interactivity in lectures using an electronic voting system. Journal of Computer Assisted Learning, 20(2), 81-94.

• JISC (2004) http://www.jisc.ac.uk/media/documents/publications/effectivepracticeelearning.pdf

• Kanter, R.M., Stein, B.A. & Jick, T.D. (1992) The Challenge of Organizational Change. Free Press. New York

• Kennedy, G. E., & Cutts, Q. I. (2005). The association between students' use of an electronic voting system and their learning outcomes. Journal of Computer Assisted Learning, 21(4), 260-268

• Lorimer, J., & Hilliard, A. (2009). Use of a Electronic Voting System (EVS) to Facilitate Teaching and Assessment of Decision Making Skills in Undergraduate Radiography Education. Paper presented at the Proceedings of 8th European Conference on e-Learning, Bari, Italy

• Moore, G. A. (1991). Crossing the Chasm. New York: Harper Business.

• Nicol, D., & Draper, S. (June 2009). A blueprint for transformational organisational change in higher education: REAP as a case study. In J. T. Mayes (Ed.), Transforming Higher Education through Technology-Enhanced Learning.

• Oliver, M. (2006). New pedagogies for e-learning? ALT-J: Research in Learning Technology, 14(2), 133 - 134.

• Robins, K., (2011) EVS in the Business School, University of Hertfordshire Internal Report

• Thornton, H. A. (2009). Undergraduate Physiotherapy students’ choice and use of technology in undertaking collaborative tasks. Open University, UK, Milton Keynes

• Twetten, J., Smith, M.K., Julius, J. and Murphy-Boyer, L. (2007) Successful Clicker Standardization EDUCAUSE QUARTERLY • Number 4 2007

• Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. B. (2003). User acceptance of information technology: toward a unified view. MIS Quarterly, 27(3), 425–478.

• Willis, J. (2009) Using EVS in the School of Life Sciences, University of Hertfordshire Internal Report