CAA in UK HE:

CAA in UK HE:CAA in UK HE:The State of the ArtThe State of the Art

Bill WarburtonBill WarburtonUniversity of SouthamptonUniversity of SouthamptonEmail: [email protected]: [email protected]

Research & Graduate School of Education

The 2003 surveyThe 2003 survey IntroductionIntroduction DemographyDemography Findings:Findings:

Question banksQuestion banks CAA softwareCAA software Pros & ConsPros & Cons Question typesQuestion types VLEs & MLEsVLEs & MLEs InteroperabilityInteroperability

SummarySummaryResearch & Graduate School of Education

The 2003 surveyThe 2003 survey

Why do a CAA survey now?Why do a CAA survey now? Increasing prevalence of web-based deliveryIncreasing prevalence of web-based delivery Pervasiveness of Virtual Learning Environments Pervasiveness of Virtual Learning Environments

(VLEs)(VLEs) Emergence of Managed Learning Environments Emergence of Managed Learning Environments

(MLEs)(MLEs) Emergence of Interoperability standards e.g. IMS/QTIEmergence of Interoperability standards e.g. IMS/QTI Four year gap e.g. between 1999 & 1995 CAA Four year gap e.g. between 1999 & 1995 CAA

surveyssurveys


The 2003 surveyThe 2003 survey Similarities with the (1999) CAA Centre Similarities with the (1999) CAA Centre

Survey*Survey* For congruence, a superset of the 1999 For congruence, a superset of the 1999

surveysurvey Dissimilarities with 1999 surveyDissimilarities with 1999 survey

Extra questions on VLEs, MLEs & Extra questions on VLEs, MLEs & InteroperabilityInteroperability

Targeted CAA enthusiasts rather than non-Targeted CAA enthusiasts rather than non-usersusers

Delivered purely by web-based CAA systemDelivered purely by web-based CAA system

*Bull, 1999; Bull & McKenna, 2000; Bull & Hesketh, 2001


The 2003 survey: The 2003 survey: Who were the Who were the respondents?respondents? Targeted people who were likely to be using CAA Targeted people who were likely to be using CAA Incentive- prize draw of two £25 book tokens Incentive- prize draw of two £25 book tokens Survey questionnaire was distributed using three JISC mailbase lists: Survey questionnaire was distributed using three JISC mailbase lists:

CAACAA TOIA TOIA ALTALT

50 responses received by initial deadline of 1850 responses received by initial deadline of 18thth May May Will be issued more widelyWill be issued more widely


The 2003 survey: The 2003 survey: Who were the Who were the respondentsrespondents?? CAA Involvement:CAA Involvement:

A constituency of enthusiastsA constituency of enthusiasts

0%

10%

20%

30%

40%

50%

60%

70%

80%

Not currentlyinvolved with

CAA

Currentlyinvolved with

CAA

Intend to beinvolved with

CAA

No longerdinvolved with

CAA


The 2003 survey:The 2003 survey: Job FunctionJob Function Little more than half the 2003 respondents were Little more than half the 2003 respondents were

academics (1999: academics (1999: 77% were academics)77% were academics) An increased proportion of pedagogic & ICT An increased proportion of pedagogic & ICT

supportsupport

0%

10%

20%

30%

40%

50%

60%

Academicteacher

ICT support other pedagogicsupport


The 2003 Survey: The 2003 Survey: Who supports CAA Who supports CAA now?now?

0%

10%

20%

30%

40%

50%

Solely in acentral unit

Solely withinfaculties

Solely inacademic

departments

In a central unitand within

faculties and/ordepartments

Elsewhere

CAA support is largely centralised Even where support has been devolved to faculties /

departments, there is usually a central CAA service


The 2003 survey: The 2003 survey: Who uses CAA now?Who uses CAA now?Usage by Subject : broadly the same as 1999Usage by Subject : broadly the same as 1999

20032003 199919991 1 Computing/IT2 Biomedical Science

3 Other science

4 Geological science5 Business=8 Psychology=8 Leisure=8 Study skills

1 Computing/IT2 Biomedical Science

3 Geological science4 Maths/Engineering

5 Other science6 Modern languages7 Psychology8 Business9 Social science

10 Humanities

11 Built environment

12 Leisure13 Study skills


2003 Survey: 2003 Survey: The use of question banksThe use of question banks

Do you use question banks?

No41%

Yes59%

Would you be interested in sharing questions with academics at other

institutions?

No12%

Yes88%

Nearly 60% said they now use question banks Nearly 90% would be interested in sharing

questions


The use of question banksThe use of question banks Pros of question sharing- shared loads:

Authoring QA Editing

Cons of question sharing: Initial time penalty difficulty of applying rigorous QA standards across multiple authors & institutions requirement for multiple formats to suit all users


The use of question banksThe use of question banks Example in Science:

The e3an project developed an item bank of more than 1500 questions for use in Electronics and Computer Science

Example in Humanities: A question bank has been developed to pre-course assess modern language

students


2003 Survey: 2003 Survey: What CAA software is used in HE What CAA software is used in HE now?now?

0

2

4

6

8

10

12

14

Perception Blackboard WebCT TRIADS i-assess Other

What CAA software do you use?

Commercial CAA systems are more dominant Many respondents used multiple CAA systems

2003 Survey: 2003 Survey: What CAA software is used in HE What CAA software is used in HE now?now?

Split between users of Commercial and Open systems Commercial systems- traditionally seen as expensive but easier to use Non-commercial systems e.g. TRIADS- free, but support-intensive


2003 Survey: 2003 Survey: Advantages of different CAA Advantages of different CAA softwaresoftware

2003 Advantages of commercial systems ranked by popularity:

2003 Advantages of Open systems:

1 Flexibility2 Support for Interoperability: “QTI…which makes it futureproof and easier to share questions”3 Student tracking- 4 Range of Q. types5 Reporting- “ability to produce wide range of reports“=9 Support=9 Scalability=9 Robustness=9 Ease of use

“TRIADS is very flexible”“AIM is tailored to mathematics. It is open source, rather than a professional package”“Ease of use (as I developed it)”


2003 Survey:2003 Survey: Disadvantages of different CAA Disadvantages of different CAA softwaresoftware

2003 Disadvantages of commercial systems ranked by unpopularity:

2003 Disadvantages of Open systems:

1 Limitations of quiz facilities built into VLEs2 Technical issues: e.g. Windows applications only, poor use of the database, non - standard user interface, cost, poor integration with MIS, poor password encryption, ad hoc patches and fixes3 non-intuitive user interfaces=5 Inflexibility: “need to use the underlying html type code to make it do what i want”=5 Restricted feedback and statistical results

“TRIADS is very complex”


2003 Survey:2003 Survey: Main benefits of CAA Main benefits of CAA Little different from 1999, considering small sample 1999 survey:

Speed of feedback to students Statistical analysis of results Automatic marking

2003 survey2003 survey Speed of feedback: Speed of feedback:

““Available 24/7 - with instant feedback”Available 24/7 - with instant feedback” Speed of automatic marking:

“50 pieces of coursework marked in a flash”” Formative use promotes student learning: Formative use promotes student learning:

““students often retake quizzes and improve their score”students often retake quizzes and improve their score”

2003 Survey:2003 Survey: Main disadvantages of Main disadvantages of CAACAA There appears to have been a shift- Learning curve and up-front investment are now seen to be the most prominent disadvantagesThere appears to have been a shift- Learning curve and up-front investment are now seen to be the most prominent disadvantages 1999 survey:

access to and reliability of hardware and software amount of time needed create and organize delivery difficulty of writing good questions

2003 survey:2003 survey: amount of time needed create and organize delivery access to and reliability of hardware and software difficulty of writing good questions


2003 Survey:2003 Survey: Does CAA save time? Does CAA save time?

If academic staff at your institution use CAA, are they saving their own time?

No49%Yes

51%

The 2003 respondents were evenly split regarding the time saved by CAA


Has this meant a significant shift in the workloads of other staff?

No71%Yes

29%

z

Surprisingly, a large majority of those who reported productivity benefits did not see a corresponding shift in workload to other staff

2003 Survey: 2003 Survey: Does CAA save time?Does CAA save time?


2003 Survey:2003 Survey: Formative vs. Summative testingFormative vs. Summative testing

There appears to be a definite shift towards formative use: less than half the 2003 tests were summative (1999: most tests were summative)

Diagnostic use is emergent: other formative usage is long-established

0%

10%

20%

30%

40%

50%

diagnostic formative self-assessment

summative

Type of assessment


2003 Survey:2003 Survey: Question typesQuestion types

1999: MCQ & MRQ most popular followed by text and numeric input, then graphic hot spot and other types

Little appears to have changed in 2003: MCQ still top

0%

10%

20%

30%

40%

50%

60%

graphicalhot spots

MCQs multi-response

other text input

What question types/formats do you use?


2003 Survey:2003 Survey: Who is doing CAA? Who is doing CAA?

0%

20%

40%

60%

80%

No-oneuses CAA

A smallgroup of

enthusiasts

20% ormore

Some nolonger use

CAA

Some may,but notaware

Which of the following describes the experience of the staff at your institution with CAA?

Moore's Gap has yet to be crossed: CAA appears still to be largely the province of ‘early adopters’

Will CAA follow VLEs and become mainstream technology?


2003 Survey:2003 Survey: The growth, influence and relevance The growth, influence and relevance of VLEsof VLEs

Does your institution already provide an institution-wide VLE?

No19%

Yes81%

Have VLEs crossed Moore’s Gap? Are now pervasive Most respondents (81%) had implemented site-wide VLEs It seems that VLE quiz tools give users a taste for CAA The uptake of VLEs may foreshadow the uptake of CAA


0%5%

10%15%20%25%30%35%40%45%

Discouraged Mildlydiscouraged

Neutral Mildyencouraged

Encouraged

Do you think your VLE has encouraged or discouraged the use of CAA?

VLEs are seen as a largely positive (60%) influence on the uptake of CAA

Cautionary note: Two reports that VLEs apparently discouraged the use of CAA

2003 Survey:2003 Survey: The growth, influence and relevance The growth, influence and relevance of VLEsof VLEs


2003 Survey:2003 Survey: Interoperability & VLEsInteroperability & VLEs

Would you like to be able to export quizzes from your VLE to

a CAA system?

I do not use my VLE's quiz

facility38%No

22%

Yes40%

The majority of VLE quiz users wanted to export their quizzes to a dedicated CAA system

One respondent wanted to import CAA items and tests into a VLE


2003 Survey:2003 Survey: tension between VLE quizzes tension between VLE quizzes and CAAand CAA Many respondents attempting to do CAA using VLE’s Quiz engine

Most commented unfavourably on VLE Quiz engines: “…testing facility is very limited, but is all we have available”

Cultural & technical differences between VLEs & CAA: Words do not always have common meanings-

Blackboard refers to a quiz as an ‘item’ In CAA an ‘item’ is usually a single question

VLE quiz items/tests often restricted to proprietary formats, & may be less secure


2003 Survey:2003 Survey: The emergence of MLEsThe emergence of MLEs

To your knowledge, is your institution aiming to develop an MLE?

Yes68%

Don't know26%

No6%

More than two thirds of respondents’ institutions have stated an intention to develop an MLE

However, the concept of an MLE is contended (JISC 7/99 projects)


2003 Survey:2003 Survey: InteroperabilityInteroperability

0

5

10

15

20

25

30

Not at allimportant

Not reallyimportant

Moderatelyimportant

Veryimportant

How important is Interoperability to you?

The majority of HE users thought interoperability to be important


2003 Survey:2003 Survey: InteroperabilityInteroperability

0%

10%

20%

30%

40%

50%

Not reallyaware

Moderatelyaware

Very aware

How aware are you of interoperability standards?

Although most HE users thought interoperability to be important, awareness of interoperability standards varied widely


SummarySummarySuperset of 1999 survey + extra questions on VLEs, MLEs & Superset of 1999 survey + extra questions on VLEs, MLEs & InteroperabilityInteroperabilityDemography: More support staff involved than in 1999Demography: More support staff involved than in 1999Question banks: Desirable but rareQuestion banks: Desirable but rareCAA software: Commercial CAA systems dominant but inflexibleCAA software: Commercial CAA systems dominant but inflexiblePros: Little changed from 1999- time savingsPros: Little changed from 1999- time savingsCons: Learning curve now more significant than costCons: Learning curve now more significant than costQuestion types: MCQs & MRQs still predominantQuestion types: MCQs & MRQs still predominantVLEs: have crossed Moore’s Gap and appear to drive CAA VLEs: have crossed Moore’s Gap and appear to drive CAA MLEs: Most HEIs intend to implements MLEs (but concept of MLE MLEs: Most HEIs intend to implements MLEs (but concept of MLE contended)contended)Interoperability: thought to be important, but awareness of Interoperability: thought to be important, but awareness of standards variesstandards varies


End of presentationEnd of presentation Questions?

Question banks: desirable but how much effort? CAA software: What features do people most want? Pros & Cons: at what point do the pros outweigh the cons? What question types do people most use? Most want? What roles do Interoperability, VLEs & MLEs play in promoting CAA? Do Interoperability, VLEs & MLEs drive CAA or vice-versa? When will CAA cross Moore’s Gap?


Documents

CAA in UK HE: