Massive Open Online Courses in Higher Education – Performance Assessment in Open Learning...

Massive Open Online Courses in Higher Education –

Performance Assessment in Open Learning Arrangements

8th International Technology, Education and Development Conference, 10th-12th March 2014, Valencia

Presenting author: Daniela Pscheida, TU Dresden, GermanyContact: daniela.pscheida@tu-dresden.de

Twitter: @danielapscheida

cMOOCs in Higher Education(c = connectivist)

Connectivist MOOCs are Open Online Courses based on the online activity and thematic exchange of learners instead of broadcasting video lectures.That‘s why this format has high potential for higher education – as it supports the training of important soft skills for the 21st century:

media literacy experiences in personal learning creation of personal knowledge

networks

Challenges of cMOOCs in Higher Education

The implementation of a cMOOC learning scenario is connected to certain challenges:

curricular integration, e.g. if there are participants from different universities and faculties

technological and media capabilities, e.g. if there are participants with different levels of experience

supervision and motivation of students documentation and assessment of learning

performance

Performance Assessment in cMOOCs

Two incompabilities between connectivist learning and higher education:

The context of higher education demands a complete collection and transparent assessment of learning outcomes along predefined learning objectives to award Credit Points – while connectivism encourages learners to focus on their own learning interests and needs (openness of learning process).

The core idea of connectivism assumes intrinsically motivated learners – while student learners are used to receive continuous feedback and orientation (guidance) from lecturers.

The SOOC13

SOOC = Saxon Open Online Course

learning innovation project by three German universities (Dresden, Chemnitz, Siegen)

nine-week course in summer term 2013, 242 registered participants, 122 students

topic: „Learning 2.0 – Individual learning and knowledge management with social media“

course organization via central weblog (www.sooc13.de), materials and impulses

blog aggregator to collect course-related blog posts from participants and tweets with hashtag #sooc13

expert talks and live sessions via Adobe Connect

Tasks and requirements

The SOOC13

regular participation in live sessions (expert talks)

writing and publishing at least one blog post in each thematic section

e-portfolio (collection of the results of research and reflection tasks and smaller experiments)

Performance recording and feedback

The SOOC13

first: RSS feed aggregator later we additionally introduced: e-portfolio-form support by two student tutors likes and individual comments on participants‘

contributions – to stimulate further discussions and debates

no feedback for assessment purposes during the course run

Performance assessment

The SOOC13

no exam assessment based on different course activities,

especially the collected e-portfolio contributions general criteria: spelling, grammar, language,

comprehension, compliance with scientific standards quality of content: understanding and correct

application of technical terms and theories, etc. ‚soft‘ criteria: aspects arising from self-directed

learning and connectivism, e.g. reflectivity and referring back

Evaluation of the SOOC

Survey conducted as online questionnaires at the beginning, in the middle and at the end of the course andvirtual evaluation workshop after the course in the form of an online written discussion

Results presented here summarize the interim and final evaluation of the

sooc13 as well asthe interim evaluation of the sooc1314.

„Do you think you complied with the requirements of the course?“

Evaluation Results from SOOC13

Yes

No

I cannot say

0% 10% 20% 30% 40% 50% 60% 70%

60%

23%

17%

N = 30

more than one third of participants were unsure about or did not think so

the main reason for uncertainty regarding the own performance was a lack of feedback (60%)

„Are you satisfied with the assessment system of the SOOC13?“

Evaluation Results from SOOC13

With 41% of participants being uncertain about that and another 21% saying „no“, there‘s a good chance for improvement.

Yes

No

I cannot say

0% 10% 20% 30% 40% 50%

38%

21%

41%

N = 29

Improvement of cMOOC assessment system by: an instrument, that allows continuous feedback for

contributions increase the transparency in terms of the assessment

criteria

... to avoid uncertainty among students

... to raise the quality of discussions

introduce digital badges for the second course run

Lessons Learned

Digital badges in SOOC1314

Digital Badges

as a formative measurement instrument „small“ badges for each reported contribution of

the e-portfolio form in states:bronzesilvergold

each badge was combined with a brief verbal judgement

to pass the course at least a bronze badge had to be received at any given task

Evaluation Results from SOOC1314

The portfolio-form is easy to use and well structured.

I understand the significance of the badges.

For me the literal assessments clarify the significance of the badges.

For me the Badges and the literal assessments are transparent methods of evaluation.

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

I fully agreeI somewhat agreeI cannot sayI rather disagreeI do not agree at all

N = 20

The evaluation with badges helps me, to appreciate my own achievements.

The volume of the literal assessment is appropriate.

The evalution method with badges and literal assessments is fair.

The literal assessment show me possibilities, where I can improve next time.

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

alwaysoftensometimesrarelynever

N = 20

Evaluation Results from SOOC1314

Evaluation results from the SOOC1314 survey prove, that the new assessment system with badges and verbal judgements is much more transparent and understandable – which also led to an increased satisfaction of participants.

Conclusion

Further innovations at the horizon: using methods of learning analytics to improve the overview and objectivity of a cMOOC in higher education.