Embed Size (px)
Citation preview
Preparing for an inter-institutional Benchmarking activity using the ACODE
Benchmarks for TELAssociate Professor Michael Sankey
Vice-President ACODE
Tuesday March 7, 2017
www.acode.edu.au
Introduction• In 2014 & 2016 ACODE ran two major Benchmarking Summits
in Sydney and Canberra, using the ACODE Benchmarks for TEL. • Unique events in Australasian HE, with 35 institutions from 5 countries
engaged so far. • Each first undertook a self-assessment of their capacity in TEL against the
Performance Indicators, then shared this with the other institutions involved. • Each assessed a minimum of two benchmarks, with many doing far more. • To maximize the value of this activity you need to start engaging with the tool
well prior to the formal benchmarking activity. • This workshop will help those new to the ACODE benchmarks:
– understand what is required when using the tool, – provide a guide to ensure participation is undertaken in a rigorous way, and– find a practical way to run an internal activity, before being involved inter-institutionally.
• We will work through a number of different scenarios to help you understand the many facets needing to be considered in undertaking this activity, and look at a plan of action for your institution to be involved.
ACODE
ACODE is a representative body; a council, not a society or guild. It exists to:– disseminate and share knowledge and expertise;– support prof development & provide networking opportunities;– investigate, develop & evaluate new approaches;– advise and influence key bodies in higher education; and– promote best practice.
Well used e-learning quality & benchmarking tools Tool Description Change Theory Validation References
ACODE Benchmarks for TEL
Set of benchmarking statements designed to assist institutions improving the quality of technology enhanced learning. Statements of good practice provided along with a ranking scale. Focus is on a team-based self-assessment. CC Licensed
Collaborative Benchmarking
Face validity supported by expert review. Revised following experience in implementation
Sankey et al. (2014, 2016)
EADTU E-xcellence Next
A framework operated by the European Association of Distance Teaching Universities. Set of quality indicators /benchmarks provided to be used to engage in self-assessments which may be referenced by external QA schemes. CC Licensed
Collaborative Benchmarking
Face validity supported by expert review. Revised following experience in implementation
Ehlers (2012)EADTU (2012)
EFMD Certification of E-learning (CEL)
3 year accreditation scheme for e-learning management operated by the European Foundation for Management Development (EFMD). It including mix of self-assessment and a detailed accreditation audit. It has an 18 month review
None Face validity supported by expert review Ehlers (2012)
EFQUEL UNIQUe Certification
A European Foundation for Quality in E-Learning quality certification , for courses, programmes, institutional systems, to certifies the whole institution. The Guidelines are supplied with supporting questions. Formalised process of of self-assessment and peer review similar to an accreditation audit. Restricted to eligible institutions.
None Face validity supported by literature review and extensive reviews undertaken by experts and quality assurance bodies
EFQUEL (2011)Ehlers (2012)
e-Learning Guidelines (eLG)
A guide to designing, implementing and enhancing eLearning. A framework of questions to encourage reflection by a range of stakeholders. No detailed good practice guidance provided. CC Licensed
None Face validity supported by expert review and literature review. Revised following experience in implementation.
Suddaby and Milne (2008)
e-Learning Maturity Model (eMM)
Quality improvement framework incorporating a benchmarking process and extensive knowledgebase. Extensive set of processes broken down into detailed organisational practice statements. CC Licensed
Maturity Model.
Process & practice revised after 3 rounds of expert consultation conducted internationally, extensive set of cases & a peer-reviewed framework.
Marshall (2012a; 2012b)Neal & Marshall (2008)
Taking the Lead
Not a quality framework, rather a tool for identifying the strategic goals for e-learning to be improved.
None Face validity supported by literature review and case studies.
Quality Matters (QM)
Quality checklist designed to improve individual online courses through a form of audit process. Checklist items supported by descriptions of good practice applied by reviewers, after training. Focus is on PD of staff for online teaching and quality assurance of courses. Not for profit framework requiring a license to use.
None Face validity supported by literature review and case studies.
Varonis (2014)
The Benchmarks• Originally developed
back in 2004• Used many times• In 2014 we shifted the
focus away from ‘eLearning’ to ‘Technology Enhanced Learning’ (TEL)
• Developed the self-assessment templates
• Provide guidelines for using the instrument
1. Institution-wide policy and governance for technology enhanced learning;
2. Planning for institution-wide quality improvement of technology enhanced learning;
3. Information technology systems, services and support for technology enhanced learning;
4. The application of technology enhanced learning services;5. Staff professional development for the effective use of technology
enhanced learning;6. Staff support for the use of technology enhanced learning;7. Student training for the effective use of technology enhanced
learning;8. Student support for the use of technology enhanced learning.
The 8 Benchmarks for TEL
Extension • The methodology provides an institutions with:1. a platform to self-access their standing against
some/all of the 8 benchmarks, and to stimulate meaningful conversations, at a local level, around how you are using technology to support your L&T.
2. an opportunity to share & learn from each other, based on their individual institutions responses (via an inter-institutional event every two years).
3. an enduring record of this via the newly developed online tool (or site)
This resulted in…
Institution BM 1 BM 2 BM 3 BM 4 BM 5 BM 6 BM 7 BM 8Asia Pacific International College X X Auckland University of Technology X XAustralian Catholic University ¢ X¢ X¢ X¢ Australian National University ¢ ¢ Charles Sturt University ¢ ¢ Christchurch Polytechnic X X Curtin University X X Edith Cowan University ¢ ¢ ¢ ¢ ¢ ¢ ¢ ¢Federation University X X¢ X¢ ¢ X¢Flinders University X X La Trobe University ¢ ¢ ¢ Lincoln University ¢ X¢ ¢ X¢ Macquarie University ¢ X ¢ X¢ Monash College ¢ ¢ ¢ ¢ ¢ ¢ Open University - UK X X X¢ X¢ Queensland University of Technology X X RMIT University ¢ ¢ ¢Swinburne University ¢ ¢ University of Auckland ¢ X ¢ ¢ University of Canberra X¢ ¢ X ¢ ¢University of Melbourne ¢ ¢ University of New England X ¢ ¢ X¢ ¢ X XUniversity of Notre Dame ¢ ¢ University of Otago X¢ X¢ X¢ X¢ X¢ ¢ ¢ ¢University of Southern Queensland ¢ ¢ ¢ ¢ X¢ X¢ X XUniversity of South Africa X¢ X¢ X¢ University of the South Pacific X X ¢ ¢ University of the Sunshine Coast ¢ ¢ ¢ University of Tasmania ¢ ¢ ¢University of Technology Sydney X X ¢ ¢ University of Western Australia X X University of Western Sydney X ¢ X¢ X ¢University of Wollongong X X X XVictoria University (Melbourne) X ¢ ¢ ¢ X Victoria University Wellington X¢ X¢ X¢ X¢ X¢ X¢ X¢ X¢
Total (2014) 11 8 8 10 12 9 5 6Total (2016) 12 12 14 16 19 13 6 8
Total 23 20 22 26 31 22 11 14
acode.edu.au
But to get to this point…Managing the logistics
• We first had to do a self assessment
• Pull people together from different sections
• Agree on where we stood• Provide a rationale and
evidence as to why
Short activity: Benchmark 1Institution - wide policy and governance for technology enhanced learning• Performance indicator1. Institution strategic and operational plans support and promote the use of technology
enhanced learning.2. Specific plans relating to the use of technology enhanced learning are aligned with the
institution’s strategic directions and operational plans.3. Planning for the ongoing use of technology enhanced learning is aligned with the
institution’s budget process.4. Institution policies, procedures and guidelines provide a framework for how technology
enhanced learning should be used at both a course and program level.5. Policies, procedures and guidelines on the use of technology enhanced learning are well
communicated and integrated into processes and systems.6. The institution has established mechanisms for the governance of technology enhanced
learning that include representation from key stakeholders.7. Authority and responsibility for the operational management of the technologies used to
enhance learning and teaching are clearly articulated.8. The institution uses a clearly articulated policy framework and governance structure then
deciding on the adoption of new technologies.
At USQ
• Worked with the word templates
• Set up in SharePoint• Had a person .2 of a
workload to wrangle • Each shared their
individual ratings• Then collated the
feedback on the word docs
• Then entered the collated data into the online tool
Split indicators
Victoria University Wellington• They used Google
Docs as the main tool for collaborating on the assessments.
• Created by a simple copy/paste of the word template.
• The goal was to use something easy for a range of staff to edit the files and add content from a range of sources directly.
• VU staff made individual assessments based on local knowledge and evidence (policy and other documentation).
• They were completed by groups of 2-3 staff with a sub-group leader responsible for collating and reporting the assessments.
• The staff involved were drawn from across the university:· Centre for Academic Development· Information Technology Services· Vice-Provost Research· Library· Faculty of Science· Faculty of Education· Student Learning
• The initial assessments were then workshopped with all participants and held in an active learning space that encouraged collaborative editing.
• This information was uploaded into the online system prior to the Canberra workshop.
• Edits and revisions were made back into the Google Docs during the workshop and immediately after to reflect the feedback and discussion at Canberra.
• The resulting information was then edited by CAD and ITS staff to produce an internal report with 13 recommendations arranged in four major themes:– Digital governance;– Digital teaching;– Digital learning; and– Digital understanding.
• The draft report was shared with the complete team to ensure accuracy of the content and agreement that the recommendations reflected a consensus on the important elements needing improvement.
• Each benchmark performance indicator result was presented in context for Victoria against the complete data set:
• These were accompanied by a narrative outlining the rationale and evidence for the assessment and a section labelled “possibilities for improvement.” The latter section outlined what could be done to improve the individual indicator. These were used to identify the recommendations, which typically addressed a range of indicators across the benchmarks.
Key to success• Early involvement with a diverse team of staff. • Two workshop sessions were needed to introduce the tool
and complete the initial evidence gathering, each running for three hours.
• Engaged team leaders made a significant difference in the extent to which evidence was gathered and synthesised, but all teams provided a comprehensive body of evidence.
• Project leaders needed to spend a significant amount of time turning the information into a consistent report but this was useful as a means of summarising progress to date against our Digital Strategy which is due for renewal in 2017
RMIT University
• WE EVALUATE EACH BM by…
1. Form groups of 6 – 10 RMIT staff representing different areas to evaluate each BM
2. Each BM group participant completes & submits a self assessment template (via Google form)
3. Group evaluation meetings are held for each BM, where participants discuss their performance indicator ratings & evidence. The group must then agree on an overall rating for each performance indicator
4. Group evaluations & recommendations for improvement are collated into a report for the VCE
Step 1. Complete the self assessment form• The Google form link in ‘BM Group Member Information’ or
‘Benchmark Self Assessment Forms’ on the site• Select a rating of 1-5 for each performance indicator (PI)• Note your rationale & evidence to support each rating• Note suggestions for improvement at the end of the form • If you cannot answer one of the sections on the form, leave it
blank and move to the next PI (the reason we have groups)• You should not spend more than 2 hours completing the form,
or 10 minutes per PIThe self assessment must be submitted prior to group evaluation meetings
PROCESS FOR GROUP MEMBERS
1. Please note your email, so we don’t keep pestering you
2. Read the Scoping and Good Practice statements
3. These inform your consideration of how you will rate an indicator
Example: SELF ASSESSMENT
Example: SELF ASSESSMENT
1
2
Step 2. Group evaluation meeting• Each group member briefly describes their PI ratings &
supporting evidence/rationale• The group will discuss & choose overall PI ratings for RMIT• Final ratings are based on group consensus, where
possible•BM group leaders will facilitate discussions
Group evaluations will be included in the final reportSelf assessment forms will be not be reported
PROCESS FOR GROUP MEMBERS
Then we have the conversation
In 2016 on average 15 participants per institution, with some 401 people participating overall.4
Head of TEQSA
The beauty of the beast
• The beauty of benchmarking is not around which tool or set of standards you are using, it's about the dialogue that emerges and the sharing of practice that is the real winner for all concerned.
• It opens the door for further collaboration.• It serves as a mechanism to facilitate discussion at
senior leadership level.
Conclusion• Many of the issues we face can be remediated
by simply taking the time to self-assess against a set of quality performance indicators.
• We then extend this by sharing our current practice with those in similar circumstances.
• This builds stronger ties and provides our institutions with the wherewithal to meet the unique challenges of building a strong digital future.
• The ACODE Benchmarks provide a catalyst to help make this happen
• We are not alone
www.acode.edu.au
This June
http://www.open.ac.uk/acode-uk/
http://www.open.ac.uk/acode-uk/
