Report on ACODE benchmarking July 2018BackgroundAt the April meeting of TELT GG, it was approved that UTAS would participate in the national ACODE Benchmarking of TELT process, culminating in the National Summit on 25-27 June 2018 (see Appendix 1).

UTAS committed to completing an internal review against all eight of the ACODE benchmarking areas. Leaders for the eight areas were nominated and then formed small working groups to undertake the evidence collection and reflection. Evidence was collected over a period of approximately 6 weeks and the groups then reported around 20 June.

At the National Summit UTAS represented by Mr Gerry Kregor and A/Prof Leonie Ellis. There were 24 other institutions from across Australasia present. UTAS was one of only 4 Institutions that completed all eight benchmarks, most institutions completed 2-3. At the Summit each institution in turn presented a summary of the findings of their own internal reviews. This presented the opportunity to listen and learn from the experiences and reflections of other institutions. Where we heard good ideas, we noted them and have included them in the recommendations section under each benchmark area.

The comments and recommendations in this report are based on an internal evidence-collecting exercise against the ACODE benchmarks; this is not necessarily an exhaustive review of TELT (or the areas of L&T, IT operations, student experience, learning and teaching and more with which TELT intersects). This evidence base will inform the reformulation of the TELT White Paper and other learning and teaching initiatives.

Following the National Summit it is noted that UTAS is below the national average on almost every measure. In almost every category the comments from the subgroups points to a lack of funding and to under-staffing for the relative lack of activity and development.

Summary of findings and recommendationsBenchmark 1: Institution-wide policy and governance for technology enhanced learningOpen to Talent gives little specific direction on or commitment to TELT and thereby little explicit acknowledgement of the current demographic of the university in which >50% of student load is online and in which the experience of every student is strongly influenced through contact with TELT platforms, pedagogies and practices.

The Strategic Plan for Learning and Teaching is a little more specific on technology: on page 4 it is stated “Our approaches promote innovation and use technology that enhances learning and teaching”. Further the Strategic Plan for Learning and Teaching sets actions to: (1.4b) provide resources to Faculties to develop blended learning and (2.3c) Support staff to develop competence in and confidence with TELT. Other Actions such as support for innovation can be read to include technology.

In the main directions on TELT (and for a significant portion of Learning and Teaching) have been set by the TELT White Paper but, as has been recently acknowledged, the White Paper was not fully resourced. Areas where progress has been made are those with close alignment with the operational remit of the Divisions and the Faculties/Colleges.

Planning for the development of TELT has not been as structured and focussed as required. Funding for TELT initiatives is difficult to achieve (although the process is relatively transparent) and, when successful provides for capital only and not for ongoing operational costs. TELT infrastructure is in effect in a funding category with a wide range of operational areas including Student Operations and Marketing that are labelled ‘Learning and Teaching’.

TELT Governance Group was established as a sub-committee of ULTC in 2016, with ToR that includes to: “Consult widely with the academic community and other key stake-holders in order to provide advice to ULTC on the development and review of policy that supports the quality of Technology-enhanced Learning and Teaching” and to “Identify, develop and review procedures, guidelines, and approaches that facilitate implementation of the TELT White Paper Principles and development of TELT quality, work practices and platforms”. Relatively little of the business of TELT GG over the past two years has been focussed on this higher-level governance. The same observation could be made in relation to another ToR: “Monitor key indicators of Student and Staff outcomes and experience of TELT for quality assurance and continuous improvement purposes and make recommendations arising from…”. A review of governance structures and operations is currently underway.

There is a general lack of policy to guide the development and governance of the TELT platform and operations. There is little relevant policy on the adoption of new TELT technologies; ITS have general procurement policies and project management methodologies. There are some relevant guidelines for how technology should be used in units and courses (notably the Blended Learning Model and BLM 1-5 framework, the L&T website and more recently from the ePortfolio project) but operationalisation of these has been patchy despite Senate endorsement.

Authority and responsibility for operational management of the TELT platform is generally recognised but not formally defined in detail. A RASCI table of governance, ownership and operational responsibilities is currently being reviewed.

Internal Review Ratings: (details of the criteria are available in Appendix 2)

Criteria 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8Averag

eUniversity of Tasmania 3 3 3 3 2 3 3 2 2.8National Averages 3.6 3.3 3.4 2.6 2.5 3.3 2.9 2.6 2.8

High ratings: six out of eight areas were ranked 3/5

Low ratings: Communication and integration of policy and procedure, use of a policy framework to guide decisions on new technologies

Recommendations: Review TELT Governance structure, operations and effectiveness, especially in relation to leading TELT

development, influencing higher level decision-making and creation of funding bids. Define and align the language used to refer to TELT and online learning in policy and strategy

documents. It is often difficult to read the intentions for TELT in senior documents due to the diversity of language used. TELT may be framed as technology, innovation, online, fully online, distance, blended, off-campus.

Encourage explicit consideration of TELT in College Strategic documents. There is a need for future iterations of the TELT WP to be closer to the policy centre of the university –

also to have explicit senior sponsorship and to be specifically resourced. Develop a suite of policies to guide the development of the TELT platform in areas such as approval of

new technologies and use of external learning technologies. In particular look to Charles Sturt University as an example of such a policy suite.

Create TELT Systems maps, TELT platform diagrams and lists for a variety of purposes and audiences and keep them up to date. Define levels of TELT systems status (trial, pilot, production) and determine policy and procedure to progress system status.

Benchmark 2: Planning for institution-wide quality improvement of technology enhanced learningLittle data is available for the purposes of reviewing; the performance of the TELT platform, quality of TELT provision, the student and staff experience of TELT or progress towards TELT-related targets. Many of our systems and processes can generate data to support decision-making but our organisational capacity to operationalise this is underdeveloped.

TELT, and the tacit acknowledgement of the current demographic of the university in which >50% of student load is online, is frequently lacking from key strategic and quality assurance documents.

As has been noted before, large funded projects that are guided by a project board and the project management capacity of ITS are well implemented and managed. However, when project funding comes to an end and the systems are taken under business-as-usual there is not sufficient funding and staff capacity to monitor their performance in relation to broader TELT quality needs as noted above.

TELT is currently a not a significant consideration in eValuate or cyclical review of courses and units.

Internal review Ratings:(details of the criteria are available in Appendix 2)

Criteria 2.1 2.2 2.3 2.4 2.5 AveragesUniversity of Tasmania 2 5 2 1 1 2.2National Averages 2.5 3.1 3.1 2.4 2.4 2.7

High ratings were given for: Decision making and project management

Low ratings were given for: QA and integration of TELT, Planning for quality improvement, Evaluation against KPIs, reporting of outcomes to all levels.

Recommendations Clarify positioning of the TELT platform with regard to the capital funding process, in particular create

a distinct funding category for student-facing TELT systems (the TELT platform) and processes (rather than the broader category of learning and teaching systems which are mainly for the purposes of student management and marketing). Investigate funding models that will provide adequate operational funds to allow systems owners and managers to maximise system functionality and quality to realise the full value of capital investment.

Implement Quality Assurance Standards around TELT that align with the L&T Strategic Plan. Include consideration of TELT in annual and cyclical unit and course reviews. Quality Matters peer

review would provide a useful guiding framework here. To measure student satisfaction with TELT provision (systems, support, learning design and teaching

practices as appropriate) include mandatory questions about TELT in eValuate OR conduct biannual surveys on student experience of TELT (this could be in-house or leverage existing international surveys such as ECAR or JISC Student Digital Experience Tracker)

Further develop institutional capacity in learning analytics and academic analytics to generate data and information to support decision-making.

Benchmark 3: Information technology systems, services and support for technology enhanced learningThe scope of TELT operations is generally well captured in the ToR of the TELT Governance Group and its subgroups but as noted previously TELT GG has struggled to maintain a relevant high-level agenda. Review of TELT Governance is currently underway.

There is some uncertainty over the scope of the TELT Platform and the TELT process that are relevant to TELT Governance. There is also uncertainty around the business ownership and the role and authority of the business owner of the components of the TELT platform.

Little data is available to the business owner or governance committees on which to base high-level decisions. There is a good range of potential sources of data but underdeveloped understanding to extract and analyse the data, therefore data is not regularly used as an evidence base for governance or management. Individual lecturers have access to reports of student performance (within the constraints of their learning design) but these data are not currently aggregated and reported.

There are good basic working relationships between ITS, TILT, Library, Student Services and the Colleges and the groups meet regularly to discuss current matters relating to TELT systems. Management level conversations happen in a variety of meetings (including TELT PMG and TELT UG) but it is not apparent that deeper and more forward-looking strategic level discussions take place or at least these are not reported to TELT GG.

In the past there has been a comprehensive RASCI table for TELT systems, and some operational areas are still relevant but organisational structures and budgets have changed significantly so the RASCI needs renewing to clarify responsibilities for maintenance of staff and student resource, staff and student training (across all systems) and the relationship between the central units and Colleges.

As noted previously, it is recognised that there are good project prioritisation and project management practices in place, however there is a strong feeling that capital projects are frequently let down (and therefore don’t realise their potential value to the business) because operating funds are not identified, allocated and protected to sustain the capital investment (ePortfolio is a recent cogent example).

Maintenance of systems is managed through the ITS operational budget but growth in the use and functionality of systems is generally not reflected in increased staffing so existing staff have to spread their time across systems and users of those systems.

eValuate does not focus much on TELT systems or experiences so there is little student voice in TELT governance or management.

There is a useful range of support pathways, resources and at-elbow help available across technical, pedagogical and learning design. It was felt that in particular the L&T website could be better promoted to teaching staff.

For individual lecturers, student tracking and reports within MyLO inform teachers of student progress (within the limit of the learning design of the unit)

Risk management is explicitly included in the ToR of the TELT GG and its subcommittees but no agenda items relating to risk have been tabled in the 2-year life of these committees.

Internal Review Ratings(details of the criteria are available in Appendix 2)

Criteria 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 AverageUniversity of Tasmania 2 4 2 3 2 2 3 3 2.6National Averages 2.8 3.4 3.2 3.1 2.4 3.0 3.3 3.2 3.0

High Ratings: Articulation of responsibilities for implementation and maintenance of TELT systems,

Low Ratings: Data and analytics to support decision-making, articulation of responsibilities for support and training for staff and students, resources for implementation and maintenance, support for experimentation with new technologies, Professional Development for staff managing services

Recommendations: Clarify, map and communicate the components of the TELT platform. Create a range of visualisations

for a variety of audiences including; TELT Governance and business owners, managers of TELT systems, teaching staff and students.

Define the role and authority of the business owner of TELT systems. Review the RASCI diagram of operational responsibilities for TELT platforms and operations giving

consideration to all ACODE Benchmarking areas. Develop and communicate 3-5-year roadmaps for all components of the TELT platform. Develop

resource plans to enable enactment of these roadmaps. Improve information on TELT systems (e.g. system status, funding, business owner, manager) and

make this available to TELT managers, governors and eventually to users. Address budget process to allow for capital and operating on projects in order to realise maximum

strategically-aligned value for capital investment.

Define, support and communicate routes for innovation in TELT including a process to suggest and seek funding for new initiatives and technologies.

Benchmark 4: The application of technology enhanced learning servicesThe TELT White Paper sets out a good pedagogical basis and model for the incorporation of TELT in learning and teaching. This is supported by the learning design guidance and documentation on L&T website.

We have a good pragmatic design framework in Blended Learning Model 1-5 framework. In line with long-standing student demand, targets to have sophisticated and useful presence in the LMS (100% at Level 3 on BLM 1-5) have been set but they have not been met. It is not clear how widely unit coordinators are aware of the imperative to meet the Level 3 target because it has not been possible to analyse the level of achievement of BLM levels but currently an attempt is being made to generate proxy measures.

Both of the above resources are based upon and supported by UTAS engagement with Quality Matters, a highly researched and internationally validated QA and design framework.

The focus of learning design has been principally at the unit level. With a current focus on curriculum renewal in line with the Degrees of Difference framework as well as the introduction of Akari curriculum management system and an accompanying Course and Unit Builder many aspects of pedagogical design will become visible and will present new opportunities to increase the coherence of the student experience across courses.

The review found that L&T strategy in general and TELT strategy in particular, are not fully supported with adequate funding or staffing to allow achievement of their objectives. Overall it was felt that the pedagogical intent of TELT is poorly funded and poorly understood. It was felt that the pedagogical intent of the TELT White Paper and good alignment with the Blended Learning Model can only be achieved where academics are able to work with an academic/educational developer. Some Colleges/Schools have access to a larger team whereas in other areas there may be only one or two TELT support staff.

Regarding central technical support it was noted that the university employs only three staff centrally to provide maintenance, upgrade and technical support for the TELT platform and not all are expert in all systems. This is considered inadequate and unsustainable and a situation of high risk to operational continuity to a core business area and consequently to the university’s reputation. UTAS does not fund TELT infrastructure or TELT support staff at levels comparable to other Australian universities.

Staff awareness of the affordances of learning technologies was flagged as a major barrier to adoption, also skills and confidence with technologies. ELT501 includes some guidance on appropriate pedagogies as well as pathways for Professional Development and resources for TELT. This will be picked up in Benchmarks 5 and 6 below.

There are several formal and less formal groups who meet to discuss TELT. In the formal space, here is the TELT Governance structure, which aims to be representative, and other university committees as well as the Teaching Matters conference as an annual opportunity to discuss innovation in learning and teaching (of note: previous work showed that School seminar series are rarely themed on L&T). The

Building e-Learning Community of Practice is less formal but is highly active and meets university-wide on a regular basis and supports ongoing communication and group problem solving through a Facebook group

Teaching Development grants are available through School and College funds, projects are frequently TELT focussed.

Little evaluation of the impact of TELT pedagogies are undertaken. As noted previously eValuate does not adequately focus on TELT and consideration of TELT is not currently included in annual or cyclical reviews. Current evaluations are mostly superficial and frequently focus on specific applications or case studies rather than considering the student demand, student outcomes, opportunities and costs.

Internal Review Ratings(details of the criteria are available in Appendix 2)

Criteria 4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 4.9 AverageUniversity of Tasmania 3 2 2 3 2 2 2 2 2 2.2National Averages 3.3 2.8 2.9 3.2 3.0 3.1 2.3 2.2 2.6 2.8

High Ratings: Nil

Low Ratings: application of TELT in line with institutional strategy, pedagogical intent is apparent to teaching and support staff, resourcing of TELT pedagogies, sustainability of TELT pedagogies, evaluation of impact, evidence of impact used for continuous improvement, good practice examples

Recommendations: Secure funding for the initiatives called for in the current TELT White Paper and ensure that future

iterations of the TELT White Paper are backed by a funded resource plan. Undertake analysis of LMS data to derive proxy measures of adherence to the BLM 1-5 framework.

Develop a system to report these measures via the Business Intelligence System (UMAP) Support the funding of an equitable number of pedagogical and technical support staff in the

Colleges/Schools. Use the EDUCAUSE Core Data Survey or similar as a benchmark for staffing levels. Simplify access and encourage the engagement of academic staff with Quality Matters by providing

funding to TILT to support staff enrolment in the program.

Benchmark 5: Staff professional development for the effective use of technology enhanced learningStaff Development is generally aligned with the Strategic Plan for Learning and Teaching (through annual planning process) but does not use draw on a framework as such.

The current model for professional development in TELT is that professional development is delivered by TILT and staff support is delivered by staff in the Colleges, although some cross-over occurs. ITS has a strong role in support and, occasionally in professional development as part of specific funded projects. Professional development for TELT has been a strong theme at BeL CoP and TELT GG, reports and requests have been forwarded to TELT GG.

Staff Development programs are informed by the needs of academics in meeting aspects of the TPE although this is inferred by TILT staff because the feedback of staff development needs as identified in

academic performance management conversations does not occur. Additionally, Educational Developers in Colleges are aware of some staff development needs through interaction with staff, but data are not shared in any coordinated way between Colleges and central units.

It was noted that most staff development is at an introductory/awareness raising level, units in the Bachelor of Education with Professional Honours (aka Grad Cert ULT) are notable exceptions. ELT501 introduces the Blended Learning Model and additional seminars on the BLM and BLM 1-5 have been offered in Schools on request. ELT521 is a project-based unit that may be focussed on TELT at an advanced level depending on the interests of the enrolee. Two TILT Ed Devs who had significant input to units in the Grad Cert ULT are no longer with TILT.

Quality Matters is a high quality and highly relevant suite of professional development courses and resources. Since 2013 over 120 academic and professional staff have undertaken training with QM, however in the past 12 months the source of central funding has been uncertain, and staff show reluctance to seek relatively small amounts of funding from their School or College even when professional development budgets exist, so there has been little activity.

There are several flexible TELT resources available to staff on demand. The Learning and Teaching website is well aligned to the pedagogical intent of the TELT White Paper and has been recently renewed. Teaching online in MyLO is a unit available in MyLO via the Course Catalog but is currently out of date. Attempts have been to refresh and promote it but staff resource in TILT is currently not available. Quality Matters is highly scalable, fully online and available to start every week of the year.

For general digital skills development, Lynda.com is available to all staff but is not effectively promoted and is not explicitly aligned to skill needed by academics for teaching.

Overall, we have a range of high quality and relevant professional development, but the scope and reach are limited for a variety of reasons

There is currently no identified staff position dedicated to the coordination, design or delivery of TELT professional development.

Evaluation on the Grad Cert ULT is good, drawing on student feedback and staff reflection to guide continuous improvement of TELT components. Other professional development offerings are evaluated on an ad hoc basis and the data is rarely used to inform future offerings.

Internal Review Ratings(details of the criteria are available in Appendix 2)

Criteria 5.1 5.2 5.3 5.4 5.5 5.6 5.7 AverageUniversity of Tasmania 3 3 3 2 2 2 3 2.6National Averages 3.0 2.6 3.0 2.9 2.8 2.8 2.5 2.8

High Ratings: Nil

Low Ratings: Coordination of professional development offerings between TILT, ITS and Colleges, resourcing for TELT professional development, availability of a variety of levels of TELT professional development

Recommendations: Identify or develop a suitable framework for academic staff development which includes a focus on

TELT. Inclusion of a staff digital skills audit may be helpful. Greater coordination between the Colleges and TILT is needed. Develop workflows that facilitate the

forwarding of relevant (aggregated) professional development needs to TILT and/or ITS. Meetings with the Associate Heads L&T of each School or Associate Deans T&L may be productive.

Close the loop on identifying staff development needs by feeding back the aggregated outcome and development needs from academic performance management conversations.

Resource TELT professional development such that targets set in the Strategic Plan for Learning and Teaching and in the TELT White Paper can reasonably be met. o Rebuild capacity in TILT to support the ongoing interpretation of TELT within Grad Cert ULT units

as well as to be able to offer flexible staff development in TELT that is above the introductory level.

o Refresh and relaunch Teaching Online in MyLO. Create TELT Systems maps and lists for a variety of purposes and audiences (including teaching staff)

in order to raise awareness about the TELT Platform and its affordances in learning and teaching. Encourage and/or fund events such as technology showcases and learning and teaching seminars

focussed on TELT and relevant pedagogies.

Benchmark 6: Staff support for the use of technology enhanced learningSupport for TEL takes several forms: self-service resources, service desk and at-elbow support.

The provision of staff support for TELT has changed over recent years. In 2012 the MyLO 2012 project was active and there was substantial provision of support, from project funds, between ITS and TILT. Since then there has been notable decrease in central staffing as there has been an increase in comparable positions in the Faculties/Colleges. Overall TELT support staffing is well below the sector average – It was noted from a 2016 DET report that UTAS has one central support staff member per 298 academic staff whereas Deakin and Griffith had one per 22 and 20 respectively.

Support resources exist across all levels of the institution that provide a basic level of support, however these resources are provided on a reactive just-in-time basis and proactive planning is noted as lacking. Many resources are out of date (frequent systems updates necessitate frequent resource updating) and there has been greater reliance on vendor resources. Vendor resources can be useful, but they do not account for local system customisations or integrations.

With regard to evaluation of staff support, this is done within projects but is patchy and ad hoc at best in business as usual settings. ITS has undertaken surveys associated with the Brightspace Daylight upgrade and MyLO Manager to determine support needs and priorities. Surveys have also been conducted within the Pebble Pad project and Echo360 environmental scan. MyLO is capable of supplying a great deal of data that could be used to track and anticipate support and development needs but systems to exploit this intelligence are underdeveloped.

Coordination of support is also described as ad hoc. The most effective coordination occurs through informal networks and conversations. The TELT Governance structure is noted as being a slow route to achieve action.

Teaching staff can access a range of support services. ITS operate tier 1 and 2 service desk. At-elbow support is provided in Colleges although numbers of support staff and their roles vary between Colleges.

Where funding is available good progress can be made, the ePortfolio replacement project (PebblePad) was highlighted as a project where, through a dedicated project officer, coordination and communication was effective. Funding for that position has now ended and coordinated activity has all but ceased. In general, the project model of system implementation does not account for and resource ongoing support.

Internal Review Ratings(details of the criteria are available in Appendix 2)

Criteria 6.1 6.2 6.3 6.4 6.5 6.6 6.7 6.8 6.9 AverageUniversity of Tasmania 2 2 2 2 2 2 2 3 2 2.1National averages 3.1 2.6 2.4 2.9 3.2 2.9 3.1 2.8 2.4 2.8

High Ratings: Nil

Low Ratings: alignment of support with technologies, identification of support needs, evaluation of support services, coordination of support services, accessibility of support services, resourcing of support services, promotion of support services, use of data for evaluation

Recommendations: Draw on benchmark data (for example the EDUCAUSE Core Data Survey or DET staffing surveys) on

TELT support staffing levels with a view to increasing support to the national average or better and well distributed across all Colleges and Divisions.

Because software is frequently updated, there Is a constant need to review and update resources in line with changes.

Provide relevant outcomes of the ServiceDesk Benchmark survey and similar review activities, to the TELT platform business owner and TELT Governance Group

It was noted that at VUW, Ed Devs are explicitly considered to be outside of some IT policy constraints, otherwise they cannot move fast enough and innovate. This idea should be explored at UTAS to foster innovation.

Benchmark 7: Student training for the effective use of technology enhanced learningIn general responsibility for student training in TELT approaches sits between the Academic Division and the Colleges. Specific responsibilities have not been well defined, but this is within the scope of the TELT platform RASCI (which is currently being reviewed) and monitoring of student outcomes and experience is within the ToR of the TELT GG. The Student Experience Committee also has a strong interest in this area.

Unistart provides some basic awareness and demonstrations of MyLO functionality but attendance is not mandated. There is a Familiarisation Unit for BEng students and, in BDemCare, CAD001 concentrates on IT Skills with specific focus on MyLO, also there is a Common Room with resources around the use of technologies. The first two programs are very limited in time and are not assessed so there is no data to allow continuous improvement. CAD001 is regularly review as any other credit-bearing unit.

There is a self-paced ‘Getting Started in MyLO’ unit available to all students but data from the unit shows that few students do more than superficially explore the site. Also, the ‘Turnitin and Academic Honesty’ available to all students. These are not compulsory and are not assessed or specifically evaluated.

There is a general acknowledgement of little alignment of any central training offerings with the various distinct online pedagogies and teaching approaches across the many disciplines. For navigating MyLO sites and using basic TELT technologies students may be guided by individual academics but this is not easily measured and the same is true for specialised software that may be used in specific disciplines.

In terms of broader student digital skills, there has been a recommendation for a student digital skills audit in each iteration of e-learning strategy/TELT White Paper since 2011 but there has been little action. An attempt to secure a site license for Lynda.com was unsuccessful in 2012.

Regarding ethical use of social media: there is a brief mention of a general responsibilities of users in the ICT Services and Facilities Use Policy but there was no instance discovered where students are guided in ethical use of communications technologies or social media.

As far as was discovered there is little explicit training in use of technologies offered to students, there are no defined pathways to develop digital skills (outside specific relevant degree programs), what little general training does exist is not evaluated and no coordination occurs between any programs. In the main it is felt that students there is little resource, unclear responsibilities and that students are left to fend for themselves, learn from peers in a just-in-time manner or call on support services when they need help.

Internal Review Ratings(details of the criteria are available in Appendix 2)

Criteria 7.1 7.2 7.3 7.4 7.5 7.6 7.7 7.8 AverageUniversity of Tasmania 2 2 1 1 2 1 1 1 1.4National averages 2.9 2.5 2.1 2.1 2.9 2.1 2.5 2.5 2.5

High Ratings: Nil

Low Ratings: Alignment of training with technologies in use, resourcing of student training, evaluation of training, coordination of training, flexibility and pathways in training, training for ethical use of technologies,

Recommendations: Ask students what they expect regarding training in the use of TELT and digital skills, perhaps as part

of a student life cycle mapping. Define a graduate digital skills persona or 21st century skills checklist or similar. Consider a student digital skills audit in the first semester of enrolment and define pathways towards

the graduate-level skills. Resource student digital skills (and TELT skills) development with dedicated staff. Investigate a drop-in

centre for student digital skills development. Investigate a license to allow students access to Linda.com or similar. Collate and promote resources

relevant to digital skills development.

Provide access to exemplar units in the LMS for students well before start of semester, including for overseas students, so that they know what to expect. Highlight to students the digital skills and understanding of relevant pedagogies that are necessary to commence their studies.

Monash College and Federation university performed well on this benchmark. Consider staff interchange or other communications to learn from those institutions.

Benchmark 8: Student support for the use of technology enhanced learningResponsibility for supporting students in their use of TELT is shared between ITS, Student Centre and the Colleges. Specific responsibilities are not well defined. Monitoring of student outcomes and experience is within the ToR of the TELT GG. The Student Experience Committee also has a strong interest in this area.

The support that is offered to students is moderately well aligned with the TELT technologies in use at UTAS. When student support resources were user-tested (by the sub-group members) it was found that support was somewhat fragmented, and many resources were several clicks away from the point of entry to the desired support. Support resources are available at: the MyLO support website, Service Desk, AskUs and there is no unified search across all areas or clear explanation about which site is good for which type of support. The majority of support is focussed on the LMS and Turnitin, with less support available for services like Echo, Collaborate and the other components of the TELT platform. Overall however it was felt that we offer a good range of support resources to students. The support resources are generally well accessible via mobile devices but were not fully tested to WCAG 2.0.

The outcomes of the Student Quality Survey show that students are generally satisfied with access to IT system, access to IT resources and the quality and effectiveness of online learning materials.

Teachers undoubtedly field a lot of TELT support questions from students, but this is difficult to measure.

Coordination between areas that provide TELT support is ad hoc and informal at best through networks such as BeL CoP and the same is true for coordination between training and support.

Development of resources for new technologies that are introduced by capital funded projects is generally quite good but as with staff resources the workload associated with keeping resources up to date was highlighted, and resources and staffing is frequently not available to update resources in a business-as-usual setting.

Internal Review Ratings(details of the criteria are available in Appendix 2)

Criteria 8.1 8.2 8.3 8.4 8.5 8.6 8.7 8.8 8.9 8.10 AverageUniversity of Tasmania 3 2 4 3 3 2 2 1 2 2 2.4National Averages 3.1 2.8 3.1 2.9 2.6 2.2 2.3 2.0 2.6 2.9 2.6

High Ratings: pathways for student access to support services

Low Ratings: Resourcing of TELT support, evaluation and continuous improvement of support, coordination of support and coordination with training, determination of ongoing support needs, planning for operationalisation of support for new technologies

Recommendations: Publish relevant Service Desk request data and ticket resolutions to the TELT Platform Business

Owner, TELT GG and Student Experience Committee to raise awareness and support decision making and resource allocation to student support and training. Also publish relevant data on access to support resources such as AskUs and websites.

Report Student Quality Survey results to TELT Governance Group. Aggregate TELT-related questions and results from across many local and national surveys (eValuate,

SQS, QILT, Library, and others.) to build a more unified picture of student experiences of our TELT services and publish this for reference by relevant groups.

Consider surveying unit coordinators about the TELT related questions that students ask of them or problems that students report to them.

Every MyLO unit should have a link to relevant student support resources, contextualised and extended to support any specialist technologies or learning designs used

Develop a library/repository of student support resources for a range of TELT systems and software in a central location (a MyLO unit) so that they can be copied into units as relevant.

Ask students more often about their experiences of TELT support and their ideas for improvement. Consider a ‘Use of technology’ survey such as EDUCAUSE ECAR (https://library.educause.edu/resources/2017/10/ecar-study-of-undergraduate-students-and-information-technology-2017 ) or JISC Student Digital Experience Tracker (https://www.jisc.ac.uk/rd/projects/student-digital-experience-tracker)

Look to initiatives such as UNE automatic wellness engine to monitor and guide responses to student engagement

Key RecommendationsThe following recommendations have been distilled out of the multifaceted internal review process that was aligned with the ACODE Benchmarks for TEL. These recommendations are further highlighted here because they are foundational to further progress in other areas. The more detailed recommendations above are still relevant but many are already underway or within the direct remit of operational areas of the university.

The Blended Learning Model was established in order to meet students expectations of a 21st century learning environment however from other current work it is apparent that over 40% of units at UTAS do not implement some basics of the Blended Learning Model and only a small minority are at the expected Level 3 on the BLM. It is for this reason that digital skills and other capacity building in our academic staff is considered critical to meeting a wide range of strategic targets.

Identify or develop, and fully fund and enact, a framework for academic staff development which includes a focus on TELT, digital skills and the Blended Learning Model.

Further develop institutional capacity in learning analytics and academic analytics to generate data and information to support audit, decision-making and quality improvement.

Develop a suite of policies to guide the development of the TELT platform in areas such as approval of new technologies and use of external learning technologies.

In order to enumerate and characterize our systems; create TELT Systems maps, that include system status (i.e. evaluation, pilot, production), and refresh the information that we keep on all TELT systems so that their status is better characterized and tracked.

Create and publish 3- to 5-year roadmaps for all TELT systems to support planning, funding and communication.

To measure student satisfaction with TELT provision include mandatory questions about TELT in eValuate. Also conduct biannual surveys on student experience of TELT - this could be in-house or leverage existing international surveys such as ECAR or JISC Student Digital Experience Tracker.

Raise the importance of the student experience for the 50% of our students who study online by reporting specific TELT-related measures and outcomes to Student Experience Committee and to TELT Governance Group.

Ask students what they expect regarding training and support in the use of TELT and digital skills. Create Student digital skills personas and define pathways for students to develop graduate-level 21st

century skills.

Appendix 1Briefing to TELT GG at meeting held on 10 April 2018

3.2 Benchmarking of TELT with Australasian Council on Open, Distance and e-Learning Benchmarks and participation in the National Benchmarking Summit

Responsibility: Gerry Kregor

MOTION(S):

• That the Committee endorses the University's participation in the ACODE benchmarking process including internal collection and evaluation of evidence and participation in the Benchmarking Summit in June 2018.

• That the Committee discusses and determines the scope of participation and coverage of the ACODE TELT Benchmarks (Benchmarks 1-8 or a subset).

On 25-27 June 2018 there will be a ACODE-sponsored Inter-institutional Benchmarking Summit, focussed on Technology-enhanced Learning and Teaching. It is confirmed that over 30 universities from Australia and overseas will participate.

The ACODE Benchmarks have been internationally validated and the process is extremely well supported and guided by comprehensive documentation (http://www.acode.edu.au/course/view.php?id=16). ACODE also provide mentoring support for the process.

To participate fully, the University will need to undertake an intensive internal process to collect and evaluate existing evidence and then reflect on the evidence against defined performance indicators. This will require the active participation and cooperation of several central units (TILT, Student Support, ITS, Library, etc), as well as Colleges, student representatives and senior managers. For each benchmark we would form a working group of 4-6 key stakeholders to undertake the work.

Participation at the Summit requires the University, in turn, to present its findings and reflections in an institutional self-assessment, followed by peer review and sharing of good practice aimed at the formulation of an Action Plan. The University should be represented at the Summit by 2-3 people, probably the Chairs of the working groups. A working group to progress participation, through actioning internal evidence collection, reflection and reporting, will need to be formed.

The Report and Action Plan will be tabled with TELT Governance Group approximately 6-8 weeks after the Summit.

Appendix 2 Summary of the ACODE TELT benchmark performance indicators

Benchmark 1 Institution-wide policy and governance for technology enhanced learning

Scoping Statement This applies to institution level planning, policy development and implementation in relation to the application of technology enhanced learning. It includes the delegation of authority and responsibility for developing and implementing policy, and strategic and operational plans.

Good Practice Statement The institution has established, well understood strategy, governance mechanisms and policies that guide the selection, deployment, evaluation and improvement of the technologies used to support learning and teaching.

Performance Indicators 1. Institution strategic and operational plans support and promote the use of technology enhanced

learning. 2. Specific plans relating to the use of technology enhanced learning are aligned with the

institution’s strategic directions and operational plans. 3. Planning for the ongoing use of technology enhanced learning is aligned with the institution’s

budget process. 4. Institution policies, procedures and guidelines provide a framework for how technology

enhanced learning should be used at both a course and program level. 5. Policies, procedures and guidelines on the use of technology enhanced learning are well

communicated and integrated into processes and systems. 6. The institution has established mechanisms for the governance of technology enhanced learning

that include representation from key stakeholders. 7. Authority and responsibility for the operational management of the technologies used to

enhance learning and teaching are clearly articulated. 8. The institution uses a clearly articulated policy framework and governance structure when

deciding on the adoption of new technologies.

Benchmark 2 Planning for institution-wide quality improvement of technology enhanced learning

Scoping Statement Institution-wide processes are in place, including, planning, implementation, evaluation and feedback loops, to ensure the effective use of technology enhanced learning and its alignment with external requirements.

Good Practice Statement Institutions support and encourage the sustainable, effective and efficient use of technology enhanced learning through strategic planning processes at all levels of the institution. The focus is continuous improvement through systematic and regular evaluation of implementation strategies and outcomes. Such evaluation will in turn inform future planning and align with the institutions strategic direction.

Performance Indicators 1. Institution-wide processes for quality assurance are in place and in use to integrate technology

enhanced learning at both a program and course level. 2. Comprehensive evaluation processes are in place to support decisions relating to the

implementing of technology enhanced learning services. 3. Planning for quality improvement of the institution’s technology enhanced learning systems and

procedures is resourced. 4. Evaluation cycles are in place to measure key performance indicators identified by and for all

stakeholders, and are integrated in planning for continuous improvement purposes. 5. Outcomes are reported to all levels of the institution.

Benchmark 3Information technology systems, services and support for technology enhanced learning

Scoping Statement Information technology (IT) services describe the range of systems and support required to maintain and update the institution’s approach to technology enhanced learning. This can include the use of: learning management systems and their associated systems; library systems; cloud-based tools and services; mobile technologies. It also includes hardware (computers, telecommunications and ancillary equipment) and networks, both internal and external which are used for the purposes of technology enhanced learning, for both on and off-campus environments. Out of scope. The pedagogical issues relating to the use of IT services is the domain of other benchmarks.

Good Practice Statement Technical infrastructure, both physical and virtual, is aligned with institutional learning goals and the technologies are resourced, support staff are trained and the infrastructure is implemented, maintained, administered and supported efficiently and effectively.

Performance Indicators 1. Systems and processes are in place to generate learning and educational analytic data to support

decision making. 2. There are clearly articulated processes, and responsibilities for the implementation and

maintenance of the technology enhanced learning systems. 3. Responsibilities and processes for support and training of staff and students in the use of the

technology enhanced learning systems are clearly defined. 4. Resources are allocated for the implementation and maintenance of IT services that support

technology enhanced learning. 5. Experimentation with new and emerging technologies is encouraged and resourced by the

institution and supported by procedure. 6. Professional development occurs for staff managing the services used to support technology

enhanced learning (including new and emerging technologies). 7. The institution has robust procedures and processes in place to identify and manage risk

associated with all the technology enhanced learning services. 8. Support levels and pathways for assistance for all learning technologies are clearly

communicated to staff.

Benchmark 4 The application of technology enhanced learning services

Scoping Statement This topic addresses the effective application of technology enhanced learning (TEL) services into courses and programs. It encompasses the underlying rationale and strategic intent, how it is embedded into teaching, how it is resourced, evaluated and advanced. The effective pedagogical application of these services is fundamental to the learning and teaching mission of the institution. Failure to apply TEL services in pedagogically sound ways will reduce the value of the investment placed in these services and has the potential to impact on every student and staff member. Out of scope. Technological, policy and administrative issues relating to the application of TEL services are the domain of other benchmarks.

Good Practice Statement The application of TEL services is:

• grounded in the institution’s Learning and Teaching strategy; • informed by good pedagogical practice and research; • supported adequately; • deployed and promoted effectively; • evaluated from a number of perspectives; and • advanced appropriately.

The Performance Indicators are organised to reflect these aspects of pedagogical application.

Performance Indicators 1. The application of technology enhanced learning services are grounded in the context of the

institution’s learning and teaching strategy. 2. The pedagogical intent of the application of technology enhanced learning services within

individual courses and programs is readily apparent to teaching and support staff 3. The pedagogical application of technology enhanced learning is based on sound educational

research and guidelines (including compliance with legal requirements, accessibility, and learning designs) are readily available to all teaching and support staff

4. Collegial communities exist to promote and support the use of technology enhanced learning for communicating and promoting the innovative use and its pedagogical application in learning and teaching.

5. Resources are allocated for the ongoing development of technology enhanced learning pedagogies.

6. The pedagogical application of technology enhanced learning services is sustainable. 7. The pedagogical impact of technology enhanced learning services is regularly evaluated in detail

at a course and program level. 8. Evidence of the impact of technology enhanced learning is integrated into continuous

improvement planning for courses and programmes. 9. Good practice examples advance the pedagogically sound use of TEL services in courses and

programs.

Benchmark 5 Staff professional development for the effective use of technology enhanced learning

Scoping Statement The key focus is on developing teaching staff to make effective use of a range of approaches to technology enhanced learning (TEL). Staff development activities encompass individual and group delivery, face-to-face, as well as online. Self-directed learning activities and resources are also included.

Some professional development will be designed and delivered to meet the strategic needs of the organisation, whilst other activities will be provided to meet the demands of teaching staff as they arise.

Good Practice Statement Quality learning and teaching is brought about where people are confident, enthusiastic, skilled and well supported, and learning experiences are designed to engage the learner and employ a variety of approaches. Engagement in professional development should not be limited by factors of physical location, equity or technological skills. This means that staff development is offered flexibly, accommodates a range of entry points, is evaluated and is informed by the work of related units. A good practice approach to the use of technology enhanced learning reflects an understanding of learners’ characteristics and needs as required by different discipline contexts.

Performance Indicators 1. A framework for staff development in technology enhanced learning is part of the institution's

learning and teaching strategy. 2. Processes are in place and in use to identify staff development needs in support of the

institution’s strategy for technology enhanced learning. 3. Educational and technical expertise is used to develop quality programs and resources

addressing staff development needs. 4. Coordination occurs between those areas providing staff development for technology enhanced

learning across the institution. 5. Staff development for technology enhanced learning is resourced. 6. Staff development programs are delivered flexibly and address differing skill levels. 7. Evaluation data is used to inform the planning for continuous improvement of Staff development

processes.

Benchmark 6 Staff support for the use of technology enhanced learning

Scoping Statement Staff support for the use of technology enhanced learning encompasses both technical and educational support. Technical support is required to deal with problems or needs related to the technological environment, including hardware and software, communications and connections, and performance. Educational support addresses the needs of staff who want to use technologies and/or encounter difficulties while using them, and who need to be able to get ready access to and who want to maximise student learning outcomes Out of scope. This benchmark does not include staff development which forms part of the more formal professional development framework – see Benchmark 5

Good Practice Statement Staff are made aware of and have access to comprehensive technical and educational support for the use of technology enhanced learning tools and services: prior to and during the implementation of the technology, in formal training sessions, on a just-in-time basis, and for troubleshooting purposes.

Performance Indicators 1. Technical and educational support is aligned with the current and emerging learning

technologies being deployed by the institution. 2. Procedures are in place to identify the support requirements of staff, at individual, team and

institutional levels.

3. Procedures are in place to regularly evaluate the support services and resources provided for staff.

4. Coordination occurs between those areas providing support services for staff across the institution.

5. Technology enhanced learning support services are accessible and used by staff. 6. Technology enhanced learning support services are adequately resourced. 7. Technology enhanced learning support services are promoted to staff. 8. New technology enhanced learning services are fully analysed for staff support requirements,

prior to and during the adoption process. 9. There are procedures in place that ensure that evaluation data on technology enhanced learning

support services for staff contributes to their continuous improvement.

Benchmark 7 Student training for the effective use of technology enhanced learning

Scoping Statement Technology enhanced learning services are the systems and tools used by the institution to support learning and teaching. These can include the use of: required computing equipment and software; learning management systems and associated applications; library systems; cloud-based environments; mobile technologies. Aspects of an ethical approach to technology enhanced learning are also included. Student training refers to the applied use of such technologies in a learning context. It can take many forms and be provided by many people, for example through: specific training classes; self-help resources; or as part of a unit of study. Staff providing the training need appropriate skills which require alignment to the professional/staff development benchmark. Out of Scope. Student training does not encompass training in other aspects of learning development (i.e. general study skills).

Good Practice Statement The provision of student training for the effective use of the institution’s technology enhanced learning services is aligned with the teaching approaches in use; is adequately resourced; is coordinated with other student support services; is flexible; is focused on the needs of students; covers a range of current technologies and reflects good practice in the use of technology.

Performance Indicators 1. Student training is aligned with the technologies and teaching approaches in use at the

institution. 2. Student training for technology enhanced learning is adequately resourced. 3. There are procedures in place to regularly evaluate the training and training resources provided

for students. 4. Coordination occurs between those areas providing training for students. 5. Student training programs are delivered flexibly and address differing skill levels. 6. Student training promotes an ethical approach to the use of social media and the technology

enhanced learning services provided by the institution. 7. Evaluation data is used to inform the planning for continuous improvement of student

development processes. 8. There are clearly defined pathways for students to access the training they require.

Benchmark 8 Student support for the use of technology enhanced learning

Scoping Statement Support for students in the use of technology enhanced learning services is defined as primarily technical but the learning context should also be acknowledged. Support should be considered in terms of the use of computers and mobile technologies; learning management systems and their associated applications; library systems, and; those cloud based systems and tools adopted by the institution. The requirements of on-campus and off-campus study should be considered.

Good Practice Statement Students are aware of and have access to effective and well-resourced support for the technology enhanced learning services used by the institution. Student support is responsive to student needs; is coordinated with student training; and is constantly developing in response to changing technology.

Performance Indicators 1. The provision of support for students is aligned with the technology enhanced learning services

used by the institution. 2. Student technology enhanced learning support services are resourced. 3. There are clearly defined pathways for students to access support services and these are

promoted to the student body.4. Support sites and resources are accessible from commonly used devices and the analytics of

student usage are monitored. 5. There are procedures in place to ensure that student support services and resources are

regularly evaluated. 6. There are procedures in place that ensure that evaluation data on technology enhanced learning

support services for students contributes to their continuous improvement. 7. Coordination occurs between those areas providing support for students. 8. There are procedures in place to ensure there is an alignment between student training and

student support. 9. Processes are in place to determine the ongoing support requirements of students. 10. New technology enhanced learning services are fully analysed for student support requirements,

prior to and during the adoption process.

Appendix 3Leaders of benchmarking areas for the internal review

Benchmark Leader1 Institution-wide policy and governance for technology

enhanced learningGerry Kregor

2 Planning for institution-wide quality improvement of technology enhanced learning

Leonie Ellis

3 Information technology systems, services and support for technology enhanced learning

Hilary Soloff

4 The application of technology enhanced learning services Martin Crees-Morris5 Staff professional development for the effective use of

technology enhanced learningAndrea Adam

6 Staff support for the use of technology enhanced learning Kevin Lyall7 Student training for the effective use of technology

enhanced learningDoug Colbeck

8 Student support for the use of technology enhanced learning

Matt Hingston