A sustainable digital assessment tool for post-graduate health-science students in Optometry, Nursing and

Physiotherapy

Dr. Kwang Cham

Melbourne School of Health Sciences,

Department of Optometry and Vision Sciences,

University of Melbourne

@DrKwangCham

• Ms. Anthea Cochrane (Optometry)

• Dr. Rebecca Wong (Dentistry)

• Ms. Bronwyn Tarrant (Nursing)

• Mr. Thomas Mathew (Nursing)

• Dr. David Kelly (Physiotherapy)

• Dr. Elaina Kefalianos (Speech Pathology)

• Dr. Meg Keage (Speech Pathology)

• Mr. Iain Scott (Program Developer)

• Learning and Teaching Initiative Grant at The University of Melbourne

• Faculty of Medicine, Dentistry, and Health Sciences Seed Funding

• Melbourne School of Health Sciences Equipment Support

Project Team and Support

• OSCEs are integral for assessing clinical competency

• Feedback quality and timeliness

• Workload implications

• Adapted an app for oral presentations and trialed it for OSCE suitability

Issue

App Development

• No network connectivity required when operating.

• Scored items are underpinned by automated feedback that is randomly selected to provide a feedback report to students. Google Docs are used to populate different versions of feedback (Fig. 1).

Figure 1: An example of different versions of feedback (Nursing).

App Development

• Functionalities include an inbuilt timer and student number verification before commencing. There is also a Red Flag button that indicates automatic failure (e.g. unprofessional behaviour, performing a technique that is deemed unsafe, hygiene etc.) (Fig. 2).

Figure 2: Built-in timer.

App Development

• Grading options include both formative and summative feedback. Summative feedback provides the option of a checklist and a global score (Fig. 3, Optometry, Nursing and Physiotherapy versions).

Figure 3 : Examples of checklist and global scores (Optometry, Nursing and Physiotherapy versions).

App Development

• This is an example of the feedback report that the student will receive less than 24hrs after assessment.

• Feedback can also be edited (add on/delete) where needed to better match student performance to avoid comments being too generic. (Fig. 4).

• In Physiotherapy specific feedback is also given relative to the category of OSCE question asked e.g. length, strength, passive movement etc.

Figure 4 : An example of a feedback report (Nursing).

Preliminary Findings

SA / A

91%

96%

93%

71%

84%

76%

84%

71%

78%

80%

89%

Preliminary Findings

SA / A

69%

91%

84%

64%

73%

82%

91%

Preliminary Findings

SA / A

100%

100%

88%

100%

75%

63%

100%

100%

100%

100%

75%

Preliminary Findings

SA / A

63%

88%

63%

100%

100%

Discussion

• Traditional paper-based OSCEs are time-consuming to collate and subject to human error. The ability to deliver personalized student feedback is limited.1

• This iPad-based OSCE tool provides timely and specific written feedback in formative and summative assessment in a time- and resource-constrained educational environment. This technological intervention has been well-received by both users and receivers.

• More research is currently being undertaken to evaluate the value, impact, and engagement of the digital tool.

• A comparison of feedback retention and accuracy between verbal and written feedback is also being explored.2,3

References1.Snodgrass, S. J., Russell, T., Ashby, S. E., & Rivett, D. A. (2014). Implementation of an electronic Objective Structured Clinical Exam for assessing practical skills in pre-professional physiotherapy and occupational therapy programs: Examiner and course coordinator perspectives. Australasian Journal of Educational Technology, 30(2): 152-166.2.Humphrey-Murto, S., Mihok, M., Pugh, D., Touchie, C., Halman, S. & Wood, T. J. (2016) Feedback in the OSCE: What Do Residents Remember?, Teaching and Learning in Medicine, 28(1): 52-60.3.Denison, A., Bate, E., Thompson, J. (2016). Tablet versus paper marking in assessment: feedback matters. Perspectives on Medical Education, 5: 108-113.

What’s next

• Pool the data from Physio (110 students) and Nursing (56 students) and present it as Nursing and Allied-health outcomes

• Working with Dentistry and Speech Pathology for their version

• Further funding to work on more flexibility - ability to adjust different criteria and weightings for a particular OSCE