Embed Size (px)
Citation preview
930 Journal of Dental Education ■ Volume 80, Number 8
A Cavity Preparation Evaluation System in the Skill Assessment of Dental StudentsHuiru Zou, DDS; Shufeng Jin, DDS; Jianping Sun, BD; Yanmei Dai, MDAbstract: The aim of this study was to compare a computerized, laser-scanning Cavity Preparation Skill Evaluation System (CPSES) with conventional teachers’ eye-hand grading assessment of dental students’ Class I cavity preparation evaluations. Thirty-eight cavity preparations of lower left first molars made by junior dental students at a dental school in China were tested from September 2013 to November 2014. The outline and retention form, smoothness, depth, wall angulation, and cavity margin index of the preparations were evaluated by CPSES and then by teachers’ eye-hand grading. The mean difference in scores for each method was considered, as was the variability of scores within each method. Compared with the teachers’ eye-hand grading method, CPSES provided objective evaluation results that had statistically significant differences (p<0.05). A questionnaire was also designed and distributed to the students; the response rate was 100%. The results indicated that most of the students recog-nized CPSES effects in the preclinical teaching; 92.1% perceived that CPSES provided high simulation and appropriate practice guidance for them; and 94.7% reported that the evaluation results provided by CPSES gave targeted and objective recommenda-tions. These findings suggest that CPSES can consistently and reliably scan a student’s tooth preparation, compare it to a theo-retically ideal preparation, and provide objective feedback. The application of CPSES in preclinical operative training can help students better understand the desired parameters for occlusal cavity preparation skills and encourage students in their self-paced learning and independent practice.
Dr. Huiru Zou is Associate Professor, No. 2 Teaching and Research Section of Operative Dentistry and Endodontics, Stoma-tological Hospital of Nankai University, Tianjin, China; Dr. Shufeng Jin is Associate Professor, No. 2 Teaching and Research Section of Operative Dentistry and Endodontics, Stomatological Hospital of Nankai University, Tianjin, China; Jianping Sun is Lab Assistant, No. 2 Teaching and Research Section of Operative Dentistry and Endodontics, Stomatological Hospital of Nankai University, Tianjin, China; and Dr. Yanmei Dai is Professor, No. 2 Teaching and Research Section of Operative Dentistry and Endodontics, Stomatological Hospital of Nankai University, Tianjin, China. Direct correspondence to Dr. Yanmei Dai, Stoma-tological Hospital of Nankai University, No. 75 Dagu Road, Heping District, Tianjin, China 300041; 86-022-27119191-1803; [email protected].
Keywords: dental education, restorative dentistry, cavity preparation, assessment
Submitted for publication 11/12/15; accepted 1/26/16
Cavity preparation is a basic skill that all dental students must master. It is very important for teachers to objectively and accurately evalu-
ate students’ cavity preparation skill levels and pro-vide appropriate guidance, correcting any perceived problems a student may have. It is also important for students to effectively assess their own cavity prepa-ration work and design further training as needed. Traditional methods rely on teachers’ visual and hand-grading assessment, which is prone to variable results with uncertainty, subjectivity, and bias, even within peer group assessments.1-3 Students may not understand a teachers’ judgment and think favoritism or bias exists.4,5 Attempts to use digital cavity prepa-ration skill evaluation systems have been reported in recent years with varying degrees of success.6-11
The assessment methodology used for cavity preparations must be valid, reliable, and reproduc-ible.12 The Cavity Preparation Skills Evaluation Sys-tem (CPSES) used in this study is a three-dimensional laser scanning device with an image-processing system. This system was designed to provide ob-
jective and accurate measurement, evaluation, and scoring of a prepared cavity in order to avoid human error or bias. We hypothesized that CPSES would be useful in providing students with standardized assessment results better than a teacher’s eye-hand grading methods.
Therefore, the aims of this study were to evalu-ate the effects of CPSES in a preliminary preclinical operative dentistry setting and to compare it to the conventional teachers’ eye-hand grading assessment. To collect feedback from students about whether CPSES helped them appreciate their practice more and develop better understanding of cavity prepara-tion method and skills in the preclinical environment, a survey was conducted among the students after they finished cavity preparation training.
Materials and MethodsThis study was approved by the Medical Eth-
ics Committee of Tianjin Stomatological Hospital
August 2016 ■ Journal of Dental Education 931
lower left first molar teeth. One tooth was prepared by each student as part of the cavity preparation training course. Students are required to complete the preparation within one hour. All students were sup-plied with an ISO standard fissure bur with continual water spray and suction. The teachers then evaluated the students’ cavity preparation skill levels with the traditional assessment criteria using both the CPSES system and the hand-eye method.
The CPSES system scans, evaluates, and scores the outline form and depth of cavity preparations against a theoretical ideal using a laser beam to scan the tooth mounted on a special jig (Figure 1). The laser beam image is taken with two specialized digital cameras and is converted from analogue to digital with an image-processing system (diagram of the system is shown in Figure 2). The converted image is transferred onto a computer; then, the contour and depth of the cavity are calculated. Finally, accurate three-dimensional images of the prepared tooth and ideal tooth are displayed on the screen with the score and evaluation details, which can be printed out.
Before the elements are measured, various parameters should be set according to the cavity preparation requirements. By setting up evaluation segments that can be accurate to each array element, the area that should not be drilled or the area over the limit and demerit scores were predetermined. The
of Nankai University (Ethics ID E2012005). The CPSES scanner and software system (Version 3.0.2.2), the lower left first molar of the CPSES standard model teeth, and Class I cavity preparation master models were all purchased from UNISN Inc. (Osaka, Japan). Mouth mirrors, sharp dental explor-ers, and periodontal explorers were prepared for use during the procedure.
Participants in the study were 38 Grade 4 den-tal students at Stomatological Hospital of Nankai University, Tianjin, China. In China, for the first three years, undergraduate dental students take social science and general medical courses. They begin to take stomatological courses in Grade 4, when they are called juniors, and progress to preclinical practice in Grade 5. Junior dental students were chosen for this study because they begin performing cavity prepara-tion from level zero, and they are inexperienced with handpiece handling.
All the students were given the same intro-ductory lecture via PowerPoint presentations using CPSES standard ideal cavity images, which can zoom in and out, rotate, and display cavity preparation effects from various angles. They also received the same demonstration from one trained teacher on the design and instrumentation of a conventional Class I cavity preparation. After that, students were asked to prepare a Class I cavity on CPSES standard model
Figure 1. Jig and space calibrator set and installation of special jig with tooth mounted before evaluation
Note: Panel A shows the jig and space calibrator set; panel B shows installation of the special jig with tooth mounted.
932 Journal of Dental Education ■ Volume 80, Number 8
master tooth with an ideal cavity preparation was scanned into the program with the CPSES scanning system. Then, the students’ cavity preparations were scanned and compared to the master scan. The soft-ware measured overdrilling or drilling insufficiency and displayed various degrees of these discrepan-cies in different colors. Warning areas and critical errors entry into the system were agreed upon by all evaluators.
The outline form score was calculated accord-ing to the measurements corresponding to width and depth of the prepared cavity and desired cavity. In this way, the student outline form score was equal to: outline form overdrilling x outline form drilling insufficiency score ˆ ½; outline form overdrilling score = 100 – outline form overdrilling demerit; out-line form drilling insufficiency score = 100 – outline form drilling insufficiency demerit score. The depth score was calculated from overdrilling and drilling
Figure 3. Overdrilling and drilling insufficiency evaluation map with demerit score segment
Note: Panel A shows overdrilling; panel B shows drilling insufficiency.
Figure 2. Slit-ray projection method measurement principle diagram
August 2016 ■ Journal of Dental Education 933
was carried out in a double blind method by three experienced and calibrated faculty members. Before acting as assessors, all faculty members were trained with the same criteria and passed exams with 95% accuracy.
SPSS software (Version 17.0) was used in the statistical analysis (Friedman Test and Wilcoxon Matched-Pairs Signed-Rank Test). The mean differ-ence in scores was rated for each method as was the variability of scores within each method. Descrip-tive statistics and signed-rank tests for systematic bias were used for intra- and interrater comparisons to identify any bias or built-in error. The intraclass correlation (ICC) was used as a measure of intra- and interrater measurement reliability.
A questionnaire was designed and distributed to the students after they finished the training to assess their attitudes about CPSES usage. There were ten questions concerning the teaching effects and evalu-ation accuracy. Response options were 1=strongly disagree, 2=disagree, 3=neither disagree nor agree,
insufficiency of the preparation depth data: depth score was equal to depth overdrilling score x depth drilling insufficiency score ˆ ½; depth overdrilling score = 100 – depth overdrilling demerit score; depth drilling insufficiency score = 100 – depth drilling insufficiency demerit score. A total score was then calculated from the combined form and depth scores: total score = (outline form score x depth score) ˆ ½. After comparing the student-prepared cavity with the master standard cavity, the system highlighted any differences with the overprepared areas in several colors and the underprepared areas in other colors (Figure 3). Meanwhile, detailed comments were generated and displayed on screen such as “The lingual groove is underprepared.” Figure 3 shows the overdrilling or drilling insufficiency evaluation map and demerit score segment. Figure 4 shows the report CPSES provided to students. The system is accurate to ±35 µm.
CPSES scored the cavity preparations from 0 to 100 according to its evaluations. Eye-hand grading
Figure 4. Evaluation result sample provided by Cavity Preparation Skill Evaluation System .
934 Journal of Dental Education ■ Volume 80, Number 8
4=agree, and 5=strongly agree. The average score was calculated according to students’ selected num-bers and students’ numbers: formula = (1×students selected 1 numbers + 2×students selected 2 numbers + 3×students selected 3 numbers + 4×students se-lected 4 numbers + 5×students selected 5 numbers) ÷ total student numbers.
ResultsNone of the instructors achieved the minimum
acceptable intrarater agreement of 0.8. Interrater agreement was substantially less than intrarater agreement for the teachers’ hand-eye grading, whereas CPSES gave consistent results throughout (correlation coefficient 1.0). This finding suggests that hand-eye grading was limited by poor intrarater reliability and low interrater consistency. The CPSES showed potential as an evaluation process as it had excellent intrarater reliability and correlation. The calculation of the correlation coefficient (Table 1) demonstrated a satisfactory relationship between hand-eye grading and CPSES rating concerning outline form and Class I cavity depth.
Table 2 shows the student survey results re-garding CPSES application effects in the preclinical teaching. A response rate of 100% was obtained on the survey. The average score given to CPSES was 3.32-4.42 in the preclinical teaching scenario. Of these students, 92% agreed or strongly agreed that CPSES provided high cavity preparation similarity and appropriate practical guidance for them, and 94.7% agreed or strongly agreed that the evaluation results provided by the system gave targeted and ob-jective recommendations. Some students suggested that teachers should explain the CPSES system evalu-ation results appropriately and share their opinions with students, making the evaluation system more definitive and objective.
DiscussionTraditional cavity preparation training consists
of teacher lectures and demonstrations with stu-dents’ observations and practice, followed by visual evaluation, critique, and correction when necessary. Students need to fully understand the requirements of cavity preparation before commencing clinical practice and to be able to know objectively their skills levels afterwards. The visual evaluation of the quality of a student’s cavity preparation by teachers Ta
ble
1. R
esul
ts o
f sta
tist
ical
ana
lysi
s
Cor
rela
tion
and
Test
Sta
tistic
s C
PSES
1
CPS
ES 2
TE
HG
M 1
TE
HG
M 2
TE
HG
M 3
CPS
ES 1
cor
rela
tion
coef
ficie
nt
1.00
0 (–
) 1.
000
(–)*
0.
793
(0.0
00)*
0.
513
(0.0
01)*
0.
795
(0.0
00)*
Z
– 0.
000
(1.0
00)b
-3.0
32 (0
.002
)a -2
.032
(0.0
42)a
-3.8
79 (0
.000
)a
CPS
ES 2
cor
rela
tion
coef
ficie
nt
1.00
0 (–
)*
1.00
0 (–
) 0.
793
(0.0
00)*
0.
513
(0.0
01)*
0.
795
(0.0
00)*
Z
0.00
0 (1
.000
)b –
-3.0
32 (0
.002
)a -2
.032
(0.0
42)a
-3.8
79 (0
.000
)a
TEH
GM
1 c
orre
latio
n co
effic
ient
0.
793
(0.0
00)*
0.
793
(0.0
00)*
1.
000
(–)
0.77
7 (0
.000
)*
0.75
2 (0
.000
)*Z
-3
.032
(0.0
02)a
-3.0
32 (0
.002
)a –
-0.0
74 (0
.941
)a -0
.937
(0.3
49)a
TEH
GM
2 c
orre
latio
n co
effic
ient
0.
513
(0.0
01)*
0.
513
(0.0
01)*
0.
777
(0.0
00)*
1.
000
(–)
0.56
7 (0
.000
)*Z
-2
.032
(0.0
42)a
-2.0
32 (0
.042
)a -0
.074
(0.9
41)a
– -1
.200
(0.2
30)a
TEH
GM
3 c
orre
latio
n co
effic
ient
0.
795
(0.0
00)*
0.
795
(0.0
00)*
0.
752
(0.0
00)*
0.
567
(0.0
00)*
1.
000
(–)
Z
-3.8
79 (0
.000
)a -3
.879
(0.0
00)a
-0.9
37 (0
.349
)a -1
.200
(0.2
30)a
–
Not
e: C
avity
Pre
para
tion
Skill
Eva
luat
ion
Syst
em (C
PSES
) and
teac
her’s
eye
-han
d gr
adin
g m
etho
d (T
EHG
M) c
orre
latio
n co
effic
ient
s w
ere
dete
rmin
ed b
y Si
g. 2
-tai
led
test
s. C
PSES
and
TE
HG
M Z
s w
ere
dete
rmin
ed b
y A
sym
p. S
ig. 2
-tai
led
test
s.
*Cor
rela
tion
sign
ifica
nt a
t the
0.0
1 le
vel (
2-ta
iled)
. Tes
t sta
tistic
s ba
sed
on W
ilcox
on S
igne
d-R
ank
Test
.
a Bas
ed o
n po
sitiv
e ra
nks.
b S
um o
f neg
ativ
e ra
nks
equa
ls th
e su
m o
f pos
itive
ran
ks.
August 2016 ■ Journal of Dental Education 935
from their teachers and use this knowledge to en-hance their skills, they will become more confident in moving to the next stage of clinical practice.13 How-ever, because of other factors such as time limitations, lack of experience, and subjective assessment, it is difficult for students to consistently obtain accurate, unbiased feedback.5 Educators around the world gen-erally believe that teachers must be calibrated prior to participation in assessment in order to avoid errors in evaluation and increase the likelihood of consistency. However, even after calibration, the same evaluator may give differing assessments of a preparation on different occasions.14 A study to assess consistency of student scores in preclinical training found that almost all the students’ preparations received differ-ent evaluations from within the examiners group,2 suggesting we need to have a more objective and consistent evaluation method.
Currently, several computer-based evaluation systems are used in preclinical training, such as the NISSIN Fair Grader 100 (Japan), KaVo PrepAssis-tant (Germany), and D4D E4D (US).7-11 All of these systems can rapidly scan students’ cavity preparation samples, accurately compare them with standardized
depends entirely on observation by the naked eye and judgment based on experience. This grading system is subject to subjectivity and uncertainty,5 and as the three-dimensional geometry of a cavity is complex, the teaching, critique, and correction procedure is not easy to communicate by language alone and may not even be accurate following detailed explanation. It is therefore difficult for students to grasp accurately and quickly the essentials of cavity preparation skills. However, CPSES can acquire cavity contour and depth data and compare those data with a standard master or ideal preparation, resulting in a compre-hensive and accurate score according to quantifiable evaluation parameters and thereby providing an ob-jective evaluation of results. The results of this study suggest that CPSES can objectively and accurately measure the shape and depth of the prepared cav-ity: it generated scores that were more statistically significantly consistent than the hand-eye methods employed previously.
Accurate assessment of and objective feedback on a student’s early practical technique are very im-portant in dental education. In the preclinical stage, if students can receive consistent and accurate feedback
Table 2. Survey results on Cavity Preparation Skill Evaluation System (CPSES) application effects in preclinical teaching
Item 1 2 3 4 5 Average Score
Teachers are full of knowledge about cavity preparation 0 1 2 30 5 4.03 requirements/methods/skills. (2.6%) (5.3%) (78.9%) (13.2%)
Teachers demonstrate cavity preparation procedure accurately 0 2 4 28 4 3.89 and clearly. (5.3%) (10.5%) (73.7%) (10.5%)
Teachers’ hand-grading judgments are accurate and fair. 1 5 15 15 2 3.32 (2.6%) (13.2%) (39.5%) (39.5%) (5.3%)
The CPSES provided high simulation and appropriate practice 0 1 2 15 20 4.42 guidance for students. (2.6%) (5.3%) (39.4%) (52.6%)
The CPSES set standard and reliable evaluation parameters. 0 1 3 23 11 4.16 (2.6%) (7.9%) (60.5%) (28.9%)
The CPSES inspired my enthusiasm to master cavity preparation 0 1 3 26 8 4.08 knowledge and skills. (2.6%) (7.9%) (68.4%) (21.1%)
The simulated 3D graphics were intuitive and interesting. 0 1 2 18 17 4.34 (2.6%) (5.3%) (47.4%) (44.7%)
The evaluation results provided by the system gave targeted 0 0 2 20 16 4.37 and objective recommendations. (5.3%) (52.6%) (42.1%)
The system was helpful to cultivate our practical ability 0 0 2 18 18 4.42 and spatial imagination. (5.3%) (47.4%) (47.4%)
The CPSES was easy to manage, which permitted us to do 0 1 3 19 16 4.39 extra practice in our spare time. (2.6%) (7.9%) (50.0%) (42.1%)
Note: Response options were 1=strongly disagree, 2=disagree, 3=neither disagree nor agree, 4=agree, and 5=strongly agree. Percent-ages may not total 100% because of rounding.
936 Journal of Dental Education ■ Volume 80, Number 8
perience. The CPSES provides an objective, accurate, and efficient assessment, avoiding the potential for bias inherent in the traditional use of teachers’ visual observation and experience.
Also, this system presents the assessment and evaluation report automatically in a 3D image that is easy to understand, thereby enhancing the learning experience. That in turn saves valuable time for both students and teachers and helps increase students’ enthusiasm for the subject matter. During our study, students’ interest in learning was definitely improved, and at the same time they came to feel more confident in performing their tasks. The students were able to see improvement in their practice and evaluation reports, a process that helps lay the foundation for good clinical practice later and eases their transition to becoming fully fledged dental professionals.
Of course, CPSES does have some drawbacks, such as the need for training before usage, the need for teachers to change some of their previous non-standard teaching habits, evaluation limitations, and high cost of consumables. In the future, if the equipment suppliers are able to reduce the costs, the system could be more affordable, and it could become more widely used.
ConclusionThis study showed that CPSES can be a
powerful tool in dental students’ cavity preparation technique and training procedures. It can provide objective cavity preparation evaluation results com-pared with those of traditional eye-hand grading methods. The use of CPSES in students’ preclinical operative training can help them better understand the desired parameters for cavity preparation skills and encourage their self-paced learning and inde-pendent practice.
AcknowledgmentsThis study was funded by an educational re-
search grant from Nankai University. The funder had no role in the design and conduct of the study; the collection, management, analysis, and interpretation of data; the preparation, review, or approval of the manuscript; or the decision to submit the manuscript for publication.
DisclosureThe authors disclosed no conflicts of interest.
cavity preparation models, and then generate objec-tive 2D or 3D feedback data onscreen for the student. For a group of 50 students’ crown preparations, using the E4D method, only 3% of the variability in the students’ scores was due to which rater evaluated the tooth, while 97% of the variability was due to which tooth was being graded.9 Those results indicated that this particular digital scanning method was more pre-cise than the conventional hand-eye grading method. Using these systems can help improve students’ understanding of and ability to implement cavity preparations, enabling them to enter clinical practice with better caries management skills. However, these systems still have some problems; for example, in 3D scanning and reconstruction, design problems with the grating can lead to blind spots and a resultant scanning blind spot. Meanwhile, evaluating scores in bands such as 90-100 band A, 80-89 band B, and so on does not reflect the accuracy of a definite numerical score as given by the scanning software.
Currently, dental educators in China have started to develop a novel cavity preparation evalua-tion software system of their own, which is still at the developmental stage.15 The CPSES used in this study has several advantages: the system can completely eliminate subjective influences; it can make objective assessments; it can repeat test results without bias; and it can avoid disagreements between students and teachers regarding scores or ratings. In turn, this process provides more opportunity to discover any real weaknesses students may have that may have gone undiscovered during the traditional process of assessment. By comparing students’ efforts to the standard or ideal cavity preparation model, the system can specify whether the student preparation meets the specified requirements such as if there is over- or underpreparation or reduction and also any specific shortcomings such as showing which aspects of the preparation do not meet the specification; it also provides images and highlights to demonstrate these points. Therefore, CPSES can provide targeted feedback, which helps guide students to improve their practical skills.
Another advantage of this system is that it al-lows students to participate in unsupervised indepen-dent practice during their spare timetable sessions. Without the need for teachers’ on-site guidance, stu-dents can use the system for self-assessment, thereby gaining valuable feedback. In this way, students can spend more time on their clinical skills laboratory practice. With the feedback data from the software, students can improve their skills and gain useful ex-
August 2016 ■ Journal of Dental Education 937
tool in a preclinical dental education environment. J Dent Educ 2014;78(10):1416-23.
9. Renne WG, McGill ST, Mennito AS, et al. E4D Compare software: an alternative to faculty grading in dental educa-tion. J Dent Educ 2013;77(2):168-75.
10. Callan RS, Cooper JR, Young NB, et al. Inter- and in-trarater reliability using different software versions of E4D Compare in dental education. J Dent Educ 2015; 79(6):711-8.
11. Schwindling FS, Deisenhofer UK, Porsche M, et al. Establishing CAD/CAM in preclinical dental educa-tion: evaluation of a hands-on module. J Dent Educ 2015;79(10):1215-21.
12. Callan RS, Cooper JR, Young NB, et al. Effect of em-ploying different typodonts when using E4D Compare for dental student assessment. J Dent Educ 2015;79(6): 705-10.
13. Polyzois I, Claffey N, McDonald A, et al. Can evaluation of a dental procedure at the outset of learning predict later performance at the preclinical level? A pilot study. Eur J Dent Educ 2011;15(2):104-9.
14. Karl M, Graef F, Wichmann M, Beck N. Evaluation of tooth preparations: a comparative study between faculty members and preclinical students. Eur J Dent Educ 2011; 15(4):250-4.
15. Wang D, Zhao S, Li T, et al. Preliminary evaluation of a virtual reality dental simulation system on drilling opera-tion. Biomed Mater Eng 2015;26(Suppl 1):S747-56.
REFERENCES1. Jenkins SM, Dummer PM, Gilmour AS, et al. Evaluating
undergraduate preclinical operative skill: use of a glance and grade marking system. J Dent 1998;26(8):679-84.
2. Sharaf AA, AbdelAziz AM, El Meligy OA. Intra- and inter-examiner variability in evaluating preclinical pediatric dentistry operative procedures. J Dent Educ 2007;71(4):540-4.
3. Satterthwaite JD, Grey NJ. Peer group assessment of preclinical operative skills in restorative dentistry and comparison with experienced assessors. Eur J Dent Educ 2008;12(2):99-102.
4. Mays KA, Levine E. Dental students’ self-assessment of operative preparations using CAD/CAM: a preliminary analysis. J Dent Educ 2014;78(12):1673-80.
5. Bush HM, Schreiber RS, Oliver SJ. Failing to fail: clini-cians’ experience of assessing underperforming dental students. Eur J Dent Educ 2013;17(4):198-207.
6. Taylor CL, Grey NJ, Satterthwaite JD. A comparison of grades awarded by peer assessors, faculty, and a digital scanning device in a preclinical operative skills course. Eur J Dent Educ 2013;17(1):e16-21.
7. Cardoso JA, Barbosa C, Fernandes S, et al. Reducing sub-jectivity in the evaluation of preclinical dental preparations for fixed prosthodontics using the Kavo PrepAssistant. Eur J Dent Educ 2006;10(3):149-56.
8. Callan RS, Palladino CL, Furness AR, et al. Effectiveness and feasibility of utilizing E4D technology as a teaching
