Journal of

Clinical Medicine

Article

Validity and Reproducibility of the Peer Assessment RatingIndex Scored on Digital Models Using a Software Comparedwith Traditional Manual Scoring

Arwa Gera 1 Shadi Gera 1 Michel Dalstra 1 Paolo M Cattaneo 2 and Marie A Cornelis 2

Citation Gera A Gera S Dalstra

M Cattaneo PM Cornelis MA

Validity and Reproducibility of the

Peer Assessment Rating Index Scored

on Digital Models Using a Software

Compared with Traditional Manual

Scoring J Clin Med 2021 10 1646

httpsdoiorg103390jcm10081646

Academic Editor Letizia Perillo

Received 23 February 2021

Accepted 10 April 2021

Published 13 April 2021

Publisherrsquos Note MDPI stays neutral

with regard to jurisdictional claims in

published maps and institutional affil-

iations

Copyright copy 2021 by the authors

Licensee MDPI Basel Switzerland

This article is an open access article

distributed under the terms and

conditions of the Creative Commons

Attribution (CC BY) license (https

creativecommonsorglicensesby

40)

1 Section of Orthodontics Department of Dentistry and Oral Health Aarhus UniversityC 8000 Aarhus Denmark arwadentaudk (AG) shadigeradentaudk (SG)micheldalstradentaudk (MD)

2 Faculty of Medicine Dentistry and Health Sciences Melbourne Dental School University of MelbourneCarlton VIC 3053 Australia paolocattaneounimelbeduau

Correspondence mariecornelisunimelbeduau

Abstract The aim of this study was to assess the validity and reproducibility of digital scoringof the Peer Assessment Rating (PAR) index and its components using a software compared withconventional manual scoring on printed model equivalents The PAR index was scored on 15 casesat pre- and post-treatment stages by two operators using two methods first digitally on directdigital models using Ortho Analyzer software and second manually on printed model equivalentsusing a digital caliper All measurements were repeated at a one-week interval Paired sample t-testswere used to compare PAR scores and its components between both methods and raters Intra-classcorrelation coefficients (ICC) were used to compute intra- and inter-rater reproducibility The error ofthe method was calculated The agreement between both methods was analyzed using Bland-Altmanplots There were no significant differences in the mean PAR scores between both methods andboth raters ICC for intra- and inter-rater reproducibility was excellent (ge095) All error-of-the-method values were smaller than the associated minimum standard deviation Bland-Altman plotsconfirmed the validity of the measurements PAR scoring on digital models showed excellent validityand reproducibility compared with manual scoring on printed model equivalents by means of adigital caliper

Keywords orthodontics CADCAM PAR index dental models digital models clinical

1 Introduction

For high standards of orthodontic treatment quality to be maintained frequent moni-toring of treatment outcomes is a prerequisite for orthodontists The orthodontic indiceswidely used in clinical and epidemiological studies to evaluate malocclusion and treatmentoutcome [1ndash3] include the Index of Orthodontic Treatment Need (IOTN) [4] the Index ofComplexity Outcome and Need (ICON) [5] the American Board of Orthodontics objectivegrading system (ABO-OGS) index [6] the Peer Assessment Rating (PAR) index [7] and theDental Aesthetic Index (DAI) [8]

The PAR [7] is an occlusal index developed to provide an objective and standardizedmeasure of static occlusion at any stage of treatment using dental models Therefore thisindex is widely used among clinicians whether it is in the private or the public sectorincluding educational institutions In fact in the UK the use of this index is obligatoryin all orthodontic clinics offering public service to audit orthodontic treatment outcomeand it is used as a measure for quality assurance The assessment of malocclusion canbe recorded at any stage of orthodontic treatment such as pre- andor post-treatmentwhereas the difference in PAR scores between two stages evaluates treatment outcome Itsvalidity and reliability on plaster models have been reported in England [9] as well as inthe United States [10] It is also a valid tool for measuring treatment need [11]

J Clin Med 2021 10 1646 httpsdoiorg103390jcm10081646 httpswwwmdpicomjournaljcm

J Clin Med 2021 10 1646 2 of 11

The use of plaster models and digital calipers has been acknowledged as the goldstandard for study model analyses and measurements [1213] Traditionally PAR scoring isperformed on plaster models by means of a PAR ruler or a combination of a digital caliperand a conventional ruler Several studies have used this method to assess malocclusiontreatment need treatment outcomes and stability of occlusion [1114ndash19] However thehuman-machine interface has evolved influencing orthodontics significantly shifting froma traditional clinical workflow towards a complete digital flow where digital modelshave become more prevalent Digital models enable patientsrsquo records to be stored dig-itally and for essential orthodontic assessments such as diagnosis treatment planningand assessment of treatment outcome to be carried out virtually through several built-infeatures such as linear measurements [1320] Bolton analysis space analyses [21] treat-ment planning [22] and PAR scoring [1223] However this modern paradigm demandsadaptation and assessment of applicability in orthodontic clinical work Neverthelessassessments of the validity and reproducibility of 3-dimensional (3-D) digital measurementtools remain scarce

Digital models can be obtained either directly or indirectly and can be printed orviewed on a computer display Scanned-in plaster models are the indirect source of digitalmodels and are as valid and reliable as conventional plaster models [12132425] In thepresent study digital models were obtained directly from an intraoral scanner Emphasis onevaluating a complete virtual workflow was recently implemented by three studies [26ndash28]Brown et al [26] concluded that 3-D printed models acquired directly from intraoralscans provided clinically acceptable models and should be considered as a viable optionfor clinical applications Luqmani et al [28] assessed the validity of digital PAR scoringby comparing manual PAR scoring using conventional models and a PAR ruler withautomated digital scoring for both scanned-in models and intraoral scanning (indirect anddirect digital models respectively) The authors concluded that automated digital PARscoring was valid and that there were no significant differences between direct and indirectdigital model scores

However to our knowledge the digital non-automated PAR index scoring tool of theOrtho Analyzer software has not been previously validated Therefore the purpose of thisstudy was to assess the validity and reproducibility of digital scoring of the PAR index andits components on digital models using this software compared with conventional manualscoring on printed model equivalents

2 Materials and Methods21 Sample Size Calculation

A sample size calculation was performed using the formula given by Walter et al [29]For a minimum acceptable reliability (intra-class correlation (ICC)) of 080 an expectedreliability of 096 with a power of 80 and a significance of 005 a sample of 12 subjectswas needed It was decided to extend the sample to 15 subjects

22 Setting

The study was conducted at the Section of Orthodontics School of Dentistry andOral Health Aarhus University Denmark This type of study is exempt from ethicsapproval in Denmark (Health Research Ethics Committee-Central Jutland Denmark caseno 1-10-72-1-20)

23 Sample Collection

The study sample consisted of 15 consecutive patient records (the first record beingrandomly chosen) selected from the archives according to the following inclusion crite-ria (1) patients had undergone orthodontic treatment with full fixed appliances at thepostgraduate orthodontic clinic between 2016 and 2018 and (2) digital models before andafter treatment were available No restrictions were applied with regards to age initialmalocclusion and end-of-treatment results

J Clin Med 2021 10 1646 3 of 11

The digital models for both treatment stages pre-treatment (T0) and post-treatment(T1) had been directly generated by a TRIOS intraoral scanner (3Shape CopenhagenDenmark) as stereolithographic (STL) files imported and analyzed through Ortho Analyzersoftware (3Shape Copenhagen Denmark) Subsequently 30 digital models were printedto generate 15 model equivalents for each stage by means of model design software (ObjetStudio Stratasys Eden Prairie MN USA) and a 3-D printing machine (Polyjet prototypingtechnique Objet30 Dental prime Stratasys Eden Prairie MN USA) in the same laboratoryand with the same technique

24 Measurements

The PAR scoring was performed at T0 and T1 by two methods (1) digitally on thedirect digital models using a built-in feature of the Ortho Analyzer software (Figure 1)and (2) manually on the printed model equivalents using a digital caliper (OrthopliPhiladelphia PA USA) measured to the nearest 001 mm with an orthodontic tip accuracyof 0001 except for overjet and overbite which were measured with a conventional rulerTwo operators (AG and SG) previously trained and calibrated in the use of both techniquesperformed all the measurements independently Reproducibility was determined byrepeated measurements on all models by both methods and by both raters at a one-weekinterval and under identical circumstances

J Clin Med 2021 10 x FOR PEER REVIEW 3 of 10

(1) patients had undergone orthodontic treatment with full fixed appliances at the post-

graduate orthodontic clinic between 2016 and 2018 and (2) digital models before and after

treatment were available No restrictions were applied with regards to age initial maloc-

clusion and end-of-treatment results

The digital models for both treatment stages pre-treatment (T0) and post-treatment

(T1) had been directly generated by a TRIOS intraoral scanner (3Shape Copenhagen

Denmark) as stereolithographic (STL) files imported and analyzed through Ortho Ana-

lyzer software (3Shape Copenhagen Denmark) Subsequently 30 digital models were

printed to generate 15 model equivalents for each stage by means of model design soft-

ware (Objet Studio Stratasys Eden Prairie MN USA) and a 3-D printing machine (Polyjet

prototyping technique Objet30 Dental prime Stratasys Eden Prairie MN USA) in the

same laboratory and with the same technique

24 Measurements

The PAR scoring was performed at T0 and T1 by two methods (1) digitally on the

direct digital models using a built-in feature of the Ortho Analyzer software (Figure 1)

and (2) manually on the printed model equivalents using a digital caliper (Orthopli Phil-

adelphia PA USA) measured to the nearest 001 mm with an orthodontic tip accuracy of

0001 except for overjet and overbite which were measured with a conventional ruler

Two operators (AG and SG) previously trained and calibrated in the use of both tech-

niques performed all the measurements independently Reproducibility was determined

by repeated measurements on all models by both methods and by both raters at a one-

week interval and under identical circumstances

Figure 1 Peer Assessment Rating (PAR) index scoring using Ortho Analyzer software Anterior component scoring con-

tact point displacement between teeth 21 and 22

The PAR scoring was performed using the UK weighting system according to Rich-

mond et al [7] and included five components scoring various occlusal traits which con-

stitute malocclusion anterior segment posterior segment overjet overbite and centerline

(Table 1) The scores of the traits were summed and multiplied by their weight The com-

ponent-weighted PAR scores were summed to constitute the total weighted PAR score

Figure 1 Peer Assessment Rating (PAR) index scoring using Ortho Analyzer software Anterior component scoring contactpoint displacement between teeth 21 and 22

The PAR scoring was performed using the UK weighting system according to Rich-mond et al [7] and included five components scoring various occlusal traits which consti-tute malocclusion anterior segment posterior segment overjet overbite and centerline(Table 1) The scores of the traits were summed and multiplied by their weight Thecomponent-weighted PAR scores were summed to constitute the total weighted PAR scoreEssential information about each case was considered such as impacted teeth missing orextracted teeth plans for any prosthetic replacements and restorative work previouslycarried out that affected the malocclusion

J Clin Med 2021 10 1646 4 of 11

Table 1 The PAR index components and scoring

PAR Component Assessment Scoring Weighting

1 Anterior dagger Contact point displacement 0ndash4 times 1Impacted incisorscanines 52 Posterior Dagger Sagittal occlusion 0ndash2

times 1Vertical occlusion 0ndash1Transverse occlusion 0ndash4

3 Overjet sect Overjet 0ndash4 times 6Anterior crossbite 0ndash44 Overbite para Overbite 0ndash3 times 2Open bite 0ndash45 Centerline Deviation from dental midline 0ndash2 times 4

Total Unweighted PAR score Weighted PAR scoredagger Measured from the mesial contact point of canine on one side to the mesial contact point of canine on the opposite side (upper andlower arches) and recorded in millimeters as the shortest distance between contact points of adjacent teeth and parallel to the occlusalplane Dagger Measured from the distal contact point of canine to the mesial contact point of permanent molar or last molar (right and left sides)sect Measured as the largest horizontal distance parallel to the occlusal plane from the labial incisor edge of the most prominent upper incisorto the labial surface of the corresponding lower incisor para Recorded as the largest vertical overlap or open bite between upper incisors andlower incisors

25 Statistical Analyses

Data collection and management were performed by means of the Research ElectronicData Capture (REDCap) tool hosted at Aarhus University [3031] Statistical analyses werecarried out with Stata software (Release 16 StataCorp 2019 College Station TX USA)

Descriptive statistics were used to analyze the total PAR scores at different timepoints between raters and methods used Paired sample t-tests were used to comparePAR scoring between both methods and raters at a significance level of lt005 Bothmethods were assessed by ICC for intra- and inter-rater reproducibility Intra- and inter-rater variability were determined by calculation of the error of the method according toDahlbergrsquos formula [32] The agreement between the digital and manual scoring methodsperformed by the two raters was determined by a scatter plot and Bland-Altman plots

3 Results31 Validity

Paired-sample t-tests showed no significant differences in the mean total PAR scoresand in the PAR components between both methods (Tables 2 and 3) The scatter plot(Figure 2a) and Bland-Altman plots (Figure 2b) illustrate agreement of the measurementsconducted with both methods

J Clin Med 2021 10 1646 5 of 11

Table 2 Intra-rater variability (error of the method and Minimum Standard Deviation (MSD)) and reproducibility (ICC) according to time point and method for both raters Paired samplet-tests comparing both methods

Rater I Rater II

PAR Index Scoring and Timepoint ScoringMethod

Mean(SD) a

Err Methoda MSD a ICC [95 CI]

p-Value(Pairedt-Test)

Mean(SD) a

ErrMethod

aMSD a ICC [95 CI] p-Value (Paired

t-Test)

T0Total PAR

Manual 295(101) 09 101 099 099 100

016

305(107) 14 101 099 098 100

074Digital 302

(105) 09 101 100 099 100 304(104) 06 103 100 099 100

T1Manual 11 (14) 02 14 099 099 100

07212 (13) 04 12 095 084 098

049Digital 12 (13) 02 11 098 094 099 10 (09) 02 10 098 095 099

T0

Lower anterior

Manual 41 (40) 05 40 099 099 100012

44 (42) 06 41 099 097 100060

Digital 44 (36) 02 35 100 099 100 45 (37) 04 37 099 098 100

T1Manual 03 (06) 00 06 100 - -

01603 (06) 00 06 100 - -

016Digital 01 (03) 00 04 100 - - 01 (03) 00 04 100 - -

T0

Upper anterior

Manual 61 (28) 06 26 097 092 099045

63 (28) 07 26 098 093 099050

Digital 63 (28) 07 26 096 089 099 64 (27) 06 26 098 095 099

T1Manual 02 (04) 00 04 100 - -

01902 (04) 00 04 100 - -

019Digital 03 (05) 01 05 100 - - 03 (05) 01 05 100 - -

T0

Posterior

Manual 13 (15) 04 14 097 091 099016

14 (15) 00 15 100 - -033

Digital 15 (16) 00 16 100 - - 15 (16) 00 16 100 - -

T1Manual 02 (06) 02 06 095 084 098

06702 (06) 00 06 100 - -

-Digital 03 (06) 00 15 100 - - 03 (06) 00 06 100 - -

T0

Overjet

Manual 120 (75) 00 75 100 - --

120 (75) 00 75 100 - --

Digital 120 (75) 00 75 100 - - 120 (75) 00 75 100 - -

T1Manual 00 (00) 00 00 - - -

-00 (00) 00 00 - - -

-Digital 00 (00) 00 00 - - - 00 (00) 00 00 - - -

T0

Overbite

Manual 31 (18) 00 18 100 - -033

29 (19) 05 18 096 089 099100

Digital 30 (18) 04 18 098 094 099 29 (18) 00 18 100 - -

T1Manual 03 (07) 05 07 100 - -

-03 (07) 00 07 100 - -

-Digital 03 (07) 05 07 100 - - 03 (07) 04 05 100 - -

J Clin Med 2021 10 1646 6 of 11

Table 2 Cont

Rater I Rater II

PAR Index Scoring and Timepoint ScoringMethod

Mean(SD) a

Err Methoda MSD a ICC [95 CI]

p-Value(Pairedt-Test)

Mean(SD) a

ErrMethod

aMSD a ICC [95 CI] p-Value (Paired

t-Test)

T0

Centerline

Manual 21 (30) 00 30 100 100 100033

21 (30) 00 30 100 100 100033

Digital 20 (27) 08 26 096 089 099 19 (26) 00 26 100 - -

T1Manual 00 (00) 00 00 - -

-00 (00) 00 00 - - -

-Digital 00 (00) 00 00 - - 00 (00) 00 00 - - -

Error of the method according to Dahlbergrsquos formula ICC could not be calculated as all the scores for overjet and centerline were = 0 a Expressed in score points unit

Table 3 Inter-rater variability (error of the method and Minimum Standard Deviation (MSD)) (using the second measurements) and reproducibility (ICC) according to time point andmethod Paired sample t-tests comparing both raters

PAR Index Scoring and Timepoint Scoring Method

Rater I Rater II Rater I vs II

Mean (SD) a Mean (SD) a Err Method a MSD a ICC [95 CI] p-Value (Pairedt-Test

T0

Total PAR

Manual 296 (100) 303 (972) 18 103 099 099 100 020

Digital 302 (102) 304 (102) 08 106 100 099 100 055

T1Manual 11 (14) 11 (14) 02 12 099 096 099 100

Digital 11 (12) 10 (10) 04 10 098 095 099 026

T0

Lower anterior

Manual 41 (40) 45 (44) 06 40 099 098 100 006

Digital 44 (36) 45 (37) 07 35 099 096 099 064

T1Manual 03 (06) 03 (04) 00 06 100 - - -

Digital 00 (01) 01 (02) 00 03 100 - - -

T0

Upper anterior

Manual 61 (28) 63 (28) 05 28 099 097 100 033

Digital 64 (28) 64 (27) 06 27 099 098 100 038

T1Manual 01 (03) 01 (02) 03 03 100 049 094 -

Digital 04 (07) 01 (03) 06 03 100 064 096 -

J Clin Med 2021 10 1646 7 of 11

Table 3 Cont

PAR Index Scoring and Timepoint Scoring Method

Rater I Rater II Rater I vs II

Mean (SD) a Mean (SD) a Err Method a MSD a ICC [95 CI] p-Value (Pairedt-Test

T0

Posterior

Manual 13 (14) 14 (15) 04 14 098 093 099 058

Digital 15 (16) 15 (16) 00 16 100 - - -

T1Manual 02 (06) 02 (06) 00 06 099 096 099 033

Digital 03 (06) 02 (06) 02 06 095 084 098 033

T0

Overjet

Manual 12 (75) 12 (75) 00 75 100 - - -

Digital 12 (75) 12 (75) 00 75 100 - - -

T1Manual 00 (00) 00 (00) 00 00 - - - -

Digital 00 (00) 00 (00) 00 00 - - - -

T0

Overbite

Manual 31 (18) 31 (18) 04 18 099 097 100 016

Digital 31 (18) 29 (18) 04 18 097 092 099 067

T1Manual 03 (07) 03 (07) 00 05 100 - - -

Digital 03 (07) 03 (07) 00 05 100 - - -

T0

Centerline

Manual 21 (30) 21 (30) 00 30 100 - - -

Digital 21 (30) 18 (25) 08 26 099 097 100 033

T1Manual 00 (00) 00 (00) 00 00 - - - -

Digital 00 (00) 00 (00) 00 00 - - - -

Error of the method according to Dahlbergrsquos formula using the second measurements ICC could not be calculated as all the scores for overjet and centerline were = 0 a Expressed in score points unit

J Clin Med 2021 10 1646 8 of 11

J Clin Med 2021 10 x FOR PEER REVIEW 7 of 10

(a)

(b)

Figure 2 (a) Scatter plot of the total weighted PAR scores measured by both digital and manual methods and both raters

with a line of unity (b) Bland-Altman plots inter-rater agreement for the total PAR scores measured by the digital and

manual PAR scoring methods at T0

32 Reproducibility

ICC for the total PAR scores and the PAR components at both time points and for

both methods fell in the 095ndash100 range for intra- and inter-rater reproducibility (Tables 2

Figure 2 (a) Scatter plot of the total weighted PAR scores measured by both digital and manual methods and both raterswith a line of unity (b) Bland-Altman plots inter-rater agreement for the total PAR scores measured by the digital andmanual PAR scoring methods at T0

J Clin Med 2021 10 1646 9 of 11

32 Reproducibility

ICC for the total PAR scores and the PAR components at both time points and for bothmethods fell in the 095ndash100 range for intra- and inter-rater reproducibility (Tables 2 and 3)All error-of-the-method values for the total PAR score and its components were smallerthan the associated minimum standard deviation

4 Discussion

Orthodontic model analysis is a prerequisite for diagnosis evaluation of treatmentneed treatment planning and analysis of treatment outcome The present study assessedthe validity and reproducibility of PAR index scoring for digital models and their printedmodel equivalents

Digitization of plaster models (scanned-in) was introduced in the 1990s [33] Theadvantages of digital models include the absence of physical storage requirements instantaccessibility and no risk of breakage or wear [2234] The analysis of scanned-in modelsis as valid as that for plaster models [24] However over the last ten years technologyhas evolved drastically offering high image resolution of digital models and upgradedplatforms needed for their analyses Hence intraoral scanning has gained popularityworldwide Several studies have confirmed the accuracy of direct digital models to beas accurate as that of plaster models Consequently direct digital models are used as analternative to conventional impression techniques and materials [3536] In the presentstudy we used the direct digital model technique

Numerous intraoral scanners and software were developed over the last decade withvarious diagnostic tools In the present study Ortho Analyzer software was used forscoring the digital models and a digital caliper was used for scoring the printed modelsAnalyzing digital models can be associated with some concerns The main concern withthe use of digital software is in fact adjusting the visualization of a 3-D object on a two-dimensional screen An appropriate evaluation requires a correct model orientation Forinstance in this study cross bites were difficult to visualize and rotation of the modelwas required to fully comprehend the magnitude of the cross bite This problem was alsoreported by Stevens et al [12] In addition segmentation of the dental crowns which is aninevitable step to create a virtual setup before carrying out the scoring is time-consumingTo ensure accurate tooth displacement measures one should pay attention when placingthe points parallel to the occlusal plane rotating the model adequately to facilitate goodvisualization of the contact points

Another concern when dental measurements are performed on printed models is theconsideration of the printing technique and the model base design Two studies [2627]evaluated the accuracy of printed models acquired from intraoral scans Brown et al [26]compared plaster models with printed models using two types of 3-D printing techniquesand concluded that both digital light processing (DLP) and polyjet printers producedclinically acceptable models Camardella et al [27] compared 3-D printed models withdifferent base designs using two types of printing techniques and concluded that 3-Dprinted models from intraoral scans created with the polyjet printing technique wereaccurate regardless of the model base design By contrast 3-D printed models with ahorseshoe-shaped base design printed with a stereolithography printer showed a significanttransversal (intra-arch distances) contraction and a horseshoe-shaped base with a posteriorconnection bar was accurate compared with printed models with a regular base Thereforein the present study a polyjet printing technique and regular model base designs wereused to ensure accuracy

Luqmani et al [28] compared automated PAR scoring of direct and indirect digitalmodels (CS 3600 software Carestream Dental Stuttgart Germany) with manual scoringof plaster models using the PAR ruler The authors found that manual PAR scoring wasthe most time-efficient whereas indirect digital model scoring was the least time-efficientThe latter had minor dental cast faults that led to time-consuming software adjustmentsHowever automated scoring was more efficient than the software scoring used by Mayers

J Clin Med 2021 10 1646 10 of 11

et al [23] which required operators to identify each relevant landmark Hence indirectscoring depends on the quality of the dental casts and can be time-consuming dependingon the software used In the present study PAR scoring was not possible as the softwareused (Ortho Analyzer) does not have the feature of automated scoring

In the present study the slightly higher reproducibility of PAR scoring for the digitalmethod was not significant with both methods proving to be highly reproducible Thehigh reproducibility of the manual method coincides with the results of Richmond et al [7]The reproducibility and variability of the direct digital method were similar to the findingsdescribed by Luqmani et al [28] Furthermore the limited variability between both meth-ods demonstrated the high validity of the digital method compared with the conventionalmanual method used as the gold standard

5 Conclusions

PAR scoring on digital models using a software showed excellent reproducibility andpresented good validity compared with manual scoring considered as the gold standard

Author Contributions Conceptualization AG and SG Data curation AG Formal analysis AGFunding acquisition MAC Supervision PMC and MAC Validation AG and SG Writingmdashoriginal draft AG Writingmdashreview amp editing AG SG MD PMC and MAC All authors haveread and agreed to the published version of the manuscript

Funding This research was funded by Aarhus University Forskingsfond (AUFF) grant number 2432

Data Availability Statement The data presented in this study are available on request from thecorresponding author

Acknowledgments The authors express their gratitude to Dirk Leonhardt for his assistance inprinting the models

Conflicts of Interest The authors declare no conflict of interest The funders had no role in the designof the study in the collection analyses or interpretation of data in the writing of the manuscript orin the decision to publish the results

References1 Gottlieb EL Grading your orthodontic treatment results J Clin Orthod 1975 9 155ndash1612 Pangrazio-Kulbersh V Kaczynski R Shunock M Early treatment outcome assessed by the Peer Assessment Rating index Am

J Orthod Dentofac Orthop 1999 115 544ndash550 [CrossRef]3 Summers CJ The occlusal index A system for identifying and scoring occlusal disorders Am J Orthod 1971 59 552ndash567

[CrossRef]4 Brook PH Shaw WC The development of an index of orthodontic treatment priority Eur J Orthod 1989 11 309ndash320

[CrossRef]5 Daniels C Richmond S The development of the index of complexity outcome and need (ICON) J Orthod 2000 27 149ndash162

[CrossRef] [PubMed]6 Casko JS Vaden JL Kokich VG Damone J James RD Cangialosi TJ Riolo ML Owens SE Jr Bills ED Objective

grading system for dental casts and panoramic radiographs American Board of Orthodontics Am J Orthod Dentofacial Orthop1998 114 589ndash599 [CrossRef]

7 Richmond S Shaw WC OrsquoBrien KD Buchanan IB Jones R Stephens CD Roberts CT Andrews M The developmentof the PAR Index (Peer Assessment Rating) Reliability and validity Eur J Orthod 1992 14 125ndash139 [CrossRef] [PubMed]

8 Cons NC Jenny J Kohout FJ Songpaisan Y Jotikastira D Utility of the dental aesthetic index in industrialized anddeveloping countries J Public Health Dent 1989 49 163ndash166 [CrossRef] [PubMed]

9 Richmond S Shaw WC Roberts CT Andrews M The PAR Index (Peer Assessment Rating) Methods to determine outcomeof orthodontic treatment in terms of improvement and standards Eur J Orthod 1992 14 180ndash187 [CrossRef]

10 DeGuzman L Bahiraei D Vig KW Vig PS Weyant RJ OrsquoBrien K The validation of the Peer Assessment Rating index formalocclusion severity and treatment difficulty Am J Orthod Dentofac Orthop 1995 107 172ndash176 [CrossRef]

11 Firestone AR Beck FM Beglin FM Vig KW Evaluation of the peer assessment rating (PAR) index as an index of orthodontictreatment need Am J Orthod Dentofac Orthop 2002 122 463ndash469 [CrossRef]

12 Stevens DR Flores-Mir C Nebbe B Raboud DW Heo G Major PW Validity reliability and reproducibility of plaster vsdigital study models Comparison of peer assessment rating and Bolton analysis and their constituent measurements Am JOrthod Dentofac Orthop 2006 129 794ndash803 [CrossRef] [PubMed]

J Clin Med 2021 10 1646 3 of 11

The digital models for both treatment stages pre-treatment (T0) and post-treatment(T1) had been directly generated by a TRIOS intraoral scanner (3Shape CopenhagenDenmark) as stereolithographic (STL) files imported and analyzed through Ortho Analyzersoftware (3Shape Copenhagen Denmark) Subsequently 30 digital models were printedto generate 15 model equivalents for each stage by means of model design software (ObjetStudio Stratasys Eden Prairie MN USA) and a 3-D printing machine (Polyjet prototypingtechnique Objet30 Dental prime Stratasys Eden Prairie MN USA) in the same laboratoryand with the same technique

24 Measurements

The PAR scoring was performed at T0 and T1 by two methods (1) digitally on thedirect digital models using a built-in feature of the Ortho Analyzer software (Figure 1)and (2) manually on the printed model equivalents using a digital caliper (OrthopliPhiladelphia PA USA) measured to the nearest 001 mm with an orthodontic tip accuracyof 0001 except for overjet and overbite which were measured with a conventional rulerTwo operators (AG and SG) previously trained and calibrated in the use of both techniquesperformed all the measurements independently Reproducibility was determined byrepeated measurements on all models by both methods and by both raters at a one-weekinterval and under identical circumstances

J Clin Med 2021 10 x FOR PEER REVIEW 3 of 10

(1) patients had undergone orthodontic treatment with full fixed appliances at the post-

graduate orthodontic clinic between 2016 and 2018 and (2) digital models before and after

treatment were available No restrictions were applied with regards to age initial maloc-

clusion and end-of-treatment results

The digital models for both treatment stages pre-treatment (T0) and post-treatment

(T1) had been directly generated by a TRIOS intraoral scanner (3Shape Copenhagen

Denmark) as stereolithographic (STL) files imported and analyzed through Ortho Ana-

lyzer software (3Shape Copenhagen Denmark) Subsequently 30 digital models were

printed to generate 15 model equivalents for each stage by means of model design soft-

ware (Objet Studio Stratasys Eden Prairie MN USA) and a 3-D printing machine (Polyjet

prototyping technique Objet30 Dental prime Stratasys Eden Prairie MN USA) in the

same laboratory and with the same technique

24 Measurements

The PAR scoring was performed at T0 and T1 by two methods (1) digitally on the

direct digital models using a built-in feature of the Ortho Analyzer software (Figure 1)

and (2) manually on the printed model equivalents using a digital caliper (Orthopli Phil-

adelphia PA USA) measured to the nearest 001 mm with an orthodontic tip accuracy of

0001 except for overjet and overbite which were measured with a conventional ruler

Two operators (AG and SG) previously trained and calibrated in the use of both tech-

niques performed all the measurements independently Reproducibility was determined

by repeated measurements on all models by both methods and by both raters at a one-

week interval and under identical circumstances

Figure 1 Peer Assessment Rating (PAR) index scoring using Ortho Analyzer software Anterior component scoring con-

tact point displacement between teeth 21 and 22

The PAR scoring was performed using the UK weighting system according to Rich-

mond et al [7] and included five components scoring various occlusal traits which con-

stitute malocclusion anterior segment posterior segment overjet overbite and centerline

(Table 1) The scores of the traits were summed and multiplied by their weight The com-

ponent-weighted PAR scores were summed to constitute the total weighted PAR score

Figure 1 Peer Assessment Rating (PAR) index scoring using Ortho Analyzer software Anterior component scoring contactpoint displacement between teeth 21 and 22

The PAR scoring was performed using the UK weighting system according to Rich-mond et al [7] and included five components scoring various occlusal traits which consti-tute malocclusion anterior segment posterior segment overjet overbite and centerline(Table 1) The scores of the traits were summed and multiplied by their weight Thecomponent-weighted PAR scores were summed to constitute the total weighted PAR scoreEssential information about each case was considered such as impacted teeth missing orextracted teeth plans for any prosthetic replacements and restorative work previouslycarried out that affected the malocclusion

J Clin Med 2021 10 1646 4 of 11

Table 1 The PAR index components and scoring

PAR Component Assessment Scoring Weighting

1 Anterior dagger Contact point displacement 0ndash4 times 1Impacted incisorscanines 52 Posterior Dagger Sagittal occlusion 0ndash2

times 1Vertical occlusion 0ndash1Transverse occlusion 0ndash4

3 Overjet sect Overjet 0ndash4 times 6Anterior crossbite 0ndash44 Overbite para Overbite 0ndash3 times 2Open bite 0ndash45 Centerline Deviation from dental midline 0ndash2 times 4

Total Unweighted PAR score Weighted PAR scoredagger Measured from the mesial contact point of canine on one side to the mesial contact point of canine on the opposite side (upper andlower arches) and recorded in millimeters as the shortest distance between contact points of adjacent teeth and parallel to the occlusalplane Dagger Measured from the distal contact point of canine to the mesial contact point of permanent molar or last molar (right and left sides)sect Measured as the largest horizontal distance parallel to the occlusal plane from the labial incisor edge of the most prominent upper incisorto the labial surface of the corresponding lower incisor para Recorded as the largest vertical overlap or open bite between upper incisors andlower incisors

25 Statistical Analyses

Data collection and management were performed by means of the Research ElectronicData Capture (REDCap) tool hosted at Aarhus University [3031] Statistical analyses werecarried out with Stata software (Release 16 StataCorp 2019 College Station TX USA)

Descriptive statistics were used to analyze the total PAR scores at different timepoints between raters and methods used Paired sample t-tests were used to comparePAR scoring between both methods and raters at a significance level of lt005 Bothmethods were assessed by ICC for intra- and inter-rater reproducibility Intra- and inter-rater variability were determined by calculation of the error of the method according toDahlbergrsquos formula [32] The agreement between the digital and manual scoring methodsperformed by the two raters was determined by a scatter plot and Bland-Altman plots

3 Results31 Validity

Paired-sample t-tests showed no significant differences in the mean total PAR scoresand in the PAR components between both methods (Tables 2 and 3) The scatter plot(Figure 2a) and Bland-Altman plots (Figure 2b) illustrate agreement of the measurementsconducted with both methods

J Clin Med 2021 10 1646 5 of 11

Table 2 Intra-rater variability (error of the method and Minimum Standard Deviation (MSD)) and reproducibility (ICC) according to time point and method for both raters Paired samplet-tests comparing both methods

Rater I Rater II

PAR Index Scoring and Timepoint ScoringMethod

Mean(SD) a

Err Methoda MSD a ICC [95 CI]

p-Value(Pairedt-Test)

Mean(SD) a

ErrMethod

aMSD a ICC [95 CI] p-Value (Paired

t-Test)

T0Total PAR

Manual 295(101) 09 101 099 099 100

016

305(107) 14 101 099 098 100

074Digital 302

(105) 09 101 100 099 100 304(104) 06 103 100 099 100

T1Manual 11 (14) 02 14 099 099 100

07212 (13) 04 12 095 084 098

049Digital 12 (13) 02 11 098 094 099 10 (09) 02 10 098 095 099

T0

Lower anterior

Manual 41 (40) 05 40 099 099 100012

44 (42) 06 41 099 097 100060

Digital 44 (36) 02 35 100 099 100 45 (37) 04 37 099 098 100

T1Manual 03 (06) 00 06 100 - -

01603 (06) 00 06 100 - -

016Digital 01 (03) 00 04 100 - - 01 (03) 00 04 100 - -

T0

Upper anterior

Manual 61 (28) 06 26 097 092 099045

63 (28) 07 26 098 093 099050

Digital 63 (28) 07 26 096 089 099 64 (27) 06 26 098 095 099

T1Manual 02 (04) 00 04 100 - -

01902 (04) 00 04 100 - -

019Digital 03 (05) 01 05 100 - - 03 (05) 01 05 100 - -

T0

Posterior

Manual 13 (15) 04 14 097 091 099016

14 (15) 00 15 100 - -033

Digital 15 (16) 00 16 100 - - 15 (16) 00 16 100 - -

T1Manual 02 (06) 02 06 095 084 098

06702 (06) 00 06 100 - -

-Digital 03 (06) 00 15 100 - - 03 (06) 00 06 100 - -

T0

Overjet

Manual 120 (75) 00 75 100 - --

120 (75) 00 75 100 - --

Digital 120 (75) 00 75 100 - - 120 (75) 00 75 100 - -

T1Manual 00 (00) 00 00 - - -

-00 (00) 00 00 - - -

-Digital 00 (00) 00 00 - - - 00 (00) 00 00 - - -

T0

Overbite

Manual 31 (18) 00 18 100 - -033

29 (19) 05 18 096 089 099100

Digital 30 (18) 04 18 098 094 099 29 (18) 00 18 100 - -

T1Manual 03 (07) 05 07 100 - -

-03 (07) 00 07 100 - -

-Digital 03 (07) 05 07 100 - - 03 (07) 04 05 100 - -

J Clin Med 2021 10 1646 6 of 11

Table 2 Cont

Rater I Rater II

PAR Index Scoring and Timepoint ScoringMethod

Mean(SD) a

Err Methoda MSD a ICC [95 CI]

p-Value(Pairedt-Test)

Mean(SD) a

ErrMethod

aMSD a ICC [95 CI] p-Value (Paired

t-Test)

T0

Centerline

Manual 21 (30) 00 30 100 100 100033

21 (30) 00 30 100 100 100033

Digital 20 (27) 08 26 096 089 099 19 (26) 00 26 100 - -

T1Manual 00 (00) 00 00 - -

-00 (00) 00 00 - - -

-Digital 00 (00) 00 00 - - 00 (00) 00 00 - - -

Error of the method according to Dahlbergrsquos formula ICC could not be calculated as all the scores for overjet and centerline were = 0 a Expressed in score points unit

Table 3 Inter-rater variability (error of the method and Minimum Standard Deviation (MSD)) (using the second measurements) and reproducibility (ICC) according to time point andmethod Paired sample t-tests comparing both raters

PAR Index Scoring and Timepoint Scoring Method

Rater I Rater II Rater I vs II

Mean (SD) a Mean (SD) a Err Method a MSD a ICC [95 CI] p-Value (Pairedt-Test

T0

Total PAR

Manual 296 (100) 303 (972) 18 103 099 099 100 020

Digital 302 (102) 304 (102) 08 106 100 099 100 055

T1Manual 11 (14) 11 (14) 02 12 099 096 099 100

Digital 11 (12) 10 (10) 04 10 098 095 099 026

T0

Lower anterior

Manual 41 (40) 45 (44) 06 40 099 098 100 006

Digital 44 (36) 45 (37) 07 35 099 096 099 064

T1Manual 03 (06) 03 (04) 00 06 100 - - -

Digital 00 (01) 01 (02) 00 03 100 - - -

T0

Upper anterior

Manual 61 (28) 63 (28) 05 28 099 097 100 033

Digital 64 (28) 64 (27) 06 27 099 098 100 038

T1Manual 01 (03) 01 (02) 03 03 100 049 094 -

Digital 04 (07) 01 (03) 06 03 100 064 096 -

J Clin Med 2021 10 1646 7 of 11

Table 3 Cont

PAR Index Scoring and Timepoint Scoring Method

Rater I Rater II Rater I vs II

Mean (SD) a Mean (SD) a Err Method a MSD a ICC [95 CI] p-Value (Pairedt-Test

T0

Posterior

Manual 13 (14) 14 (15) 04 14 098 093 099 058

Digital 15 (16) 15 (16) 00 16 100 - - -

T1Manual 02 (06) 02 (06) 00 06 099 096 099 033

Digital 03 (06) 02 (06) 02 06 095 084 098 033

T0

Overjet

Manual 12 (75) 12 (75) 00 75 100 - - -

Digital 12 (75) 12 (75) 00 75 100 - - -

T1Manual 00 (00) 00 (00) 00 00 - - - -

Digital 00 (00) 00 (00) 00 00 - - - -

T0

Overbite

Manual 31 (18) 31 (18) 04 18 099 097 100 016

Digital 31 (18) 29 (18) 04 18 097 092 099 067

T1Manual 03 (07) 03 (07) 00 05 100 - - -

Digital 03 (07) 03 (07) 00 05 100 - - -

T0

Centerline

Manual 21 (30) 21 (30) 00 30 100 - - -

Digital 21 (30) 18 (25) 08 26 099 097 100 033

T1Manual 00 (00) 00 (00) 00 00 - - - -

Digital 00 (00) 00 (00) 00 00 - - - -

Error of the method according to Dahlbergrsquos formula using the second measurements ICC could not be calculated as all the scores for overjet and centerline were = 0 a Expressed in score points unit

J Clin Med 2021 10 1646 8 of 11

J Clin Med 2021 10 x FOR PEER REVIEW 7 of 10

(a)

(b)

Figure 2 (a) Scatter plot of the total weighted PAR scores measured by both digital and manual methods and both raters

with a line of unity (b) Bland-Altman plots inter-rater agreement for the total PAR scores measured by the digital and

manual PAR scoring methods at T0

32 Reproducibility

ICC for the total PAR scores and the PAR components at both time points and for

both methods fell in the 095ndash100 range for intra- and inter-rater reproducibility (Tables 2

Figure 2 (a) Scatter plot of the total weighted PAR scores measured by both digital and manual methods and both raterswith a line of unity (b) Bland-Altman plots inter-rater agreement for the total PAR scores measured by the digital andmanual PAR scoring methods at T0

J Clin Med 2021 10 1646 9 of 11

32 Reproducibility

ICC for the total PAR scores and the PAR components at both time points and for bothmethods fell in the 095ndash100 range for intra- and inter-rater reproducibility (Tables 2 and 3)All error-of-the-method values for the total PAR score and its components were smallerthan the associated minimum standard deviation

4 Discussion

Orthodontic model analysis is a prerequisite for diagnosis evaluation of treatmentneed treatment planning and analysis of treatment outcome The present study assessedthe validity and reproducibility of PAR index scoring for digital models and their printedmodel equivalents

Digitization of plaster models (scanned-in) was introduced in the 1990s [33] Theadvantages of digital models include the absence of physical storage requirements instantaccessibility and no risk of breakage or wear [2234] The analysis of scanned-in modelsis as valid as that for plaster models [24] However over the last ten years technologyhas evolved drastically offering high image resolution of digital models and upgradedplatforms needed for their analyses Hence intraoral scanning has gained popularityworldwide Several studies have confirmed the accuracy of direct digital models to beas accurate as that of plaster models Consequently direct digital models are used as analternative to conventional impression techniques and materials [3536] In the presentstudy we used the direct digital model technique

Numerous intraoral scanners and software were developed over the last decade withvarious diagnostic tools In the present study Ortho Analyzer software was used forscoring the digital models and a digital caliper was used for scoring the printed modelsAnalyzing digital models can be associated with some concerns The main concern withthe use of digital software is in fact adjusting the visualization of a 3-D object on a two-dimensional screen An appropriate evaluation requires a correct model orientation Forinstance in this study cross bites were difficult to visualize and rotation of the modelwas required to fully comprehend the magnitude of the cross bite This problem was alsoreported by Stevens et al [12] In addition segmentation of the dental crowns which is aninevitable step to create a virtual setup before carrying out the scoring is time-consumingTo ensure accurate tooth displacement measures one should pay attention when placingthe points parallel to the occlusal plane rotating the model adequately to facilitate goodvisualization of the contact points

Another concern when dental measurements are performed on printed models is theconsideration of the printing technique and the model base design Two studies [2627]evaluated the accuracy of printed models acquired from intraoral scans Brown et al [26]compared plaster models with printed models using two types of 3-D printing techniquesand concluded that both digital light processing (DLP) and polyjet printers producedclinically acceptable models Camardella et al [27] compared 3-D printed models withdifferent base designs using two types of printing techniques and concluded that 3-Dprinted models from intraoral scans created with the polyjet printing technique wereaccurate regardless of the model base design By contrast 3-D printed models with ahorseshoe-shaped base design printed with a stereolithography printer showed a significanttransversal (intra-arch distances) contraction and a horseshoe-shaped base with a posteriorconnection bar was accurate compared with printed models with a regular base Thereforein the present study a polyjet printing technique and regular model base designs wereused to ensure accuracy

Luqmani et al [28] compared automated PAR scoring of direct and indirect digitalmodels (CS 3600 software Carestream Dental Stuttgart Germany) with manual scoringof plaster models using the PAR ruler The authors found that manual PAR scoring wasthe most time-efficient whereas indirect digital model scoring was the least time-efficientThe latter had minor dental cast faults that led to time-consuming software adjustmentsHowever automated scoring was more efficient than the software scoring used by Mayers

J Clin Med 2021 10 1646 10 of 11

et al [23] which required operators to identify each relevant landmark Hence indirectscoring depends on the quality of the dental casts and can be time-consuming dependingon the software used In the present study PAR scoring was not possible as the softwareused (Ortho Analyzer) does not have the feature of automated scoring

In the present study the slightly higher reproducibility of PAR scoring for the digitalmethod was not significant with both methods proving to be highly reproducible Thehigh reproducibility of the manual method coincides with the results of Richmond et al [7]The reproducibility and variability of the direct digital method were similar to the findingsdescribed by Luqmani et al [28] Furthermore the limited variability between both meth-ods demonstrated the high validity of the digital method compared with the conventionalmanual method used as the gold standard

5 Conclusions

PAR scoring on digital models using a software showed excellent reproducibility andpresented good validity compared with manual scoring considered as the gold standard

Author Contributions Conceptualization AG and SG Data curation AG Formal analysis AGFunding acquisition MAC Supervision PMC and MAC Validation AG and SG Writingmdashoriginal draft AG Writingmdashreview amp editing AG SG MD PMC and MAC All authors haveread and agreed to the published version of the manuscript

Funding This research was funded by Aarhus University Forskingsfond (AUFF) grant number 2432

Data Availability Statement The data presented in this study are available on request from thecorresponding author

Acknowledgments The authors express their gratitude to Dirk Leonhardt for his assistance inprinting the models

Conflicts of Interest The authors declare no conflict of interest The funders had no role in the designof the study in the collection analyses or interpretation of data in the writing of the manuscript orin the decision to publish the results

References1 Gottlieb EL Grading your orthodontic treatment results J Clin Orthod 1975 9 155ndash1612 Pangrazio-Kulbersh V Kaczynski R Shunock M Early treatment outcome assessed by the Peer Assessment Rating index Am

J Orthod Dentofac Orthop 1999 115 544ndash550 [CrossRef]3 Summers CJ The occlusal index A system for identifying and scoring occlusal disorders Am J Orthod 1971 59 552ndash567

[CrossRef]4 Brook PH Shaw WC The development of an index of orthodontic treatment priority Eur J Orthod 1989 11 309ndash320

[CrossRef]5 Daniels C Richmond S The development of the index of complexity outcome and need (ICON) J Orthod 2000 27 149ndash162

[CrossRef] [PubMed]6 Casko JS Vaden JL Kokich VG Damone J James RD Cangialosi TJ Riolo ML Owens SE Jr Bills ED Objective

grading system for dental casts and panoramic radiographs American Board of Orthodontics Am J Orthod Dentofacial Orthop1998 114 589ndash599 [CrossRef]

7 Richmond S Shaw WC OrsquoBrien KD Buchanan IB Jones R Stephens CD Roberts CT Andrews M The developmentof the PAR Index (Peer Assessment Rating) Reliability and validity Eur J Orthod 1992 14 125ndash139 [CrossRef] [PubMed]

8 Cons NC Jenny J Kohout FJ Songpaisan Y Jotikastira D Utility of the dental aesthetic index in industrialized anddeveloping countries J Public Health Dent 1989 49 163ndash166 [CrossRef] [PubMed]

9 Richmond S Shaw WC Roberts CT Andrews M The PAR Index (Peer Assessment Rating) Methods to determine outcomeof orthodontic treatment in terms of improvement and standards Eur J Orthod 1992 14 180ndash187 [CrossRef]

10 DeGuzman L Bahiraei D Vig KW Vig PS Weyant RJ OrsquoBrien K The validation of the Peer Assessment Rating index formalocclusion severity and treatment difficulty Am J Orthod Dentofac Orthop 1995 107 172ndash176 [CrossRef]

11 Firestone AR Beck FM Beglin FM Vig KW Evaluation of the peer assessment rating (PAR) index as an index of orthodontictreatment need Am J Orthod Dentofac Orthop 2002 122 463ndash469 [CrossRef]

12 Stevens DR Flores-Mir C Nebbe B Raboud DW Heo G Major PW Validity reliability and reproducibility of plaster vsdigital study models Comparison of peer assessment rating and Bolton analysis and their constituent measurements Am JOrthod Dentofac Orthop 2006 129 794ndash803 [CrossRef] [PubMed]

J Clin Med 2021 10 1646 4 of 11

Table 1 The PAR index components and scoring

PAR Component Assessment Scoring Weighting

1 Anterior dagger Contact point displacement 0ndash4 times 1Impacted incisorscanines 52 Posterior Dagger Sagittal occlusion 0ndash2

times 1Vertical occlusion 0ndash1Transverse occlusion 0ndash4

3 Overjet sect Overjet 0ndash4 times 6Anterior crossbite 0ndash44 Overbite para Overbite 0ndash3 times 2Open bite 0ndash45 Centerline Deviation from dental midline 0ndash2 times 4

Total Unweighted PAR score Weighted PAR scoredagger Measured from the mesial contact point of canine on one side to the mesial contact point of canine on the opposite side (upper andlower arches) and recorded in millimeters as the shortest distance between contact points of adjacent teeth and parallel to the occlusalplane Dagger Measured from the distal contact point of canine to the mesial contact point of permanent molar or last molar (right and left sides)sect Measured as the largest horizontal distance parallel to the occlusal plane from the labial incisor edge of the most prominent upper incisorto the labial surface of the corresponding lower incisor para Recorded as the largest vertical overlap or open bite between upper incisors andlower incisors

25 Statistical Analyses

Data collection and management were performed by means of the Research ElectronicData Capture (REDCap) tool hosted at Aarhus University [3031] Statistical analyses werecarried out with Stata software (Release 16 StataCorp 2019 College Station TX USA)

Descriptive statistics were used to analyze the total PAR scores at different timepoints between raters and methods used Paired sample t-tests were used to comparePAR scoring between both methods and raters at a significance level of lt005 Bothmethods were assessed by ICC for intra- and inter-rater reproducibility Intra- and inter-rater variability were determined by calculation of the error of the method according toDahlbergrsquos formula [32] The agreement between the digital and manual scoring methodsperformed by the two raters was determined by a scatter plot and Bland-Altman plots

3 Results31 Validity

Paired-sample t-tests showed no significant differences in the mean total PAR scoresand in the PAR components between both methods (Tables 2 and 3) The scatter plot(Figure 2a) and Bland-Altman plots (Figure 2b) illustrate agreement of the measurementsconducted with both methods

J Clin Med 2021 10 1646 5 of 11

Table 2 Intra-rater variability (error of the method and Minimum Standard Deviation (MSD)) and reproducibility (ICC) according to time point and method for both raters Paired samplet-tests comparing both methods

Rater I Rater II

PAR Index Scoring and Timepoint ScoringMethod

Mean(SD) a

Err Methoda MSD a ICC [95 CI]

p-Value(Pairedt-Test)

Mean(SD) a

ErrMethod

aMSD a ICC [95 CI] p-Value (Paired

t-Test)

T0Total PAR

Manual 295(101) 09 101 099 099 100

016

305(107) 14 101 099 098 100

074Digital 302

(105) 09 101 100 099 100 304(104) 06 103 100 099 100

T1Manual 11 (14) 02 14 099 099 100

07212 (13) 04 12 095 084 098

049Digital 12 (13) 02 11 098 094 099 10 (09) 02 10 098 095 099

T0

Lower anterior

Manual 41 (40) 05 40 099 099 100012

44 (42) 06 41 099 097 100060

Digital 44 (36) 02 35 100 099 100 45 (37) 04 37 099 098 100

T1Manual 03 (06) 00 06 100 - -

01603 (06) 00 06 100 - -

016Digital 01 (03) 00 04 100 - - 01 (03) 00 04 100 - -

T0

Upper anterior

Manual 61 (28) 06 26 097 092 099045

63 (28) 07 26 098 093 099050

Digital 63 (28) 07 26 096 089 099 64 (27) 06 26 098 095 099

T1Manual 02 (04) 00 04 100 - -

01902 (04) 00 04 100 - -

019Digital 03 (05) 01 05 100 - - 03 (05) 01 05 100 - -

T0

Posterior

Manual 13 (15) 04 14 097 091 099016

14 (15) 00 15 100 - -033

Digital 15 (16) 00 16 100 - - 15 (16) 00 16 100 - -

T1Manual 02 (06) 02 06 095 084 098

06702 (06) 00 06 100 - -

-Digital 03 (06) 00 15 100 - - 03 (06) 00 06 100 - -

T0

Overjet

Manual 120 (75) 00 75 100 - --

120 (75) 00 75 100 - --

Digital 120 (75) 00 75 100 - - 120 (75) 00 75 100 - -

T1Manual 00 (00) 00 00 - - -

-00 (00) 00 00 - - -

-Digital 00 (00) 00 00 - - - 00 (00) 00 00 - - -

T0

Overbite

Manual 31 (18) 00 18 100 - -033

29 (19) 05 18 096 089 099100

Digital 30 (18) 04 18 098 094 099 29 (18) 00 18 100 - -

T1Manual 03 (07) 05 07 100 - -

-03 (07) 00 07 100 - -

-Digital 03 (07) 05 07 100 - - 03 (07) 04 05 100 - -

J Clin Med 2021 10 1646 6 of 11

Table 2 Cont

Rater I Rater II

PAR Index Scoring and Timepoint ScoringMethod

Mean(SD) a

Err Methoda MSD a ICC [95 CI]

p-Value(Pairedt-Test)

Mean(SD) a

ErrMethod

aMSD a ICC [95 CI] p-Value (Paired

t-Test)

T0

Centerline

Manual 21 (30) 00 30 100 100 100033

21 (30) 00 30 100 100 100033

Digital 20 (27) 08 26 096 089 099 19 (26) 00 26 100 - -

T1Manual 00 (00) 00 00 - -

-00 (00) 00 00 - - -

-Digital 00 (00) 00 00 - - 00 (00) 00 00 - - -

Error of the method according to Dahlbergrsquos formula ICC could not be calculated as all the scores for overjet and centerline were = 0 a Expressed in score points unit

Table 3 Inter-rater variability (error of the method and Minimum Standard Deviation (MSD)) (using the second measurements) and reproducibility (ICC) according to time point andmethod Paired sample t-tests comparing both raters

PAR Index Scoring and Timepoint Scoring Method

Rater I Rater II Rater I vs II

Mean (SD) a Mean (SD) a Err Method a MSD a ICC [95 CI] p-Value (Pairedt-Test

T0

Total PAR

Manual 296 (100) 303 (972) 18 103 099 099 100 020

Digital 302 (102) 304 (102) 08 106 100 099 100 055

T1Manual 11 (14) 11 (14) 02 12 099 096 099 100

Digital 11 (12) 10 (10) 04 10 098 095 099 026

T0

Lower anterior

Manual 41 (40) 45 (44) 06 40 099 098 100 006

Digital 44 (36) 45 (37) 07 35 099 096 099 064

T1Manual 03 (06) 03 (04) 00 06 100 - - -

Digital 00 (01) 01 (02) 00 03 100 - - -

T0

Upper anterior

Manual 61 (28) 63 (28) 05 28 099 097 100 033

Digital 64 (28) 64 (27) 06 27 099 098 100 038

T1Manual 01 (03) 01 (02) 03 03 100 049 094 -

Digital 04 (07) 01 (03) 06 03 100 064 096 -

J Clin Med 2021 10 1646 7 of 11

Table 3 Cont

PAR Index Scoring and Timepoint Scoring Method

Rater I Rater II Rater I vs II

Mean (SD) a Mean (SD) a Err Method a MSD a ICC [95 CI] p-Value (Pairedt-Test

T0

Posterior

Manual 13 (14) 14 (15) 04 14 098 093 099 058

Digital 15 (16) 15 (16) 00 16 100 - - -

T1Manual 02 (06) 02 (06) 00 06 099 096 099 033

Digital 03 (06) 02 (06) 02 06 095 084 098 033

T0

Overjet

Manual 12 (75) 12 (75) 00 75 100 - - -

Digital 12 (75) 12 (75) 00 75 100 - - -

T1Manual 00 (00) 00 (00) 00 00 - - - -

Digital 00 (00) 00 (00) 00 00 - - - -

T0

Overbite

Manual 31 (18) 31 (18) 04 18 099 097 100 016

Digital 31 (18) 29 (18) 04 18 097 092 099 067

T1Manual 03 (07) 03 (07) 00 05 100 - - -

Digital 03 (07) 03 (07) 00 05 100 - - -

T0

Centerline

Manual 21 (30) 21 (30) 00 30 100 - - -

Digital 21 (30) 18 (25) 08 26 099 097 100 033

T1Manual 00 (00) 00 (00) 00 00 - - - -

Digital 00 (00) 00 (00) 00 00 - - - -

Error of the method according to Dahlbergrsquos formula using the second measurements ICC could not be calculated as all the scores for overjet and centerline were = 0 a Expressed in score points unit

J Clin Med 2021 10 1646 8 of 11

J Clin Med 2021 10 x FOR PEER REVIEW 7 of 10

(a)

(b)

Figure 2 (a) Scatter plot of the total weighted PAR scores measured by both digital and manual methods and both raters

with a line of unity (b) Bland-Altman plots inter-rater agreement for the total PAR scores measured by the digital and

manual PAR scoring methods at T0

32 Reproducibility

ICC for the total PAR scores and the PAR components at both time points and for

both methods fell in the 095ndash100 range for intra- and inter-rater reproducibility (Tables 2

Figure 2 (a) Scatter plot of the total weighted PAR scores measured by both digital and manual methods and both raterswith a line of unity (b) Bland-Altman plots inter-rater agreement for the total PAR scores measured by the digital andmanual PAR scoring methods at T0

J Clin Med 2021 10 1646 9 of 11

32 Reproducibility

ICC for the total PAR scores and the PAR components at both time points and for bothmethods fell in the 095ndash100 range for intra- and inter-rater reproducibility (Tables 2 and 3)All error-of-the-method values for the total PAR score and its components were smallerthan the associated minimum standard deviation

4 Discussion

Orthodontic model analysis is a prerequisite for diagnosis evaluation of treatmentneed treatment planning and analysis of treatment outcome The present study assessedthe validity and reproducibility of PAR index scoring for digital models and their printedmodel equivalents

Digitization of plaster models (scanned-in) was introduced in the 1990s [33] Theadvantages of digital models include the absence of physical storage requirements instantaccessibility and no risk of breakage or wear [2234] The analysis of scanned-in modelsis as valid as that for plaster models [24] However over the last ten years technologyhas evolved drastically offering high image resolution of digital models and upgradedplatforms needed for their analyses Hence intraoral scanning has gained popularityworldwide Several studies have confirmed the accuracy of direct digital models to beas accurate as that of plaster models Consequently direct digital models are used as analternative to conventional impression techniques and materials [3536] In the presentstudy we used the direct digital model technique

Numerous intraoral scanners and software were developed over the last decade withvarious diagnostic tools In the present study Ortho Analyzer software was used forscoring the digital models and a digital caliper was used for scoring the printed modelsAnalyzing digital models can be associated with some concerns The main concern withthe use of digital software is in fact adjusting the visualization of a 3-D object on a two-dimensional screen An appropriate evaluation requires a correct model orientation Forinstance in this study cross bites were difficult to visualize and rotation of the modelwas required to fully comprehend the magnitude of the cross bite This problem was alsoreported by Stevens et al [12] In addition segmentation of the dental crowns which is aninevitable step to create a virtual setup before carrying out the scoring is time-consumingTo ensure accurate tooth displacement measures one should pay attention when placingthe points parallel to the occlusal plane rotating the model adequately to facilitate goodvisualization of the contact points

Another concern when dental measurements are performed on printed models is theconsideration of the printing technique and the model base design Two studies [2627]evaluated the accuracy of printed models acquired from intraoral scans Brown et al [26]compared plaster models with printed models using two types of 3-D printing techniquesand concluded that both digital light processing (DLP) and polyjet printers producedclinically acceptable models Camardella et al [27] compared 3-D printed models withdifferent base designs using two types of printing techniques and concluded that 3-Dprinted models from intraoral scans created with the polyjet printing technique wereaccurate regardless of the model base design By contrast 3-D printed models with ahorseshoe-shaped base design printed with a stereolithography printer showed a significanttransversal (intra-arch distances) contraction and a horseshoe-shaped base with a posteriorconnection bar was accurate compared with printed models with a regular base Thereforein the present study a polyjet printing technique and regular model base designs wereused to ensure accuracy

Luqmani et al [28] compared automated PAR scoring of direct and indirect digitalmodels (CS 3600 software Carestream Dental Stuttgart Germany) with manual scoringof plaster models using the PAR ruler The authors found that manual PAR scoring wasthe most time-efficient whereas indirect digital model scoring was the least time-efficientThe latter had minor dental cast faults that led to time-consuming software adjustmentsHowever automated scoring was more efficient than the software scoring used by Mayers

J Clin Med 2021 10 1646 10 of 11

et al [23] which required operators to identify each relevant landmark Hence indirectscoring depends on the quality of the dental casts and can be time-consuming dependingon the software used In the present study PAR scoring was not possible as the softwareused (Ortho Analyzer) does not have the feature of automated scoring

In the present study the slightly higher reproducibility of PAR scoring for the digitalmethod was not significant with both methods proving to be highly reproducible Thehigh reproducibility of the manual method coincides with the results of Richmond et al [7]The reproducibility and variability of the direct digital method were similar to the findingsdescribed by Luqmani et al [28] Furthermore the limited variability between both meth-ods demonstrated the high validity of the digital method compared with the conventionalmanual method used as the gold standard

5 Conclusions

PAR scoring on digital models using a software showed excellent reproducibility andpresented good validity compared with manual scoring considered as the gold standard

Author Contributions Conceptualization AG and SG Data curation AG Formal analysis AGFunding acquisition MAC Supervision PMC and MAC Validation AG and SG Writingmdashoriginal draft AG Writingmdashreview amp editing AG SG MD PMC and MAC All authors haveread and agreed to the published version of the manuscript

Funding This research was funded by Aarhus University Forskingsfond (AUFF) grant number 2432

Data Availability Statement The data presented in this study are available on request from thecorresponding author

Acknowledgments The authors express their gratitude to Dirk Leonhardt for his assistance inprinting the models

Conflicts of Interest The authors declare no conflict of interest The funders had no role in the designof the study in the collection analyses or interpretation of data in the writing of the manuscript orin the decision to publish the results

References1 Gottlieb EL Grading your orthodontic treatment results J Clin Orthod 1975 9 155ndash1612 Pangrazio-Kulbersh V Kaczynski R Shunock M Early treatment outcome assessed by the Peer Assessment Rating index Am

J Orthod Dentofac Orthop 1999 115 544ndash550 [CrossRef]3 Summers CJ The occlusal index A system for identifying and scoring occlusal disorders Am J Orthod 1971 59 552ndash567

[CrossRef]4 Brook PH Shaw WC The development of an index of orthodontic treatment priority Eur J Orthod 1989 11 309ndash320

[CrossRef]5 Daniels C Richmond S The development of the index of complexity outcome and need (ICON) J Orthod 2000 27 149ndash162

[CrossRef] [PubMed]6 Casko JS Vaden JL Kokich VG Damone J James RD Cangialosi TJ Riolo ML Owens SE Jr Bills ED Objective

grading system for dental casts and panoramic radiographs American Board of Orthodontics Am J Orthod Dentofacial Orthop1998 114 589ndash599 [CrossRef]

7 Richmond S Shaw WC OrsquoBrien KD Buchanan IB Jones R Stephens CD Roberts CT Andrews M The developmentof the PAR Index (Peer Assessment Rating) Reliability and validity Eur J Orthod 1992 14 125ndash139 [CrossRef] [PubMed]

8 Cons NC Jenny J Kohout FJ Songpaisan Y Jotikastira D Utility of the dental aesthetic index in industrialized anddeveloping countries J Public Health Dent 1989 49 163ndash166 [CrossRef] [PubMed]

9 Richmond S Shaw WC Roberts CT Andrews M The PAR Index (Peer Assessment Rating) Methods to determine outcomeof orthodontic treatment in terms of improvement and standards Eur J Orthod 1992 14 180ndash187 [CrossRef]

10 DeGuzman L Bahiraei D Vig KW Vig PS Weyant RJ OrsquoBrien K The validation of the Peer Assessment Rating index formalocclusion severity and treatment difficulty Am J Orthod Dentofac Orthop 1995 107 172ndash176 [CrossRef]

11 Firestone AR Beck FM Beglin FM Vig KW Evaluation of the peer assessment rating (PAR) index as an index of orthodontictreatment need Am J Orthod Dentofac Orthop 2002 122 463ndash469 [CrossRef]

12 Stevens DR Flores-Mir C Nebbe B Raboud DW Heo G Major PW Validity reliability and reproducibility of plaster vsdigital study models Comparison of peer assessment rating and Bolton analysis and their constituent measurements Am JOrthod Dentofac Orthop 2006 129 794ndash803 [CrossRef] [PubMed]

J Clin Med 2021 10 1646 5 of 11

Table 2 Intra-rater variability (error of the method and Minimum Standard Deviation (MSD)) and reproducibility (ICC) according to time point and method for both raters Paired samplet-tests comparing both methods

Rater I Rater II

PAR Index Scoring and Timepoint ScoringMethod

Mean(SD) a

Err Methoda MSD a ICC [95 CI]

p-Value(Pairedt-Test)

Mean(SD) a

ErrMethod

aMSD a ICC [95 CI] p-Value (Paired

t-Test)

T0Total PAR

Manual 295(101) 09 101 099 099 100

016

305(107) 14 101 099 098 100

074Digital 302

(105) 09 101 100 099 100 304(104) 06 103 100 099 100

T1Manual 11 (14) 02 14 099 099 100

07212 (13) 04 12 095 084 098

049Digital 12 (13) 02 11 098 094 099 10 (09) 02 10 098 095 099

T0

Lower anterior

Manual 41 (40) 05 40 099 099 100012

44 (42) 06 41 099 097 100060

Digital 44 (36) 02 35 100 099 100 45 (37) 04 37 099 098 100

T1Manual 03 (06) 00 06 100 - -

01603 (06) 00 06 100 - -

016Digital 01 (03) 00 04 100 - - 01 (03) 00 04 100 - -

T0

Upper anterior

Manual 61 (28) 06 26 097 092 099045

63 (28) 07 26 098 093 099050

Digital 63 (28) 07 26 096 089 099 64 (27) 06 26 098 095 099

T1Manual 02 (04) 00 04 100 - -

01902 (04) 00 04 100 - -

019Digital 03 (05) 01 05 100 - - 03 (05) 01 05 100 - -

T0

Posterior

Manual 13 (15) 04 14 097 091 099016

14 (15) 00 15 100 - -033

Digital 15 (16) 00 16 100 - - 15 (16) 00 16 100 - -

T1Manual 02 (06) 02 06 095 084 098

06702 (06) 00 06 100 - -

-Digital 03 (06) 00 15 100 - - 03 (06) 00 06 100 - -

T0

Overjet

Manual 120 (75) 00 75 100 - --

120 (75) 00 75 100 - --

Digital 120 (75) 00 75 100 - - 120 (75) 00 75 100 - -

T1Manual 00 (00) 00 00 - - -

-00 (00) 00 00 - - -

-Digital 00 (00) 00 00 - - - 00 (00) 00 00 - - -

T0

Overbite

Manual 31 (18) 00 18 100 - -033

29 (19) 05 18 096 089 099100

Digital 30 (18) 04 18 098 094 099 29 (18) 00 18 100 - -

T1Manual 03 (07) 05 07 100 - -

-03 (07) 00 07 100 - -

-Digital 03 (07) 05 07 100 - - 03 (07) 04 05 100 - -

J Clin Med 2021 10 1646 6 of 11

Table 2 Cont

Rater I Rater II

PAR Index Scoring and Timepoint ScoringMethod

Mean(SD) a

Err Methoda MSD a ICC [95 CI]

p-Value(Pairedt-Test)

Mean(SD) a

ErrMethod

aMSD a ICC [95 CI] p-Value (Paired

t-Test)

T0

Centerline

Manual 21 (30) 00 30 100 100 100033

21 (30) 00 30 100 100 100033

Digital 20 (27) 08 26 096 089 099 19 (26) 00 26 100 - -

T1Manual 00 (00) 00 00 - -

-00 (00) 00 00 - - -

-Digital 00 (00) 00 00 - - 00 (00) 00 00 - - -

Error of the method according to Dahlbergrsquos formula ICC could not be calculated as all the scores for overjet and centerline were = 0 a Expressed in score points unit

Table 3 Inter-rater variability (error of the method and Minimum Standard Deviation (MSD)) (using the second measurements) and reproducibility (ICC) according to time point andmethod Paired sample t-tests comparing both raters

PAR Index Scoring and Timepoint Scoring Method

Rater I Rater II Rater I vs II

Mean (SD) a Mean (SD) a Err Method a MSD a ICC [95 CI] p-Value (Pairedt-Test

T0

Total PAR

Manual 296 (100) 303 (972) 18 103 099 099 100 020

Digital 302 (102) 304 (102) 08 106 100 099 100 055

T1Manual 11 (14) 11 (14) 02 12 099 096 099 100

Digital 11 (12) 10 (10) 04 10 098 095 099 026

T0

Lower anterior

Manual 41 (40) 45 (44) 06 40 099 098 100 006

Digital 44 (36) 45 (37) 07 35 099 096 099 064

T1Manual 03 (06) 03 (04) 00 06 100 - - -

Digital 00 (01) 01 (02) 00 03 100 - - -

T0

Upper anterior

Manual 61 (28) 63 (28) 05 28 099 097 100 033

Digital 64 (28) 64 (27) 06 27 099 098 100 038

T1Manual 01 (03) 01 (02) 03 03 100 049 094 -

Digital 04 (07) 01 (03) 06 03 100 064 096 -

J Clin Med 2021 10 1646 7 of 11

Table 3 Cont

PAR Index Scoring and Timepoint Scoring Method

Rater I Rater II Rater I vs II

Mean (SD) a Mean (SD) a Err Method a MSD a ICC [95 CI] p-Value (Pairedt-Test

T0

Posterior

Manual 13 (14) 14 (15) 04 14 098 093 099 058

Digital 15 (16) 15 (16) 00 16 100 - - -

T1Manual 02 (06) 02 (06) 00 06 099 096 099 033

Digital 03 (06) 02 (06) 02 06 095 084 098 033

T0

Overjet

Manual 12 (75) 12 (75) 00 75 100 - - -

Digital 12 (75) 12 (75) 00 75 100 - - -

T1Manual 00 (00) 00 (00) 00 00 - - - -

Digital 00 (00) 00 (00) 00 00 - - - -

T0

Overbite

Manual 31 (18) 31 (18) 04 18 099 097 100 016

Digital 31 (18) 29 (18) 04 18 097 092 099 067

T1Manual 03 (07) 03 (07) 00 05 100 - - -

Digital 03 (07) 03 (07) 00 05 100 - - -

T0

Centerline

Manual 21 (30) 21 (30) 00 30 100 - - -

Digital 21 (30) 18 (25) 08 26 099 097 100 033

T1Manual 00 (00) 00 (00) 00 00 - - - -

Digital 00 (00) 00 (00) 00 00 - - - -

Error of the method according to Dahlbergrsquos formula using the second measurements ICC could not be calculated as all the scores for overjet and centerline were = 0 a Expressed in score points unit

J Clin Med 2021 10 1646 8 of 11

J Clin Med 2021 10 x FOR PEER REVIEW 7 of 10

(a)

(b)

Figure 2 (a) Scatter plot of the total weighted PAR scores measured by both digital and manual methods and both raters

with a line of unity (b) Bland-Altman plots inter-rater agreement for the total PAR scores measured by the digital and

manual PAR scoring methods at T0

32 Reproducibility

ICC for the total PAR scores and the PAR components at both time points and for

both methods fell in the 095ndash100 range for intra- and inter-rater reproducibility (Tables 2

Figure 2 (a) Scatter plot of the total weighted PAR scores measured by both digital and manual methods and both raterswith a line of unity (b) Bland-Altman plots inter-rater agreement for the total PAR scores measured by the digital andmanual PAR scoring methods at T0

J Clin Med 2021 10 1646 9 of 11

32 Reproducibility

ICC for the total PAR scores and the PAR components at both time points and for bothmethods fell in the 095ndash100 range for intra- and inter-rater reproducibility (Tables 2 and 3)All error-of-the-method values for the total PAR score and its components were smallerthan the associated minimum standard deviation

4 Discussion

Orthodontic model analysis is a prerequisite for diagnosis evaluation of treatmentneed treatment planning and analysis of treatment outcome The present study assessedthe validity and reproducibility of PAR index scoring for digital models and their printedmodel equivalents

Digitization of plaster models (scanned-in) was introduced in the 1990s [33] Theadvantages of digital models include the absence of physical storage requirements instantaccessibility and no risk of breakage or wear [2234] The analysis of scanned-in modelsis as valid as that for plaster models [24] However over the last ten years technologyhas evolved drastically offering high image resolution of digital models and upgradedplatforms needed for their analyses Hence intraoral scanning has gained popularityworldwide Several studies have confirmed the accuracy of direct digital models to beas accurate as that of plaster models Consequently direct digital models are used as analternative to conventional impression techniques and materials [3536] In the presentstudy we used the direct digital model technique

Numerous intraoral scanners and software were developed over the last decade withvarious diagnostic tools In the present study Ortho Analyzer software was used forscoring the digital models and a digital caliper was used for scoring the printed modelsAnalyzing digital models can be associated with some concerns The main concern withthe use of digital software is in fact adjusting the visualization of a 3-D object on a two-dimensional screen An appropriate evaluation requires a correct model orientation Forinstance in this study cross bites were difficult to visualize and rotation of the modelwas required to fully comprehend the magnitude of the cross bite This problem was alsoreported by Stevens et al [12] In addition segmentation of the dental crowns which is aninevitable step to create a virtual setup before carrying out the scoring is time-consumingTo ensure accurate tooth displacement measures one should pay attention when placingthe points parallel to the occlusal plane rotating the model adequately to facilitate goodvisualization of the contact points

Another concern when dental measurements are performed on printed models is theconsideration of the printing technique and the model base design Two studies [2627]evaluated the accuracy of printed models acquired from intraoral scans Brown et al [26]compared plaster models with printed models using two types of 3-D printing techniquesand concluded that both digital light processing (DLP) and polyjet printers producedclinically acceptable models Camardella et al [27] compared 3-D printed models withdifferent base designs using two types of printing techniques and concluded that 3-Dprinted models from intraoral scans created with the polyjet printing technique wereaccurate regardless of the model base design By contrast 3-D printed models with ahorseshoe-shaped base design printed with a stereolithography printer showed a significanttransversal (intra-arch distances) contraction and a horseshoe-shaped base with a posteriorconnection bar was accurate compared with printed models with a regular base Thereforein the present study a polyjet printing technique and regular model base designs wereused to ensure accuracy

Luqmani et al [28] compared automated PAR scoring of direct and indirect digitalmodels (CS 3600 software Carestream Dental Stuttgart Germany) with manual scoringof plaster models using the PAR ruler The authors found that manual PAR scoring wasthe most time-efficient whereas indirect digital model scoring was the least time-efficientThe latter had minor dental cast faults that led to time-consuming software adjustmentsHowever automated scoring was more efficient than the software scoring used by Mayers

J Clin Med 2021 10 1646 10 of 11

et al [23] which required operators to identify each relevant landmark Hence indirectscoring depends on the quality of the dental casts and can be time-consuming dependingon the software used In the present study PAR scoring was not possible as the softwareused (Ortho Analyzer) does not have the feature of automated scoring

In the present study the slightly higher reproducibility of PAR scoring for the digitalmethod was not significant with both methods proving to be highly reproducible Thehigh reproducibility of the manual method coincides with the results of Richmond et al [7]The reproducibility and variability of the direct digital method were similar to the findingsdescribed by Luqmani et al [28] Furthermore the limited variability between both meth-ods demonstrated the high validity of the digital method compared with the conventionalmanual method used as the gold standard

5 Conclusions

PAR scoring on digital models using a software showed excellent reproducibility andpresented good validity compared with manual scoring considered as the gold standard

Author Contributions Conceptualization AG and SG Data curation AG Formal analysis AGFunding acquisition MAC Supervision PMC and MAC Validation AG and SG Writingmdashoriginal draft AG Writingmdashreview amp editing AG SG MD PMC and MAC All authors haveread and agreed to the published version of the manuscript

Funding This research was funded by Aarhus University Forskingsfond (AUFF) grant number 2432

Data Availability Statement The data presented in this study are available on request from thecorresponding author

Acknowledgments The authors express their gratitude to Dirk Leonhardt for his assistance inprinting the models

Conflicts of Interest The authors declare no conflict of interest The funders had no role in the designof the study in the collection analyses or interpretation of data in the writing of the manuscript orin the decision to publish the results

References1 Gottlieb EL Grading your orthodontic treatment results J Clin Orthod 1975 9 155ndash1612 Pangrazio-Kulbersh V Kaczynski R Shunock M Early treatment outcome assessed by the Peer Assessment Rating index Am

J Orthod Dentofac Orthop 1999 115 544ndash550 [CrossRef]3 Summers CJ The occlusal index A system for identifying and scoring occlusal disorders Am J Orthod 1971 59 552ndash567

[CrossRef]4 Brook PH Shaw WC The development of an index of orthodontic treatment priority Eur J Orthod 1989 11 309ndash320

[CrossRef]5 Daniels C Richmond S The development of the index of complexity outcome and need (ICON) J Orthod 2000 27 149ndash162

[CrossRef] [PubMed]6 Casko JS Vaden JL Kokich VG Damone J James RD Cangialosi TJ Riolo ML Owens SE Jr Bills ED Objective

grading system for dental casts and panoramic radiographs American Board of Orthodontics Am J Orthod Dentofacial Orthop1998 114 589ndash599 [CrossRef]

7 Richmond S Shaw WC OrsquoBrien KD Buchanan IB Jones R Stephens CD Roberts CT Andrews M The developmentof the PAR Index (Peer Assessment Rating) Reliability and validity Eur J Orthod 1992 14 125ndash139 [CrossRef] [PubMed]

8 Cons NC Jenny J Kohout FJ Songpaisan Y Jotikastira D Utility of the dental aesthetic index in industrialized anddeveloping countries J Public Health Dent 1989 49 163ndash166 [CrossRef] [PubMed]

9 Richmond S Shaw WC Roberts CT Andrews M The PAR Index (Peer Assessment Rating) Methods to determine outcomeof orthodontic treatment in terms of improvement and standards Eur J Orthod 1992 14 180ndash187 [CrossRef]

10 DeGuzman L Bahiraei D Vig KW Vig PS Weyant RJ OrsquoBrien K The validation of the Peer Assessment Rating index formalocclusion severity and treatment difficulty Am J Orthod Dentofac Orthop 1995 107 172ndash176 [CrossRef]

11 Firestone AR Beck FM Beglin FM Vig KW Evaluation of the peer assessment rating (PAR) index as an index of orthodontictreatment need Am J Orthod Dentofac Orthop 2002 122 463ndash469 [CrossRef]

12 Stevens DR Flores-Mir C Nebbe B Raboud DW Heo G Major PW Validity reliability and reproducibility of plaster vsdigital study models Comparison of peer assessment rating and Bolton analysis and their constituent measurements Am JOrthod Dentofac Orthop 2006 129 794ndash803 [CrossRef] [PubMed]

J Clin Med 2021 10 1646 6 of 11

Table 2 Cont

Rater I Rater II

PAR Index Scoring and Timepoint ScoringMethod

Mean(SD) a

Err Methoda MSD a ICC [95 CI]

p-Value(Pairedt-Test)

Mean(SD) a

ErrMethod

aMSD a ICC [95 CI] p-Value (Paired

t-Test)

T0

Centerline

Manual 21 (30) 00 30 100 100 100033

21 (30) 00 30 100 100 100033

Digital 20 (27) 08 26 096 089 099 19 (26) 00 26 100 - -

T1Manual 00 (00) 00 00 - -

-00 (00) 00 00 - - -

-Digital 00 (00) 00 00 - - 00 (00) 00 00 - - -

Error of the method according to Dahlbergrsquos formula ICC could not be calculated as all the scores for overjet and centerline were = 0 a Expressed in score points unit

Table 3 Inter-rater variability (error of the method and Minimum Standard Deviation (MSD)) (using the second measurements) and reproducibility (ICC) according to time point andmethod Paired sample t-tests comparing both raters

PAR Index Scoring and Timepoint Scoring Method

Rater I Rater II Rater I vs II

Mean (SD) a Mean (SD) a Err Method a MSD a ICC [95 CI] p-Value (Pairedt-Test

T0

Total PAR

Manual 296 (100) 303 (972) 18 103 099 099 100 020

Digital 302 (102) 304 (102) 08 106 100 099 100 055

T1Manual 11 (14) 11 (14) 02 12 099 096 099 100

Digital 11 (12) 10 (10) 04 10 098 095 099 026

T0

Lower anterior

Manual 41 (40) 45 (44) 06 40 099 098 100 006

Digital 44 (36) 45 (37) 07 35 099 096 099 064

T1Manual 03 (06) 03 (04) 00 06 100 - - -

Digital 00 (01) 01 (02) 00 03 100 - - -

T0

Upper anterior

Manual 61 (28) 63 (28) 05 28 099 097 100 033

Digital 64 (28) 64 (27) 06 27 099 098 100 038

T1Manual 01 (03) 01 (02) 03 03 100 049 094 -

Digital 04 (07) 01 (03) 06 03 100 064 096 -

J Clin Med 2021 10 1646 7 of 11

Table 3 Cont

PAR Index Scoring and Timepoint Scoring Method

Rater I Rater II Rater I vs II

Mean (SD) a Mean (SD) a Err Method a MSD a ICC [95 CI] p-Value (Pairedt-Test

T0

Posterior

Manual 13 (14) 14 (15) 04 14 098 093 099 058

Digital 15 (16) 15 (16) 00 16 100 - - -

T1Manual 02 (06) 02 (06) 00 06 099 096 099 033

Digital 03 (06) 02 (06) 02 06 095 084 098 033

T0

Overjet

Manual 12 (75) 12 (75) 00 75 100 - - -

Digital 12 (75) 12 (75) 00 75 100 - - -

T1Manual 00 (00) 00 (00) 00 00 - - - -

Digital 00 (00) 00 (00) 00 00 - - - -

T0

Overbite

Manual 31 (18) 31 (18) 04 18 099 097 100 016

Digital 31 (18) 29 (18) 04 18 097 092 099 067

T1Manual 03 (07) 03 (07) 00 05 100 - - -

Digital 03 (07) 03 (07) 00 05 100 - - -

T0

Centerline

Manual 21 (30) 21 (30) 00 30 100 - - -

Digital 21 (30) 18 (25) 08 26 099 097 100 033

T1Manual 00 (00) 00 (00) 00 00 - - - -

Digital 00 (00) 00 (00) 00 00 - - - -

Error of the method according to Dahlbergrsquos formula using the second measurements ICC could not be calculated as all the scores for overjet and centerline were = 0 a Expressed in score points unit

J Clin Med 2021 10 1646 8 of 11

J Clin Med 2021 10 x FOR PEER REVIEW 7 of 10

(a)

(b)

Figure 2 (a) Scatter plot of the total weighted PAR scores measured by both digital and manual methods and both raters

with a line of unity (b) Bland-Altman plots inter-rater agreement for the total PAR scores measured by the digital and

manual PAR scoring methods at T0

32 Reproducibility

ICC for the total PAR scores and the PAR components at both time points and for

both methods fell in the 095ndash100 range for intra- and inter-rater reproducibility (Tables 2

Figure 2 (a) Scatter plot of the total weighted PAR scores measured by both digital and manual methods and both raterswith a line of unity (b) Bland-Altman plots inter-rater agreement for the total PAR scores measured by the digital andmanual PAR scoring methods at T0

J Clin Med 2021 10 1646 9 of 11

32 Reproducibility

ICC for the total PAR scores and the PAR components at both time points and for bothmethods fell in the 095ndash100 range for intra- and inter-rater reproducibility (Tables 2 and 3)All error-of-the-method values for the total PAR score and its components were smallerthan the associated minimum standard deviation

4 Discussion

Orthodontic model analysis is a prerequisite for diagnosis evaluation of treatmentneed treatment planning and analysis of treatment outcome The present study assessedthe validity and reproducibility of PAR index scoring for digital models and their printedmodel equivalents

Digitization of plaster models (scanned-in) was introduced in the 1990s [33] Theadvantages of digital models include the absence of physical storage requirements instantaccessibility and no risk of breakage or wear [2234] The analysis of scanned-in modelsis as valid as that for plaster models [24] However over the last ten years technologyhas evolved drastically offering high image resolution of digital models and upgradedplatforms needed for their analyses Hence intraoral scanning has gained popularityworldwide Several studies have confirmed the accuracy of direct digital models to beas accurate as that of plaster models Consequently direct digital models are used as analternative to conventional impression techniques and materials [3536] In the presentstudy we used the direct digital model technique

Numerous intraoral scanners and software were developed over the last decade withvarious diagnostic tools In the present study Ortho Analyzer software was used forscoring the digital models and a digital caliper was used for scoring the printed modelsAnalyzing digital models can be associated with some concerns The main concern withthe use of digital software is in fact adjusting the visualization of a 3-D object on a two-dimensional screen An appropriate evaluation requires a correct model orientation Forinstance in this study cross bites were difficult to visualize and rotation of the modelwas required to fully comprehend the magnitude of the cross bite This problem was alsoreported by Stevens et al [12] In addition segmentation of the dental crowns which is aninevitable step to create a virtual setup before carrying out the scoring is time-consumingTo ensure accurate tooth displacement measures one should pay attention when placingthe points parallel to the occlusal plane rotating the model adequately to facilitate goodvisualization of the contact points

Another concern when dental measurements are performed on printed models is theconsideration of the printing technique and the model base design Two studies [2627]evaluated the accuracy of printed models acquired from intraoral scans Brown et al [26]compared plaster models with printed models using two types of 3-D printing techniquesand concluded that both digital light processing (DLP) and polyjet printers producedclinically acceptable models Camardella et al [27] compared 3-D printed models withdifferent base designs using two types of printing techniques and concluded that 3-Dprinted models from intraoral scans created with the polyjet printing technique wereaccurate regardless of the model base design By contrast 3-D printed models with ahorseshoe-shaped base design printed with a stereolithography printer showed a significanttransversal (intra-arch distances) contraction and a horseshoe-shaped base with a posteriorconnection bar was accurate compared with printed models with a regular base Thereforein the present study a polyjet printing technique and regular model base designs wereused to ensure accuracy

Luqmani et al [28] compared automated PAR scoring of direct and indirect digitalmodels (CS 3600 software Carestream Dental Stuttgart Germany) with manual scoringof plaster models using the PAR ruler The authors found that manual PAR scoring wasthe most time-efficient whereas indirect digital model scoring was the least time-efficientThe latter had minor dental cast faults that led to time-consuming software adjustmentsHowever automated scoring was more efficient than the software scoring used by Mayers

J Clin Med 2021 10 1646 10 of 11

et al [23] which required operators to identify each relevant landmark Hence indirectscoring depends on the quality of the dental casts and can be time-consuming dependingon the software used In the present study PAR scoring was not possible as the softwareused (Ortho Analyzer) does not have the feature of automated scoring

In the present study the slightly higher reproducibility of PAR scoring for the digitalmethod was not significant with both methods proving to be highly reproducible Thehigh reproducibility of the manual method coincides with the results of Richmond et al [7]The reproducibility and variability of the direct digital method were similar to the findingsdescribed by Luqmani et al [28] Furthermore the limited variability between both meth-ods demonstrated the high validity of the digital method compared with the conventionalmanual method used as the gold standard

5 Conclusions

PAR scoring on digital models using a software showed excellent reproducibility andpresented good validity compared with manual scoring considered as the gold standard

Author Contributions Conceptualization AG and SG Data curation AG Formal analysis AGFunding acquisition MAC Supervision PMC and MAC Validation AG and SG Writingmdashoriginal draft AG Writingmdashreview amp editing AG SG MD PMC and MAC All authors haveread and agreed to the published version of the manuscript

Funding This research was funded by Aarhus University Forskingsfond (AUFF) grant number 2432

Data Availability Statement The data presented in this study are available on request from thecorresponding author

Acknowledgments The authors express their gratitude to Dirk Leonhardt for his assistance inprinting the models

Conflicts of Interest The authors declare no conflict of interest The funders had no role in the designof the study in the collection analyses or interpretation of data in the writing of the manuscript orin the decision to publish the results

References1 Gottlieb EL Grading your orthodontic treatment results J Clin Orthod 1975 9 155ndash1612 Pangrazio-Kulbersh V Kaczynski R Shunock M Early treatment outcome assessed by the Peer Assessment Rating index Am

J Orthod Dentofac Orthop 1999 115 544ndash550 [CrossRef]3 Summers CJ The occlusal index A system for identifying and scoring occlusal disorders Am J Orthod 1971 59 552ndash567

[CrossRef]4 Brook PH Shaw WC The development of an index of orthodontic treatment priority Eur J Orthod 1989 11 309ndash320

[CrossRef]5 Daniels C Richmond S The development of the index of complexity outcome and need (ICON) J Orthod 2000 27 149ndash162

[CrossRef] [PubMed]6 Casko JS Vaden JL Kokich VG Damone J James RD Cangialosi TJ Riolo ML Owens SE Jr Bills ED Objective

grading system for dental casts and panoramic radiographs American Board of Orthodontics Am J Orthod Dentofacial Orthop1998 114 589ndash599 [CrossRef]

7 Richmond S Shaw WC OrsquoBrien KD Buchanan IB Jones R Stephens CD Roberts CT Andrews M The developmentof the PAR Index (Peer Assessment Rating) Reliability and validity Eur J Orthod 1992 14 125ndash139 [CrossRef] [PubMed]

8 Cons NC Jenny J Kohout FJ Songpaisan Y Jotikastira D Utility of the dental aesthetic index in industrialized anddeveloping countries J Public Health Dent 1989 49 163ndash166 [CrossRef] [PubMed]

9 Richmond S Shaw WC Roberts CT Andrews M The PAR Index (Peer Assessment Rating) Methods to determine outcomeof orthodontic treatment in terms of improvement and standards Eur J Orthod 1992 14 180ndash187 [CrossRef]

10 DeGuzman L Bahiraei D Vig KW Vig PS Weyant RJ OrsquoBrien K The validation of the Peer Assessment Rating index formalocclusion severity and treatment difficulty Am J Orthod Dentofac Orthop 1995 107 172ndash176 [CrossRef]

11 Firestone AR Beck FM Beglin FM Vig KW Evaluation of the peer assessment rating (PAR) index as an index of orthodontictreatment need Am J Orthod Dentofac Orthop 2002 122 463ndash469 [CrossRef]

12 Stevens DR Flores-Mir C Nebbe B Raboud DW Heo G Major PW Validity reliability and reproducibility of plaster vsdigital study models Comparison of peer assessment rating and Bolton analysis and their constituent measurements Am JOrthod Dentofac Orthop 2006 129 794ndash803 [CrossRef] [PubMed]

J Clin Med 2021 10 1646 7 of 11

Table 3 Cont

PAR Index Scoring and Timepoint Scoring Method

Rater I Rater II Rater I vs II

Mean (SD) a Mean (SD) a Err Method a MSD a ICC [95 CI] p-Value (Pairedt-Test

T0

Posterior

Manual 13 (14) 14 (15) 04 14 098 093 099 058

Digital 15 (16) 15 (16) 00 16 100 - - -

T1Manual 02 (06) 02 (06) 00 06 099 096 099 033

Digital 03 (06) 02 (06) 02 06 095 084 098 033

T0

Overjet

Manual 12 (75) 12 (75) 00 75 100 - - -

Digital 12 (75) 12 (75) 00 75 100 - - -

T1Manual 00 (00) 00 (00) 00 00 - - - -

Digital 00 (00) 00 (00) 00 00 - - - -

T0

Overbite

Manual 31 (18) 31 (18) 04 18 099 097 100 016

Digital 31 (18) 29 (18) 04 18 097 092 099 067

T1Manual 03 (07) 03 (07) 00 05 100 - - -

Digital 03 (07) 03 (07) 00 05 100 - - -

T0

Centerline

Manual 21 (30) 21 (30) 00 30 100 - - -

Digital 21 (30) 18 (25) 08 26 099 097 100 033

T1Manual 00 (00) 00 (00) 00 00 - - - -

Digital 00 (00) 00 (00) 00 00 - - - -

Error of the method according to Dahlbergrsquos formula using the second measurements ICC could not be calculated as all the scores for overjet and centerline were = 0 a Expressed in score points unit

J Clin Med 2021 10 1646 8 of 11

J Clin Med 2021 10 x FOR PEER REVIEW 7 of 10

(a)

(b)

Figure 2 (a) Scatter plot of the total weighted PAR scores measured by both digital and manual methods and both raters

with a line of unity (b) Bland-Altman plots inter-rater agreement for the total PAR scores measured by the digital and

manual PAR scoring methods at T0

32 Reproducibility

ICC for the total PAR scores and the PAR components at both time points and for

both methods fell in the 095ndash100 range for intra- and inter-rater reproducibility (Tables 2

Figure 2 (a) Scatter plot of the total weighted PAR scores measured by both digital and manual methods and both raterswith a line of unity (b) Bland-Altman plots inter-rater agreement for the total PAR scores measured by the digital andmanual PAR scoring methods at T0

J Clin Med 2021 10 1646 9 of 11

32 Reproducibility

ICC for the total PAR scores and the PAR components at both time points and for bothmethods fell in the 095ndash100 range for intra- and inter-rater reproducibility (Tables 2 and 3)All error-of-the-method values for the total PAR score and its components were smallerthan the associated minimum standard deviation

4 Discussion

Orthodontic model analysis is a prerequisite for diagnosis evaluation of treatmentneed treatment planning and analysis of treatment outcome The present study assessedthe validity and reproducibility of PAR index scoring for digital models and their printedmodel equivalents

Digitization of plaster models (scanned-in) was introduced in the 1990s [33] Theadvantages of digital models include the absence of physical storage requirements instantaccessibility and no risk of breakage or wear [2234] The analysis of scanned-in modelsis as valid as that for plaster models [24] However over the last ten years technologyhas evolved drastically offering high image resolution of digital models and upgradedplatforms needed for their analyses Hence intraoral scanning has gained popularityworldwide Several studies have confirmed the accuracy of direct digital models to beas accurate as that of plaster models Consequently direct digital models are used as analternative to conventional impression techniques and materials [3536] In the presentstudy we used the direct digital model technique

Numerous intraoral scanners and software were developed over the last decade withvarious diagnostic tools In the present study Ortho Analyzer software was used forscoring the digital models and a digital caliper was used for scoring the printed modelsAnalyzing digital models can be associated with some concerns The main concern withthe use of digital software is in fact adjusting the visualization of a 3-D object on a two-dimensional screen An appropriate evaluation requires a correct model orientation Forinstance in this study cross bites were difficult to visualize and rotation of the modelwas required to fully comprehend the magnitude of the cross bite This problem was alsoreported by Stevens et al [12] In addition segmentation of the dental crowns which is aninevitable step to create a virtual setup before carrying out the scoring is time-consumingTo ensure accurate tooth displacement measures one should pay attention when placingthe points parallel to the occlusal plane rotating the model adequately to facilitate goodvisualization of the contact points

Another concern when dental measurements are performed on printed models is theconsideration of the printing technique and the model base design Two studies [2627]evaluated the accuracy of printed models acquired from intraoral scans Brown et al [26]compared plaster models with printed models using two types of 3-D printing techniquesand concluded that both digital light processing (DLP) and polyjet printers producedclinically acceptable models Camardella et al [27] compared 3-D printed models withdifferent base designs using two types of printing techniques and concluded that 3-Dprinted models from intraoral scans created with the polyjet printing technique wereaccurate regardless of the model base design By contrast 3-D printed models with ahorseshoe-shaped base design printed with a stereolithography printer showed a significanttransversal (intra-arch distances) contraction and a horseshoe-shaped base with a posteriorconnection bar was accurate compared with printed models with a regular base Thereforein the present study a polyjet printing technique and regular model base designs wereused to ensure accuracy

Luqmani et al [28] compared automated PAR scoring of direct and indirect digitalmodels (CS 3600 software Carestream Dental Stuttgart Germany) with manual scoringof plaster models using the PAR ruler The authors found that manual PAR scoring wasthe most time-efficient whereas indirect digital model scoring was the least time-efficientThe latter had minor dental cast faults that led to time-consuming software adjustmentsHowever automated scoring was more efficient than the software scoring used by Mayers

J Clin Med 2021 10 1646 10 of 11

et al [23] which required operators to identify each relevant landmark Hence indirectscoring depends on the quality of the dental casts and can be time-consuming dependingon the software used In the present study PAR scoring was not possible as the softwareused (Ortho Analyzer) does not have the feature of automated scoring

In the present study the slightly higher reproducibility of PAR scoring for the digitalmethod was not significant with both methods proving to be highly reproducible Thehigh reproducibility of the manual method coincides with the results of Richmond et al [7]The reproducibility and variability of the direct digital method were similar to the findingsdescribed by Luqmani et al [28] Furthermore the limited variability between both meth-ods demonstrated the high validity of the digital method compared with the conventionalmanual method used as the gold standard

5 Conclusions

PAR scoring on digital models using a software showed excellent reproducibility andpresented good validity compared with manual scoring considered as the gold standard

Author Contributions Conceptualization AG and SG Data curation AG Formal analysis AGFunding acquisition MAC Supervision PMC and MAC Validation AG and SG Writingmdashoriginal draft AG Writingmdashreview amp editing AG SG MD PMC and MAC All authors haveread and agreed to the published version of the manuscript

Funding This research was funded by Aarhus University Forskingsfond (AUFF) grant number 2432

Data Availability Statement The data presented in this study are available on request from thecorresponding author

Acknowledgments The authors express their gratitude to Dirk Leonhardt for his assistance inprinting the models

Conflicts of Interest The authors declare no conflict of interest The funders had no role in the designof the study in the collection analyses or interpretation of data in the writing of the manuscript orin the decision to publish the results

References1 Gottlieb EL Grading your orthodontic treatment results J Clin Orthod 1975 9 155ndash1612 Pangrazio-Kulbersh V Kaczynski R Shunock M Early treatment outcome assessed by the Peer Assessment Rating index Am

J Orthod Dentofac Orthop 1999 115 544ndash550 [CrossRef]3 Summers CJ The occlusal index A system for identifying and scoring occlusal disorders Am J Orthod 1971 59 552ndash567

[CrossRef]4 Brook PH Shaw WC The development of an index of orthodontic treatment priority Eur J Orthod 1989 11 309ndash320

[CrossRef]5 Daniels C Richmond S The development of the index of complexity outcome and need (ICON) J Orthod 2000 27 149ndash162

[CrossRef] [PubMed]6 Casko JS Vaden JL Kokich VG Damone J James RD Cangialosi TJ Riolo ML Owens SE Jr Bills ED Objective

grading system for dental casts and panoramic radiographs American Board of Orthodontics Am J Orthod Dentofacial Orthop1998 114 589ndash599 [CrossRef]

7 Richmond S Shaw WC OrsquoBrien KD Buchanan IB Jones R Stephens CD Roberts CT Andrews M The developmentof the PAR Index (Peer Assessment Rating) Reliability and validity Eur J Orthod 1992 14 125ndash139 [CrossRef] [PubMed]

8 Cons NC Jenny J Kohout FJ Songpaisan Y Jotikastira D Utility of the dental aesthetic index in industrialized anddeveloping countries J Public Health Dent 1989 49 163ndash166 [CrossRef] [PubMed]

9 Richmond S Shaw WC Roberts CT Andrews M The PAR Index (Peer Assessment Rating) Methods to determine outcomeof orthodontic treatment in terms of improvement and standards Eur J Orthod 1992 14 180ndash187 [CrossRef]

10 DeGuzman L Bahiraei D Vig KW Vig PS Weyant RJ OrsquoBrien K The validation of the Peer Assessment Rating index formalocclusion severity and treatment difficulty Am J Orthod Dentofac Orthop 1995 107 172ndash176 [CrossRef]

11 Firestone AR Beck FM Beglin FM Vig KW Evaluation of the peer assessment rating (PAR) index as an index of orthodontictreatment need Am J Orthod Dentofac Orthop 2002 122 463ndash469 [CrossRef]

12 Stevens DR Flores-Mir C Nebbe B Raboud DW Heo G Major PW Validity reliability and reproducibility of plaster vsdigital study models Comparison of peer assessment rating and Bolton analysis and their constituent measurements Am JOrthod Dentofac Orthop 2006 129 794ndash803 [CrossRef] [PubMed]

J Clin Med 2021 10 1646 8 of 11

J Clin Med 2021 10 x FOR PEER REVIEW 7 of 10

(a)

(b)

Figure 2 (a) Scatter plot of the total weighted PAR scores measured by both digital and manual methods and both raters

with a line of unity (b) Bland-Altman plots inter-rater agreement for the total PAR scores measured by the digital and

manual PAR scoring methods at T0

32 Reproducibility

ICC for the total PAR scores and the PAR components at both time points and for

both methods fell in the 095ndash100 range for intra- and inter-rater reproducibility (Tables 2

Figure 2 (a) Scatter plot of the total weighted PAR scores measured by both digital and manual methods and both raterswith a line of unity (b) Bland-Altman plots inter-rater agreement for the total PAR scores measured by the digital andmanual PAR scoring methods at T0

J Clin Med 2021 10 1646 9 of 11

32 Reproducibility

ICC for the total PAR scores and the PAR components at both time points and for bothmethods fell in the 095ndash100 range for intra- and inter-rater reproducibility (Tables 2 and 3)All error-of-the-method values for the total PAR score and its components were smallerthan the associated minimum standard deviation

4 Discussion

Orthodontic model analysis is a prerequisite for diagnosis evaluation of treatmentneed treatment planning and analysis of treatment outcome The present study assessedthe validity and reproducibility of PAR index scoring for digital models and their printedmodel equivalents

Digitization of plaster models (scanned-in) was introduced in the 1990s [33] Theadvantages of digital models include the absence of physical storage requirements instantaccessibility and no risk of breakage or wear [2234] The analysis of scanned-in modelsis as valid as that for plaster models [24] However over the last ten years technologyhas evolved drastically offering high image resolution of digital models and upgradedplatforms needed for their analyses Hence intraoral scanning has gained popularityworldwide Several studies have confirmed the accuracy of direct digital models to beas accurate as that of plaster models Consequently direct digital models are used as analternative to conventional impression techniques and materials [3536] In the presentstudy we used the direct digital model technique

Numerous intraoral scanners and software were developed over the last decade withvarious diagnostic tools In the present study Ortho Analyzer software was used forscoring the digital models and a digital caliper was used for scoring the printed modelsAnalyzing digital models can be associated with some concerns The main concern withthe use of digital software is in fact adjusting the visualization of a 3-D object on a two-dimensional screen An appropriate evaluation requires a correct model orientation Forinstance in this study cross bites were difficult to visualize and rotation of the modelwas required to fully comprehend the magnitude of the cross bite This problem was alsoreported by Stevens et al [12] In addition segmentation of the dental crowns which is aninevitable step to create a virtual setup before carrying out the scoring is time-consumingTo ensure accurate tooth displacement measures one should pay attention when placingthe points parallel to the occlusal plane rotating the model adequately to facilitate goodvisualization of the contact points

Another concern when dental measurements are performed on printed models is theconsideration of the printing technique and the model base design Two studies [2627]evaluated the accuracy of printed models acquired from intraoral scans Brown et al [26]compared plaster models with printed models using two types of 3-D printing techniquesand concluded that both digital light processing (DLP) and polyjet printers producedclinically acceptable models Camardella et al [27] compared 3-D printed models withdifferent base designs using two types of printing techniques and concluded that 3-Dprinted models from intraoral scans created with the polyjet printing technique wereaccurate regardless of the model base design By contrast 3-D printed models with ahorseshoe-shaped base design printed with a stereolithography printer showed a significanttransversal (intra-arch distances) contraction and a horseshoe-shaped base with a posteriorconnection bar was accurate compared with printed models with a regular base Thereforein the present study a polyjet printing technique and regular model base designs wereused to ensure accuracy

Luqmani et al [28] compared automated PAR scoring of direct and indirect digitalmodels (CS 3600 software Carestream Dental Stuttgart Germany) with manual scoringof plaster models using the PAR ruler The authors found that manual PAR scoring wasthe most time-efficient whereas indirect digital model scoring was the least time-efficientThe latter had minor dental cast faults that led to time-consuming software adjustmentsHowever automated scoring was more efficient than the software scoring used by Mayers

J Clin Med 2021 10 1646 10 of 11

et al [23] which required operators to identify each relevant landmark Hence indirectscoring depends on the quality of the dental casts and can be time-consuming dependingon the software used In the present study PAR scoring was not possible as the softwareused (Ortho Analyzer) does not have the feature of automated scoring

In the present study the slightly higher reproducibility of PAR scoring for the digitalmethod was not significant with both methods proving to be highly reproducible Thehigh reproducibility of the manual method coincides with the results of Richmond et al [7]The reproducibility and variability of the direct digital method were similar to the findingsdescribed by Luqmani et al [28] Furthermore the limited variability between both meth-ods demonstrated the high validity of the digital method compared with the conventionalmanual method used as the gold standard

5 Conclusions

PAR scoring on digital models using a software showed excellent reproducibility andpresented good validity compared with manual scoring considered as the gold standard

Author Contributions Conceptualization AG and SG Data curation AG Formal analysis AGFunding acquisition MAC Supervision PMC and MAC Validation AG and SG Writingmdashoriginal draft AG Writingmdashreview amp editing AG SG MD PMC and MAC All authors haveread and agreed to the published version of the manuscript

Funding This research was funded by Aarhus University Forskingsfond (AUFF) grant number 2432

Data Availability Statement The data presented in this study are available on request from thecorresponding author

Acknowledgments The authors express their gratitude to Dirk Leonhardt for his assistance inprinting the models

Conflicts of Interest The authors declare no conflict of interest The funders had no role in the designof the study in the collection analyses or interpretation of data in the writing of the manuscript orin the decision to publish the results

References1 Gottlieb EL Grading your orthodontic treatment results J Clin Orthod 1975 9 155ndash1612 Pangrazio-Kulbersh V Kaczynski R Shunock M Early treatment outcome assessed by the Peer Assessment Rating index Am

J Orthod Dentofac Orthop 1999 115 544ndash550 [CrossRef]3 Summers CJ The occlusal index A system for identifying and scoring occlusal disorders Am J Orthod 1971 59 552ndash567

[CrossRef]4 Brook PH Shaw WC The development of an index of orthodontic treatment priority Eur J Orthod 1989 11 309ndash320

[CrossRef]5 Daniels C Richmond S The development of the index of complexity outcome and need (ICON) J Orthod 2000 27 149ndash162

[CrossRef] [PubMed]6 Casko JS Vaden JL Kokich VG Damone J James RD Cangialosi TJ Riolo ML Owens SE Jr Bills ED Objective

grading system for dental casts and panoramic radiographs American Board of Orthodontics Am J Orthod Dentofacial Orthop1998 114 589ndash599 [CrossRef]

7 Richmond S Shaw WC OrsquoBrien KD Buchanan IB Jones R Stephens CD Roberts CT Andrews M The developmentof the PAR Index (Peer Assessment Rating) Reliability and validity Eur J Orthod 1992 14 125ndash139 [CrossRef] [PubMed]

8 Cons NC Jenny J Kohout FJ Songpaisan Y Jotikastira D Utility of the dental aesthetic index in industrialized anddeveloping countries J Public Health Dent 1989 49 163ndash166 [CrossRef] [PubMed]

9 Richmond S Shaw WC Roberts CT Andrews M The PAR Index (Peer Assessment Rating) Methods to determine outcomeof orthodontic treatment in terms of improvement and standards Eur J Orthod 1992 14 180ndash187 [CrossRef]

10 DeGuzman L Bahiraei D Vig KW Vig PS Weyant RJ OrsquoBrien K The validation of the Peer Assessment Rating index formalocclusion severity and treatment difficulty Am J Orthod Dentofac Orthop 1995 107 172ndash176 [CrossRef]

11 Firestone AR Beck FM Beglin FM Vig KW Evaluation of the peer assessment rating (PAR) index as an index of orthodontictreatment need Am J Orthod Dentofac Orthop 2002 122 463ndash469 [CrossRef]

12 Stevens DR Flores-Mir C Nebbe B Raboud DW Heo G Major PW Validity reliability and reproducibility of plaster vsdigital study models Comparison of peer assessment rating and Bolton analysis and their constituent measurements Am JOrthod Dentofac Orthop 2006 129 794ndash803 [CrossRef] [PubMed]

J Clin Med 2021 10 1646 9 of 11

32 Reproducibility

ICC for the total PAR scores and the PAR components at both time points and for bothmethods fell in the 095ndash100 range for intra- and inter-rater reproducibility (Tables 2 and 3)All error-of-the-method values for the total PAR score and its components were smallerthan the associated minimum standard deviation

4 Discussion

Orthodontic model analysis is a prerequisite for diagnosis evaluation of treatmentneed treatment planning and analysis of treatment outcome The present study assessedthe validity and reproducibility of PAR index scoring for digital models and their printedmodel equivalents

Digitization of plaster models (scanned-in) was introduced in the 1990s [33] Theadvantages of digital models include the absence of physical storage requirements instantaccessibility and no risk of breakage or wear [2234] The analysis of scanned-in modelsis as valid as that for plaster models [24] However over the last ten years technologyhas evolved drastically offering high image resolution of digital models and upgradedplatforms needed for their analyses Hence intraoral scanning has gained popularityworldwide Several studies have confirmed the accuracy of direct digital models to beas accurate as that of plaster models Consequently direct digital models are used as analternative to conventional impression techniques and materials [3536] In the presentstudy we used the direct digital model technique

Numerous intraoral scanners and software were developed over the last decade withvarious diagnostic tools In the present study Ortho Analyzer software was used forscoring the digital models and a digital caliper was used for scoring the printed modelsAnalyzing digital models can be associated with some concerns The main concern withthe use of digital software is in fact adjusting the visualization of a 3-D object on a two-dimensional screen An appropriate evaluation requires a correct model orientation Forinstance in this study cross bites were difficult to visualize and rotation of the modelwas required to fully comprehend the magnitude of the cross bite This problem was alsoreported by Stevens et al [12] In addition segmentation of the dental crowns which is aninevitable step to create a virtual setup before carrying out the scoring is time-consumingTo ensure accurate tooth displacement measures one should pay attention when placingthe points parallel to the occlusal plane rotating the model adequately to facilitate goodvisualization of the contact points

Another concern when dental measurements are performed on printed models is theconsideration of the printing technique and the model base design Two studies [2627]evaluated the accuracy of printed models acquired from intraoral scans Brown et al [26]compared plaster models with printed models using two types of 3-D printing techniquesand concluded that both digital light processing (DLP) and polyjet printers producedclinically acceptable models Camardella et al [27] compared 3-D printed models withdifferent base designs using two types of printing techniques and concluded that 3-Dprinted models from intraoral scans created with the polyjet printing technique wereaccurate regardless of the model base design By contrast 3-D printed models with ahorseshoe-shaped base design printed with a stereolithography printer showed a significanttransversal (intra-arch distances) contraction and a horseshoe-shaped base with a posteriorconnection bar was accurate compared with printed models with a regular base Thereforein the present study a polyjet printing technique and regular model base designs wereused to ensure accuracy

Luqmani et al [28] compared automated PAR scoring of direct and indirect digitalmodels (CS 3600 software Carestream Dental Stuttgart Germany) with manual scoringof plaster models using the PAR ruler The authors found that manual PAR scoring wasthe most time-efficient whereas indirect digital model scoring was the least time-efficientThe latter had minor dental cast faults that led to time-consuming software adjustmentsHowever automated scoring was more efficient than the software scoring used by Mayers

J Clin Med 2021 10 1646 10 of 11

et al [23] which required operators to identify each relevant landmark Hence indirectscoring depends on the quality of the dental casts and can be time-consuming dependingon the software used In the present study PAR scoring was not possible as the softwareused (Ortho Analyzer) does not have the feature of automated scoring

In the present study the slightly higher reproducibility of PAR scoring for the digitalmethod was not significant with both methods proving to be highly reproducible Thehigh reproducibility of the manual method coincides with the results of Richmond et al [7]The reproducibility and variability of the direct digital method were similar to the findingsdescribed by Luqmani et al [28] Furthermore the limited variability between both meth-ods demonstrated the high validity of the digital method compared with the conventionalmanual method used as the gold standard

5 Conclusions

PAR scoring on digital models using a software showed excellent reproducibility andpresented good validity compared with manual scoring considered as the gold standard

Author Contributions Conceptualization AG and SG Data curation AG Formal analysis AGFunding acquisition MAC Supervision PMC and MAC Validation AG and SG Writingmdashoriginal draft AG Writingmdashreview amp editing AG SG MD PMC and MAC All authors haveread and agreed to the published version of the manuscript

Funding This research was funded by Aarhus University Forskingsfond (AUFF) grant number 2432

Data Availability Statement The data presented in this study are available on request from thecorresponding author

Acknowledgments The authors express their gratitude to Dirk Leonhardt for his assistance inprinting the models

Conflicts of Interest The authors declare no conflict of interest The funders had no role in the designof the study in the collection analyses or interpretation of data in the writing of the manuscript orin the decision to publish the results

References1 Gottlieb EL Grading your orthodontic treatment results J Clin Orthod 1975 9 155ndash1612 Pangrazio-Kulbersh V Kaczynski R Shunock M Early treatment outcome assessed by the Peer Assessment Rating index Am

J Orthod Dentofac Orthop 1999 115 544ndash550 [CrossRef]3 Summers CJ The occlusal index A system for identifying and scoring occlusal disorders Am J Orthod 1971 59 552ndash567

[CrossRef]4 Brook PH Shaw WC The development of an index of orthodontic treatment priority Eur J Orthod 1989 11 309ndash320

[CrossRef]5 Daniels C Richmond S The development of the index of complexity outcome and need (ICON) J Orthod 2000 27 149ndash162

[CrossRef] [PubMed]6 Casko JS Vaden JL Kokich VG Damone J James RD Cangialosi TJ Riolo ML Owens SE Jr Bills ED Objective

grading system for dental casts and panoramic radiographs American Board of Orthodontics Am J Orthod Dentofacial Orthop1998 114 589ndash599 [CrossRef]

7 Richmond S Shaw WC OrsquoBrien KD Buchanan IB Jones R Stephens CD Roberts CT Andrews M The developmentof the PAR Index (Peer Assessment Rating) Reliability and validity Eur J Orthod 1992 14 125ndash139 [CrossRef] [PubMed]

8 Cons NC Jenny J Kohout FJ Songpaisan Y Jotikastira D Utility of the dental aesthetic index in industrialized anddeveloping countries J Public Health Dent 1989 49 163ndash166 [CrossRef] [PubMed]

9 Richmond S Shaw WC Roberts CT Andrews M The PAR Index (Peer Assessment Rating) Methods to determine outcomeof orthodontic treatment in terms of improvement and standards Eur J Orthod 1992 14 180ndash187 [CrossRef]

10 DeGuzman L Bahiraei D Vig KW Vig PS Weyant RJ OrsquoBrien K The validation of the Peer Assessment Rating index formalocclusion severity and treatment difficulty Am J Orthod Dentofac Orthop 1995 107 172ndash176 [CrossRef]

11 Firestone AR Beck FM Beglin FM Vig KW Evaluation of the peer assessment rating (PAR) index as an index of orthodontictreatment need Am J Orthod Dentofac Orthop 2002 122 463ndash469 [CrossRef]

12 Stevens DR Flores-Mir C Nebbe B Raboud DW Heo G Major PW Validity reliability and reproducibility of plaster vsdigital study models Comparison of peer assessment rating and Bolton analysis and their constituent measurements Am JOrthod Dentofac Orthop 2006 129 794ndash803 [CrossRef] [PubMed]

J Clin Med 2021 10 1646 10 of 11

et al [23] which required operators to identify each relevant landmark Hence indirectscoring depends on the quality of the dental casts and can be time-consuming dependingon the software used In the present study PAR scoring was not possible as the softwareused (Ortho Analyzer) does not have the feature of automated scoring

In the present study the slightly higher reproducibility of PAR scoring for the digitalmethod was not significant with both methods proving to be highly reproducible Thehigh reproducibility of the manual method coincides with the results of Richmond et al [7]The reproducibility and variability of the direct digital method were similar to the findingsdescribed by Luqmani et al [28] Furthermore the limited variability between both meth-ods demonstrated the high validity of the digital method compared with the conventionalmanual method used as the gold standard

5 Conclusions

PAR scoring on digital models using a software showed excellent reproducibility andpresented good validity compared with manual scoring considered as the gold standard

Author Contributions Conceptualization AG and SG Data curation AG Formal analysis AGFunding acquisition MAC Supervision PMC and MAC Validation AG and SG Writingmdashoriginal draft AG Writingmdashreview amp editing AG SG MD PMC and MAC All authors haveread and agreed to the published version of the manuscript

Funding This research was funded by Aarhus University Forskingsfond (AUFF) grant number 2432

Data Availability Statement The data presented in this study are available on request from thecorresponding author

Acknowledgments The authors express their gratitude to Dirk Leonhardt for his assistance inprinting the models

Conflicts of Interest The authors declare no conflict of interest The funders had no role in the designof the study in the collection analyses or interpretation of data in the writing of the manuscript orin the decision to publish the results

References1 Gottlieb EL Grading your orthodontic treatment results J Clin Orthod 1975 9 155ndash1612 Pangrazio-Kulbersh V Kaczynski R Shunock M Early treatment outcome assessed by the Peer Assessment Rating index Am

J Orthod Dentofac Orthop 1999 115 544ndash550 [CrossRef]3 Summers CJ The occlusal index A system for identifying and scoring occlusal disorders Am J Orthod 1971 59 552ndash567

[CrossRef]4 Brook PH Shaw WC The development of an index of orthodontic treatment priority Eur J Orthod 1989 11 309ndash320

[CrossRef]5 Daniels C Richmond S The development of the index of complexity outcome and need (ICON) J Orthod 2000 27 149ndash162

[CrossRef] [PubMed]6 Casko JS Vaden JL Kokich VG Damone J James RD Cangialosi TJ Riolo ML Owens SE Jr Bills ED Objective

grading system for dental casts and panoramic radiographs American Board of Orthodontics Am J Orthod Dentofacial Orthop1998 114 589ndash599 [CrossRef]

7 Richmond S Shaw WC OrsquoBrien KD Buchanan IB Jones R Stephens CD Roberts CT Andrews M The developmentof the PAR Index (Peer Assessment Rating) Reliability and validity Eur J Orthod 1992 14 125ndash139 [CrossRef] [PubMed]

8 Cons NC Jenny J Kohout FJ Songpaisan Y Jotikastira D Utility of the dental aesthetic index in industrialized anddeveloping countries J Public Health Dent 1989 49 163ndash166 [CrossRef] [PubMed]

9 Richmond S Shaw WC Roberts CT Andrews M The PAR Index (Peer Assessment Rating) Methods to determine outcomeof orthodontic treatment in terms of improvement and standards Eur J Orthod 1992 14 180ndash187 [CrossRef]

10 DeGuzman L Bahiraei D Vig KW Vig PS Weyant RJ OrsquoBrien K The validation of the Peer Assessment Rating index formalocclusion severity and treatment difficulty Am J Orthod Dentofac Orthop 1995 107 172ndash176 [CrossRef]

11 Firestone AR Beck FM Beglin FM Vig KW Evaluation of the peer assessment rating (PAR) index as an index of orthodontictreatment need Am J Orthod Dentofac Orthop 2002 122 463ndash469 [CrossRef]

12 Stevens DR Flores-Mir C Nebbe B Raboud DW Heo G Major PW Validity reliability and reproducibility of plaster vsdigital study models Comparison of peer assessment rating and Bolton analysis and their constituent measurements Am JOrthod Dentofac Orthop 2006 129 794ndash803 [CrossRef] [PubMed]

J Clin Med 2021 10 1646 11 of 11

13 Zilberman O Huggare JA Parikakis KA Evaluation of the validity of tooth size and arch width measurements usingconventional and three-dimensional virtual orthodontic models Angle Orthod 2003 73 301ndash306 [CrossRef] [PubMed]

14 Bichara LM Aragon ML Brandao GA Normando D Factors influencing orthodontic treatment time for non-surgical ClassIII malocclusion J Appl Oral Sci 2016 24 431ndash436 [CrossRef]

15 Chalabi O Preston CB Al-Jewair TS Tabbaa S A comparison of orthodontic treatment outcomes using the ObjectiveGrading System (OGS) and the Peer Assessment Rating (PAR) index Aust Orthod J 2015 31 157ndash164 [PubMed]

16 Dyken RA Sadowsky PL Hurst D Orthodontic outcomes assessment using the peer assessment rating index Angle Orthod2001 71 164ndash169 [CrossRef]

17 Fink DF Smith RJ The duration of orthodontic treatment Am J Orthod Dentofac Orthop 1992 102 45ndash51 [CrossRef]18 Ormiston JP Huang GJ Little RM Decker JD Seuk GD Retrospective analysis of long-term stable and unstable

orthodontic treatment outcomes Am J Orthod Dentofac Orthop 2005 128 568ndash574 [CrossRef]19 Pavlow SS McGorray SP Taylor MG Dolce C King GJ Wheeler TT Effect of early treatment on stability of occlusion in

patients with Class II malocclusion Am J Orthod Dentofac Orthop 2008 133 235ndash244 [CrossRef]20 Garino F Garino GB Comparison of dental arch measurements between stone and digital casts Am J Orthod Dentofac Orthop

2002 3 250ndash25421 Mullen SR Martin CA Ngan P Gladwin M Accuracy of space analysis with emodels and plaster models Am J Orthod

Dentofac Orthop 2007 132 346ndash352 [CrossRef]22 Rheude B Sadowsky PL Ferriera A Jacobson A An evaluation of the use of digital study models in orthodontic diagnosis

and treatment planning Angle Orthod 2005 75 300ndash304 [CrossRef]23 Mayers M Firestone AR Rashid R Vig KW Comparison of peer assessment rating (PAR) index scores of plaster and

computer-based digital models Am J Orthod Dentofac Orthop 2005 128 431ndash434 [CrossRef]24 Abizadeh N Moles DR OrsquoNeill J Noar JH Digital versus plaster study models How accurate and reproducible are they J

Orthod 2012 39 151ndash159 [CrossRef] [PubMed]25 Dalstra M Melsen B From alginate impressions to digital virtual models Accuracy and reproducibility J Orthod 2009 36

36ndash41 [CrossRef] [PubMed]26 Brown GB Currier GF Kadioglu O Kierl JP Accuracy of 3-dimensional printed dental models reconstructed from digital

intraoral impressions Am J Orthod Dentofac Orthop 2018 154 733ndash739 [CrossRef] [PubMed]27 Camardella LT de Vasconcellos Vilella O Breuning H Accuracy of printed dental models made with 2 prototype technologies

and different designs of model bases Am J Orthod Dentofac Orthop 2017 151 1178ndash1187 [CrossRef]28 Luqmani S Jones A Andiappan M Cobourne MT A comparison of conventional vs automated digital Peer Assessment

Rating scoring using the Carestream 3600 scanner and CS Model+ software system A randomized controlled trial Am J OrthodDentofac Orthop 2020 157 148ndash155e141 [CrossRef]

29 Walter SD Eliasziw M Donner A Sample size and optimal designs for reliability studies Stat Med 1998 17 101ndash110[CrossRef]

30 Harris PA Taylor R Minor BL Elliott V Fernandez M OrsquoNeal L McLeod L Delacqua G Delacqua F Kirby J et alThe REDCap consortium Building an international community of software platform partners J Biomed Inform 2019 95 103208[CrossRef]

31 Harris PA Taylor R Thielke R Payne J Gonzalez N Conde JG Research electronic data capture (REDCap)mdashA metadata-driven methodology and workflow process for providing translational research informatics support J Biomed Inform 2009 42377ndash381 [CrossRef] [PubMed]

32 Dahlberg G Statistical Methods for Medical and Biological Students George Allen and Unwin London UK 194033 Joffe L Current Products and Practices OrthoCADtrade Digital models for a digital era J Orthod 2004 31 344ndash347 [CrossRef]

[PubMed]34 McGuinness NJ Stephens CD Storage of orthodontic study models in hospital units in the UK Br J Orthod 1992 19 227ndash232

[CrossRef] [PubMed]35 Fluumlgge TV Schlager S Nelson K Nahles S Metzger MC Precision of intraoral digital dental impressions with iTero and

extraoral digitization with the iTero and a model scanner Am J Orthod Dentofacial Orthop 2013 144 471ndash478 [CrossRef]36 Gruumlnheid T McCarthy SD Larson BE Clinical use of a direct chairside oral scanner An assessment of accuracy time and

patient acceptance Am J Orthod Dentofac Orthop 2014 146 673ndash682 [CrossRef] [PubMed]

Minerva Access is the Institutional Repository of The University of Melbourne

Authors

Gera A Gera S Dalstra M Cattaneo PM Cornelis MA

Title

Validity and Reproducibility of the Peer Assessment Rating Index Scored on Digital Models

Using a Software Compared with Traditional Manual Scoring

Date

2021-04-01

Citation

Gera A Gera S Dalstra M Cattaneo P M amp Cornelis M A (2021) Validity and

Reproducibility of the Peer Assessment Rating Index Scored on Digital Models Using a

Software Compared with Traditional Manual Scoring JOURNAL OF CLINICAL MEDICINE

10 (8) httpsdoiorg103390jcm10081646

Persistent Link

httphdlhandlenet11343287173

File Description

Published version

License

CC BY

• Introduction
• Materials and Methods
• • Sample Size Calculation
  • Setting
  • Sample Collection
  • Measurements
  • Statistical Analyses
  • • Results
    • • Validity
      • Reproducibility
      • • Discussion
        • Conclusions
        • References