Development of Three Web-Based Computerized Versions of ... Validation Paper.pdf[CD], ODD), eating and elimination disorders (enuresis, encopresis, anorexia nervosa, bulimia, binge

Q1Q2

Q3

Q4

12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758

5960

NEW RESEARCH

616263646566676869707172737475767778798081828384858687888990919293949596979899100101

Development of Three Web-Based ComputerizedVersions of the Kiddie Schedule for Affective Disordersand Schizophrenia Child Psychiatric DiagnosticInterview: Preliminary Validity DataLisa Townsend, PhD, Kenneth Kobak, PhD, Catherine Kearney, MA, Michael Milham, MD, PhD,Charissa Andreotti, PhD, Jasmine Escalera, PhD, Lindsay Alexander, MPH, Mary Kay Gill, MSN,Boris Birmaher, MD, Raeanne Sylvester, MSW, Dawn Rice, MS, Alison Deep, MCA,Joan Kaufman, PhD

Objective: To present initial validity data on three web-based computerized versions of the Kiddie Schedule for Affective Disorders and Schizophrenia(KSADS-COMP).

Method: The sample for evaluating the validity of the clinician-administered KSADS-COMP included 511 youths 6–18 years of age who wereparticipants in the Child Mind Institute Healthy Brain Network. The sample for evaluating the parent and youth self-administered versions of theKSADS-COMP included 158 youths 11-17 years of age recruited from three academic institutions.

Results: Average administration time for completing the combined parent and youth clinician-administered KSADS-COMP was less time thanpreviously reported for completing the paper-and-pencil KSADS with only one informant (91.9 � 50.1 minutes). Average administration times for theyouth and parent self-administered KSADS-COMP were 50.9 � 28.0 minutes and 63.2 � 38.3 minutes, respectively, and youths and parents ratedtheir experience using the web-based self-administered KSADS-COMP versions very positively. Diagnoses generated with all three KSADS-COMPversions demonstrated good convergent validity against established clinical rating scales and dimensional diagnostic-specific ratings derived from theKSADS-COMP. When parent and youth self-administered KSADS-COMP data were integrated, good to excellent concordance was also achievedbetween diagnoses derived using the self-administered and clinician-administered KSADS-COMP versions (area under the curve ¼ 0.89–1.00).Conclusion: The three versions of the KSADS-COMP demonstrate promising psychometric properties, while offering efficiency in administrationand scoring. The clinician-administered KSADS-COMP shows utility not only for research, but also for implementation in clinical practice, with self-report preinterview ratings that streamline administration. The self-administered KSADS-COMP versions have numerous potential research and clinicalapplications, including in large-scale epidemiological studies, in schools, in emergency departments, and in telehealth to address the critical shortage ofchild and adolescent mental health specialists.

Key words: child and adolescent psychiatric diagnoses, computerized assessment, KSADS

J Am Acad Child Adolesc Psychiatry 2019;-(-):-–-.

T

Journal of tVolume - /

102103104105106

his article describes preliminary validity data forthree updated, web-based computerized versionsof the Kiddie Schedule for Affective Disorders and

107108109110111112113114115

Schizophrenia for school-age children (KSADS-COMP):1 aclinician-administered version, a self-administered youthversion, and a self-administered parent version. The paper-and-pencil KSADS was originally developed in 1978 as anextension of the adult version of the Schedule of AffectiveDisorders and Schizophrenia (SADS).2 The paper-and-pencilversion of the KSADS has been translated into more than 30

he American Academy of Child & Adolescent PsychiatryNumber - / - 2019

FLA 5.6.0 DTD � JAAC2574_proof �

different languages and has undergone several revisions, asreviewed elsewhere.3 The paper-and-pencil version of theKSADS has been the diagnostic instrument used in multiplestudies sponsored by the National Institutes of Health andthe pharmaceutical industry,4-8 including clinical trials thatevaluated treatments for attention-deficit/hyperactivity dis-order (ADHD),9,10 oppositional defiant disorder (ODD),11

major depressive disorder (MDD),12,13 anxiety disorders,14

early-onset bipolar disorder,6 schizophrenia,5 posttraumaticstress disorder (PTSD),15 among many others. Many of the

www.jaacap.org 1

14 June 2019 � 7:26 pm � ce

116

http://www.jaacap.org

Q7

TOWNSEND et al.

117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175

176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234

clinical trials that employed the KSADS have resulted inchanges in pediatric drug labeling by the U.S. Food andDrug Administration (FDA). It has also been used as avalidation instrument in large-scale epidemiological studies ofpsychiatric disorders in youths.16-18

Standardized interviews such as the KSADS are asso-ciated with increased identification of suicidal ideation anddisorders underreported in unstructured assessments.19,20

The paper-and-pencil KSADS has demonstrated superiordiagnostic accuracy compared with traditional unstructureddiagnostic assessments, particularly for complex, highly co-morbid cases in outpatient, emergency department, andinpatient settings.21,22 Furthermore, prior work has shownthat computerized versions of paper-and-pencil scales aresuperior to the paper-and-pencil versions, with branchingand scoring errors minimized when computerized assess-ment instruments are used.23-25

The paper-and-pencil version of the KSADS wasdesigned to assess present and past symptoms according toDSM-IV criteria.1 It is a semistructured diagnostic interviewwith probes that evaluate specific symptoms using objectivecriteria regarding symptom intensity and frequency. Theprobes for each symptom included in the instrument aredesigned to be used flexibly, giving interviewers ampleleeway for clarifying questions and probing further asneeded to score individual items.

The paper-and-pencil KSADS consists of three primarycomponents: 1) an unstructured introductory interview, 2)a diagnostic screening interview, and 3) supplements tofinalize the criteria required for each diagnosis. The un-structured introductory interview gathers demographic in-formation; family composition and history of psychiatricillness data; a brief description of the presenting problem;history of prior mental health treatment; and general in-formation about the child’s interests and adaptive func-tioning (hobbies, friendships, behavior and performance atschool), with new questions added to the unstructuredintroductory interview of the KSADS-COMP aboutbullying, sexual orientation, and gender identification. Theintroductory interview is a critical component of theKSADS because it helps to establish rapport, generate hy-potheses about likely relevant diagnoses, and establish acontext to elicit symptoms and evaluate the child’s func-tioning. The diagnostic screening interview surveys two tofour symptoms of each disorder assessed in the KSADS,with skip out criteria that determine if the supplements forthose disorders should be administered. The screen inter-view is designed to provide a good diagnostic overview andwhen completed in its entirety before moving to the sup-plements greatly facilitates differential diagnoses. Diagnosticsupplements are then administered in the chronological

2 www.jaacap.org


order in which probable diagnoses emerged except when theonset of one disorder (eg, a substance use disorder) mayhave influenced the presentation of the other diagnosis (eg,mood disorder).

All three versions of the KSADS-COMP haveretained the three primary components of the paper-and-pencil KSADS—the unstructured introductory interview,the screen interview, and the diagnostic supplements.However, four major changes were made in developingthe three KSADS-COMP versions. First, the instrumentwas updated to reflect DSM-5 diagnostic criteria; second,the instrument was computerized, including automatedscoring algorithms and data capture features; third, theKSADS-COMP was designed to generate both categoricaldiagnoses and diagnosis-specific dimensional rating scalesof current symptoms; and fourth, the scoring criteria weremodified so that the response options for all currentsymptoms are scored using the same standardized 5-pointrating scale. The frequency of all current symptoms overthe past 2 weeks is now rated on one common metric inall three versions of the KSADS-COMP (eg, not at all,rarely, several days, more than half the days, and nearlyevery day). The threshold for clinical significance variesdepending on the item. For example, the threshold forfailure to fulfill a major role obligation associated withsubstance use (eg, missing school due to substance use) islower than the threshold for depressive irritability, giventhat missing school “rarely” or only once during a 2-weekperiod can signal a potential substance misuse disorder,whereas the threshold for irritability in the depressivedisorders section is “more than half the days.” The paper-and-pencil version of the KSADS has unique scoringcriteria for every item, making training and establishingreliability in administration problematic.

Questions included in the KSADS-COMP were writtenat a sixth grade Flesch-Kincaid level. Some of the probesincluded in the KSADS-COMP were modified from thepaper-and-pencil version of the KSADS; some were devel-oped by the investigative team; and as in the development ofpast versions of the KSADS, others were developed withinput from experts in the field (see Acknowledgments sec-tion for list of experts who provided input on the devel-opment and/or refinement of KSADS-COMP probes and/or scoring criteria).

The three versions of KSADS-COMP assess the sameset of diagnoses contained in the DSM-5–updated versionof the paper-and-pencil KSADS,26 including mood disor-ders (MDD, persistent depression, mania, hypomania,cyclothymia, bipolar disorders, and disruptive mood dys-regulation disorder), psychotic disorders (schizoaffectivedisorders, schizophrenia, schizophreniform disorder, brief

Journal of the American Academy of Child & Adolescent PsychiatryVolume - / Number - / - 2019

14 June 2019 � 7:26 pm � ce


Q8

DEVELOPMENT OF THE KSADS-COMP

235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293

294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352

psychotic disorder), anxiety disorders (panic disorder,agoraphobia, separation anxiety disorder, simple phobia,social anxiety disorder, selective mutism, generalized anxietydisorder [GAD], obsessive-compulsive disorder), neuro-developmental disorders (ADHD, autism spectrum disorder,transient tic disorder, Tourette’s disorder, chronic motor orvocal tic disorder), behavioral disorders (conduct disorder[CD], ODD), eating and elimination disorders (enuresis,encopresis, anorexia nervosa, bulimia, binge eating disorder),trauma- or stressor-related disorders (PTSD, adjustmentdisorders), and alcohol use and substance use disorders aswell as numerous other specified diagnoses when full criteriafor these diagnoses are not met.

With the three versions of KSADS-COMP, a variety ofreports are available to the clinicians in real time. TheSymptoms/Comments Report provides a detailed listing ofeach symptom item administered and responses of theyouth and caregiver to each item. All comments writtenthroughout by the clinician, parent, or youth can also beprinted using this report. This is useful for summarizinginformation about how individuals describe their symptoms(eg, “I feel like a volcano sometimes”), capturing informa-tion about specific events (eg, reports of adverse childhoodexperiences), and details about clinically significant behav-iors (eg, suspensions). The Diagnosis Report provides cur-rent and past diagnoses; their associated ICD-10 codes; a listof all threshold level symptoms; and information aboutwhether the diagnosis is current, past, or in partial remis-sion. The Diagnostic Report also provides a comprehensivelist of all suicidality items and a rating according to theColumbia Classification Algorithm of Suicide Assessment(C-CASA),27 as recommended for FDA clinical trials.Additional unique features of the clinician-administered andself-administered versions of the KSADS-COMP aredescribed in “Methods.”

This article describes two studies. The first studyexamined the validity of the clinician-administered versionof the KSADS-COMP, and the second study examined thevalidity of the parent and youth self-administered versionsof the KSADS-COMP.

METHOD: STUDY ONE: VALIDATION OFCLINICIAN-ADMINISTERED KSADS-COMPProceduresThe sample for the initial validation study of the clinician-administered KSADS-COMP comprised participants fromthe Child Mind Institute Healthy Brain Network (HBN)initiative, which includes the clinician-administeredKSADS-COMP and a number of other relevant clinicalassessments in its standard assessment battery.28 Subjectsrecruited for the HBN initiative before October 2018 who



had the clinician-administered KSADS-COMP with bothinformants and all the relevant ratings scales were includedin this report. HBN is a large-scale data collection effort(target N ¼ 10,000) focused on the generation of an openresource for studying child and adolescent mental health(see HBN website at http://fcon_1000.projects.nitrc.org/indi/cmi_healthy_brain_network/index.html for details). Adata sharing agreement was signed allowing for the sharingof de-identified data for the purposes of examining thevalidity of the KSADS-COMP, and the use of the de-identified data included in this article was approved bythe Johns Hopkins Institutional Review Board.

ParticipantsA total of 511 English-speaking youth and parent dyadsfrom the HBN initiative were assessed using the clinician-administered KSADS-COMP interview by two doctoral-level clinicians. According to the clinician-administeredKSADS-COMP, 75 youths had no psychiatric diagnosis,with an overrepresentation of children with psychopathol-ogy contained in the community-based HBN cohort givenincentives for recruitment, include free psychiatric andlearning assessments and referrals for services when clinicallyindicated. The average age for the sample was 11.8 years(SD 2.7), and youths ranged in age from 6 to 18 years at thetime of the interview. There were 307 boys (60%). Of thesample, 62% identified as white (N ¼ 317), and 19.5%identified as Hispanic. Only 63% of caregivers reportedincome data that were scored categorically based on incomeearnings below and above $90,000, with 53% of thosereporting income data above this threshold.

MeasuresClinician-Administered KSADS-COMP. The clinician-administered KSADS-COMP was administered to parentand youth participants by the same clinician. The parentinterview was completed first if the youth was a preado-lescent; the order was reversed if the youth was an adoles-cent. With the clinician-administered KSADS-COMP, aswith the paper-and-pencil version of the KSADS, final di-agnoses were based on consensus ratings integrating infor-mation derived from the parent and youth interviews. Ingeneral, greater weight is given to the youth’s reports ofinternalizing symptoms and the caregiver’s report of exter-nalizing symptoms, although latitude in clinical judgment isallowed.

In addition to having the three primary components ofthe paper-and-pencil KSADS discussed in the introduction(the unstructured introductory interview, the screen interview,and the diagnostic supplements), the clinician-administeredKSADS-COMP includes computerized youth and parent

www.jaacap.org 3

14 June 2019 � 7:26 pm � ce

http://fcon_1000.projects.nitrc.org/indi/cmi_healthy_brain_network/index.htmlhttp://fcon_1000.projects.nitrc.org/indi/cmi_healthy_brain_network/index.htmlhttp://www.jaacap.org

web3C=FPO

TOWNSEND et al.

353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411

412413414415416417418419420421422423424425426427428429430431432433434

preinterview self-report ratings of the screen interview items tostreamline the administration of the clinical interview.Figure 1 depicts a screenshot of the clinician-administeredKSADS-COMP interface. The figure shows the screen thatappears when the clinician is administering the parent inter-view. The screenshot shows the youth’s interview response inthe upper left corner, the parent’s preinterview response in theupper right corner, and the clinician’s response options in thecenter of the screen. All symptoms in the KSADS-COMP areinitially surveyed for severity over the past 2 weeks. If athreshold level response is provided, the KSADS-COMPinterview progresses to inquire about the next symptom; ifa subthreshold response is given, the interviewer is promptedto inquire about the lifetime occurrence of the symptom, withthe presence of past symptoms rated dichotomously.Threshold criteria are presented below the response options,allowing the clinician to determine what responses will beabove and below threshold for that symptom. In addition, ashighlighted by the red arrow on the screen, there is also aComments dropdown option associated with each item thatallows clinicians to write notes throughout the interview.

FIGURE 1 Kiddie Schedule for Affective Disorders and SchizophClinician-Administered Interview Interface

Note: This screenshot of the KSADS-COMP shows the clinician-administered parent inpreinterview self-report ratings in the upper right corner. The availability of these datacomments section, which can be expanded if the clinician wishes to make notes in respand the KSADS-COMP.

4 www.jaacap.org


The instructions for the clinician-administeredKSADS-COMP are similar to the instructions for thepaper-and-pencil KSADS. The KSADS-COMP is likewisea semistructured instrument and designed to be adminis-tered in a conversational style. Whereas fewer sampleprobes are included in the computer version, clinicians aretold they do not need to recite the probes verbatim, thatthey are free to make stylistic changes and incorporatelanguage generated by the parent or youth when con-ducting the interview, and that they need only ask as manyquestions as is necessary to score each item. In addition,information learned in the unstructured introductoryinterview can be used to further probe individual items.

Current threshold level and past “ever” responses willtrigger the supplement for a given disorder to appear at thebottom of the dashboard. As in the paper-and-pencil versionof the KSADS, the supplements include the necessaryfollow-up questions to determine if diagnostic criteria forthe disorder are met, if more than one episode of the dis-order was experienced, and if the current disorder is inpartial remission.

renia Computerized Version (KSADS-COMP) Screenshot—

terview screen with teen interview responses in the upper left corner and parenthelp to streamline the diagnostic interview. The red arrow calls attention to theonse to this item. See the text for a more complete description of this screenshot


14 June 2019 � 7:26 pm � ce

435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470


Q9

web3C=FPO


471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529

530531532533534535536537538539540541542543544545546547548549550551552553554555556557

Figure 2 shows a screenshot of the diagnostic dashboardof the clinician-administered KSADS-COMP and high-lights several additional features of the user interface. Thediagnostic interview dashboard appears once the unstruc-tured introductory interview of the KSADS-COMP hasbeen completed. The screen modules are shown in the toptwo thirds of the figure. Screening modules that have notbeen administered appear in green; completed modulesappear in gray. Thus, clinicians can determine at a glancethe screen modules that have and have not been completed.The bottom third of the figure depicts the supplementmodules that should be completed because threshold levelresponses were given in the screen interview, alleviating theneed for clinicians to track which supplements should beadministered after the screen interview is completed. Allmodules of the KSADS-COMP do not need to be adminis-tered; there is a “choose as you go” option for clinicians. Forexample, if a clinician has prior diagnostic information for ayouth and wishes to assess for only a specific disorder or if thepreinterview ratings completed by the parent and youthsuggest the likely presence of just one disorder, that onemodule can be selected independently of the other diagnosticmodules. This feature greatly enhances the efficiency andversatility of the clinician-administered KSADS-COMP forapplication in a variety of clinical settings.

FIGURE 2 Clinician-Administered Kiddie Schedule for Affective DCOMP) Dashboard—Screen Modules and Activated Supplements

Note: This screenshot shows the dashboard of the clinician-administered KSADS-COMPcompleted. All the screen interview modules are depicted on the top two thirds, andcomplete description of this screenshot.



Validation Measures. The measures used to validate thediagnoses generated with the clinician-administeredKSADS-COMP are identical (eg, Child Behavior Check-list, Screen for Child Anxiety and Related Emotional Dis-orders) or comparable (eg, Child Depression Inventory vs.Mood and Feelings Questionnaire) to the measures used tovalidate the paper-and-pencil KSADS.1

The Mood and Feelings Questionnaire (MFQ) (longform) is a 33-item (child report) or 34-item (parent report)well-validated scale that measures depressive symptoms inyouths 6 to 17 years of age, with individual items rated on a0 (“Not true”) to 2 (“True”) point rating scale.29-33 MFQtotal scores were used in the current report to validateclinician-administered KSADS-COMP depressive disorderdiagnoses.

The Screen for Child Anxiety Related EmotionalDisorders (SCARED) is a 41-item instrument that mea-sures anxiety disorder symptoms in children and adoles-cents via youth and parent report using a point rating scale(0–2).34-36 The parent and youth SCARED Total Scoreand GAD scale scores were used in the current report foranalytic purposes.

The Child Behavior Checklist (CBCL) is one of themost widely used instruments for measuring behavioral andemotional psychopathology in youths.37-39 The Attention

isorders and Schizophrenia Computerized Version (KSADS-

interview. The dashboard appears once the unstructured introductory interview isa sample of activated supplements are depicted below. See the text for a more

www.jaacap.org 5

14 June 2019 � 7:26 pm � ce

558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588


TOWNSEND et al.

589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647

648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705

Problem, Rule Breaking, and Aggressive Behavior stan-dardized scale scores were used in the current report tovalidate behavioral diagnoses generated with the clinician-administered KSADS-COMP.

Data AnalysesUnivariate descriptive statistics were calculated to characterizestudy participants and to evaluate the frequencies of DSM-5current and lifetime diagnoses generated with the KSADS-COMP. Current diagnoses included current and partiallyremitted episodes of disorders. For analytic purposes, fourdiagnostic categories were generated for youths who metcriteria for the following current disorders: any depressivedisorders, any anxiety disorders, ADHD, and ODD or CD.To assess convergent validity of the categorical diagnosesgenerated with the clinician-administered KSADS-COMP,Wilcoxon rank sum tests were used. Youths who met currentcriteria for a particular category of disorder were comparedwith youths with no lifetime history of a disorder in thatcategory on the measures assessing symptoms of that disorder.Nonparametric statistics were used given that the outcomemeasures were non–normally distributed. Spearman rankcorrelation coefficients were also calculated to evaluate theassociations between the standardized symptom measures(MFQ, SCARED, and CBCL subscales) and the KSADS-COMP dimensional rating scales associated with these dis-orders; Table S1 (available online) lists items included in eachKSADS-COMP diagnostic-specific dimensional rating scaleexamined in this report.

RESULTS: STUDY ONE: VALIDATION OFCLINICIAN-ADMINISTERED KSADS-COMPClinician-Administered KSADS-COMP Current andLifetime DiagnosesRates of current diagnoses are depicted in Table 1. Lifetimerates of diagnoses were as follows. A total of 66 (13%)youths met criteria for a lifetime depressive disorder; di-agnoses included MDD (n ¼ 45), persistent depressivedisorder (n ¼ 6), and other specified depressive disorder(n ¼ 17). A total of 213 (42%) youths met criteria for alifetime anxiety disorder, including panic disorder (n ¼ 4),other specified panic disorder (n ¼ 8), agoraphobia (n ¼18), separation anxiety (n ¼ 52), other specified separationanxiety disorder (n ¼ 15), social anxiety (n ¼ 81), specificphobia (n ¼ 80), GAD (n ¼ 92), other specified GAD (n ¼6), obsessive-compulsive disorder (n ¼ 36), and otherspecified obsessive-compulsive disorder (n ¼ 1). A total of339 (66%) youths met criteria for a lifetime ADHD diag-nosis. Of these, 278 youths met full criteria for currentADHD, 11 met criteria for ADHD in partial remission, 24

6 www.jaacap.org


met criteria for other specified ADHD, and 26 met criteriafor past ADHD. A total of 99 (19%) youths met criteria fora lifetime ODD, and 21 youths (4%) met lifetime criteriafor CD (16 childhood onset and 5 adolescent onset). Fre-quencies for the other disorders assessed with the KSADS-COMP were much lower than the above-describeddepressive, anxiety, and behavior disorders and are notpresented here.

Convergent Validity DataTable 1 presents the means and standard deviations foryouth and parent reports on the MFQ, SCARED, andCBCL subscales for youths with a current diagnosis of aparticular category of disorder and youths with no lifetimehistory of a disorder in that category. Youths with a currentdiagnosis differed significantly from youths without a his-tory of that given disorder on all standardized scales (p <.0001, all analyses).

Correlations Between Standardized Clinical RatingScales and Clinician-Administered KSADS-COMPDimensional Scales and Associations Between Clinician-Administered KSADS-COMP Dimensional Scales andDiagnostic Group AssignmentAs noted previously, Table S1 (available online) lists itemsincluded in each of the KSADS-COMP diagnostic-specificdimensional rating scales examined in this report. Thedimensional scales for these analyses were derived from theconsensus ratings that integrated parent and youth reports.The KSADS-COMP 3-item depression scale derived fromthe consensus ratings correlated significantly with theyouth (rS ¼ .25, p < .001) and parent (rS ¼ .40, p < .001)MFQ scores; the KSADS-COMP 1-item GAD consensusratings scale correlated significantly with the youth (rS ¼.33, p < .001) and parent (rS ¼ .43, p < .001) totalSCARED scores and the youth (rS ¼ .37, p < .001) andparent (rS ¼ .45, p < .001) SCARED GAD subscalescores; the KSADS-COMP 4-item consensus ADHD scalecorrelated significantly with the CBCL Attention Problemsubscale (rS ¼ .59, p < .001); and the KSADS-COMP 2-item ODD consensus ratings scale correlated significantlywith the CBCL rule breaking (rS ¼ .56, p < .001) andaggressive behavior (rS ¼ .61, p < .001) subscales.Table S2 (available online) provides means and standarddeviations for the clinician-administered KSADS-COMPdimensional scales for each diagnostic group. For eachdisorder, youths with a positive current diagnosis scoredsignificantly higher on the corresponding dimensionalscale than youths who did not meet criteria for that diag-nosis, suggesting the clinical utility of the screen itemsincluded in the scales.


14 June 2019 � 7:26 pm � ce

706


Q10

Q11

TABLE 1 Criterion Validity Data for Clinician-Administered Kiddie Schedule for Affective Disorders and SchizophreniaComputerized Version Current Diagnostic Groups on Standard Clinical Measures (N ¼ 511)

DSM-5 Current Diagnoses Standardized MeasureDiagnosis Positive,

Mean (SD)

DiagnosisNegative,Mean (SD) Z and p

Depressive disorders (n [ 26) MFQ-C 30.69 (18.25) 12.61 (9.64) Z [ L5.32; p < .0001MFQ-P 23.27 (12.48) 8.43 (8.18) Z [ L5.89; p < .0001

Anxiety disorders (n [ 158) SCARED-C total score 31.71 (19.08) 20.63 (15.06) Z [ L6.15; p < .0001SCARED-P total score 23.16 (13.17) 10.30 (8.48) Z [ L10.46; p < .0001

ADHD (n [ 313) CBCL attention problems 66.53 (9.73) 56.07 (7.31) Z [ L12.12; p < .0001ODD/CD (n [ 78) CBCL rule breaking 63.81 (7.56) 54.61 (6.00) Z [ L9.36; p < .0001

CBCL aggressive behavior 68.41 (8.93) 55.66 (7.14) Z [ L10.55; p < .0001

Note: Youths with positive current diagnoses scored greater than youths without positive diagnoses on each of the standard clinical rating scales,providing convergent validity of the diagnoses generated with the clinician-administered KSADS-COMP. Wilcoxon rank sum tests were used toevaluate the differences between KSADS-COMP positive and negative diagnostic groups on the standardized measures. Boldface indicates significantresults. ADHD ¼ attention-deficit/hyperactivity disorder (cutoff score ¼ 65); C ¼ Child; CBCL ¼ Child Behavior Checklist; KSADS-COMP ¼ KiddieSchedule for Affective Disorders and Schizophrenia Computerized Version; MFQ ¼ Mood and Feelings Questionnaire (cutoff score ¼ 27); ODD/CD ¼oppositional defiant disorder/conduct disorder (cutoff score ¼ 65); P ¼ Parent; SCARED ¼ Screen for Child Anxiety Related Emotional Disorders(cutoff score ¼ 25).


707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755756757758759760761762763764765

766767768769770771772773774775776777778779780781782783784785786787788789790791792793794795796797798799800801802803804805806807808809810811812813814815816817818819820821822823824

METHOD: STUDY TWO: VALIDATION OFPARENT AND YOUTH SELF-ADMINISTEREDKSADS-COMPSParticipantsA total of 158 youth and parent dyads were recruitedfrom three university and clinical sites to validate the self-administered KSADS-COMPs: Kennedy Krieger Insti-tute (KKI) and other Johns Hopkins child and adolescentpsychiatry mental health programs (n ¼ 39), WesternPsychiatric Institute and Clinic (WPIC) at University ofPittsburgh Medical Center (n ¼ 71), and the Child MindInstitute (CMI) (n ¼ 48), with all the youths from CMIwho participated in the validation of the self-administeredKSADS-COMP also participants in study one, the vali-dation of the clinician-administered KSADS-COMP.A subset of 106 youths who completed the self-administered KSADS-COMP also completed a secondresearch visit to complete the clinician-administeredKSADS-COMP (see “Procedures” below regarding se-lection criteria for completing the second assessment).Inclusion criteria across the sites were 1) 11 to 17 years ofage, 2) parent available and willing to participate in theresearch, and 3) fluent in English. At the KKI and JohnsHopkins sites, all participants were required to bereceiving mental health services for study participation; atWPIC, normal controls were recruited from the offspringof healthy controls participating in the Bipolar OffspringStudy (BIOS) (n ¼ 30; grant MH060952; PI: B.B.), andyouths with psychopathology were recruited from theBIOS study and WPIC outpatient clinics; and at CMI,subjects were recruited from a pool of youths consenting



for the HBN initiative who also agreed to participate inthe KSADS-COMP study.

Subject CharacteristicsA total of 158 adolescents (n ¼ 82 [52%] boys, n ¼ 76[48%] girls) and their parents completed the self-administered KSADS-COMPs. The mean age of youthswas 13.8 years (SD 1.7). The sample was 54% white (n ¼86); 31% African American (n ¼ 49); 8% Hispanic orLatino (n ¼ 13); 1% each for American Indian or AlaskanNative (n ¼ 1), Asian (n ¼ 1), and Native Hawaiian orother Pacific Islander (n ¼ 2); and 3% other (n ¼ 5) (notepercentages equal more than 100%, as subjects couldchoose more than one racial identity). Of youths, 92% wereliving with a biological parent. The subset of 106 youthswho also completed the clinician-administered KSADS-COMP had an average age of 13.7 years (1.7); exactly 50%of the subsample (n ¼ 53) was male, and 65% (n ¼ 69)were white.

ProceduresGiven the well-documented tendency for informants toattenuate symptom reports on retest,40 to avoid systematicbias in results examining the concordance between the self-administered and clinician-administered KSADS-COMP,at the first study visit half the subjects were randomlyassigned to complete the adolescent and parent self-administered KSADS-COMP, and half the subjects wererandomly assigned to complete the clinician-administeredKSADS-COMP. Only subjects who met criteria forMDD, a bipolar diagnosis, ADHD, ODD or CD, PTSD,

www.jaacap.org 7

14 June 2019 � 7:26 pm � ce


TOWNSEND et al.

825826827828829830831832833834835836837838839840841842843844845846847848849850851852853854855856857858859860861862863864865866867868869870871872873874875876877878879880881882883

884885886887888889890891892893894895896897898899900901902903904905906907908909910911912913914915916917918919920921922923924925926927928929930931932933934935936937938939940941942

a substance use disorder, or no lifetime diagnoses duringthe first assessment were invited for a second study visit tocomplete the alternate (eg, self-report vs. clinician-administered) version of the KSADS-COMP. Among theyouths included in the reassessment sample, 53% (56 of106) completed the self-administered version of theKSADS-COMP at the initial assessment. The standardizedclinical assessment measures were completed at theconclusion of the first study visit, after completion ofeither the self-administered or clinician-administeredKSADS-COMP. Parents and youths were each compen-sated $50 for completion of the first KSADS-COMP as-sessments, and each received an additional $75 if invitedback to complete the second interview. Compensation fortravel was also provided.

When completed second, the clinician-administeredKSADS-COMP assessments were conducted blinded toinitial self-administered KSADS-COMP diagnoses and allstandardized clinical assessment measures completed duringthe first visit. The second KSADS was completed within 3weeks of the initial assessment for 98% of the cases (meanduration between KSADS-COMP assessments: 9.78 days;SD 6.33; range, 1–39 days). All study procedures receivedapproval by the Institutional Review Boards at each of theparticipating sites.

MeasuresSelf-Administered KSADS-COMP. The self-administeredversions of the KSADS-COMP are designed for youthsage 11 and older. Similar to the clinician-administeredKSADS-COMP, the self-administered versions of theinstrument contain the same three primary componentsof the paper-and-pencil KSADS, the introductory inter-view, the screening interview that evaluates key symptomsfrom each of the disorders covered in the KSADS, andsupplements that are administered for each diagnosis withabove threshold scores on the screening items of thatdiagnosis, in order to thoroughly evaluate the disordersaccording to DSM-5 criteria. The youth and parent self-report versions are administered to each informant sepa-rately. The self-administered KSADS-COMP can becompleted on-site or remotely, but was completed on-sitefor the current investigation.

The self-administered KSADS-COMP was designed toemulate the probing done by a trained clinician. As such, itcontains the same probes, response options, and scoring andbranching logic as the clinician-administered KSADS-COMP. For example, if a child endorsed long-standingdifficulties with inattention and ADHD symptoms andendorsed difficulties with concentration when completingthe depression supplement, the child would be presented

8 www.jaacap.org


with a question that asks whether the concentration diffi-culties got worse with the onset of the depressed mood. Asanother example of how the instrument was designed toemulate the probing done by a trained clinician, if the childendorsed a history of bullying and paranoid thoughts thatothers are out to get them, a question would be asked todetermine if the child feels it is just those who have beenbullying them that are out to get them or if the paranoidideation is more pervasive.

The parent version of the KSADS-COMP is just text-based. Questions in the youth self-report version areadministered with prerecorded video clips to facilitateadministration, and youths can choose a male (K.K.) orfemale (J.K.) interviewer; Figure S1 (available online)shows a screenshot of the youth report self-administeredKSADS-COMP. Youths also have the option to turnoff the video clips and simply read the probes. Parentsand youths have the ability to add comments at any timeto clarify their answers by either typing in or, with tablets,writing in with a stylus.

There is a suicide and homicide alert system that con-tacts the clinician via text or e-mail if a respondent reportssuicidal or homicidal ideation when completing the self-administered KSADS-COMP and an option to omit theseitems if the interview is being administered without aclinician on-site. Reports for the self-administered KSADS-COMP are likewise available to the clinician immediately,listing the diagnoses for which the youth met criteria; thesymptoms endorsed, including homicidality and suicidalityitems; C-CASA ratings; and all notes written in the com-ments sections.

Clinician-Administered KSADS-COMP. As described un-der “Method” for study one, the clinician-administeredKSADS-COMP is a computerized diagnostic interviewderived from the paper-and-pencil KSADS. All interviewersfor study two were licensed clinicians with extensive expe-rience administering the paper-and-pencil KSADS whoreceived a didactic training session by one of the authors(J.K.) on the administration of the KSADS-COMP and hadthe opportunity to experiment with the computer programbefore the initiation of the investigation. To establishinterrater reliability across sites, the eight assessors (eg, twoat CMI, four at UPMC, two at KKI) scored all the screenitems on two mock patient interviews that were videorecorded, with parent and youth preinterview ratingsavailable to assessors during the administration. On the firstinterview, all eight raters scored 94% (154 of 163) of theitems identically, and the eight raters agreed if the itemswere at or above the clinical threshold for 97.5% (159 of163) of the items. In rating the second interview, all eight


14 June 2019 � 7:26 pm � ce


Q12

Q13


94394494594694794894995095195295395495595695795895996096196296396496596696796896997097197297397497597697797897998098198298398498598698798898999099199299399499599699799899910001001

10021003100410051006100710081009101010111012101310141015101610171018101910201021102210231024102510261027102810291030103110321033103410351036103710381039104010411042104310441045104610471048104910501051105210531054105510561057105810591060

raters scored 96% (154 of 160) of the items identically, andthe eight raters agreed if the items were at or above theclinical threshold for 98% (157 of 160) of the items.Although diagnostic concordance was not determined, asdiagnoses are computer generated based on scores of indi-vidual KSADS-COMP items, diagnostic concordancewould thus be expected to be comparably high.

System Usability Scale. User satisfaction with the technicalaspects of the self-report versions of the KSADS-COMPwas assessed using the System Usability Scale (SUS).41,42

The SUS is a well-validated scale for assessing usabilityacross diverse types of user interfaces (eg, tablet, desktop,interactive voice response, cell phone), with good internalconsistency reliability (coefficient a ¼ .91).43 The SUScontains 10 items covering different aspects of the user’sexperience with the technology (eg, “I thought the com-puter interview was easy to use”; “The features of thecomputer interview were too complex”; “I would take acomputer interview designed like this again”). Each item israted on a 5-point scale, with anchor descriptions providedfor the endpoints (1 ¼ strongly disagree, 5 ¼ stronglyagree). A global rating of user-friendliness is also obtained.

User Satisfaction Questionnaire. The User SatisfactionQuestionnaire has been used in prior studies to assesssatisfaction with computer-administered versions of mentalhealth assessments (see scale items listed in Table 2).44

Users rated their experience on a 4-point scale (stronglyagree, agree, disagree, strongly disagree) and were asked ifthey would be willing to be interviewed with the self-reportKSADS-COMP again and whether they preferred to beasked these types of questions by computer or clinician or ifthey had no preference.

Patient Health Questionnaire. The 9-item Patient HealthQuestionnaire (PHQ-9) is a scale designed to measure keysymptoms of depression. Initially developed to screen fordepressive disorder among adults in primary care,45 the scalehas also demonstrated good psychometric properties amongadolescents.46 Response options range from 0 (not at all) to3 (nearly every day).

Brief Child Mania Rating Scale Parent and ChildReport. The 10-item Brief Child Mania Rating Scale(BCMS) was used in the current report; the BCMSdemonstrates similar psychometric properties and per-formance as the long version of the Child Mania RatingScale.47 Response options range from 0 (never/rare) to 3(very often). The original 21-item Child Mania RatingScale scale was designed to measure symptoms of bipolarspectrum illness. It has good psychometric properties andreliably distinguishes between symptoms of bipolar



disorder, characteristics of ADHD, and ratings of normalcontrols.48

GAD Scale. The 7-item GAD-7 youth report instrument isdesigned to measure key symptoms of GAD. Responseoptions range from 0 (not at all) to 3 (nearly every day).49

Strengths and Weaknesses of ADHD Symptoms andNormal Behavior Scale (SWAN). The Strengths andWeaknesses of ADHD Symptoms and Normal BehaviorScale (SWAN) was designed to measure parent-reportedsymptoms of ADHD (18 items) and ODD (12 items).50

Items are positively worded in order to measure youths’strengths as well as weaknesses. For example, parents areasked, “Compared to other children, how does your childdo the following: .” and “give close attention to detail andavoid careless mistakes.” Response options for the scaleallow for scoring of strengths and weaknesses and rangefrom �3 (far above average) to 3 (far below average).Primary Care PTSD Screen. The Primary Care PTSDscreen is a 4-item instrument with dichotomized responseoptions (yes/no) of core PTSD DSM-5 symptoms that wasdeveloped for adults, but has been shown to be an effectivescreening tool in adolescents as well.51,52 One point isassigned for each “yes” answer.

Data AnalysesUnivariate statistics were used to examine demographic char-acteristics, examine responses to the user satisfaction scales,and describe the frequencies of selected diagnoses. Bivariatestatistics examined associations between selected diagnosticgroups attained on the parent and youth self-report interviewsand the standardized assessment instruments and KSADS-COMP dimensional rating scales. Given that most of theoutcome measures were non–normally distributed, nonpara-metric statistics were used to examine differences betweendiagnostic groups and associations between scales. Percentagreement, Cohen’s k,53 and Gwet’s first-order agreementcoefficient (AC1) statistics54 were calculated to examineconcordance between parent and youth self-report andclinician-generated diagnoses for selected current psychiatricdisorders, with both Cohen’s k and Gwet’s AC1 statisticscalculated, as Gwet’s AC1 is less affected by prevalence andmarginal probability than Cohen’s k.53-55

Given the expected and observed high rates of informantvariance, multinomial logistic regression analyses were con-ducted to derive weights for integrating parent and youth datafrom the self-administered KSADS-COMP to predict di-agnoses derived from the clinician-administered KSADS-COMP, with the items selected for entry in the regressionmodels from the self-administered KSADS-COMP data

www.jaacap.org 9

14 June 2019 � 7:26 pm � ce


TABLE 2 Satisfaction Ratings With Self-Administered KSADS-COMP

Item

Youth Responses Parents Responses

Mean (SD)Percent Agree orStrongly Agree Mean (SD)

Percent Agree or StronglyAgree

1. I was comfortable answeringquestions on the computer

3.3 (0.7) 91% 3.6 (0.5) 99%

2. The questions were clearlystated and understandable

3.2 (0.9) 85% 3.5 (0.6) 94%

3. The computer did a good jobasking me about my feelings

3.2 (0.7) 90% 3.4 (0.6) 96%

4. I felt less embarrassedanswering these questions onthe computer than I wouldhave with a clinician

2.8 (1.0) 71% 2.6 (1.0) 54%

5. I found the computer interviewto be a helpful process to gothrough

3.2 (0.7) 89% 3.4 (0.6) 96%

Note: Overall, both parents and youths felt comfortable answering the questions via computer, found the questions clearly stated, and found theinterview a helpful process. Satisfaction rating scale: 1 ¼ strongly disagree, 2 ¼ disagree, 3 ¼ agree, 4 ¼ strongly agree. KSADS-COMP ¼ KiddieSchedule for Affective Disorders and Schizophrenia Computerized Version.

TOWNSEND et al.

10611062106310641065106610671068106910701071107210731074107510761077107810791080108110821083108410851086108710881089109010911092109310941095109610971098109911001101110211031104110511061107110811091110111111121113111411151116111711181119

11201121112211231124112511261127112811291130113111321133113411351136113711381139114011411142114311441145114611471148114911501151115211531154115511561157115811591160116111621163116411651166116711681169117011711172117311741175117611771178

generated by the clinical experience of the investigators.Receiver operating characteristic curve analyses were thenconducted to determine the accuracy of the multinomiallogistic models generated using the self-administeredKSADS-COMP data in predicting clinician-deriveddiagnoses.

RESULTS: STUDY TWO: VALIDATION OFPARENT AND YOUTH SELF-ADMINISTEREDVERSIONS OF KSADS-COMPAdministration Time Self-Administered and Clinician-Administered KSADS-COMPThe mean (SD) interview duration times for the parent andyouth self-report KSADS-COMP were 63.15 (38.3) mi-nutes and 50.92 (28.0) minutes, respectively. The self-administered KSADS-COMP was completed by 81% ofthe parents and 90% of the youths within 90 minutes. Theparent and youth portions of the clinician-administeredKSADS-COMP had mean (SD) duration times of 50.3(29.9) minutes and 41.5 (28.5) minutes, respectively, withthe combined parent and youth clinician-administeredKSADS-COMP completed in less than 1 hour for 31%of the dyads, less than 90 minutes for 59.4% of the dyads,less than 2 hours for 75.5% of the dyads, and less than 3hours for 95.3% of the dyads.

User SatisfactionBoth youths and parents expressed high satisfaction with thetechnical features of the self-administered KSADS-COMP;

10 www.jaacap.org


on the SUS, the mean parent rating was 90.0 (corre-sponding to “best possible”), and the mean youth rating was81.7 (between “good” and “excellent”). Similarly, the globalrating of “user friendliness” of the technology was high: 5.5(SD 1.2) for adolescents and 5.8 (SD 0.7) for parents (7-point scale: 1 ¼ worst possible, 2 ¼ awful, 3 ¼ poor,4 ¼ OK, 5 ¼ good, 6 ¼ excellent, 7 ¼ best imaginable).Ratings of parent and youth satisfaction with the KSADS-COMP are presented in Table 2. Overall, both parentsand youths felt comfortable answering the questions viacomputer, found the questions clearly stated, and found theinterview a helpful process. Among the youths, 85% statedthey were willing to be interviewed by computer again, andwhen asked if they would prefer to be asked these types ofquestions by computer or clinician after completing the self-administered KSADS-COMP, 54% said computer, 11%said clinician, and 35% had no preference. Among theparents, 99% (n ¼ 132) said they would be willing to beinterviewed again by computer. In terms of interviewpreference, 28% of the parents stated they preferred thecomputer, 22% stated they preferred a clinician, and 50%had no preference.

Convergent Validity DataTable 3 presents the means and standard deviations foryouth and parent report on the PHQ-9, BCMS, GAD-7,SWAN (ADHD and ODD), and PTSD measures foryouths who did and did not meet criteria for the corre-sponding current diagnoses generated by youth and parent


14 June 2019 � 7:26 pm � ce



11791180118111821183118411851186118711881189119011911192119311941195119611971198119912001201120212031204120512061207120812091210121112121213121412151216121712181219122012211222122312241225122612271228122912301231123212331234123512361237

1238123912401241124212431244124512461247124812491250125112521253125412551256125712581259

report on the self-administered KSADS-COMP. Wilcoxonsigned rank tests indicated that for all these disorders,youths who generated a positive diagnosis by parent reportscored significantly higher on the corresponding symptommeasure than youths who were not rated by their parents ashaving that diagnosis. The same pattern emerged for youth-generated diagnoses except for youth report of ODD.

Associations Between Self-Report KSADS-COMPDimensional Scales and Diagnostic Group Assignmentand Correlations Between KSADS-COMP DimensionalScales With Standardized MeasuresTable S3 (available online) provides means and standarddeviations for the KSADS-COMP dimensional scales bydiagnostic group assignment. For each disorder, youths with apositive current diagnosis scored significantly higher on thecorresponding dimensional scale than youths who did notmeet lifetime criteria for that diagnosis. All comparisons werestatistically significant for parent-rated and youth-rated di-agnoses. As depicted in Table 4, all KSADS-COMP youth-

TABLE 3 Scores on Standardized Clinical Measures by Current DSelf-Administered KSADS-COMP

Never, Mean (SD)Major depressive disorder(PHQ-9)Youth diagnosis 3.59 (5.35) (n [ 96)Parent diagnosis 3.80 (5.45) (n [ 88)

Bipolar 1 or 2 (BCMS)Youth diagnosis 3.81 (4.39) (n [ 113)Parent diagnosis 2.35 (3.48) (n [ 124)

Anxiety disorder (GAD-7)Youth diagnosis 1.82 (2.91) (n [ 82)Parent diagnosis 2.83 (4.89) (n [ 77)

ADHD (SWANeparent report)Youth diagnosis 9.59 (11.96) (n [ 87)Parent diagnosis 3.75 (6.05) (n [ 72)

ODD (SWANeparent report)Youth diagnosis 8.24 (8.97) (n [ 102)Parent diagnosis 3.77 (5.54) (n [ 74)

PTSD (PTSD-PC)Youth diagnosis .58 (1.03) (n [ 106)Parent diagnosis .81 (1.25) (n [ 101)

Note: Youths with positive diagnoses scored greater than youths without poconvergent validity of the diagnoses generated with the self-administered Kexception to this pattern of findings. Wilcoxon rank sum tests evaluated diffstandardized measures (PHQ-9, BCMS, GAD-7, SWAN, and PTSD scale.). Theself-report on the KSADS-COMP. The “Parent diagnosis” row represents dihyperactivity disorder; BCMS ¼ Brief Child Mania Rating Scale; GAD-7 ¼ 7-Administered Kiddie Schedule for Affective Disorders and Schizophrenia ComPrimary Care Screen for Posttraumatic Stress Disorder, DSM-5 version; PHQWeakness of ADHD Symptoms and Normal Behavior–parent report.



generated and parent-generated dimensional rating scales werealso significantly positively associated with their correspond-ing same informant standardized measure. Table S4 (availableonline) presents correlations between clinician-administeredand youth and parent self-administered KSADS-COMPdiagnostic specific dimensional rating scales, which likewiseshowed significant correlations.

Diagnostic Concordance Among InformantsTable 5 depicts the concordance between informants. Asindicated in the “Percent Negative Agreement” column in thetable, agreement between informants was greatest when adiagnosis was not present. Concordance was lower in ratingthe presence of each diagnosis, with the highest concordancebetween informants found in diagnosing ADHD. Gwet’sAC1 concordance ratings between diagnoses generated usingthe parent and youth self-administered KSADS-COMPranged from 0.76 to 0.89; Gwet’s AC1 concordance ratingsbetween diagnoses generated using the clinician and youthself-administered KSADS-COMP ranged from 0.80 to 0.91,

iagnostic Groups Determined by Youth and Parent Report on

Current, Mean (SD) Z p

7.65 (6.06) (n [ 20) L3.904 .016.38 (5.03) (n [ 21) L2.770 .01

8.35 (6.38) (n [ 20) L3.56 .016.20 (3.77) (n [ 15) L3.61 .01

12.40 (5.44) (n [ 10) L4.951 .016.13 (4.85) (n [ 15) L3.394 .01

14.69 (9.58) (n [ 29) L2.929 .0121.67 (9.65) (n [ 48) L8.33 .001

11.59 (9.09) (n [ 17) L1.50 .1316.29 (8.26) (n [ 49) L7.61 .001

3.08 (1.26) (n [ 13) L5.44 .0012.40 (1.34) (n [ 5) L2.77 .01

sitive diagnoses on each of the standard clinical rating scales, providingSADS-COMP. Youths who generated ODD diagnoses was the onlyerences between KSADS-COMP negative and positive groups on the“Youth diagnosis” row represents diagnostic groups generated by youthagnostic groups generated by parents. ADHD ¼ attention-deficit/item Generalized Anxiety Disorder Scale; KSADS-COMP ¼ Self-puterized Version; ODD ¼ oppositional defiance disorder; PC-PTSD-5 ¼-9 ¼ 9-item Patient Health Questionnaire; SWAN ¼ Strengths and

www.jaacap.org 11

14 June 2019 � 7:26 pm � ce

1260126112621263126412651266126712681269127012711272127312741275127612771278127912801281128212831284128512861287128812891290129112921293129412951296


TABLE 4 Spearman Rank Correlations Between KSADS-COMP Self-Report Dimensional Scales and StandardizedSymptom Measures

KSADS Depression Scale(3 items) PHQ-9 (9 items)Youth self-report .56**

KSADS Bipolar Scale (3 items) BCMS (10 items)Youth self-report .55**Parent report .61**

KSADS GAD Scale (1 item) GAD-7 (7 items)Youth self-report .51**

KSADS ADHD Scale (4 items) SWAN (18 items)Parent report .76**

KSADS ODD Scale (2 items) SWAN-ODD (12-items)Parent diagnosis .73**

KSADS PTSD Scale (3 items) PTSD-PC Total (4 items)Youth self-report .56**

Note: The brief self-administered KSADS-COMP diagnostic specificrating scales correlated significantly with all standard clinical ratingscales examined. ADHD ¼ attention-deficit/hyperactivity disorder;BCMS ¼ Brief Child Mania Rating Scale; GAD ¼ generalized anxietydisorder; GAD-7 ¼ 7-item Generalized Anxiety Disorder Scale; KSADS ¼Kiddie Schedule for Affective Disorders and Schizophrenia; KSADS-COMP ¼ Kiddie Schedule for Affective Disorders and SchizophreniaComputerized Version; PC-PTSD-5 ¼ Primary Care Screen for Post-traumatic Stress Disorder, DSM-5 version; PHQ-9 ¼ 9-item PatientHealth Questionnaire; SWAN ¼ Strengths and Weakness of ADHDSymptoms and Normal Behavior–parent report; SWAN-ODD ¼Strengths and Weaknesses of ADHD symptoms and Normal Behavior,Oppositional Defiant Symptoms–parent report.**p < .01.

TOWNSEND et al.

12971298129913001301130213031304130513061307130813091310131113121313131413151316131713181319132013211322132313241325132613271328132913301331133213331334133513361337133813391340134113421343134413451346134713481349135013511352135313541355

13561357135813591360136113621363136413651366136713681369137013711372137313741375137613771378137913801381138213831384138513861387138813891390139113921393139413951396139713981399140014011402140314041405140614071408140914101411141214131414

and Gwet’s AC1 concordance ratings between diagnosesgenerated using the clinician and parent self-administeredKSADS-COMP ranged from 0.86 to 0.94. The k valueswere consistently lower for all comparisons.

Predicting Clinician-Derived Diagnoses Using Youth andParent Self-Administered KSADS-COMP DataThe parent and youth self-administered KSADS-COMPitems used to predict clinician-administered KSADS-COMPdiagnoses are depicted in Table 6, together with the results ofthe receiver operating characteristic curve analyses conductedto determine the accuracy of the prediction models. Overallgood to excellent concordance was achieved between di-agnoses derived using the self-administered and clinician-administered KSADS-COMP when parent and youthself-administered KSADS-COMP data were integrated.56

DISCUSSIONResults from this initial validity study of the clinician-administered and self-administered versions of the KSADS-COMP are promising. As evidence of convergent validity,

12 www.jaacap.org


youths with current KSADS-COMP–generated mood, anxi-ety, ADHD, and ODD and CD diagnoses scored signifi-cantly higher on the relevant standardized measure for theparticular diagnosis than youths without that diagnosis. Thevalidity of the brief dimensional measures constructed fromKSADS-COMP screen items was also supported by signifi-cant differences on the scales between diagnosis-positive anddiagnosis-negative groups. There were also significant corre-lations between the brief KSADS-COMP dimensional ratingscales and established standardized clinical rating scales, whichwere higher when the informants on the measures were thesame on the KSADS-COMP and standardized rating scales(Table 4). Longitudinal data collection will be required todetermine if the brief KSADS-COMP diagnostic specificdimensional rating scales are useful in tracking treatmentresponse and symptoms over time.

There are currently no plans to compare the paper-and-pencil version of the KSADS with the clinician-administeredKSADS-COMP. Existing literature suggests that puttingpaper-and-pencil versions of clinician-administered diagnosticinterviews on an electronic platform improves reliability andvalidity substantially by reducing missing data and eliminatinghuman error in branching and choosing appropriate interviewquestions.23 In addition, clinician tallying when using paper-and-pencil versions of structured interviews has been foundto contribute to significant errors, and research has shown thatcomputerized versions of structured diagnostic instrumentsexceed the psychometric performance of their paper-and-pencil counterparts.24,25

There are three primary limitations to the currentinvestigation: the restriction of interrater reliability assess-ments to the items in the screen interview, restriction of thediagnoses present in the validation samples and the numberof youths who met criteria for each of the diagnosesexamined, and the somewhat extended period of time be-tween self-administered and clinician-administered KSADS-COMP assessments. Despite these limitations, the move toan electronic format and the other modifications made tothe KSADS offer many advantages over the paper-and-pencil version of the instrument.

For example, the clinician-administered KSADS-COMP addresses several limitations that have been notedpreviously with the paper-and-pencil version of the KSADS.One such limitation is that the interview can be excessivelytime-consuming.3,57 The mean administration time for thecombined parent and youth clinician-administered KSADS-COMP was 91.9 minutes, which is less time than haspreviously been reported for completing the paper-and-pencil version of the KSADS with only one informant.58

Administration time is reduced and the assessmentstreamlined by the youth and parent self-administered


14 June 2019 � 7:26 pm � ce


TABLE 5 Youth, Parent, and Clinician Concordance in Current Diagnoses (N ¼ 106)Parent and Youth Concordance

Current DiagnosisPercent

Agreement Cohen’s kGwet’sAC1

ParentDiagnosisFrequency

YouthDiagnosisFrequency

PercentPositive

Agreement

PercentNegative

AgreementMDD 82 .31 .76 22 21 43 89Bipolar spectrum 83 .16 .80 11 20 20 94Social anxiety 91 .27 .89 12 7 43 93GAD 86 .22 .84 16 11 36 91OCD 63 .14 .38 28 52 29 85ADHD 71 .32 .52 50 30 67 73ODD 66 .14 .47 50 17 59 67CD 81 .17 .76 18 18 28 89PTSD 89 .06 .87 5 13 8 97No diagnoses 73 .32 .56 41 35 NA NA

Clinician and Youth Concordance



ClinicianFrequency

YouthReport

Frequency

PercentPositive

Agreement

PercentNegative


Clinician and Parent Concordance



ClinicianReport

Frequency

ParentReport

Frequency

PercentPositive

Agreement

PercentNegative


Note: Consistent with prior research, considerable variability was noted across informants. Gwet’s AC1 is considered the most reliable concordancestatistic when the prevalence and marginal probability of diagnosis are low. AC1 ¼ first-order agreement coefficient; Q16ADHD ¼ attention-deficit/hyperactivity disorder; Bipolar spectrum ¼ bipolar 1, bipolar 2, and other specified bipolar disorder; CD ¼ conduct disorder; GAD ¼ generalizedanxiety disorder; OCD ¼ obsessive-compulsive disorder; ODD ¼ oppositional defiant disorder; PTSD ¼ posttraumatic stress disorder.


14151416141714181419142014211422142314241425142614271428142914301431143214331434143514361437143814391440144114421443144414451446144714481449145014511452145314541455145614571458145914601461146214631464146514661467146814691470147114721473

14741475147614771478147914801481148214831484148514861487148814891490149114921493149414951496149714981499150015011502150315041505150615071508150915101511151215131514151515161517151815191520152115221523152415251526152715281529153015311532

preinterview screen items, the automated branching andscoring, and the computer tracking of the supplements to becompleted.



Another limitation of the paper-and-pencil KSADS wasthe need for extensive clinician training to establish inter-rater reliability given that each symptom on the paper-and-

www.jaacap.org 13

14 June 2019 � 7:26 pm � ce


TABLE 6 Predicting Clinician Current Diagnoses Using Youth and Parent Self-Administered KSADS-COMP Data (N ¼ 106)Depression Bipolar Disorder ADHD ODDYouth KSADS Depression Scale; youth reportof suicidality; parent and youth adaptivefunctioning measures (eg, drop in grades,extracurricular activities, friendships);antidepressant medication

Youth report decreasedneed for sleep; youthreport elation; familyhistory of bipolar

disorder; mood stabilizeror atypical antipsychotic;inpatient hospitalization

Parent KSADS ADHDScale; age of ADHD

onset; ADHDmedication; GAD

diagnosis

Parent KSADS ODD Scale;parent report ofsuspensions anddetentions; GAD

diagnosis; criterion Atrauma history

AUC [ 0.877 (p < .001) AUC [ 1.00 (p < .001) AUC [ 0.977 (p < .001) AUC [ 0.913 (p < .001)Sensitivity [ 0.94 Sensitivity [ 1.00 Sensitivity [ 0.92 Sensitivity [ 0.92Specificity [ 0.67 Specificity [ 1.00 Specificity [ 0.91 Specificity [ 0.91

Note: Multinomial logistic regression analyses were conducted to derive weights for integrating parent and youth data from the self-administeredKSADS-COMP to predict diagnoses derived from the clinician-administered KSADS-COMP, with the items selected for entry in the regression modelsgenerated by the clinical experience of the investigators. Overall good to excellent concordance was achieved between diagnoses derived using theself-administered and clinician-administered KSADS-COMP when parent and youth self-administered KSADS-COMP data were integrated. AUC ¼area under the curve; ADHD¼attention-deficit/hyperactivity disorder; GAD ¼ generalized anxiety disorder; KSADS ¼ Kiddie Schedule for AffectiveDisorders and Schizophrenia; KSADS-COMP ¼ Kiddie Schedule for Affective Disorders and Schizophrenia Computerized Version; ODD ¼ opposi-tional defiant disorder.

TOWNSEND et al.

15331534153515361537153815391540154115421543154415451546154715481549155015511552155315541555155615571558155915601561156215631564156515661567156815691570157115721573157415751576157715781579158015811582158315841585158615871588158915901591

15921593159415951596159715981599160016011602160316041605160616071608160916101611161216131614161516161617161816191620162116221623162416251626162716281629163016311632163316341635163616371638163916401641164216431644164516461647164816491650

pencil version of the KSADS was assessed using uniquerating criteria. The uniform rating scale used to assess allcurrent symptoms in the KSADS-COMP (eg, not at all,rarely, several days, more than half the days, and nearlyevery day) and the automated branching features of theKSADS-COMP reduce the need for such training, whilestill allowing clinicians the flexibility to use their clinicaljudgment in probing and rating symptoms. The uniformrating scale for assessing current symptoms likely also ac-counts for the excellent interrater reliability in scoring items.

There are multiple additional features that render theclinician-administered KSADS-COMP more feasible than itspaper-and-pencil predecessor for routine clinical practice. Forexample, the “choose as you go” modular format of theKSADS-COMP allows clinicians to select a subset of modulesof interest rather than completing the entire interview. Theself-administered preinterview screen items of the clinician-administered KSADS-COMP can inform module selection,and the unstructured introductory interview provides anexcellent initial assessment of adaptive functioning that lendsgreater confidence to the selection of the subset of modules tobe administered as well as providing other relevant informa-tion needed for clinical reports (eg, family, school, treatmenthistory). The availability of diagnostic reports in real timefurther addresses efficiency concerns and allows clinicians toprovide meaningful feedback to children and families in atimely fashion.

When comparing youth-generated, parent-generated, andclinician-generated diagnoses derived with the self-administered and clinician-administered KSADS-COMPs,consistent with research findings in the field, there was a lack

14 www.jaacap.org


of strong concordance between informants, with the concor-dance observed in this investigation comparable to or betterthan that observed in prior studies.59-62 Prior studies havereported parent-child k values in diagnosing depressive dis-orders of 0.09 or less61,62 compared with the k value of 0.31 indiagnosing MDD observed in the current study. Parent-childconcordance in rating the other major diagnoses was essen-tially comparable to that observed in prior investigations.61,62

Diagnostic concordance between the parent and clinician werehigher than between the parent and child across all diagnoses,with youth and clinician concordance highest for social anx-iety and PTSD. Overall, agreement between informants onthe self-administered and clinician-administered KSADS-COMP was highest when a diagnosis was not present.

Ultimately, in clinical practice in making treatmentdecisions, cross-informant variance needs to be reconciledand, to date, relies on clinical judgment to do this. Thereceiver operating characteristic curve analyses reported inthis article provide proof of concept that parent and youthdata from the self-administered KSADS-COMP can beintegrated and used to derive diagnoses with good toexcellent concordance with clinician-derived diagnoses.However, further refinement, replication, and validation ofthe models used to integrate parent and youth self-reportdata to generate diagnoses similar to clinician-derived di-agnoses are required in larger scale representative samples.At the present time, in treatment settings, such as busyemergency departments, the diagnostic information attainedwith the self-report KSADS-COMP can best be used toexpedite evaluations and help clinicians finalize diagnosticimpressions.


14 June 2019 � 7:26 pm � ce


Q5

Q17

Q6


16511652165316541655165616571658165916601661166216631664166516661667166816691670167116721673167416751676167716781679168016811682168316841685168616871688168916901691169216931694169516961697169816991700170117021703170417051706170717081709

17101711171217131714171517161717171817191720172117221723172417251726172717281729173017311732173317341735173617371738173917401741174217431744174517461747174817491750175117521753175417551756175717581759176017611762

Whereas there are validated internet-based mental healthscreens for adolescents,63 unvalidated diagnostic internet-basedassessment tools available for purchase,64 preliminary workthat has been conducted on the development and validation ofthe internet and voice Diagnostic Interview Schedule forChildren for DSM-IV,65,66 and more extensive workcompleted on the Development and Well-Being Assessment(DAWBA) instruments,67-69 there are many features that areunique to the three KSADS-COMP instruments that enhancetheir utility. To the best of our knowledge, the three versions ofthe KSADS-COMP are the only computer-administered childand adolescent psychiatric diagnostic interviews that use in-formation attained in the introductory interview to guideprobing of symptoms (eg, information about bullying to guidequestions generated when probing about paranoid ideation)and the only assessment tools to include a screen interview thatprovides a comprehensive diagnostic overview to facilitatedifferential diagnoses before surveying the full range of symp-toms associated with the different diagnoses. The clinician-administered KSADS-COMP is also the only computerizeddiagnostic interview that includes a parent and youth self-report preassessment to streamline interviewing and the onlytool to give the clinician access to the preinterview responsesand the responses of the other informant (eg, teen) whenconducting the interview (Figure 1). The youth self-reportKSADS-COMP is also the only psychiatric diagnostic instru-ment with video clips to facilitate administration. To date,Spanish, Dutch, and Danish translations of the KSADS-COMP instruments have been produced, with automatedmethods developed to create future translations.

Beyond determining categorical psychiatric diagnoses,there is growing interest in the field since the initiation of theNational Institute ofMental Health Research Domain Criteria(RDoC) program in using dimensional assessments that mapmore clearly onto distinct neural circuits rather than hetero-geneous categorical diagnoses.70 With the diagnostic-specificdimensional scales included within the KSADS-COMP andplans to create KSADS-COMP transdiagnostic rating scalesand refine the algorithms to integrate parent and youth self-report data to derive categorical diagnoses that more closelyapproximate clinician diagnoses, the KSADS-COMP mayhelp serve as a bridge between DSM and RDoC diagnosticperspectives.

The RDoC, however, is a research framework withthe goal of generating the necessary database to helpderive a new psychiatric nomenclature informed by



neuroscience, genetics, and psychology.70 In the interim,clinicians are required to generate DSM or ICD-10 di-agnoses for assessment, treatment, and billing purposes.The clinician-administered KSADS-COMP shows utilitynot only for research but also for implementation inclinical settings, with the self-report preinterview ratings,choose-as-you-go module options, and automated scoringto streamline assessments and shorten administrationtime. The self-administered versions of KSADS-COMPhave numerous potential research and clinical applica-tions, including use in large-scale epidemiological studies,in schools and busy emergency departments, and in tel-ehealth to address the critical shortage of child andadolescent mental health specialists in many areas of theUnited States.

1

Accepted May 13, 2019.

Drs. Townsend, Kearney, and Kaufman are with the Kennedy Krieger Institute,Baltimore, MD. Drs. Townsend and Kearney are also with Johns HopkinsSchool of Medicine, Baltimore, MD. Dr. Kobak and Ms. Deep are with Centerfor Telepsychology, Madison, WI. Drs. Milham, Andreotti, Escalera and Ms.Alexander are with Child Mind Institute, New York. Dr. Milham is also withNathan Kline Institute, Orangeburg, NY. Dr. Birmaher and Mss. Gill, Sylvester,and Rice are with Western Psychiatric Institute and Clinic, University of Pitts-burgh School of Medicine, PA.

KSADS-COMP Access Information: To test a demo or obtain access to theKSADS-COMP diagnostic instruments go to ��.This work was funded by the National Institutes of Health/National Institute onDrug Abuse (NIH/NIDA; grant R44 MH094092 to Drs. Kaufman and Kobak) andthe National Institute of Mental Health (NIMH; grant MH060952 toDr. Birmaher), with additional support provided by the Zanvyl and IsabelleKrieger Fund (Dr. Kaufman).

Dhananjay Vaidya, PhD, of Johns Hopkins School of Medicine, served as thestatistical expert for this research.

The authors extend appreciation to the consultants who contributed to thisinstrument, including Deanna M. Barch, PhD, of Washington University; MarcoGrados, MD, of Johns Hopkins School of Medicine; Daniel Hoover, PhD, ofKennedy Krieger Institute; Ellen Leibenluft, MD, of National Institute of MentalHealth; Danny Pine, MD, of National Institute of Mental Health; Kenneth Sher,PhD, of University of Missouri; and Susan F. Tapert, PhD, of University ofCalifornia San Diego.

Disclosure: Dr. Kobak may in the future receive royalties from KSADSCOMP,LLC. Dr. Milham has received grant funding from NIH. Dr. Birmaher hasreceived grant funding from NIH. Dr. Kaufman has received grant funding fromNIH, has served as a consultant for Pfizer and Otsuka Pharmaceuticals, and mayin the future receive royalties from KSADSCOMP, LLC. Drs. Townsend,Andreotti, and Escalera and Mss. Kearney, Alexander, Gill, Sylvester, Rice, andDeep report no biomedical financial interests or potential conflicts of interest.

Correspondence to Joan Kaufman, PhD, Center for Child and Family TraumaticStress, Kennedy Krieger Institute, John Hopkins School of Medicine, 1741Ashland Avenue, Room 434, Baltimore, MD 21205; e-mail: [email protected]

0890-8567/$36.00/ª2019 Published by Elsevier Inc. on behalf of the AmericanAcademy of Child and Adolescent Psychiatry.

https://doi.org/10.1016/j.jaac.2019.05.009

17631764

REFERENCES

17651766

1. Kaufman J, Birmaher B, Brent D, et al. Schedule for Affective Disorders andSchizophrenia for School-Age Children–Present and Lifetime Version (K-SADS-

PL): initial reliability and validity data. J Am Acad Child Adolesc Psychiatry. 1997;36:980-988.

www.jaacap.org 15

4 June 2019 � 7:26 pm � ce

17671768

mailto:[email protected]:[email protected]://doi.org/10.1016/j.jaac.2019.05.009http://www.jaacap.org

Q14

TOWNSEND et al.

17691770177117721773177417751776177717781779178017811782178317841785178617871788178917901791179217931794179517961797179817991800180118021803180418051806180718081809181018111812181318141815181618171818181918201821182218231824182518261827

1828182918301831183218331834183518361837183818391840184118421843184418451846184718481849185018511852185318541855185618571858185918601861186218631864186518661867186818691870187118721873187418751876187718781879188018811882188318841885

2. Endicott J, Spitzer RL. A diagnostic interview: the schedule for affective disorders andschizophrenia. Arch Gen Psychiatry. 1978;35:837-844.

3. Ambrosini PJ. Historical development and present status of the schedule for affectivedisorders and schizophrenia for school-age children (K-SADS). J Am Acad Child AdolescPsychiatry. 2000;39:49-58.

4. Robb AS, Cueva JE, Sporn J, Yang R, Vanderburg DG. Sertraline treatment of childrenand adolescents with posttraumatic stress disorder: a double-blind, placebo-controlledtrial. J Child Adolesc Psychopharmacol. 2010;20:463-471.

5. Findling RL, Robb A, Nyilas M, et al. A multiple-center, randomized, double-blind,placebo-controlled study of oral aripiprazole for treatment of adolescents with schizo-phrenia. Am J Psychiatry. 2008;165:1432-1441.

6. Findling RL, Pathak S, Earley WR, Liu S, DelBello MP. Efficacy and safety of extended-release quetiapine fumarate in youth with bipolar depression: an 8 week, double-blind,placebo-controlled trial. J Child Adolesc Psychopharmacol. 2014;24:325-335.

7. Findling RL, Nyilas M, Forbes RA, et al. Acute treatment of pediatric bipolar I disorder,manic or mixed episode, with aripiprazole: a randomized, double-blind, placebo-controlled study. J Clin Psychiatry. 2009;70:1441-1451.

8. Biederman J, Krishnan S, Zhang Y, McGough JJ, Findling RL. Efficacy and tolerabilityof lisdexamfetamine dimesylate (NRP-104) in children with attention-deficit/hyperactivity disorder: a phase III, multicenter, randomized, double-blind, forced-dose,parallel-group study. Clin Ther. 2007;29:450-463.

9. Schulz E, Fleischhaker C, Hennighausen K, et al. A double-blind, randomized, placebo/active controlled crossover evaluation of the efficacy and safety of Ritalin LA in childrenwith attention-deficit/hyperactivity disorder in a laboratory classroom setting. J ChildAdolesc Psychopharmacol. 2010;20:377-385.

10. Santisteban JA, Stein MA, Bergmame L, Gruber R. Effect of extended-release dexme-thylphenidate and mixed amphetamine salts on sleep: a double-blind, randomized,crossover study in youth with attention-deficit hyperactivity disorder. CNS Drugs. 2014;28:825-833.

11. Dell’Agnello G, Maschietto D, Bravaccio C, et al. Atomoxetine hydrochloride in thetreatment of children and adolescents with attention-deficit/hyperactivity disorder andcomorbid oppositional defiant disorder: a placebo-controlled Italian study. Eur Neuro-psychopharmacol. 2009;19:822-834.

12. Emslie GJ, Mayes T, Porta G, et al. Treatment of Resistant Depression in Adolescents(TORDIA): week 24 outcomes. Am J Psychiatry. 2010;167:782-791.

13. Brent D, Emslie G, Clarke G, et al. Switching to another SSRI or to venlafaxine with orwithout cognitive behavioral therapy for adolescents with SSRI-resistant depression: theTORDIA randomized controlled trial. JAMA. 2008;299:901-913.

14. Fluvoxamine for the treatment of anxiety disorders in children and adolescents. TheResearch Unit on Pediatric Psychopharmacology Anxiety Study Group. N Engl J Med.2001;344:1279-1285.

15. Runyon MK, Deblinger E, Steer RA. Group cognitive behavioral treatment for parentsand children at-risk for physical abuse: an initial study. Child Fam Behav Ther. 2010;32:196-218.

16. Kessler RC, Avenevoli S, Green J, et al. National comorbidity survey replicationadolescent supplement (NCS-A): III. Concordance of DSM-IV/CIDI diagnoses withclinical reassessments. J Am Acad Child Adolesc Psychiatry. 2009;48:386-399.

17. Findling RL, Youngstrom EA, Fristad MA, et al. Characteristics of children with elevatedsymptoms of mania: the Longitudinal Assessment of Manic Symptoms (LAMS) study.J Clin Psychiatry. 2010;71:1664-1672.

18. Axelson D, Birmaher B, Strober M, et al. Phenomenology of children and adolescentswith bipolar spectrum disorders. Arch Gen Psychiatry. 2006;63:1139-1148.

19. Lauth B, Levy SR, Juliusdottir G, Ferrari P, Petursson H. Implementing the semi-structured interview Kiddie-SADS-PL into an in-patient adolescent clinical setting:impact on frequency of diagnoses. Child Adolesc Psychiatry Ment Health. 2008;2:14.

20. Sorensen MJ, Nissen JB, Mors O, Thomsen PH. Age and gender differences indepressive symptomatology and comorbidity: an incident sample of psychiatricallyadmitted children. J Affect Disord. 2005;84:85-91.

21. Jensen-Doss A, Youngstrom EA, Youngstrom JK, Feeny NC, Findling RL. Predictorsand moderators of agreement between clinical and research diagnoses for children andadolescents. J Consult Clin Psychol. 2014;82:1151-1162.

22. Miller PR. Inpatient diagnostic assessments: 2. Interrater reliability and outcomes ofstructured vs. unstructured interviews. Psychiatry Res. 2001;105:265-271.

23. Kobak KA, Greist JH, Jefferson JW, Katzelnick DJ. Computer-administered clinicalrating scales. A review. Psychopharmacology (Berl). 1996;127:291-301.

24. Kobak KA, Taylor LH, Dottl SL, et al. A computer-administered telephone interview toidentify mental disorders. JAMA. 1997;278:905-910.

25. Brodey BB, First M, Linthicum J, Haman K, Sasiela JW, Ayer D. Validation of theNetSCID: an automated web-based adaptive version of the SCID. Compr Psychiatry.2016;66:67-70.

26. Kaufman J, Birmaher B, Axelson D, Pereplitchikova F, Brent D, Ryan N. The KSADS-PL DSM-5. Baltimore, MD: Kennedy Krieger Institute; 2016.

16 www.jaacap.org


27. Posner K, Oquendo MA, Gould M, Stanley B, Davies M. Columbia Classification Al-gorithm of Suicide Assessment (C-CASA): classification of suicidal events in the FDA’spediatric suicidal risk analysis of antidepressants. Am J Psychiatry. 2007;164:1035-1043.

28. Alexander LM, Escalera J, Ai L, et al. An open resource for transdiagnostic research inpediatric mental health and learning disorders. Sci Data. 2017;4:170181.

29. Angold A, Costello EJ, Messer S, Pickles A, Winder F, Silver D. Development of a shortquestionnaire for use in epidemiological studies of depression in children and adolescents.Int J Methods Psychiatr Res. 1995;5:237-249.

30. Thapar A, McGuffin P. Validity of the shortened Mood and Feelings Questionnaire in acommunity sample of children and adolescents: a preliminary research note. PsychiatryRes. 1998;81:259-268.

31. Kuo ES, Stoep AV, Stewart DG. Using the short Mood and Feelings Questionnaire todetect depression in detained adolescents. Assessment. 2005;12:374-383.

32. Daviss WB, Birmaher B, Melhem NA, Axelson DA, Michaels SM, Brent DA. Criterionvalidity of the Mood and Feelings Questionnaire for depressive episodes in clinic andnon-clinic subjects. J Child Psychol Psychiatry. 2006;47:927-934.

33. Kent L, Vostanis P, Feehan C. Detection of major and minor depression in children andadolescents: evaluation of the Mood and Feelings Questionnaire. J Child Psychol Psy-chiatry. 1997;38:565-573.

34. Birmaher B, Khetarpal S, Brent D, et al. The Screen for Child Anxiety Related EmotionalDisorders (SCARED): scale construction and psychometric characteristics. J Am AcadChild Adolesc Psychiatry. 1997;36:545-553.

35. Birmaher B, Brent DA, Chiappetta L, Bridge J, Monga S, Baugher M. Psychometricproperties of the Screen for Child Anxiety Related Emotional Disorders (SCARED): areplication study. J Am Acad Child Adolesc Psychiatry. 1999;38:1230-1236.

36. Muris P, Merckelbach H, Ollendick T, King N, Bogie N. Three traditional and threenew childhood anxiety questionnaires: their reliability and validity in a normal adolescentsample. Behav Res Ther. 2002;40:753-772.

37. Hughes CW. Child and Adolescent Measures for Diagnosis and Screening. Washington,DC: American Psychiatric Association; 2008.

38. Achenbach TM, Rescorla LA. Manual for ASEBA School-Age Forms and Profiles.Burlington, VT: University of Vermont, Research Center for Children, Youth, &Families; 2001.

39. Nakamura BJ, Ebesutani C, Bernstein A, Chorpita BF. A psychometric analysis of theChild Behavior Checklist DSM-Oriented Scales. J Psychopathol Behav Assess. 2009;31:178-189.

40. Lucas CP, Fisher P, Piacentini J, et al. Features of interviews questions associated withattenuation of symptom reports. J Abnorm Child Psychol. 1999;27:429-437.

41. Bangor A, Kortum P, Miller J. Determining what individual SUS scores mean: adding anadjective rating scale. J Usability Stud. 2009;4:114-123.

42. Brooke J. SUS: a ‘quick and dirty’ usability scale. In: Jordan PW, Thomas B,Weerdmeester BA, McClelland IL, eds. Usability Evaluation in Industry. London: Taylor& Francis; 1996:189-194.

43. Bangor A, Kortum P, Miller J. The system usability scale (SUS): an empirical evaluation.International Journal of Human-Computer Interaction. 2008;24:574-592.

44. Kobak KA, Reynolds WM, Griest JH. Computerized and clinician assessment ofdepression and anxiety: respondent evaluation a

Documents

Development of Three Web-Based Computerized Versions of ... Validation Paper.pdf[CD], ODD), eating and elimination disorders (enuresis, encopresis, anorexia nervosa, bulimia, binge