Journal of Fluency Disorders 33 (2008) 167–179

The effect of stuttering measurement training on judging stutteringoccurrence in preschool children who stutter

Jóhanna Einarsdóttir a,∗, Roger J. Ingham b

a University of Iceland, Reykjavik, Icelandb University of California, Santa Barbara, United States

Received 16 December 2007; received in revised form 22 May 2008; accepted 28 May 2008

Abstract

Purpose: To evaluate the efficacy of a standardized training program to improve preschool teachers’ ability to identify occurrencesof stuttering accurately and reliably in preschool children who stutter (CWS).

Method: An Icelandic version of the Stuttering Measurement Assessment and Training (SMAAT) program [Ingham, R. J.,Cordes, A. K., Kilgo, M., & Moglia, R. (1998). Stuttering measurement assessment and training (SMAAT). Santa Barbara, CA:University of California, Santa Barbara] was developed using 2-min audio-visual recordings of nine Icelandic-speaking CWS(3–5 years). Twenty preschool teachers from preschools in Iceland volunteered to participate and were randomly allocated to anexperimental and control group. The preschool teachers judged stuttering on nine pre-judged, interval-classified speech samples ontwo occasions 2–3 weeks apart; only the experimental group received judgment training between Occasion 1 and 2.

Results: The experimental and control groups displayed, on average, above 80% accuracy in identifying stuttering in the sampledCWS, even prior to training. After training the experimental group showed significantly higher mean percent judgment accuracy(p < .01), while the control group’s mean accuracy level showed no significant difference between Occasion 1 and 2.

Conclusions: The interval-based training program did improve the accuracy with which the judges identified stuttering inpreschool CWS. However, the findings also showed that the preschool teachers possessed relatively satisfactory stuttering judgmentskills, even without training.

Educational objectives: The reader will be able to (a) explain the development of a procedure for establishing standardized trainingmaterial for measuring and identifying stuttering in preschool children, (b) evaluate whether preschool teachers are generally accuratejudges of stuttering in young children, and (c) describe how their stuttering judgment accuracy can be improved through training.© 2008 Elsevier Inc. All rights reserved.

Keywords: Stuttering; Measurement training; Preschool children

1. Introduction

A standardized system for training judges to identify stuttering has been developed for use by clinicians working withadults and adolescents who stutter (PWS). Known as the Stuttering Measurement Assessment and Training (SMAAT)

∗ Corresponding author.E-mail address: jeinars@hi.is (J. Einarsdóttir).

0094-730X/$ – see front matter © 2008 Elsevier Inc. All rights reserved.doi:10.1016/j.jfludis.2008.05.004

168 J. Einarsdóttir, R.J. Ingham / Journal of Fluency Disorders 33 (2008) 167–179

program (Ingham, Cordes, Kilgo, & Moglia, 1998), it is based on the use of time interval measurement methodology(for clarity the original SMAAT program is hereafter referred to as SMAAT-adult). In a series of investigations theSMAAT-adult program has been shown to improve the accuracy and interjudge agreement levels among untrainedjudges when identifying stuttering in adults and adolescents (Cordes & Ingham, 1994b; Cordes & Ingham, 1999;Ingham, Cordes, & Gow, 1993). However, no reliable standardized system currently exists for identifying stuttering inyoung children who stutter (CWS) or preschoolers.1 An unpublished investigation by Bothe (personal communication)used the SMAAT-adult program to train clinicians to identify stuttering in young children. But Bothe reported thatthis program was less successful in improving clinician judgments with CWS than it was when used with PWS. In theabsence of a standardized stuttering judgment training program it is possible, therefore, that researchers and clinicianscannot be confident that what they choose to label as stuttering in a child’s speech will be the same as what otherresearchers or clinicians label as stuttering.

1.1. The importance of accurate stuttering identification in young children

There is a clinically important reason for identifying stuttering in preschoolers. It has been shown that young CWSrespond well to direct intervention, thereby helping to prevent the disorder from developing into a more intractablechronic form (Bothe, Davidow, Bramlett, & Ingham, 2006; Onslow, Packman, & Harrison, 2003). Delayed treatment(more than 15 months after the onset of stuttering) seems to be less effective than treatment that begins earlier—evenallowing for a high level of unassisted recovery during the first months after its onset (Ingham & Cordes, 1999;Yairi, Ambrose, Paden, & Throneburg, 1996). And stuttering treatment with children over 6 years of age has beendocumented to be substantially less effective than when applied with younger CWS (Ingham & Cordes, 1999). Clearly,early treatment depends on early identification. It is rather obvious, therefore, that it would be useful if professionalswho routinely interact with young children were able to identify early stuttering accurately and reliably.

1.2. Time interval measurement of stuttering

Time interval measurement is one of many methods used for measuring behavior (see Baer, Wolf, & Risley, 1987)and one that was originally proposed by Ingham (1990) as way to improve stuttering measurement. Its distinguishingfeature is that it measures behavior by recording whether a target or problem behavior either does or does not occurduring predetermined time intervals. The method was essentially introduced to stuttering research by Cordes andIngham (1994a) in order to improve the reliability of stuttering judgments and, by extension, the measurement ofstuttering (Cordes & Ingham, 1994a; Cordes & Ingham, 1994b; Cordes & Ingham, 1994c; Cordes & Ingham, 1995;Cordes & Ingham, 1996; Cordes & Ingham, 1999; Cordes, Ingham, Frank, & Ingham, 1992; Ingham & Cordes, 1997a;Ingham et al., 1993). The method relies on real time computer-recorded stuttering event judgments. Parts of speechare judged as either stuttered or nonstuttered by an observer who depresses a computer mouse button for the durationof each perceived stuttering event. However, the time when the mouse button is not pressed is, by default, regarded asnonstuttered speech. Those button presses are then automatically detected (and recorded) as occurring within and/oracross pre-classified intervals (usually 5 s) that are virtually superimposed over the sample recording by customizedsoftware. Interval measurement can be used to measure both the frequency and severity of stuttering (as reflected in theduration of stuttering events). Severe moments of stuttering (events that may have a long duration) will likely extendacross more than one interval. More importantly, interval measurement can also be used for training purposes. Thejudgments of pools of appropriate judges on select samples can be recorded and stored within computer memory. Thatrecord can then be used, via appropriate software, as a source for signaling in real time when judgments are “correct”or “incorrect.” Of course, that record can also be used to accumulate a library of judgments for clinical and researchpurposes.

The first systematic investigation of interval-based stuttering measurement was conducted by Cordes et al. (1992).Three groups of judges, inexperienced students, experienced students and experienced professionals judged stutteringevents in five 1-min speech samples over 12 repeated presentations of the samples. Results showed, rather predictably,that repeated sample exposure and judging experience interacted to produce improved inter- and intra-judge total count

1 Dr. Anne Bothe at the University of Georgia is currently developing a SMAAT-child program for American-English CWS.

J. Einarsdóttir, R.J. Ingham / Journal of Fluency Disorders 33 (2008) 167–179 169

agreement. Later studies showed that inexperienced judges (undergraduate students) could be trained to be highlyaccurate (above 90%) inter- and intra-judge agreement by using time interval judgments (Cordes & Ingham, 1994a;Cordes & Ingham, 1994b; Cordes & Ingham, 1994c; Cordes & Ingham, 1996; Ingham et al., 1993).

The preceding research led ultimately to the development of the SMAAT-adult program for adolescents and adults(Ingham et al., 1998). This computer-based training program was developed on the basis of stuttering judgments madeby 10 recognized authorities in the field of stuttering treatment and research. Their judgments were incorporated in theSMAAT-adult program which included nine 2-min audio-visually recorded spontaneous speech samples. The efficacyof the program was established by Cordes and Ingham (1999) in a study using inexperienced university students. Thisstudy, which used repeated assessments of speech samples before and after training in a multiple baseline experimentaldesign, basically confirmed the following findings from the earlier studies: (1) inexperienced judges could be trained torecognize stuttering more accurately in a speech sample, and (2) their mean interval-by-interval interjudge agreementcould be improved from around 80–85% before training to about 90–91% after training.

The present study used the methodology developed for the SMAAT-adult program in order to produce a versionof the program suitable for use with Icelandic CWS, and to test whether the program’s effects were replicable whenused with individuals responsible for the welfare of young Icelandic children. To test the efficacy of the SMAAT-childpreschool teachers were selected to be judges because they had daily contact with young children, but were not expectedto have special expertise about stuttering. Preschool teachers also play an important role; they refer possible CWS tospeech-language pathologists (SLPs) and should, therefore, have some basic awareness of the disorder. Studies haveshown lack of information on stuttering among preschool teachers (Carey, Block, Ross, Borg, & O’Halloran, 2008)and educational professionals (Christie, 2000). On the other hand training and information on stuttering have beenshown to alter referral rates among primary care professionals (Christie, 2000; Lees, Stark, Birse, & Nicoll, 2004).

Preschool teachers have served as one source of referral for many studies on preschool CWS (see, for example, Yairi& Ambrose, 1999; Yairi, Ambrose, & Niermann, 1993; Zebrowski & Conture, 1989). Preschools in Iceland are partof the national educational system which means that almost every Icelandic child attends preschool (Statistics Iceland,2007). The Ministry of Education Science and Culture provides National Curriculum Guidelines for the schools andall preschool teachers graduate from a 3-year University level course (Menntamálaráðuneytið, 2006). Their trainingincludes courses on developmental disorders, learning difficulties or behaviors that might impact normal development.Consequently, the accuracy with which these teachers can recognize stuttering – and thereby decide whether to referthat child for therapy services – is important. Nonetheless, there is no information as to just how accurately preschoolteachers are able to identify stuttering in preschool CWS, or if training would improve their identification skills. And ofcourse it is not known whether a standardized training program can be developed that effectively improves the accuracyof identifying stuttering occurrences in young children. Therefore this study was designed to address the followingquestions:

(1) Is it possible to develop and evaluate a standardized system for identifying and measuring stuttering in Icelandic3–5-year-old CWS?

(2) How accurate are preschool teachers in assessing stuttering in 3–5-year-old CWS?(3) Does training with SMAAT-child program improve the accuracy with which preschool teachers identify stuttering?

2. Method

Hence, this study was designed to develop standardized training material and to test the effect of the trainingprogram on preschool teachers. As mentioned earlier, the training material used in this study was an Icelandic versionof a software system known as SMAAT or SMAAT-adult (Ingham et al., 1998). The system uses Icelandic preschoolCWS and is known hereafter as SMAAT-child (Ingham et al., 2006).

2.1. Judges

Judges for this study were 20 female preschool teachers selected from 10 Icelandic preschools; nine were located inReykjavík and one in the neighboring city Kópavogur. Their ages ranged from 27 to 58 years (mean = 39 years) and allreported normal hearing and vision. Their preschool teaching experience ranged from 0 to 36 years (mean = 11 years).All except one teacher reported having encountered CWS in their classroom; the rest reported having had between one

170 J. Einarsdóttir, R.J. Ingham / Journal of Fluency Disorders 33 (2008) 167–179

and five CWS in their classrooms. After completing consent forms and providing professional background information,they were randomly allocated to an experimental group (mean age 39 years) and a control group (mean age 39.9 years).

2.2. Materials

2.2.1. The development of the SMAAT-child system in IcelandThe stimulus material for the SMAAT-child program was developed using speech samples selected from recordings

of 20 Icelandic preschool CWS. The criteria for participant inclusion were age (3–5 years) and that both parents andthree SLPs (including the first author) agreed that the child had a stuttering problem. Only stuttering during Icelandicspeech was investigated. Bilingual children2 and children with an obvious neurological disorder or severe languageimpairment were not included.

2.2.2. Speech sample recording procedureThe children were audio-visually recorded in clinic conditions (one child was recorded in his home). Recordings

were obtained using a digital camera (Canon Elura 65) equipped with a “shotgun” microphone (directional stereomicrophone-DM 50). Attempts were made to ensure that the recordings contained a representative sample of eachchild’s speech and exemplars of the child’s stuttering. The child sat on a chair at a table so as to minimize unnecessarymovement. Each recording showed only the child’s face and shoulders while speaking and playing with different toys.The duration of each recording ranged from 30 to 120 min, depending on the amount of the child’s speech. Everythingthat each child said was later orthographically transcribed. One minute samples were selected during which the childmostly used connected speech and where interruptions by the investigator and/or parent were minimal. Each selectedrecording was then edited using the Windows Movie Maker® program. Seven 1-min speech samples were selected fromthe pool of samples from each of the 20 children so that the child was constantly within the video frame, speaking insentences and displaying SLP-judged stuttering. These speech samples (7 min from 20 children, or 140 1-min samples)formed the bases for further development of the standardized system.

2.2.3. The selection of speech samplesTwo SLPs and the first author viewed the 20 videotapes individually. They identified at least one unambiguous

stuttering on each videotape and made a global rating of the child’s stuttering severity on a 1–9-point scale thatintegrated perceptual (speech) and visual (struggle) behaviors (Lewis & Sherman, 1951; O’Brian, Packman, Onslow,& O’Brian, 2004). Spearman’s rho correlations among the three SLPs by judging each child’s stuttering severity onthe 9-point severity scale were generally high (rho mean = 0.86 (0.93 (SLP 1 and SLP 2), 0.86 (SLP 1 and SLP 3),and 0.81 (SLP 2 and SLP 3))). Based on their scale score, a child was allocated to one of three groups: mild (1–3),moderate (4–6) or severe (7–9). The three SLPs then chose recordings from nine children (the same number of samplesas in the SMAAT-adult program) that they agreed showed a wide range of different characteristics and severities ofstuttering. This resulted in samples from three females and six males, with three samples displaying mild stuttering,three moderate stuttering and three severe stuttering (see Appendix A).

2.2.4. Interval configuring of the speech samplesFollowing the Cordes and Ingham (1995) procedure, each of the nine 7-min recordings was partitioned into 5 second

(s) intervals with a 5-s silent interval inserted after each speaking interval so as to permit time to make an observerjudgment. The resulting 756 5-s intervals (84 from each child) constituted the judgment stimuli. The intervals wereconverted to a DVD format and organized in two different randomized orders for each judgment occasion. The DVDrecordings required approximately 126 min of judgment time.

2.2.5. SLPs interval judgments of speech samplesTen experienced Icelandic SLPs judged each of the 756 5-s speaking intervals as either “Stuttered” or “Nonstuttered”

on 2 occasions approximately 2 months apart. The selection of the Icelandic SLPs was based on a survey conducted

2 Only monolingual children were selected because concerns have been expressed that the identification of stuttering in bilingual children’s mightbe complicated by their use of code switching, word finding difficulties or language formulation problems (Van Borsel, Maes, & Foulon, 2001).

J. Einarsdóttir, R.J. Ingham / Journal of Fluency Disorders 33 (2008) 167–179 171

by Shapiro et al. (2004); they were selected according to those reporting most experience in assessing and treatingstuttering in Iceland (excluding the first author). All SLPs finished a criterion test showing that they were able to identifyexemplars of stuttering (135 5-s intervals) in nine English-speaking CWS (age 2–8 years). The judgment accuracy datafor these CWS was based on judgments provided by recognized authorities on children’s stuttering (see Bothe, 2008).The SLPs’ mean accuracy score was 88.5% (range 81.4–94.7%).

The SLPs, four males and six females, were educated in five different countries; their mean age was 49.9 years(range 35–65 years). Before making their judgments on the first occasion the SLPs were not told they would be asked tomake judgments on a second occasion. They were instructed to watch and listen to each DVD alone without discussingit with others or asking others for their judgments. But they were permitted to replay any interval in order to be sureof their judgment decision. No definition of stuttering was provided for the SLPs; but instructions similar to thoseprovided by Cordes and Ingham (1995). As shown below these instructions3 emphasized that not all disruptions orinterruptions in speech should be considered to be stuttering:

Your task during this experiment is to decide whether each sample either contains or does not contain anyevidence of stuttering. If you decide that the sample did contain any stuttering then mark “S” (Stuttered) nextto that sample number on your answer sheet. If you decide that the sample did not contain stuttering, then mark“N” (Nonstuttered) next to that sample number. Stuttering will not be defined for you. You should be aware,however, that not all disruptions or interruptions in speech are stutterings. Some “disfluencies” are quite normaland acceptable in the speech of both persons who stutter and persons who do not stutter. Please do not countnormal or acceptable disruptions as stuttering. Any 5 second sample that you judge to contain normal disfluencies,but not stuttered disfluencies, should be judged “N” (Nonstuttered).

The SLPs’ interval judgments were then organized into three categories: Agreed Stuttered, Agreed Nonstuttered orDisagreed. The “Agreed” criterion was that 80% of the SLPs judgments across two occasions agreed that a particularinterval was either stuttered or nonstuttered (Cordes & Ingham, 1995). In other words, an interval was classified asAgreed Stuttered or Agreed Nonstuttered if, across the two occasions, at least 16 out of a possible 20 judgments were cat-egorized, respectively, “Stuttered” or “Nonstuttered.” The remaining intervals were classified as “Disagreed.” The SLPsjudgments agreed on approximately 80% of the intervals; the rest were classified as Disagreed intervals (see AppendixB). The number of intervals judged to be stuttered varied considerably across the SLPs (for example, on Occasion 1they ranged from 199 for SLP #5 to 399 for SLP #8). However, intrajudge agreement among the 10 SLPs was generallyhigh with mean self-agreement being 89.6% (range 76.2–98.8%) (Einarsdóttir & Ingham, submitted for publication).

2.2.6. Selection of samples for the SMAAT-child systemEach of the nine CWS provided two 1-min samples for inclusion within the SMAAT-child program. Table 1

shows the number of Agreed Stuttered, Agreed Nonstuttered and Disagreed intervals that were finally used within theSMAAT-child training program.

Across the 216 intervals there were 68 Agreed Stuttered intervals, 98 Agreed Nonstuttered intervals, and 50 Disagreedintervals. In other words, approximately 77% of the intervals were “agreed intervals” (Stuttered or Nonstuttered) andapproximately 23% were “disagreed intervals.” The number of Agreed Stuttered intervals varied across the children,ranging from 19 (subject # 8) to 3 (subject # 5). All but subject # 1, showed similar amounts of stuttering during theirfirst and second minute sample.

2.3. Training procedure

The procedures that followed were virtually the same as those used in earlier studies on the SMAAT-adult program(Cordes & Ingham, 1995; Cordes & Ingham, 1999; Ingham & Cordes, 1997a; Ingham & Cordes, 1997b). The teachersparticipated individually on each occasion, with their two assessments occasions separated by 2–3 weeks. The trainingsession for the experimental group began immediately after the first assessment session or on the following day. In theSMAAT-child training program the first and the second minute samples combined are used for the assessment task,but the training task only employs the first minute sample.

3 These instructions were translated into Icelandic.

172 J. Einarsdóttir, R.J. Ingham / Journal of Fluency Disorders 33 (2008) 167–179

Table 1Number of Agreed Stuttered (S), Agreed Nonstuttered (N), and Disagreed (D) intervals in the SMAAT-child training program

Child Minute 1 (training/probe) Minute 2 (generalization) Total

S N D S N D S N D

1 7 1 4 3 7 2 10 8 62 3 8 1 1 9 2 4 17 33 2 7 3 2 6 4 4 13 74 4 5 3 6 6 0 10 11 35 1 9 2 2 6 4 3 15 66 4 6 2 5 5 2 9 11 47 2 6 4 2 5 5 4 11 98 10 0 2 9 3 0 19 3 29 2 5 5 3 4 5 5 9 10

Total 35 47 26 33 51 24 68 98 50

The intervals were agreed on at least 16 out of 20 judgments.

At the beginning of each session, the preschool teacher (hereafter, the judge) sat before a computer housing theSMAAT-child program wearing computer-attached headphones. Following Cordes and Ingham (1999), the judgeswere instructed that they would be watching and listening to 2-min speech samples from nine children who hadbeen diagnosed as developmental stuttering speakers. They were told that whenever stuttering was detected the rightmouse button was to be pressed immediately and remain pressed for the duration of that stutter. As per the precedinginstructions, stuttering was not defined; judges were simply informed that not all disruptions or interruptions in speechare stuttering and that normal or acceptable disruptions are not to be recorded as stuttering (see above).

The experimenter first verified that the judge understood these instructions and then provided time for her to practicethe task. When the judge was ready, her judgments were then collected for all nine 2-min speech samples. The entireassessment task was completed three times (trial run 1, run 2 and run 3), with a 5-min break between each taskpresentation. The duration of the entire assessment task on each occasion was approximately an hour. Sample orderwas automatically randomized by the SMAAT-child software for each judge and for each trial.

Each Training Module consisted of two parts: Probe tasks and Training tasks and both tasks used only the firstminute in the nine samples. The Probe tasks were designed to evaluate the judge’s stuttering identification skillsindependent of training. Based on the accuracy of the judges during the Probe tasks the program selected the samplesfor the training. The samples that the judges judged least accurately were selected for training. During these tasks thejudge was simply instructed to press the right mouse button for the duration of each perceived stuttering event.

During the Training tasks (which began immediately after the initial Probe tasks) the judges repeated their mousebutton recording of perceived stuttering. However, if the program detected that a judge’s button press did not occurduring an Agreed Stuttered interval then the monitor screen immediately relayed that their judgment was incorrect; thevideo recording froze, then replayed the 5-s interval containing the misjudged stuttering and displayed text highlightingthe correct judgment response. A similar event occurred if a judge pressed a button indicating the presence of stutteringduring an Agreed Nonstuttered interval.

The training with each speaker in the program was completed successfully when no incorrect judgments wererecorded on 3 consecutive runs, or after 36 unsuccessful attempts to complete 3 consecutive error-free runs (as perCordes & Ingham, 1999). As per Cordes and Ingham (1999), no preschool teacher was trained more than twice on anyspeaker within training conditions. The training task was limited to four training tasks and their associated four probetasks. The overall duration of training ranged from 58 to 102 min.

This study was approved by the Icelandic Data Protection Authority and the Icelandic National Bioethics Committee.All the selected judges and the CWS parents completed an approved consent form before participating.

2.4. Data analyses

The SMAAT-child program incorporates software for analyzing the recorded judgments from each preschool teacher(judge). That record contains the exact time the right computer mouse button was pressed and released. Each judge’s

J. Einarsdóttir, R.J. Ingham / Journal of Fluency Disorders 33 (2008) 167–179 173

response per interval was able to be compared directly to the consensus judgments of 10 SLPs. If the judge pressed theright button of the mouse during an Agreed Stuttered or did not press the mouse button during an Agreed Nonstutteredinterval it was identified as correct. The Disagreed intervals were ignored and treated as neither correct nor incorrect.Judgment accuracy was calculated by determining the percentage of correct responses for each judge. The correctidentification of stuttered and nonstuttered intervals was divided by the total number of predetermined “agreed intervals”for each child thereby avoiding score inflation due to the inclusion of Disagreed interval responses. A mean correctresponse score was derived for each judge and for each run. The correct response data were calculated separately forthe assessment session and for the treatment session.

A two-factor repeated measure analysis of variance (ANOVA) (Winer, Brown, & Michels, 1991) was used tocompare the child training (experimental) and no training (control) groups’ scores before and after the SMAAT-childtraining period. Pairwise comparisons of means (t-tests) were used to identify significant effects between the traininggroup before and after training. The parametric analyses were calculated using SAS® statistical software version 9.1.Cohen’s d was calculated for the pairwise comparison of means (t-test score) and related to a “large,” “medium” or“small” effect size, as specified by Cohen (1988).

3. Results

3.1. Assessment session

Table 2 shows the preschool teachers mean percentage accuracy scores for assessing correctly the combined AgreedStuttered and Agreed Nonstuttered intervals. Their mean accuracy is presented for the three runs on Occasion 1 (beforetraining) and for the three runs on Occasion 2 (after training). The most obvious and important finding was that evenbefore training both the experimental and the control groups displayed what could be regarded as satisfactory levelsof judgment accuracy—both groups correctly identified approximately 80% of the agreed intervals (stuttered andnonstuttered).

Comparisons were then made between the judgments skills of the 10 preschool teachers trained with the SMAAT-child program and the control group who received no training between Occasion 1 and 2. A repeated measurestwo-factor ANOVA showed a significant main effect between the groups in the level of accuracy in judging AgreedStuttered and Agreed Nonstuttered intervals [F(1, 18) = 13.20, p < .001]. There was also a significant Occasion effect[F(1, 92) = 25.79, p < .0001], plus a significant Training × Occasion interaction effect [F(1, 92) = 9.27, p < .01]. Thisindicates that the training program did significantly improve the teachers’ judgment accuracy levels. There were,however, no significant differences between trial runs within each of the two judgment occasions [F(1, 92) = 1.70,p = .19] which, in turn, indicates that simply judging the speech samples repeatedly did not improve judgment accuracy.

Table 2The assessment sessions in the SMAAT-child training program

Teacher Training Non-training

Occasion 1 (%) Occasion 2 (%) Occasion 1 (%) Occasion 2 (%)

1 84.9 92.9 77.2 77.02 87.8 85.1 82.9 82.23 85.3 89.3 83.0 83.74 80.4 88.7 80.0 91.35 82.1 87.1 74.5 74.56 81.5 84.8 80.9 82.97 85.9 88.0 86.1 82.48 84.0 89.0 81.5 85.99 87.2 90.9 84.1 84.710 82.7 90.2 82.3 79.1

Mean 84.2 88.6 81.3 82.4

The mean percentages of correct judgments for each occasion (each occasion included three task trial runs) for preschool teachers (n = 10) trainedwith the SMAAT-child program and preschool teachers who did not receive training (n = 10). Training occurred between Occasion 1 and 2.

174 J. Einarsdóttir, R.J. Ingham / Journal of Fluency Disorders 33 (2008) 167–179

Fig. 1. Preschool teachers’ (n = 20) judgment accuracy in assessing stuttering during speech samples from nine preschool CWS three trial runs onOccasion 1 and three trial runs on Occasion 2. The mean percent correct judgments of the experimental group (n = 10) before and after SMAAT-childtraining and the mean percent correct judgments of the control group (n = 10). Training occurred between Occasion 1 and 2.

Post hoc t-test comparisons for the experimental group showed a significant difference between mean accuracylevels on Occasion 1 (M = 84.2, S.D. = 3.1) and Occasion 2 (M = 88.6, S.D. = 2.6) [Estimate = −4.43, S.E. = 0.75,t(29) = −6.30, p < .0001; 95% confidence interval = −5.9 to −3.0]. The Cohen effect size score obtained for the beforeand after training difference in accuracy score means was also considered to be large (d = 1.5). By contrast, the controlgroup who received no training did not show a significant difference between their accuracy scores on Occasion 1(M = 81.3, S.D. = 4.2) and Occasion 2 (M = 82.4, S.D. = 5.1) [Estimate = −1.11, S.E. = 1.20, t(29) = −1.17, p = .25; 95%confidence interval = −3.04 to 0.83].

Notwithstanding the randomization procedure used for judge-group allocation in this study, it still emerged thatthe two groups displayed significantly different levels of judgment accuracy before Occasion 1 [Estimate = −2.91,S.E. = 0.95, t (58) = −3.07, p < .01; 95% confidence interval = 0.14–5.69]. Two teachers in the control group showedbelow 80% judgment accuracy, but all teachers in the training group showed over 80% accuracy in assessing AgreedStuttered and Agreed Nonstuttered intervals. The fact that even the group with the higher score on average beforetraining improved after training strengthens claims that the training was effective. The effect of the training programis graphically shown in Fig. 1.

3.2. Training session

Table 3 shows the effect of the training during the course of the training sessions. This was determined by examiningthe levels of judgment accuracy produced by each preschool teacher from the initial to the final probe. All membersof the experimental group improved their level of judgment accuracy from the initial (M = 86.7, S.D. = 2.9) to thefinal training run (M = 91.8, S.D. = 4.0). These changes were significant [Estimate = −5.11, S.E. = 1.20, t(9) = −5.16,p < .001; 95% confidence interval = −7.35 to −2.87].

4. Discussion

This study replicates earlier research findings that have shown that training does improve the ability to iden-tify occurrences of stuttering during connected speech (Cordes & Ingham, 1994b; Cordes & Ingham, 1999;Ingham et al., 1993). It has been shown that a stuttering judgment training program based on the use of inter-val measurement can improve interjudge agreement for the judgment of stuttering in preschool CWS—just ashas been previously shown for judging stuttering in PSW (SMAAT-adult; Cordes & Ingham, 1999; Ingham etal., 1998). One noteworthy finding was that even without training the preschool teachers employed in this studywere capable of identifying stuttering with what many researchers consider satisfactory levels of judgment accu-

J. Einarsdóttir, R.J. Ingham / Journal of Fluency Disorders 33 (2008) 167–179 175

Table 3Mean percent correct (plus range) judgments for the first and the final probe phase in the SMAAT-child training program by the experimental groupof preschool teachers (n = 10)

Teacher Initial probe Final probe

Mean correct (%) Range Mean correct (%) Range

1 91.4 (85–100) 98.3 (85–100)2 87.7 (75–100) 89.3 (77–100)3 87.1 (75–100) 94.9 (81–100)4 81.0 (62–100) 88.7 (71–100)5 89.0 (75–100) 91.8 (81–100)6 85.1 (75–100) 87.3 (75–100)7 88.8 (75–100) 92.8 (72–100)8 84.7 (72–100) 85.9 (70–100)9 85.2 (50–100) 95.6 (85–100)10 86.7 (72–100) 93.2 (81–100)

Mean 86.7 (50–100) 91.8 (70–100)

racy (Kazdin, 1982). The findings did show that judgment accuracy could still be improved through training,and that merely making repeated judgments on a child’s speech sample was not sufficient to improve judgmentaccuracy.

4.1. The development of a standardized training program

A standardized system for training stuttering assessment, such as the SMAAT program, relies on a corpus of reliablestuttering judgments that come from sources with high face validity. High intra- and inter-judge agreement amongthose providing judgments is also considered to be a fundamental requirement. The original SMAAT-adult program,for instance, was based upon the combined judgments of a group of highly recognized authorities in stuttering.Their mean intrajudge agreement level was 89.1% indicating that their judgments displayed satisfactory consistency(Cordes & Ingham, 1995). Accordingly, the development of the SMAAT-child standardized training program forIcelandic CWS was ultimately based upon the judgments of individuals who ostensibly had equivalent face validity:the judgments of 10 experienced Icelandic SLPs. All had documented experience in treating and assessing stutteringin young children, but their experience was still somewhat limited; only two had assessed or treated more than 50CWS or PWS. Nonetheless, this group of SLPs also displayed high levels of judgment consistency. Their meanintrajudge agreement level (89.6%) was actually very similar to the mean intrajudge agreement level achieved by therecognized stuttering authorities (89.1%) who provided the basic data for the SMAAT-adult program. Furthermore, theIcelandic SLPs also displayed generally high levels of accuracy. As mentioned earlier, they all showed that they wereable to identify exemplars of stuttering in English-speaking CWS (age 2–8 years). Using 135 5 s intervals from nineCWS (3 females; 6 males) in the Bothe (2008) study, the SLPs the mean accuracy for the Agreed Stuttered intervalswas 86.6% (range 66.1–100%) and their mean accuracy in identifying the nonstuttered intervals was 90.6% (range74.1–98.1%).

In the present study the Icelandic SLPs actually achieved considerably higher levels of interjudge agreement whenjudging the Icelandic preschool CWS than was the case for the recognized authorities who judged stuttering in English-speaking PWS (Cordes & Ingham, 1995). The mean interjudge agreement level among the Icelandic SLPs (acrossjudgment occasions) was 80.1%. Counterpart data from adult stuttering authorities, as reported by Cordes and Ingham(1995), was only 67.2%. And Bothe (2008) recently reported that when authorities on stuttering in children judgedrecorded intervals of speech from English-speaking stuttering children (3–8 years) their mean interjudge agreementwas 73.2%. In all three studies interjudge agreement was derived from an analysis of intervals that were agreed tobe either stuttered or nonstuttered by 80% of the judges. However, while the Icelandic SLPs did show somewhathigher interjudge agreement levels, conclusions based on a direct comparison across studies is hazardous becauseof differences among the speech samples employed across these studies. Also, it is likely that identifying stuttering

176 J. Einarsdóttir, R.J. Ingham / Journal of Fluency Disorders 33 (2008) 167–179

produced by children is an easier task than identifying stuttering by adults where the target behavior is embeddedwithin samples that likely contain much faster speaking rates than those produced by very young children. It is likelythat young children are also less capable than some adults of obscuring their stuttering which in turn may contributeto an easier judgment task.

The accuracy with which teachers in this study identified stuttering may have some implications for continuingdebates about the definition of stuttering behavior. The teachers in this study were not provided with any definitionon stuttering, except to be told to try to avoid counting normal disfluencies as stutterings. Presumably the events theyobserved exceeded some personal perceptual level that prompted them to label the event as “stuttering” rather than(implicitly) a normal disfluency. In other words, they likely used, what Martin and Haroldson (1981) have termed, aperceptual threshold definition. The present study’s findings may also be relevant to findings from an investigationof awareness and understanding of stuttering among preschool teachers (n = 193) in Scotland. Approximately 75%of these teachers claimed they could confidently recognize stuttering in young children, while only 5% reported theywere not confident (Lees et al., 2004).

Time interval procedures have previously been shown to be a useful method for improving the interjudge andintrajudge agreement of stuttering measurement with adult and adolescent PWS (Cordes & Ingham, 1999; Cordes etal., 1992). But its usefulness may be of value in other areas of speech-language pathology. The method has recentlybeen used to examine the intra- and inter-observer agreement in the occurrence and duration of social communicationbehaviors (Olswang, Svensson, Coggins, Beilinson, & Donaldson, 2006). This illustrates another application thatmakes it possible to establish judges’ intra- and inter-judge reliability in recording complex behaviors and find highlyagreed exemplars that can be used for training other judges.

In the future it will obviously be important to test the efficacy of the SMAAT-child training program using a muchlarger sample of judges, and also with judges who display a lower level of judgment accuracy. In addition, the preschoolteachers’ skill in recognizing stuttering would certainly be better tested if in future studies the CWS’s samples weremixed with samples from CWNS. Such studies will help solve the continuing need for a standardized method for thetraining clinicians and researchers to identify stuttering accurately and reliably.

Acknowledgements

This manuscript was prepared as part of the requirements for the first author’s doctoral program at the Universityof Iceland. This work was supported by the University Eimskipafélag Fund, Graduate Research Fund of The IcelandicCentre for Research, Educational Grant of Landsbanki Íslands, Reykjavik Educational fund, Oddur Ólafsson Grantand Málbjörg, the Icelandic Stuttering Association. Special thanks are offered to Dr. Þórarinn Sveinsson for his helpfulcomments on data analyses, and to all of the children, parents, teachers and SLPs who contributed their time for thisstudy. Correspondence should be addressed to Jóhanna Einarsdóttir, Háskóli Íslands, Menntavísindasvið, Bolholti 5,105 Reykjavík, Iceland. Email address: jeinars@hi.is.

CONTINUING EDUCATION

The effect of stuttering measurement training on judging stuttering occurrence in preschool children whostutter

QUESTIONS

1. Why is the identification of stuttering in young children important?a. because the parents have to learn to estimate the duration of each stuttering eventb. because young children respond well to direct interventionc. for the categorization of the repetitions and prolongationsd. because parents need to ignore their child’s stuttering-like-disfluencies so as to prevent the development of

chronic stuttering

J. Einarsdóttir, R.J. Ingham / Journal of Fluency Disorders 33 (2008) 167–179 177

2. A standardized training program was developed using time interval procedures because:a. judges are then able to learn to identify different types of stuttered intervalsb. agreed intervals can be used for training purposesc. the judges can be trained to predict the likelihood of unassisted recoveryd. the judges learn to estimate the time of each stuttering within each interval

3. The inter- and intra-judge reliability in this study:a. shows the level of agreement among the judgesb. provides face validity for the stuttering judgmentsc. was higher among the SLPs than among the preschool teachersd. was relatively high among the SLPs

4. The preschool teachers in the study:a. only showed high accuracy in identifying stuttering after trainingb. improved their judgment accuracy by repeatedly judging the speech samplesc. were selected because previous studies have shown that they identify occurrences of stuttering with high levels

of accuracyd. significantly improved their judgments accuracy through training rather than by simply repeatedly judging speech

samples5. Identifying stuttering in young children:

a. is only possible if the judges are provided with a standard definition of stutteringb. is only possible if the judges know the difference between the prolongation and repetitionsc. is able to be improved by a standardized training programd. is more difficult than identifying stuttered occurrences in adults

Appendix A

See Table A1.

Table A1Descriptive data on the children used in the development of the SMAAT-child training program

Child Gender Age (months) Severity Selection

1 F 65 Moderate Selected2 M 68 Mild3 M 61 Moderate4 M 42 Moderate5 M 55 Mild Selected6 M 71 Mild7 F 56 Mild Selected8 F 61 Moderate9 F 66 Moderate

10 M 57 Severe Selected11 M 56 Mild Selected12 F 47 Severe13 F 50 Moderate Selected14 M 43 Moderate Selected15 M 54 Mild16 M 54 Mild17 M 44 Moderate18 M 66 Severe19 M 64 Severe Selected20 M 49 Severe Selected

178 J. Einarsdóttir, R.J. Ingham / Journal of Fluency Disorders 33 (2008) 167–179

Appendix B

See Table A2.

Table A2The percent of intervals with 80% agreement classified as Stuttered and Nonstuttered

Child Occasion 1 (%) Occasion 2 (%) Occasion 1 and 2 (%)

1 85.7 88.1 84.52 85.7 90.5 88.13 69.1 76.2 70.24 89.3 85.8 85.75 77.4 83.3 79.86 73.8 81.0 81.07 79.8 78.6 76.28 82.1 83.3 78.69 77.4 83.3 77.4

Mean 80.0 83.3 80.2

Each child’s sample consisted of 84 intervals judged by Icelandic SLPs (n = 10) on two occasions.

References

Baer, D. M., Wolf, M. M., & Risley, T. R. (1987). Some still-current dimensions of applied behavior analysis. Journal of Applied Behavior Analysis,20, 313–327.

Bothe, A. K. (2008). Identification of children’s stuttered and nonstuttered speech by highly experienced judges: Binary judgments and comparisonswith disfluency-types definitions. Journal of Speech, Language, and Hearing Research, 51, 867–878.

Bothe, A. K., Davidow, J. H., Bramlett, R. E., & Ingham, R. J. (2006). Stuttering treatment research, 1970–2005. I. Systematic review incorporatingtrial quality assessment of behavioral, cognitive, and related approaches. American Journal of Speech-Language Pathology, 15, 321–341.

Carey, B., Block, S., Ross, F., Borg, V., & O’Halloran, P. (2008). Preschool teachers and stuttering: A survey of knowledge, attitudes and referralpractices. Australian Communication Quarterly, 10, 4–7.

Christie, E. (2000). The primary healthcare workers project. London: British Stammering Association.Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). New York: Lawrence Erlbaum Associates.Cordes, A. K., & Ingham, R. J. (1994a). The reliability of observational data. II. Issues in the identification and measurements of stuttering events.

Journal of Speech and Hearing Research, 37, 279–294.Cordes, A. K., & Ingham, R. J. (1994b). Time interval measurement of stuttering: Effects of interval duration. Journal of Speech and Hearing

Research, 37, 779–788.Cordes, A. K., & Ingham, R. J. (1994c). Time interval measurement of stuttering: Effects of training with highly-agreed or poorly-agreed exemplars.

Journal of Speech and Hearing Research, 37, 1295–1307.Cordes, A. K., & Ingham, R. J. (1995). Judgments of stuttered and nonstuttered intervals by recognized authorities in stuttering research. Journal

of Speech and Hearing Research, 38, 33–41.Cordes, A. K., & Ingham, R. J. (1996). Time-interval measurement of stuttering: Establishing and modifying judgment accuracy. Journal of Speech

and Hearing Research, 39, 298–310.Cordes, A. K., & Ingham, R. J. (1999). Effects of time-interval judgment training on real-time measurement of stuttering. Journal of Speech,

Language, and Hearing Research, 42, 862–879.Cordes, A. K., Ingham, R. J., Frank, P., & Ingham, J. C. (1992). Time interval analysis of interjudge and intrajudge agreement for stuttering event

judgments. Journal of Speech and Hearing Research, 35, 483–494.Einarsdóttir, J., & Ingham, R. J. (submitted for publication). Does language influence the accuracy of judgments of stuttering in children?Ingham, R. J. (1990). Research on stuttering treatment for adults and adolescents: A perspective on how to overcome a malaise. In J. A. Cooper (Ed.),

Research needs in stuttering: Roadblocks and future directions (pp. 91–95). Rockville, MD: American Speech-Language-Hearing Association.Ingham, R. J., & Cordes, A. K. (1997a). Identifying the authoritative judgments of stuttering: Comparisons of self-judgments and observer judgments.

Journal of Speech, Language, and Hearing Research, 40, 581–594.Ingham, R. J., & Cordes, A. K. (1997b). Self-measurement and assessing treatment efficacy. In R. F. Curlee & G. M. Siegel (Eds.), Nature and

treatment of stuttering: New directions (2nd ed., pp. 413–438). Boston: Allyn and Bacon.Ingham, R. J., & Cordes, A. K. (1999). On watching a discipline shoot itself in the foot: Some observations on current trends in stuttering treatment

research. In N. Bernstein Ratner & E. C. Healey (Eds.), Stuttering research and practice: Bridging the gap (pp. 211–230). Mahwah, NJ: LawrenceErlbaum.

Ingham, R. J., Cordes, A. K., & Gow, M. L. (1993). Time interval measurement of stuttering: Modifying interjudge agreement. Journal of Speechand Hearing Research, 36, 503–515.

J. Einarsdóttir, R.J. Ingham / Journal of Fluency Disorders 33 (2008) 167–179 179

Ingham, R. J., Cordes, A. K., Kilgo, M., & Moglia, R. (1998). Stuttering measurement assessment and training (SMAAT). Santa Barbara, CA:University of California, Santa Barbara.

Ingham, R. J., Cordes, A. K., Kilgo, M., Moglia, R., Einarsdóttir, J., & Heliker, D. (2006). Stuttering measurement assessment and training (SMAAT)children. Santa Barbara, CA: University of California, Santa Barbara.

Kazdin, A. E. (1982). Single-case research designs: Methods for clinical and applied settings. New York: Oxford University Press.Lees, R., Stark, C., Birse, S., & Nicoll, P. (2004). Some factors influencing early referral of dysfluent children. In A. Packman, A. Meltzer, & H. F.

M. Peters (Eds.), Theory, research and therapy in fluency disorders. Proceedings of the fourth world congress on fluency disorders (pp. 96–101).Nijmegen: Nijmegen University Press.

Lewis, D., & Sherman, D. (1951). Measuring the severity of stuttering. Journal of Speech and Hearing Disorders, 16, 320–326.Martin, R. R., & Haroldson, S. K. (1981). Stuttering identification: Standard definition and moment of stuttering. Journal of Speech and Hearing

Research, 24, 59–63.Menntamálaráðuneytið (2006). Tillögur starfshóps um framtíðarskipan kennaramenntunar. [Proposal for future arrangements for teachers’ edu-

cation, prepared by a working group for the Ministry of Education]. Retrieved March 30, 2007, from http://www.menntamalaraduneyti.is/menntamal/tiuskref/.

O’Brian, S., Packman, A., Onslow, M., & O’Brian, N. (2004). Measurement of stuttering in adults: Comparison of stuttering-rate and severity-scalingmethods. Journal of Speech and Hearing Research, 47, 1081–1087.

Olswang, L. B., Svensson, L., Coggins, T. E., Beilinson, J. S., & Donaldson, A. L. (2006). Reliability issues and solutions for coding socialcommunication performance in classroom settings. Journal of Speech, Language, and Hearing Research, 49, 1058–1071.

Onslow, M., Packman, A., & Harrison, E. (2003). The Lidcombe program of early stuttering intervention: A clinician’s guide. Austin, TX: Pro-Ed.Shapiro, D. A., Abbink, M., Bortz, M., Bruna, A. V., Cook, F., Dhu, P., et al. (2004). A multinational investigation of stuttering intervention:

Assumptions practices, and lessons. In A. Packman, A. Meltzer, & H. F. M. Peters (Eds.), Theory, research and therapy in fluency disorders.Proceedings of the fourth world congress on fluency disorders (pp. 117–123). Nijmegen: Nijmegen University Press.

Statistics Iceland (2007, April 25). Retrieved June 21, 2007. From http://www.statice.is/Home/Statistics/Education/Pre-primary institutions.Van Borsel, J., Maes, E., & Foulon, S. (2001). Stuttering and bilingualism: A review. Journal of Fluency Disorders, 26, 179–205.Winer, B. J., Brown, D. R., & Michels, K. M. (1991). Statistical principles in experimental design (3rd ed.). New York: McGraw-Hill.Yairi, E., & Ambrose, N. (1999). Early childhood Persistency and recovery rates. Journal of Speech, Language, and Hearing Research, 42, 1097–1112.Yairi, E., Ambrose, N., & Niermann, R. (1993). The early months of stuttering: A developmental study. Journal of Speech and Hearing Research,

36, 521–528.Yairi, E., Ambrose, N. G., Paden, E. P., & Throneburg, R. N. (1996). Predictive factors of persistence and recovery: Pathways of childhood stuttering.

Journal of Communication Disorders, 29, 51–77.Zebrowski, P., & Conture, E. (1989). Judgments of disfluency by mothers of stuttering and normally-fluent children. Journal of Speech and Hearing

Research, 32, 625–634.

Jóhanna Einarsdóttir is an Assistant Professor at the University of Iceland-School of Education and a Ph.D. student at the University of Iceland,Medical Department. Her main advisor is Dr. Roger Ingham. She has been employed as a Speech-Language Pathologist since 1987, specializing inthe assessment and treatment of developmental stuttering since 1994.

Roger J. Ingham, PhD, is a Professor in the Department of Speech and Hearing Sciences at the University of California, Santa Barbara, Professor ofResearch at the University of Texas San Antonio and a Faculty Associate in the Research Imaging Center at the University of Texas Health ScienceCenter at San Antonio. His research has focused on the development of stuttering treatments, measures of stuttering, the neurology of stutteringand, most recently, its genetic basis.