Lexical Considerations for Standardized Vocabulary Testing ... · phonological or orthographic form, such as penguin in English and pinquino in Spanish (Kohnert, Wind-sor, & Miller,

202 Contemporary Issues In CommunICatIon sCIenCe and dIsorders • Volume 42 • 202–214 • Fall 2015

ABSTRACT: Purpose: The purpose of this study was to examine errors made by Spanish–English-speaking chil-dren on the Peabody Picture Vocabulary Test—Fourth Edition (Dunn & Dunn, 2007), specifically exploring (a) the relationship between norm-based difficulty level and frequency of errors for English language learners (ELLs) as compared to monolinguals and (b) the rela-tionship between the word familiarity index, cognate status, and percentage of error on test items. Method: The research aims were carried out in 2 studies—1 exploratory and a subsequent replication study. The exploratory study included 100 Spanish–English-speaking children in kindergarten and a com-parison group of 25 English-speaking monolinguals, followed by replication with 51 Spanish–English-speaking children in kindergarten and a comparison group of 50 English-speaking monolinguals.

A

Contemporary Issues In CommunICatIon sCIenCe and dIsorders • Volume 42 • 202–214 • Fall 2015 © NSSLHA 1092-5171/15/4202-0202

Lexical Considerations for Standardized Vocabulary Testing With Young Spanish–English Speakers

Carla Wood Vanessa PeñaFlorida State University, Tallahassee

child who speaks Spanish at home to communicate with his or her parents and later learns English as a second language

Results: The intended difficulty level of the vocabu-lary item was a significant predictor of performance, but the strength of the relationship differed between the 2 groups. The ELLs showed an uneven distribu-tion of errors with a moderate positive correlation (R² = .43, exploratory study; R² = .53, replication study), whereas the monolinguals demonstrated a strong positive relationship between the intended difficulty level and error rate (R² = .91, exploratory study; R² = .81, replication study). Implications: Findings substantiate the need for school personnel and child-language researchers to consider linguistic interplay when assessing young ELLs.

KEY WORDS: cultural and linguistic diversity, devel-opment, diagnostics, language, language disorders

(ESOL) is considered an English-language learner (ELL). There are a growing number of young chil-dren in U.S. elementary schools who are considered ELLs. Spanish-speaking families are among the fastest growing ELL group, representing more than 65% of non English–speaking immigrants in the United States (National Center for Education Statis-tics, 2004). According to the U.S Census Bureau’s

American Community Survey in 2006, 78% of the 39,540,935 Latinos (age 5 and over) living in the United States reported that they spoke Spanish at home, and 39% reported that they spoke English less than “very well.”

Given the cultural and linguistic diversity in the United States, school personnel such as general edu-cators, ESOL instructors, and speech-language pathol-ogists (SLPs) are encouraged to become knowledge-able about culturally responsive bilingual assessment procedures and to employ recommended practices in

Wood & Peña: Lexical Considerations for Standardized Vocabulary Testing 203

assessment (Gutierrez-Clellen & Peña, 2001; Gutier-rez-Clellen, Restrepo, & Bedore, 2000; Peña, 1992; Saenz & Huer, 2003). The use of naturalistic, au-thentic assessment is highly recommended for young children as a culturally responsive practice (Bagnato, 2005; Bagnato, Neisworth, & Pretti-Frontczak, 2010; Crais, 1995, 2011; Division for Early Childhood of the Council for Exceptional Children [DEC], 2007). Naturalistic observations in everyday contexts and alternative assessment measures are considered to be best practice, particularly to capture the interplay between the dual languages and to obtain meaningful, functional information about children’s communica-tion in typical activities (Crais, 1995; DEC, 2007; Peña, 1992).

Despite recommended practice guidelines and potential advantages of alternative assessment mea-sures, current assessment practices with ELLs largely involve English-normed standardized assessment measures (Caesar & Kohler, 2007; Kaderavek, 2010; Senaga & Inglebret, 2003). Generally speaking, stan-dardized tests are one of the most commonly used as-sessment tools among SLPs (Caesar & Kohler, 2007; Kaderavek, 2010). When evaluating language skills in children, SLPs reportedly use norm-referenced assess-ments 80% of the time (Kaderavek, 2010). In a sur-vey of 130 SLPs who had assessed bilingual students, most reported using traditional standardized testing procedures for assessing bilingual students (Caesar & Kohler, 2007). An analysis of the formal assessment measures that SLPs identified on the survey indicated that the vast majority (98%) of measures were Eng-lish-published measures. The Peabody Picture Vocab-ulary Test—Third Edition (PPVT–III; Dunn & Dunn, 1997) was used by the highest number of respondents (51%) who reported assessing students who were dual-language learners.

Given the ongoing predominant use of English standardized assessment measures for diagnostic, clini-cal, and research purposes, additional examination of lexical considerations for ELLs is warranted. The unique lexical considerations for dual-language learn-ers, such as the interplay between the child’s first-lan-guage (L1) and second-language (L2) lexicons, may be underrecognized in using standardized tests and interpreting results on vocabulary measures (Bialystok, Luk, Peets, & Yang, 2012), specifically influencing the intended difficulty level and the appropriateness of us-ing the recommended basals and ceilings.

Although ease of administration may be a com-mon consideration in the decision to use an assess-ment measure, other linguistic factors may negate the utility of the information derived from the measure. In other words, if an assessment measure does not have good validity for use with ELLs, there may be

shortfalls in its ability to identify specific areas for intervention, and it may over- or underidentify ELLs as having language disorders (Kaderavek, 2010). To further illustrate, several considerations unique to ELLs will be highlighted. Among lexical consider-ations derived from the literature, we highlight the (a) interplay between languages (e.g., as a facilitator or competition in linguistic processing), (b) cultural and linguistic bias in item selection, and (c) uneven distribution of difficulty.

Interplay Between LanguagesOne unique aspect of dual-language development as opposed to monolingual development is the dynamic interplay between children’s L1 and L2. Researchers have offered theories and models of bilingual word learning to explain the unique interplay of L1 and L2 for vocabulary acquisition, storage, and retrieval (Cummins, 1984; deGroot & Hoeks, 1995; Kroll & Stewart, 1994; Potter, So, Von Eckardt, & Feldman, 1984). One such developmental model is the revised hierarchical model (RHM; Kroll & Stewart, 1994). This model emphasizes the transfer of knowledge between the L1 and L2 by dual-language learners. The RHM hypothesizes that ELLs gain access to vocabulary concepts in their L2 through their L1 lexi-con base. Similar to the RHM, the Cummins model (Cummins, 1984) emphasizes the interplay between Spanish (L1) and English (L2) for word process-ing and word learning. The premise of a common underlying proficiency (CUP) suggests that children use their conceptual understanding of lexical items in their L1 as a knowledge base to facilitate L2 acquisi-tion (Cummins, 1984).

Following the premises of the RHM and CUP, if a lexical item is presented in English only (L2), then Spanish–English-speaking children may activate and use both their Spanish and English lexicons eo decode English (Kroll, Gerfen, & Dussias 2008; von Studnitz & Green, 2002). When a lexical assessment measure is conducted in English, the interplay of lan-guages may affect an ELL’s lexical performance. This tendency toward dual-lexical activation for ELLs has potential implications for the assessment of ELLs’ vocabulary knowledge. The perceived difficulty of a lexical item may be partially mediated by overlap-ping characteristics between the L1 and L2. In other words, some words may be perceived as more or less difficult, depending on how many features are shared with the child’s L1. Similarity between words in the child’s L1 and L2 may be expected to facilitate ease of recognition or, conversely, similarity may cause interference from an unintended sense of the word in the L1 (von Studnitz & Green, 2002).


One example of explicit overlap between the L1 and L2 is seen in cross-linguistic cognates. The term cross-linguistic cognate is used to refer to a word pair in two languages that shares both meaning and phonological or orthographic form, such as penguin in English and pinquino in Spanish (Kohnert, Wind-sor, & Miller, 2004). Researchers have suggested that English words sharing more features with Spanish are more readily learned or easier for young Spanish–Eng-lish-speaking ELLs than English words sharing fewer features (Kohnert et al., 2004). Given the potential mediation of cross-linguistic overlap, the lexical rela-tionship between Spanish and English for young dual-language learners may be expected to influence their performance on vocabulary assessment measures.

With unique cross-language interactions in mind, it is not surprising that previous researchers have called to question the construct validity of standard-ized tests designed for monolinguals that have been administered to ELLs. In 1984, Sattler and Altes used the Peabody Picture Vocabulary Test—Revised (PPVT–R; Dunn & Dunn, 1981) and the McCarthy Perceptual Performance Scale (McCarthy, 1970) to assess a group of typically developing (TD) 3- to 6-year-old Spanish–English-speaking Hispanic chil-dren. They found that the children’s scores on the PPVT–R, whether administered in English or Spanish, were significantly below the norms. Yet, all of the children’s scores on the McCarthy Perceptual Perfor-mance Scale indicated that the children had normal intelligence. In this case, the PPVT–R scores did not correspond with the children’s performance on the McCarthy Perceptual Performance Scale, which sug-gests incongruence in the evaluation of Spanish–Eng-lish-speaking children’s language skills. This incon-gruence may be additionally influenced by the fact that the normative samples largely consist of Cauca-sian, middle-class, monolingual English speakers, and also due to potential linguistic bias in item selection and design (Kester & Peña, 2002).

Cultural and Linguistic BiasBias is another potential influencing factor on ELLs’ performance on standardized vocabulary assessment measures that warrants consideration. According to Laing and Kamhi (2003), content bias occurs when “test stimuli, methods, or procedures reflect the as-sumption that all children have been exposed to the same concepts and vocabulary or have had similar life experiences” (p. 45). Laing and Kamhi sug-gested that children from culturally and linguistically diverse backgrounds may perform below the norm on standardized assessment measures because their life experiences or social behaviors vary from those of

mainstream culture. Linguistic bias is linked to the discrepancy between the language or dialect used by the child, the language or dialect used by the assess-ment tool, and the language or dialect that the child is expected to use when responding (Laing & Kamhi, 2003).

A potential cultural bias of the PPVT–III was noted in the findings of Bialystok et al. (2012). These authors examined the PPVT–III performance of 1,738 children between 3 and 10 years old, of whom 772 were English-speaking monolinguals and 966 were dual-language learners, primarily East Asian (n = 329) and non-Asian languages (n = 247), including 34 lan-guages. Although this study did not focus on Spanish-speaking children, the results are relevant to cultural bias of the standardized test. The results suggested that children from linguistically diverse backgrounds achieved lower standard scores on the PPVT–III than English-speaking monolinguals. Bialystok et al. also examined performance on specific categories of words for 161 participants (age 6;0 [years;months] to 6;11). The findings demonstrated that the dual-language learners scored significantly lower on home category items (e.g., food and household items) than the English-speaking monolinguals (p < .003). The performance of dual-language learners and monolin-guals was similar for items related to school context (e.g., professions, animals or plants, shapes, and instruments) and school experiences (p < .06).

Uneven Distribution of DifficultyGiven potential bias, standardized tests of vocabu-lary that have been normed on monolingual English speakers may have reduced construct validity for use with young ELLs. One critical assumption in the standardized design of vocabulary measures is that vocabulary items get increasingly more difficult as the test items progress. This progression of difficulty allows examiners to use a basal and ceiling score in computing a final score and interpreting each child’s performance. A basal is achieved when a child attains a designated number of items correct. Conversely, a ceiling is reached when a child makes a designated number of errors (e.g., six out of eight incorrect within a set). If the test is well designed, the exam-iner does not need to administer items above the ceil-ing or below the basal because the examiner can as-sume with confidence that items that occur before the basal are known items and individual items occurring after the ceiling are unknown items. For ELLs, it is possible that standardized tests of vocabulary do not adhere to the same progression of difficulty as in-tended due to linguistic differences in the phonotactic nature of the words and/or familiarity of the word.


Other examples of potential lexical considerations on standardized vocabulary assessment measures relate to item selection and test design that are not appropriately managed by translation. Direct transla-tions often do not consider the linguistic differences between English and Spanish and assume that certain target features are developed in the same order in both languages (Kester & Peña, 2002; Restrepo & Silver-man, 2001). When translated tests are used, the items on these assessment measures may be organized in the order of difficulty based on familiarity rankings of the initial language only (Kester & Peña, 2002). In other words, the content validity of these assessment mea-sures may be at risk in the translation process.

The uneven distribution of errors or uncon-trolled difficulty level of items can also occur for several additional reasons. Several potential factors may negatively impact the difficulty of lexical items on the PPVT–4 for ELLs, including differences in word familiarity, semantic boundaries, and the lack of translation equivalence. A word in one language may not have a direct translation equivalent in the other or may have a different categorical boundary, including or excluding items that the other language does not (Elston-Güttler & Williams, 2008; Prior, MacWhinney, & Kroll, 2007; Tokowicz, Kroll, de Groot, & van Hell, 2002). For example, toe in Eng-lish refers exclusively to a portion of the foot, but the translation equivalent in Spanish, dedo, refers to both fingers and toes. When semantic boundaries dif-fer between the L1 and L2, ELLs may exhibit errors characterized by overextensions or underextensions in meaning. As a result, when asked to point to the toe and include fingers in that translation, Span-ish–English-speaking ELLs may show overextensions. A previous article (Gathorcole & Moawad, 2010) provides an in-depth discussion of this interplay in which distinctive features of one language influence the other.

Although the lack of translation equivalents could be expected to make a lexical item more difficult to recognize, in other instances, dual-lexical activation may also facilitate performance, as discussed earlier, making some lexical items easier to recognize than in-tended in the L2. In other words, the conceptual space shared by the two languages may serve to facilitate recognition of a novel word in the L2. Similar to what Cummins described as CUP (1984), children who have limited exposure to a lexical item in their L2 may use the shared conceptual foundation from their L1 to decode the meaning of the word in their L2.

With the demand for differential diagnostic as-sessment of ELLs on the rise, it is imperative to critically evaluate ELLs’ performance on a standard-ized vocabulary assessment measure and consider

factors that may be associated with high/low per-formance on specific items. For ELLs, it is possible that standardized tests of English vocabulary do not adhere to the same even progression of difficulty as intended in test construction due to linguistic differ-ences in the phonotactic nature of the words and/or cultural familiarity of the word. Some words may be less familiar than intended, and other words may be more or less difficult than intended depending on how many features are shared with the child’s L1 and cultural background. In the current study, we set out to explore and describe (a) the relationship between norm-based difficulty level and frequency of error for ELLs as compared to monolinguals and (b) the rela-tionship between the word familiarity index, cognate status, and percentage of error on individual items.

METhod

Participants An initial exploratory study involved a retrospec-tive review of performance on the PPVT–4 with 100 kindergarten ELLs who were attending public schools in northern Florida. The ELL group consisted of 41 boys and 59 girls who ranged in age from 3;4 to 7;2, with a mean age of 65.1 months (approximately 5;5). The participants were sequential-language learners, with Spanish spoken as the L1 at home; however, the children likely varied in the amount of English ex-posure at home depending on the English proficiency of their siblings and parents and television exposure, which was not quantified due to the retrospective nature of the exploratory study. All of the ELLs were from low socioeconomic backgrounds, indicated by eligibility for the free lunch program. All of the children participated in annual well-child checkups and hearing screenings through the local migrant education program. The participants did not have any known medical diagnoses or identified speech-lan-guage or sensory disorders, and none had received any special education classification services.

Although monolinguals were not of specific inter-est to this study, because normative data and item analysis of monolinguals is readily available in the test norms, a regional influence could not be ruled out without the inclusion of local English-speaking monolinguals. Therefore, we included a comparison group of 25 monolingual kindergarten children from the same county and school district. The first 25 monolingual children to return consent forms were included in the study. No one was excluded from participation. The comparison group consisted of 11 boys and 14 girls who ranged in age from 4;3 to 6;6,


with a mean age of 61.5 months (approximately 5;0). Of the English monolingual comparison group, 15 were African American and 10 were Caucasian. Ninety percent had attended preschool before kindergarten, and 42% received free or reduced lunch. None of the participants had any known medical diagnoses or identified communication disorders. Signed consent for participation was obtained from a parent or guardian.

We subsequently replicated the exploratory study in a prospective inquiry with a new group of 51 ELLs in kindergarten from the same school district. Invitations were distributed to kindergarten class-rooms in four elementary schools in the same school district as the initial exploratory study. Signed con-sent was received for 55 children; however, four were later excluded after phone interviews indicated that the four households reportedly did not use any Span-ish at home on weekday evenings or weekends. The final group of ELLs in the replication study were all in kindergarten and consisted of 25 boys and 26 girls who ranged in age from 5;1 to 6;11, with a mean age of 68 months (approximately 5;8). All of the ELLs were from low socioeconomic backgrounds and quali-fied for the free lunch program. None of the partici-pants had any known medical diagnoses or identified speech-language or sensory disorders, and none had received any special education classification services.

Monolingual English-speaking children who were eligible for free and reduced lunch were specifi-cally targeted from the same partnering schools in an attempt to facilitate homogeneity between the two groups on socioeconomic status. The first 50 mono-lingual children in kindergarten who returned the con-sent form were included in the study. The comparison group consisted of 25 boys and 25 girls who were 51–83 months old, with a mean age of 67 months (SD = 8 months). Of the English monolingual com-parison group, 38 were African American, 11 were Caucasian, and 2 were Hispanic. None of the partici-pants had any known medical diagnoses or identified communication disorders.

Research assistants conducted phone interviews with all Spanish–English dual-language learners to gather detailed information regarding the percentage of Spanish and English use after school on weekdays and throughout the weekend. We used a standard par-ent interview that was consistent with the procedures protocol of a screening tool detailed by Bedore et al. (2012). As part of the protocol, we asked the parents to report on their children’s current use of English and Spanish and asked them to describe language input during a typical weekday (hour by hour) and similarly for a weekend day. The percentage of use was derived using the procedures described in previ-ous studies (Bedore et al., 2012).

For any segments of the day for which the parents reported “both Spanish and English use” occurred, we divided equally between Spanish and English. To compute the percentage of Spanish and English use, weekday data was weighted by five, weekend data was weighted by two, and the sum was divided by the total waking hours in a weekday and weekend combined. Children ranged in the percent-age of Spanish use from 2.19% to 63.54%, averaging 42.00% (SD = 15.3).

MeasuresThe PPVT–4 was administered to all of the partici-pants. The PPVT–4 is a norm-referenced, single-word receptive vocabulary test. We chose to use the PPVT–4 because it has been identified as the most frequently used standardized assessment measure with bilingual students (Caesar & Kohler, 2007). This test contains 228 test items that cover 20 categories of content areas and parts of speech (PPVT–4 Publica-tion Summary Form). On this test, the examiner pro-duces a word and the child is asked to point to the picture that best represents the word, given a choice of four pictures. As the examiner proceeds through consecutive items on the test, the test is designed to become increasingly difficult, with lexical items be-coming more difficult and/or less common or famil-iar. The test was normed on 536 African Americans, 546 Hispanics, 2,244 White Caucasians, and 214 indi-viduals classified as “other” but including American Indians, Alaska Natives, Asian Americans, and Pacific Islanders.

Word familiarity. The CELEX word form da-tabase was used for word-form frequency values (Baayen, Piepenbrock, & van Rijn, 1993). The word frequency is intended to reflect the regularity, fre-quency, or likelihood that a child would be exposed to a specific word. The CELEX word frequency values use a database of 17.9 million tokens to identify the number of times the word occurs in text and phone conversations per 1 million words. For the items on the PPVT–4, words occurring more frequently (e.g., foot and mouth) have a high word familiarity index (116.59 and 134.69, respectively). In contrast, less frequently occurring words such as turtle and juggling have a low word familiarity index (2.18 and 0.95, respectively).

Cognate status. We assigned each item on the PPVT a value relating to cognate status based on a systematic scale of phonological similarity, the Crosslinguistic Overlap Scale for Phonology (COSP; Kohnert et al., 2004). Using the scale, we assigned numeric values across categories based on the amount of phonological similarity between the English and


Spanish equivalent. Some word pairs received a small total number of points, sharing only the initial phoneme, for example. Other items on the PPVT, in contrast, shared overlapping consonants, had the same number of syllables, and had 80% overlapping vowels. These pairs demonstrated a higher degree of cognate status (e.g., a panda – un panda, cactus – cactus) based on the systematic scale in the pub-lished literature (Kelley & Kohnert, 2012; Kohnert et al., 2004).

For this task, the second author, a native Spanish speaker, completed the initial translations. The initial coding using the scale produced 20 cognates with COSP scores as high as 10. Independent assignment of points by a separate translator, however, produced several different word translations. For example, the item banana was routinely translated in the local area as plátano, banana, and banano. It was noted that the parents of our child participants were also from dif-ferent dialectal backgrounds and may have modeled a variety of translation equivalents. Due to difficulty ensuring consensus on the most common transla-tion of the target words for children given different dialectal influences in Florida, we kept the cognate status variable as a categorical variable in the current study. As a categorical variable, any PPVT item that demonstrated phonological similarity in one or more categories on the COSP was designated as a Span-ish–English cognate because the pairs demonstrated phonological similarity.

ProcedureFor the exploratory study, we reviewed the children’s records from the previous year. All of the items on the PPVT–4 that the children attempted (between the basal and ceiling) were included and were entered in a database for monolingual and ELL participants. For the ELL group, a total of 72 items on the PPVT–4 (specifically, items 1 through 72) were entered in a database. In the replication study 2 years later, data were gathered prospectively, allowing the collection of language experience and exposure through parent interviews. Aside from the addition of the thorough phone interviews with families, the materials, proce-dures, and analyses described in the preliminary study were replicated in the follow-up study.

RESuLTS of RETRoSPECTIVE ExPLoRAToRY STudY

We computed descriptive statistics for the 100 chil-dren who were ELLs as well as for the comparison group of monolingual peers. The overall standard

score on the PPVT–4 for the ELLs ranged from 31 to 110 (M = 77.35; SD = 14.73. This means that the mean standard score for the children in the sample was below the average expected limits (85–115 is considered average range); however, the children had not been identified as having any existing language-learning disorders. The standard scores of the mono-lingual participants ranged from 80 to 142 (M = 103.68; SD = 16.24).

We conducted a regression analysis to evaluate how item-level difficulty predicted the percentage of errors for the monolinguals. An analysis of 143 items (between basal and ceiling) indicated that the dif-ficulty level of the items was significantly associated with the percentage of errors, F(1, 142) = 1424.12, p < .001. The effect size, as estimated by adjusted R², was 0.91, indicating that item difficulty (numeric level) on the test explained a large proportion (91%) of the variance in accuracy for monolinguals. This relationship is evidenced in the monolinguals’ re-sponses, as depicted in Figure 1.

Similarly, we conducted a regression analysis to evaluate how item-level difficulty predicted the percentage of errors for the ELLs. Item-level diffi-culty on the 71 test items between basal and ceiling was a significant predictor of accuracy, F(1, 70) = 54.89, p < .001, but did not explain as much of the variance in performance as it did for the monolin-guals. The adjusted R² value was 0.43, indicating that the intended difficulty level accounted for a smaller portion (43%) of the predicted variance in the rate of errors for the ELLs. A scatterplot of the percentage of errors predicted by the intended difficulty level is provided in Figure 2 for the ELLs.

In an attempt to further explain additional vari-ability in the distribution of errors for the ELLs, we conducted a follow-up examination of other potential influencing factors. Visual inspection of the outli-ers was conducted (see Figure 2) to identify indi-vidual items that demonstrated an unexpectedly high percentage of errors, as well as those reflecting an unexpectedly low percentage of errors. Seven items appeared to be outliers. These items are numbered on Figure 2 with their corresponding item number on the PPVT–4. Items that did not have specific transla-tion equivalents in Spanish (e.g., buckle, toe) had a higher expected percentage of errors compared to the PPVT–4 rank-ordered difficulty ranking. Two items that were missed less frequently than expected com-pared to the test’s rank-ordered difficulty were close cognates in Spanish (e.g., diamond–diamonte and penguin–penguino).

Given that some outliers appeared to be cognates, we conducted an analysis of variance (ANOVA) to examine if there were differences in the rate of errors


figure 1. Scatterplot of the relationship between item difficulty (order) and error rate for the monolingual English-speaking comparison group (n = 25).

figure 2. Scatterplot of the relationship between item difficulty (order) and error rate for the Spanish–English dual-language learners (n = 100). Numbers were used to highlight item numbers on the Peabody Picture Vocabulary Test, Fourth Edition (PPVT–4) that appeared to be outliers in error rate for the English-language learners (ELLs) based on the intended item difficulty level.


between cognates and noncognates. No significant effect of cognate status on the mean percentage of errors was found at the .05 level, F(1, 70) = .024, p = 0.878. The mean percentage incorrect across all noncognates was 29.9% (SD = 17.5), which was not significantly different than the children’s performance on cognates (M = 29.3%; SD = 16.3). However, the relatively small number of items that were cognates (n = 20) resulted in limited power to detect a sig-nificant difference. Additionally, the distribution of difficulty level across cognates and noncognates may have been uneven. Data on the individual cognate items are provided in Table 1 (indicated in bold),

with corresponding difficulty level and percentage of error. Cognate items had an average intended dif-ficulty level of 41.80 (SD = 22.9). Noncognate items ranged similarly in intended difficulty (M = 34.46; SD = 20.0).

Last, we examined the relationship between word familiarity index values and children’s error rates. For ELLs, English word familiarity index values showed a significant but small negative correlation, with er-ror rate r = –0.30 (p < .001). In other words, higher word familiarity index values were associated with a lower percentage of errors or incorrect responses. Refer to Table 1 for a complete list of familiarity

Table 1. Word familiarity index (WFI) and percentage of errors for the Peabody Pic-ture Vocabulary Test, Fourth Edition (PPVT–4) items.

Item # WFI % Incorrect Item # WFI % Incorrect

1 92.96 5.9 37 66.15 22.52 71.73 8.8 38 3.85 0.03 12.74 26.5 39 31.28 35.04 116.59 23.5 40 10.17 20.05 10.95 26.5 41 1.06 41.36 4.13 8.8 42 15.64 28.87 14.47 11.8 43 0.95 26.38 60.84 17.6 44 9.22 36.39 67.09 23.5 45 32.35 28.810 64.53 5.9 46 68.16 11.311 27.71 2.9 47 22.35 41.312 134.69 20.6 48 46.42 62.513 15.75 6.1 49 0.50 53.114 1.68 12.2 50 7.65 18.515 8.72 34.7 51 13.24 53.116 2.18 18.4 52 53.69 45.717 188.94 20.4 53 18.77 39.518 17.49 18.4 54 7.82 16.019 2.51 20.4 55 7.37 33.320 180.56 16.3 56 2.46 70.421 9.61 49.0 57 1.40 35.822 21.17 40.8 58 0.84 6.223 50.95 30.6 59 4.86 40.724 54.86 22.4 60 8.10 39.525 33.07 14.7 61 27.09 44.126 9.66 16.0 62 33.07 63.227 13.02 21.3 63 10.34 63.228 21.28 30.7 64 11.28 60.329 92.12 20.0 65 5.03 45.630 22.29 25.3 66 2.12 35.331 81.96 41.3 67 6.42 38.232 135.36 10.7 68 18.27 48.533 152.79 14.7 69 3.24 48.534 59.22 13.3 70 27.88 50.035 3.52 18.7 71 13.07 47.136 23.41 30.7 72 3.35 66.2

Note. For the WFI, high numbers represent more familiar words based on the frequency of occurrence in children’s literature. English–Spanish cognates are identified by bold type.


values by item difficulty and child performance. An interaction between word familiarity, difficulty level, and cognate status could not be ruled out given the small number of cognate items (n = 20).

RESuLTS of REPLICATIon STudY

The results of the initial study proved interesting; however, the ability to draw conclusions was limited because the retrospective nature of the study nega-tively affected the feasibility of gathering additional ethnographic data on the ELLs, specifically regard-ing their use of language at home. To strengthen the research inquiry, we replicated the study.

We computed descriptive statistics for 51 addi-tional children who were ELLs as well as the compari-son group of 50 monolingual peers. The raw scores for the ELLs ranged from 16 to 108 (M = 67.04; SD = 18.60), and their overall standard score ranged from 44 to 106, with a mean of 82.18 (SD = 12.10). The standard scores of the monolingual participants ranged from 78 to 142 (M = 100.18; SD = 14.90).

Next we examined the relationship between the percentage of errors and the level of item difficulty for each group and found a strong correlation (r = .90) between item-difficulty level and percentage of er-rors for the monolinguals. We conducted a regression analysis to evaluate how item level difficulty predicted the percentage of errors. For the monolinguals, an analysis of item-level percentage accuracy indicated that the difficulty level of items was significantly associated with the percentage of errors, F(1, 142) = 607.89, p < .0001. The effect size, as estimated

by adjusted R², was .81, indicating that item dif-ficulty (numeric level) on the test explained a large proportion (81%) of the variance in accuracy for the monolinguals. Similarly, we conducted a regression analysis to examine the intended difficulty level as a predictor of errors for the ELLs. Item-level difficulty on the same test items was a significant predictor of accuracy, F(1, 106) = 123.46, p < .0001, but did not explain as much variance in performance as it did for the monolinguals. The adjusted R² value was 0.53, indicating that the intended difficulty level accounted for a smaller portion (53%) of the predicted vari-ance in the rate of errors for ELLs when compared to monolinguals (see Figure 3).

Item Analysis of Errors In an attempt to further explain additional variability in the distribution of errors for the ELLs, we con-ducted a follow-up examination of other potential influencing factors. Similar to the exploratory study, items that did not have specific translation equiva-lents in Spanish (e.g., buckle, toe) had a higher ex-pected percentage of errors compared to the PPVT–4 rank-ordered difficulty ranking. For example, buckle is relatively easy in difficulty listing, but 68% of the respondents answered incorrectly. Similarly, toe is ranked as an easier item, number 21 in the progres-sion of difficulty and among the starting set of items intended for 4-year-old participants, yet 30% of the ELLs who were administered that item responded incorrectly.

Other potential explanations for errors were ex-plored, including cognate status, word familiarity, and

figure 3. Replication study of item-level performance between groups of ELLs and monolingual English speakers.


the context of words (e.g., home or school environ-ments). We conducted an ANOVA to examine if there were differences in the rate of incorrect responses between cognates and noncognates. There was no significant effect of cognate status on the mean ac-curacy of responses at the .05 level, F(1, 70) = .053, p = 0.818. We examined the relationship between word familiarity index values and children’s accuracy for the replication data set. For the ELLs, English word familiarity index values showed a significant but small correlation with the percentage of correct responses, r = .36 (p = .002). In other words, higher word familiarity index values were associated with higher percentages of correct responses.

Last, we considered the context of home and school environment. Not all of the items could be clearly or reliably designated by environment. Numer-ous items (n = 50), such as happy, ball, and fence, transcend across both environments and were there-fore excluded in the comparison. Using the word categories identified by Bialystok et al. (2012), items found at school (n = 31) included items common to school routines (e.g., pencil, bus, whistle), shapes, animals, and instruments. The home category (n = 28) included items common to home routines (e.g., lamp, dressing), foods (e.g., banana), body parts, and clothing (e.g., shoes, belt). There was a significant difference for ELLs in accuracy on home-designated words (M = 64% correct, SD = 22) compared to school-designated words (M = 77% correct, SD = 19.1), F(1, 57) = 5.962, p = 0.018. Several school-designated words appeared to be associated with a higher percentage of accuracy in English. For exam-ple, violin is listed as number 73 in difficulty level out of the 108 items; however, 30 out of 37 ELLs (81%) responded correctly.

dISCuSSIon

Construct validity of a standardized assessment measure is supported by the fact that the test items become progressively more difficult as the items advance. This expected trend was demonstrated for the monolingual participants, whose error rate showed a strong positive relationship with intended difficulty level in both the exploratory and replication study. In other words, as the examiner advanced through items on the test, the items became increasingly more dif-ficult in fairly even increments. In contrast, the cor-relation between intended difficulty level and percent-age of errors was weaker for the ELLs. As evidenced in both the exploratory and replication study, lexical items did not advance as evenly or consistently in difficulty as observed for the monolingual children.

Recognizing that the PPVT–4 does not claim to be designed specifically for ELLs, potential weaknesses in construct validity for ELLs are consistent with an earlier study by Restrepo and Silverman (2001), which found discrepancies in the difficulty levels of the items on the Spanish PPVT–III when translated from the original English version.

The discrepancies in difficulty level for the ELLs across the progression of items substantiates the need for lexical considerations, including cultural and linguistic bias in item selection, and the interplay between languages in word recognition and identifica-tion. In the item pool of the PPVT–4 used in the cur-rent study, the interplay between Spanish and English would be expected to make some items easier or hard-er than intended. In some cases, Spanish similarities appeared to facilitate recognition of the desired lexical item. For other items, Spanish knowledge may have provided conflicting lexical information or competition in word recognition and identification caused by shared features of the distractor items in Spanish and false translation equivalents. These findings are consistent with the theoretical assumptions of the RHM (Kroll & Stewart, 1994), which emphasizes the interplay of children’s dual-language knowledge with transfer between the L1 and L2. In interpreting the findings of the current study, it appears reasonable to consider that young ELL participants’ access to English vocabulary was filtered through a Spanish lens or was, to some extent, affected by their knowledge of Spanish.

Although both the exploratory and replication studies identified differences in difficulty congruence between monolinguals and ELLs, the difference was notably greater in the exploratory study than in the replication study. Possible explanations may include the fact that the participants were more homogeneous in the replication study, particularly in terms of age. The exploratory study was retrospective in nature and included some 4-year-olds in the review of records. In contrast, the replication study was restricted to children who were currently enrolled in kindergar-ten. It is possible that the contrast in performance between the ELLs and monolinguals is greater in preschool than in kindergarten. Another consideration in the replication study was the possible inclusion of children with undiagnosed language learning disor-ders. In the monolingual comparison group, the range in standard score performance was 78–142, suggest-ing that there was at least one outlier with below-normal expectations for TD monolinguals.

The below-average standard score performance of the ELLs in both the exploratory and replication stud-ies proved to be consistent with predicted and previ-ous findings in the literature that Spanish-speaking ELLs tend to score below average on standardized


tests (Bialystok et al., 2012; Jackson, Schatschneider, & Leacox, 2014; Jackson-Maldonado, 1999). Possible explanations for the below-average scores of the ELLs overall cannot be derived from the current study; how-ever, explanations supported by the previous literature include the low socioeconomic background of partici-pants (Hart & Risley, 1995), the fact that English-pub-lished assessment measures are not designed to take into account the interplay between languages (Gathor-cole & Moawad, 2010), and the low representation of bilingual Hispanic Latino individuals on the PPVT–III normed sample (Restrepo & Silverman, 2001).

Although the cause(s) of the observed incongru-ence in difficulty level for ELLs cannot be deter-mined from the current study, the descriptive results lend insights into factors to consider, including cog-nate status, word familiarity, and context. Research studies have indicated that cognates may facilitate bilingual word processing by simultaneously activat-ing both languages even though the stimulus word is presented in only one language (Hermans, Bongaerts, de Bot, & Schreduer, 1998; Kroll et al., 2008). After performing the cognate analysis, it appeared that the difficulty level of the test items did not differ based on cognate status. Instead, the items were fairly evenly difficult. The lack of a significant finding may be due to the fact that there were relatively few cog-nates (n = 20) compared to noncognates (n = 52). In addition, the cognates varied in how many phonotac-tic properties were shared. A few were close cognates (penguin–pinguino), whereas other word pairs shared only the initial consonant, which would be interesting to consider on a continuum of overlap in a follow-up researcher-designed probe of cognates. This study only focused on the presence or absence of cognates in the item pool (scored 1 or 0, respectively); there-fore, the results may underestimate the effect of cross-linguistic cognates on ELLs’ performance.

An alternative explanation of the incongruence in difficulty level across items may be offered by the difference in accuracy observed between home and school words. The finding that words associated with school had a higher percentage of recognition in English than words associated with home substanti-ated the recent findings of Bialystok et al. (2012). Although the Bialystok et al. study was not specific to the interplay between Spanish and English, it high-lighted the notion that dual-language learners distrib-ute their lexicon across multiple languages that may be environmentally influenced by the primary context in which the lexical item occurs. This explanation has been highlighted in other studies, suggesting that ELLs know fewer words in each language exclusively because their vocabulary is distributed across two languages (Oller, Pearson, & Cobo-Lewis, 2007).

LimitationsResults of the current study should be interpreted cautiously due to several limitations. The ELLs in this study were from low socioeconomic back-grounds; therefore, results may not be generalizable to the performance of other ELLs from middle or high socioeconomic backgrounds. It is possible that the combination of being a dual-language learner and from a low socioeconomic background had a unique additive effect that would not be present with either of the conditions alone. In addition, the Spanish-lan-guage assessment data indicated that participants in the ELL group were generally slightly below average in their Spanish skills compared to their monolin-gual Spanish-speaking peers. Therefore, it cannot be assumed that the results are representative of the per-formance of children with different levels of Spanish ability. In a subsequent study, it would be interesting to examine performance on the cognate items across children representing various levels of Spanish skills.

Perhaps the biggest limitation, however, was that we restricted our focus to lexical items on one standardized vocabulary test that was widely used in school practice but was not constructed to measure the lexical interplay between English and Spanish and did not have a large number of opportunities for cognate influence. Also, the degree of each item’s cognate status was not considered in this study due to the small number of cognates. In other words, some items were very close cognates and shared all but one phoneme (e.g., dentist and dentista), whereas others only shared one phoneme (e.g., red and rojo). This variance in degree of overlap of cognates has been examined in other studies (Kohnert et al., 2004) and would be important to consider in constructing a researcher-made assessment instrument.

Implications

Despite noted limitations, the results of the current study substantiate the need to carefully consider con-struct validity when using tests designed for monolin-guals with children from culturally and linguistically diverse backgrounds. The findings support the need for SLPs to interpret test scores cautiously and to reconsider the exclusive use of standardized English-language assessment scores for children from cultur-ally and linguistically diverse backgrounds (Gutierrez-Clellen & Peña, 2001; Gutierrez-Clellen et al., 2000; Peña, 1992).

Given the uneven difficulty of specific lexical items for ELLs on the standardized test, one sug-gestion for clinicians that has been offered in the literature previously is to consider testing beyond the


ceiling in order to gather additional descriptive infor-mation about word knowledge (Peña, 1992).

The current findings substantiate recommenda-tions of other studies suggesting that SLPs should exercise caution in reporting standard scores from tests that have been normed on monolinguals, given that the construct validity of the test may not be as strong for Spanish–English-speaking ELLs. Recog-nizing that standardized test performance may not accurately reflect a bilingual client’s ability, the use of dynamic assessment measures (Gutierrez-Clel-len & Peña, 2001) with ELLs is also encouraged in the literature and supported by the current findings. Although effective solutions or adaptations cannot be derived from the current descriptive study alone, sev-eral options for adaptations recommended in previous literature are highlighted below:

• Consider language sampling and other alterna-tive authentic assessment techniques (Banerjee & Guiberson, 2012; Gutierrez-Clellen et al., 2000; Notari-Syverson, Losardo, & Lim, 2003).

• Examine specific errors of ELLs on standardized tests to consider possible transfer or interference of the child’s dual lexicon (Peña, 1992).

• Conduct informal dynamic assessment measures and test–teach–retest procedures in addition to standardized assessment measures (Gutierrez-Clellen & Peña, 2001).

• Employ recommended practices by assess-ing ELLs’ performance in both languages and through the lens of both cultures (American Speech-Language-Hearing Association, 1999; Banerjee & Guiberson, 2012; Individuals with Disabilities Education Act, 2004).

REfEREnCES

American Speech-Language-Hearing Association. (1999). Guidelines for the roles and responsibilities of school-based speech-language pathologists. Rockville, MD: Author.

Baayen, H., Riepenbrock, R., & van Rijn, H. (1993). The CELEX lexical database [CD-ROM]. Philadephia, PA: University of Pennsylvania, Linguistic Data Consortium.

Bagnato, S. J. (2005). The authentic alternative assessment in early intervention: An emerging evidence-based prac-tice. Journal of Early Intervention, 28, 17–22. doi:10.1177/105381510502800102

Bagnato, S. J., Neisworth, J. T., & Pretti-Frontczak, K. (2010). LINKing authentic assessment and early child-hood intervention: Best measures for best practices (2nd ed.). Baltimore, MD: Brookes.

Banerjee, R., & Guiberson, M. (2012). Evaluating young children from culturally and linguistically diverse

backgrounds for special education services. Young Excep-tional Children, 15(1), 33–45. doi:10.1177/1096250611435368

Bedore, L. M., Peña, E. D., Summers, C. L., Boerger, K. M., Resendiz, M. D., Greene, K., … Gillam, R. B. (2012). The measure matters: Language dominance profiles across measures in Spanish–English bilingual children. Bilingualism: Language and Cognition, 15(3), 616–629. doi:10.1017/S1366728912000090

Bialystok, E., Luk, G., Peets, K. F., & Yang, S. (2012). Receptive vocabulary differences in monolingual and bi-lingual children. Bilingualism: Language and Cognition, 13(4), 525–531. doi:10.1017/S1366728909990423

Caesar, L. G., & Kohler, P. D. (2007). The state of school-based bilingual assessment: Actual practice versus recommended guidelines. Language, Speech, and Hearing Services in Schools, 38, 190–200. doi:10.1044/0161-1461 (2007/020)

Crais, E. R. (1995). Expanding the repertoire of tools and techniques for assessing the communication skills of in-fants and toddlers. American Journal of Speech-Language Pathology, 4, 47–59.

Crais, E. R. (2011). Testing and beyond. Strategies and tools for evaluating and assessing infants and toddlers. Language, Speech, and Hearing Services in Schools, 42, 341–364. doi:10.1044/0161-1461(2010/09-0061)

Cummins, J. (1984). Wanted: A theoretical framework for relating language proficiency to academic achievement among bilingual students. In C. Rivera (Ed.), Language proficiency and academic achievement (pp. 2–19). Clev-edon, UK: Multilingual Matters.

deGroot, A. M. B., & Hoeks, J. C. J. (1995). The devel-opment of bilingual memory: Evidence from word trans-lation by trilinguals. Language Learning, 45, 683–724.

Division for Early Childhood of the Council for Excep-tional Children. (2007). Promoting positive outcomes for children with disabilities: Recommendations for curricu-lum, assessment, and program evaluation. Retrieved from www.decsped.org/uploads/docs/about_dec/position_con-cept_papers/Prmtg_Pos_Outcomes_Companion_Paper.pdf

Dunn, L. M., & Dunn, L. M. (1981). Peabody Picture Vocabulary Test: Revised. Circle Pines, MN: AGS.

Dunn, L. M., & Dunn, L. M. (1997). Peabody Picture Vocabulary Test, Third Edition. Circle Pines, MN: AGS.

Dunn, L. M., & Dunn, D. M (2007). Peabody Picture Vocabulary Test, Fourth Edition manual. Bloomington, MN: NCS Pearson.

Elston-Güttler, K., & Williams, J. N. (2008). L1 poly-semy affects L2 meaning interpretation: Evidence for L1 concepts active during L2 reading. Second Language Research, 24, 167–187.

Gathercole, V. C., & Moawad, R. A. (2010). Semantic interaction in early and late bilinguals: All words are not created equally. Bilingualism: Language and Cognition, 13(4), 385–408. doi:10.1017/S1366728909990460

Gutierrez-Clellen, V. F., & Peña, E. D. (2001). Dynamic assessment of diverse children: A tutorial. Language,


Speech, and Hearing Services in Schools, 32, 212–224. doi:10.1044/0161-1461 (2001/019)

Gutierrez-Clellen, V. F., Restrepo, M. A., & Bedore, L. M. (2000). Language sample analysis in Spanish-speak-ing children: Methodological considerations. Language, Speech, and Hearing Services in Schools, 31, 88–98.

Hart, B., & Risley, T. R. (1995). Meaningful differences in the everyday experiences of young American children. Baltimore, MD: Brookes.

Hermans, D., Bongaerts, T., de Bot, K., & Schreduer, R. (1998). Producing words in a foreign language: Can speakers prevent interference from their first language? Bilingualism: Language and Cognition, 1(3), 213–229.

Individuals with Disabilities Education Act of 2004, Pub. L. No. 108-446, 118 Stat. 2647 (2004).

Jackson, C. W., Schatschneider, C., & Leacox, L. (2014). Longitudinal analysis of receptive vocabulary growth in young Spanish English-speaking children from migrant families. Language, Speech, and Hearing Services in Schools, 45, 40–51. doi:10.1044/2013_LSHSS-12-0104

Jackson-Maldonado, D. (1999). Early language assess-ment for Spanish-speaking children. Bilingual Review, 24, 35–53.

Kaderavek, J. N. (2010). Language disorders in children: Fundamental concepts of assessment and intervention. Upper Saddle River, NJ: Allyn & Bacon.

Kelley, A., & Kohnert, K. (2012). Is there a cognate advantage for typically developing Spanish-speaking English-language learners? Language, Speech, and Hear-ing Services in Schools, 43, 191–204. doi:10.1044/0161-1461(2011/10-0022)

Kester, E. S., & Peña, E. D. (2002). Limitations of current language testing practices for bilinguals. College Park, MD: ERIC Clearinghouse on Assessment and Evaluation.

Kohnert, K., Windsor, J., & Miller, R. (2004). Crossing borders: Recognition of Spanish words by English-speak-ing children with and without language impairment. Applied Psycholinguistics, 25, 543–564. doi:10.1017/S0142716404001262

Kroll, J. F., Gerfen, C., & Dussias, P. E. (2008). Labora-tory designs and paradigms: Words, sounds, and sen-tences. In L. Wei & M. G. Moyer (Eds.), The Blackwell guide to research methods in bilingualism and multilin-gualism (pp. 108–131). Malden, MA: Blackwell.

Kroll, J. F., & Stewart, E. (1994). Category interference in translation and picture naming: Evidence for asym-metric connections between bilingual memory representa-tions. Journal of Memory and Language, 33, 149–174. doi:10.1006/jmla.1994.1008

Laing, S. P., & Kamhi, A. (2003). Alternative assessment of language and literacy in culturally and linguistically diverse populations. Language, Speech, and Hearing Services in Schools, 34, 44–55. doi:10.1044/0161-1461 (2003/005)

McCarthy, D. (1970). The McCarthy Scales of Children’s Abilities. New York, NY: The Psychological Corporation.

National Center for Education Statistics. (2004). The

condition of education. Retrieved from http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2004077

Notari-Syverson, A., Losardo, A., & Lim, Y. S. (2003). Assessment of young children from culturally diverse backgrounds: A journey in progress. Assessment for Ef-fective Intervention, 29(1), 39–51. doi:10.1177/073724770302900105

Oller, D. K., Pearson, B. Z., & Cobo-Lewis, A. B. (2007). Profile effects in early bilingual language and literacy. Applied Psycholinguistics, 28, 191–230.

Peña, E. (1992). The application of dynamic methods to language assessment: A nonbiased procedure. The Journal of Special Education, 26(3), 269–280.

Potter, M. C., So, K., Von Eckardt, B., & Feldman, L. B. (1984). Lexical and conceptual representation in beginning and proficient bilinguals. Journal of Verbal Learning and Verbal Behavior, 23, 23–38.

Prior, A., MacWhinney, B., & Kroll, J. F. (2007). Trans-lation norms for English and Spanish: The role of lexical variables, word class, and L2 proficiency in negotiating translation ambiguity. Behavior Research Methods, 39, 1029–1038. doi:10.3758/BF3193001

Restrepo, M. A., & Silverman, S. W. (2001). Validity of the Spanish Preschool Language Scale—3 for use with bilingual children. American Journal of Speech-Language Pathology, 10, 382–393. doi:10.1044/1058-0360(2001/032)

Saenz, T. I., & Huer, M. B. (2003). Testing strategies involving least biased assessment of bilingual children. Communication Disorders Quarterly, 24(4), 184–193. doi:10.1177/15257401030240040401

Sattler, J., & Altes, L. (1984). Performance of bilingual and monolingual Hispanic children on the Peabody Picture Vocabulary Test—Revised and the McCarthy Perceptual Performance Scale. Psychology in the Schools, 21, 313–316.

Senaga, N., & Inglebret, E. (2003, November). Assessment practices used by bilingual SLPs. Paper presented at the meeting of the American Speech-Language-Hearing As-sociation, Chicago, IL.

Tokowicz, N., Kroll, J. F., de Groot, A. M. B., & van Hell, J. G. (2002). Number of translation norms for Dutch–English translation pairs: A new tool for examin-ing language production. Behavior Research Methods, Instruments, and Computers, 34, 435–451. doi:10.3758/BF03195472

U.S. Bureau of the Census. (2006). 2006 American com-munity survey. Retrieved from http://factfinder.census.gov

von Studnitz, R. E., & Green, D. W. (2002). Interlingual homograph interference in German–English bilinguals: Its modulation and locus of control. Bilingualism: Language and Cognition, 5(1), 1–23.

Contact author: Carla Wood, Florida State University, School of Communication Science and Disorders, 201 W. Bloxham, Tallahassee, FL 32306. E-mail: [email protected]

Documents

Lexical Considerations for Standardized Vocabulary Testing ... · phonological or orthographic form, such as penguin in English and pinquino in Spanish (Kohnert, Wind-sor, & Miller,