Embed Size (px)
Citation preview
Corpus‐based vocabulary testing in an English for Academic Purposes context
Dr. Magdolna Lehmann
Institute of English Studies, University of Pé[email protected]
Rationale
First‐year students of English seem to have difficulties
• in text comprehension and academic writing
• fulfilling the course requirements
• Role of vocabulary in reading comprehension
(Laufer, 1997; Nation, 1993a)( , ; , )
• Vocabulary and academic success• Vocabulary and academic success(Morris & Cobb, 2003)
H th i l k f ffi i t d t b lHypothesis: lack of sufficient or adequate vocabulary knowledge
Aims of the studyAims of the study
• what words English majors need to be familiar withto be successful in their studies
• Corpus‐based selection of relevant lexical items for tour assessment purposes
BackgroundBackground
• Usefulness of a word = its frequency of occurrence• Usefulness of a word = its frequency of occurrence (Nation &Waring, 1997)
‐ high‐frequency vocabulary (GSL)
– low‐frequency vocabulary
– specialized vocabularyp y
– sub‐technical vocabulary (AWL)(Read 2000 p 159)(Read, 2000, p. 159)
• the first 2K GSL + 570 word families of AWL:
92% of spoken language
84% of newspaper language (Nation, 2006)p p g g ( )
• to achieve reasonable comprehension:• to achieve reasonable comprehension:
98% of all the words in any text must be known
(Hirsch & Nation, 1992; Laufer, 2005a, 2005b; Nation & Waring, 1997)
• “one in every 50 words is unknown” (Schmitt, 2008, p. 330)
How many words?
• Graduate native speakers:13 200 20 700 d– 13,200‐20,700 words (Goulden, Nation & Read, 1990, p. 341)
• Undergraduate native speakers:– 14,000 – 17,000 words (D'Anna, Zechmeister & Hall, 1991; Nusbaum, Pisoni & Davis, 1984; Zechmeister, D'Anna, Hall, Paus & Smith, 1993)
EFL learners?
Dutch students of English
10,000 words minimally
for good reading comprehension(Hazenberg & Hulstijn,1996)(Hazenberg & Hulstijn,1996)
Participants
Hungarian native speakersstudents of English at UP
Preliminary study
Vocabulary size of 1st‐year English majors at UP
• 93 students
• 2003 Autunm semester
• Receptive knowledge of 4,000 – 14,000 words
(range 8‐28/50)
Mean: 7,100 words,
If minimum is:
1) 5,000 words (Laufer, 1987, 1992a, 1992b) :
→ 15% of our students face difficulties → %in reading comprehension
2) 10,000 words (Hazenberg & Hulstijn, 1996)→ 94% fall below this level→ 94% fall below this level
Instruments
Higher levels of proficiency:
vocabulary size + active/passive control y /p
are better predictors than qualitative analysisare better predictors than qualitative analysis
(Zareva, 2005, p. 560)
• Other considerations:
– filtering students with inadequate vocab size
– practicalityp y
– objectivity
computerized item analysis– computerized item analysis
– ease of scoring
– ease of administration
Vocabulary Levels Test(Nation, 1990; Schmitt, Schmitt & Clapham, 2001)
1 area
2 contract __2___ written agreement
3 definition __5___ way of doing something
4 evidence __4___ reason for believing
5 method something is or is not true
6 role
Procedures
1. Academic vocabulary (AWL) ‐ 220 studentsVocabulary Levels Test (Nation, 1990; Schmitt, Schmitt & Clapham, 2001)
2. Low‐frequency vocabulary (10k) ‐211 students
3. Words perceived as useful in English studies – 135 st
4. Corpus‐based test battery – 134 students
Lessons learnt from phases 1‐3
• AWL did not discriminate well
• 10k level contained irrelevant vocabulary
• Intuitive judgements of word frequency not as j g q yaccurate as corpus‐based counts (McCrostie, 2007)
→Solution:
C b d b lCorpus‐based vocabulary assessment
Compiling the corpus
• representative sample of authentic texts given as compulsory readings in the courses
• Corpus of Readings in English Studies (CORES)
• computerized
• three sub‐corpora
CORES
• monolingual• monolingual • written • L1 • dynamic and developing, as it is open fordynamic and developing, as it is open for continuous expansion
• un annotated• un‐annotated(based on Horváth’s (2001, p. 44) matrix of corpus typology)
Descriptive statistics of the frequency of lexis in CORES
CORES Families Types Tokens %
K1 Words (1-1000) 963 2,798 88,183 71.87
Function words (52,316) (42.64)
Content words (35,867) (29.23)
K2 Words (1001-2000): 733 1,479 6,508 5.30
1K+2K: 1,696 4,277 94,691 (77.17)
AWL Words (academic): 545 1,630 9,251 7.54
Off-List Words: ? 6,945 14,866 15,29
Overall 2,241+? 12,848 118,808 100.00
Rank orders and frequencies of the top 10 words in three corpora and the CORES corpuscorpora and the CORES corpus
CORES CIC CANCODE JPU
the (8 526) the (439 723) the (169 335) the (32 231)the (8,526) the (439,723) the (169,335) the (32,231)of (5,261) and (256,879) I (150,989) of (14,754)
to (3,526) to (230,431) and (141,206) to (11602)( ) ( ) ( ) ( )
and (3,405) a (210,178) you (137,522) and (10,835)
in (3,400) of (194,659) it (106,249) in (9,102)a (3,341) I (192,961) to (105,854) a (8,526)
that (1,864) you (164,021) a (103,524) is (6,409)
i (1 681) it (150 707) h (91 481) it (4 149)is (1,681) it (150,707) yeah (91,481) it (4,149)
as (1,336) in (142,812) that (84,930) that (4,123)for (1,099) that (124,250) of (78,207) I (3,695)for (1,099) that (124,250) of (78,207) I (3,695)
AWL content word types appearing over 40 times in CORES
TYPE RANGE FREQ AL L LCMOTIVATION 3 91 89 1 1ACQUISITION 2 75 22 53 0CULTURAL 3 73 8 2 63CULTURAL 3 73 8 2 63TENSE 2 73 9 64 0INDIVIDUAL 3 72 38 16 18PROCESS 3 70 28 30 12CULTURE 3 68 6 2 60SIMILAR 3 62 13 40 9PRINCIPLE 3 61 7 45 9ADULT 2 60 1 59 0ADULT 2 60 1 59 0INPUT 1 57 0 57 0ATTITUDES 3 56 50 1 5EVIDENCE 3 56 8 44 4STRUCTURE 3 55 2 48 5COMPLEX 3 54 12 27 15TEXT 3 54 7 30 17THEORY 3 52 10 38 4THEORY 3 52 10 38 4CONTEXT 3 50 15 28 7
Low‐frequency word types appearing six times or more in the CORES corpusin the CORES corpus
TYPE FREQ AL L LC
IMPRESSION 15 2 4 9IMPRESSION 15 2 4 9
VAST 13 2 7 4
REFERENCE 12 4 5 3
APPEAL 11 1 3 7
CONSERVATIVE 10 3 4 3
PROMINENT 10 4 2 4
LINEAR 9 2 4 3
LITERALLY 9 1 5 3LITERALLY 9 1 5 3
PROPOSITION 9 1 4 4
SCHOLAR 8 1 2 58 1 2 5
TYPE RANGE FREQ AL L LC
SEMANTIC 1 93 0 93 0
GRAMMATICAL 2 78 2 76 0
PHONOLOGICAL 1 50 0 50 0
PHRASE 3 50 4 44 2
NOVEL 3 47 2 20 25NOVEL 3 47 2 20 25
REANALYSIS 1 47 0 47 0
LINGUISTIC 3 39 10 28 1
BILINGUAL 2 38 2 36 0GU
NEGRO 1 37 0 0 37
CATHOLIC 1 35 0 0 35
COGNITIVE 2 33 3 30 0
NATIONALISM 1 32 0 0 32
VICTORIAN 1 30 0 0 30
LEXICAL 1 29 0 29 0
SYNTAX 3 29 1 27 1
SUFFIX 1 27 0 27 0
INTEGRATIVENESS 1 26 26 0 0
PHONOLOGY 2 26 3 23 0PHONOLOGY 2 26 3 23 0
PRAGMATIC 3 26 1 24 1
The CORES‐based test battery
10 l t• 10 clusters
• 30 items
• 30 distraktors
Test 1 Test 2
AWL 1 concept, estimate approach, consistent, indicate
AWL 2 appropriate, impact obtain, primary
AWL 3 i l d lAWL 3 imply demonstrate, locate
AWL 4 retain subsequent
AWL 5 fundamental, objective, pursuit exposure, sustain, target
AWL 6 edition, initiate, subsidy incentive, utility
AWL 7 comprise, successive voluntary
AWL 8 contemporary terminate arbitraryAWL 8 contemporary, terminate arbitrary
AWL 9 devote controversy, violate
advent, adversary, alleged, allot, amplify, apprentice, assault, breakthrough, chivalry, civic, conceal,
adherence, allude, appeal, apprehension, articulated, assertion, candidate, clergy, concede,
Low-frequency (rare)
congenial, contemplate, conviction, debilitating, denounce, descendant, designate, despised, doctrine, eloquence, elusive, endow, era, evoke, feasible, fragment, glimpse, gloomy, haunted, impending, lucrative, menace, merchandise, mushroom, obsession, pace portable postulate preface proliferation redeem
condemn, conspicuous, depict, disrupt, drawback, flawless, gratitude, heritage, indispensable, mandatory, preface, resemblance, retreat, revive, salient, salvation, scholar, scramble, specimen, stunning, surmount, tactful, testimony, tormented, trait tranquility uncanny unfold upheavalpace, portable, postulate, preface, proliferation, redeem,
subtletrait, tranquility, uncanny, unfold, upheaval, vernacular, wane, whimsical, wit, yearn
Desriptive statistics
CORES-based batteryTest 1 Test 2
Item number 30 30participants 86 48
Mean 18,314 16,833Variance 21,983 15,347
SD 4,689 3,918Minimum 5 10
Maximum 29 26Maximum 29 26
Median 18 17Cronbach α 0.779 0.665Cronbach α 0.779 0.665
Score distribution
16
10
12
14
6
8
10
stud
ents
Version 1Version 2
2
4
01 3 5 7 9 11 13 15 17 19 21 23 25 27 29
items correct
Item statistics
AWL i Low-frequency itemsAWL items Low frequency items
N Mean item difficulty SD Mean
DI SD N Mean item difficulty SD Mean
DI SD
Test 1 9 59,89 28,26 0,31 0,17 21 61,57 23,10 0,34 0,12
Test 2 9 65,89 22,44 0,38 0,17 21 51,95 26,26 0,28 0,18
Cronbach’s α
Phase 1 Phase 2 Phase 3 Phase 4
V1 V2 V1 V2 V1 V2 V1 V2
Itemnumber
30 30 30 30 30 30 30 30
N 154 66 94 117 61 74 86 48
Cronbach`sα
0.78 0.55 0.59 0.57 0.83 0.75 0.78 0.67
Limitations of the studyLimitations of the study
• Number of participants• Number of participants
• Range of texts
• Incapable of handling idiomatic language use
• Keyword status
ImplicationsImplications
• Corpus linguistics• Corpus linguistics
• Language pedagogy
• Language testingg g g
Implications for corpus linguisticsImplications for corpus linguistics
• description of the specificities of language use• description of the specificities of language use in this discipline
Implications for language pedagogyImplications for language pedagogy
• Frequency and keyword lists• Frequency‐ and keyword lists
• what words to focus on
• highly increases the potentials of students in academic text comprehensionacademic text comprehension
Implications for language pedagogyImplications for language pedagogy
• Autonomous learning and exam preparation• Autonomous learning and exam preparation
F d k d li t il bl f t d t• Frequency‐ and keyword lists available for students on the departmental website
• Focused intentional vocabulary learning outside the lclassroom
Implications for language pedagogyImplications for language pedagogy
• concordancers to identify and remedy• concordancers to identify and remedyindividual problem areas
• process of essay writing
• more sophisticated vocabulary use• more sophisticated vocabulary use
• more conscious vocabulary development
Implications for language testingImplications for language testing
• what words to focus on in the selection of• what words to focus on in the selection of lexis to be tested
• more effective vocabulary assessment based• more effective vocabulary assessment based on the special lexical needs of English majors
D'Anna, C. L., Zeichmeister, E. B., & Hall, J. W. (1991). Toward a meaningful definition of vocabulary size. Journal of Reading Behaviour, 23, 109‐22.Goulden, R., Nation, P., & Read, J. (1990). How large can a receptive vocabulary be? Applied Linguistics, 11 (4), 341‐363.Hazenberg, S., & Hulstijn, J. H. (1996). Defining a minimal receptive second‐ language vocabulary for non‐native university students: An empirical investigation. Applied Linguistics, 17 (2), 145‐163.Hirsch, D., & Nation, P. (1992). What vocabulary size is needed to read unsimplified
f l ? d ( )texts for pleasure? Reading in a Foreign Language, 8 (2), 689‐96.Horváth, J. (1996). The assessment of essay writing skills in the first‐year proficiency test. In M. Nikolov & J. Horváth (Eds.), Learning lessons: innovation in teacher education
d t ( 88 106) Pé Li F C tand assessment (pp. 88‐106). Pécs: Lingua Franca Csoport.Laufer, B. (1997). The lexical plight in second language reading: Words you don't know, words you think you know, and words you can't guess. In J. Coady & T. Huckin (Eds.), Second language vocabulary acquisition (pp 20 34) Cambridge: Cambridge UniversitySecond language vocabulary acquisition (pp. 20‐34). Cambridge: Cambridge University Press.Schmitt, N., Schmitt, D., & Clapham, C. (2001). Developing and exploring the behaviour of two new versions of the Vocabulary Levels Test Language Testing 18(1) 55‐88of two new versions of the Vocabulary Levels Test. Language Testing, 18(1), 55 88.
Laufer, B. (2005a). Focus on form in second language vocabulary learning. EUROSLA Yearbook 5 223 250EUROSLA Yearbook, 5, 223-250.McCrostie, J. (2007). Investigating the accuracy of teachers` word frequency intuitions. RELC Journal, 38(1), 53-66.Morris L & Cobb T (2003) Vocabulary profiles as predictors of the academicMorris, L., & Cobb, T. (2003). Vocabulary profiles as predictors of the academic performance of Teaching English as a Foreign Language trainees. System, 32 (1),75-87.Nation I S P (1993) Vocabulary size growth and use In R Schreuder & BNation, I. S. P. (1993). Vocabulary size, growth and use. In R. Schreuder & B. Weltens (Eds.), The bilingual lexicon (pp. 115-134). Amsterdam: Benjamins.Nation, I. S. P., & Waring, R. (1997). Vocabulary size, text coverage and word lists. In N. Schmitt & M. McCarthy (Eds.), Vocabulary: Description, acquisition y ( ), y p , qand pedagogy (pp. 6-19). Cambridge: Cambridge University Press. Nusbaum, H. C., Pisoni, D. B., & Davis, C.K. (1984). Sizing up the Hoosier mental lexicon: Measuring the familiarity of 20,0,000 words. Research on g ySpeech Perception, Progress Report No. 10. Bloomington, IN: Indiana University.O`Keefee, A., McCarthy, M., & Carter, R. (2007). From corpus to classroom: language use and language teaching. Cambridge: Cambridge University Press.
