Testing Grammar and

Vocabulary

A presentation by Marcia Tadjuddin

Which side are you on?

Grammar and vocabulary should be tested in isolation. (direct testing of grammar and vocabulary)

Pro or Con?Develop your arguments!

Testing Grammar•Why Test Grammar?

•As far as proficiency tests are concerned, there has been a shift towards the view that since it is language skills that are usually of interest, then it is these which should be tested directly, not the abilities that seem to underlie them.

Why test grammar?• Probably, most proficiency tests that are

administered on a large scale still retain a grammar section.

• One reason for this must be the ease with which large number of items can be administered and scored within a short period of time.

• The question of content validity: if we decide to test writing ability directly, then we are severely limited in the number of topics, style of writing and the grammatical elements that we can cover in any version of the test.

So, should we test grammar?•Wherever the teaching of grammar is

thought necessary, then consideration should be given to the advisability of including a grammar component in achievement area.

•Whether or not grammar has an important place in an institution’s teaching, it has to be accepted that grammatical ability, or rather the lack of it, sets limits to what can be achieved in the way of skills performance.

Writing specifications•Specification of content for achievement

tests:▫From the syllabus, which list the

grammatical structures to be taught.▫Inferring from textbooks and other

teaching materials•For placement test:

▫Includes all of the structures identified above

▫Also perhaps those structures even for the lowest classes

Sampling•Reflect an attempt to give the test content

validity by selecting widely from the structures specified.

•It should also take into account of what are regarded as the most important structures.

•It should not deliberately concentrate on the structure that happen to be easiest to test.

Writing items•Whichever techniques are chosen, it is

important for the text of the item to be written grammatically correct and in natural language.▫E.g. We can’t work in this class because there

isn’t enough silence.•To avoid unnatural language, it is

recommended to use corpus based examples.•4 techniques are presented here: gap filling,

paraphrase, completion and multiple choice. The first 3 require the candidate’s production, the last calls only for recognition.

Gap Filling•Ideally, gap filling items should have just

one correct response.▫E.g. The council must do something to

improve transport in the city. ______________, they will lose the next election. [otherwise]

•An item with two possible correct responses may be acceptable if the meaning is the same, whichever is used.▫E.g. My dad works in a company _________

makes Nike Shoes. [that/which]

Gap filling• An item is probably rejected if the different

possibilities give different meanings or involve quite different structures, one of which is the one that is supposed to be tested.Patient: My baby keeps me awake all night. She won’t stop crying.Doctor: __________ let her cry. She’ll stop in the end.[just, I’d, Well, Then, etc.)

• This item may be improved by including the words “then”, and “just” so that it cannot fill the gap.Doctor: Then _________ just let her cry. She’ll stop in the end. [I’d , you]

Gap filling•Adding to the context can often restrict the

number of possible correct responses to a single one.

• This may be presented in a longer passage with several gaps and can be used to test a set of related structures (such as the articles) or a variety of structures

We live in _____ old house in _______ middle of the village. There is ________ beautiful garden behind ________ house. ______ roof of ______ house is in very bad condition.

Paraphrase•Require the student to write a sentence

equivalent in meaning to one that is given.

•It is helpful to give part of the paraphrase in order to restrict the students to the grammatical structure being tested.▫When I came, my mom was setting up the

dinner table.▫When I came, the dinner table____________

Completion •Can be used to test a variety of

structures. •Usually in a form of conversational

transcript.

•See the example on pg. 177

Multiple Choice• There are times when gap filling will not test what we

want to test because there are too many possibilities.

Gap filling:They left at 7. They _________ be home by now.

Multiple Choice:A: They left at 7. They _________ be home by now.B: Yes, but we can’t count on it, can’t we?

a. can b. could c. will d. must

Multiple Choice•Can be used to test discontinuous

elements.

A: Poor man, he ________at that for days now.

B: Why doesn’t he give up?

a) was workingb) has been workingc) is workingd) had worked

Scoring Production Grammar Tests• The important thing when scoring is to be clear

about what each item is testing, and to award points for that only.

• Nothing should be deducted for non-grammatical errors, or for errors in elements of grammar which are not being tested by the item.

• If two elements are being tested in an item, then points may be assigned to each of them.

• Alternatively, it can be predetermined that both elements have to be correct for any points to be awarded.

• To ensure scoring is valid and reliable, careful preparation of the scoring key is necessary.

Testing VocabularyWhy test vocabulary?• If there is little teaching of vocabulary, it may be

argued that there is little call for achievement tests of vocabulary.

• For those who believe that systematic teaching of vocabulary is desirable, vocabulary achievement tests are appreciated for their backwash effect.

• The usefulness and feasibility of a general diagnostic test of vocabulary is not readily apparent.

• In placement tests, we would be looking for some general indication of the adequacy of the student’s vocabulary. We would not normally require a particular set of lexical items to be a prerequisite for a particular language class.

Writing Specification• If vocabulary is being consciously taught, then

presumably all the items presented to the students should be included in the specifications.

• We can add all the new items that the students have met in other activities (reading, listening, etc.)

• Words should be grouped according to whether their recognition of their production is required.

• A subsequent step is to group the items in terms of their relative importance.

• For placement tests or proficiency tests , specification refer to one of the published word lists that indicate the frequency of the words.

Sampling •Words can be grouped according to their

frequency and usefulness.•From each of these groups, items can be

taken at random, with more being selected from the groups containing the more frequent and useful words.

Writing Items: Testing recognition ability•Multiple choice can be recommended for

this type of testing problem.•Distractors are usually readily available•It seems unlikely to be any serious

harmful backwash effect since guessing meaning of vocabulary items is something that we would probably wish to encourage.

Recognizing SynonymsE.g.: which is the closest in meaning to

“gleam”?A. gather B. shine C. welcome

D. clean

•Which distractors do you think are likely to be chosen?

•Whether distractors would work as intended would only discovered through trialling.

•Note that all of the options are words that the candidates are expected to know.

Recognizing Synonyms•Another way is to have a common word as the

stem with 4 less frequent words as options.

E.g.: which is the closest in meaning to “shine”?A. malm B. gleam C. loam D. snarl

•The drawback: what distractors to use? How do we ensure that the distractors are unfamiliar words to the test takers? If the test taker knows them, they will not distract.

Recognizing DefinitionsLoathe meansa) dislike intenselyb) become seriously illc) search carefullyd) look very angry

• Note that all of options are of about the same length.• Test takers who are uncertain of which option is

correct will tend to choose the one which is noticeably different from the others.

• In the example above, the writer has included some notion of intensity in all of the options.

Recognizing Definitions• The less frequent word or the difficult word could also

be used (although with the same concerns)

▫One word that means to dislike intensely isA. growl B. screech C. sneer D. loathe

• Thrasher (Internet) believes that vocabulary is best tested in context. A better way to test knowledge of loathe would be:

▫Bill is someone I loathe for the humiliation that he has caused me and my family.A. like very much C. respectB. dislike intensely D. fear

Recognizing appropriate word for context•Context, rather than a definition or a

synonym, can be used to test knowledge of a lexical item.▫E.g. The strong wind _______ the man’s efforts

to put up the tent.▫a. disabled b. hampered c. deranged d.

regaled

•Note that the context should not itself contain words that the candidates are unlikely to know.

Recognizing appropriate word for context• Since learners and language users in general

normally meet vocabulary in context, providing context in an item makes the task more authentic and perhaps result in a more valid measure of the candidate’s ability.

• The context may help activate a memory of the word, in the sane way as meeting it when reading in a non-test situation.

• There could be some negative backwash when words are presented in isolation.

• However, when we test vocabulary by means of multiple choice, the range of possible distractors will be wider if words are presented in isolation.

Testing Production AbilityUsing Pictures

•The main difficulty in testing productive lexical ability is the need to limit the candidate to the lexical item that we have in mind using only words. One way around this is to use pictures.

•However, this method is obviously restricted to concrete nouns that can be unambiguously drawn.

Testing Production AbilityDefinitions

•May work for a range of lexical items.•But not all items can be identified

uniquely from a definition (excluding all synonyms)

•Not all words can de defined entirely in words more common or simpler than themselves.

Testing Production AbilityGap filling

•This can take the form of one or more sentences with a single word missing.

•Often there is an alternative word to the one we have in mind. This problem can be solved by giving the first letter of the word (possibly more) and even an indication of the number of letters.

Reminders:•While grammar and vocabulary contribute

to communicative skills, they are rarely to be regarded as ends in themselves.

•It is essential that tests should not accord them too much of importance, and so create a backwash effect that undermines the achievement of the objectives of teaching and learning where these are communicative in nature.