1

Developing Statistic-based and Rule-based Grammar Checkers for Chinese ESL LearnersHoward Chen

Department of EnglishNational Taiwan Normal Universityhjchen@ntnu.edu.tw

2

The Needs to Provide Feedback on Second Language Writing More and more tests ask ESL/EFL students

to demonstrate their writing abilities SLA Researchers would suggest that

learners would need more practices and corrective feedback.

However, who can provide them useful feedback on meaning and forms?

3

Use the Existing Grammar Checkers? Teachers are the best feedback providers. However, so many essays to correct….

Microsoft grammar checker General impressions from ESL/EFL learners=

it is NOT very useful. The two new commercial packages: Vantage

MyAccess and ETS Criterion The feedback quality for ESL learners are not so accurate and

comprehensive. (perhaps because it does not target at any L1 group and it is mainly targeted at native speakers)

4

A More Through Review on E-rater- ETS Criterion Japanese college researcher Junko Otoshi (2005)

from Ritsumeikan University Use 28 Japanese adult students’ TOEFL writing

essays to explore what Criterion can and cannot do with regard to providing feedback on the essays.

Criterion’s critique function was compared with a human instructor’s error feedback focusing on five error categories: verbs, word choice, nouns, articles, and sentence structures.

5

Errors Marked by Criterion and Human Instructors (Means) Error Type Criterion Human Instructors Verbs 0.47 0.84 Nouns 0.00 0.94 Articles 0.07 2.00 Word Choice 0.11 2.32 Sentence Structure 0.32 6.31

6

Rather Disappointing Results and Possible Reasons The results revealed that Criterion

experienced difficulties in detecting errors in all of the five categories.

Does it aim for higher accuracy and has lower recall? More conservative approach

The size the reference corpus? Another program MyAccess has similar

problems, though the general impression from review reports was that they can detect more errors.

7

Trying to Combine Different Approaches: Plan A and B for Grammar Checkers With the funding from NSC in Taiwan, we

planned to develop two grammar checkers. Different approaches= parser-rules-statistics

Plan A: we will use the ngram to help to identify the errors

Plan B: we will use the rule-based grammar checker to identify errors.

If possible, plan A and B will be merged and it should be able to capture more errors.

In this paper, we will only discuss the plan A.

8

What’s the Ngram (statistical) Checker? We will not write specific grammar rules. The computer helps to calculate all the

possible combinations of word strings (2-word and 3-word) in a very large native corpus. Language models building.

All these saved to a large database. Then when students write and submit an

essay to the ngram checker, the system can quickly detect the word strings that do not exist in the native corpus.

9

Ngram-based Checker: advantages The key idea is simple but powerful No need to write rule More robust in detecting errors. Large and suitable corpus might make this

very useful. (ETS, they used 30-million news)

10

The Procedure of Developing an Ngram Checker (corpora and tools) 1. Find suitable and large corpus (e.g BNC;

wikipedia, and Google) 2. Extract the ngrams (NLP tools SRI tool ) 3. Build a large ngram database 4. Develop and test different highlighting

methods 5. Highlight the possibly problematic ngrams

in learners’ writing

11

Grammar Checker Online

The links http://140.122.83.250:4000/main (BNC) http://140.122.83.250/search.php (Google) http://140.122.83.245/ngram-check/ (BNC)

12

The Web Interface of Ngram Checker

13

14

15

A Simple Example

16

Evaluate the Checker Performances: Any Standard Way of Evaluating Checkers? What kind of errors should be used to test the

grammar checker? Fair assessment- same set of sentences. How many sentences? Many different categories and errors Lexical factors. NLP researchers: F-measure and precision

and recall

17

Test with CLEC Corpus from China The size of the Chinese learners of English

Corpus. 1 million error-tagged learner corpus. With about 60 error types. We decided to single out some sentences (10

sentences) from the learner corpus and then throw them into our ngram checkers.

18

1. Form

19

2. Verb Phrases (Tense)

20

3. Noun Phrases

21

4. Pronouns

22

5. Adjective Phrases

23

6. Prepositions- seems to be a difficult area

24

7. Conjuncts Errors

25

8. Word Errors

26

9. Collocation Errors

27

10. Sentence Structure Errors

28

The Strengths of NTNU Ngram Checkers: Ngram is good at detecting errors in the “local”

or adjacent domains. It can indeed find many errors in CLEC.

Spellings Word forms Verb phrases Noun phrases Adj phrases Collocations

29

The Weakness of Ngram Checkers It failed to catch the followings effectively:

Tense errors Conjuncts errors Fragments Pronoun errors Preposition errors The run on sentences The missing words

30

The Poor Performance of Ngram Checkers for Tense and Conjuncts

31

Rule-based Checker can Perform Better for Some Nonlocal Errors

32

Wintertree Grammar Checker

33

BUT Ngram Performed Better for the Local Errors I have some book. The informations are so rich. Th

ese researches are excellent. He is new friend. He cutted his finger. He enjoys to eat. He wants jumping into the river. I cannot decided about this. These reason are too simple. I has three answers.

34

What Can We Do to Improve Feedback from Ngram Checkers? Only Highlighting and No detailed feedback?? We are facing a bigger challenge. How to recommend correct usage? How we can find

the correct examples for students? If students only see the errors highlighted, they

might still fail to correct the errors. For agreement errors, tense errors, confusing words,

Students might be able to self-correct.However, if there are some tense errors, collocations

errors or preposition errors, learners might need more specific suggestions.

35

Find the Proper Collocates: increase and improve life

36

Confusion between accept and receive your apology

37

Future Directions for Improvement1. Test with many different errors and find the

strengths and limitations of Ngram-based checkers and Rule-based checkers

2. Use Tagged learner corpus to find the error patterns from learner languages

3. Feedback can be added in for ngram-based Checkers on the major error patterns

4. Better integration of the rule- based system and ngram checkers

38

Thanks for your attention Questions and Discussions

hjchen@ntnu.edu.tw National Taiwan Normal University