Click here to load reader

1/26 Writing II Word processors Spell checkers Grammar checkers

  • View
    222

  • Download
    1

Embed Size (px)

Text of 1/26 Writing II Word processors Spell checkers Grammar checkers

  • Slide 1

1/26 Writing II Word processors Spell checkers Grammar checkers Slide 2 2/26 Word processors Typical functions (with linguistic relevance) Text formatting via a graphical user interface Automatic completion/expansion/correction Spelling correction Grammar and style correction Dictionary and thesaurus functions Sorting and collating (of tables) Count words Compare and merge documents All of the above according to local norms As determined by language and/or area Slide 3 3/26 Low-level functions Text justification Usually involves inserting spaces, though not randomly Arabic allows letter shapes to be stretched Counting words What counts as a word? Collation, sorting Alphabetical order may differ from language to language Slide 4 4/26 Low-level functions Hyphenation Printing conventions (dont hyphenate short words, or isolate too few characters) There are rules, which differ by language Eng (morph) present-ation ~ Fr (phon) prsen-tation Some rules change spelling Swe trafikk+kultur = trafikultur, but trafikk-kultur Ger ck k-k, eg dicke dik-ke Swiss Ger ss can be split, but not in a compound, eg Stras-se but gross- artig (* gros-sartig) Hyphen is repeated on second line in some languages Hyphenation should avoid misleading the reader, or other unfortunate consequences, eg mish-ap, leg-end, Arse-nal Slide 5 5/26 Spelling checkers First developed in early 1980s Now standardly available for all text-based software (not just word processors) Available for many languages (at least, those for which spelling is a relevant concept) Nevertheless, still quite crude in design and application I have a spelling checker, It came with my PC, It plain lee marks four my revue Miss steaks aye can knot sea. Eye ran this poem threw it, Your sure reel glad two no. Its vary polished in it's weigh, My checker tolled me sew. [...] Candidate for a Pullet Surprise or Owed to a Spell Checker Jerrold H. Zar http://www.bios.niu.edu/zar/poem.pdf Slide 6 6/26 Spelling checkers Operate only at the word level Checking each word independently against a word list (dictionary) For most languages this implies some knowledge of morphology for handling inflections Though see what happens when you add a word to the dictionary Slide 7 7/26 Slide 8 8/26 Slide 9 9/26 Slide 10 10/26 Spelling checkers For not-found words, possible alternatives are suggested Calculated using Levenshtein distance Simple string difference calculation May or may not take account of likely errors due to Transpositions of symbols (eg langauge ) Transpositions of neighbouring keys (eg levture ) Phonetic misspellings (eg fizix ) Slide 11 11/26 Slide 12 12/26 Levenshtein distance Smallest number of substitutions, insertions and deletions needed to change one string into another Most efficient computer algorithm for calculating this discovered by V.I. Levenshtein in 1965 (Particular) substitutions, transpositions, etc. may be weighted to bias the score Considering size of dictionary, processing must be lightning fast Slide 13 13/26 Spelling checkers Display of suggestions may or may not take account of likelihood considering Levenshtein distance score (is the word with the lowest score necessarily the likeliest correction?) Frequency of use Matching part of speech Readability (long list of alternatives not helpful to a bad speller, eg dyslexic) Slide 14 14/26 Spelling checkers In general do not handle true homograph errors They could quite easily deal with Very frequent errors that can be identified by immediate context (eg its~its, there~their, no~know,...) (Some) errors that can be identified by part-of-speech tagging (eg practice~practise) More difficult to deal with errors that depend on meaning Slide 15 15/26 Spelling checkers Dictionary size: the bigger the better? Including rare words disadvantageous especially if they are same as common misspellings (eg bhat) They clutter up the list of suggestions Most spelling checkers now compromise 90,000 entries according to Wikipedia Sensible handling of morphology (inflections and derivations) can reduce size considerably Slide 16 16/26 Spelling checkers for other languages Concept of spelling not appropriate for some writing systems If writing system is really phonetic, spell checker only has to deal with true typos (miskeying), not alternative phonetic realisations Compounding rules in languages like Ger, Dutch mean many new words checker should not flag these if they are potentially correct Spelling is much less standardized for some languages, eg Heb Iraq Languages with very rich morphology have potentially infinite different word forms, so simple dictionary lookup is not appropriate Slide 17 17/26 Grammar checking Grammar as in good grammar? Early grammar checkers were really style checkers Still word-based, will flag use of weak words like nice, very, etc. and use of clichs, and mechanical errors, eg double words, apparent punctuation errors Now grammar checking involves genuine text analysis Several companies were involved but Microsoft has now become dominant Arguably resulting in stagnation (see Wikipedia) Slide 18 18/26 Grammar checking Who wants it and what for? What mistakes do native speakers make? Borderline between style and grammar? media/data is/are; less~fewer; compared to/with Comma after subject of sentence however as a conjunction Some mistakes clear-cut Do people type ungrammatical sentences? Mistakes introduced by editing Slide 19 19/26 Grammar checking for learners Language learners have different needs from ordinary users Mistakes are somewhat predictable They make different mistakes They might also like an explanation or link to a grammar (in the pedagogic sense) tutorial Grammar checker can be predictive, i.e. go looking for specific mistakes Could be set at an appropriate level Slide 20 20/26 Grammar checking True grammar checking would involve syntactic analysis... Needs a dictionary indicating parts of speech Morphological processing (as before) Rules of grammar... and possibly some semantic processing Actually, its too hard to do completely But a lot can be done Slide 21 21/26 Grammar checking Subject-verb agreement Modifier-noun agreement (Eng this~these etc, but more extensive for other langauges) Verb complement checking (wait for, depend on, etc) Inclusion of a main clause All of the above only if the sentence is fairly simple Slide 22 22/26 Grammar checking For real grammar-checking, use a tagger and/or parser (see later) Some things can be done with statistical models Learn probability of word sequences (n- grams) from a large corpus Use this model to judge grammaticality of text Slide 23 23/26 Slide 24 24/26 Using a language model Language model can also be used to distinguish between Homophones Near synonyms In either case by looking at collocations Again, n-grams Or co-occurrence of words in the sentence Slide 25 25/26 Using a language model: n-grams Counting hits on Google Homophone distinction principle reason (110k) ~ principal reason (1.03m) stationary cupboard (831) ~ stationery cupboard (37.7k) could of gone (27.7k) ~ could have gone (1.92m) I wonder weather (2.78k) ~ I wonder whether (1.74m) dessert + camel (307k) ~ desert + camel (2.07m) Near synonym distinction strong coffee (443k) ~ powerful coffee (668) strong engine (86k) ~ powerful engine (614k) strong + coffee (17.6m) ~ powerful + coffee (8.9m) strong + engine (28.6m) ~ powerful + engine (28.8m) Slide 26 26/26 Using a language model: collocation Homophone distinction dessert + camel (307k) ~ desert + camel (2.07m) Near synonym distinction strong + coffee (17.6m) ~ powerful + coffee (8.9m) strong + engine (28.6m) ~ powerful + engine (28.8m) Similar distinctions can also be measured with reference to a structured thesaurus such as WordNet ( next weeks topic)

Search related