Upload
others
View
9
Download
0
Embed Size (px)
Citation preview
Described our research on developing Ainu language analysis toolkit.
Described recent improvements to the previous system as well as other
enhancements made with an aim to help Ainu language researchers.
In particular:
• enhanced POS tagger with analysis of morphological information,
• added translation support tool for Ainu language translators,
• created simple shallow parser (chunker).
In the near future we plan to:
- Compare different tokenization approaches (ex. Huang et al., 2007).
- Add other dictionaries (Nakagawa, 1995; Tamura, 1998).
- Perform a robust evaluation of the annotations with the help of several
experts and Ainu native speakers.
- Create Deep Syntactic Parser
- Add Named Entity Recognition
- Improve the system for even better performance.
- Apply to machine translation.
A Toolkit for Analysis of Ainu Language
Michal Ptaszynski1 Fumito Masui1 and Yoshio Momouchi2
1) Kitami Institute of Technology, Department of Computer Science 2) Professor Emeritus at Hokkai-Gakuen University
Ainu language is a language of Ainu people living in northern Japan. It is critically endangered.
However, it has a large heritage including myths, stories and poetry. In ongoing numerous
linguistic research these resources have been analyzed, till now by hand. We present a toolkit
for the most necessary linguistic analysis to help Ainu language researchers and translators.
Abstract
Tokenization DL-LSM: (Dictionary Lookup with Longest String Matching)
based on Longest Match Principle
POS Tagging CON-POST: (Contextual Part of Speech Tagging)
based on higher order HMM trained on dictionary examples (* HMM = bigrams, higher order HMM = bigrams, trigrams and longer)
Token Translation CON-ToT: (Contextual Token Translation)
translation selected specifically for the word selected in CON-POST
System Description
© Michał Ptaszyński 2013
Output Options
Linguistic Studies: • collections of Ainu epic stories and myths (Chiri, 1978; Kayano, 1998; Piłsudski and
Majewicz,2004)
• dictionaries and lexicons (Batchelor, 1905; Hattori, 1964; Chiri, 1975-1976; Nakagawa, 1995;
Kayano, 1996; Tamura, 1998; Kirikae, 2003)
• grammar descriptions (Chiri, 1974; Murasaki, 1979; Refsing, 1986; Kindaichi, 1993; Sato, 2008)
NLP-related Studies:
• attempt to transform Ainu language dictionary into an online database (Bugaeva, 2010)
• automatically gather word translations from texts (Echizen-ya et al., 2004)
• analysis / retrieval of hierarchical Ainu-Japanese translations (Azumi and Momouchi, 2009a,b)
• annotating Ainu “yukar” stories for machine translation system (Momouchi et al. 2008)
• a system for translation of Ainu topological names (Momouchi and Kobayashi 2010)
Our previous work:
• created POST-AL, a simple POS tagger for Ainu language (Ptaszynski and Momouchi, 2012)
Previous Research on Ainu language
• [default] Vertical
(typical for POS taggers)
View Selection
1. Skye Hohmann. 2008. The Ainu’s modern struggle. In Worlds Watch, Vol 21., No. 6. 2. Christopher Moseley (ed.). 2010. Atlas of the World? Languages in Danger, 3rd ed. Paris,
UNESCO Publishing. Online version:http://www.unesco.org/culture/languages-atlas/ 3. Yukie Chiri. 1978. Ainu shin-yoshu. Tokyo, Iwanami Shoten. 4. Shigeru Kayano. 1998. Kayano no ainu shinwa shuusei [A collection of Ainu myths by
Kayano]. vol. 1-10, Tokyo, Heibonsha. 5. Bronisław Piłsudski (Author), Alfred F. Majewicz (Editor). 2004. The Collected Works of
Bronislaw Pilsudski: Materials for the Study of the Ainu Language and Folklore, v.3, Pt. 2: Materials for the Study of the Ainu, (Trends in Linguistics: Documentation). Mouton de Gruyter (Oct 2004)
6. Shiroo Hattori (ed.). 1964. An Ainu dialect dictionary. Tokyo, Iwanami Shoten. 7. Mashiho Chiri. 1975-1976. Bunrui ainugo jiten [A classificational dictionary of the Ainu
language], vol. 1-3, Tokyo, Heibonsha. Reprint of works from 1953, 1954 and 1962. 8. Hiroshi Nakagawa. 1995. Ainugo Chitose Hogen Jiten: The Ainu-Japanese Dictionary:
Chitose Dialect [In Japanese]. Sofukan. 9. Shigeru Kayano. 1996. Kayano Shigeru no ainugo jiten [An Ainu dictionary by Kayano
Shigeru]. Tokyo, Sanseido. 10. Suzuko Tamura. 1998. Ainugo Chitose Hogen Jiten: The Ainu-Japanese Dictionary: Saru
Dialect [In Japanese]. Sofukan.
11. Hideo Kirikae. 2003. Ainu shin-yoshu jiten: tekisuto bumpo kaisetsu tsuki (Lexicon to Yukie Chiri’s Ainu Shin-yosyu (Ainu Songs of Gods) with Text and Grammatital Notes) [In Japanese]. Sapporo: Hokkaido Daigaku Bungakubu Gengogaku.
12. Mashiho Chiri. 1974. Ainu goho gaisetu (An outline of Ainu grammar). In Chiri Mashiho chosakushuu (Collection of works by Machiho Chiri) [In Japanese], vol. 4, 3-197. Tokyo, Heibonsha. Reprint from 1936.
13. Kyoko Murasaki. 1979. Karafuto ainugo. Bunpo-hen (Sakhalin Ainu. Grammar volume) [In Japanese]. Tokyo, Kokushokan-kokai.
14. Kirsten Refsing. 1986. The Ainu language. The morphology and syntax of the Shizunai dialect. Aarhus, Aarhus University Press.
15. Kyosuke Kindaichi. 1993. Ainu yukara goho tekiyo (An outline grammar of Ainu epic stories) [In Japanese]. In Ainugogaku kogi 2 (Lectures on Ainu studies 2). Kindaichi Kyosuke zenshu. Ainugo I, v. 5, 145-366. Tokyo, Sanseidoo. Reprint from Ainu jojishi yukara no kenkyu (Research on Ainu epic stories) 2, 1-233, Tokyo: Toyo Bunko, 1931.
16. Tomomi Sato. 2008. Ainugo bunpo no kiso (The basics of Ainu grammar) [In Japanese]. Tokyo, Daigakushorin.
17. Anna Bugaeva. 2010. Internet Applications for Endangered Languages: A Talking Dictionary of Ainu. Waseda Institute for Advanced Study Research Bulletin,No.3, pp. 73-81.
18. Hiroshi Echizen-ya, Kenji Araki, Yoshio Momouchi and Koji Tochinai. 2004. Acquisition of Word Translations Using Local Focus-Based Learning in Ainu-Japanese Parallel Corpora. Lecture Notes in Computer Science, Springer-Verlag, Vol. 2945, pp.300-304.
19. Michal Ptaszynski and Yoshio Momouchi, "Part-of-Speech Tagger for Ainu Language Based on Higher Order Hidden Markov Model", Expert Systems With Applications, Vol. 39, Issue 14 (2012), pp. 11576–11582, Elsevier, 2012.
20. Yasunori Azumi and Yoshio Momouchi. 2009a. Development of Analysis Tool for Hierarchical Ainu-Japanese Translation Data [In Japanese]. Bulletin of the Faculty of Engineering at Hokkai-Gakuen University, No.36, pp.175-193.
21. Yasunori Azumi and Yoshio Momouchi. 2009b. Development of Tools for retrieving and analyzing Ainu-Japanese translation data and their applications to Ainu-Japanese machine translation system [In Japanese]. Engineering Research: The Bulletin of Graduate School of Engineering at Hokkai-Gakuen University, No.9, pp.37-58.
22. Yoshio Momouchi, Yasunori Azumi and Yukio Kadoya. 2008. Research Note: Construction and Utilization of Electronic Data for “Ainu Shin-yosyu” [In Japanese]. Bulletin of the Faculty of Engineering at Hokkai-Gakuen University, No. 35, pp. 159-171.
23. Yoshio Momouchi and Ryosuke Kobayashi. 2010. Dictionaries and Analysis Tools for the Componential Analysis of Ainu Place Names [In Japanese]. Engineering Research: The Bulletin of Graduate School of Engineering at Hokkai-Gakuen University, No.10, pp.39-49.
24. Huang, C., Simon, P., Hsieh, S., & Prevot, L. (2007). Rethinking Chinese Word Segmentation: Tokenization, Character Classification, or Word break Identification, In Proceedings of the ACL 2007 Demo and Poster Sessions, pages 69-72, 2007
25. John Batchelor. 1905. An Ainu-English-Japanese dictionary (including a grammar of the Ainu language). Tokyo Methodist Pub.House.
Shallow Parser
• Ainu language is a language of Ainu people, mostly living in northern Japan.
• Population of Ainu = about 23 thousand people.
• Number of native speakers = less than hundred (Hohmann, 2008).
• Ainu language is critically endangered (Moseley, 2010).
Purpose of this research:
↓ Create language analysis toolkit including POS tagger,
translation support tool and shallow parser.
↓ Help in linguistic and language anthropology research
and support translators of Ainu texts.
Contribute to the process of reviving Ainu language.
Introduction
1. POS standard selection • -n Nakagawa (1995) / compact
• -t Tamura (1998) / sophisticated
• [default]Kirikae(2003) / balanced
• -s Simple (Japanese) / no POS
description • -en English / English POS tag names
• -e English (Abbreviated) / abbreviated
English POS tag names (useful for
linguists)
2. Morphological Analysis -m analysis of morphemes constituting
each word
Ima
ge
so
urc
e: h
ttp
://e
n.w
ikip
edia
.org
/wik
i/A
inu_pe
ople
Ima
ge
so
urc
e: h
ttp://w
ww
.am
azo
n.c
o.jp
• -h Horizontal
(useful for language anthropologists) Additional options:-ts Translation support
*) ORIGINAL TRANSLATION (私はそれを見て、安心をし流れに沿うて帰つて来た.)
Conclusions and Future Work
Applied the POS information from
POS tagger to create a simple
shallow parser, or chunker, which:
1. Divides the sentence into
clauses containing the longest
possible string of consecutive
nouns and verbs and closely
related morphemes (particles,
prefixes, suffixes).
2. Divides the clauses into noun
phrases (NP) and verb phrases
(VP).
Difficult to create
a fully functional
deep parser.
Ainu language grammar is sophisticated:
• polysynthetic language (more morphemes than word stems)
• words relate to each other and change meanings in context
• incomplete model of grammar.
However, the sentence structure of Ainu language
is generally similar to Japanese (SOV).
General rules for
sentence
chunking should
work.
TAMURA STANDARD WITH MORPHOLOGICAL ANALYSIS
• base dictionary for POST-AL:
Ainu shin-yoshu jiten (Lexicon to
Yukie Chiri’s Ainu Shin-yosyu (Ainu
Songs of Gods)) by Kirikae (2003)
• transform dictionary information
to XML database:
1. token (word, morpheme, etc.)
2. part of speech
3. meaning (in Japanese)
4. usage examples (not for all cases)
5. reference to the story it appears in (not for all cases)
Dictionary Construction Ima
ge
so
urc
e: h
ttp://w
ww
.am
azo
n.c
o.jp
Tokenization
DL-LSM: (Dictionary Lookup with Longest String Matching)
based on Longest Match Principle
DL-P-LSM: (Dictionary Lookup with Partial-LSM)
based on LMP with caesurae
POS Tagging
S-POST: (Statistical Part of Speech Tagging)
all words of the same form are treated as one list,
choose POS with the highest occurrence
CON-POST: (Contextual Part of Speech Tagging)
based on higher order HMM trained on dictionary examples
(* HMM = bigrams, higher order HMM = bigrams, trigrams and longer)
Token Translation
RAN-ToT: (Random Token Translation)
translation selected randomly from the list of words
of the same POS (S-POST extension)
CON-ToT: (Contextual Token Translation)
translation selected specifically for the word selected in CON-POST
System Description input
tokenization
DL-LSM | DL-P-LSM
POS tagging
S-POST | CON-POST
token translation
RAN-ToT | CON-ToT
output
References
• 13 Ainu stories (yukar) from Ainu shin-yoshu
(Ainu Songs of Gods) gathered by Chiri (1978).
• all stories are tokenized (by Kirikae, 2003)
• one yukar is annotated with POS and
translations (by Momouchi et al., 2008)
Yukar 10: Pon Okikirmuy yayeyukar “kutnisa kutunkutun”
(The “Kutnisa kutunkutun” story told by Small Okikirmuy himself)
Evaluation Dataset Description
Ima
ge
so
urc
e: h
ttp://w
ww
.am
azo
n.c
o.jp
Score Calculation
Calculate score as balanced
F1 score for all parts of POST-AL
Results
Tokenization
DL-LSM was slightly
Better (98.46%).
Token translation
Contextual token translation
was much better (98.36%)
than statistical (90.11%).
Evaluation
POS tagging
Contextual POS tagging
was much better (96.96%)
than statistical (90.11%).
Additional options:-o Additional information