Co-funded by the 7th Framework Programme and the ICT Policy Support Programme of the European Commission through the contracts T4ME, CESAR, METANET4U, META-NORD (grant agreements no. 249119, 271022, 270893, 270899).
Language Technology for Multilingual Europe: Current State and Future Plans
Georg Rehm
Network Manager META-NET German Research Center for Artificial Intelligence (DFKI), Berlin, Germany
European Languages in the Age of Technology: Quo vadis? – Vilnius, Lithuania
November 14, 2012
Outline
q Introduction
q Language White Paper Series
q Strategic Research Agenda
q Conclusions
http://www.meta-net.eu 2
Multilingual Europe
3 http://www.meta-net.eu
q Challenge: Providing each language community with the most advanced technologies for communication and information so that maintaining their mother tongue does not turn into a disadvantage.
q While research has made considerable progress in recent years, the pace of progress is not fast enough to meet the challenge within the next 10-20 years.
q All stakeholders – researchers, LT user and provider industries, language communities, funding programmes, policy makers – should team up for a major dedicated push.
Objectives
META-NET is a network of excellence dedicated to fostering the tech-nological foundations of the European multilingual information society.
http://www.meta-net.eu 4
Four EU-Funded Projects
q Initial project: T4ME (FP7; 13 partners, 10 countries)
q Three ICT-PSP consortia since Feb. 2011: CESAR, METANET4U, META-NORD
q All EU member states and several non-member states covered.
q META-NET in Nov. 2012: 60 members in 34 countries.
http://www.meta-net.eu 5
http://www.meta-net.eu/members
Language White Paper Series META-VISION
http://www.meta-net.eu 6
Language White Paper Series
http://www.meta-net.eu 7
q Reports on the state of our languages in the digital age and the level of support through language technology.
q Series covers 30 languages.
q Key communication instruments to address decision makers and journalists.
q Inform about societal and technological problems and challenges as well as economic opportunities.
q >2 years in the making.
q >200 national experts as contributors. q >8.000 copies printed and distributed to
politicians and journalists.
30 Languages Covered
q Basque q Bulgarian* q Catalan q Czech* q Danish* q Dutch* q English* q Estonian* q Finnish* q French*
q Galician q German* q Greek* q Hungarian* q Icelandic q Irish* q Italian* q Latvian* q Lithuanian* q Maltese*
q Norwegian q Polish* q Portuguese* q Romanian* q Serbian q Slovak* q Slovene* q Spanish* q Swedish* q Croatian
http://www.meta-net.eu 8
* = Official EU language
Cross-Lingual Ranking
q In four application areas, each language is assigned to one of five clusters, ranging from excellent LT support to weak/no support:
1. Machine Translation
2. Speech Processing
3. Text Analysis
4. Resources
q Results finalised at a meeting in Berlin with representatives of all 30 languages (October 21/22, 2011).
http://www.meta-net.eu 9
MT
http://www.meta-net.eu 10
English
good
French, Spanish
moderate fragmentary
Catalan, Dutch, German, Hungarian, Italian, Polish, Romanian
weak or no support
Basque, Bulgarian, Croatian, Czech, Da-nish, Estonian, Finnish, Galician, Greek, Icelandic, Irish, Latvian, Lithuanian,
Maltese, Norwegian, Portuguese, Serbian, Slovak, Slovene, Swedish
excellent
Czech, Dutch, Finnish, French, German,
Italian, Portuguese, Spanish
moderate fragmentary
Basque, Bulgarian, Catalan, Danish, Estonian, Galician, Greek,
Hungarian, Irish, Norwegian, Polish, Serbian, Slovak, Slovene, Swedish
weak or no support
Croatian, Icelandic, Latvian, Lithuanian, Maltese, Romanian
excellent
English
good
Spee
ch
English
good
Dutch, French, German, Italian,
Spanish
moderate fragmentary
Basque, Bulgarian, Catalan, Czech, Danish, Finnish, Galician, Greek, Hungarian, Norwegian, Polish, Portuguese, Romanian, Slovak,
Slovene, Swedish
weak or no support
Croatian, Estonian, Icelandic, Irish, Latvian, Lithuanian, Maltese, Serbian
excellent
English
good
Czech, Dutch, French, German, Hungarian,
Italian, Polish, Spanish, Swedish
moderate fragmentary
Basque, Bulgarian, Catalan, Croatian, Danish, Estonian, Finnish, Galician,
Greek, Norwegian, Portuguese, Romanian, Serbian, Slovak, Slovene
Icelandic, Irish, Latvian, Lithuanian, Maltese
weak/no support excellent
Res
ourc
es
Text
Ana
lysi
s
Europe’s Languages and LT
http://www.meta-net.eu 11
Dutch French German Italian
Spanish
Catalan Czech
Finnish Hungarian
Polish Portuguese
Swedish
Basque Bulgarian
Danish Galician
Greek Norwegian Romanian
Slovak Slovene
Croatian Estonian Icelandic
Irish Latvian
Lithuanian Maltese Serbian
English
good support through Language Technology
weak or no support
Not enough R&I on European languages
➔ LT research on European languages, except for English, is too weak and too slow
➔ Many languages are badly covered
0
50
100
150
200
250
300
350
400
450
English
Ch
inese
Germ
an, Stand
ard
Fren
ch
Spanish
Japane
se
Arabic
Dutch
Portugue
se
Czech
Danish
Swed
ish
Hind
i Ko
rean
Turkish
Ita
lian
Russian
Finn
ish
Hebrew
Hu
ngarian
Sloven
e Urdu
Romanian
Zulu
Bulgarian
Catalan-‐Va
lencian-‐Ba
lear
Greek
Thai
Welsh
Estonian
Basque
Ge
rman, Swiss
InukStut
Indo
nesia
n Ineseñ
o LaSn
Marathi
Malay
Pushto
Serbian
Syria
c Tamil
UgariS
c Ukrainian
Uspanteko
Vietnamese
Languages treated in the 2010 editions of Journal of Computational Linguistics and Conferences of ACL, EMNLP and COLING. Many European languages without any reference: Slovak, Maltese, Lithuanian, Irish, Albanian, Croatian, Galician etc.
Key Observations
http://www.meta-net.eu 13
q When it comes to Language Technology support, there are massive differences between Europe’s languages and technology areas.
q LT support for English is ahead of any other language.
q Even support for English is far from being perfect.
q The gap between English and the other languages keeps widening!
q Several languages – Icelandic, Latvian, Lithuanian, Maltese – receive this weakest score in all four areas!
q At least 21 European languages in danger of digital extinction!(Languages put into the “weak or no support” category at least once.)
White Paper Press Campaign
q Headline of press release:
At Least 21 European Languages in Danger of Digital Extinction. Good News and Bad News on the European Day of Languages.
q Sent out to journalists, politicians and other stakeholder groups before the European Day of Languages (September 26).
q Overwhelmed by the huge interest in the topic and our key findings!
q 520+ mentions in the online and traditional press.
q 40+ interviews with META-NET representatives (television, radio).
q News came in from 41 countries in 35 different languages.
http://www.meta-net.eu 14
Coverage by Country
http://www.meta-net.eu 15
Spain, 15.90%
Bulgaria, 10.80%
International, 7.90%
Latvia, 5.30%
Netherlands, 4.80%
Greece, 4.60% Romania, 4.40%
Serbia, 4.40%
Italy, 4.20%
Germany, 3.50%
Russia, 3.50%
Estonia, 2.90%
France, 2.60%
Slovenia, 2.40%
Iceland, 2.20% Malta, 2% USA, 1.50%
Denmark, 1.30%
Latin America, 1.30%
Lithuania, 1.30%
Ireland, 1.30% UK, 1.10%
Belgium, 0.90%
Finland, 0.70% Sweden, 0.70%
Poland, 0.70%
Norway, 0.40%
Mexico, 0.40%
Brazil, 0.40%
Slovakia, 0.40%
Basque Country, 0.40% Portugal, 0.40% Austria, 0.20%
New Zealand, 0.20%
Hungary, 0.20%
Bosnia and Herzegovina, 0.20%
Costa Rica, 0.20%
Cyprus, 0.20%
Canada, 0.20%
Australia, 0.20%
Spain
Bulgaria
International
Latvia
Netherlands
Greece
Romania
Serbia
Italy
Germany
Russia
Estonia
France
Slovenia
Iceland
Malta
USA
Denmark
Latin America
Lithuania
Response: Examples
q Austria: Der Standard. q Denmark: Politiken, Berlingske Tidende. q Finland: Tiede. q Germany: Heise Newsticker, Süddeutsche Zeitung. q Greece: in.gr, Πρώτο Θέµα, Prosilipsis. q Iceland: Fréttablaðið, Morgunblaðið. q Italy: Wired. q Lithuania: Delfi, Elektronika, KaunoDiena. q Norway: Computerworld. q Slovenia: Delo, Dnevnik, Demokracija. q Serbia: Politika. q Spain: El Mundo. q UK: Huffington Post. q USA: Mashable, NBC News, Reddit.
http://www.meta-net.eu 16
Date 30 September 2012 Page 16
Copyright material. This may only be copied under the terms of a Newspaper Licensing Agency agreement (www.nla.co.uk) or with written publisher permission. For external republishing rights see www.nla-republishing.com
Press Campaign: Examples
http://www.meta-net.eu 17
38
!!"# $"%&'() *+,-) .*#… µ&/,0# *//"#&(1, 2+34 ('& '5(*!64 1//*4 *753+'8(64
9/:;;*4, ;0µ%3#' µ* +'#*753+'8-() 6(<*;" µ* !"# 7+,95'%) 200 ('& +/6,# *&.&(:#. = ;79(*(5&µ6#" µ*/6-!" ."µ,;&*0!"(* '+2 !, *+&;!"µ,#&(2 .>(!7, ?@AB-C@A µ* '%,5µ) !" -!*-;&#) @753+'8() =µ65' D/3;;:#.
D&' !&4 '#19(*4 !"4 65*7#14 !,74, 9/3;;,/29,& '+2 34 -:5*4 !"4 D"-5'&14 =+*>5,7 E'<µ,/29";'# !&4 .&'<6;&µ*4 9/3;;&(64 7+"5*;>*4 ('& ."µ&,059";'# 6#' «F*7(2 G&-E/>,» 9&' (1<* *753+'8() 9/:;;'. H!" µ*/6!" !,74, ,& *&.&(,> '#'I)!"-;'# µ*!'J0 1//3# !6;;*5' E';&(1 "/*(!5,#&(1 *59'/*>', ."/'.) !"# 0+'5J" '7!2µ'!"4 µ*!1%5';"4, !" .7#'!2!"!' %3#"!&()4 '//"/*-+>.5';"4 ('& $"%&'()4 '#1/7;"4 (*&µ6#,7, *#: !'7!2-5,#' .&*5*7#)-<"(* ('& " .&'<*;&µ2!"!' 9/3;;&(:# +253# ) +"9:#.
H* +5:!" %1;" *J6!';'# !&4 &;!,-;*/>.*4 +,7 *+&!56+,7# ;!,74 -5)-;!*4 #' (1#,7# µ*!'%51;*&4 online, 2+34, 9&' +'51.*&9µ', " 7+"5*;>' !,7 (,/,;;,0 +/"5,%,5&()4 Google Translate. A"# >.&' :5', *J*!1;!"(* ('& " «*+&(,#>'» !3# *//"#2%3-#3# -5";!:# µ* !&4…;7;(*764 !,74, 2+34 9&' +'51.*&9µ' " .7#'!2!"!'
#' «µ&/);*&» (1+,&,4 ;!, GPS ;!" µ"!5&() !,7 9/:;;'. K& *5*7#"!64 ('!6/"J'# ;!, ;7µ+65';µ' 2!& 7+15-,7# !6!,&*4 ;7;(*764, '//1 .*# *>#'& !2;, .&'.*.,µ6#*4 2;, ,& '99/2%3#*4. A, «-57;2» µ*!1//&, ('!'(!1,
2+34 *>#'& 1//3;!* ('& /,9&(2, " '99/&() 9/:;;'. K& '99/2%3#,& -5)-;!*4 6-,7# !"# ('/0!*5" .7#'!) !*--#,/,9&() 7+,;!)5&J", (1!& !, ,+,>, *7#,*> !"# +*5'&!653 *J1+/3;" !"4 9/:;;'4. B+2 «!*-#,/,9&(2 '+,-(/*&;µ2» (&#.7#*0,7# +*5&;;2!*5, " &;/'#.&(), " /*!,#&(), " /&<,7'#&() ('& " µ'/!6I&(" 9/:;;', *#: ;* />9, ('/0!*5" µ,>5' E5>;(,#!'& " *//"-#&(), " E,7/9'5&(), " ,7995&() ('& " +,/3#&(), +,7 2+34 '#'%65*& " 65*7#' 6-,7# «'+,;+';µ'!&()» !*--#,/,9&() 7+,;!)5&J".
«?6!5&'» -'5'(!"5>I*!'& " 7+,-;!)5&J" -5";!:# ;* ,//'#.&(), 9'/-/&(), 9*5µ'#&(), &!'/&() ('& &;+'#&() 9/:;;'. K& *+&(*%'/)4 !"4 *+&;!"-µ,#&()4 ,µ1.'4, !"#$ %&'()*+,- ('& .(/0*1( 2/µ, '#'%65,7# -'5'(!"5&-;!&(1: «L+15-,7# .5'µ'!&(64 .&'%,-564 ;!"# 7+,;!)5&J" !"4 9/3;;&()4
!*-#,/,9>'4 '#1µ*;' ;!&4 .&1%,5*4 *753+'8(64 9/:;;*4. A, -1;µ' µ*!'-J0 “µ&(5:#” ('& “µ*91/3#” 9/3;;:# ,/,6#' ('& .&*750#*!'&. M56+*& #' *J';%'/>;,7µ* !,# *%,.&';µ2 !3# µ&(52!*53# ('& /&92!*5, +/,0;&3# ;* $"%&'(,04 +25,74 9/3;;:# µ* !&4 '+'5'>!"!*4 E';&(64 !*-#,/,9>-*4. N&'%,5*!&(1, ,& 9/:;;*4 '7!64 *>#'& ('!'.&(';µ6#*4 ;* $"%&'() *J'%1#&;"».
?1/&;!', ,& *&.&(,> !,#>I,7# 2!& -3-5>4 '+,%';&;!&() .51;" ,& 9/:;;*4 '7!64 .0;(,/' <'… *+&E&:;,7# ;!,# $"%&'(2 (2;µ,7 !,7 21,7 '&:#'. = (. 3"*4" ."5*,6748)&, µ6/,4 !"4 *+&-;!"µ,#&()4 ,µ1.'4 '+2 !, O#;!&!,0!,
@+*J*59';>'4 !,7 F29,7 @5*7#"!&(2 P6#!5, B<"#1, /6*& ;!,# «@.A.»: «= 65*7#' '7!) .*# /6*& 2!& .*# <' I);*& " *//"#&() 9/:;;' ) 2!& (&#.7#*0*& µ* *J'%1#&;"». = *&.&(24 *J"9*> 2!& 2;, 7+15-,7# 1#<53+,& +,7 µ&/1-#*, 951%,7# ('& *+&(,#,0# µ* µ&' 9/:;;', !2!* '7!) <' ;7#*->;*& #' 7+15-*&. @>#'& ;"µ'#!&(2, 2µ34, #' 6-,7# 2/,& ,& -5);!*4 !" .7#'!2!"!' #' «µ&/);,7#» ;!&4 µ"-'#64, 2+34 !' GPS !,74, ;!' *//"#&(1 ('& #' 6-,7# ;!" .&1<*;) !,74 9/3;;&(1 *59'/*>' "/*(!5,#&(:# 7+,/,9&;!:#.
?*!'J0 '7!:# !3# «*59'/*>3#» *>#'& ,& .&,5<3!64 ,5<,95'%&(:# ('& ;7#!'(!&(:# /'<:#, +,7 -5";&µ,+,&-,0#!'& ('<"µ*5 '+2 *('!,#!1.*4 @//"#*4 -5);!*4 ('& E';>I,#!'& ;!" 9/3;;&() !*-#,/,9>'. M'5’ 2/' '7!1, !,#>I*& 2!& " $"-
%&'() *J1+/3;" µ&'4 9/:;;'4 *>#'& ;"µ'#!&() «N*# *>#'& ;!' -65&' !,7 µ6;,7 -5);!". K& *(1;!,!* (7E*5-#);*&4, " @753+'8() @#3;" ('& , &.&3!&(24 !,µ6'4 +56+*& #' -5"µ'-!,.,!);,7# !"# '#1+!7J" '7!)4 !"4 !*-#,/,9>'4 9&' 2/*4 !&4 9/:;;*4», '#'%65*& ('& ;7#*->I*&: «K& -5);!*4, 2µ34, +56+*& #' '+'&!,0# #' 7+15--,7# ('& ;!" 9/:;;' !,74 !' µ6;' '7!1 ('& #' µ"# &('#,+,&,0#!'& µ* !' '99/&(1». Q
!"µ#$% 27 &'#$'µ()*+, 2012 !"!#$!%&' (#)&'
LifeMKFF@H @LRSMBTP@H DFSHH@H U@SRKLCABO A@VCKFKDOPB… W@M@RBH?@C@H
!" #$%&'() "*'%+,&-$ (&,./,"01/, 2' "33$,&(+
!"!*+ ,!%-&#[email protected]
= 9/:;;' !"4 '+,J6#3;"4…
XX GREEKLISH
-. /0012345+. 6)78$'9 "6+,5 $%5 :/1;$')% <,5/$7 $'65+1+0.:7 ,#+8$7).=%, 0'0+529 #+, ',5+'* $%5 #')/.$")4 '=>#148% $%9 01?88/9
.! GREEKLISH /9,(),#:-#);# 97<)# µ<=: µ6#&µ+--:# > email ), 9/*,==0-/*), #<), -6$ ?@*"$ µ"$. A"*+ -) 1/1)#0$ 0-, -" -/7/&--"4" ?*0#," &9+*?)&# -" 17:==,(+ /*1"7/4", -" )9)4" /9,-*<9)&# -6 ?*>=6 -6$ /776#,(>$ 1*"µµ"-)-=/,*+$, <B65), (", #<), /#>7,(/$ B"4#/-", 0-, 8/# <?)&# «"1("7,+=/,» "&-<$ -,$ -/?#)7)14/$. % ("C6-16->$ .7:==)7)14"$, (. .,@*1)$ 39"µ9,#,@-6$, 7</, =-)# «D.E.»: «E" greeklish /4#", 9*0576µ" 1," -6# /776#,(> 17@==", ,84:$ 1," "#C*@9)&$ #<"$ 67,(4"$ 1," <#"# ("C"*+ 17:==,(0 701). 3/ -6 ?*>=6 -:# greeklish "9)F/#@#)#-", "90 -6 µ)*B> -6$ 7<F6$ > 09:$ 7<µ/ -) /-&µ)7)1,(0 4#8"7µ" 9)& 867@#/-", µ/ -6# )*C)1*"B4" -6$ 7<F6$ (", =</-", (", µ/ -6 =6-µ"=4" -6$ 7<F6$ (", µ/ -6# 9*)<7/&=> -6$». % (4#8&#)$, µ/ -)# )9)4) <*?)#-", "#-,-µ<-:9), ), #<), +#C*:9),, /4#", 6 "9)F<#:=6 "90 -6 1*"9-> µ)*B> -6$ 17@=-="$. G&-> 6 «),(/,0-6-"», 0µ:$, 5)6C+/, (", =-6# ("-"#06=6 -6$ =6µ"=4"$ "77+ (", -6# 9*)<7/&=6 -6$ 7<F6$. «G&-> 6 "9)F<#:=6 8/# /4#", +#/& =6µ"=4"$», "#"B<*/, ) /,8,(0$, ) )9)4)$ /F61/4 0-, 6 8,"8,("=4" -6$ 1*"B>$ 5)6C+/, #" /#-&9:-C/4 6 7<F6 (", #" =/C/4 µ/ +77/$ )µ0**,'/$ 7<F/,$. «%-"# ?*6=,µ)9),/4-", "&-> 6 µ)*B> /9,(),#:#4"$, ("--"=-*<B)#-",, "-)#);#. H/# /4#", 9*)$ C+#"-), "77+ C" (+#/, '6µ,+», "#"B<*/, ) (. 39"µ9,#,@-6$, ) )9)4)$ =&µ5)&7/;/, -)&$ ?*>=-/$ #" /9,7<1)&# -6# /776#,(> 1*"µµ"-)=/,*+.
-/01234 .56µ5/7/0894.
Website: Visitors Overview
http://www.meta-net.eu 18
began sending out press release
European Day of Languages
unusually high traffic
Website: Visitors’ Cities
http://www.meta-net.eu 19
Strategic Research Agenda META-VISION
http://www.meta-net.eu 20
Three Ingredients
21
Appropriate Programme
Vision & Agenda
Appropriate Actors
Research & Commercialisation
Appropriate Support
Funding
http://www.meta-net.eu
Strategic Research Agenda
http://www.meta-net.eu 22
q META-NET Strategic Research Agenda for Multilingual Europe 2020.
q Addresses the problems we found when preparing the white papers.
q Three priority research themes and application/innovation scenarios.
q Can put Europe ahead of its competitors in this technology area.
q 190+ contributors.
q Final version to be ready in Nov. 2012.
q SRA will be presented to the EC and national bodies.
Strategic Research Agenda
http://www.meta-net.eu 23
Priority Themes: 3 + 2
q Three Priority Research Themes: § Translation Cloud
§ Social Intelligence and e-Participation
§ Socially-Aware Interactive Assistant
q Two additional themes: § European Language Technology
Platform
§ Core Technologies for Language Analysis and Production
http://www.meta-net.eu 24
Gartner Hype Cycle 2012
13 of the 48 emerging technologies are
language technologies!
Conclusions META-NET
http://www.meta-net.eu 26
Conclusions
q Our white paper press campaign shows that Europe is extremely interested in and passionate about its languages.
q Two Parliamentary Questions in the European Parliament on the “digital extinction of languages” topic.
q Now is the time to move forward with a continent-wide, systematic push and to invest in strategic research.
q A modest investment is required.
q This push will generate a countless number of opportunities.
q Horizon 2020 and Connecting Europe Facility can provide sufficient resources to make our visions for Europe’s citizens and economy a reality.
http://www.meta-net.eu 27
Labai ačiū!
Thank you very much! [email protected] http://www.meta-net.eu http://www.facebook.com/META.Alliance
28
Q/A
2013 2020
Maturity
Database Technology Language TechnologyNetwork Technology
Language Technology
Web Technology