Proceedings of the 2nd Workshop on Building Educational Applications Using NLP,pages 53–60, Ann Arbor, June 2005.c©Association for Computational Linguistics, 2005

Direkt Profil: A System for Evaluating Texts of Second Language Learnersof French Based on Developmental Sequences

Jonas Granfeldt1 Pierre Nugues2 Emil Persson1 Lisa Persson2

Fabian Kostadinov3 Malin Ågren1 Suzanne Schlyter1

1Dept. of Romance Languages 2Dept. of Computer Science 3Dept. of Computer ScienceLund University Lund University University of Zurich

Box 201, 221 00 Lund, Sweden Box 118, 221 00 Lund, Sweden CH-8057 Zurich, Switzerland{Jonas.Granfeldt, Malin.Agren, Suzanne.Schlyter}@rom.lu.se

emil.person@telia.com nossrespasil@hotmail.comPierre.Nugues@cs.lth.se fabian.kostadinov@access.unizh.ch

Abstract

Direkt Profil is an automatic analyzer oftexts written in French as a second lan-guage. Its objective is to produce an eval-uation of the developmental stage of stu-dents under the form of a grammaticallearner profile. Direkt Profil carries outa sentence analysis based on developmen-tal sequences, i.e. local morphosyntacticphenomena linked to a development in theacquisition of French.The paper presents the corpus that we useto develop the system and briefly, the de-velopmental sequences. Then, it describesthe annotation that we have defined, theparser, and the user interface. We con-clude by the results obtained so far: on thetest corpus the systems obtains a recall of83% and a precision of 83%.

1 IntroductionWith few exceptions, systems for evaluating lan-guage proficiency and for Computer-Assisted Lan-guage Learning (CALL) do not use Natural Lan-guage Processing (NLP) techniques. Typically, ex-isting commercial and non-commercial programsapply some sort of pattern-matching techniques toanalyze texts. These techniques not only reduce thequality and the nature of the feedback but also limitthe range of possible CALL applications.

In this paper, we present a system that imple-ments an automatic analysis of texts freely writtenby learners. Research on Second Language Acqui-sition (SLA) has shown that writing your own textin a communicative and meaningful situation witha feedback and/or an evaluation of its quality andits form constitutes an excellent exercise to developsecond language skills.The aim of the program, called Direkt Profil, is

to evaluate the linguistic level of the learners’ textsin the shape of a learner profile. To analyze sen-tences, the program relies on previous research onsecond language development in French that item-ized a number of specific constructions correspond-ing to developmental sequences.

2 The CEFLE Lund CorpusFor the development and the evaluation of the sys-tem, we used the CEFLE corpus (Corpus Écrit deFrançais Langue Étrangère de Lund “Lund WrittenCorpus of French as a Foreign Language”). Thiscorpus currently contains approximately 100,000words (Ågren, 2005). The texts are narratives ofvaried length and levels. We elicited them by ask-ing 85 Swedish high-school students and 22 youngFrench to write stories evoked by a sequence of im-ages. Figure 1 shows pictures corresponding to oneof them: Le voyage en Italie “The journey to Italy”.The goal of the system being to analyze French asa foreign language, we used the texts of the Frenchnative speakers as control group.The following narrative is an example from a be-

53

ginner learner:

Elles sont deux femmes. Elles sont aitalie au une vacanse. Mais L’Auto esttrès petite. Elles va a Italie. Au l’hothelelles demande une chambre. Un homme ale clé. Le chambre est grande avec deuxlies. Il fait chaud. C’est noir. Cettedeux femmes est a une restaurang. Dansune bar cet deux hommes. Ils amour lesfemmes. Ils parlons dans la bar. Ils onttres bien. Le homme et la femme partic-ipat a un sightseeing dans la Rome. Ilsachetons une robe. La robe est verte. Lafemme et l’homme reste au un banqe. Ilsc’est amour. La femme et l’homme estau une ristorante. es hommes va avec lesfemmes. L’auto est petite.

This text contains a certain number of typicalconstructions for French as a foreign language:parataxis, very simple word order, absence of ob-ject pronouns, basic verb forms, agreement errors,spelling mistakes. Research on the acquisition ofFrench as a foreign language has shown that theseconstructions (and others) appear in a certain sys-tematic fashion according to the proficiency level ofthe learners. With Direkt Profil, we aim at detectingautomatically these structures and gathering themso that they represent a grammatical learner profile.This learner profile can ultimately be used to assesslearners’ written production in French.

3 Direkt Profil and Previous WorkDirekt Profil is an analyzer of texts written in Frenchas a foreign language. It is based on the linguisticconstructions that are specific to developmental se-quences. We created an annotation scheme to markup these constructions and we used it to describethem systematically and detect them automatically.The analyzer parses the text of a learner, annotatesthe constructions, and counts the number of occur-rences of each phenomenon. The result is a text pro-file based on these criteria and, possibly, an indica-tion of the level of the text. A graphical user inter-face (GUI) shows the results to the user and visual-izes by different colors the detected structures. It isimportant to stress that Direkt Profil is not a gram-mar checker.

The majority of the tools in the field can be de-scribed as writing assistants. They identify andsometimes correct spelling mistakes and grammat-ical errors. The line of programs leading to PLNLP(Jensen et al., 1993) and NLPWin (Heidorn, 2000)is one of the most notable achievements. The gram-matical checker of PLNLP carries out a completeparse. It uses binary phrase-structure rules and takesinto account some dependency relations. PLNLP istargeted primarily, but not exclusively, to users writ-ing in their mother tongue. It was created for En-glish and then applied to other languages, includingFrench.Other systems such as FreeText (Granger et al.,

2001) and Granska (Bigert et al., 2005) are rele-vant to the CALL domain. FreeText is specificallydesigned to teach language and adopts a interactiveapproach. It uses phrase-structure rules for French.In case of parsing failure, it uses relaxed constraintsto diagnose an error (agreement errors, for exam-ple). Granska, unlike FreeText, carries out a par-tial parsing. The authors justify this type of analysisby a robustness, which they consider superior andwhich makes it possible to accept more easily incor-rect sentences.

4 An Analysis Based on DevelopmentalSequences

The current systems differ with regard to the typeof analysis they carry out: complete or partial. Thecomplete analysis of sentences and the correction oferrors are difficult to apply to texts of learners with(very) low linguistic level since the number of un-known words and incorrect sentences are often ex-tremely high.We used a test corpus of 6,842 words to evalu-

ate their counts. In the texts produced by learnersat the lowest stage of development, Stage 1, nearly100% of the sentences contained a grammatical er-ror (98.9% were incorrect1) and 24.7% of the wordswere unknown.2 At this stage of development, anycomplete analysis of the sentences seems very diffi-cult to us. On the other hand, in the control group the

1An “incorrect sentence” was defined as a sentence contain-ing at least one spelling, syntactic, morphological, or semanticerror.

2An “unknown word” is a token that does not appear in thelexicon employed by the system (ABU CNAM, see below)

54

Figure 1: Le voyage en Italie “The journey to Italy”.

corresponding figures are 32.7% for incorrect sen-tences and 10.6% for unknown words. More impor-tantly, this analysis shows that using a quantificationof “unknown words” and “incorrect sentences” onlyis insufficient to define the linguistic level of learn-ers’ texts. Learners at Stage 3 have in fact fewer in-correct sentences than learners from Stage 4 (70.5%vs. 80.2%). Moreover, the percentage of unknownwords in the control group (the natives) is slightlyhigher than that of learners from the Stage 4 (10.6%vs. 10.4%). Thus, the simple count of errors isalso insufficient to distinguish more advanced learn-ers from natives. To identify properly and to definelearners of various linguistic levels, we need moredetailed analyses and more fine-grained measures.This is exactly the purpose of the developmental se-quences and learner profiles implemented in DirektProfil.

5 Developmental Sequences in FrenchDirekt Profil carries out an analysis of local phenom-ena related to a development in the acquisition ofFrench. These phenomena are described under theform of developmental sequences. The sequencesare the result of empirical observations stemming

from large learner corpora of spoken language (Bart-ning and Schlyter, 2004). They show that certaingrammatical constructions are acquired and can beproduced in spontaneous spoken language in a fixedorder. Clahsen and al. (1983) as well as Piene-mann and Johnston, (1987) determined developmen-tal sequences for German and spoken English. Forspoken French, Schlyter (2003) and Bartning andSchlyter (2004) proposed 6 stages of developmentand developmental sequences covering more than 20local phenomena. These morphosyntactic phenom-ena are described under the form of local structuresinside the verbal or nominal domain. Table 1 showsa subset of these phenomena. It is a matter of currentdebate in field of SLA to what extent these devel-opmental sequences are independent of the mothertongue.The horizontal axis indicates the temporal devel-

opment for a particular phenomenon: The develop-mental sequence. The vertical axis indicates the setof grammatical phenomena gathered in such waythat they make up a “profile” or a stage of acqui-sition. To illustrate better how this works, we willcompare the C (finite verb forms in finite contexts)and G (object pronouns) phenomena.

55

At Stage 1, the finite and infinitive forms coexistin finite contexts. As the main verb of the sentence,we find in the learners’ production je parle (tran-scription of /je parl/ analyzed as a “finite form”) aswell as /je parle/ i.e. *je parler or *je parlé. The cur-rent estimation is that in Stage 1, there are between50 and 75% of finite forms in finite contexts. AtStage 4, the percentage of finite forms has increasedto 90–98%. For this morphological phenomenon,the developmental sequence describes a successive“morphologization”.The G phenomenon concerns the developmental

sequence of object pronouns. The first object pro-nouns are placed in a postverbal position accordingto the scheme Subject-Verb-Object (SVO), e.g. *jevois le/la/lui (instead of je le/la vois). At Stage 3,learners can produce phrases according to the SvOVscheme (Pronoun-Auxiliary-Object-Verb): Je veuxle voir (correct) but also *j’ai le vu (incorrect). AtStage 5, we observe je l’ai vu. For this syntactic phe-nomenon, the developmental sequence describes achange in the linear organization of the constituents.

6 Annotation

The concept of group, either noun group or verbgroup, correct or not, represents the essential gram-matical support of our annotation. The majority ofsyntactic annotation standards for French takes suchgroups into account in one way or another. Gendneret al. (2004) is an example that reconciles a greatnumber of annotations. These standards are how-ever insufficient to mark up all the constructions inTable 1.We defined a text annotation specific to Direkt

Profil based on the inventory of the linguistic phe-nomena described by Bartning and Schlyter (2004)(Table 1). We represented these phenomena by de-cision trees whose final nodes correspond to a cate-gory of analysis.The annotation uses the XML format and anno-

tates the texts using 4 layers. Only the 3rd layer isreally grammatical:

• The first layer corresponds to the segmentationof the text in words.

• The second layer annotates prefabricated ex-pressions or sentences (e.g. je m’appelle).

These structures correspond to linguistic ex-pressions learned “by heart” in a holistic fash-ion. It has been shown that they have a greatimportance in the first years of learning French.

• The third layer corresponds to a chunk anno-tation of the text, restricted to the phenomenato identify. This layer marks up simulta-neously each word with its part-of-speechand the verb and noun groups to which theybelong. The verb group incorporates subjectclitic pronouns. The XML element spanmarks the groups and features an attributeto indicate their class in the table. The tagelement annotates the words with attributes toindicate the lemma, the part-of-speech, andthe grammatical features. The verb group inthe sentence Ils parlons dans la bar extractedfrom the learner text above is annotated as:<span class="p1_t1_c5131"><tagpos="pro:nom:pl:p3:mas">Ils</tag><tag pos="ver:impre:pl:p1">parlons </tag></span> dans labar. The class denoted p1_t1_c5131corresponds to a “finite lexical verb, noagreement”.

• The fourth layer counts structures typical of anacquisition stage. It uses the counter XMLelement, <counter id="counter.2"counter_name="passe_compose"rule_id="participe_4b"value="1"/>.

7 Implementation

The running version of Direkt Profil is restricted tothe analysis of the verb groups and clitic pronouns.For each category in Table 1, the program identifiesthe corresponding constructions in a text and countsthem.The analyzer uses manually written rules and a

lexicon of inflected terms. The variety of the con-structions contained in the corpus is large and in or-der not to multiply the number of rules, we chosea constraint reinforcement approach. Conceptually,the analyzer seeks classes of phrase structures inwhich all the features are removed. It graduallyidentifies the structures while varying the feature

56

Ph. Stages 1 2 3 4 5 6A. % of sentences

containing a verb(in a conversa-tion)

20–40% 30–40% 50% 60% 70% 75%

B. % of lexicalverbs showing+/-finite opposi-tion (types)

No opp.;% in finiteforms1–3sg

10–20%of types inopposition

About 50%in opposi-tion

Most in op-position

All in op-position

+

C. % of finiteforms of lexicalverbs in oblig-atory contexts(occurrences)

Finiteforms50%–75%

Finiteforms70–80%

Finiteforms:80–90%

Finiteforms:90–98%

Finiteforms:100%

+

D. 1st, 2nd, 3rd

pers. sing.(copula/aux)est, a, va

No opposi-tion:J’ai/ c’est

Oppositionj’ai – il aje suis – ilest

Isolated er-rors *je va,*je a

+ + +

E. % of 1st pers.plural S-V agree-ment nous V-ons(occurrences)

– 70–80% 80–95% Errors incomplexconstruc-tions

+ +

F. 3rd pers. pluralS-V agreementwith viennent,veulent, pren-nent

– –ils *prend

Isolatedcases ofagreement

50% ofcases withagreement

Someproblemsremain

+

G. Object pronouns(placement)

– SVO S(v)oV SovVappears

Productive + (y, en)

H. % of genderagreementArticle-Noun(occurrences)

55–75% 60–80% 65–85% 70–90% 75–95% 90–100%

Table 1: Developmental sequences adapted from Schlyter (2003); Bartning and Schlyter (2004).Legend: – = no occurrences; + = acquired at a native-like level; aux = auxiliary; pers. = person; S-V =Subject-Verb

57

values. The recognition of the group boundaries isdone by a set of closed-class words and heuristicsinside the rules. It thus follows an old but robuststrategy used in particular by Vergne (1999), interalia, for French.

Direkt Profil applies a cascade of three sets ofrules to produce the four annotation layers. Thefirst unit segments the text in words. An interme-diate unit identifies the prefabricated expressions.The third unit annotates simultaneously the parts-of-speech and the groups. Finally, the engine creates agroup of results and connects them to a profile. Itshould be noted that the engine neither annotates allthe words, nor all segments. It considers only thosewhich are relevant for the determination of the stage.The engine applies the rules from left to right thenfrom right to left to solve certain problems of agree-ment.

The rules represent partial structures and are di-vided into a condition part and an action part. Thecondition part contains the search parameters. It canbe a lemma, a regular expression, or a class of inflec-tion. The engine goes through the text and appliesthe rules using a decision tree. It tests the conditionpart to identify the sequences of contiguous words.Each rule produces a positive (“match”) or negative(“no match”) result. The rules are applied accord-ing to the result of the condition part and annotatethe text, count the number of occurrences of the phe-nomenon, and connect to another rule. By traversingthe nodes of the tree, the engine memorizes the rulesit has passed as well as the results of the conditionparts of these rules. When arriving at a final node,the engine applies the action parts of all the rules.

The engine finds the words in a dictionary ofinflected terms. It does not correct the spellingmistakes except for the accents and certain stems.Learners frequently build erroneous past participlesinferring a wrong generalization of stems. An exam-ple is the word *prendu (taken) formed on the stemprend|re and of the suffix -u.We used a lexicon available from the As-

sociation des Bibliophiles Universels’ web site(http://abu.cnam.fr/) that we corrected and trans-posed into XML. We also enriched it with verbstems.

8 InterfaceDirekt Profil merges the annotation levels in a resultobject. This object represents the original text, theannotation, the trace of the rule application, and thecounters. The result object, which can be saved, isthen transformed by the program to be presented tothe user. The display uses the XHTML 1.1 spec-ifications which can be read by any Web browser.Direkt Profil has a client-server architecture wherethe server carries out the annotation of a text and theclient collects the text with an input form and inter-acts with the user.Figure 2 shows a screenshot of Direkt Profil’s

GUI displaying the analysis of the learner textabove. The interface indicates to the user by dif-ferent colors all the structures that the analyzer de-tected.

9 Results and EvaluationWe evaluated Direkt Profil with a subset of the CE-FLE corpus. We chose 20 texts randomly distributedon 4 learner stages. We also used 5 texts comingfrom the control group. In this version, we did nottest the correction of the misspelled words: accentand stems. Table 2 shows some statistics on the sizeof the texts and Table 3 shows the results in the formof recall and precision.The results show that Direkt Profil detects well

the desired phenomena. It reveals also interestingdifferences according to the levels of the texts. Theresults show that Direkt Profil analyzes better thelearner texts than the texts from the native Frenchadolescents (control group). Without knowing ex-actly why, we note that it suggests that the adoptedstrategy, which aims at analyzing texts in French asa foreign language, seems promising.

10 Conclusion and Future WorkWe presented a system carrying out a machine anal-ysis of texts based on developmental sequences. Thegoal is to produce a learner profile. We built a parserand developed a set of rules to annotate the texts. Di-rekt Profil is integrated in a client-server architectureand has an interface allowing the interaction with theuser.The results show that it is possible to describe the

vast majority of the local structures defined by the

58

Figure 2: The graphical user interface.

Stage 1 Stage 2 Stage 3 Stage 4 Control TotalNumber of analyzed texts 5 5 5 5 5 25Word count 740 1233 1571 1672 1626 6842Sentence count 85 155 166 126 107 639Average text length (in words) 148 247 314 334 325 274Average length of sentences (in words) 8.7 7.9 9.5 13.3 15.2 10.9

Table 2: Test corpus.

Stage 1 Stage 2 Stage 3 Stage 4 Control TotalReference structures 23 97 101 119 85 425Detected structures 27 98 100 112 92 429Correctly detected structures 15 81 89 96 73 354Non detected structures 5 16 12 20 11 ()64Overdetected structures 10 17 11 17 19 ()74Recall 65% 84% 88% 81% 86% 83%Precision 56% 83% 89% 86% 79% 83%F-measure 0.6 0.83 0.89 0.83 0.82 0.83

Table 3: Results.

59

developmental sequences under the form of rules.Direkt Profil can then detect them and automaticallyanalyze them. We can thus check the validity of theacquisition criteria.In the future, we intend to test Direkt Profil in

teaching contexts to analyze and specify, in an au-tomatic way, the grammatical level of a learner. Theprogram could be used by teachers to assess studenttexts as well as by the students themselves as a self-assessment and as a part of their learning process.A preliminary version of Direkt Pro-

fil is available on line from this addresshttp://www.rom.lu.se:8080/profil

ReferencesMalin Ågren. 2005. Le marquage morphologiquedu nombre dans la phrase nominale. une étude surl’acquisition du français L2 écrit. Technical report, In-stitut d’études romanes de Lund. Lund University.

Inge Bartning and Suzanne Schlyter. 2004. Stades etitinéraires acquisitionnels des apprenants suédophonesen français l2. Journal of French Language Studies,14(3):281–299.

Johnny Bigert, Viggo Kann, Ola Knutsson, and JonasSjöbergh. 2005. Grammar checking for Swedish sec-ond language learners. In CALL for the Nordic Lan-guages, Copenhagen Studies in Language, pages 33–47. Copenhagen Business School, Samfundslitteratur.

Harald Clahsen, Jürgen M. Meisel, and Manfred Piene-mann. 1983. Deutsch als Fremdsprache. DerSpracherwerb ausländischer Arbeiter. Narr, Tübin-gen.

Véronique Gendner, Anne Vilnat, Laura Mon-ceaux, Patrick Paroubek, and Isabelle Robba.2004. Les annotations syntaxiques de référencepeas. Technical report, LIMSI, Orsay.http://www.limsi.fr/Recherche/CORVAL/easy/PEAS_reference_annotations_v1.6.html.

Sylviane Granger, Anne Vandeventer, and Marie-JoséeHamel. 2001. Analyse de corpus d’apprenants pourl’ELAO basé sur le TAL. Traitement automatique deslangues, 42(2):609–621.

George E. Heidorn. 2000. Intelligent writing assistance.In Robert Dale, Hermann Moisl, and Harold Somers,editors, Handbook of Natural Language Processing.Marcel Dekker.

Karen Jensen, George E. Heidorn, and Stephen D.Richardson. 1993. Natural Language Processing:The PLNLP Approach. Kluwer Academic Publishers.

Manfred Pienemann and Malcolm Johnston. 1987. Fac-tors influencing the development of second languageproficiency. In David Nunan, editor, Applying sec-ond language acquisition research, pages 45–141. Na-tional Curriculum Resource Centre, Adelaide.

Suzanne Schlyter. 2003. Stades de développe-ment en français L2. Technical report, Insti-tut d’études romanes de Lund, Lund University.http://www.rom.lu.se/durs/STADES_DE_DEVELOPPEMENT_EN_FRANCAIS_L2.PDF.

Jacques Vergne. 1999. Étude et modélisation de lasyntaxe des langues à l’aide de l’ordinateur. Analysesyntaxique automatique non combinatoire. Synthèse etRésultats. Habilitation à diriger des recherches, Uni-versité de Caen, 29 septembre.

60