Automated Topical Component Extraction Using Neural NetworkAttention Scores from Source-based Essay Scoring

Haoran ZhangDepartment of Computer Science

University of PittsburghPittsburgh, PA 15260

colinzhang@cs.pitt.edu

Diane LitmanDepartment of Computer Science & LRDC

University of PittsburghPittsburgh, PA 15260

litman@cs.pitt.edu

Abstract

While automated essay scoring (AES) can re-liably grade essays at scale, automated writingevaluation (AWE) additionally provides forma-tive feedback to guide essay revision. How-ever, a neural AES typically does not provideuseful feature representations for supportingAWE. This paper presents a method for link-ing AWE and neural AES, by extracting Top-ical Components (TCs) representing evidencefrom a source text using the intermediate out-put of attention layers. We evaluate perfor-mance using a feature-based AES requiringTCs. Results show that performance is compa-rable whether using automatically or manuallyconstructed TCs for 1) representing essays asrubric-based features, 2) grading essays.

1 Introduction

Automated essay scoring (AES) systems reliablygrade essays at scale, while automated writing eval-uation (AWE) systems additionally provide forma-tive feedback to guide revision. Although neuralnetworks currently generate state-of-the-art AESresults (Alikaniotis et al., 2016; Taghipour and Ng,2016; Dong et al., 2017; Farag et al., 2018; Jin et al.,2018; Li et al., 2018; Tay et al., 2018; Zhang andLitman, 2018), non-neural AES create feature rep-resentations more easily useable by AWE (Roscoeet al., 2014; Foltz and Rosenstein, 2015; Crossleyand McNamara, 2016; Woods et al., 2017; Madnaniet al., 2018; Zhang et al., 2019). We believe thatneural AES can also provide useful information forcreating feature representations, e.g., by exploitinginformation in the intermediate layers.

Our work focuses on a particular source-basedessay writing task called the response-to-text as-sessment (RTA) (Correnti et al., 2013). Recently,an RTA AWE system (Zhang et al., 2019) was builtby extracting rubric-based features related to the

use of Topical Components (TCs) in an essay. How-ever, manual expert effort was first required to cre-ate the TCs. For each source, the TCs consist ofa comprehensive list of topics related to evidencewhich include: 1) important words indicating theset of evidence topics in the source, and 2) phrasesrepresenting specific examples for each topic thatstudents need to find and use in their essays.

To eliminate this expert effort, we propose amethod for using the interpretable output of theattention layers of a neural AES for source-basedessay writing, with the goal of extracting TCs. Weevaluate this method by using the extracted TCsto support feature-based AES for two RTA sourcetexts. Our results show that 1) the feature-basedAES with TCs manually created by humans ismatched by our neural method for generating TCs ,and 2) the values of the rubric-based essay featuresbased on automatic TCs are highly correlated withhuman Evidence scores.

2 Related Work

Three recent AWE systems have used non-neuralAES to provide rubric-specific feedback. Woodset al. (2017) developed an influence estimation pro-cess that used a logistic regresion AES to identifysentences needing feedback. Shibani et al. (2019)presented a web-based tool that provides formativefeedback on rhetorical moves in writing. Zhanget al. (2019) used features created for a randomforest AES to select feedback messages, althoughhuman effort was first needed to create TCs froma source text. We automatically extract TCs usingneural AES, thereby eliminating this expert effort.

Others have also proposed methods for pre-processing source information external to an es-say. Content importance models for AES predictthe parts of a source text that students should in-clude when writing a summary (Klebanov et al.,

arX

iv:2

008.

0180

9v1

[cs

.CL

] 4

Aug

202

0

Source Excerpt: Today, Yala Sub-District Hospital has medicine, free of charge, for all of the most common diseases. Wateris connected to the hospital, which also has a generator for electricity. Bed nets are used in every sleeping site in Sauri...Essay Prompt: The author provided one specific example of how the quality of life can be improved by the Millennium VillagesProject in Sauri, Kenya. Based on the article, did the author provide a convincing argument that winning the fight against poverty isachievable in our lifetime? Explain why or why not with 3-4 examples from the text to support your answer.Essay: In my opinion I think that they will achieve it in lifetime. During the years threw 2004 and 2008 they made progress.People didnt have the money to buy the stuff in 2004. The hospital was packed with patients and they didnt have alot of treatmentin 2004. In 2008 it changed the hospital had medicine, free of charge, and for all the common dieases. Water was connectedto the hospital and has a generator for electricity. Everybody has net in their site. The hunger crisis has been addressed withfertilizer and seeds, as well as the tools needed to maintain the food. The school has no fees and they serve lunch. To me thatssounds like it is going achieve it in the lifetime.

Table 1: A source excerpt for the RTAMV P prompt and an essay with score of 3.

Prompt RTAMV P RTASpace

Score 1 852 538(29%) (26%)

Score 2 1197 789(40%) (38%)

Score 3 616 512(21%) (25%)

Score 4 305 237(10%) (11%)

Total 2970 2076

Table 2: The Evidence score distribution of RTA.

2014). Methods for extracting important keywordsor keyphrases also exist, both supervised (unlikeour approach) (Meng et al., 2017; Mahata et al.,2018; Florescu and Jin, 2018) and unsupervised(Florescu and Caragea, 2017). Rahimi and Litman(2016) developed a TC extraction LDA model (Bleiet al., 2003). While the LDA model considers allwords equally, our model takes essay scores intoaccount by using attention to represent word impor-tance. Both the unsupervised keyword and LDAmodels will serve as baselines in our experiments.

In the computer vision area, attention croppedimages have been used for further image classifi-cation or object detection (Cao et al., 2015; Yuxinet al., 2018; Ebrahimpour et al., 2019). In the NLParea, Lei et al. (2016) proposed to use a genera-tor to find candidate rationale and these are passedthrough the encoder for prediction. Our work issimilar in spirit to this type of work.

3 RTA Corpus and Prior AES Systems

The essays in our corpus were written by studentsin grades 4 to 8 in response to two RTA sourcetexts (Correnti et al., 2013): RTAMV P (2970 es-says) and RTASpace (2076 essays). Table 1 showsan excerpt from RTAMV P , the associated essaywriting prompt, and a student essay. The boldingin the source indicates evidence examples that ex-

perts manually labeled as important for studentsto discuss (i.e., TC phrases). Evidence usage ineach essay was manually scored on a scale of 1to 4 (low to high). The distribution of Evidencescores is shown in Table 2. The essay in Table 1received a score of 3, with the bolding indicatingphrases semantically related to the TCs from thesource text.

To date, two approaches to AES have been pro-posed for the RTA: AESrubric and AESneural. Tosupport the needs of AWE, AESrubric (Zhang andLitman, 2017) used a traditional supervised learn-ing framework where rubric-motivated featureswere extracted from every essay before model train-ing - Number of Pieces of Evidence (NPE) 1, Con-centration (CON), Specificity (SPC) 2, Word Count(WOC). The two aspects of TCs introduced in Sec-tion 1 (topic words, specific example phrases) wereused during feature extraction.

Motivated by improving stand-alone AES perfor-mance (i.e., when an interpretable model was notneeded for subsequent AWE), Zhang and Litman(2018) developed AESneural, a hierarchical neuralmodel with the co-attention mechanism in the sen-tence level to capture the relationship between theessay and the source. Neither feature engineeringnor TC creation were needed before training.

4 Attention-Based TC Extraction: TCattn

In this section we propose a method for extract-ing TCs based on the AESneural attention leveloutputs. Since the self-attention and co-attentionmechanisms were designed to capture sentenceand phrase importance, we hypothesize that theattention scores can help determine if a sentence or

1An integer feature based on the list of topic words foreach topic.

2A vector of integer values indicating the number of spe-cific example phrases (semantically) mentioned in the essayper topic.

No. Sentences attnsent attnphrase

1 People didn’t have the money tobuy the stuff in 2004.

0.00420 0.23372

2 The hunger crisis has been addressedwith fertilizer and seeds, as well asthe tools needed to maintain the food.

0.08709 0.62848

3 The school has no fees and theyserve lunch.

0.10686 0.63369

Table 3: Example attention scores of essay sentences.

phrase has important source-related information.To provide intuition, Table 3 shows examples

sentences from the student essay in Table 1. Boldedare phrases with the highest self-attention scorewithin the sentence. Italics are specific examplephrases that refer to the manually constructed TCsfor the source. Attnsent is the text to essay atten-tion score that measures which essay sentenceshave the closest meaning to a source sentence.Attnphrase is the self-attention score of the boldedphrase that measures phrase importance. A sen-tence with a high attention score tends to include atleast one specific example phrase, and vice versa.The phrase with the highest attention score tendsto include at least one specific example phrase ifthe sentence has a high attention score.

Based on these observations, we first extract theoutput of two layers from the neural network: 1)the attnsent of each sentence, and 2) the output ofthe convolutional layer as the representation of thephrase with the highest attnphrase in each sentence(denoted by cnnphrase). We also extract the plaintext of the phrase with the highest attnphrase ineach sentence (denoted by textphrase). Then, ourTCattn method uses the extracted information in3 main steps: 1) filtering out textphrase from sen-tences with low attnsent, 2) clustering all remain-ing textphrase based on cnnphrase, and 3) generat-ing TCs from clusters.

The first filtering step keeps all textphrase wherethe original sentences have attnsent higher thana threshold. The intuition is that lower attnsent

indicates less source-related information.The second step clusters these textphrase based

on their corresponding representations cnnphrase.We use k-medoids to cluster textphrase into Mclusters, where M is the number of topics in thesource text. Then, for textphrase in each topiccluster, we use k-medoids to cluster them into Nclusters, where N is the number of the specificexample phrases we want to extract from each topic.The outputs of this step are M ∗N clusters.

The third step uses the topic and example clus-

Layer Parameter Name ValueEmbedding Embedding dimension 50Word-CNN Kernel size 5

Number of filters 100Sent-LSTM Hidden units 100Modeling Hidden units 100Dropout Dropout rate 0.5Others Epochs 100

Batch size 100Initial learning rate 0.001

Momentum 0.9

Table 4: Hyper-parameters for neural training.

Figure 1: An overview of four TC extraction systems.

tering to extract TCs. As noted earlier, TCs in-clude two parts: topic words, and specific examplephrases. Since our method is data-driven and stu-dents introduce their vocabulary into the corpus,essay text is noisy. To make the TC output cleaner,we filter out words that are not in the source text.To obtain topic words, we combine all textphrasefrom each topic cluster to calculate the word fre-quency per topic. To make topics unique, we assigneach word to the topic cluster in which it has thehighest normalized word frequency. We then in-clude the top Ktopic words based on their frequencyin each topic cluster. To obtain example phrases,we combine all textphrase from each example clus-ter to calculate the word frequency per example,then include the top Kexample words based on theirfrequency in each example cluster.

5 Experimental Setup and Results

Figure 1 shows an overview of four TC extractionmethods to be evaluated. TCmanual (upper bound)uses a human expert to extract TCs from a sourcetext. TCattn is our proposed method and automat-ically extracts TCs using both a source text andstudent essays. TClda (Rahimi and Litman, 2016)(baseline) builds on LDA to extract TCs from stu-dent essays only, while TCpr (baseline) builds onPositionRank (Florescu and Caragea, 2017) to in-stead extract TCs from only the source text.

Since PositionRank is not designed for TC ex-

Prompt Component Parameter TClda TCpr TCattn

RTAMV P

Topic WordsNumber of Topics 9 19 16Number of Words 30 20 25

Example PhrasesNumber of Topics 20 1 18Number of Phrases 15 20 15

RTASpace

Topic WordsNumber of Topics 15 20 10Number of Words 10 10 20

Example PhrasesNumber of Topics 10 1 9Number of Phrases 20 50 20

Table 5: Parameters for different models.

traction, we needed to further process its output tocreate TCpr. To extract topic words, we extractall keywords from the output. Next, we map eachword to a higher dimension with word embedding.Lastly, we cluster all keywords using k-medoidsinto PRtopic topics. To extract example phrases,we put them into only one topic and remove allredundant example phrases if they are subsets ofother example phrases.

We configure experiments to test two hypotheses:H1) the AESrubric model for scoring Evidence(Zhang and Litman, 2017) will perform compara-bly when extracting features using either TCattn

or TCmanual, and will perform worse when us-ing TClda or TCpr; H2) the correlation betweenthe human Evidence score and the feature values(NPE and sum of SPC features)3 will be compa-rable when extracted using TCattn and TCmanual,and will be stronger than when using TClda andTCpr. The experiment for H1 tests the impact ofusing our proposed TC extraction method on thedownstream AESrubric task, while the H2 experi-ment examines the impact on the essay representa-tion itself.

Following Zhang and Litman (2017), we stratifyessay corpora: 40% for training word embeddingsand extracting TCs, 20% for selecting the best em-bedding and parameters, and 40% for testing. Weuse the hyper-parameters from Zhang and Litman(2018) for neural training as shown in Table 4. Ta-ble 5 shows all other parameters selected using thedevelopment set.

Results for H1. H1 is supported by the results inTable 6, which compares the Quadratic WeightedKappa (QWK) between human and AESrubric Ev-idence scores (values 1-4) when AESrubric usesTCmanual versus each of the automatic methods.TCattn always yields better performance, and evensignificantly better than TCmanual.

Results for H2. The results in Table 7 supportH2. TCattn outperforms the two automated base-

3These features are extracted based on TCs.

Prompt TCmanual (1) TClda (2) TCpr (3) TCattn (4)RTAMV P 0.643 (2,3) 0.614 (3) 0.525 0.648 (1,2,3)RTASpace 0.609 (3) 0.615 (3) 0.559 0.622 (1,3)

Table 6: The performance (QWK) of AESrubric us-ing different TC extraction methods for feature cre-ation. The numbers in the parentheses show the modelnumbers over which the current model performs signif-icantly better (p < 0.05). The best results betweenautomated methods in each row are in bold.

Prompt Feature TCmanual TClda TCpr TCattn

RTAMV PNPE 0.542 0.482 0.587 0.639SPC (sum) 0.689 0.585 0.365 0.679

RTASpaceNPE 0.484 0.513 0.494 0.625SPC (sum) 0.601 0.574 0.533 0.598

Table 7: Pearson’s r comparing feature values com-puted using each TC extraction method with human(gold-standard) Evidence essay scores. All correlationvalues are significant (p ≤ 0.05). The best results be-tween automated methods in each row are in bold.

lines, and for NPE even yields stronger correlationsthan the manual TC method.

Qualitative Analysis. The manually-createdtopic words for RTAMV P represent 4 topics,which are “hospital”, “malaria”, “farming” and“school”4. Although Table 5 shows that the au-tomated list has more topics for topic words andmight have broken one topic into separate topics,a good automated list should have more topics re-lated to the 4 topics above. We manually assign atopic for each of the topic words from the differentautomated methods. TClda has 4 related topics outof 9 (44.44%), TCpr has 6 related topics out of 19(31.58%), and TCattn has 10 related topics out of16 (62.50%). Obviously, TCattn preserves morerelated topics than our baselines.

Moving to the second aspect of TCs (specificexample phrases), Table 8 shows the first 10 spe-cific example phrases for a manually-created cat-egory that introduces the changes made by theMVP project5. This category is a mixture of dif-ferent topics because it talks about the “hospital”,“malaria”, “school”, and “farming” at the same time.TCattn has overlap with TCmanual on differenttopics. However, TClda mainly talks about “hospi-tal”, because the nature of the LDA model doesn’tallow mixing specific example phrases about dif-ferent topics in one category. Unfortunately, TCpr

4All Topic Words generated by different models can befound in the Appendix A.1.

5All Specific Example Phrases generated by different mod-els can be found in the Appendix A.2.

TCmanual TClda TCpr TCattn

progress just four years running water electricity brighter future hannah electricity running water irrigation setmedicine most common diseases water connected hospital generator electricity millennium villages project poor showed treatment school supplies

water connected hospital patients afford unpaved dirt road farmers could crops afford bedhospital generator electricity rooms packed patients probably bar sauri primary school electricity hospital

bed nets used every sleeping site share beds future hannah better fertilizer medicine enough alsohunger crisis addressed fertilizer seeds recieve treatment sauri primary school rooms packed patients

tools needed maintain food supply doctor clinical officer running hospital villages project food fertilizer crops get supplyno school fees doctors clinical millennium development goals five net costs 5

school attendance rate way up water fertilizer knowledge village leaders nets net bed freekids go school now receive treatment dirt road running water supplies schools almost

... ... ... ...

Table 8: Specific example phrases for the RTAMV P progress topic.

does not include any overlapped specific phrase inthe first 10 items; they all refer to some generalexample phrases from the beginning of the sourcearticle. Although there are some related specificexample phrases in the full list, they are mainlyabout school. This is because the PositionRankalgorithm tends to assign higher scores to wordsthat appear early in the text.

6 Conclusion and Future Work

This paper proposes TCattn, a method for using theattention scores in a neural AES model to automat-ically extract the Topical Components of a sourcetext. Evaluations show the potential of TCattn

for eliminating expert effort without degradingAESrubric performance or the feature represen-tations themselves. TCattn outperforms baselinesand generates comparable or even better resultsthan a manual approach.

Although TCattn outperforms all baselines andrequires no human effort on TC extraction, annota-tion of essay evidence scores is still needed. Thisleads to an interesting future investigation direc-tion, which is training the AESneural using thegold standard that can be extracted automatically.

One of our next steps is to investigate the im-pact of TC extraction methods on a correspondingAWE system (Zhang et al., 2019), which uses thefeature values produced by AESrubric to generateformative feedback to guide essay revision.

Currently, the TClda are trained on student es-says, while the TCpr only works on the sourcearticle. However, TCattn uses both student essaysand the source article for TC generation. It mightbe hard to say that the superior performance ofTCattn is due to the neural architecture and atten-tion scores rather than the richer training resources.Therefore, a comparison between TCattn and amodel that uses both student essays and the sourcearticle is needed.

Acknowledgments

We would like to show our appreciation to everymember of the RTA group for sharing their pearlsof wisdom with us. We are also immensely gratefulto all members of the PETAL group and reviewersfor their comments on an earlier version of thepaper.

The research reported here was supported, inwhole or in part, by the Institute of EducationSciences, U.S. Department of Education, throughGrant R305A160245 to the University of Pitts-burgh. The opinions expressed are those of theauthors and do not represent the views of the Insti-tute or the U.S. Department of Education.

ReferencesDimitrios Alikaniotis, Helen Yannakoudakis, and

Marek Rei. 2016. Automatic text scoring using neu-ral networks. In Proceedings of the 54th AnnualMeeting of the Association for Computational Lin-guistics (Volume 1: Long Papers), volume 1, pages715–725.

David M Blei, Andrew Y Ng, and Michael I Jordan.2003. Latent dirichlet allocation. Journal of ma-chine Learning research, 3(Jan):993–1022.

Chunshui Cao, Xianming Liu, Yi Yang, Yinan Yu,Jiang Wang, Zilei Wang, Yongzhen Huang, LiangWang, Chang Huang, Wei Xu, et al. 2015. Lookand think twice: Capturing top-down visual atten-tion with feedback convolutional neural networks.In Proceedings of the IEEE International Confer-ence on Computer Vision, pages 2956–2964.

Richard Correnti, Lindsay Clare Matsumura, LauraHamilton, and Elaine Wang. 2013. Assessing stu-dents’ skills at writing analytically in response totexts. The Elementary School Journal, 114(2):142–177.

Scott A Crossley and Danielle S McNamara. 2016.Adaptive educational technologies for literacy in-struction. Routledge.

Fei Dong, Yue Zhang, and Jie Yang. 2017. Attention-based recurrent convolutional neural network for au-tomatic essay scoring. In Proceedings of the 21stConference on Computational Natural LanguageLearning (CoNLL 2017), pages 153–162.

Mohammad K Ebrahimpour, Jiayun Li, Yen-Yun Yu,Jackson Reesee, Azadeh Moghtaderi, Ming-HsuanYang, and David C Noelle. 2019. Ventral-dorsal neu-ral networks: Object detection via selective attention.In 2019 IEEE Winter Conference on Applications ofComputer Vision (WACV), pages 986–994. IEEE.

Youmna Farag, Helen Yannakoudakis, and Ted Briscoe.2018. Neural automated essay scoring and coher-ence modeling for adversarially crafted input. InProceedings of the 2018 Conference of the NorthAmerican Chapter of the Association for Computa-tional Linguistics: Human Language Technologies,Volume 1 (Long Papers), pages 263–271.

Corina Florescu and Cornelia Caragea. 2017. Position-rank: An unsupervised approach to keyphrase ex-traction from scholarly documents. In Proceedingsof the 55th Annual Meeting of the Association forComputational Linguistics (Volume 1: Long Papers),pages 1105–1115.

Corina Florescu and Wei Jin. 2018. Learning featurerepresentations for keyphrase extraction. In Thirty-Second AAAI Conference on Artificial Intelligence.

Peter W Foltz and Mark Rosenstein. 2015. Analysisof a large-scale formative writing assessment systemwith automated feedback. In Proceedings of the Sec-ond (2015) ACM Conference on Learning@ Scale,pages 339–342. ACM.

Cancan Jin, Ben He, Kai Hui, and Le Sun. 2018.Tdnn: a two-stage deep neural network for prompt-independent automated essay scoring. In Proceed-ings of the 56th Annual Meeting of the Associationfor Computational Linguistics (Volume 1: Long Pa-pers), pages 1088–1097.

Beata Beigman Klebanov, Nitin Madnani, Jill Burstein,and Swapna Somasundaran. 2014. Content impor-tance models for scoring writing from sources. InProceedings of the 52nd Annual Meeting of the As-sociation for Computational Linguistics (Volume 2:Short Papers), pages 247–252.

Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016.Rationalizing neural predictions. In Proceedings ofthe 2016 Conference on Empirical Methods in Natu-ral Language Processing, pages 107–117.

Xia Li, Minping Chen, Jianyun Nie, Zhenxing Liu,Ziheng Feng, and Yingdan Cai. 2018. Coherence-based automated essay scoring using self-attention.In Chinese Computational Linguistics and NaturalLanguage Processing Based on Naturally AnnotatedBig Data, pages 386–397. Springer.

Nitin Madnani, Jill Burstein, Norbert Elliot,Beata Beigman Klebanov, Diane Napolitano,Slava Andreyev, and Maxwell Schwartz. 2018.Writing mentor: Self-regulated writing feedback forstruggling writers. In Proceedings of the 27th Inter-national Conference on Computational Linguistics:System Demonstrations, pages 113–117.

Debanjan Mahata, John Kuriakose, Rajiv Ratn Shah,and Roger Zimmermann. 2018. Key2vec: Auto-matic ranked keyphrase extraction from scientific ar-ticles using phrase embeddings. In Proceedings ofthe 2018 Conference of the North American Chap-ter of the Association for Computational Linguistics:Human Language Technologies, Volume 2 (Short Pa-pers), pages 634–639.

Rui Meng, Sanqiang Zhao, Shuguang Han, DaqingHe, Peter Brusilovsky, and Yu Chi. 2017. Deepkeyphrase generation. In Proceedings of the 55thAnnual Meeting of the Association for Computa-tional Linguistics (Volume 1: Long Papers), pages582–592.

Zahra Rahimi and Diane Litman. 2016. Automati-cally extracting topical components for a response-to-text writing assessment. In Proceedings of the11th Workshop on Innovative Use of NLP for Build-ing Educational Applications, pages 277–282.

Rod D Roscoe, Laura K Allen, Jennifer L Weston,Scott A Crossley, and Danielle S McNamara. 2014.The writing pal intelligent tutoring system: Usabilitytesting and development. Computers and Composi-tion, 34:39–59.

Antonette Shibani, Simon Knight, and Simon Bucking-ham Shum. 2019. Contextualizable learning ana-lytics design: A generic model and writing analyt-ics evaluations. In Proceedings of the 9th Interna-tional Conference on Learning Analytics & Knowl-edge, pages 210–219. ACM.

Kaveh Taghipour and Hwee Tou Ng. 2016. A neuralapproach to automated essay scoring. In Proceed-ings of the 2016 Conference on Empirical Methodsin Natural Language Processing, pages 1882–1891.

Yi Tay, Minh C Phan, Luu Anh Tuan, and Siu Che-ung Hui. 2018. Skipflow: incorporating neural co-herence features for end-to-end automatic text scor-ing. In Thirty-Second AAAI Conference on ArtificialIntelligence.

Bronwyn Woods, David Adamson, Shayne Miel, andElijah Mayfield. 2017. Formative essay feedbackusing predictive scoring models. In Proceedings ofthe 23rd ACM SIGKDD International Conferenceon Knowledge Discovery and Data Mining, pages2071–2080. ACM.

Peng Yuxin, He Xiangteng, and Zhao Junjie. 2018.Object-part attention model for fine-grained imageclassification. IEEE transactions on image process-ing: a publication of the IEEE Signal Processing So-ciety, 27(3):1487–1500.

Haoran Zhang and Diane Litman. 2017. Word embed-ding for response-to-text assessment of evidence. InProceedings of ACL 2017, Student Research Work-shop, pages 75–81.

Haoran Zhang and Diane Litman. 2018. Co-attentionbased neural network for source-dependent essayscoring. In Proceedings of the Thirteenth Workshopon Innovative Use of NLP for Building EducationalApplications, pages 399–409.

Haoran Zhang, Ahmed Magooda, Diane Litman,Richard Correnti, Elaine Wang, LC Matsmura,Emily Howe, and Rafael Quintana. 2019. erevise:Using natural language processing to provide forma-tive feedback on text evidence usage in student writ-ing. In Proceedings of the AAAI Conference on Arti-ficial Intelligence, volume 33, pages 9619–9625.

A Appendices

A.1 Topic Words ResultsTable 9 shows all topic words for the RTAMV P

from TCmanual. Table 10 shows all topic wordsfor the RTAMV P from TClda. Table 11 showsall topic words for the RTAMV P from TCpr. Ta-ble 12 shows all topic words for the RTAMV P

from TCattn.

A.2 Specific Example Phrases ResultsTable 13 shows all specific example phrases forthe RTAMV P from TCmanual. Table 14 shows allspecific example phrases for the RTAMV P fromTClda. Table 15 shows all specific example phrasesfor the RTAMV P from TCpr. Table 16 shows allspecific example phrases for the RTAMV P fromTCattn.

Topic 1 Topic 2 Topic 3 Topic 4care bed farmer school

health net fertilizer supplieshospital malaria irrigation fee

treatment infect dying studentdoctor bednet crop midday

electricity mosquito seed mealdisease bug water lunchwater sleeping harvest supplysick die hungry book

medicine cheap feed papergenerator infect food pencil

no biting energydie freekid childrenbed kid

patient goclinical attendofficer

running

Table 9: Topic words of TCmanual.

Topi

c1

Topi

c2

Topi

c3

Topi

c4

Topi

c5

Topi

c6

Topi

c7

Topi

c8

Topi

c9

help

keny

apo

vert

yfo

odm

oney

scho

olpe

ople

hosp

ital

year

spo

orlik

eth

ink

fert

ilize

rne

edki

dssa

uri

med

icin

eaf

rica

wor

ldbe

tter

auth

orcr

ops

nets

supp

lies

mal

aria

hosp

itals

proj

ect

good

know

lifet

ime

wat

erth

ing

child

ren

sick

wat

ervi

llage

sth

ings

life

artic

lefa

rmer

saf

ford

scho

ols

2008

free

saur

itim

ehe

lppo

ssib

lene

eded

dona

telu

nch

dise

ase

elec

tric

ityvi

llage

wor

kth

ink

conv

ince

dgr

owri

ght

educ

atio

n20

04di

seas

eshe

lped

hard

saur

ifig

htdy

ing

dolla

raf

ford

nets

med

icin

esch

ange

goin

gliv

epr

over

typr

oble

mtr

eatm

ent

ener

gym

osqu

itoes

doct

ors

lives

alot

clot

hes

said

fam

ilysu

rviv

ele

arn

getti

ng20

08go

als

reas

onst

ates

achi

evab

lefa

mili

esne

eds

stud

ents

says

gave

impr

oved

happ

enpl

ace

time

stop

stuf

fw

ent

year

sdo

ctor

2015

help

ing

heal

thco

nvin

cela

ckpe

rson

adul

tspr

ogre

ssex

ampl

eshe

lpgo

alim

port

ant

belie

vehu

nger

caus

efe

esdi

ed20

04ch

ange

dbe

lieve

feel

hann

ahto

ols

patie

nts

pare

nts

text

shap

eye

arpr

oble

ms

happ

ysh

ows

seed

spr

ovid

e20

04aw

aycu

rech

ange

sco

untr

ies

tell

reas

ons

plan

tsco

stlu

nche

sm

osqu

itos

runn

ing

star

ted

diff

eren

ceca

reco

nvin

cing

fert

ilize

rsbe

dsbo

oks

prev

ent

trea

tgr

eat

plac

essh

oes

fight

ing

farm

ing

mea

nsho

me

trea

ted

supp

ort

mill

enni

umch

ange

stor

yw

rote

able

dont

wan

ted

diei

ngco

mm

onpr

ogre

sslit

tleam

eric

ast

ory

solv

eddo

llars

chor

essa

idbe

dsca

me

impr

ove

way

sag

ree

supp

lym

edic

alm

eal

com

epa

tient

sgi

rlco

untr

yw

ants

sayi

ngir

riga

tion

jobs

woo

dni

ght

said

2025

achi

eve

mak

esop

inio

nw

ont

ever

yday

mat

eria

lsbi

tege

nera

tor

plac

eho

pecl

othi

ngw

inni

ngaf

ford

gone

lear

ning

deat

hcl

ean

prog

ram

help

sco

mm

unity

sach

shu

ngry

doct

ors

able

slee

pel

ectr

icty

tells

ever

ybod

yec

onom

ypr

ogre

sspl

ant

lots

supl

ies

impo

veri

shed

givi

ngsm

all

star

thi

stor

yco

nclu

sion

look

sick

ness

mea

lsliv

ing

drin

km

illen

ium

easy

para

grap

hsa

ysfa

rms

live

pape

ram

azin

gcu

res

read

mak

ing

that

sfu

ture

feed

fact

atte

ndan

ceea

sily

evid

ence

happ

ened

Tabl

e10

:Top

icw

ords

ofTC

lda.

Topi

c1

Topi

c2

Topi

c3

Topi

c4

Topi

c5

Topi

c6

Topi

c7

Topi

c8

Topi

c9

Topi

c10

Topi

c11

Topi

c12

Topi

c13

Topi

c14

Topi

c15

Topi

c16

Topi

c17

Topi

c18

Topi

c19

irri

gatio

nro

addi

seas

esad

ults

fight

deve

lopm

ent

joyf

ulpe

ople

mid

day

villa

gem

illen

nium

back

spl

enty

doct

orth

ing

pape

ren

dw

ork

slee

ping

fert

ilize

rbr

ight

erm

edic

ine

lifet

ime

villa

ges

dirt

kids

scho

olw

omen

acce

ssho

spita

lsu

pplie

sw

orld

bed

farm

ers

futu

rem

alar

iapr

ojec

tju

mp

fees

grou

ndca

resh

ape

chor

esne

tcr

ops

hann

ahdi

seas

ego

als

bar

stud

ents

bana

nas

med

icin

espa

tient

sbo

oks

nets

plan

tca

rm

osqu

itoes

plan

mus

icm

eal

clot

hsc

hool

str

eatm

ent

penc

ilssi

tese

eds

saur

ich

arge

econ

omy

sing

ing

ener

gym

othe

rsto

day

offic

erou

tcom

em

arke

tqu

ality

ever

yone

lunc

hfe

etsu

pply

wat

erla

ckye

arsu

ppor

ters

danc

ing

clot

hing

area

sel

ectr

icity

tool

stim

ehe

lpda

yki

ndge

nera

tor

plac

ehe

alth

room

sye

ars

advi

cefa

mily

pove

rty

item

slif

eta

rget

sco

mm

uniti

esde

ath

lead

ers

nigh

tgl

imps

eco

sts

afri

cadi

ech

emic

als

know

ledg

eso

lutio

nsfo

odm

illio

nspa

rent

s

Tabl

e11

:Top

icw

ords

ofTC

pr.

Topi

c1

Topi

c2

Topi

c3

Topi

c4

Topi

c5

Topi

c6

Topi

c7

Topi

c8

Topi

c9

Topi

c10

Topi

c11

Topi

c12

Topi

c13

Topi

c14

Topi

c15

Topi

c16

pove

rty

way

year

slu

nch

goal

sel

ectr

icity

supp

lies

affo

rdm

any

free

scho

olho

spita

lbe

dpr

ojec

tsu

pply

fert

ilize

rfig

htw

ould

four

serv

espr

oble

ms

wat

erfo

odlif

etim

epe

ople

med

icin

esc

hool

s20

04ne

tsw

orld

mai

ntai

nse

eds

win

ning

rate

villa

ges

pare

nts

day

gene

rato

rne

tco

uld

keny

acr

ops

fees

dise

ase

used

mill

enni

umdi

seas

esad

dres

sed

atte

ndan

ce80

atte

ndcl

oth

also

room

sac

hiev

able

saur

ich

arge

stud

ents

yala

ever

yvi

llage

hung

erir

riga

tion

help

prog

ress

pass

edth

ree

runn

ing

pack

edto

geth

erpe

ncils

farm

ers

slee

ping

acro

ssliv

esne

cess

ary

kids

last

mad

een

ergy

patie

nts

mal

aria

afri

cam

edic

ines

site

wor

kad

ults

tool

sen

ough

occu

rred

book

sco

nnec

ted

need

edta

keye

tm

idda

yen

dlif

ela

ckbe

tter

year

2015

5fu

ture

sach

sm

eal

wor

rydy

ing

plen

tygo

chan

ges

know

ledg

eke

epw

orke

dth

ough

dram

atic

supp

orte

rsde

ath

plan

tge

tou

tcom

ele

arn

poor

care

feed

chan

getim

eaw

ayco

mm

onpl

ace

toda

yon

efiv

efa

mily

two

clin

ical

2025

trea

ted

beco

me

solu

tions

first

like

hard

heal

thof

ficer

hist

ory

real

lyal

ong

com

ego

odse

tta

ttere

dse

lling

targ

ets

little

doct

orcr

isis

clot

hing

see

trea

tmen

tei

ther

area

sch

emic

als

die

min

imal

who

leite

ms

mal

aria

lhu

ngry

alm

ost

save

prev

enta

ble

danc

ing

harv

est

mill

ions

trea

tabl

ew

alke

dsh

owed

easy

cost

sba

rech

eap

met

feet

ever

hann

ahar

ound

impo

veri

shed

mos

quito

esen

cour

agin

gea

sily

prob

ably

Tabl

e12

:Top

icw

ords

ofTC

attn

.

Cat

egor

y1

Cat

egor

y2

Cat

egor

y3

Cat

egor

y4

unpa

ved

road

sun

ited

natio

nsin

terv

entio

nya

lasu

bdi

stri

ctho

spita

lm

alar

iaco

mm

ondi

seas

epr

even

tabl

etr

eata

ble

tatte

red

clot

hing

safe

rhea

lthie

rbet

terl

ife

thre

eki

dsbe

dtw

oad

ults

room

spa

cked

patie

nts

mos

quito

esca

rry

mal

aria

infe

ctpe

ople

bitin

gba

refe

etou

tpov

erty

stab

ilize

econ

omy

qual

itylif

eco

mm

uniti

esno

tmed

icin

etr

eatm

entc

ould

affo

rdki

dsdi

em

alar

iaad

ults

sick

2000

0da

yle

ssth

an1

dolla

rday

afri

cake

nya

saur

ino

doct

oron

lycl

inic

alof

ficer

runn

ing

hosp

ital

bed

nets

mos

quito

esaw

aype

ople

save

mill

ions

lives

goal

sm

et20

1520

25no

runn

ing

wat

erel

ectr

icity

bed

nets

cost

5do

llar

80vi

llage

sac

ross

sub-

saha

raaf

rica

sad

peop

ledy

ing

near

deat

hpr

even

tabl

ech

eap

med

icin

estr

eatm

alar

iaC

ateg

ory

5C

ateg

ory

6C

ateg

ory

7C

ateg

ory

8cr

ops

dyin

gki

dsno

tatte

ndgo

scho

olpr

ogre

ssju

stfo

urye

ars

prog

ress

enco

urag

ing

supp

orte

rsno

taff

ord

fert

ilize

rirr

igat

ion

nota

ffor

dsc

hool

fees

yala

sub

dist

rict

hosp

italh

asm

edic

ine

solu

tions

prob

lem

ske

eppe

ople

impo

veri

shed

outc

ome

poor

crop

ski

dshe

lpch

ores

fetc

hing

wat

erw

ood

med

icin

efr

eech

arge

chan

gepo

vert

yst

rick

enar

eas

good

lack

fert

ilize

rwat

ersc

hool

sm

inim

alsu

pplie

sbo

oks

pape

rpen

cils

med

icin

em

ostc

omm

ondi

seas

espo

vert

yhi

stor

yno

teas

yta

skha

rden

ough

food

crop

sha

rves

tfee

dw

hole

fam

ilyhu

ngry

sick

conc

entr

ate

note

nerg

yw

ater

conn

ecte

dho

spita

lw

inni

ngag

ains

tpov

erty

poss

ible

achi

evab

lelif

etim

eno

mid

day

mea

llun

chho

spita

lgen

erat

orel

ectr

icity

bed

nets

used

ever

ysl

eepi

ngsi

tehu

nger

cris

isad

dres

sed

fert

ilize

rsee

dsto

ols

need

edm

aint

ain

food

supp

lyki

dsgo

scho

olno

wno

scho

olfe

esno

wse

rves

lunc

hst

uden

tssc

hool

atte

ndan

cera

tew

ayup

Tabl

e13

:Spe

cific

exam

ple

phra

ses

ofTC

manual.

Cat

egor

y1

Cat

egor

y2

Cat

egor

y3

Cat

egor

y4

Cat

egor

y5

wor

kha

rdlif

etim

eau

thor

conv

ince

win

ning

fight

pove

rty

achi

evab

lelif

etim

ech

ildre

nad

ults

easy

task

bette

rpla

ceun

ited

natio

nsau

thor

conv

ince

dw

inni

ngfig

htpo

vert

yac

hiev

able

lifet

ime

mos

quito

esca

rry

mal

aria

lived

dolla

rbe

tterh

ealth

unite

dst

ates

auth

orw

ants

dise

ase

calle

dm

alar

iath

ing

hist

ory

brig

hter

futu

relif

eco

mm

uniti

esau

thor

conv

ince

win

ning

fight

prov

erty

com

eni

ght

stuf

fnee

dth

ings

like

like

book

spa

perp

enci

lsw

inni

ngfig

htpr

over

tyac

hiev

able

lifet

ime

mal

aria

lmos

quito

esea

rnm

oney

thin

gsne

edle

arn

life

keny

aw

inni

ngfig

htpo

vert

yac

hiev

able

life

time

easi

lyad

ults

sick

fight

ing

pove

rty

impo

rtan

tkid

sar

ticle

brig

hter

futu

reso

lutio

nspr

oble

ms

peop

leim

pove

rish

edw

ork

chan

geth

inks

impo

rtan

tw

inin

gfig

htpo

vert

yac

hiev

able

mos

quito

esaw

ayha

rdw

ork

wan

tskn

owar

ticle

stat

esin

fect

peop

lebi

ting

agre

eau

thor

win

ning

fight

pove

rty

ache

ivab

leaw

aysl

eepi

ngw

orki

ngha

rdau

thor

prov

ided

bette

rlif

e20

08au

thor

thin

ksbe

tterl

ife2

008

base

dar

ticle

auth

orco

nvin

cere

adin

gar

ticle

conv

ince

dpo

vert

yth

ings

chan

ged

pove

rty

ache

ivab

lelif

etim

eC

ateg

ory

6C

ateg

ory

7C

ateg

ory

8C

ateg

ory

9C

ateg

ory

10at

tend

ance

rate

amaz

ing

prog

ress

year

sgo

odsh

ape

kids

adul

tsdo

nate

mon

eym

idda

ym

eal

text

says

good

educ

atio

n20

1520

25ta

ttere

dcl

othe

sse

rves

lunc

hst

uden

tste

xtsa

idw

ents

choo

lhu

ngry

sick

tatte

red

clot

hing

mid

day

mea

lsye

argi

rlar

eas

good

chea

pm

edic

ines

bare

feet

serv

edlu

nch

year

2004

tryi

nghe

lpgo

als

supp

osed

dona

ting

mon

eyst

uden

tsw

ante

dle

arn

para

grap

hsa

ysw

orke

dha

rdsa

vem

illio

nsliv

esbo

oks

penc

ilspr

ogre

sssh

ows

win

ning

fight

pove

rty

achi

evab

lese

cond

reas

onki

dsat

tend

scho

oltr

eate

dch

emic

als

seco

ndex

ampl

esc

hool

sm

inim

alpa

ragr

aph

stat

esgi

rlw

ent

scho

ols

hosp

itals

prog

ress

enco

urag

ing

supp

orte

rsm

illen

nium

villa

ges

hann

ahsa

chs

conv

ince

dw

inni

ngsc

hool

scho

olfe

esw

entk

enya

prac

tical

item

ski

dssa

uria

ttend

scho

olpa

rent

saf

ford

scho

olfe

esat

tend

ence

rate

pare

nts

mon

eyC

ateg

ory

11C

ateg

ory

12C

ateg

ory

13C

ateg

ory

14C

ateg

ory

15cl

ean

wat

ergr

owcr

ops

mill

enni

umvi

llage

proj

ect

stop

pove

rty

runn

ing

wat

erel

ectr

icity

wat

erw

ood

feed

fam

ilym

illen

ium

villa

gepr

ojec

tlo

ngtim

ew

ater

conn

ecte

dho

spita

lgen

erat

orel

ectr

icity

fres

hw

ater

need

edhe

lpm

illen

nium

villa

ges

proj

ecth

elpe

dw

orld

wor

kch

ange

patie

nts

affo

rdne

eds

help

farm

ers

wor

rych

ange

dram

atic

ally

beat

pove

rty

room

spa

cked

patie

nts

prob

ably

med

icin

esfr

eech

arge

crop

sdy

ing

affo

rdne

cess

ary

fert

ilize

rirr

igat

ion

dram

atic

chan

ges

occu

red

villa

ges

subs

ahar

anaf

rica

endi

ngpo

vert

ysh

are

beds

chor

esfe

tchi

ngfe

rtili

zerk

now

ledg

epl

ace

live

wan

tlea

rnre

ciev

etr

eatm

ent

fetc

hing

wat

erhu

nger

cris

isad

dres

sed

fert

ilize

rsee

dsto

ols

need

edm

aint

ain

food

supp

lyha

ppen

edye

ars

plac

eslik

edo

ctor

clin

ical

offic

erru

nnin

gho

spita

lfe

edfa

mili

esdr

amat

icch

ange

soc

curr

edvi

llage

ssh

ows

win

ning

fight

pove

rty

achi

evab

lelif

etim

edo

ctor

scl

inic

alhu

nger

cris

isad

ress

edm

illen

nium

deve

lopm

entg

oals

wan

tkin

dpo

vert

yw

ater

fert

ilize

rkno

wle

dge

fam

ilypl

ants

eeds

outc

ome

poor

chan

gepo

vert

ystr

icke

nar

eas

good

pove

rty

assu

reac

cess

rece

ive

trea

tmen

tfa

rmer

sw

orri

edco

min

gye

ars

runn

ing

bare

enco

urag

ing

supp

orte

rsm

illen

nium

villa

ges

proj

ect

affo

rdtr

eatm

ent

occu

rred

villa

ges

subs

ahar

anC

ateg

ory

16C

ateg

ory

17C

ateg

ory

18C

ateg

ory

19C

ateg

ory

20ya

lasu

bdis

tric

thos

pita

lmed

icin

efr

eech

arge

com

mon

dise

ases

nets

slee

ping

site

saur

ipl

anpe

ople

pove

rty

achi

eve

goal

year

sla

ter

free

lunc

haf

ford

nets

stab

ilize

econ

omy

qual

itylif

eco

mm

uniti

esre

ach

goal

took

year

sya

ladi

stri

ctas

sure

acce

sshe

alth

care

help

goin

gsc

hool

star

ted

2004

prev

enta

ble

trea

tabl

epe

ople

peop

lest

ory

says

com

mon

afri

cane

arde

ath

achi

eve

goal

sdi

seas

eslik

epo

orcr

ops

lack

com

mon

dise

ase

afri

caho

mel

ess

peop

leho

spita

lgoo

dsh

ape

dist

rict

hosp

ital

Tabl

e14

:Spe

cific

exam

ple

phra

ses

ofTC

lda.

Category 1brighter future hannah

millennium villages projectunpaved dirt road

bar sauri primary schoolfuture hannah

sauri primary schoolvillages project

millennium development goalsvillage leaders

dirt roadcar jumplittle kids

preventable diseases peoplemany kids

diseases peoplekids die

school suppliesprimary school

school feesinfect people

Table 15: Specific example phrases of TCpr.

Cat

egor

y1

Cat

egor

y2

Cat

egor

y3

Cat

egor

y4

Cat

egor

y5

Cat

egor

y6

win

ning

fight

coul

dfe

edbe

dne

taff

ord

four

year

spr

ogre

sslif

etim

eye

arfe

esst

uden

tssc

hool

supp

lies

scho

ols

saur

ikno

wle

dge

supp

lies

med

icin

espo

vert

yw

inni

ngw

orld

villa

ges

peop

lesc

hool

wor

kha

rdbo

oks

villa

ges

occu

rred

80ac

ross

alon

gsc

hool

fees

supp

lies

affo

rdfe

rtili

zer

affo

rdsc

hool

fees

bette

rmed

icin

ew

ater

ener

gyw

inni

ngfig

htpo

vert

yal

soev

ery

dise

ases

kids

heal

thne

t5to

ols

crop

ssc

hool

fees

seed

sbe

dne

tshe

lpke

epho

spita

lele

ctri

city

conn

ecte

dw

inni

ngpo

vert

ypr

even

tabl

efa

mily

peop

leca

reye

ars

man

yvi

llage

ssa

urip

roje

ctfa

rmer

sro

oms

patie

nts

crop

spe

ople

food

atte

ndan

cero

oms

end

man

ybe

dne

ts5

also

fight

pove

rty

affo

rdsc

hool

fees

bed

nets

outc

ome

poor

crop

ssc

hool

lunc

hm

ealm

idda

ysu

pplie

spr

oble

ms

also

peop

leen

ergy

man

yw

ater

elec

tric

ityho

spita

lfer

tiliz

erpo

vert

yfig

htw

inni

ngal

sow

ould

ener

gyle

arn

help

prog

ress

year

ske

nya

afri

cato

day

lunc

hst

uden

tsse

rves

mid

day

food

supp

lym

aint

ain

elec

tric

itysu

pplie

sel

ectr

icity

wat

eren

ergy

fight

pove

rty

win

ning

peop

lefe

essc

hool

farm

ers

coul

dra

tepe

ople

med

icin

e20

045

year

ske

epsc

hool

fees

bed

show

edlu

nch

coul

dw

ork

elec

tric

itym

edic

ine

villa

ges

keny

a80

farm

ers

man

ysc

hool

lunc

hsc

hool

sal

sofe

es20

04al

soye

arra

tesc

hool

bed

nets

used

coul

daf

ford

fert

ilize

rfo

urye

ars

lifet

ime

pove

rty

year

year

ssc

hool

show

edho

spita

lwat

erfa

rmer

sne

eded

food

supp

lyvi

llage

sge

nera

tore

nerg

ysc

hool

supp

lies

little

affo

rden

ough

year

sfo

urla

stfiv

eda

ysc

hool

pare

nts

atte

ndbe

dne

tsfr

eefo

odal

sofa

rmer

stw

om

any

pove

rty

scho

olm

edic

ine

fert

ilize

rhos

pita

lbed

wat

erel

ectr

icity

also

fert

ilize

rsup

plie

sal

soto

ols

year

sch

ange

sfe

rtili

zera

ddre

ssed

scho

olsc

hool

sfe

esfr

eetw

oel

ectr

icity

wat

erru

nnin

gal

soge

nera

tor

supp

lym

aint

ain

food

also

tatte

red

year

svi

llage

ske

nya

proj

ecta

ttend

ance

scho

olfe

essc

hool

slu

nch

free

gene

rato

rele

ctri

city

ener

gypo

vert

yhu

nger

elec

tric

itylu

nch

scho

olcr

ops

food

farm

ers

fert

ilize

rbed

netw

ater

wat

erfe

rtili

zere

nerg

ysc

hool

med

icin

esfe

rtili

zera

ddre

ssed

scho

olsu

pplie

scr

isis

Cat

egor

y7

Cat

egor

y8

Cat

egor

y9

Cat

egor

y10

Cat

egor

y11

Cat

egor

y12

elec

tric

ityru

nnin

gw

ater

irri

gatio

nse

tbe

dsh

owed

dise

ases

help

stud

ents

supp

lies

peop

lesc

hool

sye

ars

four

free

scho

ols

med

icin

em

edic

ine

elec

tric

ityto

ols

fert

ilize

rmed

icin

essc

hool

sal

sosc

hool

stud

ents

atte

ndan

cepo

orsh

owed

trea

tmen

tsch

ools

uppl

ies

lunc

hm

eale

nerg

ype

ople

year

sfo

urth

ree

thou

ghsc

hool

scho

ols

free

supp

lies

fees

wat

erel

ectr

icity

conn

ecte

dsc

hool

sru

nnin

gfr

eech

arge

scho

olm

aint

ain

supp

lyfa

rmer

sco

uld

crop

saf

ford

bed

dram

atic

chan

gebe

dne

tsvi

llage

sye

ars

80po

vert

ym

any

crop

sfe

rtili

zerf

arm

ers

tool

spl

ant

stud

ents

lunc

hse

rves

scho

ol20

04cr

ops

farm

ers

2004

first

food

elec

tric

ityho

spita

lpo

vert

ybe

tterl

ives

mad

em

any

wor

ked

toge

ther

end

wat

erel

ectr

icity

supp

lies

scho

olen

ergy

med

icin

ecr

ops

free

hosp

itala

lso

lack

fert

ilize

rsch

oolb

edne

tsbe

tterf

ertil

izer

med

icin

een

ough

also

achi

evab

lelif

etim

esa

uri

penc

ilsst

uden

tssu

pplie

sye

tm

edic

ine

scho

olsu

pplie

sye

ars

hung

ersc

hool

supp

lies

farm

ers

atte

ndan

cecr

ops

bed

nets

year

sho

spita

lro

oms

pack

edpa

tient

sm

alar

iago

odbe

dne

tuse

dvi

llage

sm

any

keny

asa

uri8

0fe

rtili

zerc

rops

lack

farm

ers

wat

erw

ater

supp

lies

scho

ols

free

hosp

ital

hosp

itald

isea

sefo

urye

ars

2004

food

fert

ilize

rcro

psge

tsup

ply

bed

net

year

sfo

odsu

pply

hung

ercr

isis

fert

ilize

rirr

igat

ion

crop

sm

edic

ine

wat

ersc

hool

scr

ops

supp

lies

free

char

geev

ery

slee

ping

site

five

netc

osts

5co

mm

ondi

seas

essa

urin

etm

edic

ines

scho

olm

edic

ine

fert

ilize

rfre

esc

hool

scho

ols

lunc

hal

sofr

eesc

hool

bed

also

occu

rred

80ne

tsne

tbed

free

wor

kto

geth

erpo

vert

yne

t5fr

eech

arge

med

icin

esc

hool

med

icin

essc

hool

fees

scho

ols

year

sfo

ursc

hool

sla

stst

uden

tsru

nnin

gw

ater

supp

lies

scho

ols

alm

ost

hosp

italg

osc

hool

coul

daf

ford

scho

olsu

pplie

site

ms

seed

spl

antc

rops

fert

ilize

rsc

hool

fees

lunc

hsc

hool

supp

lies

scho

ols

also

2004

bed

supp

lies

know

ledg

em

edic

ines

affo

rdpr

ojec

tpro

gres

sm

ade

food

good

sach

sm

any

free

scho

ols

lunc

hsc

hool

char

gelu

nch

scho

ols

scho

olse

eds

food

crop

sfa

rmer

ssc

hool

spr

ojec

tals

osu

pplie

sfo

odsu

pply

farm

ers

wat

eral

soho

spita

ldoc

torc

linic

alsh

owed

bed

nets

wat

erfe

rtili

zerm

edic

ines

scho

olfe

essc

hool

sfr

eelu

nch

hosp

italy

ears

med

icin

esc

hool

wat

ersu

pplie

sm

idda

ysc

hool

food

hung

erye

ars

mad

em

alar

iata

kech

ange

sfr

eech

arge

med

icin

esc

hool

fert

ilize

rsc

hool

ssu

pplie

sel

ectr

icity

farm

ers

fert

ilize

rfr

eech

arge

scho

ols

year

sm

eal

man

yfo

odco

uld

bette

rfut

ure

peop

lelu

nch

crop

sfa

rmer

sfe

rtili

zere

lect

rici

tykn

owle

dge

stud

ents

lunc

hm

edic

ine

hosp

italm

ade

scho

olfe

essc

hool

sfr

eem

edic

ines

scho

ols

scho

olfa

rmer

scr

ops

bed

free

char

gesc

hool

year

shu

nger

Cat

egor

y13

Cat

egor

y14

Cat

egor

y15

Cat

egor

y16

Cat

egor

y17

Cat

egor

y18

bed

nets

villa

ges

afri

cam

illen

nium

80ac

ross

supp

lybo

oks

seed

sfe

rtili

zera

ddre

ssed

food

med

icin

een

ough

wou

ldw

ork

hard

bette

rw

ater

conn

ecte

dho

spita

lw

ater

runn

ing

med

icin

em

edic

ines

supp

lies

80vi

llage

sac

ross

elec

tric

ityw

ater

seed

ssu

pply

fert

ilize

rcro

pspl

enty

peop

lew

orld

saur

ikid

spo

vert

yne

tsbe

dus

edcr

ops

affo

rdbe

dne

tsm

edic

ine

crop

sel

ectr

icity

pove

rty

fight

peop

leke

nya

end

pove

rty

man

yliv

eshu

nger

ever

yfe

rtili

zers

eeds

crop

sm

any

peop

lepo

vert

yco

uld

take

mid

day

mea

lsa

urif

ree

bed

nets

wor

ld20

15di

seas

esla

ckw

ater

day

ever

yto

ols

fert

ilize

rke

nya

wou

ldbe

tterw

alke

dba

rem

idda

ym

eall

unch

crop

sfe

rtili

zerp

lant

food

irri

gatio

npo

orvi

llage

saur

iad

ults

one

bed

two

last

crop

sfa

rmer

sal

sow

ater

coul

dpo

vert

ypr

oble

ms

cris

isth

ough

man

ybe

dne

tsus

edbe

dne

tsev

ery

wat

erm

edic

ine

wel

lpro

ject

villa

ges

poor

end

peop

lefo

odw

ork

man

yen

ergy

crop

sse

eds

wat

erne

eded

peop

leke

nya

targ

ets

80vi

llage

sbe

dev

ery

slee

ping

site

net

fert

ilize

rcro

psw

ater

keep

tool

sac

hiev

able

keny

avi

llage

svi

llage

scho

olpe

ople

man

yad

dres

sed

fert

ilize

rsee

dsal

mos

tkid

sdi

epe

ople

bed

nets

ever

yus

edsc

hool

keny

abe

dne

tsm

any

villa

ges

peop

lepr

oble

ms

keny

asc

hool

food

scho

ols

hosp

italp

eopl

ese

eds

fert

ilize

rfoo

dal

sow

ater

rate

way

prog

ress

bette

rafr

ica

hosp

italw

ater

runn

ing

clin

ical

offic

erbe

dne

tsal

soad

ults

proj

ectv

illag

eske

nya

villa

gepe

ople

year

sch

ange

sfo

urfr

eeoc

curr

edse

eds

fert

ilize

rwat

erat

tend

ance

rate

way

wat

erho

spita

lbed

nets

saur

ibed

nets

goal

sfo

urye

ars

met

need

edw

ater

ever

yw

ork

scho

olfe

esfe

rtili

zerf

ood

see

wor

ldbe

dne

tsco

uld

keep

ever

ybe

dne

tspo

vert

yvi

llage

fight

afri

casa

uri

year

sho

spita

lvill

ages

char

geco

nnec

ted

fert

ilize

rirr

igat

ion

nece

ssar

yfa

rmer

sto

ols

gohu

ngry

getp

eopl

eco

uld

bed

nets

used

ever

ysl

eepi

ngdi

seas

esm

edic

ine

med

icin

esco

mm

onpr

even

tabl

eat

tend

ance

rate

way

selli

ngco

me

food

mai

ntai

nsu

pply

elec

tric

itysu

pplie

sfe

rtili

zers

eeds

irri

gatio

nfa

rmer

sla

ckge

tfoo

dw

ork

wou

ldpr

obab

lyho

spita

lcha

rge

bed

nets

prev

enta

ble

nets

bed

wat

ersa

uriy

ears

wor

kw

orld

help

last

toge

ther

2015

2025

dyin

ghu

nger

deat

hfe

rtili

zerl

ack

crop

sbe

com

esa

uri

wor

ldw

inni

ngfig

htw

aypl

ace

crop

sfe

rtili

zere

noug

hfa

rmer

spo

vert

ym

any

2015

mill

enni

umpr

ogre

ssdi

seas

esm

alar

iape

ople

easi

lysa

urih

isto

ryw

aysi

tesa

uri

help

peop

lepo

vert

ypl

ace

man

y

Tabl

e16

:Spe

cific

exam

ple

phra

ses

ofTC

attn

.