prev

next

of 39

Published on

10-May-2015View

1.313Download

0

DESCRIPTION

Fridolin Wild, Gaston Burek, Adriana Berlanga

Transcript

1.Language Technology Enhanced Learning Fridolin Wild The Open University, UK Gaston Burek University of Tbingen Adriana Berlanga Open University, NL2. Workshop Outline 1 | Deep IntroductionLatent-Semantic Analysis (LSA) 2 | Quick Introduction Working with R 3 | Experiment Simple Content-Based Feedback 4 | Experiment Topic Proxy # 3. Latent-Semantic Analysis LSA 4. Latent Semantic Analysis Assumption: language utterancesdohavea semantic structure However, this structure is obscured by word usage (noise, synonymy, polysemy, ) Proposed LSA Solution:map doc-term matrixusing conceptual indicesderived statistically (truncatedSVD )and make similarity comparisonsusing e.g.angles 5. Input (e.g., documents) { M } =Deerwester, Dumais, Furnas, Landauer, and Harshman (1990):Indexing by Latent Semantic Analysis, In: Journal of the AmericanSociety for Information Science, 41(6):391-407 Only the red terms appear in morethan one document, so strip the rest. term = feature vocabulary = ordered set of features TEXTMATRIX 6. Singular Value Decomposition = 7. Truncated SVD we will get a different matrix (different values,but still of the same format as M). latent-semantic space 8. Reconstructed, Reduced Matrix m4:Graph minors : Asurvey 9. Similarity in a Latent-Semantic Space (Landauer, 2007) Query Target 1 Target 2 Angle 2 Angle 1 Y dimension X dimension 10. doc2doc - similarities Unreduced = pure vector space model - Based onM = TSD - Pearson Correlation over document vectors reduced - based onM 2= TS 2 D - Pearson Correlationover document vectors 11. (Landauer, 2007) 12. Configurations 4 x 12 x 7 x 2 x 3=2016 Combinations 13. Updating: Folding-In SVD factor stability Different texts different factors Challenge: avoid unwanted factor changes(e.g., bad essays) Solution: folding-in instead of recalculating SVD is computationally expensive 14 seconds (300 docs textbase)10 minutes (3500 docs textbase) and rising! 14. The Statistical Languageand Environment R R 15. 16. Help > ?'+' > ?kmeans > help.search("correlation") http://www.r-project.org=> site search => documentation Mailinglist r-help Task View NLP: http://cran.r-project.org/-> Task Views -> NLP 17. Installation & Configuration install.packages("lsa", repos="http://cran.r-project.org") install.packages("tm", repos="http://cran.r-project.org") install.packages("network", repos="http://cran.r-project.org") library(lsa) setwd("d:/denkhalde/workshop") dir() ls() quit() 18. The lsa Package Available via CRAN, e.g.: http://cran.at.r-project.org/src/contrib/Descriptions/lsa.html Higher-level Abstraction to Ease Use Five core methods: textmatrix()/query() lsa() fold_in() as.textmatrix() Supporting methods for term weighting, dimensionality calculation, correlation measurement, triple binding 19. Core Processing Workflow tm = textmatrix(dir/) tm = lw_logtf(tm) * gw_idf(tm) space = lsa(tm, dims=dimcalc_share()) tm3 = fold_in(tm, space) as.textmatrix(tm) 20. A Simple Evaluation of Students Writings Feedback 21. Evaluating Student Writings External Validation? Compare to Human Judgements! (Landauer, 2007) 22. How to do it... library( "lsa ) # load package # load training texts trm = textmatrix( "trainingtexts/ ) trm = lw_bintf( trm ) * gw_idf( trm ) # weighting space = lsa( trm ) # create an LSA space # fold-in essays to be tested (including gold standard text) tem = textmatrix( "testessays/", vocabulary=rownames(trm) ) tem = lw_bintf( tem ) * gw_idf( trm ) # weighting tem_red = fold_in( tem, space ) # score an essay by comparing with# gold standard text (very simple method!) cor( tem_red[,"goldstandard.txt"], tem_red[,"E1.txt"] ) => 0.7 23. Evaluating Effectiveness Compare Machine Scoreswith Human Scores Human-to-Human Correlation Usually around .6 Increased by familiarity between assessors, tighter assessment schemes, Scores vary even stronger with decreasing subject familiarity (.8 at high familiarity, worst test -.07) Test Collection: 43 German Essays, scored from 0 to 5 points (ratio scaled), average length: 56.4 words Training Collection: 3 golden essays, plus 302 documents from a marketing glossary, average length: 56.1 words 24. (Positive) Evaluation Results LSA machine scores: Spearman's rank correlation rho data:humanscores[names(machinescores), ] and machinescoresS = 914.5772, p-value =0.0001049 alternative hypothesis: true rho is not equal to 0sample estimates: rho0.687324Pure vector space model: Spearman's rank correlation rho data:humanscores[names(machinescores), ] and machinescoresS = 1616.007, p-value =0.02188 alternative hypothesis: true rho is not equal to 0sample estimates: rho0.4475188 25. Concept-Focused Evaluation (using http://eczemablog.blogspot.com/feeds/posts/default?alt=rss) 26. Visualising Lexical Semantics Topic Proxy 27. Network Visualisation Term-2-Termdistance matrix = = Graph t 1 t 2 t 3 t 4 t 1 1 t 2 -0.2 1 t 3 0.5 0.7 1 t 4 0.05 -0.5 0.3 1 28. Classical Landauer Example tl = landauerSpace$tk %*% diag(landauerSpace$sk) dl = landauerSpace$dk %*% diag(landauerSpace$sk) dtl = rbind(tl,dl) s = cosine(t(dtl)) s[which(s