Language Technology Enhanced Learning

  • View
    1.314

  • Download
    0

Embed Size (px)

DESCRIPTION

Fridolin Wild, Gaston Burek, Adriana Berlanga

Transcript

  • 1.Language Technology Enhanced Learning Fridolin Wild The Open University, UK Gaston Burek University of Tbingen Adriana Berlanga Open University, NL

2. Workshop Outline

  • 1 | Deep IntroductionLatent-Semantic Analysis (LSA)
  • 2 | Quick Introduction Working with R
  • 3 | Experiment Simple Content-Based Feedback
  • 4 | Experiment Topic Proxy

# 3. Latent-Semantic Analysis LSA 4. Latent Semantic Analysis

  • Assumption: language utterancesdohavea semantic structure
  • However, this structure is obscured by word usage (noise, synonymy, polysemy, )
  • Proposed LSA Solution:
    • map doc-term matrix
    • using conceptual indices
    • derived statistically (truncatedSVD )
    • and make similarity comparisonsusing e.g.angles

5. Input (e.g., documents) { M } =Deerwester, Dumais, Furnas, Landauer, and Harshman (1990):Indexing by Latent Semantic Analysis, In: Journal of the AmericanSociety for Information Science, 41(6):391-407 Only the red terms appear in morethan one document, so strip the rest. term = feature vocabulary = ordered set of features TEXTMATRIX 6. Singular Value Decomposition = 7. Truncated SVD we will get a different matrix (different values,but still of the same format as M). latent-semantic space 8. Reconstructed, Reduced Matrix m4:Graph minors : Asurvey 9. Similarity in a Latent-Semantic Space (Landauer, 2007) Query Target 1 Target 2 Angle 2 Angle 1 Y dimension X dimension 10. doc2doc - similarities

    • Unreduced = pure vector space model
    • - Based onM = TSD
    • - Pearson Correlation over document vectors
    • reduced
    • - based onM 2= TS 2 D
    • - Pearson Correlationover document vectors

11. (Landauer, 2007) 12. Configurations 4 x 12 x 7 x 2 x 3=2016 Combinations 13. Updating: Folding-In

  • SVD factor stability
    • Different texts different factors
    • Challenge: avoid unwanted factor changes(e.g., bad essays)
    • Solution: folding-in instead of recalculating
  • SVD is computationally expensive
    • 14 seconds (300 docs textbase)
    • 10 minutes (3500 docs textbase)
    • and rising!

14. The Statistical Languageand Environment R R 15. 16. Help

  • > ?'+'
  • > ?kmeans
  • > help.search("correlation")
  • http://www.r-project.org
  • => site search
  • => documentation
  • Mailinglist r-help
  • Task View NLP: http://cran.r-project.org/-> Task Views -> NLP

17. Installation & Configuration

  • install.packages("lsa", repos="http://cran.r-project.org")
  • install.packages("tm", repos="http://cran.r-project.org")
  • install.packages("network", repos="http://cran.r-project.org")
  • library(lsa)
  • setwd("d:/denkhalde/workshop")
  • dir()
  • ls()
  • quit()

18. The lsa Package

  • Available via CRAN, e.g.: http://cran.at.r-project.org/src/contrib/Descriptions/lsa.html
  • Higher-level Abstraction to Ease Use
    • Five core methods:
    • textmatrix()/query()
    • lsa()
    • fold_in()
    • as.textmatrix()
    • Supporting methods for term weighting, dimensionality calculation, correlation measurement, triple binding

19. Core Processing Workflow

  • tm = textmatrix(dir/)
  • tm = lw_logtf(tm) * gw_idf(tm)
  • space = lsa(tm, dims=dimcalc_share())
  • tm3 = fold_in(tm, space)
  • as.textmatrix(tm)

20. A Simple Evaluation of Students Writings Feedback 21. Evaluating Student Writings External Validation? Compare to Human Judgements! (Landauer, 2007) 22. How to do it...

  • library( "lsa ) # load package
  • # load training texts
  • trm = textmatrix( "trainingtexts/ )
  • trm = lw_bintf( trm ) * gw_idf( trm ) # weighting
  • space = lsa( trm ) # create an LSA space
  • # fold-in essays to be tested (including gold standard text)
  • tem = textmatrix( "testessays/", vocabulary=rownames(trm) )
  • tem = lw_bintf( tem ) * gw_idf( trm ) # weighting
  • tem_red = fold_in( tem, space )
  • # score an essay by comparing with
  • # gold standard text (very simple method!)
  • cor( tem_red[,"goldstandard.txt"], tem_red[,"E1.txt"] )
  • => 0.7

23. Evaluating Effectiveness

  • Compare Machine Scoreswith Human Scores
  • Human-to-Human Correlation
    • Usually around .6
    • Increased by familiarity between assessors, tighter assessment schemes,
    • Scores vary even stronger with decreasing subject familiarity (.8 at high familiarity, worst test -.07)
  • Test Collection: 43 German Essays, scored from 0 to 5 points (ratio scaled), average length: 56.4 words
  • Training Collection: 3 golden essays, plus 302 documents from a marketing glossary, average length: 56.1 words

24. (Positive) Evaluation Results

  • LSA machine scores:
  • Spearman's rank correlation rho
  • data:humanscores[names(machinescores), ] and machinescores
  • S = 914.5772, p-value =0.0001049
  • alternative hypothesis: true rho is not equal to 0
  • sample estimates:
  • rho
  • 0.687324
  • Pure vector space model:
  • Spearman's rank correlation rho
  • data:humanscores[names(machinescores), ] and machinescores
  • S = 1616.007, p-value =0.02188
  • alternative hypothesis: true rho is not equal to 0
  • sample estimates:
  • rho
  • 0.4475188

25. Concept-Focused Evaluation (using http://eczemablog.blogspot.com/feeds/posts/default?alt=rss) 26. Visualising Lexical Semantics Topic Proxy 27. Network Visualisation

  • Term-2-Termdistance matrix

= = Graph t 1 t 2 t 3 t 4 t 1 1 t 2 -0.2 1 t 3 0.5 0.7 1 t 4 0.05 -0.5 0.3 1 28. Classical Landauer Example tl = landauerSpace$tk %*% diag(landauerSpace$sk) dl = landauerSpace$dk %*% diag(landauerSpace$sk) dtl = rbind(tl,dl) s = cosine(t(dtl)) s[which(s