Analysing Dialogue for Diagnosis and Predic3on in Mental

Cognitive Science Research Grouphttp://cogsci.eecs.qmul.ac.uk

AnalysingDialogueforDiagnosisandPredic3oninMentalHealth

Ma8hewPurverQueenMaryUniversityofLondon

withChris3neHowes,RoseMcCabe,ClaireKelleher,NiallGunter,JulianHough


MentalHealth&Language•  Communica3onisimportantinmentalhealth:

–  Linguis3cindicatorsofcondi3ons–  Communica3onduringtreatment:

•  Communica3onqualityassociatedwithoutcomes•  Conversa3onstructure(how)andcontent(what)

–  CanNLPtechniqueshelpusanalyse&understandtherapy/condi3ons?

•  PPATproject:–  schizophrenia:face-to-faceoutpa3entconversa3on

•  AOTDproject:–  depression&anxiety:onlinetext-basedtherapy

•  SLADEproject:–  demen3a:face-to-faceclinicalconversa3on

•  (Howes,McCabe,Purver2012-2014,2018toappear)


Ques3ons•  Doeslanguagecorrelatewithand/orpredictsymptoms&

outcomes?–  Canweusethistohelpdiagnosisand/ortreatment?

•  Whataretheinforma3vefeatures?–  Topic?–  Sen3ment/emo3onalcontent?–  Conversa3onstructure?

•  Canwedetectthemautoma3cally?–  Accurately–  Robustly–  Usingexis3ngNLPtechniques/tools

•  Howcanwedobe8er?


PPAT:Face-to-FaceDialogue•  Transcriptsoftherapyforschizophrenia•  Measuresofsymptomseverity

–  posi0ve(delusions,hallucina3ons,beliefs)–  nega0ve(withdrawal,bluntedaffect,alogia)

•  Recordedrelatedoutcomes–  ra3ngsofcommunica3onquality–  futureadherencetotreatment(6monthslater):

•  non-adherence:riskofrelapse3.73meshigher–  sharedunderstandingknowntobearelatedfactor

•  Manualannota3on&sta3s3calanalysis•  McCabeetal(2013)

•  Automa3cNLPprocessing&machinelearning•  Howesetal(2012;2013)


UsingBruteForce•  Classifyen3redialogues(pa3entturnsonly)withSVMs,ngrams

–  Predictnon-adherencetotreatment6monthslater

•  Similarforsymptoms,someoutcomese.g.HAS,PEQ•  Humanpsychiatristgivensametask:

•  Buthowwellwillthisgeneralise?Andwhatdoesitmean?

Features P(%) R(%) F(%)

Classofinterest 28.9 100.0 44.8

Baselinefeatures 27.0 51.9 35.5

Bestngramfeatures 70.3 70.3 70.3

Data P(%) R(%) F(%)

Texttranscripts 60.3 79.6 68.6

Transcripts+video 69.6 88.6 78.0

Images:wikipedia,coursera.com


UsingBruteForce•  Classifyen3redialogues(pa3entturnsonly)withSVMs,ngrams

–  Predictnon-adherencetotreatment6monthslater

•  Similarforsymptoms,someoutcomese.g.HAS,PEQ•  Humanpsychiatristgivensametask:

•  Buthowwellwillthisgeneralise?Andwhatdoesitmean?

Features P(%) R(%) F(%)

Classofinterest 28.9 100.0 44.8

Baselinefeatures 27.0 51.9 35.5

Bestngramfeatures 70.3 70.3 70.3

Data P(%) R(%) F(%)

Texttranscripts 60.3 79.6 68.6

Transcripts+video 69.6 88.6 78.0

Images:wikipedia,coursera.com


Manualtopicsegmenta3on


TopicModelling•  LatentDirichletAlloca3on(Bleietal,2003)•  UnsupervisedBayesianmodel:

–  textsasmixturesof“topics”–  topicsasdistribu3onsoverwords

•  Nopriorknowledgeoftopics–  numberoftopics–  likelydistribu3onshapes–  (automa3callyop3mised)

•  Successfulapplica3oninawiderangeofdomains&tasks


LDAtopicmodelling•  Infer20lexical“topics”:


LDAtopicmodelling•  LDAtopicsgivenmanual“interpreta3ons”:–  (someincludeposi3ve/nega3vesen3mentaspect)


ManualvsLDAtopiccorrela3on


Outcomepredic3onusingtopics•  Usetopicweightperdialogue,withgeneralDr/Pfactors,asfeatures:

Measure ManualAcc(%)

LDAAcc(%)

HASDr 75.8 75.0

HASP 59.0 53.7

PANSSposi3ve 61.1 58.8

PANSSnega3ve 62.1 56.1

PANSSgeneral 59.5 53.4

PEQcommunica3on 59.7 56.7

PEQcommbarriers 61.9 60.4

PEQemo3on 57.5 57.5

Adherence(balanced) 66.2 54.1


Linguis3canalysis:Repair

•  Manuallinguis3canalysis–  Significantroleofrepair–  Pa3ent-ini3atedother-repair&self-repair


Compareotherdialoguecontexts

•  Therapy:moreself-repair,lessother-repair&ini3a3on


Pa3ent-doctorcomparison

•  Pa3ents:moreself-repair,lessother-repair&ini3a3on


But…•  Experimentswithautoma3cother-repairdetec3ondidn’thelp:–  Averysparseproblem(e.g.<1%ofturns)–  Needsageneralmeasureofparallelism–  Needsvocabulary-independence

•  Sosparse,evenperfectperformancewouldn’thavehelpedpredic3on

•  Wecandobe8ernow!–  (seelater)


Schizophrenia:Summary•  Predic3ngfutureadherencetotreatment:–  words&ngrams(phrases):70%–  humans:70%(transcripts),80%(video)–  topics:66%(manual),54%(auto)–  i.e.wecandoit,butwedon’treallyunderstandhow…

•  Topicmodellingprovidesusefulfeatures:–  topicscorrelatewellwithhuman-annotatedtopics–  topicspredictsymptomseverity–  topicspredicttherapeu3crela3onshipra3ngs–  topics&emo3on/stanceinterrelate

•  Repaircorrelateswithadherence–  butautoma3cdetec3onisdifficult–  …andit’saverysparsephenomenon


OnlineText-basedTherapy•  Text-basedtherapyfor

depression&anxiety–  IESODigitalHealthLtd

•  Cogni3veBehaviouralTherapy–  2,000sessions,500pa3ents,

mean5.65sessions/pa3ent•  Anonymisa3onusingRASP

–  (Briscoeetal,2006)–  Non-trivial

•  Outcomemeasure–  Pa3entHealthQues3onnaire

(PHQ-9)–  Currentseverity,progress

sincestart


Topics•  Themesincludefamily,sleep,symptoms,progress,process:0 3mesessionsorrytodaygreatsendnextnowoneworkthanksseethank

pleasehelpmakeableperhapslook

1 feellifethinkknowwaythingsnowlikewantmakeselffeelingspeoplechangemaybesomeonemuchneedothers

2 rightwellgreatsureappointmentfeelthankjustloltonightpleaseknowgetsorrysaybyemee3nglastthough

3 ea3ngeatfoodweightsickdrinkmealnowlunchcontrolgreatchocolateabsolutelydayhealthydinnerputusereally

4 3mehusbandmumfamilyfeelchildrennowdadwantseesaidfriendsalsokidshomelifegotschooldaughter

5 peoplesayangrysitua3onangersitua3onssaidwaysocialotherslikeonefriendstalksomeonepersonbehavioursayingknow

6 getgoknowlikeneedthingsgoingjustthinktrywantonesomething3megoodnowmakedaystart


TopicsvsSchizophreniaSleeppa8erns daysleepweek3mebedwork

moodnightgetthingsdayssleepday3mefeelbedbitthingshoursmorningsleepingnight

Family 3mehusbandmumfamilyfeelchildrennowdadwantseesaidfriendsalsokidshomelife

mummoneydadbrothershoppingdiedenjoytabletsbloodbaddaughtersister

Food/weight ea3ngeatfoodweightsickdrinkmealnowlunchcontrolgreatchocolateabsolutelydayhealthy

weightstoneeatmedica3ongainhospitaltwelveweighexercisecutgym

Nega3vefeelings feellifethinkknowwaythingsnowlikewantmakeselffeelings

feelmedica3onfeelingthoughts3memoodlowheadpastillness

Crises gethelpgpdepressionpainknowmedica3onhealththerapysorryappointmentlastfacemoment

rememberdoctorhospitalreasonpolicepeoplememoryringshakingheadachesdoor

Socialstress workjob3megoodstressworkinggetschoollifemoneywifeissues

thingsbackplaceyearsthoughtbitagohomeputdaycoming


Topicvsseverity&progress0 Materials,self-help,procedures − 10 Unhelpfulthinking/habits

1 Feelings/effectsofrela3onshipsonsenseofself

+ + 11 Work/training/educa3onissues/goals

2 Posi3vereac3ons/encouragement 12 Agenda/goalsetng&review

3 Issuesaroundfood 13 Panica8ackdescrip3on/explana3on − −

4 Family/rela3onships&issueswith(mostlynega3ve)

+ 14 Otherhealthcareprofessionals,crises,risk,interven3ons

++

5 Responsestosocialsitua3ons 15 Sleep/dailyrou3ne +

6 Breakingthingsdownintosteps + 16 Posi3veprogress,improvements −− −

7 Worries/fears/anxie3es − 17 Feelings,specificoccasions/thoughts

8 Managingnega3vethoughts/mindfulness

18 Explaining/framingintermsofCBTmodel

+

9 Fears,checking,rituals,phobias − − 19 Techniquesfortakingcontrol − −


Sen3ment/Emo3onDetec3on•  Detectposi3ve&nega3vesen3ment

–  seee.g.(DeVaultetal,2013)•  Detectanger

–  challenge&emo3onelicita3oninCBTprocess•  Comparedexis3ngtools

–  Manuallyannotated85u8erancesin1session•  posi0ve/nega0ve/neutral(inter-annotatoragreementκ=0.66)

•  Dic3onary-basedLIWC–  sen3ment34-45%;angerrecall=0

•  Data-based(RNNs)Stanford,trainedonnewstext(85%)–  sen3ment51-54%(noanger)

•  Data-based(SVMs),trainedonTwi8ertext–  sen3ment63-80%


Sen3ment/Emo3onvsPHQ

•  Moreposi3vesen3mentèbe8erPHQ,progress•  Morevariablesen3mentèworseprogress•  More/morevariableangerèworsePHQ

Severity(PHQ) Progress(ΔPHQ)

Sen3mentmean −− −

Sen3mentstddev +

Angermean/max +

Angerstddev +


Predic3ngfinaloutcomes•  Changesinlevelshelppredic3ngfinalin/out-of-caseness:–  usingfeaturesfromini3aland/orfinalsessions:

•  Featureschosenareinforma3ve:–  Levelsofsen3ment&anger,progress&crisis/risktopics

•  Deltasbetweensessions–  PHQscoresatassessmentandini3altreatmentsessions

FinalIn-caseness

Baselinepropor0on 26.8%

First+lastsessionfeatures,incldeltas 0.71(0.48)

IncludingearlyPHQscores 0.76(0.51)


Predic3ngdropout•  Canwepredictdropout&non-engagement?

–  148of500didnotenterorstayintreatment

•  >70%accuracyusingini3alsessionfeatures

–  Butonlybyincludingfine-grainedwordfeatures

Dropout

Baselinepropor0on 29.6%

Assessmentsessionfeatures 0.65(0.26)

Treatmentsessionfeatures 0.70(0.59)

Bothsessions 0.73(0.64)


Predic3ngtherapyquality•  Canwedis3nguish“good”from“bad”therapists?

–  Top25%vsbo8om25%basedonnumberofpa3entsrecovered

•  Goodaccuracyusingini3al&finalsessionfeatures

–  Butmostlybyincludingfine-grainedwordfeatures

Dropout

Baselinepropor0on 50%

Onlyhigh-levelfeatures 0.67(0.63)

Incudinglexicalfeatures 0.78(0.74)


Depression/Anxiety:Summary•  Topicmodellingprovidesusefulfeatures:

–  correlatewellwithhuman-annotatedtopicsandpreviousstudy–  topicscorrelatewithsymptomseverityandprogress

•  Emo3ondetec3onprovidesusefulfeatures:–  levelsandvariabilitypredictsymptomsandprogress–  needscarechoosing&trainingtools

•  Predic3ngusefuloutcomemeasures:–  recovery:71%,76%withPHQinforma3on

•  emo3onlevels&variability;talkaboutprogress&dealingwithcrises•  (arewestar3ngtounderstandwhat’sgoingon?)

–  dropout:73%–  therapistquality:78%

•  detailsofcontent&structure:wes3lldon’tunderstandthese…


SLADE:Demen3aDiagnosis•  U.Exeterdataset

–  148diagnosisconversa3onswithdoctor(&carer)•  70posi3vediagnosisofdemen3a•  78nega3vediagnosis(MildCogni3veImpairmentinsomecases)

–  AyerreferralfromGP,memorytests/scans–  Givendiagnosis,advice

•  Rela3velyearlystage–  Canweaiddiagnosis?


Demen3a&Language•  Vocabularyreduc3on(e.g.Hirst&Feng,2012)

–  Authorsoverlong3mescales•  Contentreduc3on(e.g.Orimayeetal2014)

–  Fewerpredicates,feweru8erances,shortersentences–  Demen3aBank:74%

•  Speechfeatures(Jarroldetal,2014)–  Includinglexicalclassfeaturese.g.pronoun/noun/verbfrequencies–  Smallset,healthycontrols:80-90%

•  Combinedfeatures(Fraseretal,2016)–  Impairment:seman3c,syntac3c,informa3on,acous3c–  Demen3aBank:c.80%

•  Butwehaveshort3mescales,diagnosis-dependentcontent…–  Adviceondriving,legalrequirements,futureplanning–  Andmanyotherfeaturese.g.length–  Needcontent-independentfeatures


Conversa3on-basedstudies•  ManyCA-likestudies(Watsonetal1999…Jonesetal2015)

•  Indica3vedialogue-structuralfeatures:–  “Lackoffluency”

•  Self-repair•  Lackoftopiccoherence

–  Other-repair•  Types,appropriateness,answeringbehaviour,lackofcorrec3ons

–  Ques3on-answering•  Avoidancestrategies,contentlessness

–  Pausingbehaviour•  Intra-andinter-u8erance

–  Backchannelbehaviour•  Morecontentlessu8erancesvsloweruseofcon3nuants?

–  Laughter


Tes3ngInterac3onalFeatures•  Addinterac3vityfeaturestoFraseretal(2016)model:

–  Self-repairindices:•  Pauses,filledpauses,incompletewords,repe33on,editterms•  Similarityofpost-fillerterms

–  Other-repairindices:•  Ques3onforms,inter-turnpauses•  Backchannels,answerlength

–  Generalindices:•  Laughter•  “Don’tknow”answers•  Par3cipantturn&ques3onfrequency/ra3os

•  Top3mostpredic3vefeaturesareinterac3onal(Kelleheretal,inprep)•  Performanceimprovement:

–  Benchmarkreplica3on:F=73.8%–  Addinginterac3vefeatures:F=79.4%


Doingbe8er:Other-Repair•  (HowesetalSIGDIAL2012;Purveretal,2018toappear)

•  Discrimina3veclassifica3on–  Weightedclassifierstocombatsparsity–  Per-turnincrementality–  Assumeadjacentantecedent-repairpairs

•  85%ofcases(Purveretal,2003)

•  Definefeaturesmanually,extractautoma3cally–  Linguis3cally/observa3onallyinformed:

•  Wh-ques3onwords,closedclassrepairwords•  Backchannelbehaviour,fillers,pauses,overlaps•  Lexicalparallelism,syntac3c(POS)parallelism•  Seman3cparallelism(Mikolovetal2013;Turianetal2010)

–  Bruteforce“ceiling”:allunigrams


Other-repairresults•  Resultsonrealunbalanceddata:

•  Worseonsomedatasetse.g.MapTaskF-score38-50(PRC0.55)

Target Features P(%) R(%) F(%) PRC

PCC (baseline) 1 100 3 0.01

PCC Allhigh-level 44 43 44 0.40

PCC Allfeatures 46 47 46 0.48

BNC (baseline) 4 100 8 0.04

BNC Allhigh-level 55 55 55 0.52

BNC Allfeatures 57 62 60 0.61

SWBD (baseline) 0.2 100 1 0.00

SWBD Allhigh-level 54 52 53 0.50

SWBD Allfeatures 52 60 56 0.58


Doingbe8er:Self-repair•  (Hough&Purver,EMNLP2014;Purveretal,2018toappear)•  “Disfluencydetec3on”forspeechrecogni3on

AflighttoBoston–uh,Imean,toDenverè AflighttoDenver

•  Per-wordincrementaloutput,maintainingseman3ccontext:Theinterviewwas–itwasalrightIwentswimmingwithSusan–orrather,surfing

•  Domain-general,informa3on-theore3cfeatures:–  Similari3esbetweenprobabilitydistribu3ons–  Changesinprobability&entropygivenrepairhypotheses–  Combinedinrandomforestclassifiers


Self-repairresults•  Designed&trainedonSwitchboardcorpus:

–  State-of-the-artaccuracyF=0.85(P0.93>>R0.79)–  Per-u8erancecorrela3on0.96–  Fasterincrementality

•  Transfertomentalhealthdomain(PPAT):–  F=0.62(P0.66>R0.59)–  Per-u8erancecorrela3on0.81–  Per-dialoguecorrela3on0.94

•  Othercorporalessimpressive(BNC,Colman&Healey2011):–  F=0.42(P0.40<R0.44)–  Per-u8erancecorrela3on0.58(p<0.001)


Summary•  Wecanpredictusefuloutcomemeasures

–  (diagnosis,severity,adherence)–  butwe’dreallylikeaninterpretablemodel

•  Topic&emo3ondetec3onisausefulstep–  needscarechoosing&trainingtools–  goodforpredic3ngsymptomsandprogress–  notgoodforotheroutcomes(therapyquality,adherence…)

•  Interac3onmodellingisanotherusefulstep–  par3cularlytheroleofrepair(self-andother-)–  approxima3onshelpdemen3adiagnosis–  generalmodelsarenowinastatetobeapplied!


AcknowledgementsTheCMSIprojectwassupportedbyCrea3veWorksLondon,aKnowledgeExchangeHubfortheCrea3veEconomyfundedbythe

ArtsandHumani3esResearchCouncil;andcompletedincollabora3onwithCha8erboxLabsLtd&theBarbican

ThePPAT&AOTDprojectsweresupportedbyQueenMaryUniversityofLondon’sEPSRC-fundedPump-PrimingandInnova3onFunds,andPsychologyOnlineLtd;andcompletedincollabora3onwithPsychologyOnlineLtdandiLexIRLtd.

TheprojectConCreTeacknowledgesthefinancialsupportoftheFutureandEmergingTechnologies(FET)programmewithinthe

SeventhFrameworkProgrammeforResearchoftheEuropeanCom-mission,underFETgrantnumber611733§

Documents

Analysing Dialogue for Diagnosis and Predic3on in Mental