Don’t Let Notes Be Misunderstood: A Negation Detection Method for Assessing Risk of Suicide in...

Don't Let Notes Be Misunderstood:

A Negation Detection Method for Assessing Risk of Suicide in

Mental Health RecordsGeorge Gkotsis, Sumithra Velupillai, Anika Oellrich

Harry Dean, Maria Liakata and Rina Dutta

Biomedical Research Centre Nucleus – King’s College London

e-HOST-ITElectronic health records to predict HOspitalised Suicide attempts:Targeting Information Technology solutions

AimTo determine whether structured and free-text data in Electronic Health Records (EHRs) can be used to quantify changes in symptoms, behaviour patterns and health service-utilisation and predict serious suicide attempts

Motivation

• Health records contain many structured fields, such as:• Personal information/contact• Diagnosis• Prescription• Interventions• Scans & Measurements

Most of the structured fieldsare left blank

The mysterious case of health records (1/2)

Max Weber

Theory ofFormal Rationality

The mysterious case of health records (2/2)• Free text contains a lot of information

• Traditional information access technology returns many false positives

Example1. Patient is suicidal

2. Patient is not suicidal

• Meaning can be expressed in multiple ways

Example1. He has suicidal thoughts2. He wants to end his life3. She wants to kill herself

CRIS database• 226,000 patients• 18.6 million documents (Event)

• Suicide-related data• 783,000 documents contain the word suicid*• 111,000 patients

Anonymous Reviewer:“Overall,I think the paper is well thought out and written, and I am

envious of their access to such a large patient dataset”

Problem description

Negation detection – definition (1/2)

“The determination of whether a finding or disease mentioned within narrative medical

reports is present or absent”*

Negex, Chapman et al.Journal of Biomedical Informatics, 2001

Negation detection – definition (2/2)

Negation Detection

Sentence

Target keyword

Positive/Negative

Towards negation detection resolution• Fundamental NLP taskReduced to identifying the scope of negation

Examples:

No issues other than her indicating that she might commit suicide

He continues to deny any suicidal thoughts and is happy to come to the XXX for medical review tomorrow

+

-

State-of-the-art: Negex (1/2)• Lexical-based approach

• Collection of negation cues/expressions• Pseudo-negation expressions• Termination cues for scope• Search scope of 6 words surrounding the target keyword

• pyConTextNLP

State-of-the-art: DEEPEN (2/2)• Wrapper over NegEx• Applied over the (predicted to be) negated sentences• Uses a dependency parse tree

“Negation’s Not Solved”*• Optimizable, but not generalizable• Annotation guidelines are different• Spans considered can be nouns or whole phrases• Amount of overlap allowed (or not)

*Wu et al.PLOS One, 2014

Proposed solution

Workflow

Annotated Dataset

Preprocessing

CoreNLPSentence

Parse tree

Target Node

Negation Detection

1. Pruning

2. Identification of

dominating

subordinate

clause

3. Identification of

negation

governing the

target-node

4. Negation

resolution

Positive/Negative

TargetKeyword‘suicid*’

MHRs

‘suicid*’

Annotation

Dataset and annotation

Proposed Methodology

2941

3125

Positive Negative

Dataset and annotation• Generation

• Random sampling from SLAM Events of 6k sentences containing the word “suicid*”

• Annotation• One expert annotated the complete corpus• Another expert repeated the annotations for 25% of the sentences𝛋=0.93 (IAA=97.9%)

Limitations• Linguistic focus• Patient-agnostic

1. PruningLabels• Subordinate conjunctions• ,• S• SBAR• SINV

2. Identification of dominating subordinate clause

SBAR

3. Identification of governing nodesS

Evaluation

Comparison1. Proposed Model (uses 15 negation words)2. NegEx (uses 272 rules)3. pyConTextNLP-N (uses [2])4. pyConTextNLP-O (uses [1])

no, without, nil,not, n't, never, none, neith, nor, nondeny, reject, refuse, subside, retract

Results (1/3)

Positive NegativePositive 2782 331Negative 159 2794

Total 2941 3125Pred

ictio

n

Class

Results (2/3)

Precision Recall FM AccuracyNegEx 93.4 92.1 92.8 93

pyContextNLP-N 94.1 92.9 93.5 93.7pyContextNLP-O 80.7 86 83.2 83.2

Proposed 89.4 94.6 91.9 91.9

Results (3/3)

Discussion & Conclusion

Distribution of sentence length

Proper punctuation and sentence chunking are crucial!

Discussion• Corpus of 6k sentences from Mental Health Records• Annotation of high quality• Evaluation - focus on positive cases

• Parse trees+Require minimum number of negation keywords+Further potential

• Statement extraction (subject-predicate-object)• Temporal characteristics• Degree of suicidality

- Expensive - Error prone for long sentences

Future work (1/2)Expand on expressions of suicidality

oStudy how negation detection can be used to strengthen predictive power of mental health recordsoOngoing cohort study (pupils with ASD)o Large scale study based on hospitalisation events

Future work (2/2)• Evaluate our tool against other datasets/domains

• Consider syntactic dependency parser (instead of constituency-based)• spaCy• SyntaxNet

https://github.com/gkotsis/negation-detection

Source code