The Role of Social Media and Artificial Intelligence for Disaster Response

TheRoleofSocialMediaandAr2ficialIntelligenceforDisasterResponse

MuhammadImranQatarCompu+ngResearchIns+tute

HamadBinKhalifaUniversityDoha,Qatar

May25,2016

h<p://mimran.me/

Fromtradi*onaltoemergingtoolsforcrisisresponse

ThisTalkisAbout…•  TheRoleofInforma2oninTime-cri2calSitua2ons

–  Naturaldisastersandtheirdestruc+ons–  Man-madedisastersandmassconvergenceevents

•  TheRoleofSocialMediaforDisasterResponse–  Par+cularfocusonmicro-bloggingplaJorms–  Availabilityofvarioustypesofinforma+onandopportuni+es

•  TheRoleofAr2ficialIntelligenceforDisasterResponse–  HowAIisusefulindisasterresponse–  VariousAItechniques,approaches,andtools–  Workofcrisiscompu+nggroupatQCRI–  Ongoingresearch–  Futuredirec+ons

Source:UNISDR

MostAffectedCountriesbyNaturalDisasters(1995-2015)

HumansSufferingandEconomicDamagebyDisasters

Humanssufferingfromtheimpactsofdisasters,crises,andarmedconflict

Millionsofpeopleaffectedeachyearbydisasters;Atanannualcosttotheglobaleconomythatexceeds$300billion

PlanandPrepare



Disastersareunavoidablebutplanningcanlessen

theireffects

PlanandPrepare



Providehelpinghand…

-  Between2008and2014,184millionpeopledisplacedbynaturaldisasters

-  Over60millionduetoconflicts

-  Anaverage26.4millioneachyear

TheUrgencytoActandPlan

Informa2on:ALifelineDuringDisasters

Theopaquenessinducedbydisastersisoverwhelming

Peopleneedinforma2onasmuchaswater,food,medicineorshelter

Lackofinforma+oncanmakepeoplevic2msofdisasterandtargetsofaid

DataorDialogue?TheRoleofInforma2oninDisasters

IainLogan,formerheadofdisasteropera+ons,Interna+onalFedera+on:“Theveryfirstthingyouneedtodoisclimbintoahelicopter.Youcan’tgettoseehowmanypeopleareburiedbutyougetaneagle’seyeview…Youcanseewhichairfieldsareworking,whichbridgesaredown.Aderthreetofourhoursinahelicopter,Ihadacompleteoverviewofthegeographicalextentofthedisaster(earthquakeinElSalvador,2001),thelogis+csinvolved,thepopula+oncenters–alsowherenottosendpeople.Thenyoumustgetonthegroundtogetthequality.”

-Informa+onBestowsPower-

TheRoleofSocialMedia

SocialMediaUseDuringChristchurchEarthquake

Aself-organizedworkforceof10,000volunteersgatheredonFacebook

TheRoleofTwi<erDuringThailandFloods

[AlisaKongthonetal.2011]

Twi<erBreaksEventsFasterFirstreport

Breaksthestory33minutesbeforelocalTV

HudsonPlaneCrash

WestgateMallA<ack

Twi<erBreaksEventsFasterFirstreportonTwi<er Aaer1minute

Aaer2minutes

BostonBombing

CrisisCommunica2onsBeforeandNow

GeraldBaron

AnalysisofTwi<erCrisis-RelatedDataAnIn-depthStudy

Twi<erCrisis-RelatedData(2012)

Source:QatarCompu+ngResearchIns+tute-PublishedinWorldHumanitarianDataandTrends2014(UNOCHA)

Twioerdatafrom13crises;Analyzedover100,000tweets;Informa+ontypesandsources

Twi<erCrisis-RelatedData(2013)


Twioerdatafrom13crises;Analyzedover100,000tweets;Informa+ontypesandsources

Twi<erCrisis-RelatedData(All)

-  Twioerdatafrom13recentcrises

-  Over100,000tweets

-  Informa2ontypes

-  Typesofsources


UNOCHAHumanitarianClusters-RelatedInforma2ononTwi<er

TyphoonYolanda–OCHAClusters

-  Performedanalysisof

morethan440,000tweetsduringthefirst48hours

-  15%ofthetweetsfoundpoten2allyrelevant


TyphoonYolanda–OCHAClusters


-  Performedanalysisof

morethan440,000tweetsduringthefirst48hours

-  15%ofthetweetsfoundpoten2allyrelevant

SandyHurricaneTwi<erDataAnalysis

@NYGovCuomoordersclosingofNYCbridges.OnlyStatenIslandbridgesunaffectedatthis+me.Bridgesmustcloseby7pm.#Sandy#NYC.

rt@911buff:publichelpneeded:2boys2&4missingnearly24hoursadertheygotseparatedfromtheirmomwhencarsubmergedinsi.#sandy#911buff

freakingout.homealone.willjustwatchtv#Sandy#NYC.

400Volunteersareneededforareasthat#Sandydestroyed.





Personal

Informa+ve






Personal

Informa+ve

Cau+onandAdvice

Casual+esandDamage

Dona+ons






Personal

Informa+ve

Cau+onandAdvice

Casual+esandDamage

Dona+ons


MERSOutbreakTwi<erDataAnalysis

MiddleEastRespiratorySyndrome(MERS)Twioerdatacollec+onfrom:2014-04-27to2014-07-14usinghashtag#MERS(Total=215,370)Dataanalysis:Reportsofsymptoms Affectedpeoplereports Deathreports

DiseasetransmissionreportsPreven+onques+ons Treatmentques+ons

Reportsofsignsorsymptomssuchasfever,coughorques+ons

ReportsofaffectedpeopleduetotheMERSdisease

ReportsofdeathsduetotheMERSdisease

Ques+onsorsugges+onsrelatedtothepreven+onofdisease

Reportsorques+onsrelatedtothetransmissionofthedisease

Ques+onsorsugges+onsregardingthetreatmentofthedisease

SocialMediaDuringMERSOutbreak

RT@abeoel:TwoworkersatFLhospitalexposedtoapa+entwithMiddleEastRespiratorySyndromeareshowingflu-likesymptoms

Coronavirussymptomsinclude:fever,coughing,shortnessofbreath,conges2oninthenoseandthroat,andinsomecasesdiarrhea.MERS#MERSisarela+velynewrespiratoryillness,spreadb/wpeopleinclosecontact.Symptomsarefever,cough,&shortnessofbreath.SaudiArabiafindsanother32MERScasesasdiseasespreads:RIYADH(Reuters)-SaudiArabiasaidonThursday...hop://t.co/cPhm0uTRCo

Signsandsymptoms

Signsandsymptoms

Signsandsymptoms

Affectedindividuals

SocialMediaDuringMERSOutbreak

FirstCaseofDeadlyMiddleEasternVirusFoundinU.S.:TheCentersforDiseaseControlhasconfirmedthatacaseofthedeadlyMidd...

ThirdCaseofMERSConfirmedintheU.S.:TheU.S.CentersforDiseaseControlandPreven+onconfirmedonSat...hop://t.co/Sb8PMyxVUnNocleartransmissionlinkbtwncamelsandhumansforMERS.94%Egyp+ancamelsseroposi+vebutnohumancasesyet.Hmm#asm2014Saudihealthauthori+esannouncedonMondaythatthedeathtollfromtheMERScoronavirushasreached115sincetherespiratorydisease...

Transmission

Deathreports

Affectedindividuals

Affectedindividuals

ISCRAMCallforPapers

AidisOutThere!

AiderisOutThere!

AIDRisAlsoOutThere!

TheRoleofAr2ficialIntelligence

2013PakistanEarthquakeSeptember28at07:34UTC

2010Hai2EarthquakeJanuary12at21:53UTC

DataandOpportuni2es

SocialMediaPlamorms

AvailabilityofImmenseData:

Around16thousandstweetsperminutewerepostedduringthehurricaneSandyintheUS.

Opportuni2es:-  Earlywarningandeventdetec2on

-  Situa2onalawareness

-  Ac2onableinforma2on

-  Rapidcrisisresponse-  Post-disasteranalysis

Diseaseoutbreaks

ProcessingSocialMediaData

Filter:removing,duplicates,spamandmessagesfrombots

Classify:categoriza+onofitemsintoinforma+ontypes

Cluster:iden+fytrendingandemergingtopic

Aggregate:makingsensebyconnec+ngdifferentpieces

Extract:shortsnippetsoffocusedinforma+on

Summarize:learningabiggerpictureofanevent

WAIT!Beforeapplyinganytechnique?Please!Havealookatyourdatafirst

DataCharacteris2csandPrepara2on

•  Single-wordslangs:pls(please),srsly(seriously)•  Mul2-wordslangs:imo(inmyopinion)•  Misspellings:missin(missing),ovrcme(overcome)•  Phone2csubs2tu2on:2morrow(tomorrow)•  Wordwithoutspaces:prayfornepal(prayfornepal)

Canyouguess?“ruokm8”??

>>“AreyouOK,mate?”

ToolstoProcessSocialMediaData

SystemsforCrisis-RelevantDataProcessing

Twitris[PurohitandSheth2013]Twioer;seman+cenrichment,classifyautoma+cally,geotag

SensePlace2[MacEachrenetal.2011]Twioer;geotag,visualizeheat-mapsbasedongeotags

EMERSEEnhancedMessagingfortheEmergencyResponseSector[Carageaetal.2011]TwioerandSMS;machine-translate,classifyautoma+cally,alerts

ESAEmergencySitua+onAwareness[Yinetal.2012;Poweretal.2014]Twioer;detectbursts,classify,cluster,geotag

SystemsforCrisis-RelevantDataProcessing

Twitcident[Abeletal.2012]TwioerandTwitPic;seman+cenrichment,classify

CrisisTracker[Rogstadiusetal.2013]Twioer;cluster,annotatemanually

Tweedr[Ashktorabetal.2014]Twioer;classifyautoma+cally,extractinforma+on,geotag

AIDR:Ar2ficialIntelligenceforDisasterResponse[Imranetal.2014a]Twioer;annotatemanually,classifyautoma+cally

Ar2ficialIntelligenceforDisasterResponse

Informa2onProcessing

Datacollec+on

1 2Humanannota+onsonsampledata

Machinetraining

3Classifica+on

4

DisasterTimeline:

DATACOLLECTION

Humansalonecannotprocesslargeamountsofdata,soweonlyusethemtohelpprocessasubset

Wetrainmachineusinghumaninputtoautoma+callyprocesslargeDataathighspeed

ForexampleusingKeywords,hashtagsetc.

ImpactandResponseTimeline

DepartmentofCommunitySafety,QueenslandGovt.&UNOCHA,2011

Disasterresponse(today) Disasterresponse(ourtarget)

Requiresreal-2meprocessingofdata

Datacollec+on

1 2Humanannota+ons Machinetraining

3Classifica+on

4

ONLINEAPPROACH

DATACOLLECTION

HA

Learning-1

CLASSIFICATIONOFDATA&DECISIONMAKINGPROCESS

Learning-2 Learning-3 … Learning-n

Humanannota+on-1

Humanannota+on-2

Humanannota+on-3 … Human

annota+on-n

Firstfewhours

Informa2onProcessing(Real-2me)

BigChallenges–4Vs

•  VolumeScaleofdata(20mtweetsin5daysTyphoonOklahoma)

•  VelocityAnalysisofstreamingdata(16k/minduringSandy)

•  VarietyDifferentforms/typesofdata(informa+ontypes)

•  VeracityUncertaintyofdata

MachineLearning+Crowdsourcing

hop://aidr.qcri.org/

AIDR=Machinelearning+Crowdsourcing

CrowdsourcedStreamProcessingCombininghumanandmachinecomputa2on

Difficult, ambiguous items to be labeled by crowd

Automatic processing


output output

Performing verification

Providing training data

a: Split automatic/manual processing b: Detect-verify paradigm


Automatic processing output

c: Improving quality through active learning

input input

input




output output







input input

input




output output







input input

input

Qualityassuranceloops:humanprocessingelementsdothework,automa+cprocessingelementscheckforconsistencyProcess-verify:workisdoneautoma+cally,humanschecklow-confidenceorborderlinecasesOnlinesupervisedlearning:humanstrainmachinestoperformworkautoma+cally


AIDR—Ar+ficialIntelligenceforDisasterResponse—isafree,open-source,andeasy-to-useplaJormtoautoma+callyfilterandclassifyrelevanttweetspostedduringhumanitariancrises.

1 2 3

Collect Curate Classify

AwardedtheGrandPrizeintheOpenSourceSoawareWorldChallenge2015

AIDR:FromEnd-usersPerspec2ve

Collec2on Classifier(s)

•  Keywords,hashtags•  Geographicalboundingbox•  Languages•  Followspecificsetofusers

Acollec2onisasetoffilters Aclassifierisasetoftags•  Dona2onsrequests&offers•  Damage&causali2es•  Eyewitnessaccounts•  …

2stepsapproach1 2


Real-2meClassifica2oninAIDR


Trainer

AIDR–Collec2onSexngCollec2ondetaildashboard


GeographicalregionfilterLanguagefilter

Collec2ondefini2on


AIDR–ClassifiersSexng

AIDR–ClassifierSexng(cont.)


HumanAnnota2oninAIDRInternalTaggingInterface


HumanAnnota2onUsingMicroMappersMicroMapperInterface(webclicker)


Mobileclicker

TaggedItemsandMachineOutput


Trainingexamples Classifiers’output

High-levelArchitecture


Items Collector Feature Extractor Classifier(s)

Learner Crowdsourcing Task GeneratorStream of incoming

items from data sources

Item & featuresItem

An expert defines classifiers by givinga name and description for each category

Expert

Items

Crowd workers/volunteers

Model parameter

ClassifiedItem

A list of classified items by categoryand classifier’s confidence

Labelingtasks

Labeleditem

Datasource

Datasource

Quality,Cost,andPerformanceofAIDR

Qualityvs.CostinAIDR


Goal:Maximizingqualitywhileminimizingcost•  Quality•  classifica+onaccuracy•  Precision

•  Cost(humanlabels)•  monetaryincaseofpaid-workers•  +meincaseofvolunteers

Qualityvs.CostinAIDR


Qualityvs.costusingpassivelearningandde-duplica2on

Qualityvs.costusingac2velearningandde-duplica2on

Performance


Intermsofthroughputandlatency

Latencyoffeatureextractor,classifier,andthesystem

Throughputoffeatureextractor,classifier,andthesystem

TyphoonHAGUPIT(2014)

UNICEFU-ReportandAIDR-SMS

AItoAnswerHeathQueriesviaSMS

•  EveryhourZambianyouthgetinfectedwithHIV/AIDS

•  UNICEFlaunchedU-ReportprojectinZambia

•  UsageofU-ReportplaJormhasrecentlyincreased300%

UNICEFU-ReportinZambia

Manualprocessingandrou+ngofSMS

Counselors(expertsofHIV,STIs)

SMSservice

1 2

3

4

5

6

Vulnerablepeople

UNICEFU-ReportinZambia+AIDR

Manualprocessingandrou+ngofSMS

Counselors(expertsofHIV,STIs)

SMSservice

1 2

3

4

5

6

Vulnerablepeople

NewScien2stFeaturedThisWork

MediaCoverage

HumanAnnota2onSelec2onandschedulingforsupervisedclassifica2onsystem

OngoingWork

HumanAnnota2on-Challenges

1-Labelingtaskselec2on• Whichtaskstopickforlabeling?•  Noduplicatetasksshouldbelabeled•  Priori2zetasksthatarelikelytoincreaseclassifier’saccuracy

Crowdsourcingisabigresearchtopic.Weaddresstwochallengeshere:

Twi<erCrisesDatasets

1.   Joplin-2011•  Consistsof206,764tweetscollectedusing(#joplin)

2.   Sandy-2012•  Consistsof4,906,521tweetscollectedusing(#sandy,hurricanesandy,…)

3.   Oklahoma-2013•  Consistsof2,742,588tweetscollectedusing(Oklahoma,tornado,…)

Distribu2onofTweetsintoPhases

Pre:preparednessphaseImpact:phasecorrespondstotheperiodinwhichthemaineffectsarefeltPost:correspondstoresponseandrecoveryphase

Joplin(led),Sandy(center),andOklahoma(right).Numberoftweetsperdayinalldatasets.

LabelingTaskSelec2onExperiment:Isde-duplica2onnecessary?

Phase Train Phase Test AUC(withoutde-duplica2on)

AUC(withde-duplica2on)

S1(pre) 1,500 S1(pre) 500 0.78 0.74

S1(pre) 500 S1(pre) 500 0.73 0.72

S2(impact) 500 S2(impact) 500 0.80 0.72

S3(post) 500 S3(post) 500 0.79 0.73

S4(post’) 500 S4(post’) 500 0.70 0.64

•  29-74%oftweetsarere-tweets&60-75%arenearduplicates•  Duplica+oncausesanar2ficialincreaseinaccuracy•  Necessarytoreduceclassifierbias.Otherwiselearningonafewerconcepts•  Necessarytoimproveworkersexperience

[Rogstadiusetal.2011]

LabelingTaskSelec2onExperiment:Passivelearningvs.Ac2velearning

JOPLIN

SANDY

OKLAHOMA

S1 S2 S3 S4

AUCstabilizewithfewertrainingitemsusingac+velearning

LabelingTaskScheduling

•  All-at-oncelabeling•  Obtain1,500labelsonS1anduseallfortraining

•  Cumula2velabeling

•  Obtain500labelsineachofS1,S2,andS3andtrainonlabelsavailableuptoeachphase

•  Independentlabeling•  Obtain500labelsineachofS1,S2,andS3andusethemostrecentlabelsfortraining,discardingold.

2-Labelingtaskscheduling

LabelingTaskSchedulingExperiment:Whichlabelingstrategytofollow?

JOPLIN

SANDY

OKLAHOMA

Informa2ve Informa2ve(50%) Dona2ons

All-at-onceapproachdominatesininforma+veandcumula+vestrategyseemsbeoerfordona+ons

DomainAdapta2onAbilityofasystemtoapplyknowledgeandskillslearnedinpreviousdomainstonoveldomains

OngoingWork

OurGoal:Tobuildasystemthatcanunderstandnaturallanguage

DomainAdapta2onLabeledsource,butunlabeledtarget

Featureextractor

Machinelearningalgorithm

Featureextractor

Classifiermodel

Inputdocuments(bluedomain)Featurevectors

Labels

FeaturevectorsMachineclassifieditems

Inputdocuments(orangedomain)

Training

Predic2on

Sourceeventdata Targeteventdata

SameDomainLearning

Trainingdata Machinelearningmodel Tes+ng

datainfer predict

Apples Apples

Apples

Oranges

Differentshapes,colors,skins,tastes,etc.

Sourcedomain Targetdomain

Oranges

Oranges

BUT

Crisis-relatedDataClassifica2on

Trainingdata Machinelearningmodel Tes+ng

datainfer predict

ItalyearthquakeQueenslandfloodsSandyhurricane

CostaRicaearthquakeColoradofloodsTyphoonHaiyan

Differentevents,languages,needsetc.

Sourcedomain Targetdomain

DomainAdapta2on

ModelAdapta2onExperiments

•  Modeladapta+onusingsinglesource– Usingboth:in-domainandcross-domain

•  Modeladapta+onusingmul2plesources–  In-domain– Mul+plesourceeventswithoutthetarget– Mul+plesourceeventswiththetarget

•  Modeladapta+oninspecialcases– Samelanguages– Similarlanguages

Observa2ons&Findings

•  Datafromearlyhoursofacrisishelp•  Pasteventsofsametypeareuseful•  Samelanguagedataastargeteventisalsouseful•  Similarlanguagesarealsouseful•  Cross-domaintrainingdoesnotshowsignificantimprovements

RapidCrisisResponseFutureDirec2ons

ImageProcessingforDamageAssessment

Tasks•  Imagecategoriza2on•  E.g.building,bridge,roaddamage

•  Damageandseverityassessment•  Givenadamageimage,iden+fyseverityofdamage(low,mild,high)

DeepLearningtoImproveClassifica2on

Tasks•  Improvetextclassifica2onperformance•  Availabilityofbigdata•  Automa+cfeatureslearning•  Binaryandmul+-classclassifica+on•  Tes+ngdatafrommul+plepastevents

TransferLearningDifferencesinclassifica2ontasks:•  Differentclassifica+ontasks•  Differenttypesofdisasters,stakeholders,informa+onneeds

Task:•  Learnfromsourcetoclassifytarget•  Seman+csimilaritybetweentasks•  Instancessimilaritybetweendomains•  Instanceweigh+ng

Summariza2onandPriori2za2onofAc2onableInforma2on

Informa2onneeds&problem:

•  Differentstakeholders•  Differentgoals,requirements,andinfo.needs

Generalsitua2onalawarenessvs.Targetsitua2onalawareness•  High-levelgeneralupdatesfromanevent•  Specificupdates(infrastructuredamages)

Resources,Datasets,AndTools

TowardsStandardBaselinesandDatasets

CrisisNLP.qcri.org

-  Accessto52milliontweets-  Around50klabeledtweetsintohumanitariancategories-  Largestword2vecembeddingstrainedon52mcrisis-relatedtweets-  Out-of-vocabularydic2onaries

TowardsStandardBaselinesandDatasets

UpcomingBook

DigitalHumanitarians:Book

ACMCompu2ngSurvey

ProcessingSocialMediaMessagesinMassEmergency:ASurvey[Imranetal.2015]

Conclusions•  Informa2onbestowspowerfordisasterresponse

–  Peopleneedinforma+onasmuchaswater,shelter,andfood–  Disastersareunavoidable,butplanningcanlessentheireffects

•  Socialmediaas2me-cri2calinforma2onsource–  Earlywarnings,eventdetec+on,eventmonitoring–  Availabilityofinforma+onopensnewopportuni+es

•  Ar2ficialIntelligenceforDisasterResponse–  Appliedresearchatitsbest–  AI+humans-in-the-loopcanenablerapidcrisisresponse–  AItechniquesusefulfor:

•  Situa+onalawareness•  Ac+onableinforma+onextrac+on•  Summariza+on

THANKYOU!h<p://mimran.me/

MuhammadImran

CrisisNLP.qcri.orgAIDR.qcri.org

Technology

The Role of Social Media and Artificial Intelligence for Disaster Response