Internetdiscussionforumsformedicaladvice

JenniCole,ChrisWatkins

Interactionbetweenspreadofanepidemicandofnewsabouttheepidemic2006-2010EPSRCgrant‘AmorphousComputation’ £400,000VincentJansen(Biology,PI),ChrisWatkins(ComputerScience)(Iwasonthisgrantbyaccident)

SebastianFunk(PhDstudent)nowalectureratLondonSchoolofHygieneandTropicalMedicineErez Gilad(PDRA)nowatUnitofNuclearEngineering,Ben-GurionUniversity,Israel

Introducedtheoreticalmodelsshowingthatthelocalspreadofnewsaboutanepidemiccansloworsuppressthespreadoftheepidemic(inprinciple).

Idea:diseasespreadsononenetwork,andinformationaboutthediseasespreadsonanotherrelatednetwork.

Aspeoplegetsick,localinformationaboutthediseaseisgeneratedandspreadsontheinformationnetwork;asuninfectedpeoplegetinformationaboutthepresenceofthedisease,theychangetheirbehaviour toavoidcatchingit.

Somenicemodels.Notapplicableinpracticesofar.

Funk,Gilad,Watkins,JansenProceedingsoftheNationalAcademyofSciences106(16),2009(345citationsJ )

OnthefeasibilityofusingtwomobilephonesandWLANsignaltodetectco-locationoftwousersforepidemicpredictionNguyen,Luo,WatkinsProgressinLocation-BasedServices2014,63-78(InternallyfundedPhDatRoyalHolloway)

NPIs include:- personal protective measures

- voluntary home isolation of ill persons - respiratory etiquette, and hand hygiene- voluntary home quarantine of exposed household members - use of face masks when ill

- community measures aimed at increasing social distancing - school closures and dismissals - social distancing in workplaces- postponing or cancelling mass gatherings

- environmental measures - routine cleaning of frequently touched surfaces.

Several new elements [in] the 2017 guidelines ... anewsectiononcommunityengagementhasbeenincludedtohighlightthatthetimelyandeffectiveuseofNPIsdependsoncommunityacceptanceandactiveparticipation.

Givencurrentvaccinetechnology,apandemicvaccinemightnotbeavailableforupto6months

Casefatalityratio>1%Casehospitalisation >7%

• Promoting public understanding that individual action is essential ... NPI recommendations might change as new knowledge is gained.

• ...establishing communication channels that enable members of the public to ask questions and express concerns (e.g., call centers or social media sites)

• Ensuring that school dismissals and other NPIs are acceptable to the community during a pandemic

• Coordinating with local partners to support households complying with voluntary home quarantine (e.g., providing necessary food and supplies)

• Identifying strategies for mitigating the secondary consequences of school dismissals and other social distancing measures (e.g., modifications or cancellations of mass gatherings)

• Minimizing intervention fatigue* during a pandemic

PublicacceptanceofNPIs

Manythingswouldneedtobediscussedbymanypeopleduringashortperiodoftime.

TheRoleofOnlineDiscussionForumsduringaPublicHealthEmergency

ResearchbyJenniCole,SeniorResearchFellow,RoyalUnitedServicesInstitutespecialising inNationalSecurity,Resilience,EmergencyManagement

PhDstudentship~£50,000internallyfundedbyRoyalHolloway,2013-2017supervisedbyChrisWatkins(ComputerScience),DorotheaKleine (ICT4D),andAlPinkerton(Geography)

• InterviewstudyofEbolawitnesses&whatinformationtheysoughtandneededduringtheepidemic

• Accuracyandusefulnessofadviceinon-linediscussionforums,assessedbypanelsofdoctors.

• ObservationsandinterviewsofRedditmoderators– howtheymanageforumdiscussions,skillsrequired

• Studiedthehistoryandgrowthofreddit.com/r/ebola duringthe2014Ebolaoutbreak

Internationalemployees’concernsduringseriousdiseaseoutbreaksandthepotentialimpactonbusinesscontinuity:Lessonsidentifiedfromthe2014–15WestAfricanEbolaoutbreak.J Cole,CWatkins.Journalofbusinesscontinuity&emergencyplanning9(2),149-162,2015

HealthAdvicefromInternetDiscussionForums:HowBadIsDangerous?JCole,CWatkins,DKleine.JournalofmedicalInternetresearch18(1)2016

Informationsystemsduringapandemic(PHEIC)FromtheEbolawitnessinterviewswelearned:

• Differentinfoisneededbypeoplefar-at-risk,near-at-risk,andreal-at-risk• Peoplereal-at-riskhavedetailedpracticalquestions,notfullyansweredbyofficialadvice

Requirementsforaninformationsystem:

• Standardapproachtopublichealthinformationisone-waycommunicationfromexpertstopublic:thisisnotenough

• Needaccuratecollaborativequestion-answeringonahugescale

• Fromzerotomillionsofusersinlessthanamonth

• Shouldbeabeaconofqualityamidothersocialmediadiscussion• FacebookandTwitterdonothavequalitycontrolbymoderatorsorcollaborativevoting

reddit.com• FourthmostvisitedwebsiteintheUK(alexa.com),afterGoogle,Youtube,andFacebook

• acollectionofmoderateddiscussionforumsoneverysubject,fromdisreputabletohigh-minded• someforumslarge:worldnews has17millionsubscribers,manymorereaders

• userscanpostlinksorself-writtenquestions:reddit supportsnewsaggregation,question-answering,anddiscussioninsameforum

• moderatedbyvolunteers:anyonecanstartanewforum(subreddit)• moderatorssetthepolicyandtoneforeachforum• candeletepostsandbanundesirableusers• manyskills:moderationteamscallinhelpfromothersubreddits asneeded

• subtle‘two-dimensional’votingsystemforprominenceandquality• qualityofdiscussionispolicedbymoderators,andnudgedbyvotingincentives

Ordervaluebeginstobeinfluencedfromthefirstactiveupvotecast

0

Itisnotuntilthethirdactivedownvotethattheordervaluebeginstobeaffected

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

-10 -5 0 5 10 15

Relativehot-scorevsupvotes - downvotesTop-levelposts areorderedby‘hot-score’

relative_hot_score =sgn(upvotes– downvotes )*log(abs(1+upvotes – downvotes ))

hot_score =time+relative_hot_score

Oneunitoftime– a‘reddit day’– is45000seconds.

Atleast3downvotes areneededbeforerelative_hot_score changesOriginallyabug,nowvaluedasafeature

Eachpost hasanassociatedtreeofcomments

Ateachnodeofthecommenttree,thechild-nodesareorderedbythe‘best’scoring

best_score =lower_confidence_bound_of_Bernoulli_parameter(#upvotes,#downvotes )

OriginallysuggestedbyRandallMunroe(authorofXKCD)Noadditionaltheory(?)

Commentsdown-votedbelowthresholdaredisplayedincollapsedform:noteventrollslikethis.

Norestructuringofcomment-treeaccordingtovotes.Child-commentscanhavehigherscoresthantheirparents,butscoresnotpropagateduptree

Aimof‘best_score’istoallowgoodlatercommentstoovertakeearliermediocrecommentsPartiallysuccessful.

Allvotescoresnownotfullyvisibletopreventgamingbybots:onlypartiallysuccessful.

Redditislikearackofmagazines• 10,000sofsubreddits,eachonelikeacontinuouslypublishedmagazine

• Rangefromthescurrilousandpornographictothesoberandhigh-minded• Subreddits suchas/r/conspiracyand/r/The_Donald likefakenewsandgenerateit• Subreddits suchas/r/worldnews seektodiscussonlycorrectandreliablenews• Hypothesis:thenatureandpurposeofasubreddit ismadecleartousers,andusersknowwhattypeofmaterialtheyarereading,inthesamewaythattheyexpectdifferentnewsin“TheSun”and“TheGuardian”

• Userscanavoidanechochamberbysubscribingtoamixofsubreddits

• eachsubreddit isentrepreneuriallycreated,promoted,andrunbyitsmoderators• moderatorscannot affectvoting(integrityofvotingisimportant)• moderatorscandeletepostsandcomments,andsuspend/banusers,andsetthetoneofdiscussion

INFLUENCEPOORQUALINFO INFLUENCEHIGHQUALINFO

Admins ClosedownsubredditsForceoffensivesubredditstogoprivateRemovealeadmoderator

Handoveranabandoned/poorlymoderatedsubtoanewmoderator

Moderators- accessto

moderationtoolsandsystemprivileges

CounterclaimsinpostsandcommentsRemovecontentBanpostersBandomainsFlairlowqualitySetrulesDownvote

StartsubredditsDesignateapprovedpostersAssignothermoderatorsPostgoodqualityinformationCommentonotherpostsUpvote

Approvedposters- somesystem

privileges

CounterclaimsinpostsandcommentsReportbadinformationDownvote

PostgoodinformationimmediatelyCommentfavourablyUpvotegoodinformation

Registeredusers Makecomments(maybedelayed)ReportbadinformationDownvote

Postgoodinformation(maybedelayed)CommentfavourablyUpvote goodinformation

Levelsofqualitycontrolonreddit

Amodelofhowusersevaluatesourcesofon-lineinformation(Wathen andBurkell 2002)

Trafficonr/ebola in2014Contextshifts:changesintypesofquestionaskedassituationchanges

Communityshifts:changesinthetypesofpeopleusingthesubreddit

SUBSCRIBERS UNIQUEVIEWS PAGEVIEWS

Jul2014 130 1,698 6,839

Aug2014 1,073 26,195 167,087

Sep2014 4,636 55,793 556,473

Oct2014 14,192 446,497 5,017,046

Manynewignorantusers=>lowerqualityvoting,manyrepeatedquestions

e.g.RepeatedpostsimplyingthatEbolacouldbecomeairbornewerestronglyupvoted,eventhougheachsuchpostwasimmediatelyrefutedinthecommentsbelow

Problemsofscale– needtorecruitextramoderators.

Problemsofracism,marketing,trolling.(Changesneededtoautomoderator,spamfilter;specialistmoderatorsrecruitedtodealwith

racismandtrolling.)

FIG 26: Subscriber growth on r/ebola. The first upward curve

corresponds with the first cases of Ebola on U.S. soil and the

second, more vertical upward curve, corresponds to the first

cases diagnosed on U.S. soil.

Correlationbetweeneachdoctor’sscorewiththemeanofotherdoctorsandeachdoctor’scorrelationwiththe(scaled)reddit bestscoreordering.

Qualityofmedicalinformationonreddit:assessmentbydoctors

Commentrankingasassessedby27doctors:

Informationquality,completeness,andusefulness(entirethreadsfromchronicconditionsubreddits)

Smallstudyusingassessmentbydoctors:

- Informationinchronicconditionsubreddits mostlyaccurate,complete,useful

- Aminorityofdoctorswereunwillingtorateanyinformationfromadiscussiongrouphighly

Amorerobustfinding,inagreementwithpreviousliterature:

Extensiveactivesearchingforbadorharmfulinformationinsoberdiscussiongroupsfailedtofindany.

Conclusions

• medicaladviceinanonymouslymoderatedreddit forumsisaccurateanduseful,withhighrespectforscientificmedicine

• reddit forumscanhandlesuddengrowthinthenumberofusersand‘contextshift’inthetypesofquestionsasked• callinspecialistvolunteersfromthecommunityofreddit moderators• requestexpertadvicefromsciencesubreddits• caneffectivelypolicediscussionquality(aidedby2Dvoting)

• BUT:amateurs;patchycoverage;legalvacuum.

Policyrecommendations

TheNHSandPublicHealthEnglandshouldincreasetheirsupportandliaisonwithmoderatedinternetforumsandthecommunitiesofvolunteermoderators,toenable:

• moderatedforumsonspecificdiseasesandmedicalconditions

• rapidimplementationof‘beacon’subreddit(s)forcollectivelyintelligentnewsaggregationandquestion-answeringduringapandemic

Theworld’smosteffective‘volunteercommunities’?

Wikipedia – 130,000peoplehaveeditedinthelast30days10,000editorswithspecialpermissions

Reddit – 26,000moderators:typicallyspendonehourperday

StackOverflow – 40,000,000visitorspermonth13,000,000questionsaskedsofar

TripAdvisor – contentprovidedbyreviewers.

Ineachcase,asmallcompany/foundationimplementsa‘gamifiedconversation’

Contentisprovided,moderated,andevaluatedbyahierarchicalcommunityoftensofthousandsofvolunteers.

Whatelseispossible?Couldcomment-boardsinnewspapersandblogsbe