Co-funded by the Horizon 2020 Framework Programme of the European Union

Workshop5:ThePolicingofTrolling,Hate,andLiesOnline

Deliverable2.7

UniversityofWarwick

VersionControlSheet

Title Workshop5Report:Trolling

PreparedBy Dr.KaterinaHadjimatheou

ApprovedBy

VersionNumber

Contact

RevisionHistory:

Version Date SummaryofChanges Initials ChangesMarked

TheresearchleadingtotheseresultshasreceivedfundingfromtheEuropeanUnion’sHorizon2020ResearchandInnovationProgramme,underGrantAgreementno700281.

ContentsExecutiveSummary..................................................................................................................................................i

1. Introduction.......................................................................................................................................................2

1.1 MEDI@4SEC............................................................................................................................................2

1.2 Work Package 2: Implementation: Improved Dialogue, Collaboration andPractices....................................................................................................................................................................2

1.3 Deliverable2.7.......................................................................................................................................3

2. WorkshopSetupandMethod...................................................................................................................4

2.1 WorkshoponTrolling..........................................................................................................................4

2.2 Method.......................................................................................................................................................5

3. IntroductiontoTrolling...............................................................................................................................7

3.1 Keyactorsandstakeholdersinanti-trollingefforts...........................................................7

4. SWOTAnalysis:LawEnforcementAgenciestacklingtrolling.............................................11

4.1 StrengthsandOpportunities.......................................................................................................12

4.2 WeaknessesandThreats...............................................................................................................13

5. RelevantTools,BestPractices,Recommendations....................................................................16

5.1 Relevanttoolsandbestpractices..............................................................................................16

5.2 Actions,tools,andrecommendations.....................................................................................17

6. ConclusionandOutlook............................................................................................................................20

Appendix1 WorkshopAgenda...................................................................................................................21

Appendix2 WorkshopEvaluation............................................................................................................22

i

Executive Summary

Trollingrepresentstheuglysideofsocialmediause.Asmuchassocialmediaconnectspeopleandbringsthemtogether,sotoocanitisolate,alienate,anddemeanpeople.Thelevelsofabuse,hatred,harassment,andbullyingonsocialmediatodayareatanall-timehigh,asareconcernsabouttheeffectsofonlinenastinessonthementalhealthofchildrenin particular. The toxic environment on some social media sites has even begun todiscourage people from expressing legitimate views online or even maintaining apresenceonsocialmedia.Thisworkshopexploredthechallengesandopportunitiesforpublicsecurityprovidersintacklingtrollingonsocialmedia.Itbroughttogetherover50practitioners,specialists,industryrepresentativesandresearchersfrommorethan15EUcountries and combined short, stimulating presentations with intense breakoutdiscussions. This report summarises the insights from that workshop and draws outrecommendationsforkeystakeholdersinthefieldofanti-trolling.

Theworkshop featured sevenpresentations frompolicing, civil society, academia andindustry.Thesewerefollowedbytwobreakoutsessionsforfocuseddiscussionandafinalplenarysessioninwhichkeymessageswerereflecteduponandrefined.

Thebreakout sessionsaddressed each stageofanti-trolling interventions: prevention,disruption,andenforcement,consideringthecurrentroleandpotentialcontributionofkey stakeholders in each of these. The workshop considered both human andtechnologicalmeansoftacklingtrollingonsocialmedia.Theethicalandlegalaspectsofanti-trolling interventions are considered in a separate report, also available on theMEDI@4SECwebsite.

Key stakeholders and actors in efforts to tackle trolling include national andsupranational governments; social media and gaming platforms; law enforcementagencies; schools andparents; victimsof trolling;civil societyorganisations includingNGOs, charities and campaign organisations; and employers in sectors particularlyaffectedbytrolling.ThecurrentroleandpotentialcontributionofthesegroupstotacklingtrollingisoutlinedinSection3ofthisreport.

ASWOT(Strengths,Weaknesses,OpportunitiesandThreats)analysisoftrollingfromtheperspectiveof lawenforcementagencies ispresented inSection4.Thishighlights theimportance of a victim-centred approach to anti-trolling, the need for a coordinatedapproachtoanti-trollingthatrecognizesthemanychallengestoeffectiveprosecutionandthereforetheneedtodeliverbothcriminaljusticeresponsesandotherdisruptiveandpreventivemeasures.

Section 5 puts forward recommendations for future action by LEAs and all otherstakeholders,consideringexamplesofbestpractice,especiallyaroundpartnershipandcooperationbetweenstakeholders.

1. Introduction

1.1 MEDI@4SEC

MEDI@4SECfocusesuponenhancingunderstandingoftheopportunities,challengesandethicalconsiderationofsocialmediauseforpublicsecurity:thegood,thebadandtheugly. The good comprises using social media for problem solving, fighting crime,decreasing fear of crime and increasing the quality of life. The bad is the increase ofdigitisedcriminalityand terrorismwithnewphenomenaemerging throughtheuseofsocialmedia.Theuglycomprisesthegreyareaswheretrolling,cyberbullying,threats,orlive video-sharing of tactical security operations are phenomena to deal with duringincidents.Makinguseofthepossibilitiesthatsocialmediaoffer,includingsmart‘work-arounds’iskey,whilerespectingprivacy,legislation,andethics.Thischangingsituationraisesaseriesofchallengesandpossibilitiesforpublicsecurityplanners.MEDI@4SECwill explore this through a series of communication anddissemination activities thatengageextensivelywitharangeof end-users tobetterunderstandtheusageofsocialmediaforsecurityactivities.MEDI@4SECwillseekabetterunderstandingofhowsocialmedia can, and how social media cannot be used for public security purposes andhighlight ethical, legal and data-protection-related issues and implications. Activitiescentre around six relevant themes: DIY Policing; Everyday security; Riots and massgatherings:Thedarkweb;Trolling;andInnovativemarketsolutions.MEDI@4SECwillfeedinto,supportandinfluencechangesinpolicy-makingandpolicyimplementationinpublic security that can be used by end-users to improve their decision making. Bystructuring our understanding of the impact of social media on public securityapproaches in a user-friendly way MEDI@4SEC will provide an evidence-base androadmapforbetterpolicymakingincluding:bestpracticereports;acatalogueofsocialmedia technologies; recommendations for EU standards; future training options; and,ethicalawarenessraising.

1.2 Work Package 2: Implementation: Improved Dialogue, Collaboration and Practices

Following the results of the first work package, which focused on identifying bestpractices and providing a state-of-the-art overview, the second work package of theMEDI@4SECprojectsaimstodistributethesefindingsandtoengageadiscussionwiththewider community of public security. The work package thereby contributes to theimplementation of socialmedia for public security and fosters an improved dialogue,collaborationandthesharingofexistingpractices.

Whiletheworkpackagealsowill identifyopportunitiesandchallengesinsocialmediause,analyseavailabletechnologiesanddiscusstheneedforstandards,acorecomponentoftheworkpackagetwoisaseriesofworkshops,onededicatedtoeachofthethemesmentionedbefore.

TheworkshoponTrollingthatwereportoninthisdeliverableisthefifthinthisseries.

Interested parties can follow the activities of the project and register for upcomingworkshops on the project website www.media4sec.eu, and in our LinkedIn group:https://www.linkedin.com/groups/12000103.

1.3 Deliverable 2.7

ThisdocumentprovidesresultsfromthePolicingofTrolling,Hate,andLiesOnlineheldinLondonon15thMay.Theworkshopbroughttogetherstakeholdersfromacrossvarioussectors includinglocalauthorities’ representatives,academicresearch, industry, socialmediabusinessandotherauthorities,aswellaslawenforcementagencies,todiscussthechallengesthatsocialmediahascreatedinrespectofonlinehateandtrolling,toexchangeexperience and to think about recommendations for future policy, practice andnetworkingtoaddressthisgrowingareaofconcern.

2. Workshop Setup and Method

2.1 Workshop on Trolling

Theaimof thisworkshopwas tobring togetheranti-trollingactorsandstakeholders-fromthetechindustry,LEAs,NGOsandvictimgroups,andacademia-toshareknowledgeofbestpractices,identifykeyopportunitiesandchallengesandproposeactionsthatcouldbe taken to improve efforts to tackle trolling. A structured discussion session in theafternoonwasdesignedtocollectinsightsaboutthreekeyapproachestoanti-trolling:prevention (stopping trollingoccurring); disruption (stopping trolls in their tracks sotheycannotachievetheiraims);andenforcement(bringingtrollstojustice).

TheworkshoptookplaceonMay15th,2018inLondon,UKattheAlanTuringInstituteintheBritishLibrary.ItwashostedbytheMEDI@4SECcoordinatingpartnertheUniversityofWarwick,andcollaborativelypreparedforbytheentireMEDI@4SECconsortium.

Fig.1Introductiontotheworkshop

Participationwas limited and interested partieswere invited to apply with an emailexplaining theirbackgroundandmotivation for theirparticipation. In total,50peopleparticipated in the workshop, of which 30 were external (not part of theMEDI@4SECproject). Participants came from France, Denmark, Portugal, Germany,Greece,Italy,theNetherlands,Romania,Slovenia,Austria,Switzerland,Sweden,andtheUK. Non MEDI@4SECparticipants included police officers (12), local and centralgovernment(5),NGOs(4),thetechindustry(4)andresearchorganisations(5)

2.2 Method

‘Trolling’ refers to a verybroad categoryof onlinenastiness, ranging fromabuse andbullying,tomischievousbutdisruptivefun-poking,totheconcertedbutcoverteffortsofsome governments to use social media to manipulate information and politicalmovements.ThefocusoftheMEDI@4SECprojectisprimarilyonsocialmediaforpublicsecurityratherthannationalsecurity.Forthisreasontheworkshopfocusedonthekindsoftrollingthatviolatesthemoralandlegalrightsofindividuals,ratherthanthekindoftrollingthatunderminespoliticalmechanismsandnorms.

Thefirsthalfofthedaywasdevotedtopresentationsdesignedtoprovokeandstimulatethinking amongst the audience. In the first of two morning sessions, anti-trollingpractitionersfrompolicingandcivilsocietydiscussedthechallengesandopportunitiesthey face in theirwork. In the second, academics talked about different typologies oftrollingandtheresponsestothem.Themorningsessionclosedwithapresentationofananti-trollingtechnologicalsolutionbysomeonefromthetechindustry.

Fig.2MEDI@4SECTrollingWorkshopAgenda

As with previous MEDI@4SECworkshops, the afternoon was spent discussing thechallenges and opportunities, strengths and weaknesses of current anti-trollingapproaches,andthedifferentrolesandresponsibilitiesofstakeholdersandactors,fromparents and teachers to law enforcement and socialmedia platforms. Two rounds ofdiscussioninsmall,groupseachofwhichincludedaspreadofexpertisewerefollowedbya finalsession inwhich the entiregroupreflectedonandrefined themain take-awaymessagestheworkshophadidentified.

Applicants to the workshop who were not selected to attend and the widerMEDI@4SECvirtualcommunitycouldfollowtheday’seventsonTwitter,whichhavealsobeen captured in a Storify on the workshop website:http://media4sec.eu/workshops/trolling/

Fig3:Breakoutdiscussions

Fig4.Templateforbreakoutdiscussions

Aswithpreviousworkshops,thebreakoutsessionswerefacilitatedbymembersoftheMEDI@4SECconsortium, while other members acted as note-takers. Based on astructured reporting template, note-takers gatheredexpertise and synthesised resultsfromthegroupdiscussions.Asmentionedabove,thesereportingtemplatewasdesignedto collect insights about three key approaches to anti-trolling: prevention (stoppingtrollingoccurring);disruption(stoppingtrollsintheirtrackssotheycannotachievetheiraims);andenforcement(bringingtrollstojustice).

Theremainderofthisreportpresentstheideasandinsightsgatheredfromtheworkshopandthenotesofthebreakoutsessionsinparticular.Thecontentofthereportdoesnotpresentanagreedconsensusamongstallparticipants,butratherreflectsthemainissuesraisedduringtheworkshop.

Keep in mind different kinds of trolling: abuse online; digilantism; cyberbullying; naming-and-shaming; troll bots; political trolling; hate-based trolling; misogyny, etc.

What can and should be done to prevent trolling?

What methods of disruption are available/desirable?

How can action against trolling be enforced?

Which actors should be involved and what responsibilities should they assume?

Which working processes, methods, practices should be in place?

Which tools, capabilities, capacities, knowledge should be in place?

Which opportunities, strengths, and best practices can we build upon?

What are the weaknesses, threats, and barriers and how can we address them?

What are the ethical and legal considerations/ challenges?

3. Introduction to Trolling

Trollingcanbeusefullydefinedasthetargetingofdefamatoryandantagonisticmessagestowards users of social media (Williams and Pearson, 2014:4). Trolling broadlyunderstoodincludes:cyberbullying;cyberhate;cyberstalking;cyberharassment;revengeporn; sextortion;digital vigilantismor ‘digilanism’; namingand shaming; and flaming.Cyberhateistrollingtargetedtowardsthosewithminoritystatus.Theterm‘flaming’isusedbydifferentactorstorefertodifferentactivities:someuseittorefertoextremelyprovocative language, designed to start a fight (Hardaker 2010; Herring et al. 2002;WilliamsandPearson, 2014:11); othersuse it to refer to thosewhopost offensiveorprovocativematerialfortheirownentertainmentorgratification(Bishop,2013).Alltheactivitieslistedhereinvolvethetargetingofindividualsforabuseonline.Revengeporn,sextploitationanddoxinginvolvespecificallyexposingaspectsofsomeone’sprivatelifeonlineandtherebyruiningtheirreputation.Mostoftheseactstakeplacewithinacontextofwiderabuse, includingoffline,butsometrolls, inparticularthosewhobelongtothesociologicallydistinctgroupofindividualswhoself-identifyastrolls,carryoutthevastmajorityoftheirabuseonline.Thelegalstatusofallthetrolling-relatedactsjustlisteddiffersfromjurisdictiontojurisdictionandfromacttoact.

3.1 Key actors and stakeholders in anti-trolling efforts

Duringtheworkshop,participantsdiscussedtheroles,responsibilities, interactionandexpertiseofthedifferentstakeholdersandactorsinvolvedinoraffectedbytrollingandtheattemptstostopit.Theseincludedinparticular:

1. Victims

2. LawEnforcementAgencies(LEAs)

3. NationalandLocalGovernments

4. Parents

5. Schools

6. Employersinsectorsparticularlyaffectedbytrolling

7. NGOsandcharitiessupportingvictimsoftrolling

8. TechIndustry:SocialMediaPlatformProviders;GamingCompanies

Engagementbetweenthesestakeholdersisparticularlyimportantinthecontextofanti-trollingbecausemuchofwhatcanbedonetoprevent,disruptandenforcerulesagainsttrollinginvolvesamixoflegalandpolicysolutions.Forexample,ourworkshopheardthatalargemajorityofvictimsaresatisfiedwiththetakedownofoffendingmaterialanddonotwishtopursueaprosecution.Takedownscaninvolvepoliceactionbuttheycanalsobedonebelowthethresholdoflaw-enforcementinvolvement,bysocialmediacompaniesdirectly or via the work of dedicated NGOs running helplines for victims. Close and

productive engagementbetweenstakeholders is therefore vital to the successofanti-trollingstrategiesandinitiatives.

Thebelowtabledescribesinbrieftherolethatisandcouldbeplayedbyeachofthesestakeholders.Recommendationsforhowstakeholderscoulddevelop,stepup,ormodifytheiractivitiestobettertackletrollingareconsideredbelowinSection5.

Stakeholder Role

Industry: Social MediaPlatform Providers;Gamingproviders

Providepeoplewithaccesstocommunicationsplatformsthat are important to social life and sometimes vital toprofessionallife.Manysocialmediaandevensomegamingsitesareintegraltothesociallivesofpeople,astheyareoneofthemain forums inwhich socialising takesplace. Formanypeople,activeandfrequentengagementonsocialmediasitesisvitaltotheircareers.

Providea forum for trolling:Agreatdeal of trolling takesplaceonsocialmediaplatformsandonlinegamingplatforms.Gamingsitesofferaplacewhereyoungchildrencanviewandengageinanonymousorpseudonymouschatwithteenagersandadults.Theage-appropriatenessofgamesindicatedintheadvertising and product description is determined by thecontentofthegameitself,notbythechatpostedbyusers.Soyoungchildrenplayingthegamemayfindthemselvesexposedtochatthatincludeshateful,racist,orabusivespeech.

Report criminalbehaviour toLEAs: Platformsof all kindshave a legal responsibility to report crimes on their site topolicewhentheybecomeawareofthese;

Design and employ measures to enforce ‘community’norms: Someplatformsemploy a combinationof automaticfilters and human moderators to examine and takedownoffendingmaterial; others encourage users to self-police bymoderatingtheirowncommunitiesandthreads;manyallowuserstoblockandreportothers.

Engagewith each other, with NGOs, andwith LEAs andlegislators todefinestandardsandbringchangesto lawandtopolicy

Schools,Parents Educate children: in ways designed both to prevent themfrom becoming trolls (respect and responsibilities online,raisingawarenessoftheharmsoftrolling)andtohelpthembecomeresilienttotrolling,includingbyreportingit.

Schools have particular insights into the culture of childrenandof cyberbullying. See for example a recent report citingviewsofUKheadteachersonthe impactofcyberbullyingon

mentalhealth.1Schoolsshouldproactivelyshareexpertiseandinsightwithparents,LEAs,NGOsandotheranti-trollingstakeholderstoeducatethemoncurrenttrendsandpractices.

National/LocalGovernments

Designandpasslegislationandregulationtooutlawcriminalkindsofspeechandharassmentanddeterminethelegalresponsibilitiesofplatforms,defineanti-trollingstrategiesandmakefundingavailabletopursuethem.

Puttingpressureonprivatesectortodomore

Facilitatingandlobbyingforbettercollaborationbetweenstakeholders

Designnationalcurriculaforschoolstoincluderesponsibleandsafeuseofsocialmedia.

Lawenforcementagencies

Interpretthelawinordertodefineinpracticethresholdsforcriminalbehaviour

Enablereportingofcrimeandinvestigatereportsofcrime,engagewithvictimsandprosecuteoffenders.

EngagewithNGOsthatrefercrimesupwardstopolice

Engagewith socialmedia and other platforms on whichtrolling occurs to agree thresholds, reporting mechanisms,takedowns of offending material, and methods foridentificationofoffenders.

Victims ReporttrollingtoNGOs,socialmediaplatformsand/orpolice.

Definewaysofbuildingresiliencetotrollingandsharetheirknowledgewithothers.

Decidewhethertopursueprosecutionaftertakedown(andthereforeimpactontherateofprosecutionsandimpactontrolls)

Engageincounter-speechtodiscouragetrollsandbuildsocialcapital.

Employersofpeopletrolledforreasonstodowiththeirprofessionalsocialmediapresence

Exposeemployeestotrollingintheworkplaceorbecauseoftheirwork(e.g.journalists)

Informemployeesabouttherisksofsocialmediauseandofbeingtrolledonsocialmediabecauseoftheirwork

1 See, for example, this recent report citing concerns of headteachers in the UK about the effects of cyberbullying on children. ‘Social media sites are damaging children’s mental health, headteachers warn’ The Independent Newspaper 9 March 2018, https://www.independent.co.uk/news/education/education-news/headteachers-social-media-children-mental-health-school-association-college-a8246456.html

Upskillandeducateemployeesabouthowtoprotectthemselvesfromtrolling,reportit,andbuildresiliencetoit.

Supportemployeesbothpsychologicallyandinpracticaltermsifandwhentrollingoccurs

NGOsandcharities Provideachannelforvictimstoreporttrolling

Providepsychological,practical,andlegalsupporttovictims.

Monitorandresearchtrollingtrendstocampaignandinformfuturepolicy

Interfacebilaterallyandmultilaterallywithsitesthathosttrollingmaterial(e.g.pornsites)toagreemodusoperandiontakedowns(e.g.ofrevengeporn)

Engagewithallotherstakeholdersmentionedabovetohelpthemdotheirworkbetter

Techindustry Developautomatedtoolsforidentifying,flagging,blocking,removingdifferentkindsoftrollsanddifferentkindsofhatespeechorabuse,reducingneedforhumanstotakeondifficultmoderationwork.

We return to the role of stakeholders in Section 5 where we provide specificrecommendations for each group. First, we focus in on the role of one particularstakeholder group, namely Law Enforcement, to consider the particular strengths,weaknesses, opportunities and threats facing their role in tackling trolling. Lawenforcement are the largest single stakeholder group represented in theMEDI@4SECcommunity. Performing a SWOTanalysis from theirperspective also allowsus to seemore clearly the limits and constraints on their role as anti-trolling actors, and theimportanceofothernon-criminal-justiceinterventions.

4. SWOT Analysis: Law Enforcement Agencies tackling trolling

Inthefollowing,wedescribetheinternalStrengthsandWeaknesses,aswellasexternalOpportunities and Threads (SWOT) for law enforcement agencies (LEAs) looking totackletrolling.TheSWOTanalysisisaninstrumentdevelopedforstructuredsituationalassessmentandplanning.Itiscommonlyusedinvariousorganizationalsettings.ThecoreideaofSWOTistolookatanissuefrombothaninternalandanexternalperspective,understanding the internal strengths and weaknesses of an organisation, as well asexternalopportunitiesandthreats,posedby thecontextorenvironment inwhich theorganisationoperates.Thegoalistodevelopinsightsintohoworganisationscanbenefitfromstrengthsandopportunitieswhilemitigatingweaknessesandthreats.

AsthisanalysisiscreatedformultipleLEAsthatvarysignificantlyintheirsocialmediatechnologyadaptation,wefindcaseswherecertainaspectsareseeminglycontractingandarelistedbothasastrengthandweakness,dependingonthestateofimplementation.Therefore,wealsodonotclaimthatallourfindingsapplytoallLEAs.Thepointsthatwemake,nevertheless,shouldhelpLEAstoassesstheirsituation.

ThetablebelowsummarisestheSWOTanalysis(Table4.1),allaspectsaredetailedinthefollowing.

SWOTANALYSIS:LAWENFORCEMENTAGENCIESTACKLINGTROLLINGSTRENGTHSS1ADVERTISINGPOLICESUCCESSESFORDETERRENCEANDREASSURANCES2:LEARNINGFROMVICTIMS

S3:BUILDINGONEXISTINGVICTIM-CENTREDAPPROACHES

S4:BUILDINGONEXISTINGCOMMUNITYPOLICINGRELATIONSHIPS

S5:SHAPINGLEGALFRAMEWORK

WEAKNESSESW1:LACKOFSOCIALMEDIAANDGAMINGKNOWLEDGE,TRAININGANDEXPERIENCE

W2:BADPOLICEUSEOFSOCIALMEDIA/TROLLINGCULTUREINLEAs

W3:LACKOFESTABLISHEDPROCESSESFORPRIORITISINGCASES

W4:POORDECISIONSONWHOTOPROSEUCTEUNDERMINESCONFIDENCE

W5:LONGPROSECUTIONTIMELINES

OPPORTUNITIESO1:IMPROVEMENTSINDE-ANONYMISATION

O2:AUTOMATEDSYSTEMS

O3:WORKOFNGOsCANHELPREACHVICTIMSRELUCTANTTOAPPROACHPOLICE

O4:INCREASINGENGAGEMENTOFSOCIALMEDIA/GAMINGPLATFORMSWITHSTATEAGENCIES

THREATST1:LACKOFAMATURELEGALFRAMEWORK:

T2:RELUCTANCEOFSOMEGAMINGANDSOCIALMEDIASITESTOTAKERESPONSIBILITY

T3:COUNTER-SPEECHCANFUELFURTHERTROLLING

T4:VICTIMSREFUSINGTOPROSECUTECANUNDERMINEDETERRENCE

T5:THEGENERATIONGAP

T6:DIFFICULTIDENTIFICATIONOFTROLLS.

T7:THE‘DISINHIBITIONEFFECT’

T8:LACKOFINTERNATONALCOOPERATION

4.1 Strengths and Opportunities

ThoughEULEAsdivergewidelyintheirapproachtohatecrimeandharassmentaswellas in their confidence and experience inpolicingonline crimes,manyalready have anumberofstrengthswithintheirorganisationswhichtheycouldbuilduponusefullytoimprovethewayinwhichtheytackletrolling.ThebelowsuggeststhatLEAscandevelopthesestrengthsinwaysthatwillhelpdetertrolling,improveunderstandingofthecrimeand its effects, andprovide amore responsive andeffective approach todealingwithvictims,thuspromotingconfidenceinthepolicingofabuseonline.Therearealsoarangeof opportunities presented by the context in which police pursue their anti-trollingagendas,chieflytodowiththepotentialofnewtechnologiesandofcollaborationwithkeypartnersandstakeholders.

S1:ADVERTISINGPOLICESUCCESSESFORDETERRENCEANDREASSURANCE:Policecanbehighlyeffectiveatpublicisingsuccesses.TrollscaneasilycometofeelthattheInternetisasafespaceforthemtopursuetheirharmfulactivities.Theyoftenwanttoprovokeareactionamongstvictimsandreceivesatisfactionfrombeingblocked.Buttheydon’twanttobearrestedandprosecuted.LEAscanusesocialmediatoadvertisetheirsuccessesinorder to deter trolls. This may also reassure victims and the public that police takeseriouslytrollingcrimesandthattrollscanfaceseriousconsequencesfortheiractions.Reassuranceofthiskindisimportantforconfidenceinpolicing.

S2: LEARN FROM VICTIMS: by actively soliciting their views on the criminal justiceprocess, and by involving them and their stories in trainings for police asspeakers/”experts”.Thereisresearchsuggestingthatpolicehavelackedawarenessoftheharmsoftrolling,leadingthemtobedismissiveofvictimsandtheirreports(HerMajesty’sInspectorateofConstabulary,2015;CitroninThinkProgress,2014).Policecancapitaliseon their access to victims to draw on their experiences to improve education andawarenessamongststaff.

S3:BUILDONVICTIM-CENTREDAPPROACHES:alreadydevelopedbypoliceforuseinotherareastoimprovehandlingofvictimsandreducenegativepsychologicalimpactonthem. Police in many countries are already developing victim-centred approaches topolicingforuseinrelationtocrimessuchasdomesticabuseandhumantrafficking.Many

alsohavededicatedspecialistswhoareexpertsinworkingwithchildvictimsofcrime.Theseexistingskillsandprocesseshelptoensurethatthepursuitofcriminalsdoesnotcomeattheexpenseofvictims.Policecantransferskillsandknowledgebetweentheseandhatecrime/cyberteamsinrecognitionthatvictimisationofothersiscoretotrollingandthatthoseaffectedaresometimesvulnerable.

S4:BUILDONEXISTINGCOMMUNITYPOLICINGRELATIONSHIPS:especiallywithyoungpeople,totacklechallengesonline,canhelpbridgethegenerationgapandincreasetrustinpoliceandbuildculturalknowledgeamongstpoliceofsocialmediaandgaming.Policecan build police/community trust by negotiating, rather than assuming, a commonunderstandingofhatecrime,especiallywherethelawisunclear(Masonetal,2015).

O1:IMPROVEMENTSINDE-ANONYMISATION:Techpossibilitiesforde-anonymisationofcommunicationsonlineleavefewerplacesfortrollstohide,thoughtheseoptionsareexpensiveandwouldonlybereservedfortheworstkindofoffenders.

O2:AUTOMATEDSYSTEMS:forflagging,blocking,takingdownmaterialarepromisingwaysofpreventingvictimisationandreducingtheburdenofcasesreferredtopolice.Themore that is done by platforms on which trolling occurs, the better from an LEAperspective,becausepolicemonitoringofsocialmedianetworksisnotonlyexpensivebut legally problematic, raising data protection and human rights issues that areexaminedinmoredepthinD4.5ReportonEthicalandLegalIssuesinPolicingTrollingonSocialMedia.

O3:WORKOFNGOsCANHELPREACHVICTIMSRELUCTANTTOAPPROACHPOLICE:forexample, those whomight be concerned about their immigration status or whomaysimplynotfeelconfidentdealingwithpolice.CollaborationwithandsupportforNGOscanhelpachievecriminaljusticeforthesevictimswithoutincreasingtheresourceburdenonpolice.Forexample,theUK’sMETPolicehaveanofficialpartnershipwithStopHateUKaspartoftheirflagshipOnlineHateCrimeHub.

O4: INCREASINGENGAGEMENTOFPLATFORMSWITHSTATEAGENCIES canbebuiltupon to further common understandings of key trolling offences, e.g. the EuropeanCommission’s2016CodeofConductonCounteringIllegalHateSpeechOnline,draftedincollaborationwithmainstreamproviders.

4.2 Weaknesses and Threats

W1:LACKOFSOCIALMEDIAANDGAMINGKNOWLEDGE,TRAININGANDEXPERIENCE:NotallLEAsareawareofsocialmediatechnologies,notallofthemhavethebackground,knowledgeandexperiencetopolicesocialmedia.Insomeplaces,evennewrecruitsdonotreceivesocialmediatraining.Olderpersonnelwhonolongercan/wanttoworkonthestreetsarestaffingtheofficesdoingdeskresearch,andtheyareoftentheleastwell-versedinsocialmedia.

W2: BAD POLICE USE OF SOCIALMEDIA/TROLLING CULTURE IN LEAs: As researchshows,policeofficers themselvesaresometimes trollsorbehave irresponsiblyonline.Bettertraininganddirectaddressingofthisissue,forexamplethoughincorporationof

socialmedia-focusedappliedscenariosinCodeofEthicstrainingcouldhelpbuildabetteronlinecultureamongstLEAs.

W3:LACKOFESTABLISHEDPROCESSESFORPRIORITISINGCASES:LEAsoftendonothaveaclearframeworkthatwouldsupportthemindecidingwhichcasestopursuewithwhatlevelofresources.Forexample,theUKlawdefinesillegalityasinvolvingspeechthatis ‘grosslyoffensive’but there isn’tawell-enoughestablishedinterpretationof thisbypolice.

W4: POOR DECISIONS ON WHO TO PROSEUCTE UNDERMINES CONFIDENCE: Often,police(orthemedia)makeanexampleofprosecutedcasesinvolvingindividual‘idiots’ratherthanthemoreprofessionalororganizedtrolls,whotendtogetawaywithwhattheydo.Thiscanundermineconfidenceinpolicing.

W5:LONGPROSECUTIONTIMELINES:candetervictimsfrompursuingandstickingwithprosecution.Evenarecentcasecitedasasuccessfulexampleofawomansuinganex-partnerforrevengeporntookfourandahalfyearstoresolve.2Quicker,morestreamlinedkindsofenforcementwouldencouragevictimstopursueprosecutionsandgeteverythingthecriminaljusticesystemhastooffer.

T1:LACKOFAMATURELEGALFRAMEWORK:Asbothonlineharassmentandhatecrimehave only been recognized as forms of criminality relatively recently, policing is notbackedupbyamaturelegalframeworkofthekindthatwouldenableLEAstodetermineconfidently what types of trolling are prosecutable and which are not. Some LEAdiscussantsfeltthatthethresholdiscurrentlytoohigh,leavingpoliceunabletorespondtoagreatdealofnastinessonline.

T2: RELUCTANCE OF SOME GAMING AND SOCIAL MEDIA SITES TO TAKERESPONSIBILITY: When the threshold for criminalisation is very high, enforcementmechanisms for those kinds of online nastiness that fall short of crime take on extraimportance.Butworkshopparticipantsrepeatedby-now-familiarcomplaintsthatsomeSocialMediaplatformsandsomegamingcompaniesinparticulararenottakingproactivemeasurestoestablishrulesofconductontheirplatform,toeducateusersandtomonitorand moderate content, let alone report hatespeech to the police. Even those withreportingmechanismshavelongbeencriticisedforbeingtooslowtotakedownoffensiveandhatefulmaterial,sothatbythetimetheyhavethedamagehasalreadybeendone(Phillips,2015:72).

T3: COUNTER-SPEECH CAN FUEL FURTHER TROLLING: as it encourages people toengagewithtrollsandfeedthem,andalsotendstodescendintoabusivenessitself.Ourworkshopheardreportsofarisein ‘digitalvigilantism’or ‘digilantism’:citizenstakingproactivemeasurestonameandshametrollsorotherperceivedwrongdoersonline,inways that are themselves questionable at best. But we also heard that- when usedtactically-counter-speechcanalsohelpvictimsfeellessalienatedandimprovethetoneofdebateonline(RAND,2016;CentrefortheAnalysisofSocialMedia,2015).

2 ‘YouTube star wins damages in landmark UK 'revenge porn' case’ The Guardian Newspaper, 17 Jan, 2018. https://www.theguardian.com/technology/2018/jan/17/youtube-star-chrissy-chambers-wins-damages-in-landmark-uk-revenge-porn-case

T4:VICTIMSREFUSINGTOPROSECUTECANUNDERMINEDETERRENCE:Weheardfromourpractitionerpresentersthatvictimscontactingthepoliceabouttrollingtendtodropout as soonas the content is removed.This leaves trolls unpunished, undeterredandhighly likelytogoon tovictimiseothers in the future.While it isunderstandable thatvictimswouldnotwanttogivetrollsanymoreoftheirtimeandenergy,thereisariskthatthisisunderminingdeterrence.Italsoencouragesdigitalvigilantestostepintofillthepunitivegap.Othermeansofbringingtrollstojusticethatarenottooonerousforvictims,suchasclassactionsuits,mightbeexploredasanalternative.

T5:THEGENERATIONGAP:andtheslowuptakeintechnologytakentogethermeanthatpolicewillalwaystendtolagbehindcurrenttrends.

T6: IDENTIFICATION OF CRIMINAL TROLLS: is still a real challenge with the moresophisticated,organisedoffendersmovingfromVPNtoVPN.Othermethodsofidentifyingindividualsonline,suchastheuseofforensiclinguisticsoftwarethatcreatesaprofileofa troll’s speechpatterns and tries todetect themelsewhereon theweb on lesswell-protectedsites,couldbepotentiallyusefulhere.Butdecisionsmustbemadeaboutwhichcriminaltrollstopursueaggressivelyandwhichtotargetfordisruption.

T7:THE‘DISINHIBITIONEFFECT’:appearstobegrowing,leadingtoamoretoxicclimateandnormalisationoftrollingonline.Thischallengespoliceeffortstodeterminewhatisprosecutableandwhatisnot,notleastbecausetheseoftenrelyonpoliceinterpretationsof highly variable and relative terms such as ‘gross offensiveness’. The apparentlyincreasinguseofoffensivespeechbypoliticiansandotherauthority figures insocietyfurtherinfluencesthewaybenchmarksofoffensivenessshift.

T8: LACKOF INTERNATONALCOOPERATIONand legal harmonizationon combattingtrollingbehaviouracrossbordersleadstoimpunityfortrolls.Thisiscompoundedbythefactthattheevidenceofcrimescommittedagainstapersoninonejurisdictionareheldonaserverinanotherjurisdiction(mostoftentheUSA).

5. Relevant Tools, Best Practices, Recommendations

5.1 Relevant tools and best practices

Theworkshopsawsomepresentationsofoutstandinggoodpracticeinthecivilsocietysector, such as the Revenge PornHelpline3which helps victims report and takedownmaterialandhasdevelopedcloserelationshipswithpornsitestoenablethistohappenquicker.Thisandotherhelplines,includingRomania’sSaferInternetCentre4helpline,aretrustedflaggerswhosereportscanbetakeningoodfaithbyplatforms.Thelatterhasalsoemployedatrainedpsychologisttomanthehelplinetooffersupporttochildrenwhocallandothervictimsofcrime.Themoreeffectivetheworkofsuchagenciesis,thelessLEAsareburdenedwithanoverwhelmingvolumeofcases.

Alsodiscussedwereguidelinesforemployingcounter-speechinwaysthatareeffectiveingivingvoiceandsupporttovictimsandwhichdonotfeedthetrolls.TheworkoftheInstituteofStrategicDialogue5inthisareaisanexampleofarelevanttoolinthisrespect.

LEAshave alsodevelopedtheir own reporting toolswhich enable victimsnot only toreporttrollingcrimesbutalsotogatherevidenceofthemandpassthisinausableformtopolice.TheMETpolice’sOnlineHateCrimeHubisanexampleofthis.Theyarealsosupporting research that aims to correlate hate crime online with events offline andtherefore prepare better for a coordinated LEA and partner response to predictablespikesinboth.

Aswithotherworkshops, theneed for internationalcooperation to identifytrollsandbringthemtojusticewasemphasized.MLAT-mutuallegalassistancetreaties-whichaideffortstopolicetheinternetacrossborders,andInterpol’sGlobalComplexforInnovation,which builds relationships andmutual understanding between national police forces,werementionedasexamplesofrelevanttoolsandgoodpracticeinthisarea.

Collaborationandpartnershipbetweenrelevantstakeholdersandactorswasidentifiedasparticularlyimportanttotheeffectivecounteringoftrollingonline,inpartbecausesomuchof thenastiness thatoccurs fallsbelowthethreshold forcriminalization, inpartbecausethevolumeofcrimeissohighthatpolicecannotcopealone.Examplesofgoodpractice building common understandings of key trolling types such as Hate Speechinclude theEuropeanCommission’s2016CodeofConduct onCountering IllegalHateSpeechOnline6,draftedincollaborationwithmainstreamproviders.

Socialmediaplatformsarealsoincreasinglydevelopingtheirowntoolstocountertrollingthrough wholesale blocking of individuals, introduction of user-calibrated settings,intense moderation, reporting tools, and automatic takedowns. These are all useful

3 https://revengepornhelpline.org.uk/ 4 http://oradenet.salvaticopiii.ro/ 5 Counter-Conversations, 2018, http://www.isdglobal.org/wp-content/uploads/2018/03/Counter-Conversations_FINAL.pdf 6 http://ec.europa.eu/newsroom/just/item-detail.cfm?item_id=54300

methodsofprotectingpeopleandreducingtheamountofnastinessonline,andinthiswaycombattingthedegradationofonlinepublicspaces.

Specific actions, tools, and recommendations identified from the discussion are listedbelow.

5.2 Actions, tools, and recommendations

Thissectionliststhekeyactions,toolsandrecommendationsthatcanbetakenawayfromthe workshop. One of the key issues emerging from the workshop is the need forcollaboration but also proactive intervention from many actors. The belowrecommendationsarethereforeorganizedbystakeholder,asthishelpstohighlightthedifferentbutequallyvitalrolesplayedbyallanti-trollingactors.

Actions,ToolsandRecommendationsforLEAs

Shouldincreasetheirvisibilityinthedigitalspacetoincreasetrustandenforcelawfulbehaviour.

Should proactively advertise their successes in terms of prosecutions to send amessage to trolls that they arenot safeonline and to victims that their experiencesmatterandaretakenseriously

Shouldtakeproactivemeasurestocreateandmaintainacultureofrespectamongstpoliceratherthandisdainforthepublicasthiscanleadtobadpolicebehaviourandeventrollingbypolice

Should avoid naming-and-shaming of criminals online as this tacitly sanctionssimilarbehaviourbythepublicintheformofdigilantismthatcanturnveryuglyandevenresultinillegalbehaviourorsuicide

Should enable knowledgeable young recruits to educate the old and embed socialmediatrainingintorecruitmentandupskillingprogrammes.

Should use existing community relations with young people and interaction withvictims to learn from them about social media use and abuse, negotiatingunderstandingsofhatecrime

Shouldeducate officers onwhat trolling is, the harms for victims, thementality oftrolls,butalsoininvestigativemethods

Shouldeducatecitizensonhowtoprotectthemselves,whatthelawrequires,andteachvictimshowtocollectevidenceoftheirownvictimisationtoreportmoresuccessfully.

Shouldtransferknowledgeonvictim-centredapproachestopolicing fromotherfieldstopolicingoftrolling

Should collaborate with platforms to ensure continuity between prioritisationimplementedthereandlegalrequirements

ShouldcontinuetopartnerwithrelevantNGOsandcivilsocietygroupstoprovidejusticeresponsestovictimsoftrollingthatdoesnotmeetthethresholdforcriminalityorwhich

Stakeholder Recommendedactions

Industry:SocialMedia(SM)PlatformProviders;Gamingproviders

Continue to develop automated systems for flagging,blocking,removingcontent.

SMplatformsshouldidentifytrustedflaggers(publicuserswhoflagcontentconsistently)andgivethemcontenttocheck

SM companies should bemore transparent about theirrulesofengagement

Gamingcompaniesshouldputgamificationtechniquestousetoteachtheirusershowtorespectthecommunityrulesonanyspecificplatform.

Require stronger certification of user accounts so as toremovepossibilitiesforanonymity.

Cooperatetocollectdatainordertofacilitateprosecution

SMcompaniesshouldestablishasharedcodeofconduct(forcontentanalysis,contentremoval,reportingtoLEAs)

SM companies should improve working conditions formoderatorstopromote‘responsiblemoderation’.

SM companies should provide end-users with bettercontroltoblockcontentand/orusersandtoopt-intoseecontentflaggedasoffensive

Schools Runonlinerespectandresiliencesessionsforveryyoung

children,astheyareexposedtotrollingongamingsites.

Runsessionsforparents,asschoolshaveuniqueaccesstothishard-to-reachstakeholdergroup.

Supranational/National/LocalGovernments

Designandpass legislationandregulation to:a)outlawthreatstosharesensitivemateriale.g.revengeporn,andnotonlyactualsharingandb)applytogamingspaces.

Designnationalcurriculaforschoolstoincluderesponsibleandsafeuseofsocialmedia.

Establish an EU standard for social media companieswantingtooperateintheEU

Fundbetterresearchintocausesandpsychologicalprofilesof trolls, tobetter informanti-trollingpreventiveeducationandrehabilitation

Lawenforcementagencies

Interpret the law inordertodefine inpractice thresholdsforcriminalbehaviour

Enablereportingofcrimeandinvestigatereportsofcrime,engagewithvictimsandprosecuteoffenders.

EngagewithNGOsthatrefercrimesupwardstopolice

Engagewithsocialmediaandotherplatformsonwhichtrolling occurs to agree thresholds, reportingmechanisms,takedowns of offending material, and methods foridentificationofoffenders.

Victims Candomoretohelpprotectothersbysupportingprosecutionsandbecominginvolvedintrainingofpoliceandothersas‘experts’

Shouldnotusecounter-speechtodescendintoabusivenessthemselves.

Employersofpeopletrolledforreasonstodowiththeirprofessionalsocialmediapresence

Shouldadoptadutyofcaretostaffsimilartowhatisalreadythereintermsofoccupationalhazardsandhealthandsafety,includinginparticularthefollowingactions:

• Informemployeesabouttherisksofsocialmediauseandofbeingtrolledonsocialmediabecauseoftheirwork

• Upskillandeducateemployeesabouthowtoprotectthemselvesfromtrolling,reportit,andbuildresiliencetoit.

• Supportemployeesbothpsychologicallyandinpracticaltermsifandwhentrollingoccurs

• Investintechthathelpsemployeestoblocktrolls

NGOsandcharities Developguidanceonusingcounter-speechinwaysthatarehelpfulandproductiveanddonotcontributefurthertoatoxicenvironmentonline.

Buildongoodpracticesofferingtrainedpsychologiststorespondtohelplinereports

Buildbilateralrelationshipswithsitestofacilitatetakedownsandblockingofusers

6. Conclusion and Outlook

Trollingis,unfortunately,becomingformoreandmorepeopleadailyhazardofonlinecommunications. Quite apart from the premeditated, targeted trolling used bysophisticatedabusersandcriminals,moreandmorepeopleareengaginginhatefulorabusivespeechasthisisnormalizedbyanincreasinglytoxicatmosphereonline.Whilepolice are right to insist that their role in anti-trolling should be limited to tacklingcriminalbehaviour,thereisstillasignificantlackofclarityaboutwherethelinebetweenanti-socialandcriminalbehaviorshould lie.This inevitablyresults inpolicehaving tofieldhugenumbersofreportsoftrolling,andfailingtosatisfysomevictims.Andthatiswhyeffectivecollaborationbetweenawiderangeofanti-trollingactors,includingcivilsociety partners and social media platforms is so vital to building an effective andcoherentresponse.Governmentandnationallawenforcementauthoritieshaveakeyroletoplayheretoo,becausetheycanhelptocoordinateactorsinacoherentanti-trollingstrategy.Theoutputsfromthisreportwouldusefullyfeedintoanysuchinitiative.

Formanydelegatesattheworkshopthiswasthefirsttimetheyhavehadtheopportunitytoexplorethistopicwithstakeholdersfromotherfields.Asthefeedback(Appendix2)illustrates, the opportunity to actively engage with such a wide group and exchangeexperiencesand ideas isverywelcomedbyall.Moving forwardsitwillbecritical thatsuchaforumismaintainedandinparticularthateventssuchasthiscontinuetoenableface-to-faceinteractionswhichhelpembednewandgrowingnetworkstoassistindealingwiththeseissues.

The workshop helped to illuminate the strengths, weaknesses, responsibility for andproper place of different kinds of anti-trolling interventions. Taking down offendingmaterial is not easy, but it often satisfies victims and it is certainly far easier thanmounting an effective prosecution. As a result, much ant-trolling activity is currentlydisruptive,butanti-trollinglaws(againsthatespeech,harassment,threatsetc)areoftenleftunenforced.Thisisachallengefordeterrenceandforeffortstoreducetheamountoftrollingonline,whenallitdoesisdisplacethetrollstootherplacesandothervictims.

Similarly,counter-speechmaycreatesolidarityamongstvictimsandothersympatheticgroupsagainsttrolls,butsuchgroupscanthemselvesturnnasty,andoftentheirattentionispreciselywhattrollsaimtoprovoke.Fightingtrollswithmethodsshortofprosecutionisdifficultwhentrollsaimtoprovokeandroutinelyfindwaysofweaponisingtoolsusedagainstthem,turningthembackonthosewhoaimtostopthem.

Perhapsmorethananyoftheothertopicsconsideredinthisseriesofworkshops,trollingisaphenomenonthatreflectsonsocietyasawholeandwhichitistheresponsibilityofeveryonetotackle.Lawenforcementagenciesplayacrucialroleinanti-trollingefforts,but only in a much broader web of collaborations (sometimes with unexpected andunusual partners such as campaign groups or porn sites) partnerships and proactiveindependent activitiesby a rangeof actors fromparents to schools andemployers togamingcompanies.

Appendix 1 Workshop Agenda

Appendix 2 Workshop Evaluation

Overalltheworkshopwaspositivelyreceived.Delegatesindicatedthatithasbeenaveryeffectiveuseoftheirtime,enablingthemtonetworkwithotherstakeholderswithwhomtheydonotnormallyinteract.Thiswasfurtherreflectedinfeedbackaboutthebreakoutgroupswherethebreadthofinputwasappreciatedbymostdelegates.Therewaspositivefeedbackaboutthespeakerswhocontributedtotheevent.Thebreadthofsubjectmatterwasgenerallywellreceivedalthoughafewdelegatesraisedtheneedforfurthertopicstobeshowcasedduringthesesessions.

The immediate impact of thisworkshop ispositive. The responses showed thatmostdelegateshadimprovedtheirunderstandingandknowledgeofthestateoftheartinthisfield.Most importantly a significantnumber of delegates indicated that they hadmetpeoplefromoutsidetheirexistingnetworkswhomightformthebasisofnewprofessionalcontacts.Aclearmessageemergingfromdelegateswastheneedformoreoftheseevents.The time was too limited in which to discuss everything in such a varied field anddelegatesvaluedtheopportunitytotaketimeoutoftheirjobs,meetothersstakeholderswithwhomtheydonotnormallyworkandtobuildnewprofessionalnetworks.