Upload
others
View
6
Download
0
Embed Size (px)
Citation preview
©CopyrightJASSS
SimonDAngusandBehroozHassani-Mahmooei(2015)
MonashUniversity,Australia
"Anarchy"Reigns:AQuantitativeAnalysisofAgent-BasedModellingPublicationPracticesinJASSS,2001-2012
JournalofArtificialSocietiesandSocialSimulation 18(4)16<http://jasss.soc.surrey.ac.uk/18/4/16.html>DOI:10.18564/jasss.2952
Received:03-Feb-2015Accepted:17-Sep-2015Published:31-Oct-2015
AbstractAgentBasedModelling(ABM),apromisingscientifictoolset,hasreceivedcriticismfromsome,inpart,duetoaclaimedlackofscientificrigour,especiallyinthecommunicationofitsmethodsandresults.Totesttheveracityoftheseclaims,weconductastructuredanalysisofover900scientificobjects(figures,tables,orequations)thatarosefrom128ABMpaperspublishedintheJournalofArtificialSocietiesandSocialSimulation(JASSS),duringtheperiod2001to2012inclusive.Regrettably,wefindconsiderableevidenceinsupportofthedetractorsofABMasascientificenterprise:elementaryplottingattributesareleftoffmoreoftenthannot;basicinformationsuchasthenumberofreplicatesorthebasisbehindaparticularstatisticarenotincluded;andfew,ifany,establishedmethodologicalcommunicationstandardsareapparent.Inshort,'anarchyreigns'.WhilstthestudywasconfinedonlytoABMpapersofJASSS,weconcludethatiftheABMcommunitywishesitsapproachtobeacceptedfurtherafield,authors,reviewers,andeditorsshouldtaketheresultsofourworkasawake-upcall.
Keywords:AgentBasedModelling,SocialSciences,Simulation,Publishing
Introduction
"Whilethetheoreticalandexperimentalfoundationsofagent-basedsystemsarebecomingincreasinglywellunderstood,comparativelylittleefforthasbeendevotedtounderstandingthepragmaticsof(multi-)agentsystemsdevelopment—theeverydayrealityofcarryingoutanagent-baseddevelopmentproject.Asaresult,agentsystemdevelopersareneedlesslyrepeatingthesamemistakes,withtheresultthat,atbest,resourcesarewasted—atworst,projectsfail."
"Itiswidelyacknowledgedthat…agent-basedmodels,canplayanimportantroleinfosteringunderstandingofthedynamicsofcomplexsystems.…However,current[agent-based]modellingpracticehastwosubstantialshortcomings:(1)Thereasoningbehindthechoiceofacertainhumandecisionmodelisoftennotwelldocumented;insufficientempiricalortheoreticalfoundationsaregiven;orthedecisionmodelisonlyassumedonanad-hocbasis…(2)Oftenthemodelisnotdescribedinatransparentmanner(clearandcomplete)thatwouldallowforreproducibilityandfacilitatethecommunicationofthemodelanditsresults."
1.1 ThetwoquotationsaboveconciselydescribeatragedyinthestoriedhistoryofAgent-BasedModelling(ABM).Thetragedybeingthat,whilstdescribingessentiallythesameparadox–thepromiseofABMapproachesinthesocialsciencesjuxtaposedagainstthelackofwelldevelopedpracticesinABMscience–thefirstquote(Wooldridge&Jennings1998)predatesthesecond(Mülleretal.2013)by15years.[1]
1.2 Ofcourse,duringthisperiod–onewhichhasseenanewglobalfinancialcrisis(GFC),theemergenceofseveralpotentiallypandemicinfectiousdiseases,andtheriseandriseofnetwork-basedsocialmediaplatforms–thechampionsofABMmethodologiesfromallrelevantfieldshavere-emphasised,inprominentpublishingplatforms,theneedfor,andrenewedrelevanceof,ABMmethods:DoyneFarmerandDuncanFoley(2009),MarkBuchanan(2009),andJean-PhilippeBouchard(2008)allleveragedthe'meltdown'ofthe2007–
http://jasss.soc.surrey.ac.uk/18/4/16.html 1 25/01/2016
2009GFCtocallforABMsinEconomicsatNature;whilstJoshuaEpstein,alsowritinginNature,pointedtothenetworkcomplexityinheritintheemergingH1N1outbreakof2009topointouthowsuitedABMsaretomodellinginfectiousdiseases(Epstein2009).Inshort,thepromiseofthemethodologyseems,duringthisperiod,aliveandwell.
1.3 Meanwhiletimelyandcompellingarticlesofthegenericform"FieldX,meetABM…,"werewrittenfor,amongstothers,the'humansystems'modellingcommunitybyEricBonabeauintheUSflagship,ProceedingsoftheNationalAcademyofSciences(Bonabeau2002),themind—actionsocio-economiccommunitybyNigelGilbertandPietroTerna(2000)inthe(thenbrandnew)MindandSocietyjournal,andforthesociologycommunitybyMichaelMacyandRobertWillerintheAnnualReviewofSociology(2002),eacharrivingatthestartofthiscrucialperiod.
1.4 Intermsofresults,afewABMpractitionershavefoundareceptiveaudiencefortheirinsightfulworks.Creative,unashamedlyABM,papershavefoundtheirwayintotopfield(e.g.intheEconomicsciences:AmericanEconomicReview(albeit,Papers&ProceedingsoftheAEA),JournalofEconomicDynamicsandControl,JournalofEconomicBehaviourandOrganisation)andgeneralist(e.g.Science)journalsduringthisperiod(Geanakoplosetal.2012;Howitt&Clower2000;Dosietal.2010;Limetal.2007).
1.5 However,outsideofthesefewhigh-pointsoftheABMsocialsciences,ABMstudiesingeneralhavefaredratherlesswell.Leombrunietal.'s((2005)surveyofthetop20Economicand10Socialsciencesjournalsfoundonlyahandfulofpublishedstudies(7intheformer,11inthelatter)uptothatpointintimewhichusedanABMmethodology.Recentpublishingpracticesseemlittledifferent.
1.6 VariousexplanationsaregivenforABM'sdifficultreceptioninfieldjournals.Chiefamongsttheseseemtobeaperceivedlackofwhatmightbecalled'intuitivetransparency'relativetotheclosed-form,deductive,toolsetsavailable.JoshuaMEpstein(2006),inhisclassiccontributiontotheHandbookofComputationalEconomics(vol2),writes(ch34),
"Therealreasonsomemathematicalsocialscientistsdon'tlikecomputationalagent-basedmodellingisnotthattheapproachisempiricallyweak(innotableareas,it'sempiricallystrongerthantheneoclassicalapproach).It'sthatitisn'tbeautiful."
1.7 Inasimilardirection,LeombruniandRichiardi(2005)considerthetwokeyperceivedproblemsof'Economists'withABMstudies(lackofgeneralisability,identificationorestimationproblems)standbehindtheheadlinecommentthatABMs,'don'tproveanything'.(Caseinpoint,LeombruniandRichiardi'spaperwaspublishednotbyaprogressive,forward-looking,Economicsfieldjournal,butratherbyPhysicaA:statisticalmechanicsanditsapplications–not,oneassumes,acommonlyreadjournalbymosteconomists.)
1.8 Whilstthethe'proof'(or'beauty')problemisratherover-statedbyfieldeditorsandreviewers(andishandledverywellbytheauthorsjustmentioned),alingeringproblemremains:thecommunicationofABMmethodsandresults–thisisanABMsciencepracticeissue.
1.9 ThesimplefactisthatABMstudiesoftendrawtheirheritageapartfromthedeductivemathematicalsciences,instead,theybuildonthesciencesofsoftwaredesignandnumericalsimulation.Andhere,ABMpractitionersinthesocialsciencesseemtohavesufferedheavilyfromalackofwell-establishedcommunicationtoolsandstandards.Richiardietal.(2006)capturethispointwell(paragraph1.5),
"Agent-basedmodelshavesolidmethodologicalfoundations.However,thegreaterfreedomtheyhavegrantedtoresearchers(intermsofmodeldesign)hasoftendegeneratedinasortofanarchy(intermsofdesign,analysisandpresentation)."[emphasisadded]
1.10 Theygoontoelaboratethisanarchyasfollows(paragraph1.5),
"a)Thereisnoclearclassificationofthedifferentwaysinwhichagentscanexchangeandcommunicate:everymodelproposesitsowninteractionstructure.
"b)Thereisnotastandardwaytotreattheartificialdatastemmingfromthesimulationruns,inordertoprovideadescriptionofthedynamicsofthesystem,andmanyarticlesseemtoignorethebasicsofexperimentaldesign.Often,thecomparisonbetweenartificialandrealdataisoverlynaïf,andtheparameters'valuesarechosenwithoutproperdiscussion;and
"c)Toooften,itisnotpossibletounderstandthedetailsoftheimplementationofanagent-basedsimulation.Thismakesreplicationadifficult,sometimesimpossibletask,thusviolatingthebasicprincipleofscientificpracticeandconfiningtheknowledgegeneratedbyagent-basedsimulationstonomorethananecdotalevidence."[enumerationadded]
1.11 …Anarchyindeed.Sadly,andbringingusbacktotheopeningquotationfromWooldridgeandJenning's(1998)article,thisexactproblemwasidentified(understandablythen)forthe'newfield'ofABMeightyearsprior(section8.2),
"Inafieldasnewasagentsystems,therearefewestablishedstandardsthatadevelopercanmakeuseofwhenbuildingtheagentspecificcomponentsofanapplication."
1.12 Sincethistime,therehave,ofcourse,beenrealattemptstoorderthe'anarchy'ofABMdevelopmentandcommunication.Asearlyas2000thepowerfulUnifiedModelingLanguage(UML)wasarticulatedtoincludeagents(Odelletal.2000)andsoofferedapotentialstandardisationcandidate(atleastofoneaspectofABMdevelopment),whilsttheOverview,Designconcepts,andDetails(ODD)(Grimmetal.2006)andODD+D(ODD+'Decision',Mülleratal.2013)protocolsof2006and2013respectivelyarosefromaninitiativewhichsoughttoresponddirectlytothe'anarchy'problemofABMdevelopment.Meanwhile,otherauthorshavefocussedtheircontributionstospecificareasofABMpracticesuchasmodelvalidation(Law2005;Windrumetal.2007;Moss2008)orvalidationandreplication(Heathetal.2009)and2-Dimensional,spatialvisualisation(Kornhauseretal.2009).Whilsttheseapproachesareallhighly
http://jasss.soc.surrey.ac.uk/18/4/16.html 2 25/01/2016
relevantandimportant(andinsomefocussedareasoftheliteratureappeartobehavinganimpactonthequalityofABMpractice,Grimmetal.2010),wewishtoprovidethecommunitywithaslightlymorequantitativeperspectiveonthestateofaffairs.
1.13 Specifically,inthisstudy,wefocusonthethird(c)aspectofRichiardietal.'s(2006)'anarchy'enumeration–thatofthebasiccommunicationofmethodologiesandresultsviaastudy'svisualshort-hand–thetables,figuresandequationsthatitemploystodescribeitsscience(whatweshallcallapaper's'objects').Weconsiderthatthissimpleaspecthasreceivedlittleattentioninthestandardisationeffortstodate.WherepreviousauthorshavespenttheirtimeonthedevelopmentanddesignaspectsofABMs,itwouldappearthatthecommunityhaslargelyassumedthatthequalityofbasiccomponentsofABMcommunicationwasingoodorder.Wecontendthatevenifamodelisdescribedaccuratelyandhelpfully,perhapsusingtheODDorODD+Dprotocol,andevenifthemodelhasbeenwellvalidated,itwillstillfailasapieceofscienceifitskeymethodologicalandresultsobjectsareofpoorquality.Thisdistinctionmattersperhapsmoresoinourvisually-orientated,short-attention-span,eraofscientificpublishing.Indeed,theso-called'mega-journal'PLOS-One,exemplifiesthistrend,inaskingauthorstonominatea'strikingimage'(forourstudy:'object')duringsubmission.Theypresumablyknowwellthevalueofsuchanimagefortheirsocialmediaandmarketingplatformintegrations.
1.14 Weconductastructuredanalysisofover900scientificobjects(figures,tables,orequations)thatarosefrom128ABMpaperspublishedintheJASSS,duringtheperiod2001to2012inclusive.Byfocussingonasingleoutlet,wereduceanyinter-journalvarianceonpublishingandeditorialstandardsandallowauthors'ownpracticestocometothefore.
1.15 ThejournalJASSSwaschosenforthestudyfortwoprinciplereasons:first,itiswell-regardedasageneralsocialsciencesABMstudy'clearing-house'withanactiveandengagedreadership,sostandswellforcross-disciplinarysocialsciencesABMpublishingtrends;andsecond,JASSS,beinganopen-access,online,HTML-based,journallendsitselftofacilemulti-yearstudyofthatwhichwepropose–paperscouldbeidentified,harvestedandanalysedwithease.WenotealsothatJASSSactsasacrucial'reverberationboard'forthetransmissionofrelevantideasbetweenthesocialandphysicalsciences(Squazzoni&Casnici2013).
1.16 Finally,thestudyperiod2001-2012coversthiscrucial'earlydecade'ofsocialsciencesABMscience.Asmentionedearlier,notonlyweremultiple,keycontributionsmadetointroduceABMmethodstovarioussocialsciencesfieldsatthestartofourstudyperiod,butalsotheperiodcoversalmostallofthemajorcontributionstothestandardisationprogramofABMsocialsciencemodeldevelopmentandcommunicationwhichhasbeenongoingsinceatleast2000.IftheseeffortshavehadanyearlyimpactsonABMcommunicationpractices,theseshouldbeevidentinourstudy.
TheABMPaperSample
EnrollingJASSSPublicationsintothePrimaryDatabase
2.1 World-wide-web(WWW)linkstoallscientificarticles(notreviews,forumetc.)publishedinJASSSfromvolume1to15(1998-2012)wereobtainedfromtheindexpage(e.g.http://jasss.soc.surrey.ac.uk/1/1/1.html),producing487links.
2.2 Ascriptwaswrittentodownloadthe.htmlfileofeacharticlefromtheJASSSsiteandthenprocessthemeta-dataoftheHTMLfilestoobtainthepaper'sfields(title,authors,keywords,abstract,volume,issue,dateofpublication).Insevencases,themeta-dataformatdeviatedfromtheJASSSstandard,causingthearticlestobedroppedfromthesample(e.g.http://jasss.soc.surrey.ac.uk/11/4/3.html).Thisproduced480articleswithsuccessfullyharvestedmeta-data.
2.3 Next,acorpusofuniquekeywords(the'Subject'meta-datacontentintheJASSStemplate,foranexample,seeAppendixA)(note,keywordscanbephrases)wasbuilt,takingcareofnon-materialkeywordvariationssuchasthepresenceorabsenceofahyphen.Inall,1501keywordsweregatheredinthisstep.
2.4 Finally,anarticlewasenrolledintotheprimarydatabaseifitcontainedoneofthefollowing'ABM'keywordschosenbytheauthorsafterreviewingtheuniquekeywordset,eitherexactly,orasakeywordroot(e.g.'multiagents',or'multiagentsystems'wouldmatchwith'multiagent')(numberofmatchesinparentheses):'agentbased'(151),'multiagent'(39),'socialsimulation'(31),'individualbased'(9),'multiagent'(8),'agentsimulation'(2),and'artificialagents'(1).Ofthe480papers,220uniquepaperscontainedamatchingkeywordwereenrolledintheprimarydatabase.Themajorityofpapers(202)hadasingleABMkeywordmatch,16matchedtwoABMkeywords,andtwomatchedthree.
2.5 Toassistwithreplication,afulllistingoftheresulting937objects,theirhomepaperID,andbasicdescriptorsisgiveninacomma-separatedvaluefileonlinewiththiswork.
Methods
Defining&ValidatingtheObjectTaxonomy
3.1 Sincenocleartaxonomyofobjectsexiststoourknowledgeintheliterature,theauthorssetaboutbuildingausefulandfaciletaxonomytodescribethepublicationpracticesineachABMpaper.Ourmethodologyhasbeenguidedbyexperiencearisingfromrelatedsocialsciencestaxonomy/encodingexercises,albeitofatextualnature(Haraetal.2000;Rourke&Anderson2004).Buildingthetaxonomyorganicallyproceededin6cyclicalsteps:
1. Twoauthorsreviewedthemethodsandresultsobjectscontainedwithinarandomlyselected20papersub-sampleofthe
http://jasss.soc.surrey.ac.uk/18/4/16.html 3 25/01/2016
primarydatabase(SampleA).2. Adrafttaxonomywascompiledcollaboratively,includinghierarchicaldescriptions.3. Thedrafttaxonomywasthenapplied,independentlybytwoauthors(SDA,BH-M)toSampleA.4. Thesameauthorsthenmettodiscussdisagreementsandimperfectionsinthedrafttaxonomyleadingtoarefinedtaxonomy.5. Therefinedtaxonomywasthenappliedindependentlybytwoauthors(SDA,BH-M)toafurther20paperrandomsub-sample
fromtheprimarydatabase(SampleB).6. Asecondmeetingwasthenconvenedbetweenthetwoauthorstovalidateandfurtherrefinethetaxonomyleadingtothefinal
taxonomy,andclarifyitsapplicationtothepooled(SamplesA+B)40papersampleusedinprevioussteps.
3.2 Note,sinceourfocusofanalysisisonthedecisionsofauthorsastohowtheypresentthemethodsandresultsofABMstudies,weskipduplicateobjecttypesfoundinanyarticle.Thatis,thefirstinstanceofagivenobjectisstudied,withsecond,third,andsubsequentobjectshavingthesamegeneralattributesasthefirst,notincludedintheanalysis.Typically,subsequentobjectsofthesamekindwerepresentedwithidenticalfeatures(orlackoffeatures)asthefirstobject,presumablysinceauthorscreatefigures,ortables,via'templates'.
SummaryoftheObjectTaxonomy
Table1:SummaryoftheObjectTaxonomy.Countsofobjectsateachlevelofthehierarchyaregiveninthecells.
http://jasss.soc.surrey.ac.uk/18/4/16.html 4 25/01/2016
3.3 Table1providesasummarypresentationoftheObjectTaxonomy.Ascanbeseen,alltheobjectsarefirstcategorisedbasedonwhethertheyareusedintheMethodsorResultssections.AgivenobjectisthenlabelledasFigure,TableorEquationbasedonwhattheauthorsoftheoriginalpapershavenamedthem.Finally,foreachobject,dependingonwhereithasappearedanditstype,wecollectedarangeoffurtherinformationrelevanttotheobjectaspresentedinthetable.
ApproachtoanObject'sSurroundingContext
3.4 Ifspecificcharacteristicsoftheobject'snature(e.g.the'Target'ofaMethodsFigureorTable,ortheGranularityofaResultsFigureorTable,referTable1)werenotobviousfromthecontentsoftheobject,oritscaption,theinformationwassoughtfromthesurroundingtext.However,inthespecificcasewherethesimulationresultsfigureswerestudiedfortheirquality(seefinalResultssection,'QualityofABMresultsplottingovertime'below)amorestringenttestwasappliedrequiringthekeyinformationwewerelookingfortobepresentinthefigureitselforfigurecaptiononly.Weareoftheviewthataresultsfigureshouldbe,asmuchaspossible,self-contained.Here,weareinfluencedbythewidelyused(e.g.bytheeditorsoftheProceedingsoftheNationalAcademyofSciencesoftheUSA)"ScientificStyleandFormat:TheCSEManualforAuthors,Editors,andPublishers"(8thedition)(CouncilofScienceEditors2014,section30.3)(seeespeciallysection30.2.2ofthisguide,'FigureCaption').
ExamplesofTaxonomicObjects
3.5 Note:inallcases,exceptwherestatedotherwise,exampleobjectsfrompapersareprovidedwithoutexplicitattribution.Ourintentionisnottosingleoutparticularauthors,butrathertopointtogeneraltrendsinthediscipline.SourceJASSSarticlecitationinformationforanyexampleprovidedcanbesoughtfromthecorrespondingauthor.
3.6 InFigures1to9weprovideexamplesofthetopthree(byincidenceinthedatabase)objectsusedtocommunicatemethodologiesinthedatabase.
http://jasss.soc.surrey.ac.uk/18/4/16.html 5 25/01/2016
Figure1.ExampleMethodsobject'Look-uptable'.Inthiscase,eachrowofthetabledescribesfeaturesofa'resource'usedinthemodel.
Figure2.ExampleMethodsobject'Schematicdiagram'.Thefigureindicatesrelationshipsbetweenexampleagentsinthemodel('Seller001','Buyer001')andfunctions,withoutadheringtoUMLformalism,ora'Flow-chart'presentation.
http://jasss.soc.surrey.ac.uk/18/4/16.html 6 25/01/2016
Figure3.ExampleMethodsobject'Screen-shot'.Atypicalscreen-shot,inthiscase,annotatedandtakenfromtheNetLogosoftwareplatform.
3.7 InFigures1,2and3,examplesofthe'Look-upTable','SchematicDiagram',and'Screen-shot'objectsarepresented.Fortheformer,weweredirectedbythenomenclatureoftheauthors,where'Figure'wasused,the'Figure'taxonomyobjectclasseswereemployed,whilst'Table'inducedthe'Table'taxonomyclasses.'SchematicDiagram',asintheexample(Figure2),wasusedtocodeanyfigurewhichconveyedtherelationshipbetweenmultipleaspectsofthemodel(agents,procedures)butdidnotconformtoeitheraflow-diagramorUML-formalism.Wehavenotsoughttoclassifythesefiguresfurther,owingtotherichvarietyofsymbolicandrelationalelementsemployedbyauthors.
Figure4.ExampleMethodsobject'Parameterinitialisation'.Eachrowofthetableprovidesparametervaluesforthreesettingsinthemodel.
http://jasss.soc.surrey.ac.uk/18/4/16.html 7 25/01/2016
Figure5.ExampleMethodsobject'Processes(behaviours,rules)'.Eachrowofthetableprovidesdefinitionsandthresholdsforvariousrulesandquantitiesinthemodel.
Figure6.ExampleMethodsobject'Experimentalsetup'.Eachrowofthetablepresentsthechoicesofdecisionrulesandparameterstestedineachexperiment.
http://jasss.soc.surrey.ac.uk/18/4/16.html 8 25/01/2016
Figure7.ExampleEquationobjects.a:'A-temporal',b:'Discrete-time',andc:'If-else'.
3.8 Figures4,5and6provideexamplesofhighlyused'Table'classobjects,being,unsurprisingly,focussedonparameterinitialisation,ruledefinitions,and(numericalsimulation)experimentalconditions.Figure7(a,bandc)provideexamplesoftheprominentequationtypesusedbyauthorsinthemethodssection.
Training&EncodingtheentirePrimaryDatabase
3.9 Next,aresearchassistant,familiarwithABMscience,wastrainedinapplyingthefinaltaxonomy,beforeapplyingittoSampleA.Areviewandtrainingmeetingwasheldwithoneoftheauthors(BH-M)toprovideguidanceandclarificationtotheresearchassistant,beforetheyappliedthefinaltaxonomytoSampleB.Afinaltrainingmeetingwasheldwithoneoftheauthors(BH-M)tocompletethetrainingandensurestrongcoherencewiththeencodingapproachoftheauthors.
3.10 Finally,theresearchassistantwentontoencodetheentirePrimaryDatabase,recordingdecisionswithanonlineformtofacilitatedataentry.Validationoftheresearchassistant'sapplicationofthefinaltaxonomytothePrimaryDatabaseincluded:on-going,ad-hoc,conferringwithoneoftheauthors(BH-M);randomcheckingbythesameauthoroftheencoding(around10%ofthecodedobjectsinall);andthenauthorcheckingofanyresidual'flagged'objects.A'flag'wasusedwherevertheresearchassistantwashesitantforanyreasoninthecorrectencodingtouse.Finally,adifferentresearchassistantreviewedall'flagged'objectstoconfirmthatthecorrecttaxonomyhadbeenapplied.
3.11 Duringcoding,anumberofpaperswerefoundnottobeABMinnatureandweredroppedfromthedatabasefollowingasimplerule:ifthepaperdidnotconveythemethodsandresultsofascientificenquiryusinganABMmodel,itwasdropped.Forexample,somepaperswerefoundtodiscussABMtheoryorpractice(suchasDeichselandPyka's(2009)'APragmaticReadingofFriedman'sMethodologicalEssayandWhatItTellsUsfortheDiscussionofABMS')orwereintendedas'positionpapers'foralternativedisciplines(suchasAhrweiler's(2011),'ModellingTheoryCommunitiesinScience').Additionally,paperspublishedpriorto2001(1998–2000)weredroppedduetothesmallnumberofresultantABMpapersintheseyears.
3.12 Intheend,theStudyDatabasecontained937objects(139equations,599figures,199tables)drawnfrom128papers,spanningtheyears2001–2012.
Quantitativeanalysis
3.13 QuantitativeanalysiswascarriedoutwithacombinationoftoolsincludingMicrosoftExcel(pivottables)andMATLAB(R2014b)(visualisations).
Results
Summaryofthedata
Table2:Uniquepapers,byyear,intheStudyDatabase.
Year Count2001 42002 92003 132004 72005 112006 112007 142008 132009 112010 182011 82012 9Total 128
http://jasss.soc.surrey.ac.uk/18/4/16.html 9 25/01/2016
Table3:Objectcounts,byMethodorResults,andbyObjectType,byyear,intheStudyDatabase.
Year Methods Results TotalEqtn. Figure Table Eqtn. Figure Table Total
2001 10 2 20 2 342002 6 21 6 22 5 602003 16 32 20 29 8 1052004 8 19 14 20 2 632005 10 33 6 23 7 792006 16 15 11 29 9 802007 17 20 14 37 10 982008 21 29 10 37 6 1032009 9 26 9 27 8 792010 15 36 16 1 39 9 1162011 5 22 9 12 3 512012 15 20 7 21 6 69Total 138 283 124 1 316 75 937
4.1 Tables2and3presentsummarystatisticsforpapersandobjects,respectively,intheStudyDatabase.Duetotherelativelysmallnumberofpapersenrolledeachyearovertheperiod,communicationpracticesovertimewillbestudiedatamulti-yearaggregatelevelbelow.Perhapsunsurprisingly,Table3demonstratesthetendencytoconveymethodologicalaspectsofastudypredominantlywiththeuseofabalanceofequations,figuresandtables,whilstresultsareseldomcommunicated(oranalysed)inequationform(apointwereturntolater),andlargelyfindexpressioningraphical(figure)presentation.
4.2 Toanalysepublishingpracticesovertime,westudytheincidenceofpaperswhichhaveatleastoneobjectofagiventype.AspresentedinTables2and3,itisobviousthatsomeyearshaveasmall(<5)numberofpapersorgivenobjecttype.Thus,todrawmeaningfulconclusions,weaggregateoverfour,threeyear,periods:2001–2003,2004–2006,2007–2009,and2010–2012(inclusive).
Temporalpatternsinmethodologicalpresentation
Table4:ABMmethodsobjectuseovertime.Percentofpapersinthedatabasepublishedwithinthegivenperiod,havingatleastonemethodsobjectofagiventype,sortedbydescendingincidenceinfinalperiod.
Type '01-'03 '04-'06 '07-'09 '10-'12Look-up-table 50 59 58 66Schematicdiagram 58 45 58 57Screen-shot 35 31 32 37Pseudo-code 0 17 16 26XYplotofvarioustype 15 3 11 14Raw-code 15 7 11 11Flow-chart 19 28 18 9Map 12 3 5 9Matrix 0 10 11 6UML 8 10 11 3Example-agent 12 17 0 0Table(other) 12 0 0 0Misc(allothers) 23 21 11 20Relevantpapersinperiod(count) 26 29 38 35
http://jasss.soc.surrey.ac.uk/18/4/16.html 10 25/01/2016
Figure8.PatternsinABMmethodsobjectuseovertime.Eachbubblerepresentsthepercentageofpapersinthedatasetpublishedduringthegiventhreeyearbandwhichincludedatleastoneoftheobjectsindicated.Note:sizingofthebubblesisnon-linear.
4.3 InTable4,andvisualisedinFigure8,trendsinpublishingpracticesformethodologicalpresentationarestudied.Weusetheincidence-rateofanobject,definedasthefractionofrelevantpapers(i.e.thoseincludingmethodologicalobjects)inthegivenperiodintheStudyDatabasewhichutilisedtheobjectatleastonce,asthesummarymeasurethepracticesundertakenbyauthors.Wenotethatpercentagesdonothavetosumwithinacolumn,asonepapermayexhibitmorethanoneobjecttype.
4.4 Forthesakeoftheanalysis,wepoolFigureandTableobjecttypeswhichtogethergives25uniquemethodsobjects.Further,wefocusononlythoseobjectswhichobtainanincidence-rateofatleast10%inoneofthefourperiods.Wefind12suchobjecttypes(Table4)whichfitthiscriterion,coveringbetween80%and89%ofallrelevantpapersinthatperiod.Prominentobjecttypesincludedin'Misc(allothers)'include'Algorithm','3Dmodelofenvironment','Agenttypehistogram',and'Video'(recall,JASSSiswhollypublishedonlineandmulti-mediaisencouraged,thoughapparentlyusedsparingly).
4.5 AscanbeseeninFigure8,thetopobjectchoices:'Look-up-table','Schematicdiagram'and'Screen-shot'areverystableoverthestudyperiod.Whatisperhapssurprisingisthatthemoreformal,structured,modeldescriptiontoolofUMLappearstohaveallbutdisappearedasaformofcommunicationovertime.Alternatively,andperhapsmoreencouragingly,raw-code,andpseudo-code(arguablythemostinformativeanddetaileddescriptionofthemechanicsofamodel)haveretained,orslightlyincreased,theirincidence-rateovertime.
Temporalpatternsinresultspresentation
Table5:ABMresultsobjectsovertime.Percentofpapersinthedatabasepublishedwithinthegivenperiod,havingatleastoneresultsobjectofagiventype,sortedbydescendingincidenceinfinalperiod.
Type '01-'03 '04-'06 '07-'09 '10-'12Simulation-only 96 100 100 97Empirical-only 0 8 8 10Mixed:Empirical–simulation 0 0 8 13Theory-only 8 12 3 3Mixed:Theory–simulation 8 4 3 3Notreported 4 0 0 0Relevantpapersinperiod(count) 24 26 38 31
http://jasss.soc.surrey.ac.uk/18/4/16.html 11 25/01/2016
Figure9.PatternsinABMresultsobjectuseovertime.Eachbubblerepresentsthepercentageofpapersinthedatasetpublishedduringthegiventhreeyearbandwhichincludedatleastoneoftheobjectsindicated.Note:sizingofthebubblesisnon-linear.
4.6 InTable5,andvisualisedinFigure9,wepresentasimilaranalysisforresultsobjects,again,poolingFigureandTableobjecttypes.Inthisanalysis,wefocusonthebasisoftheresultspresentation–whethertheresultsaredrawnfromsimulationresultsonly(i.e.quantitiesdrawnfromartificialdatasets),empiricalresultsonly(I.e.,quantitiesdrawnfromsurvey,ormeasureddatasets),theoreticaloutcomes(i.e.numericalcalculationsofparametricsystems,typicallywithoutstochasticsources),orsomecombinationoftheabove.
4.7 Whatthedatainthetableandthefigureshowisthatthepredominantpractice,byapproximatelyninetooneistheuseofsimulationdataonlyfromwhichtodrawresults.Thatis,authorsofidentifiedABMstudies,inJASSS,almostexclusivelysupporttheirkeyresults(eitherastablesorfigures)withtheuseofsimulationdataonly.Authorsarenot,therefore,oftenfoundtobepresenting(forverification,orcomparison)otherquantitytypesalongwiththeirsimulationdata.Thereisperhapssomeevidenceofatrendtowardscomparisonofsimulationquantitiestoempiricalquantities('Mixed:empirical–simulation')coupledwithatendencyawayfromtheoreticalcomparison,butthesampleistoosmalltoasserttheseassignificanttrends.
QualityofABMresultsplottingovertime
4.8 WefurtheranalysethepresentationofresultsintheStudyDatabasebyassessingwhetheragivenresults(figures-only)objectexhibitsoneoffivequalityattributes.Thefiveattributeswere:
1. (XY1)XandYaxeslabelsincluded(textualorpro-numeralinnature);2. (XY2)XandYscaleincluded(requiresnumericmax/minorticks,plusunits);3. (TYP)Thebasisoftheplottedresultsindicatedclearlyonthefigure,inthecaption,orinthesurroundingtext(e.g.simulation-
only,theory-only,mixed…etc.);4. (PAR)Theparametersusedtogeneratedthedataincluded(eitheronthefigure,inthecaption,orinreferencetoatable);5. (NUM)Forsimulationresults,thecountofsimulations(N)usedtocreatethesummarydataincluded.
4.9 Eachfigurewasgivenonepointifitdisplayedanyoftheabovefiveattributes,leadingtoamaximumscoreoffive.Itshouldbestressedthatweconsidertheabovefivefeaturestobetheabsolutebareminimumforeffectivecommunicationofscientificresults,followingtheprincipleslaidoutintheUniversityofChicagoScientificStyleandFormatmanual(8thedition)(CouncilofScienceEditors2014,section30.3).
Figure10.Examplesimulationsresultsfigurefroma2012JASSSarticlescoringpoorly(score1/5)underthequalityscoringtool.Inthiscase,thefigurereceivedonepointforincludingtheXandYscale.Allotherfeatures(axeslabelling,resultsbasis,parameters
used,simulationN)weremissing.
http://jasss.soc.surrey.ac.uk/18/4/16.html 12 25/01/2016
Figure11.Examplesimulationsresultsfigurefroma2012JASSSarticlereceivinganaveragequalityscore(score3/5).Inthiscase,thefigurereceivedonepointeachforincludingtheXandYscale,axeslabelsandresultsbasis.Otherfeatures(parametersused,
simulationN)weremissing.
Figure12.Examplesimulationsresultsfigurefroma2012JASSSarticlereceivingahighqualityscore(score4/5).Inthiscase,thefigurereceivedonepointeachforincludingtheXandYscale,axeslabels,resultsbasisandparametersused.However,norecordof
thenumberofsimulationsusedwasprovided.
4.10 InFigures10,11and12,weprovide(anonymous)examples,takenfromtheStudyDatabase,ofalow(score:1/5),medium(score:3/5)andhigh(score:4/5)qualityresultsfigureasassessedbyourattributemethod.
http://jasss.soc.surrey.ac.uk/18/4/16.html 13 25/01/2016
Table6:Thequalityofsimulationresultsobjects.Datafor229resultsfigureobjectswhichwerelabelledas,orappearedtobe,simulation-backedinnature.
Attribute CountofObjectswithAttributes Exists%
Missing%
Note
XY2 210 91.7 8.3 XandYscaleincludedXY1 177 77.3 22.7 XandYaxeslabelledTYP 126 55.0 45.0 SimulationbasisclearlyindicatedPAR 92 40.2 59.8 ParametersincludedNUM 75 32.8 67.2 CountofsimulationrunsOfTotal 229
4.11 Inall,229resultplots,drawnfrom106uniquepapers,wereanalysedusingthequalitymetric.Theseobjectsallclaimed,orappeared,tobesimulation-basedinnature–thepredominantformofresultspresentationchoice.Byaclearmargin(Table6),themostprevalentattributesdisplayedweretheXY2andXY1attributeswithover91%and77%ofobjectsexhibitingXandYscales,andXandYaxeslabellingrespectively.Toaidthereader'sunderstanding,weprovideinFigure13arare'negative'exampleinwhichasimulationresultsobjectfailedthe'XY2'(XandYscale)attribute.However,perhapsalarmingly,over40%ofobjectsfailedtodemonstrateoneormoreoftheotherthreeattributes.
Figure13.Examplesimulationsresultsfigurefroma2007JASSSarticleinwhichtheXY2(XandYscale)attributewasmissing.Casesofthisspecificattributebeingmissingwererare,occurringinlessthan10%ofrelevantobjects.
Figure14.PatternsinABMresultsqualityovertime.Barsrepresentthepercentileofofthedistributionofaverageobjectscores,bypaper,forthegivenperiod.Thetotalcountofrelevantpapersineachperiodwas('01-'03')18,('04-'06)27,('07-'09)34,and('10-'12)
27respectively.Objectqualityscoringexplainedinthetext.
4.12 Ifoneassumesthatauthorsareultimatelyresponsibleforthepresentationofresultfigures,thenonecantakeanaveragescoreacrossallqualifyingresultsobjectsbypaper(author).InFigure14,wepresentthepercentiles,byaggregatetimeperiod,ofaveragepaper
http://jasss.soc.surrey.ac.uk/18/4/16.html 14 25/01/2016
scores,whereeachpaperisrepresentedbytheaverageofthescoresoftheresultsobjectswithinit,codedinourstudy.
4.13 Again,stressingthatweconsiderthefiveresultsfigureattributesasbasicscientificrequirements,theresultsarealarming,bothintermsoftheaveragequality,andthetendovertime.Forinstance,themedianpaperscorein2001–2003of2.50isonlymarginallyimprovedupon(3.00)by2004–2006,andthennotagaininthesubsequentyears.Thisindicatesthataround50%ofpaperspresentresultsfigureswhichmisstwoormoreofthebasicfiveattributesweidentify.Atthetopend,ahighpointpaperscoreof4.62isobtainedforthe95thpercentilein2004–2006,butthisslidesto4.00by2010-2012.
4.14 Thesedataindicatethatforwhateverreason,thequalityofsimulationresultspresentedbyABMauthorsinJASSSoverthelastdecadeisgenerallypoor,andthestandardsarenotimproving.Wediscusspotentialreasonsfortheseproblemsbelow.
Discussion&Conclusions
5.1 Thisstudysetouttostudyanunder-reportedaspectofABMscience:thecommunicationpracticesandqualityofmethodologicalandresultsobjectsinABMstudiesoverthelastdecade.Asnotedintheintroduction,whilstotheraspectsofABMdevelopmentandcommunicationhavereceivedacademicattentionandproposals,wearenotawareofanysuchreviewofthekindwecarryouthere.
5.2 Againstabackgroundandhistoryofearly'anarchy'ontheonehand,andattemptsatformalisationontheother,wewereinterestedtoseeiftherewereanytrendstowardsorderordisorderinABMpublishingpractices,andinthegeneralqualityofthesepracticesforscientificends.
5.3 Below,wedrawtogetherthreekeyconclusionsfromourwork,andprovidesometentativereflectionsontheirpossiblecauses.
Conclusion1ABMscience'smethodological'anarchy'showsnosignsofsubmissiontoformalism
5.4 WeareindebtedtoRichiardietal.'s(2006)'anarchy'descriptor.Lookingatthepracticesrevealedinthisstudy,weseenoevidencethatthestateofaffairsischanging.Threedominantmodesofvisualcommunicationwereintheascendencyin2001–2003(Look-up-tables,'Schematic'diagrams',andScreen-shots),andthesamethreesatinthesamepositionadecadelater(Figure8).Whilstlook-up-tablesandscreen-shotscouldperhapsbeleftasidefromthisanalysis,astheyservetheirown,specific,communicativepurpose,theuseofrelationaldiagramsnotfittinganyparticularformalism(whichwecall'schematics')isinteresting.Indeed,whenonenotesthattheincidence-rateofUMLandFlow-chartdiagramshasdeclinedduringthedecade,wecanonlyconcludethatthesocialsciencescommunityhasturneditsbackon,orperhaps,hasneverproperlyengagedwith,theuseofmoreformalvisuallanguagesforconveyingtothereaderthecorerelationshipsamongstmodelcomponentsandagents.
5.5 ApotentialexplanationforthislackofengagementisofferedbyHeathetal.'s(2009)'manyfieldsofstudy'conclusiontotheirsurveyof297ABMpapers,
"ABMisconnectingdiversefields.Thefieldsofbiology,business,ecology,economics,themilitary,publicpolicy,socialscienceandtraffic,amongothers,alluseABM.ThesediversefieldsaretryingtounderstandcomplexsystemsandareusingABMasaonecommontool.…afterreviewingthesurveyedarticlesitisclearthateachfieldhasdevelopedtheirownABMterminologytodescribetechniques,applicationsandresults,havetheirownABMstandardsandtheirownABMphilosophies."(paragraph4.4)
5.6 Whilstthepaperssurveyedherearealldrawnfromthe'socialsciences'communityastheywerepublishedbyJASSS,asimilardrivercouldbeconjectured.OneofthepowerfulhallmarksofJASSSisitswideembraceofagent-basedsimulationpapersfromallacrossthesocialsciences.However,thisdiversitywillbringwithitadiversityofphilosophiesofknowledge,andadiversityofexpertisewithquantitativeandcomputationalmethods.Inparticular,WooldridgeandJennings'(1998)'Agent-orientedDevelopment'pitfall#4.3comestomind,"Youforgetyouaredevelopingsoftware"[emphasisretained].WooldridgeandJenningswrite,
"Unfortunately,becausetheprocess[ofdevelopinganyagentsystem]isexperimental,itencouragesthedevelopertoforgetthattheyareactuallydevelopingsoftware.Projectplanstendtobepre-occupiedwithinvestigatingagentarchitectures,developingcooperationprotocols,andimprovingcoordinationandcoherenceofmulti-agentactivity.Mundanesoftwareengineeringprocesses—requirementsanalysis,specification,design,verification,andtesting—becomeforgotten.Theresultofthisneglectisaforegoneconclusion:theprojectflounders,notbecauseofagent-specificproblems,butbecausebasicsoftwareengineeringgoodpracticewasignored."[emphasisretained]
5.7 Doestheanthropologist,writingtheirfirsteverABMin(say)NetLogo,grippedbythenotionthattheirtheoreticalhunchcan,forthefirsttime,bemodelledandvisualisedinreal-timebeforetheireyes,thinkthattheyarenowa'softwareengineer'?(Dotheyevenknowwhatthatis?)Theanswerisobviously'no'.OuranthropologistgetsonwithusingtheABMtooltosupporttheirscientificconclusion.Alongtheway,however,thedevelopment,validation,andcommunicationoftheirmethodologyandresultswillverylikelybe'home-spun'.Andso,anotherbespokedaughterofthe'schematic'classisborn.
Conclusion2ThescientificpresentationofABMresultsneedsimmediate,remedial,attention
5.8 Thesecondconclusionrestsonthealarmingpatternsuncoveredbyoursimple'5attributes'qualitysurveyofsimulation-basedresultsplots(Table6,Figure14).That60%ofthese'results'figuresdidn'tclearlyindicatetheparametersusedtogeneratethem,northenumberofsimulationsbehindthem,strikessomewhatofamortalblowtoanyhopeofsuccessfulreplication.Or,atthepaper-levelofobservation,thatthe75thpercentileaverageresultsfigurequalityscoreislessthan3.5,duringthemostrecentperiod(2010–2012)intheStudyDatabasesuggeststheproblemofadequateresultspresentationpracticeiswide-spreadandcontinuestoday.
http://jasss.soc.surrey.ac.uk/18/4/16.html 15 25/01/2016
5.9 Atfacevalue,itwouldseemthateithertheinterpretationorenforcementofJASSS'ownauthorguidelinesisatfaultastheyappearclearonthematterofreplication,
"Authorsarestronglyencouragedtoincludesufficientinformationtoenablereaderstoreplicatereportedsimulationexperiments."
5.10 However,theongoingtextoftheseguidelinesfocussesmoreontherequestthatfullmodelcodebemadeavailablethroughathird-partysite,ratherthanthedetailsneededtoactuallyusethemodeltoreplicatetheresults.Indeed,point(4)ofJASSS'sstated'Refereeguidelines'continuesinthis'algorithmic-centric'view,askingrefereestocommentexplicitlyon,
"Ifthearticledescribesasimulationmodel,isthereenoughdetailprovidedfortherelevantoutputfromthemodeltobereplicatedbyareader(thedescriptionmightbeintheformofanalgorithm,pseudo-code,oraccesstothesimulationprogramitself)?"
5.11 Whilstthequestionis'replication',theapparentsufficientanswer,accordingtoJASSS'guidelines,isthatauthorsprovideanindicationofthealgorithmusedtogeneratetheresults.Ofcourse,ifthe'simulationprogramitself'isprovided,thenitispossible,withasuitablywrittencode-base(andrunfiles)toidentifytheexactparametersusedforeveryresultinthepaper.However,anythingshortofthiswillnotsuffice:'analgorithm',or'pseudo-code',onitsown,willnotleapintoactionandproducetheresultsinthepaper.
5.12 Here,thereseemsaclearandsimpleopportunityforJASSS(atleast)totightenitsguidelinesaroundresults,broadeningits'algorithmic-centric'viewofreplicationtoincludealltheinformationrequiredtoreplicatetheresults,includingforexample,parameters,initialisationsettings,random-numberstreamdefinitions,andperhapsevenhardwarearchitecture.Again,suchconsiderationsnodoubtariseimmediatelytothemindofthesoftwareengineer,butnot,ouranthropologist(oreconomist,orsociologist,or…)colleague(weincludeourselveshere).
Conclusion3ParametricinterpretationbyestimationofABMsimulationdataappearstobemissinginaction
5.13 CloseinspectionofTable3highlightsonefurtherpatterninthepracticesrelatedtoanalysingABMresults:thelackofestimationofABMsimulationdata.The'smoking-gun'forthisconclusionisthealmostcompletelackofequationobjectswithintheresultssectionsofthe128papersreviewed,indeed,justone'results'equationwasidentifiedbyourmethods.WhilstwereadilyacknowledgethattherearemanycaseswhereparametricestimationofABMoutputsisunnecessarytojustifyscientificconclusionsduetotheinherentunpredictabilityoremergentcomplexityofclassesofABMmodels,therearemanyothercases,(forexample,whereanABMmodelistobecomparedtoempiricaleconomicdata)whereestimationisnecessary.Thatjustonepaperappearedtogodownthispath,wouldseemthatestimation,asananalysistechnique,orasaparametertuningtool,iseffectivelyunknowninthesocialsciencesABMcommunity.
5.14 Tobefair,thereislikelyculturalandtechnicalreasonsatplaybehindConclusion3.Culturally,werefertothe'manyfieldsofstudy'argumentproposedaboveinreflectiononConclusion1.ABMsocialsciencedemands,attimes,avariedandreasonablytechnicalskill-set:notonlymustthesocialscientisthavetheirownfield-specificknowledgeandnoveltyofcontribution,aswehaveseen,theyshouldalsoideallybeawareofsomebasicsoftwareengineeringprinciples,andhere,theywouldnowappeartoneedfurthertobefamiliarwithtime-seriesanalysisandestimationtechniques.SupposingagainthatJASSSwelcomesahigherproportionof'manyfields'authorswhoareexploringABMasatooltoilluminatetheirdisciplinethanotherjournals,thatalargeproportionofpapersdonotgodownthepathofestimationisunderstandable.
5.15 However,thereisasecond,technicalreasonthatcouldbeadvanced.Supposethatwearewrong,andthatJASSSauthorspredominantlywouldliketouseestimationtechniquestoassesstheartificialdatatheirmodelsproduceandtheyarewellversedin'standard'approaches,thentheywouldquicklydiscoverthatestimationofABMscanbepresentsomeuniquechallenges.TwofeaturesofalargenumberofABMscausetrouble–first,thenormallylargenumberofparameters,andsecond,theexistenceofnon-lineardynamics(oftenthereasonforchoosingABMtechniquesinthefirstplace).Together,theseattributescauseenormousdifficultyforestimation.Thereissome'hope'onthehorizon,however:veryrecentlyGrazziniandRichiardi(2014,2015)havebeguntoprovidesomecredibleoptionsfortheestimationofergodicandnon-ergodicABMtime-seriesinthepresenceoftheseattributes.WhilstGrazzini'searlier,butstillrelativelyrecent(2012)studyinJASSSshouldbehighlightedagaintotheABMsocialscientistscommunityasitenablesanauthortoidentifywhethertheirABMdataexhibitsstationarityandergodicityinthefirstplace(crucialquestionsforchoiceofestimationtechnique).
5.16 Hence,Conclusion3shouldnotbeseenasallthatsurprising,andshouldbereadmoreasaconfirmationofthestateofABMsocialscienceingeneral.However,withtheeffortsnowbeingexpendedonthisproblem(seeespeciallythecompanionworkofLeeetal.2015),ifasimilarpaucityofequation-basedestimationanalysisofABMoutputswasdiscoveredoverthenextdecade,differentconclusions(andprescriptions)wouldneedtobemade.
5.17 Beforedrawingfinalconclusionswereturntothemostobviouslimitationofourstudy:theuseonlyofJASSSABMpapersforouranalysis.Weacknowledgeseveralproblemshere.First,selectionbias:byconfiningourselvestoJASSSwehavenovisionofthequalityofABMstudiespublishedelsewhere:paperssubmittedtoJASSScouldbeofhigherorlowerqualitythanthe'fieldmedian';orpaperscouldexpressanunusualmixofsocialscience,artificiallife,andABMmethodologies,impactingthewaythattheyarepresentedtowardsabespoke'JASSS'style.Whilstsuchbiases(andotherslikethem)areimportant,theydonot,inouropinion,diminishtheresponsibilityoftheJASSStowardsbuildingupitspublicationstandards,JASSScan(anddoes)playacriticaleducationalroleinthepreparation,presentationandprosecutionofABMscience,apointwereturntobelow.
5.18 Second,editorialflux:itispossiblethatduringourstudyperiod,editorialpoliciesweakenedortightened,orfollowedsomeotherfunctionalform,perhapstopursueother(reasonable)motivationssuchasexpandingthe'reach'or'inclusiveness'oftheJASSS
http://jasss.soc.surrey.ac.uk/18/4/16.html 16 25/01/2016
community.Indeed,JASSSitselfpublishedcurrentJASSSEditorSquazzoniandCasnici's(2013)studywhichadvocatedspecificeditorialpolicyprescriptionsforJASSStobetterserveitsaimssuchaskeepingacloserwatchonJASSS'squantifiableinter-disciplinaryimpactandpotentiallytargetingspecific,un-tapped,domainsthroughspecialissues.Whilstitishardtoobtainameasureofsuchpolicydynamics,ourstudy'smainconclusionsencourageadoptionofminimumpractices,generictoallABMpapers,regardlessoffieldorspecificeditorialfocus,hence,weagainwouldsubmitthatanynuancededitorialmovementsapplyatalayerabovetheonetowhichwearestudying.Inanycase,theEditorialoversightofJASSSduringthisperiodwasasstableasonecouldhopeforanyjournal:asingle(foundational)editoroversawthefirst17yearsofthejournal'slife,generouslyoverlappingwithourstudyperiod.Theeditor,asfounder,builtupthejournalfromtheground,andhasrightlyreceivedenormousgratitudefromthesocialsimulationcommunityfortheircontributionstoJASSS'ssuccess(Elsenbroich&Badham2015).
5.19 Onefurtherlimitationisworthmakingclear.Weacknowledgethattheoverallclarityofapaper'sscientificcontributionmayormaynotrestontheformalismofitspresentationstyle.Inthiswork,wehavemerelytabulatedthetrendsintheABMpublishingcommunity,expressedthroughJASSS.Whatwecannotconcludeiswhetherthepaperswhichadoptoneorotherformalisminthecommunicationofmethodsandresultsareactuallyanyclearerthanthosewhichdonot.Theanswertosuchaquestionwouldrequireanaltogetherdifferentmethodology,presumablyinvolvingtrainedhumanreadersand/orreplicationattemptsofthesciencethatagivenpaperpresents.Weseesuchconsiderationsasanaturalextensionofthisworkandwouldencourageotherstoimaginecreativeexperimentaldesignswhichcouldidentifythe'best'communicationformalismofmethodsandresultsforABMworks.
5.20 Inonesense,ourstudyconcludeswithoutaparticularlynovelcontribution:weappearmerelytohavefoundnewwaystocataloguetheanarchyofABMsocialsciencespublicationpractices,oralternatively,thelackofuptakeofthevariousproposalsto'order'therealm.Onthisquestion,itwouldseemthattheABMsocialsciencescommunityhasachoice:itcangoonpermittingthe'anarchy',choosingtoapplylaissez-fairepublicationpracticestandardstothecommunicationofmethodologiesandresults,or,itcanattempttoapplyorderbyenforcingoneormorestandards.
5.21 Affinityordissatisfactionwith'anarchy'willlikelyturnonone'sstancetowardsthemeritsofdiversity.Ontheonehand,proponentsofanarchicpublishingpracticescouldpointtothebenefitsofenhancedacademicfreedom,creativeexpression,andthenotionoffittingtheauthors'personalsenseofthe'right'methodologicalorresultsobjectpresentationtothescientificclaimathand,withlittlereferencetonormsorstandards.AsmentionedaboveinConclusion1,giventhediversityofgeneratingfields(eachwiththeirownfield-norms)whoseektopublishtheirABMworksinJASSS,thereisnoneedtoencouragesuchdiversity,itwillariseorganicallyduetoeacharticle'sprovence.Furthermore,whoistosaythatoneauthor'sexperimentationorinnovationinpresentingtheirABMmethodorresultswillnotbeself-evidentlybrilliant,andsoreceivewideradoptionamongstABMauthorsviaimitationforthebenefitofall?Thereareindeedpotentialmeritstoanarchy.Thatsaid,letusadvanceanalternativeperspectiveonthemeritsofanarchy.IfourreadingofJASSSisright,thatitisaplatformofchoiceformanyfirst-timeABMauthorsfromthesocialsciences,thenJASSSmustbeseenasfarmorethansimplya'socialsciencessimulationpublication',JASSSisalsoapowerfuleducationaltool:thearticlesinJASSSwillimplicitlyformacorpusofideas,methodsandpracticestolearnfrom,extend,andultimatelyimitate.ThismattersbecauseiftheABMmethodologyistomakeprogressoutsideofsimulation-specificjournalslikeJASSSandintothe'main-stream'topfieldjournalsofourrespectivedisciplines,thenJASSScanplayanimportantroleindeveloping(enforcing)thebeststandardsoftheABMdiscipline'inhouse'beforeauthorstaketheirideastoeditorsandrefereesforwhomABMwill(still)beanentirelynewmethodologicalapproach.If,overtime,socialscientistsaremadetodevelopstandardapproaches,thenthatstandardwillbelearnedandunderstoodbyfieldeditorsandreferees,annullinganeasy(rejection)chargethatABMpaperslackintelligibilityortransparencybasedonthecommunicationofthemethodologyorresultsalone.Insummary,therecouldbestronglong-runbenefitstofinding(andcontinuouslyrefining)socialscienceABM'sformalvoiceonthepresentationofmethodsandresults.
5.22 Butourstudyhasdonemorethanconfirmthisanarchy,ithasidentifiedabasicproblemwithavastnumberofABMresultspublishedinJASSSoveradecade:thequalityofpresentedresults.Thissituationcannotcontinue.Itisamatteroffundamentalscientificpracticeandreproducibility–asupposedlycherishedfeatureofABMscience.Westressthatwearenotsuggestingthatworksstudiedinoursurveydonothavegoodscientificpointstomake,theseworkshaveallpassedthepeer-reviewprocessandassuchmustconveyimportantscientificfindings.However,theABMcommunitycannotontheonehandgrumbleabouttheslowuptakeofABMscienceintopfieldjournals,whilstontheother,failatpracticingbasicscientifichygienewhenitcomestopresentingitsresults.Again,viewedthroughaneducationallens,JASSShasarealopportunity,byitsinstructionstoreferees,anditssubmissionrequirements,tosetminimumstandardsforreplicationandresultspresentation.
5.23 Welookforwardtocontributingtothefurtherdevelopmentandenforcementofbest-practicestandards,andcallonthesocialsciencesABMcommunitytodothesame.
Acknowledgements
WethankTayaAnnable,andPenelopeMealyfortheirgenerousresearchassistanceandcontributionsinpreparingthisstudy,andJu-SungLeeandMatteoRichiardiwhoprovidedhelpfulcommentsonanearlierversionofthismanuscripts.Wealsothankthefouranonymousrefereeswhoprovidedextensive,interesting,andhelpfulperspectivesonourworkwhichservedtoimprovetheclarityofitsmessage.Allerrorsaretheauthors.
Notes
1Onecouldgofurtherbackstill.Forexample,Starbuck'sfascinating(1983)paperreviewingtheprospectsforsimulationinthesocial
http://jasss.soc.surrey.ac.uk/18/4/16.html 17 25/01/2016
sciencesoffersthemodernABMpractitionermuchmaterialforsoberreflection.
AppendixA-ExampleJASSSHTMLheaderusedfortermextraction,keywordsfromthe'Subject'identifierhighlighted
Paper:Huet,S.,Edwards,M.andDeffuant,G.(2007),"TakingintoAccounttheVariationsofNeighbourhoodSizesintheMean-FieldApproximationoftheThresholdModelonaRandomNetwork",JournalofArtificialSocietiesandSocialSimulation,10(1)10http://jasss.soc.surrey.ac.uk/10/1/10.html.
HTML:
References
ARHWEILER,P.(2011).ModellingTheoryCommunitiesinScience.JournalofArtificialSocietiesandSocialSimulation,14(4),8<http://jasss.soc.surrey.ac.uk/14/4/8.html>.
BONABEAU,E.(2002).Agent-basedmodeling:Methodsandtechniquesforsimulatinghumansystems.ProceedingsoftheNational
http://jasss.soc.surrey.ac.uk/18/4/16.html 18 25/01/2016
AcademyofSciencesoftheUnitedStatesofAmerica,99(suppl3),7280–7287.[doi:10.1073/pnas.082080899]
BOUCHAUD,J.-P.(2008).Economicsneedsascientificrevolution.NatureMaterials,455(7217),1181.[doi:10.1038/4551181a]
BUCHANAN,M.(2009).Economics:Meltdownmodelling.NatureNews,460(7256),680.[doi:10.1038/460680a]
COUNCILOFSCIENCEEDITORS(2014),ScientificStyleandFormat,8thEdition,ScientificStyleandFormatonline(URL:http://www.shttp://www.scientificstyleandformat.org/Home.html),UniversityofChicagoPress.
DEICHSEL,S.,&Pyka,A.(2009).APragmaticReadingofFriedman'sMethodologicalEssayandWhatItTellsUsfortheDiscussionofABMs.JournalofArtificialSocietiesandSocialSimulation,12(4),6<http://jasss.soc.surrey.ac.uk/12/4/6.html>
DOSI,G.,Fagiolo,G.,&Roventini,A.(2010).SchumpetermeetingKeynesApolicy-friendlymodelofendogenousgrowthandbusinesscycles.JournalofEconomicDynamics&Control,34(9),1748-1767.[doi:10.1016/j.jedc.2010.06.018]
ELSENBROICH,C.&Badham,J.(2015).AStandingOvationforNigel:AnInformalStudy.JournalofArtificialSocietiesandSocialSimulation,18(1),16<http://jasss.soc.surrey.ac.uk/18/1/16.html>.
EPSTEIN,J.M.(2006)."RemarksontheFoundationsofAgent-BasedGenerativeSocialScience."Chapter34inHandbookofComputationalEconomicsII:Agent-basedComputationalEconomics,editedbyLeighTesfatsionandKJudd,North-holland.
EPSTEIN,J.M.(2009).Amodelapproach.NatureMaterials,460(7256),667.
FARMER,J.D.,&Foley,D.(2009).Theeconomyneedsagent-basedmodelling.NatureMaterials,460(7256),685.[doi:10.1038/460685a]
GEANAKOPLOS,J.,Axtell,R.,Farmer,D.J.,Howitt,P.,Conlee,B.,Goldstein,J.,etal.(2012).GettingatSystemicRiskviaanAgent-BasedModel.AmericanEconomicReview,PapersandProceedings,102(3),53–58.[doi:10.1257/aer.102.3.53]
GILBERT,N.,&Terna,P.(2000).Howtobuildanduseagent-basedmodelsinsocialscience.Mind&Society,1,57–72.[doi:10.1007/BF02512229]
GRAZZINI,J.(2012).AnalysisoftheEmergentProperties:StationarityandErgodicity.JournalofArtificialSocietiesandSocialSimulation15(2),7<http://jasss.soc.surrey.ac.uk/15/2/7.html>.
GRAZZINI,J.,&Richiardi,M.(2014).PartialIdentificationinNon-ergodicModels.TechnicalReport,iNET,OxfordMartinSchool,UniversityofOxford.
GRAZZINI,J.,&Richiardi,M.(2015).Estimationofergodicagent-basedmodelsbysimulatedminimumdistance.JournalofEconomicDynamics&Control,51,148–165.[doi:10.1016/j.jedc.2014.10.006]
GRIMM,V.,Berger,U.,Bastiansen,F.,Eliassen,S.,Ginot,V.,Giske,J.,etal.(2006).Astandardprotocolfordescribingindividual-basedandagent-basedmodels.EcologicalModelling,198(1–2),115–126.[doi:10.1016/j.ecolmodel.2006.04.023]
GRIMM,V.,Berger,U.,Deangelis,D.L.,Polhill,J.G.,Giske,J.,&Railsback,S.F.(2010).TheODDprotocol:Areviewandfirstupdate.EcologicalModelling,221(23),2760–2768.[doi:10.1016/j.ecolmodel.2010.08.019]
HARA,N.,Bonk,C.J.,&Angeli,C.(2000).Contentanalysisofonlinediscussioninanappliededucationalpsychologycourse.InstructionalScience,28(2),115–152.[doi:10.1023/A:1003764722829]
HEATH,B.,Hill,R.andCiarallo,F.(2009).ASurveyofAgent-BasedModelingPractices(January1998toJuly2008).JournalofArtificialSocietiesandSocialSimulation12(4),9<http://jasss.soc.surrey.ac.uk/12/4/9.html>.
HOWITT,P&Clower,R,(2000).Theemergenceofeconomicorganization,JournalofEconomicBehaviour&Organization41,55–84.
KORNHAUSER,D.,Wilensky,R.&Rand,W.(2009).Designguidelinesforagentbasedmodelvisualisation.JournalofArtificialSocietiesandSocialSimulation12(2),1<http://jasss.soc.surrey.ac.uk/12/2/1.html>.
LAW,A.(2005).Howtobuildvalidandcrediblesimulationmodels.Proceedingsofthe37thConferenceonWinterSimulation,24–32.[doi:10.1109/wsc.2005.1574236]
LEE,J.,Filatova,T.,Ligmann-Zielinska,A.,Hassani-Mahmooei,B.,Stonedahl,F.,Lorscheid,I.,Voinov,a.,Polhill,J.G.,Sun,Z.&Parker,D.C.(2015).TheComplexitiesofAgent-BasedModelingOutputAnalysis.JournalofArtificialSocietiesandSocialSimulation18(4),4<http://jasss.soc.surrey.ac.uk/18/4/4.html>.
LEOMBRUNI,R.,&Richiardi,M.(2005).Whyareeconomistsscepticalaboutagent-basedsimulations?PhysicaA:StatisticalMechanicsandItsApplications,335,103–109.
LIM,M.,Metzler,R.,&Bar-Yam,Y.(2007).GlobalPatternFormationandEthnic/CulturalViolence.Science,317(5844),1540–1544.
MACY,M.W.,&Willer,R.(2002).FromFactorstoActors:ComputationalSociologyandAgent-BasedModeling.AnnualReviewofSociology,28(1),143–166.[doi:10.1146/annurev.soc.28.110601.141117]
MOSS,S.(2008).Alternativeapproachestotheempiricalvalidationofagent-basedmodels.JournalofArtificialSocietiesandSocialSimulation,11(1),5<http://jasss.soc.surrey.ac.uk/11/1/5.html>.
http://jasss.soc.surrey.ac.uk/18/4/16.html 19 25/01/2016
MÜLLER,B.,Bohn,F.,Dreßler,G.,Groeneveld,J.,Klassert,C.,Martin,R.,etal.(2013).Describinghumandecisionsinagent-basedmodels–ODD+D,anextensionoftheODDprotocol.EnvironmentalModelling&Software,48,37–48.[doi:10.1016/j.envsoft.2013.06.003]
ODELL,J.,Parunak,H.,&Bauer,B.(2000).ExtendingUMLforagents.InG.Wagner,Y.Lesperance,&E.Yu.PresentedattheProc.oftheAgent-OrientedInformationSystemsWorkshopatthethNationalconferenceonArtificialIntelligence,Austin,TX:AnnArbor.
RICHIARDI,M.,R.Leombruni,N.Saam,andM.Sonnessa.(2006).ACommonProtocolforAgent-BasedSocialSimulation.JournalofArtificialSocietiesandSocialSimulation9(1),15<http://jasss.soc.surrey.ac.uk/9/1/15.html>.
ROURKE,L.,&Anderson,T.(2004).Validityinquantitativecontentanalysis.EducationalTechnologyResearchandDevelopment,52(1),5–18.[doi:10.1007/BF02504769]
SQUAZZONI,F.&Casnici,N.(2013).IsSocialSimulationaSocialScienceOutstation?ABibliometricAnalysisoftheImpactofJASSS.JournalofArtificialSocietiesandSocialSimulation16(1),10<http://jasss.soc.surrey.ac.uk/16/1/10.html>.
STARBUCK,W.H.(1983).Computersimulationofhumanbehavior.BehavioralScience,28(2),154–165.[doi:10.1002/bs.3830280207]
WINDRUM,P.,Fagiolo,G.,&Moneta,A.(2007).Empiricalvalidationofagent-basedmodels:Alternativesandprospects.JournalofArtificialSocietiesandSocialSimulation,10(2),8<http://jasss.soc.surrey.ac.uk/10/2/8.html>.
WOOLDRIDGE,M.,&Jennings,N.(1998).PitfallsofAgent-OrientedDevelopment(pp.385–391).PresentedattheProceedingsofthesecondinternationalconferenceonAutonomousagents.[doi:10.1145/280765.280867]
http://jasss.soc.surrey.ac.uk/18/4/16.html 20 25/01/2016