The Use of Software Tools in Linked Data Publication and ... Use of Software... · International Journal on Semantic Web and Information Systems Volume 13 • Issue 4 • October-December

DOI: 10.4018/IJSWIS.2017100104

International Journal on Semantic Web and Information SystemsVolume 13 • Issue 4 • October-December 2017

Copyright©2017,IGIGlobal.CopyingordistributinginprintorelectronicformswithoutwrittenpermissionofIGIGlobalisprohibited.

The Use of Software Tools in Linked Data Publication and Consumption:A Systematic Literature ReviewArmando Barbosa, Federal University of Alagoas, Maceió, Brazil

Ig Ibert Bittencourt, Federal University of Alagoas, Maceió, Brazil

Sean Wolfgand Matsui Siqueira, Federal University of the State of Rio de Janeiro (UNIRIO), Rio de Janeiro, Brazil

Rafael de Amorim Silva, Federal University of Alagoas, Maceió, Brazil

Ivo Calado, Federal Institute of Alagoas, Rio Largo, Brazil

ABSTRACT

ToreducethecomplexityintrinsictoLDmanipulation,softwaretoolsareusedtopublishorconsumedataassociatedtoLDactivities.However,fewdevelopershaveabroadunderstandingofhowsoftwaretoolsmaybeusedinpublicationorconsumptionofLinkedData.ThegoalofthisworkistoinvestigatetheuseofsoftwaretoolsinLinkedDatapublicationandconsumptionprocesses.Morespecifically,understandinghowthesesoftwaretoolsarerelatedtoprocessofpublicationorconsumptionofLD.Inordertomeettheirgoal,theauthorsconductedaSystematicLiteratureReview(SLR)toidentifythestudiesontheuseofsoftwaretoolsintheseprocesses.TheSLRgathered6473studies,ofwhichonly80studiesremainedforfinalanalysis(1.25%of theoriginalsample).Thehighlightsof thestudyare:(1)initialstepsofthepublicationprocessarefairlysupportedbythesoftwaretools;(2)Non-RDFserializationisfairlysupportedinpublicationandconsumptionsprocessbythesoftwaretools;and(3)therearenon-supportedstepsinconsumptionandpublicationprocessesbythetools.

KeyWoRDSConsumption Process, Linked Data, Ontology, Publication Process, Software Tools

1. INTRoDUCTIoN

Inlastdecade,theevolutionoftheWebhasincreasedtheconsumptionandproductionofcontent.Thisgrowthimpliesinseveralchallengessuchasalongertimerequiredtoperformsearchesorbrowsing,dataconsistencyproblems,andmisinterpretations.Theusersandmachinesperceivethesechallenges,astheyarenecessarytoaccessasetofwebpagestofindrelevantinformation.Inaddition,mostofthedataavailableontheWebareunstructured(images,blogs,webcontent,etc.),whichmeansthatmachinesarenotable toautomaticallyunderstandsuchdatacontent.Therefore, it isessential toprovidealternativesforstructuringdatainamachineunderstandablemanner.

LinkedData(LD)isaneffectiveoptiontostructuresuchdata,presentingasetofbestpracticesforguidingpublishingandconnectingthesestructureddataontheWeb(Hyland&Wood,2011).LDadoptsagenericabstractdatamodelfordescribingresourcescalledResourceDescriptionFramework

68


69

(RDF).Thisframeworkisresponsiblefordescribingtermsandtheirrelationships,byusingtriplesofsubjects,predicatesandobjects.Additionally,ontologiesmayprovideenrichmenttoLinkedDatabyaddingsemanticstotheirRDFstatements(Heath&Bizer,2011).Moreover,LDbringssomebenefitswhichmaybeappliedinmultipleresearchfields,suchasdatainteroperabilityandtermsdefinition(Bauer&Kaltenböck,2011).SomeapplicationsofLD includeOpenGovernment (Ding,Lebo,Erickson,DiFranzo,Williams,Li,&Hendler,2011;Shadbolt,O’Hara,Berners-Lee,Gibbins,Glaser,&Hall,2012)(e.g.toimproveoftransparencyandprovenance),Education(Dietze,Yu,Giordano,Kaldoudi,Dovrolis,&Taibi,2012)(e.g. to improveeducationrepositoriesandcontentsharing),andBusiness(Barnes&Martin,2002)(e.g.toimprovebusinessopportunitiesandbusinessvalue).

LD is defined by both publication and consumption processes. Recently, W3C (2014) haspublishedalistof10stepstofulfillLDpublicationrequirements:(1)topreparestakeholders;(2)toselectdatasetforreuse;(3)tomodelthedatatorepresentdataobjectsandtheirrelationships;(4)tospecifyasuitableopendatalicense;(5)todefineURIforlinkeddata;(6)todescribeobjectswithstandardvocabularies;(7)toconvertdatatoalinkeddatarepresentation;(8)toprovidemachineaccesstodata;(9)toannouncetothepublic;and(10)toestablishasocialcontractofalinkeddatapublisher.Ontheotherhand,BauerandKaltenböck(2011)havepublishedalistof7stepsrelatedtoLDconsumptionprocess:(1)tospecifyconcreteusecasesfornewservicesorapplications;(2)toevaluaterelevantdatasourcesanddatasets;(3)tochecklicensesforuseandreuseprovidedbydataowners;(4)tocreateconsumptionpatternstospecifywhatdataarereusedfromacertaindatasource;(5)tomanagealignment,cachingandupdatemechanisms;(6)tocreatemashupsandapplicationstoprovideuser-friendlygraphicaluserinterfaces(GUIs)andpowerfulservicesforendusers;and(7)toestablishsustainablepartnerships.

Both aforementioned processes are essentials to improve data quality, interoperability, anddiscoverabilityofdata, aswell as to support thedevelopmentof richapplications.Additionally,consideringthenumberofstepsandactivities,itisimportantthedevelopmentoftoolsforsupportingLinkedDatapublicationandconsumptionprocesses.ConsideringthehugenumberofLinkedData-relatedtoolsproposedrecently(Bizer,Heath,&Berners-Lee,2009),itisatime-consumingtask,forpublishersorconsumers,toknowtheavailablefeaturesofeachtoolinordertomakeproperlydecisionaboutsoftwaretoolsthatshouldbeusedaccordingtoeachstep.Forthisreason,ahigh-levelquestionwasraised:Whichsoftwaretoolshavebeenusedtosupportthepublicationand/orconsumptionofLinkedData?Inthiscontext,ourpurposeistoidentifywhattoolshavebeenusedtoperformtheaforementionedtasks,providingabroadvisionaboutLinkedDatasoftwaretoolstopublishand/orconsumelinkeddata.Additionally,weusethecategoriesprovidedbyW3C1toclassifythesoftwaretoolsaccordingtotheirfeatures.

TherearesomesimilarworksthatinvestigatedtheuseofsoftwaretoolstopublishorconsumeLinkedData(Corlosquet,Delbru,Clark,Polleres,&Decker,2009;Bizeretal.,2009).However,thesestudiesdonotcaptureall theaspectsandevidencesthatweareinterested,suchassupportandformats.Moreover, theirworkissomewhatlimited,astheydonotsystematicallyreviewtheliteraturetoinvestigatetheworksthatjointlycoversoftwaretoolsandLinkedDataPublicationandConsumptionProcess.

Therefore,inthispaperweusetheSystematicLiteratureReview(SLR)methodtounderstandwhichsoftwaretoolssupportpublicationandconsumptionoflinkeddata.Weutilizethemethoddefinedin(Kitchenham&Charters,2007)toidentify,evaluate,interpretandsynthesizetheavailablestudies in order to answer particular research questions on the symbiosis of software tools forpublicationand/orconsumptionoflinkeddata,andtoestablishthestateofevidencewithanin-depth


70

analysis.Thoughttheanalysiswegeneratesixquestions.Thesequestionsmayguidenewresearchline,softwaresolutions,anddiscussion.

Therestofthispaperisorganizedasfollows:Section2describestheSLRmethodusedinthisreviewandprimaryresults.Section3presentsthequalityassessment.Section4presentsresults.Section5presentsadiscussionabouttheresults.Section6presentsthreatstovalidity.Finally,inthelastsectionwepresenttheconclusionsandfutureworkofthisSLR.

2. SySTeMATIC LITeRATURe ReVIeW

Asystematicliteraturereview(SLR)demonstratesthestateoftheartinsomeparticularresearchfield,evaluatingfindingsrelatedtoagivenresearchissue,atopicarea,orevenaphenomenon.ItisworthtodifferentiateaSLRfromaSystematicMapping.AccordingtoKitchenhametal.(2007),SystematicMappingsorScopingStudiesprovideawideoverviewofaresearcharea.TheresultsofamappingstudycanidentifyareassuitabletoconductingaSLRandareaswhereaprimarystudyismoreappropriate.

Inthiswork,weadopttheguidelinesproposedbyKitchenhametal.(2007)toconductourSLR.WealsoadoptthetoolcalledStArt(StateoftheArtthroughSystematicReviews)(LaPES,2012)toconductourSLR,reducingerrorsinherenttotheSLRprocess.Thistoolisvalidatedin(Hernandes,Zamboni,Fabbri,&Thommazo,2012),obtainingahighperformanceontheexecutionofaSLR.

Inthiswork,weinvestigatethefollowingresearchMainQuestion(MQ):

MQ→Whichsoftwaretoolssupportthepublicationand/orconsumptionofLinkedData?

AccordingtoKitchenhametal.(2007),themainstepforanysystematicreviewisthedefinitionoftheresearchquestions,sincetheyareresponsibleforguidingthewholereviewprocess.Basedonsuchassumption,andsimilarlytoothersystematicreviews(Dermeval,Vilela,Bittencourt,Isotani,&Brito,2014),weelaboratethreeresearchquestionsinordertoenableustoanswertheMQ.

• RQ1→Which are the linked data-based publication / consumption steps utilized by software tools in current literature?

• RQ2→Which serialization formats does the tool support?• RQ3→Is the usage of software tool empirically evaluated?

RQ1providesastartingpointtounderstandthesoftwaretoolsandthemainapproach(publish/consume)supportedbythem.Thisquestionisresponsibleforidentifyingthestepsofthepublicationandoftheconsumptionprocesssupportedbythesoftwaretools.RQ2identifiesthemainserializableformats(RDF,OWL,JSON,CSV,XMLandothers)usedforpublishingorconsuminglinkeddata.Finally,RQ3intendstoanalyzeifsuchstudiesprovideanempiricalevaluationofthesoftwaretools.Besides,Table1presentsthecentralmotivationforeachresearchquestion.

ThestepIIdefinesinclusionandexclusioncriteriaadoptedinthiswork.AccordingtoKitchenhametal.(2007),criteriaintheSLRprocessidentifythosepapersthatprovidedirectevidenceaboutresearchquestions,reducingapossiblebias.WeadoptedtwotypesofcriteriaintheSLR:(1)inclusioncriteria;and(2)exclusioncriteria.TheformerdefinesallthenecessaryrequirementsinorderanarticlemustbeconsideredintheSLRprocess.Thelatterdefinesalltherequirementsforanarticlenottobeincludedinthisprocess.Weconsideredprimarypapers(thosethatpresentsomeproposalinthearea)intheinclusioncriteriawhereassecondarypapers(i.e.thosethatonlyreviewanareatopicsuchasaSLR)intheexclusioncriteria.Regardingtopublicationdate,studiesareeligibleforinclusionintheSLRprocesswhethertheywerepeer-reviewedandpublishedfromJanuary2006


71

untilDecember2015.Intheotherhand,apaperisexcludedfromtheprocesswhetheritbelongstosomeexclusioncriteriadefinedinTable2.

ToperformthestepIII,weconsideredthefollowingelectronicdatabases:

• ScienceDirect(http://www.sciencedirect.com);• ISIWebofKnowledge(http://apps.webofknowledge.com);• Scopus(htttp://www.scopus.com);• SpringerLink(http://link.springer.com/);• ACMDigitalLibrary(http://dl.acm.org);• IEEEXplore(http://ieeexplore.ieee.org);• EngineVillage(http://www.engineeringvillage.com).

Figure1showsthesystematicreviewprocessandthenumberofpapersidentifiedateachstage.It isworthhighlighting the steps related to thepaper selectionprocess.Thus,at theStep I,

Weextractedthepapersfromelectronicdatabasesusingthefollowingsearchterms:(1)softwaretool terms,containing“Environment”OR“Authoring”OR“Architecture”OR“Framework”OR“Software”OR“Application”OR“Program”;(2)linkeddataterms,containing“LinkedData”OR“LinkedOpenData”;and(3)ontologyterms,containing“Ontology”OR“Ontologies”.Fromthesearchresults(6,473papers),weautomaticallyaccessed,downloadedandorganizedwithaidingoftheStArttool.Thesetermswerecombinedasfollows:

(1)AND(2)AND(3)

Table 1. Research Question Motivation

RQ Motivation Research Question

RQ1Ingeneral,toolssupportonlysomestepsofthepublication/consumptionprocess.Inthiscontext,itisimportanttodetermineforeachanalyzedtooltheexactsetofsupportedsteps.

Whicharethelinkeddata-basedpublication/consumptionstepsutilizedbysoftwaretoolsincurrentliterature?

RQ2 Consideringtherearedifferentserializationformatsavailable,wewanttodeterminetheformatsthateachanalyzedtoolsupports.

Whichserializationformatsdothetoolsupport?

RQ3Theevaluationofapublication/consumptiontoolisessentialtoquantifytheeffectivenessoftheanalyzedtool.So,weintendtoverifywhichempiricalevaluationswereappliedforeachtool.

Istheusageofsoftwaretoolempiricallyevaluated?

Table 2. Inclusion and exclusion criteria

# Inclusion Criteria # Exclusion Criteria

1 Papersabouttoolsforpublishingand/orconsuminglinkeddata 1 PapersinotherlanguagethanEnglish

2 PrimaryStudies 2 ShortPapers

3 ComputerSciencePapers 3 DuplicatedStudies

4 OnlyPapersinEnglish 4 SecondaryStudies

5 PapersbetweenJan.2006andDec.2015 5 Papersonlyabouttools

6 Paperswhichanswertheresearchquestions 6 Papersonlyaboutlinkeddata

7 Papersonlyaboutsoftwaretoolsfocusedopendata(withoutLinkedData)


72

AttheStepII,wedetectedandremovedduplicatedpapersusingtheStArttool,remainingasetof5,068papers.

AttheStepIII,wereviewedtitlesandkeywordsofeachpaperandexcludedthosenotrelatedtotheresearchquestions(1,733papers).Theremainingsetofpapers(3,335)wereconsideredforthenextstepsincedatawasinsufficienttodecideonremoval.

AttheStepIV,weanalyzedtheabstractsandexcludedpapersaccordingtotheexclusioncriteria(3,190papers).SimilarlytoStepIII,thepaperswithinsufficientinformationtobeclassifiedontheexclusioncriteriaremainedinthelisttobeanalyzedinthenextStep.

At theStepV,we retrieved thecomplete text from the remainedpapers (144), and readbothintroductionandconclusionsectionsand,additionally,performedafull-textscreenedofeachoneoftheselectedpapers.Fromthisstep,64paperswereexcludedinthisstepaccordingtotheexclusioncriteria.

AfterfinishingStepV,80papersremainedtothedataextractionprocess,whichhastwoactivities:(1)readingthefull-textofeachpaperand(2)savingextracteddata(i.e.inaspreadsheet,database,textfile,etc.).Therefore,allremainingpapersofStepVweresubmittedforfull-textreading.Tosavethedata,weusedStArttool.Additionally,duringthisstage,weextracteddataaccordingtoanextractionform(seeTable6inAppendixA),whichenabledustorecorddetailsaboutthereviewedpapers.

3. QUALITy ASSeSSMeNT

Thequalityassessmentoftheselectedstudiesisusefultoincreasetheaccuracyofthedataextractionresults.Inaddition,itisimportanttodeterminethevalidityoftheinferencesandinascertaining

Figure 1. Paper selection flow chart


73

credibilityandcoherentsynthesisofresults.Table3presentsthequalityassessment(Kitchenhametal.,2007)criteria.Criteria2,4and8onlyreceiveYes(Y)orNo(N)asscore.Additionally,weallowedonlyYes(Y)andPartially(P)scorestothelastcriteriasinceweareinterestedinthepossiblebenefitstoindustry.

TherawdataofthequalityassessmentispresentedinTable72oftheAppendixB,accordingtoassessmentquestiondescribedinTable3.Thescoresof11papersarelessthan50%andtheaveragescoreis7.08seeFigure2.Regardingthescoreofproposedcriteria,Q1,Q2andQ3receivedthehighestaveragesscores(>0.9),whileQ4,Q5,Q8,Q9,Q10andQ11receivedintermediatescoresbetween0.55and0.83.CriteriaQ6andQ7receivedthelowestaveragescore,0.14and0.13respectively.

4. ReSULTS

Thissectionpresents the resultsobtainedby running theSRLmethodpreviouslydescribed.Weprovideadetailedoverviewofgeneralfeaturesofthesestudies:sourcetypes,yearofpublication,andapplicationcontext.ThereviewedpaperswerepublishedbetweenJanuaryof2006andDecemberof2015.Additionally,thedistributionofselectedstudiesoverpublicationsources,includingpublicationname,type,count(i.e.thenumberofselectedstudiesfromeachsource),andthepercentageofselectedstudiesareavailable3.The80selectedstudiesaredistributedin49publicationsources,suggestingthatlinkeddatatoolshavebeenawidespreadconcerninthecommunity.Thestudiesincludedinthisreviewbelongthefollowingcategories:(1)journal;(2)conference;(3)workshop;or(4)bookchapters.Weidentifiedthatmoststudiesbelongtothecategory(2)(38.75%or31studies),followedbycategory(4)(27.5%or22studies),category(3)(17.5%or14studies)andcategory(1)(16.25%or13studies).Wealsoobservedthatmostoftheseselectedworkswerepublishedin2012(22.5%)

Table 3. Quality assessment criteria

# Questions Possible Answers

Q1 Istherearationaleforwhythestudywasundertaken? Y=1,N=0,P=0.5

Q2 Isthepaperbasedonresearch? Y=1,N=0

Q3 Isthereaclearstatementoftheresearchgoals? Y=1,N=0,P=0.5

Q4 Doesthestudyreusesomeframework(software,sourcecode)? Y=1,N=0

Q5 Isthereanadequatedescriptionofthecontextinwhichtheresearchwascarriedout? Y=1,N=0,P=0.5

Q6 Howmanystepsofthepublicationprocessdoesthestudysupport? Number of publication steps

Total of publication steps

Q7 Howmanystepsoftheconsumptionprocessdoesthestudysupport?

Number of consumption steps

Total of consumption steps

Q8 Wasthestudyempiricallyevaluated? Y=1,N=0

Q9 Isthereadiscussionabouttheresultsofthestudy? Y=1,N=0,P=0.5

Q10 Arethelimitationsofthisstudyexplicitlydiscussed? Y=1,N=0,P=0.5

Q11 Doestheresearchalsoaddvaluetotheindustrialcommunity? Y=1,P=0.5


74

followedby2010(15%),2013(15%),2014(13.8%),2011(12.7%),2009(6.3%),2008(3.8%),2000(1.3%),2007(1.3%).

ThequestionRQ1identifiesthepurposeofeachsoftwaretool.Weclassifiedthetoolsintotwocategories:publicationandconsumption(seeFigure4).Thesoftwaretoolscanbeclassifiedinoneorbothcategories,accordingtothesupportedsteps.Figure3depictsthat9softwaretools(11.11%)providesupporttooneormorestepsofbothlinkeddataconsumptionandpublicationprocesses.Thesetoolscanbeclassified,underPublicationpointofview,accordingtothefollowingfeatures:RDFpublisher(S01,S19,S25,S37,andS38),RDFsynchronizer(S33),RDFcompactor(S18),LinkedDataconsultinglanguage(S21)andsearchfederatedengine(S27).Table4describesdetailssuchassource-codelanguage,platform,publisher,supportedserializableformats,evaluationmethodandpublicationyear.Figure4depictsthedistributionofthese9studies,inpublicationandconsumptionprocesses.Mostsolutions(77.77%;7studies)arefocusedonprovidingmachineaccess(endpoint,publishingdatasets,etc.)todata(SPARQL,RDFandRDFa).Ontheotherhand,onlyonesoftwaretool(11.11%,1study)isconcernedwiththeconversionofstructureddatatolinkeddataandanotherone(11.11%)isconcernedwiththreesteps:definitionofURIsforLinkedData,announcementofdatasetsandtoestablishsocialcontractofalinkeddatapublisher.

ThequestionRQ2identifiesserializableformatssupportedbysoftwaretools.Figure5showsthe relation between serialization formats and publication, as well as consumption process. WecannotethegrowingsupporttoRDFserializationformats,wherethegreatestamountofsoftwaretoolsrelatedtopublicationprocessfocusesonprovidingmachineaccess(27studies).Ontheotherhand,regardingtotheconsumptionprocess,theconsumptionpatterns(28studies)andalignment(28studies)arethetwomainstepshandled.Itisalsoimportanttopayattentiontothelowsupporttonon-RDFserializationsinStepvii(Convertdata)oftheLDpublicationprocess.InFigure5,itisworthtonotethesupportisconcentratedinthesteps4and5oftheconsumptionprocess.Infact,themostsupportedserializationsareRDF(33.75%,27studies)andSPARQL(12.5%,10studies).Ontheotherhand,themostsupportednon-RDFareXML(8.75%,7studies),CSV(7.5%,6studies)andJSON(5%,4studies).

The question RQ3 enables the identification of evaluated-empirically software tools. Theclassificationoftheseevaluationsisobtainedfromeachpaper.ThenumberofstudiesforeachcategoryisshowninFigure5.Accordingtotheresults,40%ofthestudies(32studies)werenotevaluated,andtheotherstudiesaredistributedbetweenexperiment(36.25%,29studies),usecase(11.25%,9

Figure 2. Overview of the studies. Paper quality assessment distribution.


75

Figure 3. Classification of software tools in publication and consumption

Figure 4. Software tools distribution in steps of publication and consumption processes


76

studies),proofofconcept(2.5%,2studies),benchmark(1.25%,1study)andsimulation(1.25%,1study).Figure7depictshowtheevaluationsaredistributedalongpublicationandconsumption.Itisworthtonotethat50%(23studies)oftheworksdealingwiththepublicationprocesswerenotevaluated.

5. DISCUSSIoN

Thissectionpresentsadiscussionontheobtainedresults.Weintendtoprovideourperspectiveabouteachresearchquestion,highlightingthefaultsinthecurrentstateofsoftwaretoolsforpublishingandconsumingLinkedData.

Theresultsindicateanunevendistributionofsoftwaretoolsamongthepublicationsteps.Wenotethat63.82%ofthestudieswerefocusedonprovidingmachineaccess,followedbydataconversiontolinkeddataandthedefinitionofURIsforlinkeddata(with34.04%each).Ontheotherhand,wehavedetectedagaponsolutionstopreparestakeholders,supportopendatalicensespecification(2.12%,onestudy),selectadatasetforreuse(10.63%,fivestudies),usestandardsvocabulariesfordescribingobjects(12.76%,sixstudies)andmodelthedatatorepresentdataobjectsandtheirrelationships(24.89%,sevenstudies).Aswecansee,mostoftheproposedsoftwaretoolssupportonlythefinalstepsofthepublicationprocess(44studies)whileonly24studiessupporttheinitialsteps.

Theresultsarepartiallyexplainedbythehigh-levelvisionofthepublicationprocessdetailsprovidedbyW3Cdocuments.AlthoughW3C(2014)pointsouttheimportancetouse/determinedata

Table 4. Publication and consumption software tools

Study Name Developer Year Platform Language Supported Serializable Formats

Evaluation Method

S01 OrganiKproject Industry 2010 WEB PHP RDF Experiment

S18 RDSZ Academy 2014 Notdescribed Python RDF Experiment

S19 RDFextensionforDRUPAL Industry 2000 WEB PHP RDFa,RDFS,OWL

Doesnotpresentempiricalevaluation

S21 FREyA Industry 2012 Notdescribed FREyA SPARQL,RDF Experiment

S25 LinkedLab Academy 2011 WEB Notdescribed

RDF,OWL,XML,JSON,CSV UseCase

S27 KonneXSALT Academy 2008 WEB Java RDF,SPARQL


S33 RDFSync Industry 2007 Notdescribed

Notdescribed RDF,SPARQL Experiment

S37 Rhizomer Academy 2011 WEBJava,HTML,JavaScript

SPARQL


S38 Bioqueries Academy 2011 WEB PHP RDF,SPARQL,RDFS



77

license,itlacksdiscussingdetailsonhowtoimplementsuchtaskorwhichvocabularytoadopt.Asnoneanalyzedtoolcoverstheinitialstepsofthepublicationprocess,thereisnotoolabletofulfillthewholepublicationprocess.Basedontheaforementioneddiscussion,wecanpointouttwoquestionsthatcanberaisedforaresearchagenda.

1. HowtodevelopsoftwaretoolstocopewiththeinitialphasesofthepublicationprocessofLinkedData?2. HowtoprovideanintegratedsolutiontosimplifythepublicationofLinkedData?

RegardingtheLDconsumptionprocess,thepresentresultsindicatethatconsumptionsoftwaretools are concentrated in creating consumption patterns to specify what data are reused from acertaindatasource(85%,34studies),followedbymanagingalignments(85%,34studies).Further,thespecificationofusecasesfornewservicesorapplications(2.5%,1study)anddatasourcesanddatasetsevaluation(5%,2studies)aresupportedbyfewstudies,whiletheremainingstepswerenotcoveredbythereviewedstudies.Sincedevelopersandcompaniesusuallyhavespecificobjectivestoreuseadataset,theydeveloptheirownconsumptionpatternsandalignments.Consequently,wecandrawathirdquestionforaresearchagenda:

3. HowtodevelopandsharereusableconsumptionpatternstofacilitatetheconsumptionofLinkedData?

In addition, it is important toprovidegoodURIs4, convertdata to linkeddata, andprovidemachineaccess inpublicationprocess.Considering that thesestepssupportahigherquantityofserializationformats,theresultspointoutadisperseuseofserializableformatsinbothprocesses.Thus,theresultsindicatethatthesupportofserializableformatsimpactsdirectlyonhowinstitutionspublishtheirdata,aswellasconsumersreusethepublisheddataontheWeb.Anotherimportant

Figure 5. Evaluation distribution


78

pointisthelowsupportprovidedbythesoftwaretoolstopreviousandsubsequentstepsof“tocreateconsumptionpatternstospecifywhatdataarereusedfromacertaindatasource”and“tomanagealignment,cachingandupdatemechanisms”.

Similarly,tothepublicationprocess,wewerenotabletodetermineanytoolcapableofsupportingthewholeconsumptionprocess.Thus,itbringsusanimportantchallengethatwesummarizeinthefollowingquestiontotheresearchagenda:

4. HowtoprovideanintegratedsolutiontosimplifytheconsumptionofLinkedData?

ThecreationofconsumptionpatternsandthemanagementofalignmentmechanismsarethestepsofLDconsumptionprocessthatmoresupporttheserializationformats.ItisworthtonotethetworectanglespresentedinFigure6,sincetheycontainthehighestnumberofsupportedstepsandserializableformats.Therefore,thelowsupportforserializationformatsimpactsnegativelyinthe

Figure 6. Relation between publication and consumption processes and serializable formats


79

consumptionprocess,thusitisnecessarytoproposeapproachestoincreasethenumberofsupportedformats(structuredandRDFserialization).Thisleadsustoafifthquestion:

5. HowtoprovideasolutiontosupportsRDFandnon-RDFserializationformatsinLinkedDataPublicationandConsumptionprocess?

Theempiricalevaluationofanytoolisessentialtoensurequality,aswellastohighlightthetoolfeatures.Nevertheless,almosthalfofevaluatedtools(45%,36studies)donotpresentanykindofevaluation(empiricalornot).Therefore,itisimportanttoquantifytheeffectivenessofeachanalyzedtooltoperformtheintendedtaskproperly,whichbringsustothelastquestion:

6. Howtovalidatetheeffectivenessoftheproposedstudiesaccordingtosupportedsteps?

Inthebigpicture,wecanpointoutanincreasing,albeitbadlydistributed,offerofsoftwaretoolstosupplythepublicationandconsumptionofLinkedData.Inaddition,aconsiderablepartofthesetoolsdoesnotconsiderexistingproductionandconsumptionprocessesduringdevelopment.Thus,itisimportanttoprovideanewsetofsoftwaretoolsthattakeintoaccountthecurrentLinkedDatastateoftheart.

Figure 7. Evaluation by purpose


80

Throughoutthisstudy,wepresentedaconsiderablenumberoftoolstoassistthepublisherinbothprocessesofpublishingandconsuminglinkeddata.Thoughtheimportantcontributionspresented,accordingtoBizeretal. (2009) thefollowingquestionsarestillopen:1) thesupportof licenses(Heath&Bizer,2011;Schmachtenberg,Bizer&Paulheim,2014),2)theautomaticevaluationofdataquality(Maα&Qi,2013;Moss,Corsar,&Piper,2012),and3)thedataalignment(Bleiholder&Naumann,2009;Homoceanu,Kalo,&Balke,2014).Besides, it isnoteworthy theneedofanintegrationenvironmenttosimplifytheprocessexecution.

6. THReATS To VALIDITy

AsthreatstothevalidityofthisSLR,themainconstructsinthisreviewarethethreeconcepts“SoftwareTools”,“LinkedData”and“Ontologies”.Forthefirstconcept,weusedtheterms“environment”,“authoring”,“architecture”,“framework”,“software”,“application”and“program”tocollectstudiesrelatedtosoftwaretoolapproaches.Forthesecondconcept,theterms“LinkedData”and“LinkedOpenData”wereusedtocoverthepotentiallyrelevantstudiesonLinkedDatafromthedatabasesearch.Forthethirdconcept,theterms“ontology”anditsplural“ontologies”wereusedtocollectstudiesrelatedtoontologyapproaches.However,wedidnotusecontrolpapersduringthesearchondigitallibraries,whichmaybeconsideredasthemainthreattovalidity.Additionally,wedidnotperformacomplementarymanualsearch.

7. CoNCLUSIoN

Inthiswork,weconductedaSLRtoinvestigatethesupportofsoftwaretoolstopublishandconsumeLinkedData.Ourgoalwastoimprovetheunderstandingofhowthesesoftwaretoolssupporteachstepofbothprocesses.

Eightystudieswerefullyinvestigated,ofwhichnineprovidesupportinoneormorestepsoftheconsumptionandpublicationprocesses.Inregardstothepublicationprocess,itispossibletoevidencethelowsupportintheinitialsteps.Moreover,intheconsumptionprocess,itispossibletoevidencethesoftwaretoolsfocusthesupportofthedevelopmentofconsumptionpatterns,aswellasdataalignment.

Aspreviouslydiscussed,weraisedasetofquestions,whichinvolvesthestepsinprocesses,software tools,serializationformatsandtherelationsamongthem.Finally,wehighlightedsomeimportantquestionsthatarerevisedandsummarizedinTable5inordertosuggestaresearchagendaonlinkeddata.

Basedon(Kitchenhametal.,2007)wecanhighlightsomecontributionsprovidedbythisstudy.Suchcontributionsinclude:

• Theidentificationofgapsincurrentresearchinordertosuggestareasforfurtherinvestigation;• Theprovisionofaframework/backgroundinordertoproperlypositionnewresearchactivities;• Thesummarizationofexistingevidence,benefitsandlimitations.

The result presented in thisSystematicLiteratureReviewcanbeveryuseful to linkeddatacommunity,sinceitgathersevidencesfromprimarystudiesincludedinthereview,formingabodyofknowledgeregardingtheuseofsoftwaretoolstopublishorconsumelinkeddata.Wecanoutlinethecontributionofthisworkasfollows:

• ResearchagendabasedonSRLobservationsandaccordingtogapsinbothprocesses.Besides,wepointedouttheneedofanintegrationenvironmenttosimplifytheprocessexecution.


81

Asfuturework,weintendtofurtherinvestigatethesoftwaretoolintegration.Moreover,weintendtoextendthissystematicreviewaimingtoclarifythecommunicationinterfaceamongsoftwaretoolsbetweenstepsineachprocess.

Table 5. Questions and motivation

Motivation Question

Theinitialstepofthepublicationprocessisbadlyexploredbytheavailabletools.

HowtodevelopsoftwaretoolstocopewiththeinitialphasesofthepublicationprocessofLinkedData?

Nonesolutionsupportsthewholeprocessofpublicationofconnecteddata.

HowtoprovideanintegratedsolutiontosimplifythepublicationofLinkedData?

Thereisnotastandardandin-depthapproachtotheconsumptionprocessofconnecteddata.

HowtodevelopandsharereusableconsumptionpatternstofacilitatetheconsumptionofLinkedData?

Nonesolutioncoversthewholeprocessofdataconsumption.

HowtoprovideanintegratedsolutiontosimplifytheconsumptionofLinkedData?

Thereisneedtoincreasethenumberofsupportedserializationformatsforbothpublicationandconsumptionprocesses.

HowtoprovideasolutiontosupportsRDFandnon-RDFserializationformatsinLinkedDataPublicationandConsumptionprocess?

Lownumberofsoftwaretoolswasevaluated.Additionally,theseevaluationswarenotfocusedonsupportpublicationorconsumptionprocess.

Howtovalidatetheeffectivenessoftheproposedstudiesaccordingtosupportedsteps?


82

ReFeReNCeS

Auer,S.,Bühmann,L.,Dirschl,C.,Erling,O.,Hausenblas,M.,Isele,R.,&Williams,H.(2012).Managingthelife-cycleoflinkeddatawiththeLOD2stack.InThe Semantic Web–ISWC 2012(pp.1–16).SpringerBerlinHeidelberg.doi:10.1007/978-3-642-35173-0_1

Barnes,M.,&Martin,R.(2002).Businessdatalinking:Anintroduction.Economic Trends,581,34–41.

Bauer,F.,&Kaltenböck,M.(2011).Linkedopendata:Theessentials.Editionmono/monochrom,Vienna.

Bizer,C.,Cyganiak,R.,&Heath,T.(2007).HowtopublishLinkedDataontheWeb.RetrievedSeptember18,2016,fromhttp://wifo5-03.informatik.uni-mannheim.de/bizer/HowtoPublishLinkedData.htm

Bizer,C.,Heath,T.,&Berners-Lee,T.(2009).LinkedData-TheStorySoFar.International Journal on Semantic Web and Information Systems,5(3),1–22.doi:10.4018/jswis.2009081901

Bleiholder, J., & Naumann, F. (2009). Data fusion. ACM Computing Surveys, 41(1), 1–41.doi:10.1145/1456650.1456651

Corlosquet,S.,Delbru,R.,Clark,T.,Polleres,A.,&Decker,S.(2009).ProduceandConsumeLinkedDatawithDrupal! InProceedings of the International Semantic Web Conference (pp.763-778).SpringerBerlinHeidelberg.doi:10.1007/978-3-642-04930-9_48

Dermeval,D.,Vilela,J.,Bittencourt,I.I.,Castro,J.,Isotani,S.,&Brito,P.(2014,September).Asystematicreviewontheuseofontologiesinrequirementsengineering.InProceedings of the 2014 Brazilian Symposium on Software Engineering (SBES)(pp.1-10).IEEE.doi:10.1109/SBES.2014.13

Dietze,S.,Yu,H.Q.,Giordano,D.,Kaldoudi,E.,Dovrolis,N.,&Taibi,D.(2012,March).LinkedEducation:interlinkingeducationalResourcesandtheWebofData.InProceedings of the 27th annual ACM symposium on applied computing(pp.366-371).ACM.doi:10.1145/2245276.2245347

Ding,L.,Lebo,T.,Erickson,J.S.,DiFranzo,D.,Williams,G.T.,Li,X.,&Hendler,J.A.(2011).TWCLOGD:Aportalforlinkedopengovernmentdataecosystems.Web Semantics: Science, Services, and Agents on the World Wide Web,9(3),325–333.doi:10.1016/j.websem.2011.06.002

Heath,T.,&Bizer,C.(2011).Linkeddata:Evolvingthewebintoaglobaldataspace.Synthesis lectures on the semantic web: theory and technology,1(1),1-136.

Hernandes,E.,Zamboni,A.,Fabbri,S.,&Thommazo,A.D.(2012).UsingGQMandTAMtoevaluateStArt-atoolthatsupportsSystematicReview.CLEI Electronic Journal,15(1),3–3.

Homoceanu,S.,Kalo,J.C.,&Balke,W.T.(2014,June).PuttingInstanceMatchingtotheTest:IsInstanceMatchingReadyforReliableDataLinking?InProceedings of theInternational Symposium on Methodologies for Intelligent Systems(pp.274-284).SpringerInternationalPublishingdoi:10.1007/978-3-319-08326-1_28

Hyland,B.,&Wood,D.(2011).Thejoyofdata-acookbookforpublishinglinkedgovernmentdataontheweb.InLinking government data(pp.3–26).SpringerNewYork.doi:10.1007/978-1-4614-1767-5_1

Kitchenham,B.,&Charters,S. (2007).Guidelinesforperformingsystematic literaturereviews insoftwareengineering.InTechnical report, Ver. 2.3.EBSETechnicalReport.EBSE.

LaPES.(n.d.).Start-stateoftheartthroughsystematicreviewtool.RetrievedSeptember04,2016,fromhttp://lapes.dc.ufscar.br/tools/start_tool

Moss,L.,Corsar,D.,&Piper,I.(2012,June).Alinkeddataapproachtoassessingmedicaldata.InProceedings of the 2012 25th International Symposium on Computer-Based Medical Systems (CBMS) (pp. 1-4). IEEE.doi:10.1109/CBMS.2012.6266391

Schmachtenberg,M.,Bizer,C.,&Paulheim,H.(2014,October).Adoptionofthelinkeddatabestpracticesindifferenttopicaldomains.InProceedings of theInternational Semantic Web Conference(pp.245-260).SpringerInternationalPublishing.doi:10.1007/978-3-319-11964-9_16

Shadbolt,N.,OHara,K.,Berners-Lee,T.,Gibbins,N.,Glaser,H.,Hall,W.,&schraefel,.(2012).Linkedopengovernmentdata:Lessonsfromdata.gov.uk.IEEE Intelligent Systems,27(3),16–24.doi:10.1109/MIS.2012.23

http://dx.doi.org/10.1007/978-3-642-35173-0_1

http://wifo5-03.informatik.uni-mannheim.de/bizer/HowtoPublishLinkedData.htm

http://dx.doi.org/10.4018/jswis.2009081901

http://dx.doi.org/10.1145/1456650.1456651

http://dx.doi.org/10.1007/978-3-642-04930-9_48

http://dx.doi.org/10.1109/SBES.2014.13

http://dx.doi.org/10.1145/2245276.2245347

http://dx.doi.org/10.1016/j.websem.2011.06.002

http://dx.doi.org/10.1007/978-3-319-08326-1_28

http://dx.doi.org/10.1007/978-1-4614-1767-5_1

http://lapes.dc.ufscar.br/tools/start_tool

http://lapes.dc.ufscar.br/tools/start_tool

http://dx.doi.org/10.1109/CBMS.2012.6266391

http://dx.doi.org/10.1007/978-3-319-11964-9_16

http://dx.doi.org/10.1109/MIS.2012.23


83

Armando Barbosa is a Master’s student in Informatics graduate program, Computing Institute at the Federal University Alagoas (Brazil) and a member of the Center of Excellence for Social Technologies. Armando’s main research line is Semantic Web, Linked Data.

Ig Ibert Bittencourt is Professor at the Federal University of Alagoas (Brazil) and CoDirector of the Center of Excellence for Social Technologies. He received his PhD in 2009 from the Federal University of Campina Grande, Brazil and his Post-Doctoral degree in 2013 at the University of Campinas (UNICAMP). He is also the vice-president of the Special Committee of Computers and Education from Brazilian Computer Society and W3C Advisory Committee Representative. Ig Bittencourt also founded a Company in Intelligent Tutoring Systems awarded as the most innovative Brazilian company in 2014 (USP Innovation Prize). He believes in innovative social entrepreneurship as a model for promoting a sustainable economic and social development. His main interests are research, development and innovation in Computers and Education, Ontologies and Social Entrepreneurship.

Sean W.M. Siqueira is Professor at the Department of Applied Informatics, Federal University of the State of Rio de Janeiro (UNIRIO), Brazil, where he teaches courses in Information Systems, Databases, Web Science and Social & Semantic Web. He holds an MSc (1999) and a PhD (2005) in Computer Science, both from the Pontifical Catholic University of Rio de Janeiro (PUC-Rio), Brazil. His research interests include knowledge representation and management; collaborative systems; social web; semantic web; ontologies; information integration; semantic models; user models; and learning & education. He has experience in the Computer Science area, with focus on Web Science, Information Systems and Technology Enhanced Learning. He has participated in some international research projects and has written more than 130 papers for conferences, journals, and book chapters. He was the coordinator of the Graduate Program in Information Systems at UNIRIO from July 2012 to September 2014, the coordinator of the Program Committees of the Brazilian Symposium on Information Systems (SBSI 2015), the Brazilian Symposium on Computers and Education (SBIE 2012 and SBIE 2014), member of the steering committee of the Brazilian Congress of Computers and Education (CBIE 2015), co-editor-in-chief of the iSYS: Brazilian Journal of Information Systems as well as the Brazilian Journal on Computers in Education, and member of the special committee on Computers and Education (CEIE) and coordinator of the special committee on Information Systems (CESI), both from the Brazilian Computer Society.

Rafael de Amorim Silva received his BS degree in Information System from the Faculty of Alagoas (2005), Brazil. He received his MS degree in Computer Science from the Federal University of Pernambuco (2008), Brazil, and his PhD degree in Electronics Engineering from the Aeronautics Institute of Technology, Brazil. He is currently professor of the Federal University of Alagoas, Brazil. His main scientific interests include Computer Networks, Informatics in Education, and Artificial Intelligence.

Ivo Calado holds a BSc (2008) and MSc (2010) in Computer Science at Federal University of Alagoas and Federal University of Campina Grande, respectively. He also holds a PhD in Electrical Engineering at Federal University of Campina Grande, Brazil (2015). Currently, Ivo Calado is a Professor at Federal Institute of Education, Science and Technology of Alagoas, Brazil. His main research interests include Computer Networks, Internet of Things and Semantic Web.

Villazón-Terrazas, B., Vilches-Blázquez, L. M., Corcho, O., & Gómez-Pérez, A. (2011). Methodologicalguidelinesforpublishinggovernmentlinkeddata.InLinking government data(pp.27–49).SpringerNewYork.doi:10.1007/978-1-4614-1767-5_2

WorldWideWebConsortium.(2014).Bestpracticesforpublishinglinkeddata.RetrievedSeptember04,2016,fromhttps://www.w3.org/TR/ld-bp/

eNDNoTeS

1 https://www.w3.org/2001/sw/wiki/Category:Tool2 http://bit.ly/2c7UlHc3 http://bit.ly/2cB9CzE4 http://www.w3.org/TR/ld-bp/#HTTP-URIS

http://dx.doi.org/10.1007/978-1-4614-1767-5_2

https://www.w3.org/TR/ld-bp/


84

APPeNDIX A

extraction Form

Table 6. Extraction form

# Study Data Description Relevant RQ

1 StudyIdentifier Uniqueidforthestudy Studyoverview

2 Dateofdataextraction,Authors,Year,Title,Country Studyoverview

3 ArticleSource Studyoverview

4 TypeofArticle Journal,Conference,workshop,bookchapter Studyoverview

5 Softwaretoolname Whatarethesoftwaretoolname? Studyoverview

6 Version Whatarethesoftwaretoolversion? Studyoverview

7 LastUpdate Whenwasthelastupdateofthesoftwaretool? Studyoverview

8 License Whatisthelicense? Studyoverview

9 Platform Web,Mobile,Desktop Studyoverview

10 Language Whatare/istheusedlanguagetoimplementthesoftwaretool? Studyoverview

11 ApplicationContext Industrial,Academic Studyoverview

12 Purpose Publication,Consumption RQ1

13 Stepsfollowed Whatarethestepsfollowedofthepurpose? RQ1

14 Serializableformats Whataretheserializableformatssupportedbythesoftwaretool? RQ2

15 ResearchMethod Controlledexperiment,Casestudy,Usecase,Conceptualproof,Benchmark,Simulation,Notapplicable RQ3


85

APPeNDIX BQuality Assessment

Table 7. Quality assessment

ID References Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10 Q11 Total Score Quality

S01 (Aastrand,Celebi,&Sauermann,2010) 1 1 1 1 1 0,4 0,3 1 1 0,5 1 9,2 84%

S02 (An,Kim,Lee,&Lee,2013) 1 1 1 1 0,5 0 0,3 1 1 1 1 8,8 80%

S03

(Dimou,DeVocht,VanGrootel,VanCampe,Latour,Mannens&VandeWalle,2014)

1 1 1 1 1 0 0,1 1 1 0 0,5 7,6 69%

S04

(Lehmann,Furche,Grasso,Ngomo,Schallhart,Sellers,&Auer,2012)

0 0 1 0 0 0,1 0 0 1 1 1 4,1 37%

S05 (Aprosio,Giuliano,&Lavelli.2013) 1 1 1 0 0,5 0,3 0 1 1 1 0,5 7,3 66%

S06(Araujo,Houben,Schwabe,&Hidders,2010)

1 1 1 1 0 0 0,4 0 1 0 0,5 5,9 54%

S07(Assaf,Louw,Senart,Follenfant,Troncy,&Trastou,2012)

1 1 1 1 1 0,2 0 1 1 1 1 9,2 84%

S08(Auer,Dietzold,Lehmann,Hellmann,&Aumuelle,2009)

1 1 1 0 1 0,4 0 1 1 1 1 8,4 76%

S09 (Augenstein,Padó,&Rudolph,2012) 1 1 1 1 0,5 0,5 0 1 1 1 1 9 82%

S10 (Lehmann&Bühmann,2011) 1 1 1 1 0,5 0,1 0 1 1 1 0,5 8,1 74%

S11 (Bottoni&Cerian,2014) 1 1 1 0 0,5 0 0,4 0 0 0 0,5 4,4 40%

S12 (Chang,Xu,Zhou,Shao,Li,&Yan,2013) 1 1 1 1 0,5 0,4 0 0 0 1 0,5 6,4 58%

S13

(Chen,Ding,Wang,Wild,Dong,Sun,&Sankaranarayanan,2010)

1 1 1 1 0,5 0,4 0 1 1 1 0,5 8,4 76%

S14 (Becker&Bizer,2009) 0,5 1 1 0 0,5 0 0,3 0 0 1 0,5 4,8 44%

S15

(Bizer,Lehmann,Kobilarov,Auer,Becker,Cyganiak,&Hellmann,2009)

1 1 1 1 0,5 0,4 0 0 1 1 0,5 7,4 67%

S16 (Ciravegna,Gentile,&Zhang,2012) 1 1 1 0 0,5 0 0,3 0 0 0 0,5 4,3 39%

S17

(Coletta,Castanier,Valduriez,Frisch,Ngo,&Bellahsene,2012)

1 1 1 1 0,5 0,2 0 0 1 0 0,5 6,2 56%

continued on following page


86


S18

(Fernández,Arias,Sánchez,Fuentes-Lorenzo,&Corcho,2014)

1 1 1 1 0,5 0,1 0 1 1 1 1 8,6 78%

S19 (Corlosquetetal.,2009) 1 1 1 1 1 0,4 0 1 1 0 0,5 7,9 72%

S20 (Tao,Song,Sharma,&Chute,2013) 1 1 1 1 0,5 0,1 0 1 1 1 0,5 8,1 74%

S21(Agatonovic,Cunningham,&Damljanovic,2012)

1 0 1 1 0,5 0,1 0 1 1 0 0,5 6,1 55%

S22(Curry,O’Donnell,Corry,Hasan,Keane,&O’Riain,2013)

1 1 1 0 1 0,3 0 0 1 0 1 6,3 57%

S23(DiNoia,Mirizzi,Ostuni,Romito,&Zanker,2012)

0,5 0 0 0 0,5 0 0,3 1 1 0 0,5 3,8 34%

S24(DeVocht,Softic,Mannens,Ebner,&VandeWalle,2014)

1 1 1 0 1 0 0,3 1 1 1 0,5 7,8 71%

S25 (Darari&Manurung,2011) 0,5 1 1 0 0,5 0,5 0 0 1 1 0,5 6 55%

S26 (Dastgheib,Kochut,&Mesbah,2013) 1 1 1 1 0,5 0 0,1 0 1 1 0,5 7,1 65%

S27(Groza,Handschuh,Möller,&Decker,2008)

1 1 1 1 1 0,2 0 0 1 1 0,5 7,7 70%

S28 (Delbru,2009) 1 1 1 0 1 0 0,3 1 1 0 0,5 6,8 62%

S29 (Auer,Dietzold,&Doehring,2010) 1 1 1 1 0,5 0,1 0,3 1 1 1 1 8,9 81%

S30(Paret,VanWoensel,Casteleyn,Signer,&DeTroyer,2011)

1 1 1 0 0,5 0 0,3 1 1 1 0,5 7,3 66%

S31 (Elze,Hesse,&Martin,2011) 1 1 1 1 1 0 0,3 0 1 1 0,5 7,8 71%

S32(Emaldi,Lázaro,Laiseca,&López-de-Ipiña,2012)

0,5 1 0,5 1 0,5 0 0,3 1 1 1 0,5 7,3 66%

S33

(Tummarello,Morbidoni,Bachmann-Gmür,&Erling,2007)

1 1 1 0 0,5 0,3 0 1 1 1 1 7,8 71%

S34 (Baritakis,Fafalios,&Tzitzikas,2014) 1 1 1 1 0,5 0 0,3 1 1 0 0,5 7,3 66%

S35 (Farouk&Ishizuka,2012) 1 1 1 0 0,5 0,2 0 1 1 1 0,5 7,2 65%

S36(Fernández-Tobías,Cantador,Kaminskas,&Ricci,2011)

0,5 1 0,5 0 0,5 0 0,3 0 1 1 0,5 5,3 48%

S37(García,Brunetti,López-Muzás,Gimeno,&Gil,2011)

1 1 1 1 0,5 0,1 0 0 1 1 0,5 7,1 65%

Table 7. Continued



87


S38

(Aldana-Montes,García-Godoy,&Navas-Delgado,2011)

1 1 1 1 1 0,4 0 0 1 1 0,5 7,9 72%

S39 (Graves,2010) 1 1 1 1 0,5 0,4 0 0 1 1 0,5 7,4 67%

S40(Wang,Liu,Penin,Fu,Zhang,Tran,&Pan,2009)

1 1 1 1 0,5 0 0,3 1 1 1 0,5 8,3 75%

S41 (Chen,Hsueh,&Huang,2011) 1 1 1 0 1 0,1 0 0 0 0 0,5 4,6 42%

S42(Fisteus,García,Fernández,&Fuentes-Lorenzo,2014)

1 1 1 1 0,5 0,1 0 1 1 1 0,5 8,1 74%

S43 (Han,Finin,Parr,Sachs,&Joshi,2008) 0,5 1 1 0 0,5 0,4 0 1 0 1 0,5 5,9 54%

S44 (Bauer,Kaltenböck,&Recheis,2011) 1 1 1 0 0,5 0,4 0 0 0 0 0,5 4,4 40%

S45 (Auer,Hladky,&Khalili,2012) 1 1 1 1 0,5 0,1 0 1 1 1 0,5 8,1 74%

S46 (Auer&Khalili,2013) 1 1 1 1 0,5 0,1 0 1 1 1 0,5 8,1 74%

S47 (Consens&Khatchadourian,2010) 1 1 1 0 0,5 0 0,3 1 1 0 0,5 6,3 57%

S48 (Grollios,Mitkas,&Vavliakis,2013) 1 1 1 0 0,5 0 0,1 1 1 1 0,5 7,1 65%

S49 (Konstantinou,Kouis,&Mitrou,2014) 1 1 1 1 0,5 0,4 0 1 1 1 0,5 8,4 76%

S50(Hernandez,Lappalainen,&Sicilia,2013)

1 1 1 0 1 0 0,7 0 0 1 0,5 6,2 56%

S51

(Lehmann,Furche,Grasso,Ngomo,Schallhart,Sellers,&Auer,2012)

1 1 1 0 0,5 0 0,1 1 1 0 0,5 6,1 56%

S52 (Lowe,2009) 1 1 1 1 1 0,2 0 1 1 0 0,5 7,7 70%

S53(Lynden,Kojima,Matono,&Tanimura,2011)

1 1 1 1 0,5 0,2 0 1 1 1 0,5 8,2 75%

S54 (Machado&ParentedeOliveira,2011) 1 1 1 0 0,5 0,4 0 0 1 1 1 6,9 63%

S55 (Narasimha,Kappara,Ichise,&Vyas,2011) 1 1 1 1 0,5 0 0,3 1 1 1 0,5 8,3 75%

S56 (Curé,2014) 1 0 1 1 1 0 0,1 0 1 1 0,5 6,6 60%

S57(Ostuni,DiNoia,Mirizzi,Romito,&DiSciascio,2012)

0,5 1 1 0 0,5 0 0,3 0 1 1 0,5 5,8 53%

S58(Otero-Garcıa,Vidal,Lama,Bugarın,&Domenech)

1 1 1 0 1 0 0,3 0 1 0 0,5 5,8 53%

S59 (Merialdo&Papotti,2012) 1 1 1 0 0,5 0,4 0 1 1 1 0,5 7,4 67%

Table 7. Continued



88


S60 (Duval&ParraKlerkx,2010) 1 1 1 1 1 0 0,1 1 1 1 0,5 8,6 79%

S61 (Passant,2010) 0,5 0 1 0 0,5 0 0,3 1 1 1 0,5 5,8 53%

S62 (Brambilla,Ceri,&Quarteroni,2013) 0,5 1 1 1 0,5 0,2 0 1 1 0 0,5 6,7 61%

S63

(Radzimski,Sánchez-Cervantes,Rodríguez-González,Gómez-Berbís,&García-Crespo,2012)

1 1 1 1 0,5 0,2 0 0 0 0 0,5 5,2 47%

S64 (Seguran,Senart,&Trastour,2012) 1 1 1 0 1 0,1 0 0 0 0 1 5,1 46%

S65 (Shakya,Takeda,&Wuwongse,2009) 1 1 1 1 0,5 0,1 0 0 0 1 0,5 6,1 55%

S66 (Chapman,Skarka,&Solanki,2012) 1 1 1 0 1 0,1 0 0 1 1 0,5 6,6 60%

S67(Solomou,Kalou,Koutsomitropoulos,&Papatheodorou,2011)

1 1 1 1 0,5 0 0,3 1 1 1 1 8,8 80%

S68(Sorrentino,Bergamaschi,Fusari,&Beneventano,2013)

1 1 1 1 1 0,1 0 0 0 1 1 7,1 65%

S69 (Stadtmüller,Speiser,Harth,&Studer,2013) 1 1 1 0 0,5 0 0,3 1 1 1 1 7,8 71%

S70 (Stolz&Hepp,2013) 1 1 1 1 0,5 0 0,3 1 1 1 1 8,8 80%

S71 (Norheim,Roman,&Stuhr,2011) 1 1 1 1 0,5 0,1 0 1 1 1 1 8,6 78%

S72 (d’Aquin,Motta,&Tiddi,2014) 1 1 1 0 0,5 0 0,1 1 1 1 0,5 7,1 65%

S73 (Tramp,Heino,Auer,&Frischmuth,2010) 1 1 1 0 0,5 0,1 0 1 1 1 0,5 7,1 65%

S74

(Tummarello,Cyganiak,Catasta,Danielczyk,Delbru,&Decker,2010)

1 0 1 0 0,5 0 0,3 0 0 1 0,5 4,3 39%

S75 (Veres,2012) 0,5 1 1 0 0,5 0 0,1 1 1 1 0,5 6,6 60%

S76(Montoya,Ibáñez,Skaf-Molli,Molli,&Vidal,2014)

1 1 1 0 0,5 0 0,3 1 1 1 0,5 7,3 66%

S77(Gómez-Pérez,Vila-Suero,&Villazón-Terrazas,2012)

1 1 1 1 0,5 0,4 0 1 1 1 1 8,9 81%

S78 (Zhang,Song,Li,&Liu,2014) 1 1 1 0 0,5 0,5 0 1 1 1 0,5 7,5 68%

S79

(Kaoudi,Koubarakis,Kyzirakos,Miliaraki,Magiridou,&Papadakis-Pesaresi,2010)

1 1 1 1 0,5 0 1 1 1 1 0,5 9 82%

S80 (Čerāns&Būmans,2010) 1 1 1 1 0,5 0,1 0 1 1 1 0,5 8,1 74%

Average 0,92 0,92 0,97 0,55 0,61 0,14 0,13 0,62 0,83 0,74 0,61 7,08 64%

Table 7. Continued

Documents

The Use of Software Tools in Linked Data Publication and ... Use of Software... · International Journal on Semantic Web and Information Systems Volume 13 • Issue 4 • October-December