29
IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data Integration Deliverable ID D3.2 Work Package Reference WP3 Issue 1.0 Due Date of Deliverable 30/11/2017 Submission Date 15/11/2017 Dissemination Level 1 PU Lead Partner Terranea Contributors - Grant Agreement No 700256 Call ID H2020-DRS-1-2015 Funding Scheme Collaborative I-REACT is co-founded by the Horizon 2020 Framework Programme of the European Commission under grant agreement n. 700256 1 PU = Public, PP = Restricted to other programme participants (including the Commission Services), RE = Restricted to a group specified by the consortium (including the Commission Services), CO = Confidential, only for members of the consortium (including the Commission Services)

Report on Open Data and Historical Events Data Integration · IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data

  • Upload
    others

  • View
    4

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Report on Open Data and Historical Events Data Integration · IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data

IMPROVINGRESILIENCETOEMERGENCIESTHROUGH

ADVANCEDCYBERTECHNOLOGIES

ReportonOpenDataandHistoricalEventsDataIntegration

DeliverableID D3.2

WorkPackageReference WP3

Issue 1.0

DueDateofDeliverable 30/11/2017

SubmissionDate 15/11/2017

DisseminationLevel1 PU

LeadPartner Terranea

Contributors -

GrantAgreementNo 700256

CallID H2020-DRS-1-2015

FundingScheme Collaborative

I-REACTisco-foundedbytheHorizon2020FrameworkProgrammeoftheEuropeanCommissionundergrantagreementn.700256

1PU=Public,PP=Restrictedtootherprogrammeparticipants(includingtheCommissionServices),RE=Restrictedtoagroupspecifiedbytheconsortium(includingtheCommissionServices),

CO=Confidential,onlyformembersoftheconsortium(includingtheCommissionServices)

Page 2: Report on Open Data and Historical Events Data Integration · IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data

ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies

Project:I-REACT ReportonOpenDataandHistoricalEventsDataIntegration

DeliverableID:D3.2

GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:2of29

Preparedby Reviewedby Approvedby

G.Zeug W.Stemberger C.Rossi

Issue Date Description Author(s)

0.01 13/10/2016 Provisionoftemplate W.Stemberger

0.02 21/10/2016 Updateofreportstructure/TOC G.Zeug

0.03-0.09

06/01/2017 –10/11/2017

Continuousreportupdates G.Zeug

1.0 10/11/2017 Finalversion G.Zeug

Page 3: Report on Open Data and Historical Events Data Integration · IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data

ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies

Project:I-REACT ReportonOpenDataandHistoricalEventsDataIntegration

DeliverableID:D3.2

GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:3of29

TABLEOFCONTENTS1 INTRODUCTION............................................................................................................................5

1.1 PurposeoftheDocument.....................................................................................................5

1.2 StructureoftheDocument...................................................................................................5

1.3 Acronymslist.........................................................................................................................6

1.4 References.............................................................................................................................6

2 OPENDATA..................................................................................................................................7

2.1 BackgroundaboutOpenDatainEurope...............................................................................7

2.2 UseofOpenDatainIREACT..................................................................................................7

2.2.1 DirectIntegrationofOpenData.....................................................................................7

2.2.1.1 CriticalInfrastructureData.......................................................................................9

2.2.2 IndirectIntegrationofOpenData................................................................................12

2.2.2.1 Downscalingofstatistics–Population...................................................................12

2.2.2.2 Downscalingofstatistics–ForestStatistics...........................................................15

2.2.2.3 Floodvulnerabilitylayer.........................................................................................16

2.2.2.4 BurnedAreaMapping............................................................................................17

2.3 Interfaces.............................................................................................................................20

3 HISTORICALDATA......................................................................................................................21

3.1 HistoricalHazard/DisasterEventDatasets.........................................................................21

3.1.1 Identificationofavailableopendatasources...............................................................21

3.1.2 Geocodingofthehazardevents...................................................................................22

3.1.3 Definitionofadatamodelanddataharmonization.....................................................23

3.1.4 Integrationintoacommondataset..............................................................................23

3.2 DisasterEventTracker.........................................................................................................24

3.3 Interfaces.............................................................................................................................25

4 CONCLUSIONS............................................................................................................................26

5 LISTOFOPENDATASETS............................................................................................................27

5.1 ListofOpenDatasets...........................................................................................................27

Page 4: Report on Open Data and Historical Events Data Integration · IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data

ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies

Project:I-REACT ReportonOpenDataandHistoricalEventsDataIntegration

DeliverableID:D3.2

GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:4of29

LISTOFFIGURESFigure1:SampleofOSMdatainPBFformat...................................................................................10

Figure2:ModelschemetoautomaticallydownloadandextractOSMdata...................................11

Figure3:OSM-derivedpolicestations(blue)andfirebrigadestations(red)intheUK..................12

Figure4:Allocationaccordingtoproportion...................................................................................14

Figure5:AsubsetoftheCatalonianpopulationgridbasedondasymetricmapping......................14

Figure6:AsubsetofJRC'stotalpopulationgridofCatalonia..........................................................15

Figure7:Forestvaluegridbasedonforesttypeandwoodprices.Pricerangesfromlightgrey(lowprice)todarkgrey(highprice),blackarebackgroundcells.............................................................16

Figure8:BurnedAreaMappingworkflow.......................................................................................18

Figure 9: Burned Area Map produced using Copernicus Sentinel-2 imagery. The black outlineenclosestheburnedarea.Theblackenedarearepresentscharredvegetationinthepost-firefalse-colorimage(channels8,4,3)............................................................................................................19

Figure10:ExampleofaBurnSeveritymapfromtheCalifornianwildfires,October2017.............20

Figure11:WebinterfacetodisplayandqueryhistoricalwildfireeventsinEurope.......................25

Figure12:PrototypewebinterfacefortheDisasterEventTrackingservice...................................26

LISTOFTABLESTable1:SelectedOpenDatathemesforI-REACTuse.......................................................................8

Table2:ThemesandrelatedsectorsselectedfortheI-REACTcriticalinfrastructuredataset..........9

Table3:MaximumdamagesperlanduseintheUK,accordingtoJRCGlobalflooddepth-damagefunctions..........................................................................................................................................17

Table4:NaturalhazardeventdatabasesusedasinputfortheI-REACThistoricaleventsdataset.22

Table5:DatastructureoftheHistoricalEventsdataset..................................................................23

Table6:Listofidentifiedandpossiblyusefuldatasets....................................................................28

Page 5: Report on Open Data and Historical Events Data Integration · IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data

ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies

Project:I-REACT ReportonOpenDataandHistoricalEventsDataIntegration

DeliverableID:D3.2

GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:5of29

1 INTRODUCTION

1.1 PURPOSEOFTHEDOCUMENTThisreportsummarisestheactivitiesofthe I-REACTtasksonOpenData(T3.3)andonHistoricalEvents(T3.7).Itdescribestheidentificationandprocessingstepsfortheprovisioningofopendataproducts. Moreover, it describes the process of collecting historical disaster event data fromdifferentsourcesandtheirintegrationintoasingledataset.

1.2 STRUCTUREOFTHEDOCUMENTThedocumentisorganizedasinthefollowing:

• Chapter1istheintroductionanddescriptionofthedocumentitself;

• Chapter2includestheactivitiesonopendata;• Chapter3illustratesthehistoricaleventdatacollectionandprocessing;• Chapter4istheconclusion;• Chapter5 lists the identifiedopendatasets thatwere thebasis for theselectionprocess

withintheprojectteam.

Page 6: Report on Open Data and Historical Events Data Integration · IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data

ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies

Project:I-REACT ReportonOpenDataandHistoricalEventsDataIntegration

DeliverableID:D3.2

GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:6of29

1.3 ACRONYMSLISTAOI AreaofInterest API ApplicationProgrammingInterface ADRC AsianDisasterReductionCenter BAI BurnedAreaIndex BBK BundesamtfürBevölkerungsschutzundKatastrophenhilfe CLC CorineLandCover CRED Centre for Research on the Epidemiology of Disasters of the University of

LouvaininBrussels(Belgium)

CSV CommaSeparatedValues EEA EuropeanEnvironmentAgency ESA EuropeanSpaceAgency FAO FoodandAgriculturalOrganisationoftheUnitedNations FBK FondazioneBrunoKessler GEOSS GlobalEarthObservationSystemofSystems GHSL GlobalHumanSettlementLayer GPE GeopoliticalEntities IFRC InternationalFederationoftheRedCross JRC JointResearchCenter NBR NormalisedBurnRatio NDVI NormalisedDifferenceVegetationIndex OFDA OfficeofUSForeignDisasterAssistence OSM OpenStreetMap PBF ProtocolbufferBinaryFormat REST Representationalstatelesstransfer UNDP UnitedNationsDevelopmentProgramm UN-ISDR UnitedNationsInternationalStrategyforDisasterRiskReduction UN-OCHA TheUnitedNationsOfficefortheCoordinationofHumanitarianAffairs UA UrbanAtlas UNESCO UnitedNationsEducational,ScientificandCulturalOrganization USGS UnitedStatesGeologicalSurvey WMO WorldMeteorologicalOrganisation WMS WebMapService

1.4 REFERENCESID TitleChuviecoetal.,2002

Chuvieco,E,Martin,MP,Palacios,A,2002,Assessmentofdifferentspectralindicesinthered-near-infraredspectraldomainforburnedlanddiscrimination.Int.J.RemoteSensing,Vol23,No23,5103-5110.

Freireetal.2015

Freire,S,Halkia,M,2015,Towardsa100-m,GHSL-basedpopulationgridinEurope.ProceedingsofEFGS2015.

Huizingaetal.2017

Huizinga,J,deMoel,H,Szewczyk,W,2017,Globalflooddepth-damagefunctions.Methodologyandthedatabasewithguidelines.JRCTechnicalReports.EUR28552EN.

LopezGarciaandCaselles,1991

MJLopezGarcia,VCaselles,1991,MappingburnsandnaturalreforestationusingThematicMapperdata.GeocartoInternational,vol6,No1,31-37.

Sikkink2015 Sikkink,PG,2015,ComparisonofsixfireseverityclassificationmethodsusingMontanaandWashingtonwildlandfires.In:Keane,RobertE.;Jolly,Matt;Parsons,Russell;Riley,Karin.Proceedingsofthelargewildlandfiresconference;May19-23,2014;Missoula,MT.Proc.RMRS-P-73.FortCollins,CO:U.S.DepartmentofAgriculture,ForestService,RockyMountainResearchStation.p.213-226.

Page 7: Report on Open Data and Historical Events Data Integration · IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data

ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies

Project:I-REACT ReportonOpenDataandHistoricalEventsDataIntegration

DeliverableID:D3.2

GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:7of29

2 OPENDATA

2.1 BACKGROUNDABOUTOPENDATAINEUROPEPublic sector information is considered as one important resource supporting the Europeaneconomic growth and contributing to the development of a knowledge and innovation basedeconomy.Openpublicsectorinformationisalsoconsideredmakingthepublicadministration,itsworkflows and decisions more transparent. This allows citizens to contribute to governanceprocesses.Aspublicsectorinformationisusuallycreatedandmanagedbygovernmentagenciesitis also referred to as government data. In 2014, the European Commission published theCommunication‘Towardsathrivingdata-driveneconomy’.Itissketchingthefeaturesofthedata-driven economy of the future and setting out some operational conclusions to support andaccelerate the transition towards it. Data is considered being at the center of the futureknowledge economy and society. It is therefore necessary to facilitate the data exploitation, toreduce transaction costs and toharmonize the rulesondata re-use.Moreover, theprincipleof'openbydefault' and theneed tomakedata freely andopenly re-usableboth for humans andmachinesisstressed.

The open access to public data was already regulated on European level in 2001 throughRegulation (EC) No 1049/2001 regarding public access to European Parliament, Council andCommission documents.Main objective of this regulationwas to increase transparency and toallowcitizensparticipatingmorecloselyinthedecision-makingprocesses.WiththeregulationtheEuropeanCommissionconsidered that transparencyandopennesswould contribute to creatinggreater legitimacyof decision-makingprocesses and to strengthen theprinciples of democracy.TwoyearslaterDirective2003/98/EC,commonlyknownasPSIDirective,settherulesforthere-useofsuchpublicsectorinformationthroughouttheEuropeanUnion.

Since thenmanyopendata initiativeswere set-uponnational and international levels. Besidestransparencyandcitizenparticipation,thegreateconomicpotentialistodayinfocus.Publicdatahas significant potential for re-use in new products and services and havingmore data openlyavailablewillhelpdiscoveringnewandinnovativesolutions.

The I-REACTprojectmakesdifferentuseofandbenefits fromOpenData.Ontheonehand it isusedasdirectinformationsourceforend-users.Ontheotherhand,itisbasisfornewlydevelopeddataproductsandservicesbuiltthroughtheintegrationofopendatafromdifferentsources.Thefollowingreportprovidesanoverviewoftherelatedactivitiesinworkpackage3(WP3).

2.2 USEOFOPENDATAINI-REACT

2.2.1 DIRECTINTEGRATIONOFOPENDATAI-REACTaimsatbuildingnewdecisionsupporttoolsforcrisismanagementgoingbeyondstate-of-the art. In amap interface, different data are used to provide decisionmakerswith actionable

Page 8: Report on Open Data and Historical Events Data Integration · IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data

ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies

Project:I-REACT ReportonOpenDataandHistoricalEventsDataIntegration

DeliverableID:D3.2

GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:8of29

information. Available open data is used as background information and basis for analysis.Moreover,partnersuseitasinputtotheirprojectactivities.

Inaninitialstep,theEuropeanopendata landscapewasreviewedforsuitabledataanda listofavailablemap themes related to the I-REACT topicswere compiledandprovided to theprojectpartners.Ofspecialinterestweredatasets,whichpartnersofotherworkpackageswouldrequirefortheirdevelopments.DatasourcesthatwereinvestigatedincludedamongotherstheEuropeanINSPIRE portal, national open data portals, European Environment Agency (EEA), GEOSS, JointResearch Centre (JRC), and UNESCO. OpenStreetMapwas also investigated for its suitability toproviderelevantdata.Manygovernmentsandorganizationshavechangedtheirpoliciesregardingre-useofpublic information,whichhasresultedinasignificantincreaseofdataandinformationthatareeasilyaccessibleandopenlyre-usable.However,itisimportanttounderstandthatopendatadoesnotnecessarilycorrespondtofreeaccesstodata.Instead,governmentdata(e.g.largescale cadastre information) is still consideredas valuable economic resourceproviding a steadyincome to the administration. Moreover, charges can be applied for data handling. PositiveexamplestobementionedareSpain,FinlandandforpartstheUKhavingprogressiveopendatapolicies. In contrary it wasmore challenging to identify and obtain data from Italy andMalta.Through their personal local contacts some project partners could support this activity. Afterseveralfeedbackloopstheprojectteamchoseafinallistofdatasets.AspartofT3.3theselecteddatasets were collected from their sources, required areas of interest were extracted andprojectedintotheagreedcoordinatereferencesystems(EPSG3857).Table1showstheselecteddatasets.

Table1:SelectedOpenDatathemesforI-REACTuse

Name Description Source Format CDDA The European inventory of nationally

designated areas holds informationaboutprotectedareas.

EEA Vector(sqlite)

DEM Digitalelevationmodelsforall I-REACTAOIs(1m–5mspatialresolution)

NationalMappingAgenciesofES,UK,FI,MT.ITdataprovidedthroughARPA

Raster(tiff)

Criticalinfrastructures

Criticalinfrastructures OpenStreetMapandnationalsources

Vector(sqlite)

Natura2000 Natura 2000 network of protectedareas

EEA Vector(geojson)

Population Populationgrid JRC-GHSLandTerranea Raster(tiff)

Railways Railwaynetwork OpenStreetMap Vector(sqlite)

Rivers Rivernetwork OpenStreetMap,EEA Vector(sqlite)

Roads Roadnetwork OpenStreetMap Vector(sqlite)

UNESCOWorld UNESCOWorldHeritageSites UNESCO Vector

Page 9: Report on Open Data and Historical Events Data Integration · IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data

ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies

Project:I-REACT ReportonOpenDataandHistoricalEventsDataIntegration

DeliverableID:D3.2

GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:9of29

Name Description Source Format Heritage (geojson)

The data were made available to all project partners for further integration into their ownworkflows.SomeofthedatawasimportedtotheI-REACTORtobeavailableasstaticbackgroundlayer.

2.2.1.1 CRITICALINFRASTRUCTUREDATACriticalInfrastructuresarefacilitiesofhighimportancetosociety.Theirfailuree.g.duetoanaturaldisaster would lead to supply shortfalls and possible disturbance of public security or otherdramaticconsequences.OneimportantgoalrelatedtoT3.3wastocompileadatasetaboutcriticalinfrastructuresforallI-REACTtestareas.However,asthenameimplies,criticalinfrastructuresareof high importance for society functioning. Therefore, many countries do not provide relatedinformationasopendata. ItwasdecidedtouseOpenStreetMap(OSM)as initialdatasourcefortheextractionofinformationoncriticalinfrastructures.

AlistoftheGermanFederalAgencyforCivilProtection(BBK)definingcriticalinfrastructuretypeswas used as basis for selecting the themes of interest. Table 2 shows the themes that werechosen.

Table2:ThemesandrelatedsectorsselectedfortheI-REACTcriticalinfrastructuredataset

Sector Theme Administration Firebrigade,policestations Energy Powerplants Healthcare Hospitals,Pharmacies School Schools Telecommunication Telecomtowers Transportation Bridges,airport Water Watersupply,wastewater

OSM data is available in the native data format PBF (Protocolbuffer Binary Format). It isconsidered as an alternative to XML.Daily dumps of theOSMdatabase are e.g. available fromservice providers like Geofabrik (http://download.geofabrik.de/). The osm.pbf files of Geofabrikare un-filtered OSM and contain all data and metadata available in OSM. Some data isautomatically transferred into SHP format, a commonly used GIS vector format. The GeofabrikSHP-filescontainanumberofshapelayers.However,incontrasttothePBF-files,theSHP-filesarenot complete but only a selection of features and attributes ismade available. To fully benefitfromtheOSMinformationcontent,itisnecessarytoworkwiththe'raw'PBFdata.

Thefollowingfigure1showsaPBFdatasnippet. It includesgeographic informationofa feature(Lat/Loncoordinates),aversionnumberandtimestamp,theuserwhocreatedthedataandatag,describingthefeaturetype('power''tower').

Page 10: Report on Open Data and Historical Events Data Integration · IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data

ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies

Project:I-REACT ReportonOpenDataandHistoricalEventsDataIntegration

DeliverableID:D3.2

GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:10of29

Figure1:SampleofOSMdatainPBFformat

An OSM dataset includes points, lines, multi-lines and polygon features at the same time. Toextract relevant information, thedifferent featureshadtobequeried forspecificattributesandtagscontainingadditional information.Forexample,pipelinesbelongtothe linefeaturesetandare considered as <man-made> = "pipeline". A pole of a transmission line belongs to the pointfeaturesetandcanbeidentifiedunder<other_tags>="power"=>"tower".

To avoid manual queries of the large OSM database, automatic workflows were developed toextract and download the features of interest. Figure 2 shows a related model that wasimplementedinQGIS.Foreachcriticalinfrastructuretypesuchamodelwasset-up.Throughthisapproachseveralthousandindividualfeatureswereselectedforeachcountry(e.g.forUK231.980features).Figure3showsallpolicestationsandfirebrigadesintheUK.ThemodelscanbererunafterdefinedperiodstogatherOSMupdates.

Page 11: Report on Open Data and Historical Events Data Integration · IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data

ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies

Project:I-REACT ReportonOpenDataandHistoricalEventsDataIntegration

DeliverableID:D3.2

GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:11of29

Figure2:ModelschemetoautomaticallydownloadandextractOSMdata

OSMisacrowdsourcedproduct.Itsqualitybutalsolayerstatusdiffersduetothelargenumberofcontributors.Toharmonisethedataadditionalmanualprocessingwasrequired.Moreover,dataduplicates (in terms of attributes and geometries) were identified and cleaned. Due to theheterogeneityof theOSMdata itwasdecidedtotryto improvethe initialdatasetsthroughtheintegrationofnationaldatasets.Suitabledatawasfoundtoupdateandreplacetheoriginaldata:

• Spain:hospitalsandschools;

• Italy:firebrigadesandhospitals;

• UK:firebrigadesandschools.

ThedataoncriticalinfrastructureswasuploadedtotheI-REACTserverthroughtheIDI-interface.The data can be further used for risk assessments before and for damage assessments in theaftermathofadisastrousevent.

Page 12: Report on Open Data and Historical Events Data Integration · IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data

ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies

Project:I-REACT ReportonOpenDataandHistoricalEventsDataIntegration

DeliverableID:D3.2

GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:12of29

Figure3:OSM-derivedpolicestations(blue)andfirebrigadestations(red)intheUK

2.2.2 INDIRECTINTEGRATIONOFOPENDATABesides the direct use of open data as information layers it was also taken as input for thedevelopmentofadded-valuedataproducts.

2.2.2.1 DOWNSCALINGOFSTATISTICS–POPULATIONNational statistics provide a valuable data source for many sectors (population, agriculture,forestry, environment). However, statistics are often aggregated on larger administrative units(e.g.districts)beforepublication.Riskhowever,maychangeinmuchsmallerunits.Thistaskaimedat demonstrating the disaggregation and downscaling of publicly available statistics to spatiallyassesspeopleatriskatlargerscale.Moreover,itwasplannedtotesttheapproachforothertypesofstatisticstodeterminee.g.thepotentialfinanciallossofforestareasthroughawildfire.

The disaggregation of population statistics is a common approach to get detailed informationabout the spatial distribution of non-spatial census figures. Many different approaches existranging from simple areal interpolation to complex statistical modelling. Areal interpolationmethodsfollowazonetransformationapproachtransformingcensusdatafromonesetofspatialunitstoanotherapplyinginterpolationordisaggregationtechniques.Statisticalmodellingismore

Page 13: Report on Open Data and Historical Events Data Integration · IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data

ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies

Project:I-REACT ReportonOpenDataandHistoricalEventsDataIntegration

DeliverableID:D3.2

GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:13of29

complex aiming at inferring the relationship between population and other variables for thepurposeofestimatingthetotalpopulationforanarea.Thestatisticalmodellingisdataintensiveandusingsocioeconomicvariablesforpopulationestimation.

For the I-REACT purposes a dasymetric mapping approach was applied. It aims at transferringpopulation figures from a source zone (administrative unit) to a target zone (100m grid ofresidential areas). The approach followed a binary assumption, i.e. there exist populated andunpopulatedareasandcensusdatashouldberedistributedonpopulatedareasonly.Thismethodis robust and characterised through zone homogeneity. The population figures (counts) areproportionallyallocatedtothetargetzones.Nofurtherassumptionsliketheassignmentaccordingtobuilt-updensitiesweremade.Otherapproachesexistwherealltypesoflandcoverclasses(e.g.forest,agriculture,etc.)areconsideredbeingpopulated.Suchassumptionrequiresweightingsforeach land cover class which have to be either subjectively determined or through sampling oftraining areas. The appliedbinarydasymetricmapping is robust and applicable to all test areaswithoutrequiringspecificdatawhichmightbedifficulttoobtainforsomecountries.

Thedatarequiredforthedasymetricapproachwerecensusdatacollectedfromnationalstatisticalagencies,a spatialdatasetwithadministrativeunits forwhich thecensuswascollectedand theCorine LandCover (CLC) classes for residential areas (classes111 -Continuousurban fabric and112-Discontinuousurbanfabric).CLCisavailableat25haminimummappingunit.Incomparison,theCopernicusUrbanAtlas(UA)has10mresolutionandwouldallowmorepreciseallocationofpopulation data.However, theUA covers 695 functional urban areas and their surroundings. AlargepartoftheI-REACTAOIsarenotcoveredbyUA.ToavoidadatamergeofdatawithdifferentscalesitwasdecidedtoworkwithCLConly.

In a first processing step target and source zones were overlaid. This overlay allows theidentification of residential land cover polygons within each administrative source zone. Forassigning census figures, an areal weighted method was applied to each intersection zone asfollows:

WhereP is thepopulation,a is theareaandsubscriptssandt refer tosourceandtargetzonesrespectively.Withthisapproach,anyintersectionofatargetzonewithinasourcezonewillhavethesamepopulationdensityleadingtoahomogeneousdistributionofpopulationfigures.Forgridcells that intersect administrative boundaries, population figures were allocated based on theproportionoftheareaofeachunitlocatedinthegridcell(seefigure4).

Page 14: Report on Open Data and Historical Events Data Integration · IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data

ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies

Project:I-REACT ReportonOpenDataandHistoricalEventsDataIntegration

DeliverableID:D3.2

GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:14of29

Figure4:Allocationaccordingtoproportion.

Figure5showsasubsetoftheCatalonianpopulationgrid.Besidesgridsoftotalpopulation,twomorelayerswerecreatedcoveringthepopulationagegroupsof0-5yearsand>65years.

Figure5:AsubsetoftheCatalonianpopulationgridbasedondasymetricmapping

Asecondapproachfordownscalingpopulationfigureswasinvestigated.InputwastheEuropeanpopulationgridprovidedbytheJRC(GlobalHumanSettlementLayer-Population).Thedifferencetothefirstapproachis,thattheJRCpopulationdatasetassignspopulationfiguresderivedfromapopulationdatasetof Eurostat at 1 km2 resolution tobuilt-updensitiesderived from itsGlobalHumanSettlementLayer.Thus,aweightedpopulationdistributionisachieved.LandusedatafromtheUAandCLCwereusedasancillaryreferencetoweightpopulationaccordingtothefunctionalcharacteristics of the settlement area. No further distinction between residential and non-residentialareasaremade.Moreover,itisassumedthatallbuilt-upareaswillbeinhabited(Freireetal.2015).

TheoriginalJRCpopulationgridrepresentingthetotalpopulationwastakenasinput.Populationstatisticsaccordingtoagegroups(0-5yearsand>65years)peradministrativeunitswerederivedfromnationalcensus.Theagegroupproportionsinrelationtopopulationtotalswereappliedtothe original population grid. Figure 6 shows a subset of the second population disaggregation

Page 15: Report on Open Data and Historical Events Data Integration · IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data

ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies

Project:I-REACT ReportonOpenDataandHistoricalEventsDataIntegration

DeliverableID:D3.2

GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:15of29

approach based on JRC's total population grid over Catalonia. The figure shows that thepopulationfiguresarenothomogeneouslyspreadbutweightedaccordingtothevariablebuilt-updensity.

Figure6:AsubsetofJRC'stotalpopulationgridofCatalonia

Both disaggregation approaches are valid methods having advantages and disadvantages. ThepopulationgridswereprovidedasinputtotheriskmodellingactivitiesinWP4.

2.2.2.2 DOWNSCALINGOFSTATISTICS–FORESTSTATISTICSAnotherusecaseaimedatdownscalingforeststatisticstodeterminethepotentialfinanciallossofforestareasthroughawildfire.ThisapproachwastestedforaGermansite.TheapproachaimedatcollectingpriceinformationabouttreespeciesaswellasthedistributionofrelatedtreespeciesandtodisaggregatethepricevaluestoahomogeneousgridovertheCLCforestclasses.Incaseofawildfire,thepotential financial lossaccordingtotheexistingtreespeciescouldbedeterminedthroughsuchapproach.

In a first step wood prices were collected from the Bavarian Ministry of Agriculture. Due tochangingpricesoverayearanaveragepricewastaken.However,thelimitingfactorwasthatthewoodpriceswereprovidedpercubicmeters,only.Sincetherewasnospatialdataabout forestvolumes available, a literature search provided simple relationship classes about typical forestvolumes for each forest type in Bavaria (e.g. 1 ha coniferous corresponds to 300m3 ofwood).Thus, thewoodpriceper km2 couldbe calculatedandassigned to relatedCLC classes. Figure7showstheresultingforestvaluegridwithmaximumforestpricesof€1.725.360.

Page 16: Report on Open Data and Historical Events Data Integration · IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data

ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies

Project:I-REACT ReportonOpenDataandHistoricalEventsDataIntegration

DeliverableID:D3.2

GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:16of29

Dueto theunavailabilityofhomogeneouswoodpricedata forEurope (Eurostat),andespeciallymissingforestvolumeinformationitwasdecidedtonotfurtherfollowtheideaofforestvaluation.Moreover,duringdiscussionswithforestersitwashighlighted,thatwoodpriceschangenotonlyin relation to tree species but also to the quality and intended use of the wood. High qualityveneersarecostlierthanwoodusedfortheproductionofwoodchips. Ifatall,dataaboutsuchdifferentiationisonlyavailablefromlocalforestinventories,whicharedifficulttoobtainforlargeareas.

Figure7:Forestvaluegridbasedonforesttypeandwoodprices.Pricerangesfromlightgrey(lowprice)todarkgrey(highprice),blackarebackgroundcells.

Thetestcasesdemonstratedthepossibilitiesandlimitationsofspatiallydisaggregatingstatisticaldata.Suchapproachesarealsoappliedbyinsurancesfordownscalingriskrelatedinformationthatisaggregatede.g.perpostalcodeareastofinergranularity.Nevertheless,ameaningfulresult ismainlydependentontheavailabilityofsuitablestatistics.

2.2.2.3 FLOODVULNERABILITYLAYERRiskisunderstoodasafunctionofhazard,vulnerabilityandexposure.Whilehazardandexposuredata is often available at reasonable scale, information about vulnerability is lacking. Damagecurvesarebuilt toevaluate the relationshipbetweenhazard intensitiesand relateddamages.Atypical example is a damage curve for earthquake intensities related to building types andconstructionmaterials.

Many countries have developed flood-related damage curves based onwater depth. However,they are not available for all regions.Moreover, due to differentmethodologies employed for

Page 17: Report on Open Data and Historical Events Data Integration · IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data

ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies

Project:I-REACT ReportonOpenDataandHistoricalEventsDataIntegration

DeliverableID:D3.2

GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:17of29

variousdamagemodelsindifferentcountries,damageassessmentscannotbedirectlycomparedwitheachother.TheJRCaddressedtheseproblemsand,basedonanextensiveliteraturesurvey,developed a globally consistent database of depth-damage curves describing fractional damagefunctionsofwaterdepthandmaximumdamagevaluesforavarietyofassetsandlanduseclasses(Huizingaetal.2017).Tosupportflood-relatedriskassessments,itwasdecidedtodevelopspatialflood vulnerability layers based on JRC's damage curves integrated in land cover data. Table 3exemplifiesthemaximumflooddamagesfortheUnitedKingdom.Dependingontheflooddepth,thedamagevaries.Thedamagevaluesarealsoavailableforlanduseclasses.CLCwasthebasisforassigningrelateddamagepotentialspercountry.

Table3:MaximumdamagesperlanduseintheUK,accordingtoJRCGlobalflooddepth-damagefunctions

Residential Commercial Industrial Agriculture Transport 152€/m2 316€/m2 257€/m2 655€/ha 688€/m2

Thedevelopmentofthefloodvulnerabilitylayerwasdecidedonlyinsummer2017.Therefore,thefinallayersarestillindevelopment.

2.2.2.4 BURNEDAREAMAPPINGAnotheraimofT3.3wasthetimeseriesanalysisandfrequentupdateof landcover informationderivedfromSentinel-2data.Afterseveraldiscussionsanditerations,theprojectteamdecidedtomodifythetasksincetherewasnorealuseoffrequentlandcovermaps.Instead,itwasdecidedtodevelopaservicefortheautomatedmappingofburnedareasintheaftermathofwildfireevents.Wildfirescanaffectlargeareasthatneedtobemappedinordertoassessandevaluateeconomicand environmental damages, to model trace gas emissions, and to plan for the landscaperecovery.Burnedareamapsarealsoimportanttofeed-backintothewildfirenowcastandforecastmaps.Furthermore,toprepareandmitigateforthefutureitisnecessarytolearnfrompasteventsinordertounderstanddriversandrisksandtoidentifyactionablerecommendationstoenhancedisaster resilience. Remote sensing is the standard tool used to delineate the affected burnedarea.Aspartof theMODISActiveFireandBurnedAreaProducts,burnedareasaremappedatglobal scale investigating spectral, temporal, and structural changes. The application of variousspectral indices such as theNormalisedDifferenceVegetation Index (NDVI), BurnedArea Index(BAI),andNormalizedBurnRatio(NBR)havebeenwidelyusedtomonitorfire-inducedvegetationchanges.

The development of the burned area mapper was decided only in summer 2017. Itsimplementation is still under development. The service will make use of the large spatialresolutionofSentinel-2datathatwillboosttheaccuracyofthecurrentMODISproductat250mspatialresolution.Theburnedareamappingwillbeautomatedandtriggeredassoonasawildfireisterminated.Amessageissentfromtheservicebusincludingtheindicativefirelocation.Basedon the location information the search for available post-event Sentinel-2 imagery on theCopernicus Open Access Hub of the European Space Agency (ESA) is activated. In case severalimagesareavailable,thosewithlowestcloudcoverageareselectedandonlytherequiredbands

Page 18: Report on Open Data and Historical Events Data Integration · IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data

ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies

Project:I-REACT ReportonOpenDataandHistoricalEventsDataIntegration

DeliverableID:D3.2

GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:18of29

are downloaded to the cloud processing backend. After an atmospheric correction and cloudmaskingtheburnedareaindex(BAI)iscalculatedassecondstep.BAIisbasedonthereflectanceintheRedandNear-Infraredchannelsandcalculatedasshowninthefollowingformula(Chuviecoetal.,2002):

IncomparisontotheBAItheNormalizedBurnRatio(NBR)usesnear-infrared(NIR)andshortwave-infrared (SWIR) wavelengths to discriminate between burned and unburned vegetation (LopezGarciaandCaselles,1991).TestsarecurrentlyperformedtoevaluatetheoutputqualityofBAIandNBR to finally decide about the index to be applied for the automated mapping. To finallydistinguishbetweenburnedandunburnedareas, thresholdshave tobeapplied to the resultingBAI output. Finding an adequate threshold is still subject to current test runs. To remove smallisolatedpixelsamajorityfilterisapplied.Arastertovectorconversionfollowedbyasmoothingofthepolygonoutlinecompletestheprocessingchain.ThefinalresultisreturnedtotheI-REACTOR.Figure8showsthegeneralprocessingworkflow.

Figure8:BurnedAreaMappingworkflow

Theprocessingchainiscurrentlyimplemented.ItisforeseentovalidatethemappedburnedareasusingreferencedataprovidedbytheCopernicusEmergencyManagementserviceoftheEuropeanCommission.Figure9showsanexampleofaderivedburnedarearesulting froma forest fire inPortugalin2017.

Page 19: Report on Open Data and Historical Events Data Integration · IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data

ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies

Project:I-REACT ReportonOpenDataandHistoricalEventsDataIntegration

DeliverableID:D3.2

GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:19of29

Figure9:BurnedAreaMapproducedusingCopernicusSentinel-2imagery.Theblackoutlineenclosestheburnedarea.Theblackenedarearepresentscharredvegetationinthepost-firefalse-colorimage(channels8,4,3).

Wildfires do not always destroy the vegetation cover completely. It is therefore necessary toassesstheseverityoftheimpacttoplantheremediation.Suchevaluationcanbecarriedoute.g.throughgroundtruthing.Remotesensingprovidesanotheropportunitycombiningpre-andpost-fire imageryandapplying theNormalizedBurnRation (NBR) index.TheNBRusesNIRandSWIRinformationforthecalculationoftheindex.Pre-andpost-firedatahavetobeprocessedbeforedeterminingtheΔNBR(dNBR)throughthedifferencebetweenthepre-andpost-fireNBR.Finally,dNBR is classified according to the United States Geological Survey (USGS) standard for BurnSeverityassessment(Sikkink2015).Figure10showsanexampleofaburnseveritymapprocessedafterthelargeCalifornianwildfiresinOctober2017.

Page 20: Report on Open Data and Historical Events Data Integration · IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data

ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies

Project:I-REACT ReportonOpenDataandHistoricalEventsDataIntegration

DeliverableID:D3.2

GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:20of29

Figure10:ExampleofaBurnSeveritymapfromtheCalifornianwildfires,October2017

Theautomatedburnseverityassessmentwillbeimplementedasasecondstepintheburnedareamappingworkflow.

2.3 INTERFACESThe open data products were integrated in the I-REACTOR through their upload via the IDI-interface.PopulationgridswerealsoprovideddirectlytothepartnerFBKtobeusedintheirriskmodellingtask.

The burned areamapping service is still under development and the final service logic (trigger,execution, upload of results) will be defined together with the project partners during theupcomingmonths.

Page 21: Report on Open Data and Historical Events Data Integration · IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data

ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies

Project:I-REACT ReportonOpenDataandHistoricalEventsDataIntegration

DeliverableID:D3.2

GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:21of29

3 HISTORICALDATA

3.1 HISTORICALHAZARD/DISASTEREVENTDATASETSTheknowledgeabout the locationsofhistoricalhazardanddisastereventsand their impacts iscrucialtoimproveriskassessmentsandmitigationefforts.Moreover,historicaldatacanbeusedto validate riskmodels and is important information for insurances to evaluate premiums. ForfloodstheneedforunderstandinghistoricaleventsisunderlinedbytheEuropeanFloodsDirective(2007/60/EC) that requires also the review of past flood events as part of the overall aim toidentifyareasatriskofflooding.Whileseveralcountriesrecordfloodeventsandtheirimpacts,thecollected information and sources are very heterogeneous. Besides national sources, researchinstitutions collect hazard event data and sometimes share it as open data. Often, such datacollections are project based and not part of a sustainable process. Large insurances and re-insuranceshave similar services running.However, their data is usually for internal use and forclients'useonly.

TheHistoricalEventsDatabaseof I-REACTaimsat collecting informationaboutpastdisasters inEuropefromopendatasources.Theinformationwillhelpidentifyingnaturalhazardhotspots,toenhanceforecastandnowcastmodels,andtovisualize,communicateandmanagenaturalhazardrisks better.While the initial objective included only data about historical European flood andwildfireeventsitwasdecidedtocoveralsootherhazardtypesforamorecomprehensivepicture.Theactivitiesincluded:

• Identificationofavailableopendatasources

• Geocodingofthehazardevents

• Definitionofadatamodelanddataharmonization

• Integrationintoacommondataset

• IntegrationintoI-REACTOR

• Designofaweb-interfaceforexploringandaccessingthecatalogue

Thefollowingsectionsprovidemoredetailsaboutthesetasks.

3.1.1 IDENTIFICATIONOFAVAILABLEOPENDATASOURCESAn intensive web search was conducted to identify possible open data sources about hazardevents. Severalpublicdatabasesexist,providing relevant information.However,notallof themallow commercial reuse. The following databaseswere identified (Table 4) and chosen asmaininput.

Page 22: Report on Open Data and Historical Events Data Integration · IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data

ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies

Project:I-REACT ReportonOpenDataandHistoricalEventsDataIntegration

DeliverableID:D3.2

GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:22of29

Table4:NaturalhazardeventdatabasesusedasinputfortheI-REACThistoricaleventsdataset

Name Hazard Owner License CopernicusEMS Allhazards JRC Opendata DartmouthFloodObservatory

Floods UniversityofColoradoBoulder

CreativeCommons3.0

EEAFloodsDatabase Floods EEA public EuropeanForestFireInformationSystem

Wildfires ECJRC public

EuropeanHistoricalFloodDatabase

Floods EOXPLORE&Terranea backgroundIPtotheproject

ExtremeWindStormsCatalogue

Windstorms MetOffice,UniversityofReadingandUniversityofExeter

CreativeCommonsCCBY4.0InternationalLicence

GLIDE Allhazards AsianDisasterReductionCenter(ADRC)

public

SmithsonianInstitutionGlobalVolcanismProgram

Volcanoes SmithsonianInstitution–NationalMuseumofNaturalHistory

publicwebsite

USGSEarthquakeHazardsprogram

Earthquakes USGS publicdomain

Thedatabasesapplydifferentcriteriaforintegratingevents.Moreover,someofthemareupdatedregularly while others (e.g. EEA flood database) is currently a one-time product (status 2017).Besidesthementioneddatabases,additionalnationalsourceswereinvestigatedandfloodeventdatasetsfromtheUK,FinlandandItalycouldbefound.

3.1.2 GEOCODINGOFTHEHAZARDEVENTSManyoftheavailabledatasetsincludegeospatialinformation.However,especiallyinthecaseoffloods,oftenonlynamesofflood-affectedplacesareavailable.ThoseplacenamesweregeocodedusingtheGoogleMapsgeocodingservice.ArelatedAPIprovidesaccesstotheservicethroughanHTTP-request.Whenaplace is identifiedalocation isreturned informofacoordinatepair.Thestandard API allows the geocoding of 2.500 addresses per day. Since amuch larger number ofplacenameshadtobegeocoded,thebillingservicehadtobeused.

TheEEAFloodDatabaseturnedouttobeadatasetwheremanydifferentnationaldatasourceswere copied into a single file. However, the national sources followeddifferent approaches forfloodrecording.Insomecases,riversections(distancefrommouth)wereprovidedaslocations.Inother cases, watershed names or only flood names or dates (e.g. January flood 2005) wereavailable.Manyoftheseeventscouldnotbegeolocated,yet.Otherexamplesexistwhereplaceswerewrongly geocoded and placed inwrong countries or even continents. Such errors relatedeither to realexistingplacenames in the relatedcountriesormisspellingsand foreign languageplacenames.Someoftheseerrorscouldbesolvedthroughmanualintervention.

Page 23: Report on Open Data and Historical Events Data Integration · IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data

ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies

Project:I-REACT ReportonOpenDataandHistoricalEventsDataIntegration

DeliverableID:D3.2

GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:23of29

3.1.3 DEFINITIONOFADATAMODELANDDATAHARMONIZATIONAfterevaluationoftheoriginaldatasetsandincollaborationwiththeprojectpartnersacommondatamodelwasdefined (seeTable5). It canbeconsideredas reasonablecompromisecoveringthemajorityofrequiredinformationthatisavailablefromallsources.

Table5:DatastructureoftheHistoricalEventsdataset

Field Description Field Description

ID UniqueID Type Hazardtype

Countrycode 2-digitISOcode Areaaffected(km2) Impactedarea

Countryname Countryname Peopleaffected Noofpeopleaffected

Province Provincename Peoplekilled Noofpeoplekilled

Location Placename Estimateddamage Estimateddamage

Start Date Source ofinformation

Sourceofinformation

End Date Comment Anycomment

X_COORD[EPSG:3857]

Coordinate Y-COORD[EPSG:3857]

Coordinate

Especiallytheinformationabouttheimpactedarea,numberofpeopleaffectedandkilledaswellastheestimateddamageswererarelyavailablefromthedatasources.

Duetothedifferentoriginaldatastructuresalargeeffortwasrequiredtocleanandharmonisethedatabefore itcouldbe integrated intoadatabase.Spellingmistakesandclericalerrorsofplacenames had to be corrected. They probably derived from a manual data input of an operator.Datasets were available in different languages. French and Bulgarian (in Cyrillic) could lead togeocoding problems and some places were not identified. Other information like country andprovincenameswasdeterminedthroughaGISintersectionwithaspatial layerofadministrativeboundaries.

3.1.4 INTEGRATIONINTOACOMMONDATASETAll pre-processed and geocoded input datasets were finally integrated into a single file. It iscurrentlyavailableinCSVformatbutitistheaimtoset-upadatabaseforthispurposeinthenearfuturetoallowamoreprofessionaldatahandling.Around35.000dataentriescouldbeintegratedsofar.

All event data was uploaded to the I-REACTOR backend using its REST-API and theEmergencyEventfunction.

Page 24: Report on Open Data and Historical Events Data Integration · IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data

ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies

Project:I-REACT ReportonOpenDataandHistoricalEventsDataIntegration

DeliverableID:D3.2

GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:24of29

3.2 DISASTEREVENTTRACKERThe initialaimof task3.7was tocollectonlydataabouthistoricaldisasterevents fromexistingdatasets. As shown above, several of such datasets exist and are accessible as open data.However, the update cycle of these datasets is usually slow. In some cases it is not clear anddocumented,ifupdateswillbemadeavailableatall.Itwasthereforedecidedtoaddanadditionalcomponent to the task and to monitor relevant information channels about newly reporteddisasters. As these events will be historical in the future, they can be recorded and the eventdataset thus automatically updated avoiding also repeating the laborious data processing fromdifferentsources.

Threedifferentdatasourcesforinformationaboutdisastereventswereidentified:

• ReliefWeb (https://reliefweb.int) isa serviceprovidedbyUNOCHA. It isa leadingonlinesource for reliable and timely information on global crises and disasters. Addressing thehumanitarianaidcommunityasmainusersReliefWebprovidesdisasteranddevelopmentrelatedinformation,includingreports,maps,infographicsandvideosfromtrustedsources.ReliefWeboffers all its content throughanAPI includingalso informationaboutdisasterevents.

• GLIDE (http://glidenumber.net): accessing disaster information can be a time consumingand laborious task. Not only is data scattered but also frequently identification of thedisastercanbeconfusingincountrieswithmanydisasterevents.Toaddressbothoftheseissues,theAsianDisasterReductionCenter(ADRC)proposedagloballycommonuniqueIDcodefordisasters.ThisideawassharedandpromotedbytheCentreforResearchontheEpidemiology of Disasters (CRED) of the University of Louvain in Brussels (Belgium),OCHA/ReliefWeb,OCHA/FSCC,ISDR,UNDP,WMO,IFRC,OFDA-USAID,FAO,LaRedandtheWorld Bank and was jointly launched as a new initiative "GLIDE". On the website newdisaster events are registered and related event information is collected. Events can bequeriedbutthewebsitedoesnotprovideanAPIfordataaccess.Though,thedatabasecanbedownloaded.

• ECHOFlash isaserviceprovidedbytheEmergencyResponseCoordinationCentreof theEuropean Commission (http://erccportal.jrc.ec.europa.eu/ECHO-Flash). The websitepublishes daily news about disaster and crisis eventsworldwide. Thewebsite is in htmlformat.NoAPIorotherdownloadinterfaceexists.

TheeventtrackervisitsthethreewebsitestogatherrecentnaturaldisastereventsinEurope.Itiscapableoffilteringandreducingdescriptiontexts,findgeopoliticalentities(GPE)andretrievegeo-coordinates, provided through the Google Geocoding-API. The event tracker can be executedmanuallyor run fullyautomatically.Aweb-scrappingparthandlesall informationaccesspoints,stored as string URLs. Since each of the current implemented services has to be hailed by adifferent interface,aheadlessRSS-feed reader (DailyFlash),aheadlessbrowserwith included JSsupport(GlideDB)andregularAPIrequests(ReliefWeb)wereusedforinformationretrieval.

Page 25: Report on Open Data and Historical Events Data Integration · IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data

ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies

Project:I-REACT ReportonOpenDataandHistoricalEventsDataIntegration

DeliverableID:D3.2

GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:25of29

Since the emergency events are uploaded to the I-REACTOR database by REST-API access theinformationhastobesortedafterretrievalaccordingtotheprovidedREST-APIspecificationsforthefinalPOSTrequest.Duetothevaryingsourcetypes,eachoftheserviceshastobetreatedinits uniqueway.When all the information has been sorted the description or comment can befilteredforusefulinformation.Moreover,theeventdescriptionsaresearchedforGPEsandaddedto the location field. The natural text-processing library Spacy provides the backgroundarchitecture for determining additional locations. A final PostObject, holding a single event iscompletedandstoredinaPostObjectCollectionwithalltheotherevents.TheGooglegeocoding-APIisusedtoretrievecoordinatesofallGPEs.

The I-REACTOR REST-API is accessed as described in its documentation(https://bitbucket.org/mobilesolutionsismb/ireact-rest-api). After authentication the gatheredPostObjects are pushed by POST request as json dictionaries. An error handling and re-authorizationhasbeenimplementedaswell.

3.3 INTERFACESBesidesthedata integration intothe I-REACTOR,task3.7 includesthedesignofaweb-interfaceforexploringandaccessingthecatalogue.Twoprototypesweredevelopedforthatpurpose.Thefirst webviewer visualises all historical European forest fires from EFFIS(http://wildfire.terranea.de/).Eventscanbequeriedbycountry,dateandaffectedarea.Foreachfire,availableattributesareprovided.Dependingonthechosenzoomlevel,eventsareshownaspointfeaturesorpolygonfeatures.OpenStreetMapandGoogleMapsareavailableasbackgroundlayers. The Copernicus HRL forest typemap is integrated through aWMS service.Moreover, aforestfirefrequencymapwascreatedsummarisingallforestfireswithin1km2gridcells.Figure11showsascreenshotoftheforestfireviewer.

Figure11:WebinterfacetodisplayandqueryhistoricalwildfireeventsinEurope.

Page 26: Report on Open Data and Historical Events Data Integration · IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data

ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies

Project:I-REACT ReportonOpenDataandHistoricalEventsDataIntegration

DeliverableID:D3.2

GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:26of29

Asecondwebsitewasdevelopedtovisualisetheevents,collectedwiththeDisasterEventTracker.TheeventsstoredinaMongodatabasewererenderedusingtheinteractivevisualizationPythonlibraryBokeh,whichisusefulforquicklyandeasilycreatinginteractiveplots,dashboards,anddataapplications.Thewebsitevisualisestheeventsaspointsonamap.Theycanbequeriedaccordingtoeventtype,countryanddate.Figure12showsthewebinterfaceoftheDisasterEventTrackingservice.

Figure12:PrototypewebinterfacefortheDisasterEventTrackingservice

4 CONCLUSIONS

During the past years, many government organisations opened their data infrastructures andpublishedtheirdataunderopendatapolicies.OnEuropeanlevel,thereexistevenregulationsfordisclosingenvironmentrelatedinformation.However,thewaytoimplementregulationsisusuallynotstandardised.Therearenationalandregionaldisparitiesinopendataapproaches.Whilesomemembercountriesareproactiveothersareratherreluctantandconsidergovernmentdataratherasasecretoraspossiblesourceof income.Thus,datausersaimingat integratingthedata intotheir analysis and services still face challenges, be it through fragmented data sources, non-standard interfaces, non-machine-readable formats, and diverse license conditions. Open datadoesnotautomaticallymeanthatitisfreelyaccessible.Sometimes,datahandlingorservicefeesarecharged.Moreoverthequalityofopendata,evenfromgovernmentsources,canbepoorandtheaccuraciesarenotalwaysdocumented.

Nevertheless,theI-REACTtasks3.3onopendataand3.7onhistoricaleventsdemonstratedthatinnovative services and data products can be developed using open data. These products gobeyondpureresearchbuthavethepotentialtobeofferedascommercialservicestoend-users.

Page 27: Report on Open Data and Historical Events Data Integration · IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data

ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies

Project:I-REACT ReportonOpenDataandHistoricalEventsDataIntegration

DeliverableID:D3.2

GrantAgreementNo:700256 CallID:H2020-DRS-1-2015 Page:27of29

5 LISTOFOPENDATASETS5.1 LISTOFOPENDATASETS

Theme Source Coverage Spatialresolution

UpdateFrequency

Format URL License

HazardMapAvailability

UK England unclear quaterly WMS,WFS,SHP,GMS,TAB

https://data.gov.uk/dataset/flood-map-for-planning-rivers-and-sea-flood-zone-2

open

FI

ES ES unclear unclear SHP,WMS http://aca-web.gencat.cat/aca/appmanager/aca/aca?_nfpb=true&_pageLabel=P41800177491338804489122

open

MT noinfo noinfo noinfo noinfo noinfo noinfo

Risk UK England unclear asneeded WMS,WFS,SHP,GMS,TAB

https://data.gov.uk/dataset/flood-risk-areas open

CriticalInfrastructure

OSM Global unclear,candiffer

unclear,very

irregular

PBF,SHP http://download.geofabrik.de/ open

Nationaldatasources

UK, ES,FI,MT,IT

unclear,candiffer

Demographic

JRC Global 250m unclear geotiff http://ghsl.jrc.ec.europa.eu/ghs_pop.php open

DigitalElevationModel

EUDEM EEA38 25m unclear Tiff,WMS http://land.copernicus.eu/pan-european/satellite-derived-products/eu-dem/eu-dem-v1.1/view

open

UKnational

UK 0.25m,1m,2m

unclear ASCII,JPG,WMS

http://environment.data.gov.uk/ds/catalogue/index.jsp#/catalogue open

ES ES 2m unclear ASC http://centrodedescargas.cnig.es/CentroDescargas/buscadorCatalogo.do?c open

Page 28: Report on Open Data and Historical Events Data Integration · IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data

ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies

Project:I-REACT ReportonOpenDataandHistoricalEventsDataIntegration

DeliverableID:D3.2

GrantAgreementNo:700256 CallID:H2020-DRS-1-2015 Page:28of29

national odFamilia=MDT05

FInational FI 2m unclear Asciigrid,GeoTiff

http://www.maanmittauslaitos.fi/en/digituotteet/elevation-model-2-m open

Natura2000sites

EEA EEA38 1:100000 irregular/annual

shp,spatialite

open

Nationallydesignatedareas(CDDA)

EEA EEA38

irregular/annual

shp,spatialite http://www.eea.europa.eu/data-and-maps/data/nationally-designated-areas-national-cdda-11#tab-gis-data

open

Rivernetworks EEA EEA38 1:250000 irregular mdb(personalgeoDB)spatialite

http://www.eea.europa.eu/data-and-maps/data/european-catchments-and-rivers-network#tab-gis-data

open

OSM Global unclear unclear,very

irregular

PBF,SHP

open

CopernicusUrbanAtlas

2012

EEA EEA38 100mgrid every5years

SHP,WMS http://land.copernicus.eu/local/urban-atlas/urban-atlas-2012/view open

CopernicusForestType

EEA EEA38 100mgrid every5years

Tiff,WMS http://land.copernicus.eu/pan-european/high-resolution-layers/forests/forest-type/view

open

CopernicusImperviousness

2012

EEA EEA38 100mgrid every5years

Tiff,WMS http://land.copernicus.eu/pan-european/high-resolution-layers/imperviousness/view

open

CorineLandCover2012

EEA EEA38 100m/250mgrid

every5years

Tiff,WMS http://land.copernicus.eu/pan-european/corine-land-cover/clc-2012/view open

CopernicusRiparianZonesLandCover

EEA EEA38 100mgrid every5years

Shp,WMS http://land.copernicus.eu/local/riparian-zones/land-cover-land-use-lclu-image/view

open

UNESCOWorldHeritagesites

UNSESCO/Geonue/Wikipedia

Global unlear(pointdata)

unclear Shp,GML,CSVGeoJson,KML

http://www.geonue.com/unesco-world-heritage-sites-interactive-map/https://en.wikipedia.org/wiki/World_Heritage_Site

open

Table6:Listofidentifiedandpossiblyusefuldatasets.

Page 29: Report on Open Data and Historical Events Data Integration · IMPROVING RESILIENCE TO EMERGENCIES THROUGH ADVANCED CYBER TECHNOLOGIES Report on Open Data and Historical Events Data

ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies

Project:I-REACT ReportonOpenDataandHistoricalEventsDataIntegration

DeliverableID:D3.2

GrantAgreement:700256 CallID:H2020-DRS-1-2015H2020-DRS-1-2015 Page:29of29

ENDOFTHEDOCUMENT