Upload
others
View
6
Download
0
Embed Size (px)
Citation preview
IMPROVINGRESILIENCETOEMERGENCIESTHROUGH
ADVANCEDCYBERTECHNOLOGIES
“ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID D3.1
WorkPackageReference WP3
Issue 1.00
DueDateofDeliverable 30/11/2017
SubmissionDate 15/11/2017
DisseminationLevel1 PU
LeadPartner GeoVille
Contributors EOXPLORE,CSI,TUW
GrantAgreementNo 700256
CallID H2020-DRS-1-2015
FundingScheme Collaborative
I-REACT is co-fundedby theHorizon2020FrameworkProgrammeof theEuropeanCommissionundergrantagreementn.700256
1PU=Public,PP=Restrictedtootherprogrammeparticipants(includingtheCommissionServices),RE=Restrictedtoagroupspecifiedbytheconsortium(includingtheCommissionServices),CO=Confidential,onlyformembersoftheconsortium(includingtheCommissionServices)
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:2of84
Preparedby Reviewedby Approvedby
W.Stemberger C.Klug C.Rossi
Issue Date Description Author(s)
0.01 13/10/2016 Setupoftemplate W.Stemberger
0.1 10/11/2016 Tableofcontents F.Nagl
0.2 26/03/2017 Provisionofcontentsforchapter2 F.Nagl
0.3 07/11/2017 Provisionofcontentsforchapter5 I.Ali
0.4 08/11/2017 Provisionofcontentsforchapter2 F.Nagl,M.Schwandner
0.5 09/11/2011 Provisionofcontentsforchapter4 M.Leonelli,M.Velluto
0.6 13/11/2017 Provisionofcontentsforchapter3 C.Bielski
1.0 13/11/2017 Reviewandcorrections C.Klug
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:3of84
TABLEOFCONTENTS1 INTRODUCTION...........................................................................................................................9
1.1 PurposeoftheDocument....................................................................................................9
1.2 StructureoftheDocument..................................................................................................9
1.3 AcronymsList.....................................................................................................................10
1.4 ReferenceandapplicableDocuments...............................................................................10
2 COPERNICUSEMSINTEGRATION..............................................................................................12
2.1 ServiceOverviewofCopernicusEMS................................................................................12
2.1.1 RapidMapping...........................................................................................................14
2.1.1.1 ReferenceMaps..................................................................................................15
2.1.1.2 DelineationMaps................................................................................................16
2.1.1.3 GradingMaps......................................................................................................17
2.1.2 RiskandRecoveryMapping.......................................................................................19
2.1.2.1 ReferenceMaps..................................................................................................19
2.1.2.2 Pre-disasterMaps...............................................................................................19
2.1.2.3 Post-disasterMaps..............................................................................................19
2.2 CopernicusEMSDataPortal..............................................................................................20
2.2.1 OverviewonDataProvision.......................................................................................22
2.3 CopernicusEMSIntegration..............................................................................................24
2.3.1 DesignofEMSIntegrationModule............................................................................25
2.3.2 ImplementationofEMSIntegrationModule.............................................................26
3 EUROPEANEARLYWARNINGSYSTEMSINTEGRATION.............................................................29
3.1 ServicesOverviewoftheCopernicusEWS........................................................................29
3.1.1 TheEuropeanFloodAwarenessSystem(EFAS).........................................................29
3.1.2 TheEuropeanForestFireInformationSystem(EFFIS)...............................................30
3.2 CopernicusEFASDataPortal.............................................................................................33
3.3 CopernicusEFASIntegration.............................................................................................34
3.4 I-REACTprocessingofEFASData.......................................................................................35
3.4.1 AffectedCommunitiesBasedontheEFAS5-yearReturnPeriodForecast................36
3.4.2 AffectedCommunitiesBasedontheEFAS20-yearReturnPeriodForecast..............39
3.5 CopernicusEFFISDataPortal.............................................................................................42
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:4of84
3.6 CopernicusEFFISDataPortal.............................................................................................43
3.6.1 TheFireWeatherIndex..............................................................................................43
3.6.2 AffectedCommunitiesBasedontheFWI...................................................................44
3.6.3 AffectedCommunitiesBasedontheFWI–NUTSlevel3...........................................47
3.6.4 FireHotSpotDetectionacrossEurope......................................................................47
3.6.5 AffectedCommunitiesBasedontheVIIRSfirehotspotdata.....................................48
3.7 Discussion..........................................................................................................................52
4 EXISTINGLOCALEMSINTEGRATION.........................................................................................53
4.1 OverviewofLocalEMS......................................................................................................53
4.2 SurveyofexistingEmergencyManagementSystems.......................................................55
4.2.1 Finland........................................................................................................................55
4.2.2 Italy.............................................................................................................................56
4.2.3 Spain...........................................................................................................................60
4.2.4 UnitedKingdom..........................................................................................................61
4.3 DefinitionandDevelopmentofIntegrationServices........................................................63
5 SENTINELDATAPROCESSINGINTEGRATION............................................................................68
5.1 Satellitedataprocurement................................................................................................68
5.1.1 Dataandsensorspecifications...................................................................................69
5.1.2 Datastoragesystem...................................................................................................70
5.1.3 Datacoverageandavailability....................................................................................71
5.1.4 Historicalremotesensingdata...................................................................................72
5.1.5 EODCprocessingenvironmentandplatform.............................................................72
5.1.6 EODCdatastorageandSentinel-1dataflow..............................................................73
5.2 Sentinel-1watermappingalgorithmdesign......................................................................74
5.2.1 Userinterfaceandfloodmonitoringactivationmethod............................................75
5.2.2 Sentinel-1basednear-realtimefloodmappingservicelogicimplementation..........75
5.2.2.1Developmenttools,supportingproductsandqualitycontrol..................................75
A. TUWienSARToolbox.................................................................................................75
B. SupportingProductsandModules.............................................................................76
C. Dataacquisitionandmanagement............................................................................77
5.2.2.2Concepts,RequirementsandInfrastructure.............................................................77
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:5of84
A. Conceptandbigpicture.............................................................................................77
B. Requirements.............................................................................................................78
C. Infrastructureandservicelogicimplementationscheme..........................................78
D. Prototype....................................................................................................................79
6 CONCLUSIONS...........................................................................................................................83
LISTOFFIGURES
Figure2-1:CopernicusEMSserviceoverview..................................................................................13
Figure2-2:Exampleofareferencemapdepictingthepre-eventsituationregardingseverefloodingsinJune2016inBavaria(Germany).Themapdisplaysbasictopographicfeatures.Itwasproducedusingpublicdatasetsandpre-eventimagery..................................................................................16
Figure2-3:ExampleofaflooddelineationintheareaofCarlisle(UnitedKingdom)producedafterStormDesmondcausedsignificantsurfacefloodinginDecember2015.Themapshowstheextentofthefloodedareaattwotimes.Itwasobtainedfrompost-eventimageryusingasemi-automaticapproach..........................................................................................................................................17
Figure2-4:ExampleofadamageassessmentfortheareaofAmatrice(Italy)aftertheearthquakeoccurred on 24th August 2016. The map displays the assessment of damage to buildings andtransportation infrastructureaswellasfurthercrisis information,which indicatesblockedroads,landslides,debrisandtents.Thecrisisinformationwasderivedfrompost-eventsatelliteimagebyvisualinterpretation.........................................................................................................................18
Figure2-5:CopernicusEMSDataPortal;RapidMappingactivationprovidingmetadataandaccesstorasterandvectorproducts..........................................................................................................21
Figure2-6:Mostlyavailablegeographicfeatures,labeledwithbasenames,andcommonattributes...........................................................................................................................................................23
Figure2-7:........................................................................................................................................26
Figure2-8:........................................................................................................................................26
Figure3-1:TheEFASforecastingwebsitethatispubliclyaccessible.Notethatthepublicservicedoesnotprovidecurrentfloodforecasts.................................................................................................30
Figure3-2:TheEFFISCurrentSituationviewer.ThisviewerprovidescurrentseasonwildfiresituationinformationacrossEurope...............................................................................................................31
Figure3-3:TheEFFIShistoricalwildfireviewershowingresultsfor2014.......................................32
Figure3-4:FloodprotectionlevelsacrossEuropebasedontheJongmanetal.(2014)methodology...........................................................................................................................................................35
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:6of84
Figure3-5:DescriptionoftheprocessingchainfortheI-REACTtask3201fortheproductionofthe5-yearreturnperiodfloodhazardforecastoutput..........................................................................37
Figure3-6:Final5-yearfloodhazardforecastoutputinGeoJSONfileformat.Thedarkerthebluecolour,thegreaterthefloodhazardforcast....................................................................................39
Figure3-7:DescriptionoftheprocessingchainfortheI-REACTtask3202fortheproductionofthe20-yearreturnperiodfloodhazardforecastoutput........................................................................39
Figure3-8:Comparinga5-year(leftside)and20-year(rightside)returnperiodfloodhazardforecast.Herethe'raw'dataispresentedatthepixellevel...........................................................................40
Figure3-9:Five-andtwenty-yearreturnperiodsforthefloodhazardforecastoverlapping.PleaserefertoFigure8fortheseparatefloodhazardforecasts................................................................40
Figure3-10:5-and20-yearfloodhazardforecastwithNUTSlevel5datainthebackground........41
Figure3-11:Thefinal20-yearfloodhazardforecastinGeoJSONformat.Theblueshadingprovidesinformation about the number of models forecasting a flood disaster. Darker = more modelsforecastingflood..............................................................................................................................41
Figure3-12:TheprocessingchainforI-REACTtask3203basedontheECMWFFWIdata.............44
Figure3-13:TheI-REACTtask3203outputbasedonECMWFFWIdata.ThetablepresentssomeofthedetailsstoredintheGeoJSONpolygons....................................................................................46
Figure3-14:TheI-REACTtask3206outputbasedonECMWFFWIdata.ThetablepresentssomeofthedetailsstoredintheGeoJSONpolygons.....................................................................................47
Figure3-15:TheprocessingstepsoftheI-REACTtask3204toprocessVIIRSbasedactivefiredatatodelineateaffectedcommunities......................................................................................................48
Figure3-16:TheVIIRSactivefireproductusedtoproducedelineationsoftheNUTS3levelregionswheretheywereidentified.Thetableshowsthateachofthepolygonsprovidesthenumberofhotspotpixelsfoundwithinthatregion(count)....................................................................................50
Figure3-17:TheprocessingstepstoproducetheI-REACTtask3205GeoJSONoutput..................51
Figure3-18:TheGeoJSONoutputoftheI-REACTtask3205showingtheVIIRSactivefiredatapointswithintheNUTSlevelpolygons........................................................................................................51
Figure4-1:WarningsissuedbytheFinnishMeteorologicalInstitute..............................................55
Figure4-2:Meteo/HydrogeologicalwarningbulletinissuedbyARPAPiemonte............................57
Figure4-3:LivestormserviceissuedbyARPAPiemonte.................................................................58
Figure4-4:WarningsonsensorsissuedbyARPAPiemonte............................................................58
Figure4-5:Monitoringhydrostations–RiverPo............................................................................59
Figure4-6:PoriverstretchbetweenFerraraandPanaro................................................................59
Figure4-7:MeteowarningbulletinissuedbyMetOffice................................................................62
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:7of84
Figure4-8:FloodriskwarningsissuedbyEnvironmentAgency......................................................62
Figure4-9:LEMSintegrationmoduleandtheI-REACTsystem........................................................63
Figure4-10:LEMSdatabase.............................................................................................................64
Figure4-11:ARPAHydrogeologicalbulletininI-REACT...................................................................66
Figure 5-1: EODC data acquisition strategy. Satellite data are transferred from the satellite toCollaborative Ground Segment, then to a rolling archive system at ZAMG and stored to EODCinternalstoragesystem....................................................................................................................68
Figure5-2:Sentinel-1AdatacoverageforIWGRDacquisitionmode..............................................71
Figure5-3:Sentinel-1AdatacoverageforIWGRDacquisitionmode..............................................71
Figure5-4:ENVISATASARcoveragemapforWideSwathmode.....................................................72
Figure5-5:EODCinfrastructurecomponents..................................................................................73
Figure5-6:ENVISATASAR/Sentinel-1datapre-processingworkflow............................................74
Figure5-7:ENVISATASAR/Sentinel-1schemeforproductiongeneration.....................................74
Figure5-8:Anexampleoverviewof theSyntheticApertureRadar (SAR)datapre-processingandproductgeneration(NeusiedlLake:Austrian–Hungarianborder)...................................................75
Figure5-9:ApplicationofHandIndexmasktoremovethetopographicnoise...............................76
Figure5-10:AnexampleS1bordernoiseremovalmask.................................................................77
Figure5-11:AnoverviewofI-REACTORframework........................................................................78
Figure 5-12: An overview of infrastructure and service logic for near-real time floodmapping/monitoring........................................................................................................................79
Figure5-13:Anexampleofmulti-temporalfloodfrequencyproduct(Italy)...................................80
Figure5-14:Anexampleofmulti-temporalfloodfrequencyproduct(UnitedKingdom)................80
Figure5-15:ExampleofSentinel-1productuploadedtoI-REACTdatabase...................................81
Figure 5-16: Floodmonitoring framework (top) andprototypeof final product displayedon thefront-endoftheI-REACTwebpage(bottom)...................................................................................81
LISTOFTABLES
Table2.1:Targeteddeliverytimesforthethreemaptypesatthetwoservicelevels....................15
Table2.2:Mainitemsofreferencemaps........................................................................................15
Table2.3:Additionalavailablefeaturesofdelineationmaps..........................................................16
Table2.4:Gradingmap-specificfeatures.........................................................................................18
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:8of84
Table2.5:Countofactivations.........................................................................................................20
Table2.6:Specificationsofprovidedmapproducts........................................................................22
Table3.1:Floodhazardclassificationmatrixbasedontheproceduredescribedinthetext..........34
Table3.2:AvailableEFFISdatathroughWMS..................................................................................42
Table4.1:Connectedsources..........................................................................................................54
Table4.2:Catalunya.........................................................................................................................60
Table4.3:I-REACTDataInterfaceattributes...................................................................................64
Table4.4:GeoJSONattributes.........................................................................................................65
Table4.5:DatastructureofUKserviceprovidingfloodwarnings...................................................67
Table5.1:Sentinel1sensorspecifications.......................................................................................69
Table5-2:Sentinel1:specificationsofdifferentdataacquisitionmodes.......................................70
Table6.1:Verylargetable...................................................................Error!Bookmarknotdefined.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:9of84
1 INTRODUCTION
1.1 PURPOSEOFTHEDOCUMENT
Thegoalofthisdocumentistoreportonthedesignandimplementationdetailsofinterfacestowardexternal data sources such as Copernicus EMS, European Flood & Fire Systems, local EMS andSentineldatawhicharepartoftheworkpackage3(WP3)“ExternalServicesandDataIntegration”ofthe I-REACTproject.Theoverallobjectiveof thisWP is to integrateexistingdatasourcesandsystems related tonatural hazards, suchas floods and fires, and therefore to “fuel” theoverallsystemwithusefulinformationsources.TheWPissplitintoseveraltasksaccordingtothetypeofdataandsystems,whichareintegratedwithinI-REACT.Ingeneral,datastreamsandserviceswillbeimplementedfollowingtheas-a-serviceapproach,sothatnewdataisingestedautomaticallyintoI-REACTwithouthavingusertoactivateadatastream.
1.2 STRUCTUREOFTHEDOCUMENT
Thedocumentisorganizedasfollows,representingtaskswithindeliverable3.1ofWP3.
• Chapter 1 is an introductory description of the document itself, which should help toorganizereadingincomprehensiveway.
• Chapter2givesadetailedandthoroughdescriptionontheworkingprogressandactivitieswithinTask3.1“COPERNICUSEMSINTEGRATION”.
• Chapter 3 describes indetail the activitieswithin Task3.2 “EUROPEANEARLYWARNINGSYSTEMSINTEGRATION”.
• Chapter4,“EXISTINGLOCALEMSINTEGRATION”comprisesasynopsisofactivitieswithinTask3.4.
• Chapter5isadescriptionofTask3.5“SENTINELDATAPROCESSINGINTEGRATION”.
• Chapter6containstheconclusions.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:10of84
1.3 ACRONYMSLIST
CAMS CopernicusAtmosphereMonitoringService DSS DecisionSupportSystemEMS EmergencyManagementService EODC EarthObservationDataCentre ERCC EuropeanResponseandCoordinationCentre ESA EuropeanSpaceAgency EWS EarlyWarningSystem EO EarthObservation FWI ForestFireWeatherIndex GWIS Global Wildfire Information System I-REACT ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologiesLEMS LocalEmergencyManagementSystemsNRT NearReal-TimeSGRT SARGeophysicalRetrievalToolboxWMS WebMapServiceWP Workpackage ZAMG ZentralanstaltfürMeteorologieundGeodynamik
1.4 REFERENCEANDAPPLICABLEDOCUMENTS
ID Title Revision Date
[RD01] http://emergency.copernicus.eu/mapping/ems/cite-copernicus-
ems-mapping-portal online
accessed 2016
[RD02] https://web.jrc.ec.europa.eu/callsfortender/index.cfm?action=a
pp.tender&id=2588 onlineaccessed 2017
[RD03] http://publications.jrc.ec.europa.eu/repository/bitstream/JRC83
027/lb-na-26072-en-n.pdf onlineaccessed 2017
[RD04]
http://www.copernicus.eu/main/emergency-management onlineaccessed 2017
[RD05]
https://www.efas.eu/efas-archive.html onlineaccessed 2017
[RD06]
http://effis.jrc.ec.europa.eu/about-effis/data-license/ onlineaccessed 2017
[RD07]
http://effis.jrc.ec.europa.eu/applications/fire-history/ onlineaccessed 2017
[RD08]
http://effis.jrc.ec.europa.eu/applications/data-and-services onlineaccessed 2017
[RD09]
http://ec.europa.eu/eurostat/web/nuts onlineaccessed 2017
[RD10]
http://effis.jrc.ec.europa.eu/api-test/#/rest onlineaccessed 2017
[RD11] http://docs.oasis-open.org/emergency/cap/v1.2/CAP-v1.2-os.pdf
onlineaccessed 2017
[RD12]
http://en.ilmatieteenlaitos.fi/about-us online
accessed 2017
[RD13]
http://meteoalarm.eu/ online
accessed 2017
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:11of84
[RD14]
https://www.eodc.eu/ online
accessed 2017
[RD15] Elefante,S.,Wagner,W.,Briese,C.,Cao,S.,&Naeimi,V.:High-
performancecomputingforsoilmoistureestimation.Proceedingsofthe2016conferenceonBigDatafromSpace(BiDS'16).
-- 2016
[RD16]
A.Nobre,L.Cuartas,M.Hodnett,C.Renn,G.Rodrigues,A.Silveira,M.Waterloo,andS.Saleska.Heightabovethenearestdrainageahydrologicallyrelevantnewterrainmodel.JournalofHydrology,vol.404,no.1,pp.13–29,011.http://www.sciencedirect.com/science/article/pii/S0022169411002599.
online
accessed 2017
[RD17]
I. Ali, S. Cao, V.Naeimi, C. Paulik, andW.Wagner.Methods toremovebordernoisefromSentinel-1 interferometricwideswathgroundrangedetectedsyntheticapertureradardata:Implicationsandimportancefortimeseriesanalysis. IEEEJournalofSelectedTopicsinAppliedEarthObservationsandRemoteSensing.
-- 2017
[RD18]
B. Jongman, S. Hochrainer-Stigler, L. Feyen, J.C.J.H. Aerts, R.Mechler, W.J.W. Botzen, L.M. Bouwer, G. Pflug, R. Rojas, P.J.Ward.Increasingstressondisaster-riskfinanceduetolargefloods. Nat.Clim.Change,4,pp.264-268,10.1038/nclimate2124.
-- 2014
[RD19]
V.Naeimi, S. Elefante, S. Cao,W.Wagner,A.Dostalova, andB.Bauer-Marschallinger. Geophysical parameters retrieval fromsentinel-1sardata:Acasestudyforhighperformancecomputingat eodc, Proceedings of the 24thHigh Performance ComputingSymposium, ser. HPC ’16. San Diego, CA, USA: Society forComputerSimulationInternational,2016,pp
-- 2016
[RD20] Details about Equi7 projection:
http://dx.doi.org/10.1016/j.cageo.2014.07.005
onlineaccessed 2017
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:12of84
2 COPERNICUSEMSINTEGRATION
2.1 SERVICEOVERVIEWOFCOPERNICUSEMS
CopernicusEmergencyManagement Service (CopernicusEMS)provides information fordisasterresponse management and supports recovery, disaster risk reduction, prevention, andpreparednessactivities.Theserviceprovidesmapsbasedonsatelliteimagery,addressingvariousemergencysituationsarisingfromnaturalorman-madedisasters,aswellasearlywarningservicesforfloodandfirerisks.MainaimoftheEuropeanUnion(EU)serviceisreinforcingcrisisresponseincaseofnationalorcross-borderdisastersinEuropebutalsoincaseoflarge-scaledisastersoutsideoftheEU[RD01].
CopernicusEMS ispartof theEuropeanCopernicusProgrammeforEarthObservation,aimedatproviding information services on environmental and security issues, based on satellite earthobservation and in situ data. Within this framework, the EMS is operated by the EmergencyResponseCoordinationCentre (ERCC)at theEuropeanCommission'sHumanitarianAidandCivilProtection department (ECHO). Technical assistance in support of service implementation isprovidedbytheJointResearchCentre(JRC).
CopernicusEMSconsistsofdifferentmodulesthatcanbeassignedtotwomaincomponents(Figure2-1)[RD01]:
• EFAS and EFFIS - delivering alerts and risk assessments of floods and forest fires - can besubsumedtoanearlywarningcomponent(describedindetailinchapter3).
• The Copernicus EMS Mapping component provides map products and analysis in case ofdisasters.Itaddressesawiderangeofemergencysituationsresultingfromnaturalorman-madedisasters,coveringinparticularfloods,earthquakes,tsunamis,landslides,severestorms,fires,industrialaccidents,volcaniceruptions,andhumanitariancrises.Theproductsaredeliveredintwodifferentmodesbydifferentmodules:
o RapidMappingprovidesrapidservicedeliveryduringtheemergencyresponsephase(withinhours/days;available24/7/365)
o Risk & Recovery Mapping is designed for on-demand provision of geospatialinformationsupportingrecovery,disasterriskreduction,prevention,andpreparednessactivitiesinpre-orpost-crisissituations(withinweeks/months)
This service structure is complemented by a dedicated validation component, aiming at theimprovementoftheEMSmappingserviceproductsregardingrobustnessandaccuracycriteria.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:13of84
Figure2-1:CopernicusEMSserviceoverview.
TheoperationalphaseoftheEmergencyManagementService(EMS)startedinApril2012withinthe former framework of the Regulation on GMES (Global Monitoring for Environment andSecurity), which has been updated by the Copernicus Regulation in April 2014. The providedinformationandproductsareaccessibleonafull,openandfreeof-chargebasistothepublic.
CopernicusEMSMapping Services address inparticular theneedsof actors in the fieldof crisismanagementandhumanitarianaidoperations.Relatedtoanemergencyevent“AuthorisedUsers”(AUs) can trigger the service provision process. AUs includemainly the national civil protectionauthoritiesfromtheparticipatingstatesoftheEUCivilProtectionMechanism(NationalFocalPoints,NFPs) as well as EU services. Associated Users (ASCUs), including - amongst others - local andregionalpublicentitiesandgovernmental andnon-governmentalorganisations, can request theservicethroughanAuthorisedUser.Inordertorequestanactivation,aServiceRequestForm(SRF)has to be completed and submitted via e-mail to the European Response Coordination Centre(ERCC).
EachrequestisthenreviewedtoensurethatanactivationiswithinthescopeoftheEMSMappingServiceregardingthetypeandmagnitudeofanevent,therelevanceofimpactandthedegreeofurgency(RapidMappingService)orregardingtherelevanceforDisasterRiskReduction,PreventionandPreparednesspurposes(RiskandRecoveryMappingService).FollowingtheauthorisationoftheactivationrequestbytheERCCmappingproductsaregeneratedbyserviceprovidersunderservicecontracts.Theprocessingofanactivationrequestfollowsadefinedworkflowinvolvingdifferentserviceprocessingstepsincollaborationandexchangebetweenserviceprovider,authoriseduser,JRC,ERCCandESA.JRCisresponsibleforthetechnicalcoordinationoftheEMSMapping,itperformstechnicalandqualitycontrolsthroughoutserviceprocessingandmanagestheserviceevaluationprocessuponanactivation.Furtheritisinchargeofthearchivinganddisseminationofgeneratedproducts. The European Space Agency (ESA) generally coordinates procurement and access to
CopernicusEmergencyManagementService(EMS)
EarlyWarning
EFAS EFFIS
Mapping
RapidMappingRiskandRecoveryMapping
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:14of84
satellite data required for the generation of deliverables. The data provision is based on theCopernicus Space Component Data Access (ESA-CSC-DA) system that offers access to satelliteimagingsensordataofCopernicusSentinelaswellasCopernicusContributingMissions(CCM).
2.1.1 RAPIDMAPPINGSupport of crisismanagement during emergency responsephase is emphasised to be themainpurpose of the rapid mapping service. Earth observation data and disaster related maps areprovidedon-demandwith strict timeliness requirements.Acquisition,processingandanalysisofsatelliteandaerialimageryandothergeospatialrasterandvectordataformthebasisforserviceprovision.
TheCopernicusEMSRapidMappingserviceisoperatedcontinuously,i.e.servicerequestscanbehandled24hoursaday,sevendaysaweek.Thaton-dutyrequesthandlingpermitsashortresponsetime for product supply within the demanded time frame of hours / days. In this regard, theavailabilityofpost-eventsatellite imagery,shortlyafteranemergencyevent, isconsideredakeyrequirement,whichshallbefulfilledwithintheframeworkoftheESACSC-DAmechanism.
Therapidmappingserviceaimsattheprovisionofassessmentsofimpactextentanddamagegradeas well as pre-emergency information particularly on infrastructure of affected areas. Thestandardisedproducts includeprintablemaps supplementedbybrief analysisona case-by-casebasis,aswellasgeospatialdatasetsforusewithinageographicalinformationsystem(GIS).
Dependingontheservicerequest,threecategoriesofmapsareprovided:
• Reference maps: based on satellite imagery acquired prior to the disaster event theydescribethesituationbeforetheemergencysituation
• Delineationmaps:outlinetheextentoftheareaaffectedbytheevent
• Gradingmaps:provideanassessmentoftheimpact/damagegradecausedbyanevent
The differentmap products are provided individually or in combination depending on the userneeds. The service also features updates for delineation and grading maps, allowing for amonitoring.
Tomeettherequirementsgivenbythelevelofurgency,eachmaptypecanberequestedintwodifferentservicelevels(Table2-1):
• Service Level 1 targets a rapid product delivery within a few hours after the receipt ofbaselinesatellitedata.Additionallytotheprovisionofprecisemapswithinthedemandedtimeframeof9to12hours(seetable)aFirstAvailableMap(FAM)isprovidedwithin3hoursfor delineation and gradingmaps representing an early information product with lowerthematicandpositionalaccuracy.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:15of84
• ServiceLevel5targetstheprovisionofmapproductswithmaximumaccuracyandlevelofdetailwithinfiveworkingdays,wherebythistimeframecanbeadapteddependingonuserneeds.
MAPTYPE SL1 SL5
Referencemap
9h 5days
Delineationmap
12h 5days
Gradingmap 12h 5days
Table2-1:Targeteddeliverytimesforthethreemaptypesatthetwoservicelevels.
2.1.1.1 REFERENCEMAPSReferenceMaps serve thepurposeof providingbackground informationon the areahit by thedisaster.UsuallytheyarebasedonsatelliteorVeryHighResolution(VHR)ortho-imagerycapturedclose before the event. They contain available comprehensive information to support disastermanagement measures, including selected basic topographic features and exposed assets andfacilities.
Referencemapsnormallyincludethefollowing(topographic)features(Table2-2):
Featureclass Examples/Itemdescription
GeneralInformation areaofinterest,sensorfootprint,sources,noimagedata
Administrativeboundaries
relevantadministrativeboundarylevels
Settlements agriculture, commercial, educational, industrial, institutional,medical,military,recreational,religious,residential,transportation
Hydrology coastline,dam,lake,reservoir,river,stream,canal,waterway
Pointofinterest educational, industrial, institutional, medical, religious,transportation,cemetery
Physiography contourlineandelevation(m),heightabovesealevel(m)
Utilities powerplant,quarry,watertreatmentplant
Transportation motorway, primary road, secondary road, local road, tramway,subway/metro,railway,station,bridge,tunnel,aerodrome,runway,harbour
LandUse–Landcover bare soil, cropland, grassland, scrub, woodland, forest, wetland,firebreak
Table2-2:Mainitemsofreferencemaps.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:16of84
DepictingtheexposurewithintheAOI,ReferenceMapsmayincludesupplementarysummarytablesprovidingoverviewinformationonexposedpopulation,settlements,transportation,utilitiesorlandusewithinthemap’sAOI(Figure2-2).
Figure2-2:Exampleofareferencemapdepictingthepre-eventsituationregardingseverefloodingsinJune2016inBavaria(Germany).Themapdisplaysbasictopographicfeatures.Itwasproducedusingpublicdatasetsandpre-eventimagery.
2.1.1.2 DELINEATIONMAPSBasedonpost-disastersatelliteimages,DelineationMapscontainanassessmentofthegeographicextentoftheevent.Ifrequestedtheevolutionofaneventisoutlined.Dependingonthetypeofevent, Delineation Maps may depict flooded or burnt areas, or they provide an outline ofearthquakeimpactareaorthelocationandextentoflandslides,forexample.
Delineationmapsdisplaycrisisinformationaswellasreferenceinformation.Henceasamainitem,additionally to the before mentioned set of topographic features included by reference maps,delineationmapscontainalayeroutliningtheaffectedarea(Table2-3).
Featureclass Examples/Itemdescription
CrisisInformation Geographicextentoftheaffectedarea;e.g.floodedarea,burntarea
Table2-3:Additionalavailablefeaturesofdelineationmaps.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:17of84
DepictingthedisasterconsequenceswithintheAOI,delineationmapsmayincludesupplementarysummarytablescontaininginformationandassessmentsregardingexposedoraffectedpopulation,settlements,transportation,utilitiesorlandusewithinthemap’sAOI(Figure2-3).
Figure2-3:ExampleofaflooddelineationintheareaofCarlisle(UnitedKingdom)producedafterStormDesmondcausedsignificantsurfacefloodinginDecember2015.Themapshowstheextentofthefloodedareaattwotimes.Itwasobtainedfrompost-eventimageryusingasemi-automaticapproach.
2.1.1.3 GRADINGMAPSGradingMapsshowtheimpactofdamagescausedbyadisasterevent.Inparticulartheyprovideinformationontheextent,typeandmagnitudeofdamages.Theassessmentsarederivedfrompre-andpost-eventsatelliteimages.
Gradingmaps include reference information as well as crisis information specific to the event.Gradingassessmentlayermaydepictdisasterextentandimpactgraderegardingaffectedareasorthedamagetoassetsaswellasotherrelevantinformation.
Providing information on disaster consequences within the AOI, grading maps may includesupplementary summary tables containing data on affected population or assets classified perdamagegrade(Table2-4,Figure2-4).
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:18of84
Featureclass Examples/Itemdescription
Areagrading severityofimpactforaffectedareas;e.g.firegrading,floodgrading
Settlementsgrading Affectedassetsclassifiedaccordingtodamagegrade
Transportationgrading Affectedassetsclassifiedaccordingtodamagegrade
Utilitiesgrading Affectedassetsclassifiedaccordingtodamagegrade
Other relevant crisisinformation
Observation relevant in the emergency context; e.g. debris, roadblock,crater
Table2-4:Gradingmap-specificfeatures.
Figure2-4:ExampleofadamageassessmentfortheareaofAmatrice(Italy)aftertheearthquakeoccurredon24thAugust2016.Themapdisplaystheassessmentofdamagetobuildingsandtransportationinfrastructureaswellasfurther crisis information,which indicatesblocked roads, landslides,debris and tents. The crisis informationwasderivedfrompost-eventsatelliteimagebyvisualinterpretation.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:19of84
2.1.2 RISKANDRECOVERYMAPPINGTheRiskandRecoveryMappingmodulesupportsdisasterriskmanagementmeasuresnotrelatedtoimmediatecrisisresponse.Withinaproductiontimeofweeksormonthsitprovidessophisticatedproducts on demand, aiming at the support of prevention, preparedness and reconstructionactivities.Inrespecttoawiderangeofemergencysituationstohandle,productsareprovidedondemandspecifictotheneedsrelatedtothetypeofeventandthephaseofcrisismanagementcycle.
Ingeneraltheinformationcontainedinmapproductscanbecategorisedintopographicfeatures,disasterriskinformationandtailoredinformationrelatedtotheeventandthecrisismanagementcyclephase.
Insupportofspecificcrisismanagementtasksrelatedtoemergencyphasesthreeproducttypesareoffered,addressing1)prevention,2)preparednessand3)recovery/reconstructionphase.
2.1.2.1 REFERENCEMAPSRelated to disaster risk reduction, reference maps contribute comprehensive and detailedinformationontheterritoryandexposedassets.Informationspecifictopreventionproductsassistintheinventoryofassetsatrisk,intheplanningofprotectivemeasuresandthedevelopmentofawareness-raising activities. Maps may contain for example information on existing mitigationmeasuresorlandusezoningplanstakingaccountofdisasterriskinformation.
2.1.2.2 PRE-DISASTERMAPSRelated to disaster preparedness, pre-disastermaps provide relevant information in support ofdisastermanagementmeasurestominimisedisasterimpact.Theproductsforexamplecanassistinthepreparationofresponseoperationsandrescuemeasures.Inthiscontextthemapsmayincludeinformation on hazard exposure, vulnerability/resilience, risk status for population and assets,evacuationplansandforecasts.
2.1.2.3 POST-DISASTERMAPS
Relatedtorecovery/reconstructionphase,post-disastermapscontaininformationrelevantbeyondimmediate response phase. The products support recovery measures aiming to re-establishconditionstopre-disasterstate.Forthatpurpose,themapsforexampleprovideassessmentsoflong-termimpactandrecoveryneedsaswellas informationonrecoveryplanning.Further,theymaycontaininformationregardinghazardexposure,vulnerabilityandresilience,particularlywithregard to new assets or they aim for a progress monitoring of reconstruction efforts andrehabilitation.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:20of84
2.2 COPERNICUSEMSDATAPORTAL
AWeb-basedportalprovidesaccesstotheCopernicusEMSMappingproductsandtotheEMSEarlyWarningmodules EFAS and EFFIS (emergency.copernicus.eu). It acts as a central repository forrelevantinformationmaterialinparticularregardingtheEMSMappingcomponentandoffersthedownloadofServiceRequestForms,UserGuide,etc.
All Rapid Mapping and Risk & Recovery Mapping activations are featured on a map and in asearchablelist,whichcanbefilteredbasedoneventtype,eventdate,activationstatusandaffectedcountry.Theavailableproductsofanactivationcanagainbe filteredbymaptypeand location.Additionally metadata is provided for each activation, containing information on event time,activation time, event type, type and number of produced maps, activation status, affectedcountries,areadescriptor,nameofactivator,activationreasonandrequestedproducts.
For the EMSMapping Services GeoRSS feeds are available to receive regular content updatesrelatingtocurrentorpastactivations.
Furthermore,theportalfeaturestheCopernicusMapCoveragePlannertool,whichfacilitatesthedefinition of the geographical extent for the preparation of a following EMS -Mapping ServiceRequest.Additionally amap is provided that aggregates activationsof several other EmergencyMappingorganizationsbesidestheCopernicusMappingservices.
TheEMSportalprovidesgeospatialinformationin196activationssofar:
NumberofActivations
RapidMappingActivations 167
Risk and Recovery MappingActivations
29
total 196
Table2-5:Countofactivations.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:21of84
Figure2-5:CopernicusEMSDataPortal;RapidMappingactivationprovidingmetadataandaccesstorasterandvectorproducts.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:22of84
2.2.1 OVERVIEWONDATAPROVISIONTheproductsofferedbytheCopernicusEMSMappingServicesareprovidedasdigitalmapoutputsandasvectordatasetsontheEMSportal.
Ingeneral,allproductsimplementastandardizedstructureasdescribedbelow.Followingspecificrequestsorforusabilitypurposescase-by-caseadaptionsmaybeapplied.
Bydefault,thedigitalmapsareavailableindifferentdeliveryformatsandresolutions:
Printablemap FullcolourISOA1,orequivalent
Resolution:high=300dpi;medium=200dpi;low=100dpi
GeoPDF
Metadatafile
Georeferencedmap
FullcolourISOA1,orequivalent
Resolution:high=300dpi;medium=200dpi;low=100dpi
GeoTIFF,GeoreferencedJPEG(withworldfile)
Metadatafile
Table2-6:Specificationsofprovidedmapproducts.
MapproductsareusuallyprovidedinUTMcoordinatesystemusingWGS84datum.Mapscaleandthe covered area of themaps vary according to user requirements and the size of the area ofinterest.Downloadable rastermap files includemetadata informationonpublicationdate,mapscale and the product version. Map file names (e.g.EMSR194_05FARINDOLA_GRADING_OVERVIEW_v1_300dpi.jpg) include activation code, productcodeandname(referringtoaffectedlocation),producttype,productversionandresolution.
Additionally,allgeographicfeaturesthatarederivedfromtheanalysisandcontainedinthemaps,areprovidedas vector files. Theavailabilityof georeferenceddatasets in vector format is a keycomponentoftheCopernicusEMSservicedelivery,asitallowsusingthegeneratedinformationinsupportoffurthergeospatialanalysis.VectorfilesaredeliveredinESRIshapefileformatorinGoogleEarthKML(orKMZ)format,includingametadatafile.Foreachmapcomponentassociatedwithanactivationafullsetofvectorfilesisdelivered.Relevantattributesfordataclassificationaccordingtothelegenditemsofassociatedrastermapsareincluded.Bydefaultself-explanatorynamesanddescriptivecontentsareassignedtoattributes.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:23of84
Availablefeatureclassesandcommonattributestructure:
Figure2-6:Mostlyavailablegeographicfeatures,labeledwithbasenames,andcommonattributes.
Detailed feature class names are applied as shown in the following example:EMSR194_05FARINDOLA_02GRADING_v1_7000_crisis_information_poly_grading.shp
Featureclassnamesgenerallyrefertothenamesusedforthelegenditems/headingsinthemapproducts.Dependingontheitem,geometrytypecanbepolygon,lineorpointandfilenamesmayindicatethegeometrytype.Incasealayerincludesgradinginformationanameaffixisaddedtothefilename.Furthermoreascaleparameterreferstothemapscale.Inherenttomapscaleisthelevelofdetail,affectingassetspolygons,especiallysettlementsdigitalisationgranularity.
Apartfromthat,thenomenclatureisconsistenttotherastermapproducts,includinginformationon activation code, product code and name (referring to affected location), product type andproductversion.
Allrelevant information,regardingtheCopernicusEMSServicewasgatheredinRD01,RD02andRD03.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:24of84
2.3 COPERNICUSEMSINTEGRATION
Allthedatainserted,arecomingfromCopernicusEMSportal,whichareautomaticallyparsedbyaPythonscripttoretrievethelatestactivationsfromthewebsite.Inthefollowingparagraphs,theaccessibilityoptionstothesedataaredescribed.
BesidesthepossibilityofdownloadingdatamanuallyfromthewebsiteofCopernicus,therearenotmanyoptionstogetautomaticaccesstodisastermapsandmetadatathereof.FTP-ServersareonlyaccessibleforuserswhohaveactivatedtheEmergencyManagementService(EMS).Therefore,thisprojectmakesuseoftheGeoRSSfeedthatisprovidedonthesamewebsiteforallemergencyeventtypes(cf.http://feeds.feedburner.com/CopernicusEMSMappingRushModeActivations).
AGeoRSSfeedisanemergingstandardthatintegrateslocationdataintoaclassicalwebfeed.TheGeoRSSfeedofCopernicusEMSisupdatedaccordingtotheactivationtimeofaspecificeventthathasbeentriggeredbytheresponsibleadministrationorauthority.Usuallythereisatimedifferencebetweenthetimewhentheeventhasoccurred(EventtimeLOC)andtheactualactivationthatmaytakeplacea fewdaysafter theevent.TheGeoRSSalertsabout themost recentactivationsandprovidesanattributelisttoidentifythesingleevents:
• Titleofactivationincludinganemergencynumber
• Detailsandmaps
• Eventtype
• Eventtime(localtime,LOC)
• Activationtime(coordinateduniversaltime,UTC)
• Affectedcountries
• Areadescriptor
• Activationreason
TheabovelistedinformationcanbeusedinEMSdataintegrationmodulefortheautomatictransferofthegeodataintothebackendsystem.Furthermore,itrepresentsvaluableinformationabouttheshowngeodatafortheenduser.
Alsofortheupdateofthedatabasetables,containingallthepreviouslymentionsinformationabouttheGeoRSSfeedaswellasthemetadatainformationabouttheprovidedmaps,anautomatedservicewillbeusedtoactivelymonitortheGeoRSSfeedforupdates.Thisprovidesatimelyresponsebythedataintegrationmodulefornewactivationsandupdatingofthedatabasewhennewmapsaremadeavailable.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:25of84
2.3.1 DESIGNOFEMSINTEGRATIONMODULE
The analysis of the data shows that disaster maps and forecast layers are vector based data.Thereforethecollectionofthesedatasetsinageospatialdatabaseisseenasviableandpracticalsolution for a data storage. A database also offers a basically unlimited and therefore scalablesolution for storing all geospatial data. Through the storage of layers in a spatial database, thearchivingofdataisenabled.Thus,historicflooddatacanbeaccessedatanytime, ifdemanded.Regardinggeodataprocessingcapabilitiesandspatialfunctions,thePostgreSQLdatabasewiththespatialextensionPostGISwasthemostreasonableandsuitabledatabasemanagementsystemforthedataintegrationmodule.EspeciallywiththespatialextensionPostGIS it ispossibletocreatehigh-performancesolutionsforgeodataprocessingandvisualization.PostgreSQLalsosupportstheconceptualworkflowofimportingtriggereddisastermapsbetterthanotherDBMSduetothenativehandlingofshapefilesandrasterfiles.
Thedevelopmentofalargelibraryofspatialfunctions,theenablingofspatialindices,thepossibilitytoautomateprocessingwithPythonscriptsandtheinteroperabilitywithGeoServerandotherOGCconform software makes PostgreSQL superior over other database system candidates for theintegrationandstorageofCopernicusEMSlayer.
Thedesignofthedatabaseislooselybasedonthepythonscript,whichrunsinthebackgroundfortheautomaticallyingestionofthedataandontheprovideddatastructureoftheCopernicusEMSwebsite.Therefore,threedifferenttableshadcreatedbutonlyonedoescontaingeospatialdata.Thefirsttablestoresallthemetadatainformationbelongingtoanactivation,suchasthetitle,dates,area,countryorthestatus.Aslongasthestatusisnotsetto“closed”,theactivationwillbecheckedineachrunofthePythonscript ifnewmapsweremadeavailable.ThesecondtablecontainsallmetadatainformationaboutaprovidedZIPfile,whichisdownloadedbythePythonscript.Here,informationaboutfilename,featurecount,geometrytype,publication,scaleandtypeofmapwillbestored.Inthelasttablealltheinformationabouttheattributesinashapefilearepresent.Sinceall provided datasets have more or less similar structure, only those attributes with the mostmeaningfulvaluesfortheend-usersarechosenfortheimportintothedatabase.Theillustrationbelowrepresenttheuseddatabasemodel(Figure2-7).
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:26of84
Figure2-7:Databasemodel.
Allthetableslinkedwithforeignkeyssothatcomplexqueriesoverallthethreetablesarepossible.Allattributes,whichcannotbefoundwithinthethreetables,havebeenneglectedforthisproject.
2.3.2 IMPLEMENTATIONOFEMSINTEGRATIONMODULEThissectionsummarisestheprocessingchainandimplementationofthesystemandstartswithanoverviewofthearchitecture:
Figure2-8:EMSIntegrationModuleprocessingchain.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:27of84
TheprocessingchainbeginswithanewdisasteractivationpublishedontheCopernicusEMSviatheGeoRSS feeds.This triggerbeginsaseriesofprocessingstepsthatarenecessarytoproducethedatabasetableentry,whichthenaredeliveredtoIDIofthecloudinfrastructure.TheGeoRSSusedtotriggertheprocessingchaincarriestheinternetaddress(URL)oftheEMSactivation.
Using this information, theprogramcan thenaccess theEMSwebsiteandharvest the requiredinformationtocontinue.EMSactivationsproducemapproductsfocusedonthecrisisregion.Thecoordinatesusedforeachofthemapproductsarethegeospatialinformationrequiredtocontinuetheprocessingchain.TheCopernicusGeoRSS feed isproviding information innear-real time foreach occurred disaster event according to the activation time. The rush mode event mappingincludesworldwidedatawhichwillbeallincludedinourdatabase.AuniqueURLdefineseachevent,whereallrelevantmapsandmetadataarepackedinZIPfiles,whicharedownloadedautomaticallythrough HTML parsing and website control functions, e.g. confirming the disclaimer of liabilitybeforeeachfiledownload.
TheparsingfunctionsdefinetheexactpathtothestoredZIPpackagesviatheCopernicusGeoRSSfeedHTMLsourcecode.ThefilesareunpackedlocallyonthedataintegrationbackendliveserverandingestedinthePostgreSQLdatabase.Meaning,eacheventincludesseveralmappinganddetailversions. They are identified through unique namespaces: One shapefile per event and detailversion is loaded feature-wise into all three tablesof thePostgreSQL system,mentioned in theprevious section. The geometryof each feature is stored in a geometry columns in thePostGISformat“geometry”,whichenablesthepossibilitytoeasilyrunGISfunctionalitiesovertheinsertedfeatures.AsthenativecoordinatesystemofCopernicusgeodataisbasedonaUTMprojection,alllayersmustbetransformedtoEPSG3857(WGS84WebMercator),aprojectionsystemforwebandmobileapplications.
The storage, processing and projection of geodatawithin the spatial database is done throughPostGIS functions thathaveproven tobe very efficient for thepurposeof this data integrationmodule.Themainadvantagefortheingestionofthoselayersintoadatabaseisalsotheprovisionofattributedata,spatialindexingandmanyotherspatialanalysisfunctionsthatmightbeusedondemand.InordertosendthepreparedandreadytousedatafromthePostgreSQLdatabasetoI-REACTinfrastructure,shapefileswithauniquenamewillbecreatedforeachshapefileprovidedbyCopernicusEMS,whichhasbeensuccessfullyingested.Thisallowsthepossibilitytoaddonlythoseattributestoshapefiles,whicharenecessaryforthemappingofthedataintheI-REACTfront-end.This step is carriedoutwith the command line toolpgsql2shp,which is included in thePostGISextensionofthePostgreSQLdatabasesystem.Withthehelpofpre-definedSQLstatements,onlythe desired data can be retrieved from the database by adding the statement to the PostGIScommandlinetool.Aftertheshapefileswerelocallyproduced,theywillbecompressedandzippedinordertofacilitatethehandlingofthedatawithintheI-REACTback-endintheAzurecloud.ViaHTTPrequestdatacanbesendviatheweb.
TheIDIoffersanAPIinordertoauthenticateusers,querythemetadatatablewithintheI-REACTinfrastructureandinsertnewdatasetstoback-endofAzurecloudsystembyusingRESTfulservicesandHTTPrequesthandler.Foreachzippedshapefile,metadatainformationwillbequeriedfrom
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:28of84
thePostgreSQLdatabaseandstoredinaso-calledPythondictionaryvariable.HereinformationfromallthreetableswillbecollectedtosatisfytherequirementsoftheIDIAPIinputparameters.Thosedataarerelevanttocreateinalaterstageametadatadocumentabouteachlayerprovidedinthefront-end.AfteronedatasetanditscorrespondingmetadatahasbeensuccessfullysendtotheIDIviaaPythonrequest,thenextdatasetwillbeprepareduntilalldatahavebeensuccessfullysend,belongingtooneactivation.This import logic istriggeredeverytimeanewactivationwasmadeavailableonCopernicusEMSwebsite.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:29of84
3 EUROPEANEARLYWARNINGSYSTEMSINTEGRATION
3.1 SERVICESOVERVIEWOFTHECOPERNICUSEWS
TheEuropeanearlywarningsystemsaremadeupoftwoCopernicusservices:theEuropeanFloodAwarenessSystem(EFAS)andtheEuropeanForestFireInformationSystem(EFFIS).Thesesystemsprovide early warning andmonitoring of flood and fire disasters respectively across Europe toauthorised users. They are both under the Copernicus emergency management service (EMS)(RD04).
3.1.1 THEEUROPEANFLOODAWARENESSSYSTEM(EFAS)TheEuropeanFloodAwarenessSystem,alsoknownastheEuropeanFloodAlertingSystem–EFAS–hasbeenindevelopmentsince2005afteraEuropeanCommissiondecisionwasmadebasedontheaftermathofthedevastatingElbeandDanubefloodsof2002.EFASwasalsothefirstCopernicusservicetobecomeoperationalwhenitbecamepartoftheEmergencyManagementServiceoftheCopernicusInitialOperationsandinsupportoftheEuropeanCivilProtection.Theservicehasbeenfullyoperationalsincethethirdquarterof2012.
Asanoperationalservice,EFASistaskedtoforecastandmonitorfloodsacrossEurope.TheprimaryclientsoftheservicearetheNationalHydrologicServicesaswellastheEuropeanResponseandCoordinationCentre(ERCC).EFASisabletoprovidethemwithprobabilisticfloodforecastswithanearlywarningwindowofupto10daysinadvance.Inordertoprovidethisoperationalservicetherearefourcentreschargedwithexecutingthedifferentpartsoftheservice:
1. TheEFASComputationalcentreiswherethefloodforecastsareproducedandtheEFAS-InformationSystemplatformishosted;
2. TheEFASDisseminationcentreiswheretheEFASoutputsareanalyseddailyandtheinformationdisseminatedtothepartnersandtheERCC;
3. TheEFASHydrologicaldatacollectioncentreiswhereriverdischargeandwaterleveldataacrossEuropeiscollected.Bothhistoricalandreal-timedata;
4. TheEFASMeteorologicaldatacollectioncentreiswherethemeteorologicaldataacrossEuropeiscollected.Again,bothhistoricandreal-timedata.
Operationally,eachof thecentreshasbeencontractedout todifferentconsortia.Detailsof thepartnersinvolvedcanbefoundonthewebsite.
Thepubliclyavailabledataareonlypastforecastsfromthesummerof2014becausethearchivetoolisonlyaprototype.Error!Referencesourcenotfound.showstheuserinterfaceofthepubliclyavailable service website [RD05]. The real-time flood forecasts are only made available to the
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:30of84
national/regional partner institutes. Furthermore, the actual EFAS forecast cannot be publishedopenlyduetoservicelicensingrestrictions.
Figure3-1:TheEFASforecastingwebsitethat ispubliclyaccessible.Notethatthepublicservicedoesnotprovidecurrentfloodforecasts.
3.1.2 THEEUROPEANFORESTFIREINFORMATIONSYSTEM(EFFIS)EFFIS-theEuropeanForestFireInformationSystemisinchargeofsupportingservicesrelatedtothe protection of forests against fires across Europe. It was established by the EC in 1998 incollaborationwiththenationalfireadministrationsandissupportedbyanetworkofexpertsfromEUcountriesaswellasneighbouringcountries.ThisExpertGrouponForestFiresisregisteredundertheSecretariatGeneraloftheEuropeanCommission.EFFISalsoprovidesservicestootherbranchesof the EC such as the ERCC and provides harmonised forest fire information to the EuropeanParliament. It became a component of the Copernicus EMS in 2015. Recently, EFFIS has beenextended toprovideglobalwildfire information through theGlobalWildfire InformationSystem(GWIS).
Fordownstreamwildfiredisasterservices,EFFISprovidesanumberofinterestingdatawithalicensethatgenerallyappearstobeopenbutnotstandard[RD06].Twoofthemostwidelyusedservicesarethefollowing:
• CurrentSituation– forecastsofup to6days inadvance frommeteorological firedangermapsforthecurrentEuropeanfireseasonaswellasupdatedmapsoffirehotspotsandfireperimeters;
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:31of84
• FireNews–dailyupdatesofnewsarticlesrelatedtowildlandfiresacrossEuropeselectedandgeo-locatedbytheEFFISteam.
Thecurrentsituationviewer(Error!Referencesourcenotfound.;RD06)providesthepossibilityforuserstovisualisesomeoftheinformationmadeavailablethroughEFFISincluding:
• FireDangerForecast;
• RapidDamageAssessmentand;
• AnalysisTools.
Figure3-2:TheEFFISCurrentSituationviewer.Thisviewerprovidescurrentseasonwildfiresituation informationacrossEurope.
However this is a visualisation tool and thedatadisplayed cannot bedirectly used as part of adownstream service. Customers of downstream services are interested in the following pre-firestateandfiredatabaseEFFISmodulesmentionedontheservicewebsite:
• FireDangerAssessment–basedon theCanadian Forest FireWeather Index (FWI)usingcurrentmeteorologicalforecasts.Wildfiredangerismappedintosixclassesfromverylowtoextreme;
• RapidDamageAssessment–includesactivefiredetectionfromsatelliteimageprocessingofthermalanomaliesandthemappingofburnedareas;
• EmissionsAssessmentandSmokeDispersion–notcurrentlypubliclyavailable;
• PotentialSoilLossAssessment–notcurrentlypubliclyavailable;
• VegetationRegeneration–notcurrentlypubliclyavailable;
• FireDatabase–detailedindividualwildfirerecordsprovidedbyEFFISnetworkcountries.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:32of84
EFFISalsoprovidesavisualisationtoolforthefiredatabase(Error!Referencesourcenotfound.;RD07)whichatthetimeofwritingprovidedupdatesonlyupto2014.
ItshouldalsobenotedthattherearesomedataavailablethroughtheGWISthatarenotprovidedthroughtheEFFISviewer.Forexample,fireemissionsspeciesthatareprovidedbytheCopernicusAtmosphereMonitoringService(CAMS).
Figure3-3:TheEFFIShistoricalwildfireviewershowingresultsfor2014
AsshownontheEFFIS‘dataandservices’webpage[RD08],allthedataiscurrentlyonlyavailableasWMSwhichisnotidealfordownstreamserviceswantingtotakeadvantageofthedatatoproducenewservicesandnotonlyviewtheinformation.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:33of84
3.2 COPERNICUSEFASDATAPORTAL
TheEFASdataportalcanpotentiallyprovideawiderangeofimportantinformationtodownstreamservicesinthecontextofflooddisasterearlywarning.BasedontheEFASarchiveprototype,thefollowinggeospatiallayersareavailable:
• Floodsummarylayers;
• Hydrologicallayers;
• Meteorologicallayers;
• InitialConditionslayers;
• Backgroundlayers;
• Flashfloodlayers.
ThemostimportantdatalayerstotheI-REACTprojectarethehydrologicallayers,flashfloodlayers,andpotentiallythefloodsummarylayers.
The hydrological layers provide the flood forecasts for Europe, which are essential to anydownstream early flood disaster warning services. EFAS has the best European wide floodforecastingsystemthatisabletoprovidefloodforecastswithleadtimesgreaterthananyoneelsein the EU. This capability would greatly enhance the I-REACT services portfolio and provide animportantbasisfornewflooddisasterrelatedservices.Unfortunately,atthistime,suchlayersarenotpubliclyavailableorprovidedasopendata.Thisisduetothelicensingissuesbetweenthelargenumberof EFASpartners thatmakes this servicepossible. It is alsodue to the fact that not allMemberStatesarecomfortablewithprovidingdisasterwarninginformationimmediatelyintothehandsofthepublicwithoutfirsthavingtheirauthorisedexpertsconfirmthefloodwarningandthenreportthewarningthroughtheestablishedchannelsfordisasterriskalerting.
Fromadownstreambusinessserviceperspective,thisisveryunfortunatebecauseitdoesnotallowfortheexploitationofpubliclyfundedservicesfornovelapplications.However, it istheprojectsunderstanding that the EC is aware of this and is working towards trying to find a mutuallyadvantageoussolutioninthenearfuture.
TheI-REACTprojectandthisparticulartaskcouldnotjustwaitandseewhethertheissuewouldberesolvedbecauseserviceswerepromisedandneedtobedelivered.Therefore,task3.2–EuropeanEarlyWarningSystemsIntegrationthatneededtocouplefloodforecastingfromEFASneededtofindadifferentsolution.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:34of84
3.3 COPERNICUSEFASINTEGRATION
Asmentionedintheprevioussection,directintegrationoftheEFASfloodforecasts,specificallythehydrological layers was and currently is not possible through direct access using commonapplicationprogramming interfaces (API)and/oropenservices.For this reason, I-REACTworkedcloselyincollaborationwiththeJRCandthesub-contractorKajoS.R.L.toovercomethisissue.Atthebeginningoftheproject,anemployeeofEOXPLORE,thetaskleaderforT3.2wasinvitedasavisitingscientisttoworkonthepremisesoftheJRCinIsprawheretheEFASteamislocated.Overthe threemonthswhen the visiting scientist wasworkingwith the EFAS team, the goal of thecooperation was to identify applicable flood forecasting results and define the processingmethodologysothatitcanbemadeavailabletotheI-REACTpartners.
This section presents the details of the procedure that translates EFAS discharge forecasts intoclassifiedfloodhazardmapsthatareingestedandprocessedinsideI-REACT.Itshouldbenotedthatwhile the JRCandthe I-REACTconsortiumareconstantly looking forways toextendtheservicecooperationbeyondtheproject timeline, it ispossible that futureaccesswillbestoppedpost-I-REACTduetocurrentdatalicensingandsharingrestrictions.
The flood forecast information that is integrated into the I-REACT service is a procedure thattranslates EFAS discharge forecasts into classified flood hazardmaps. This procedure is runningconcurrentlyandrespectingthetimescheduleoftheoperationalEFASsuite.Therefore,processeddataaremadeavailabletwicedailybasedontheEFAS00:00and12:00forecasts.TheprobabilityoftheEFASECMWFensemble (ENS)based forecasts thatexceed the5-and20-year returnperiod(derived from EFAS long term climatology) threshold is taken as ameasure of the actual floodhazard.The5-and20-yearreturnperiodwasderivedfromEFASlong-termclimatologystudies.Inthis way, the range of ensemble forecasts is taken as a measure of the probability of floodoccurrence,whereasfloodforecastreturnperiodsallowfortheestimationofthemagnitudeofthepredictedfloodevents.
TheresultsoftheEFASfloodforecastsareoutputinrasterformat.Ateachgridcell,themedianoftheensembleforecastgivenbythe latestEFASprediction is first identified.Then, themaximumdischargeofthemedianoverthefullforecastingperiod,i.e.10days,isselected.Thisselectedvalueisusedtoderivethetimingofthefloodpeak.Thefloodpeakisconsideredasthelead-time,indays,whenmaximumdischargeistooccur.
In the proposed classification (seeError! Reference source not found.), the total probability ofexceedingcertainEFASthresholdsandthepeaklead-timesareusedtoclassifythepotentialfloodhazardateverygridcellinto3distinctclasses:low,medium,high.
Table3-1:Floodhazardclassificationmatrixbasedontheproceduredescribedinthetext.
Peaklead-time
Probability <24H 48-72H >72H
10-20 Medium Low No
20-50 High Medium Low
>50 High High Medium
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:35of84
Thefirst floodhazardscenariousesthe“raw”EFAS information.RawEFAS informationassumesthatnofloodprotectionmeasuresaretakenintoaccount.Thisscenarioprovidesthefloodshazardassessmentbasedonthenaturalhydrologicalresponseofthesystemand it isuptotheusertoevaluate the potential effects based on the known local flood protection infrastructure. Animportant limitation of this approach is the fact that there is a higher number of false alertsproduced.
In the second scenario, there is the inclusion of some information related to flood protectioninfrastructure. The data on flood protection was compiled based on risk-based estimations forEuropedevelopedbyJongmanetal.(2014).Thisinformationwasintegrated,whereavailable,withtheactuallevelofprotectionfoundthroughliteraturereviewand/orassessedbylocalauthorities.
Finally,thepredictedEFASstreamflowiscomparedtothe localfloodprotection level.Rivergridcells where the protection level is exceeded are considered to activate the impact assessmentprocedure.The floodprotection levelsareprovidedas the returnperiodof themaximumfloodeventthatcanberetainedbythelocaldefencemeasures,e.g.dykes.Thefloodhazardinformationisthenclassifiedasdescribedabove(Error!Referencesourcenotfound.).
Figure3-4:FloodprotectionlevelsacrossEuropebasedontheJongmanetal.(2014)methodology.
3.4 I-REACTPROCESSINGOFEFASDATA
ThegoaloftheI-REACTprojectisnotonlytointegrateCopernicusdataintothedownstreamservicebutalsoaddvaluetoitfortheusersanduseittodevelopnewproducts.ThissectiondescribesthemethodologythattransformstheEFASfloodhazarddatatotheproductthatisvisualizedinI-REACTbytheuser.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:36of84
AnobjectiveoftheI-REACTprojecttaskT3.2–EuropeanEarlyWarningSystemsIntegration,istointegrateEFASdata.BeforedescribingthismethodologyandtoavoidconfusionitshouldbenotedthatwithintheI-REACTsystem,thewordtaskhasalsobeenadoptedtodescribeapieceofdataand/or information that isbeing ingested into the system.Whileamore in-depthdescription isfoundelsewhere (seeWorkPackage5documentation), thisnote is to clarify for the reader thedifference.AnI-REACTsystemtaskhasfourdigitstoidentifythedata.Forexample:task3201.ThisistheeasiestwaytodifferentiatebetweenprojectTasks(e.g.3.2)andI-REACTsystemtasks.
3.4.1 AFFECTED COMMUNITIES BASED ON THE EFAS 5-YEAR RETURN PERIODFORECAST
Thefirstdataprocessingworkflowtobedescribed is the ingestionofEFAS5-yearreturnperiodfloodhazardforecastdatatodelineatepotentiallyaffectedcommunities. IntheI-REACTsystem,this data task number is 3201 and involves the processing of the EFAS data to the NUTS(Nomenclature of territorial units for statistics) 3 administrative level [RD09].NUTS levelswerechosenbecausethishierarchicalsystemfordividinguptheEUterritoryhelpsharmonisethedataacrossEuropeandhavebeendelineatedforspecificdiagnoses.
Error!Referencesourcenot found.describestheprocessingchainthat isusedtogeneratetask3201.ThemajorityoftheprocessingtasksarebasedontheuseofOpenSourcetechnologiesbothtogatherandthenprocessthedata.
Asmentionedintheprevioussection,thefloodhazardclassificationdataisproducedatthesametimeastheEFAS00:00and12:00forecasts.Therefore,thetaskprocessingisscheduledtorunsoonafter the expected time that the data would be made available. Any kind of forecasting iscomputationally intensive. Furthermore, the EFAS flood forecasting is based on other weatherforecastoutputsandeventhoughtheyareoperational, issuescanoccurthateitherprolongtheprocessing or stop the forecast from being produced. Consequently, the task 3201 processingactuallychecks fornewfloodhazarddatathreetimesdaily incasethefloodforecastswerenotproducedontimeinordertocapturetheEFASdataassoonasitisavailable.ThisistoensurethatthemodelcanberunandproducethewantedoutputandfinallydeliveredtotheI-REACTIDIasquicklyaspossibleafternewfloodhazardforecastinformationisavailable.
At the moment, the scheduling of the process is not in-line with the current state-of-the-arttechnologiesthatarebeingimplementedintheI-REACTsystem.Ideally,onewouldliketheEFASdatatoautomaticallyletthetask3201processknowthatthedataisreadyforprocessing.Atthistime,thefloodhazardinputdataisprovidedviaFTPthatrequiresthetask3201tocheckwhetherthereisanewfilebasedontimestampandagreeduponfilenamingconvention.Thisissueiscausedby the fact that EFAS does not provide any public API’s to connect with the outputs that areproduced. However, this may change in the future and would certainly improve downstreamprocessing.Thepartnerswillupdatetheprocessifandwhenthereareserviceupgradeswithrespecttothisissue.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:37of84
Thediagrambelow (Error!Reference sourcenot found.) describes theprocess toproduce task3201. The EFAS flood hazard forecast data is taken directly from a secure FTP1 (File TransferProtocol)server.AccesstotheFTPserverrequiresauthenticationbecauseoftheEFASdatalicensingand sharing restrictions. Since the availability of the data varies from day to day, i.e. not allscheduledEFASrunsaresuccessful,thetask3201processneedstokeeptrackofthelastsuccessfulprocess.BykeepingtrackofthesuccessfulEFASoutputs,thetask3201startsthecurrentprocessfromthistimeplus1day.Duetolateand/orincompletefloodhazardforecastruns,thiscouldbethemodelrundateminus1to2days.
Asshownintheprocessdiagram,thereisalocaldatabasetablethatkeepsthisinformationcurrent.The PostgreSQL2 database table is queried based on in-house developed Python3 programminglanguagecodetoqueryforthelastsuccessfulfloodhazardforecastrundate.
Thetask3201processthendownloadstheEFASarchivefilecontainingtherasterdatainGeoTiff4formatted files for the floodhazard forecastbasedon the5-year returnperiodand the20-yearreturnperiod.
Figure3-5:DescriptionoftheprocessingchainfortheI-REACTtask3201fortheproductionofthe5-yearreturnperiodfloodhazardforecastoutput.
OncetheEFASdataisstoredlocallyandextractedfromthearchive,thetask3201modelusesthePython library rasterstats, to extract the statistical values. Rasterstats is a Python module for
1https://en.wikipedia.org/wiki/File_Transfer_Protocol2https://en.wikipedia.org/wiki/PostgreSQL3https://en.wikipedia.org/wiki/Python_(programming_language)4https://en.wikipedia.org/wiki/GeoTIFF
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:38of84
summarizinggeospatialrasterdatasetsbasedonvectorgeometries5.Therasterstatslibraryisrunagainst theGeoTiff rasterandavector fileof theNUTS5administrativeboundaries.ThevaluesextractedfromtheGeoTiffpixelsarethenumberoffloodforecastmodelspredictingapotentialfloodforthatspecificarea.InordertopreparetheresultsforI-REACT,apercentageiscomputedbydividingthevaluefromthepixelbythetotalpossiblenumberofmodelrunswhichis51.ThisresultisthenstoredlocallyinthePostgreSQL/PostGIS6databasethatislinkedtotheNUTS5geospatialID.
APostGISviewthenmatchestheCommuneIDtotheIDofatablecontainingtheCommunedetailsincludingthegeometry.TheresultfromthisviewisthencalledfromthePythonModeltoloadintothespatialdatalibraryGeoPandas.GeoPandas7isanopensourceprojectwhosegoal istomakeworkingwithgeospatialdatainPythoneasier.
Using GeoPandas, the model checks each geometry and flood maximum percentage value asdescribedaboveandaddsstylerelatedcolumnsandvaluesforeachrow.ThisinformationistobeusedinthefinalpartoftheprocesswheretheGeoJSONfileisproduced.
Asshowninthetask3201processfigure(Error!Referencesourcenotfound.),theprocessthensavesthedataobjecttoavectorfileofthetypeESRIShapefile8(SHP).TheShapefilecontainsalltherequireddatabeforeitisfinallyconvertedtothewantedI-REACTstandardvectorfiletypeGeoJSON.TheGeoJSON9geospatialvectordatafileformatisanopenformatandisstandardsbased.ThisfiletypewaschosentobetheprimarygeospatialdataformatforsharingwithintheI-REACTprojectspecificallybecause it is standardsbased,openandeasily ingestedbymanydifferent tools.Theadditionalstep,i.e.convertingtheshapefileintoaGeoJSONfile,isrequiredinordertoallowforthemodeltoproducetheGeoJSONdatabasedonthelateststandardsincludingtherighthandruleforvectorgeometry.
OncetheGeoJSONfilehasbeenproduced,themodelbuildstherequiredIDImetadataobjectthatdescribesthegeneratedGeoJSONfile.ThecodethenusestheI-REACTIDIservicetouploadthedataproduced.Oncecomplete,themodelstoresalogoftheprocesstothelocaldatabase.
ThefollowingFigure3-6showsthefinalGeoJSONresultforasectionofNorthWesternPolandandNorthEasternGermany.TheshapesarebasedontheNUTS5leveladministrativeboundariesthatpotentiallycouldbeaffectedbyflooding.
5https://github.com/perrygeo/python-rasterstats6https://en.wikipedia.org/wiki/PostGIS7http://geopandas.org/8https://en.wikipedia.org/wiki/Shapefile9http://geojson.org/
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:39of84
Figure3-6:Final5-yearfloodhazardforecastoutputinGeoJSONfileformat.Thedarkerthebluecolour,thegreaterthefloodhazardforcast.
3.4.2 AFFECTED COMMUNITIES BASED ON THE EFAS 20-YEAR RETURN PERIODFORECAST
TheI-REACTtask3202isthesameasthetask3201fromaprocessingperspectivebuttheEFASinputisdifferentandthereforeprovidesadifferentendresult.Therefore,manyofthedetailsprovidedintheprocessingdescriptionfortask3201willnotberepeatedhere.
Theprocessfortask3202isrunagainstthe20-yearfloodhazardforecastproducedbyEFAS.Theprocessingdiagram is presented inError!Reference sourcenot found.where the communitiespotentiallyaffectedbyfloodingaredelineatedbasedonNUTS5.
Figure3-7:DescriptionoftheprocessingchainfortheI-REACTtask3202fortheproductionofthe20-yearreturnperiodfloodhazardforecastoutput.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:40of84
Tohighlightsomeofthedifferencesinthedataandresultsofthetwotasks,thefollowingError!Referencesourcenotfound.presentstheinputdataforagivenlocationforboththe5-yearand20-yearreturnperiods.Notethattheexpectedbehavioristhatwhereverthereisa5-yearforecasta20-yearforecastshouldalsobepresent.However,thereverseisnottruebecauseundernormalconditions,alargerfloodevent(20-yearreturnperiod)shouldnotbeforecastwithouttheabilitytoforecastasmallerevent(5-yearreturnperiod).ThisisshowninError!Referencesourcenotfound.wherethe5-and20-yearreturnperiodforecastsareoverlapping.Indarkblueisshownthe20-yearwhereasthelightbluegridpointsarethe5-yearreturnperiod.
Thenextstepistotakethosegridpointsandmakethemmoreeasilytointerpretandvisualize.
Figure3-8:Comparinga5-year(leftside)and20-year(rightside)returnperiodfloodhazardforecast.Herethe'raw'dataispresentedatthepixellevel.
Figure 3-9: Five- and twenty-year return periods for the floodhazard forecast overlapping. Please refer toError!Referencesourcenotfound.fortheseparatefloodhazardforecasts.
Duetothefactthatdecisionswithrespecttohowtoreacttoaflooddisasterwarningisorchestratedbythenationaland/orlocalauthoritiesresponsible,therelevantregionisprovidedastheoutput
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:41of84
ratherthanapixel.Thisprocessingwasalsoappliedfortheother I-REACTservicesthatneedtoknowplacenamesand/orlocationstomonitorotherinformationsourcesonceawarninghasbeenposted.Forexample,havingplacenamesallowsthesocialmediamonitoringto lookoutforthenameswithrespecttofloodoccurrences.
InError!Referencesourcenotfound.,theNUTS5levelvectordataisplacedinthebackgroundtoshowhowthedifferentgridvaluesintersectwiththeadministrativeboundaries.ThesetwotypesofinformationaremergedtogetherinordertoproducethefinaloutputthatissenttotheI-REACTIDIforsharingwithotherpartnersandvisualization.
The finalGeoJSONoutput ispresentedwith the statistics foreachNUTS5 levelpolygon (Error!Referencesourcenotfound.).Thedifferencesintheblueshadingindicatesthenumberoffloodmodels indicatinga flood.Asdescribed in theprevioussection, therearea totalof51differentmodelsusedandtherefore,themoremodelsthatareforecastingaflood,thegreaterthepossibilityofanactualflooddisaster.
Figure3-10:5-and20-yearfloodhazardforecastwithNUTSlevel5datainthebackground.
Figure3-11:Thefinal20-yearfloodhazardforecastinGeoJSONformat.Theblueshadingprovidesinformationaboutthenumberofmodelsforecastingaflooddisaster.Darker=moremodelsforecastingflood.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:42of84
It is up to the national, regional, and/or local authorities to interpret the flood hazard forecastwarnings.Thedarkertheblue,themoretheyshouldbepayingattentiontotheirlocalforecastsandputting intomotion their respective action plans. Each polygon that is provided to the I-REACTsystemalso provides the distribution of the floodhazard forecast values. This essentially is thenumberofmodelsoutof51possiblethatare indicatingriverfloodpotentialbasedonthemostrecentweatherforecasts.
3.5 COPERNICUSEFFISDATAPORTAL
The European Forest Fire Information Systemprovides standardisedWebMap Service10 (WMS)interface that is OGC compliant. From the EFFIS data and services webpage11, it is possible todownloaddata.TheError!Referencesourcenotfound.providesalistingofthedifferentdatasetsmadeavailablethroughWMS.
Table3-2:AvailableEFFISdatathroughWMS.
Dataset HotSpotsMODIS
HotSpotsVIIRS BurnedAreaMODIS
BurnedAreaVIIRS
Layer Last24hours Last7days Last30days Last90days Fireseason
TheWMSstandardprotocolservesgeo-referencedmapimages.Thismeansthatthesearesimplygeo-referenced pictures and therefore they do not store the real measures that the I-REACTdownstreamserviceisinterestedin.WMSisprimarilyusedforvisualisationpurposes.However,theI-REACTprojectwantstofurtherusetheavailableEFFISwildfireinformationtointegrateitintothedifferentgeo-processingchainsandbusiness logic.Therefore,WMSisnotthestandardprotocolthatisusefulwhendownstreamservicesneedaccesstotheactualmeasuresrepresentedbytheimages.Tobetterunderstandthisissue,pleasereferbacktoError!Referencesourcenotfound.wherethecurrentsituationviewerisshowingthefiredangerforecastbasedontheweatherindex(FWI).ThedifferentcoloursofthevisualisationrepresentactualvaluesoftheFWI.However,whenaccessingthisdataviatheWMS,oneonlyknowsthatitisredorgreen.Onecannotknowtheactualpixelvaluethatcausedthatareatobecolouredred.Adifferentprotocolwouldneedtobeavailableinordertogetaccesstotherawdatathatisneededforfurtherprocessingandnotjustvisualisation.
Thereisapossibilitytoacquirethewanteddataofflinethroughadataaccessrequest.EOXPLOREtookadvantageofthisinordertobetterunderstandthedataavailablewhentheJRCwelcomedan10https://en.wikipedia.org/wiki/Web_Map_Service11http://effis.jrc.ec.europa.eu/applications/data-and-services/
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:43of84
employeeasavisitingscientist.TogetherwiththeEFFISteam,wewereabletogetaccessandseethetypeofdatathatEFFIShasintheirwildfiredatabases.Unfortunately,thisisnotaviablesolutionfordownstreamservicesbecauseofthelagtimetomakewanteddataavailable.
TheEFFISteamiscontinuingtoworkondevelopingtheirservices.ThereisatestwebsitewheretheEFFISAPIthatpotentiallywillbemadeavailableinthefutureishosted[RD10].Basedonthedetailsthat can be viewed through this test website, the I-REACT partners are very confident thateventuallyafullyautomatedconnectionthroughastandardAPIsuchasREST12willbepossiblewithEFFIS.Furthermore, therearemanynewdataandservices listed fromwhich I-REACTcouldalsobenefit.ThisisexactlyhowdownstreamservicesshouldaccessCopernicusservices.
3.6 COPERNICUSEFFISDATAPORTAL
Unfortunately, thisalsomeant that theprojectpartnerscouldnot ‘waitandsee’whetherEFFISwould implement the necessary API to access the wanted European wildfire data from theCopernicusEFFISportal.Consequently,analternativesolutionneededtobefoundinordertobesurethattheI-REACTsystemwasabletoingesttheneededEuropeanwildfireinformation.
The solution in this casewas on the one hand to use the I-REACT partners’ access toweatherforecastdataandprocessingexpertisetore-producetheEuropeanFireWeatherIndexandontheotherhandtofindanotherproviderofthefirehotspotdata.ThenexttwosectionsprovidethedetailsofhowthehotspotdataandtheFWIareprocessedandprovidedto I-REACT.ThesearespecificallyrelatedtotheI-REACTtasks3203to3206.
3.6.1 THEFIREWEATHERINDEXOneof theways thatEFFISmonitors forwildfirepotentialacrossEurope is tocompute theFireWeather Index (FWI). This is aweather forecastbased indexdevelopedby theCanadian ForestServiceandtestedbytheEFFISteamtobeoneofthebest.ItrequiresasinputaweatherforecastwhichisprimarilyprovidedbytheECMWFalthoughEFFISalsousesothernationalweatherforecastmodels.ForI-REACT,ourpartnerFMIhasaccesstotheEuropeanweatherforecastfromtheECMWFbutalsoproducesafinerspatialresolutionweatherforecastcalledtheGrandLimitedAreaModelEnsemblePredictionSystemorGLAMEPSforshort.BothoftheseforecastsareusedtocomputetheFWI.Another I-REACTpartner,Meteosim isproducing theFWIbasedon the two forecasts. Thedetailsofhowthis isdone isprovidedthroughreporting fromthe I-REACTWorkPackage4andthereforewillnotbeduplicatedhere.
TheFWI13essentiallyprovidesameasureofwildfiredangerbasedonweatherforecasts.
12https://en.wikipedia.org/wiki/Representational_state_transfer13http://cwfis.cfs.nrcan.gc.ca/background/summary/fwi
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:44of84
3.6.2 AFFECTEDCOMMUNITIESBASEDONTHEFWIThefirstI-REACTtasktoprocesstheFWIdatais3203todelineatethecommunitiesthathaveahighfire danger potential. This task is based on the computation of the FWI based on the ECMWFforecasts.ThisissimilartotheprimaryFWIproductproducedbyEFFISandavailablethroughtheirviewer.
AswiththeprevioustasksusingEFASinformation,theseprocessingchainsalsomakeuseofopensourceprojectsincluding:
• PostgreSQL/PostGIS–forgeo-spatialdatabaseaccess;
• Python–includingmanydifferentdataprocessinglibraries;
• AndGDAL–theGeospatialDataAbstractionLibrary14.
Furthermore,theseprocessingchainstakeadvantageoftheServiceBusmessagingserviceandtheIDI of the I-REACT system as it was designed and developed specifically for this project. Error!Referencesourcenotfound.showstheprocessingsteps.
Figure3-12:TheprocessingchainforI-REACTtask3203basedontheECMWFFWIdata.
AsshowninthedatalisteningsectionofError!Referencesourcenotfound.,thereisadatalistenerthat has been implemented in Python. Within a continuous loop, this listener is listening formessagesfromtheI-REACTsystem,i.e.theServiceBus,foranyupdatesregardingtheI-REACTtask
14http://www.gdal.org/
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:45of84
4302.Notethatthistaskbeginswiththenumberfourwhichmeansitisbasedonworkpackage4results.Specifically,theI-REACTtask4302istheECMWFbasedFWIdataproducedbytheprojectpartnerMeteosim in the NetCDF15 format as a substitute for the EFFISmissing data. The datalistenerfortask3203issubscribedtospecificallyreceivemessagesaboutthetask4302.
Themessaging service,as implemented in the I-REACTsystem, is thecurrent state-of-the-art toimplementingwebbasedservices.Thisallowsaserviceto immediatelyreactwhenamessage isreceived.Unliketheoldersolutionswherebyaservicewouldneedtoconstantlycheckwhetherafilewasupdatedremotely.
AsshowninthedatalisteningsectionofError!Referencesourcenotfound.,thereisadatalistenerthat has been implemented in Python. Within a continuous loop, this listener is listening formessagesfromtheI-REACTsystem,i.e.theServiceBus,foranyupdatesregardingtheI-REACTtask4302.Notethatthistaskbeginswiththenumberfourwhichmeansitisbasedonworkpackage4results.Specifically,theI-REACTtask4302istheECMWFbasedFWIdataproducedbytheprojectpartnerMeteosim in the NetCDF16 format as a substitute for the EFFISmissing data. The datalistenerfortask3203issubscribedtospecificallyreceivemessagesaboutthetask4302.
Themessaging service,as implemented in the I-REACTsystem, is thecurrent state-of-the-art toimplementingwebbasedservices.Thisallowsaserviceto immediatelyreactwhenamessage isreceived.Unliketheoldersolutionswherebyaservicewouldneedtoconstantlycheckwhetherafilewasupdatedremotely.
Whenamessageisreceiveditalertsthetask3203tothefactthatthereisnewFWIdataavailableforprocessing.ThisessentiallytriggerstheprocessingchainbasedontheECMWFFWIdata.ThefirststepistoquerytheI-REACTIDIforthespecificdataserviceUniformResourceLocator17(URL)forthetaskthatsentthemessage,i.e.4302inthisparticularcase.ThequeryreturnsalistofthedataURLscreated.Thislistismadeupofthemostcurrentdataplusalldatacreatedinthelasttwodays since.With this list in-hand, themodel chooses from the list the latest data available anddownloadsittothelocalprocessingdirectory.
Once the data has been downloaded, the data processing can begin. First, the data must beextractedfromthedownloadedNetCDFfile.Specifically,themodelneedsthemaximumFWIvaluesfor the currentdateaswell as thenext twodays. This is possiblebecause theFWI isbasedonweatherforecastdata.TheresultisthecreationofthreedifferentFWIdatasetsinGeoTIFFformat.Thenextstepistocomparetherasterdata(FWI)totheNUTSlevel5vectordata.BothtypesofgeospatialdataareintheprojectionEPSG:4326whichistheWorldGeodeticSystem(WGS)1984standard.
TheFWIrasterdataisprocessedusingthePythonrasterstatslibraryinordertoextractthepixelvaluesandthenreturnthestatisticaldataforeachNUTS5polygon.TherastervaluesincludetheFWImaximum,minimumand90thpercentilewhicharethenstoredwiththeNUTS5levelpolygon
15https://en.wikipedia.org/wiki/NetCDF16https://en.wikipedia.org/wiki/NetCDF17https://en.wikipedia.org/wiki/URL
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:46of84
IDinthelocalPostGISdatabase.APostGISviewthenmatchestheNUTSlevel5IDtotheIDofatablecontainingthepolygondetailsincludingthegeometry.TheresultfromthisviewisthencalledfromthePythonmodeltoloadintothespatialdatalibraryGeoPandas.
UsingGeoPandas,themodelcheckseachgeometryandtheFWImaxvalueandaddsstylerelatedcolumnsandvaluesforeachrow.ThisinformationisusedinthefinalstepsoftheprocesstoproducethefinalGeoJSONoutputfile.
ThenextstepintheprocesssavesthedataobjectasanESRIshapefilecontainingalltherequiredinformationbeforeconvertingthisshapefileintoaGeoJSON.ThisadditionalstepisrequiredtoallowtheprocessingmodeltoproduceastandardisedGeoJSONfile,includingtheadditionoftheright-handruleforspecifyingpolygongeometry.OncetheGeoJSONfileisready,themodelbuildstheassociatedIDImetadataobjectdescribingthethreegeneratedGeoJSONfiles.InordertosharetheGeoJSONfilescorrecly,themetadataincludestheFWIleadtimeaswellastheorderinwhichtheyshouldbevisualizedfordisplaybytheI-REACTclient.Theprocesscontinuesbyuploadingthetask3203resulttotheI-REACTserviceviatheIDI.Oncetheprocessiscomplete,themodelstoresalogoftheprocesstothelocaldatabase.
ThefollowingError!Referencesourcenotfound.presentsanexampleofthetask3203GeoJSONoutput.ThetablepresentssomeofthevaluesassociatedwiththeNUTSlevel5polygonsincludingtheFWIstatistics.
Figure3-13:TheI-REACTtask3203outputbasedonECMWFFWIdata.ThetablepresentssomeofthedetailsstoredintheGeoJSONpolygons.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:47of84
3.6.3 AFFECTEDCOMMUNITIESBASEDONTHEFWI–NUTSLEVEL3Similar to the previous section, it is also possible to produce the FWI but based on largeradministrativeboundaries.Inthiscase,theadministrativeboundariesarebasedonNUTSlevel3.ThisisusefulincaseswheresimilarFWIarecoveringlargeareasandmoregeneralnationalstatisticsarerequired.
ThisprocessingchainwasgiventheI-REACTtasknumber3206andisalsobasedontheECMWFweatherforecastFWI.Duetothefactthatthemajorityoftheprocessingissimilartothatofthetask 3203, the details will not be repeated in this section. Error! Reference source not found.presents thesameprocessingchainwith theoutputof task3206presented inError!Referencesourcenotfound..NotethemuchlargerregionshavingthesameFWIoutput.ThelocationisofsouthernFranceandnorth-easternSpain.ThetableinsidethefigurepresentsthedifferentstatisticsthatcanbequeriedthroughtheGeoJSONfileandthereforearealsoavailablethroughtheI-REACTuserinterface.
Figure3-14:TheI-REACTtask3206outputbasedonECMWFFWIdata.ThetablepresentssomeofthedetailsstoredintheGeoJSONpolygons.
3.6.4 FIREHOTSPOTDETECTIONACROSSEUROPEThe FWI provides ameasure of thewildfire danger however, this only provides information toexpertsandlocalauthoritiesaboutthepotentialofthewildfireshouldafirestart.Thefirehotspotsare locations where elevated temperatures were identified through satellite based EarthObservation (EO) techniques. They are also called ‘active fire data’ indicating the fact that apotentiallyactivefirehasbeenidentifiedinthesatelliteimagery.
ActivefiredataiscurrentlyacquiredviatwodifferentEOsensors:theModerateResolutionImagingSpectroradiometer(MODIS)andtheVisibleInfraredImagingRadiometerSuite(VIIRS).Inthefuture,itishopedthattheCopernicusSentinel-3sensorwillbeaddedtothelist.Thehotspotsoractivefiresarebasedonthermalprocessingoftheimagerytoidentify‘hot’pixels.Firedetectionsfrom
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:48of84
thetwosensorscandifferslightlyduetothespectralbandsthattheyusetoidentifyhotobjects.ThereisalsoadifferenceinthespatialresolutionbecauseMODISprovidesoutputat1kmspatialresolutionwhereastheVIIRSsensordataisabletoprovidegreaterresponseoverfiresofrelativelysmallareasduetoitsspatialresolutionof375m.Furthermore,theactivefiredataaredesignedtobeprovidedinnearreal-time.Whatthismeansisthattheteamsprocessinganddeliveringattempttoprocessandmakeavailablethehotspot informationasquicklyaspossible. Ingeneral, this iswithinthreehoursofthesatelliteacquiringtheimagery.
BothtypesoffirehotspotdataareusedwithintheTask3.2.WhileitisnotclearfromwhichservicetheEFFISteamdownloadstheirfirehotspotdata,itisknownthattheyarefromthesamesatellites.Forthisreason,thetwofirehotspotdataareintegratedintotheI-REACTsystem.
3.6.5 AFFECTEDCOMMUNITIESBASEDONTHEVIIRSFIREHOTSPOTDATAThe I-REACT task 3204 involves the processing of VIIRS hot spot point data and again the dataprocessingisasmuchaspossiblebasedonopensourcetechnologies.Suchtechnologiesareusedbothingatheringthedataaswellasprocessingit.
Atthemoment,VIIRSactivefiredataforEuropeisdownloadedfromanFTPserver.Unfortunately,there isnomessagingservicethatalertsthetask3204tonewdataandthereforethetaskmustcheckregularlyfornewinput.However,thismaychangeinthefutureifamoresuitableserviceisfound.Thefollowingdiagram(Error!Referencesourcenotfound.)presentstheprocessingstepsoftask3204.
Figure3-15:TheprocessingstepsoftheI-REACTtask3204toprocessVIIRSbasedactivefiredatatodelineateaffectedcommunities.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:49of84
OncethehotspotdatahasbeendownloadedfromtheFTPserver,whichoccursoncedaily,theprocessingcanbegin.Aswithotheroperationaldata,thereisnoguaranteethatitwillbeavailablewhenitisexpected.Therefore,asshownintheprocessingdiagram,itisnecessarytocheckfornewdataagainstlocallogstobesureunnecessaryprocessingisdone.
TheVIIRSEuropeandailydataisdeliveredintheCSV18fileformatintheEPSG:4326projection.Theavailabilityofthedatavariesfromdaytodayandsotheprocessneedstokeeptrackofthelastsuccessfulprocessandstartsthecurrentprocessfromthistimeplus1day(thiscouldbethemodelrundateminus1to2days).Thisprocessrunsasaloopandthereforeiftherewasanissueand/oragapintheavailabilityofdata,theprocesswillattempttocatchupinprocessingalltheavailabledatatothecurrentdate.
Toachievethis,thePythoncodequeriesalocaldatabasetableinPostgreSQLforthelastsuccessfulrundate.ThefirststepoftheprocessistoimportandloadthehotspotdataintoaPostGISspatialtable,convertingtheCSVdataintopointgeometries.APostGISviewisthenusedtoqueryaspatialtableoftheNUTSlevel5boundariesacrosstheEUtofindthosewhichcontainahotspotandthenumberofhotspotswithinthepolygon.TheresultfromthisviewisthencalledfromthePythonmodeltoloadintothespatialdatalibraryGeoPandas.
UsingGeoPandas,themodelcheckseachgeometryandcountsthenumberofvaluesindicatinghotspots.Thentheprocessaddsstylerelatedcolumnsandvaluesforeachrow.ThisistobeusedinthefinalpartoftheprocesswheretheGeoJSONisproduced.TheprocessthensavesthedataobjecttoanESRIshapefilecontainingalltherequireddatabeforeconvertingthisshapefileintoaGeoJSONfiletype.ThisadditionalstepisrequiredtoallowforthemodeltoproducethewantedGeoJSONoutputfileinthelateststandardincludingtherighthandruleforgeometry.
With the final GeoJSON output file ready, the model builds the required IDI metadata objectdescribingthegeneratedGeoJSONfilethatwillbedisplayedbytheI-REACTclient.ThecodethenusestheI-REACTIDIservicetouploadthefirehotspotinformationlayer.Oncecompletethemodelstoresa logof theprocess to the localdatabase.A sampleGeoJSONoutputof the task3204 ispresentedbelow(Error!Referencesourcenotfound.).
AsshownintheError!Referencesourcenotfound.,eachpolygonprovidesthenumberofhotspotpixelsidentifiedwithintheregionaswellasotherinformationaboutthepolygonsuchaslocationandID.
ThesameVIIRSactivefiredataisusedfortheI-REACTtask3205.Thistaskshowsthelocationsofthepixelsthatwereidentifiedasfirehotspots.Thetask3205involvesaccessingtheVIIRShotspotpointdataandconvertingittotherequiredGeoJSONoutputformattouploadtotheIDIandthendisplayedviatheuserinterface.Thisprocesscontinuestobebasedonanumberofopensourcetechnologiesthatareusedtogatherandthenprocessthedata.
18https://en.wikipedia.org/wiki/Comma-separated_values
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:50of84
Figure3-16:TheVIIRSactivefireproductusedtoproducedelineationsoftheNUTS3levelregionswheretheywereidentified.Thetableshowsthateachofthepolygonsprovidesthenumberofhotspotpixelsfoundwithinthatregion(count).
ThediagraminError!Referencesourcenotfound.describestheprocessingsteps.Thefirstpartoftheprocessfollowsthesamestepastask3204becauseit isbasedonthesameVIIRSactivefirepointdatathatisavailableinthePostGIStable.APostGISviewisthenusedtoqueryaspatialtableofNUTSlevel5boundariesacrosstheEUtofindthosehotspotswhicharewithinthefullextentofthecommuneboundaries.TheresultingpointdatafromthisviewisthencalledfromthePythonmodeltoloadintothespatialdatalibraryGeoPandas.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:51of84
Figure3-17:TheprocessingstepstoproducetheI-REACTtask3205GeoJSONoutput.
UsingGeoPandas,themodeladdsstylerelatedcolumnsandvaluesforeachrow.ThisistobeusedinthefinalpartoftheprocesswheretheGeoJSONoutputfileisproduced.Thetask3205processthensavesthedataobjectasanESRIshapefilecontainingalltherequireddatabeforeconvertingthisshapefileintoaGeoJSONfiletype.TheremainderoftheprocessingisthesameasthatfortheI-REACTtask3204.
Figure3-18:TheGeoJSONoutputoftheI-REACTtask3205showingtheVIIRSactivefiredatapointswithintheNUTSlevelpolygons.
TheexampleoutputshowninError!Referencesourcenotfound.showstheresultofthetask3205.TheVIIRSactivefirelocationsaremappedontopoftheresultsfromtask3204providingabettersenseofwherethehotspotsarefoundwithinthecontextofthecommunepolygon.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:52of84
3.7 DISCUSSION
TheTask3.2–EuropeanEarlyWarningSystemsIntegrationwasfirstandforemosttodependonthe Copernicus EarlyWarning Services, EFFIS and EFAS for data and information on flood andwildfiremonitoringtobeintegratedaspartoftheI-REACTdownstreamservices.Unfortunately,upto the time of writing of this report, those services are not available and/or able to provideoperationally data for downstream services as expected. However, even though this particularprojectTask3.2wasunabletodirectlyusetheEWSservicesdirectly,thenecessaryandproposed‘downstream’ services based on the expected Copernicus EWS have been implemented andintegrated into the I-REACT system.Thiswasachieved through close collaborationwith the JRCincludingthehostingofaresearchscientistfromEOXPLOREforthreemonths.Furthermore,thiscouldnotbeachievedwiththehelpofotherI-REACTpartnersthathaveaccesstotherequiredinputdata (FMI=weather forecast)and theexpertise toproduce thewildfiremonitoring information(Meteosim=FWI).
TheresultsachievedhavelaidthegroundworkforotherpossibleCopernicusservicesthatmayinthefutureprovidewantedinformationfordownstreamdisasterservicesbutalsoisreadywhenthewantedCopernicusEWSdatabecomesavailable.ConnectingtouseEFFISandEFFASdata inthefutureshouldbequitesimple.
Animportantpointtorememberwithrespecttothetask3.2isthefactthatitispossibletodeliverstandard GeoJSON disaster related information from a variety of sources. This is an importantprojectachievementbecauseitdemonstratesthestrengthsbehindthetechnologicaldesignofI-REACT.Eventhoughdataiscomingfromdifferentplacesandindifferentformats,theprocessingtaskscandealproducethewantedstandarddataproductbasedonopensourcetechnologiestobeingestedintoI-REACTandprovidefireandflooddisasterinsights.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:53of84
4 EXISTINGLOCALEMSINTEGRATION
4.1 OVERVIEWOFLOCALEMS
The integration of local Emergency Information Systems in the I-REACT framework focuses onobtaininginformationconcerningalertsandmeasurementsfrommonitoringsystems.Theriskswearefocusingon,forthisactivity,areflood,landslide,extremeweathereventsandfires;actually,forthesehazardswecandefinemoreeasilyaclearflowofinformationfromforecasttoemergencymanagement.
Alerts: Inthiscase,wemainlyextractandintegratethecontentofbulletinsthatpredictfiresand/ormeteorologicalandhydrogeologicalphenomena.
MonitoringSystems: Inthiscase,weextractandintegratebothrawmeasuresfromsensors,suchaswaterandraingauges,aswellasinformationaboutmeasuresexceedingwarningthresholds
To achieve this, we have designed and developed the LEMS (Local Emergency ManagementSystems),aspecificintegrationmodule,whichprovides:
§ connectiontoexternaldatasources;
§ appropriateprocessingofdataformat,fromatechnicalpointofview;
§ simplelogicalrules,usefultounderstandthecontent;
§ standardizationandtransfertotheI-REACTOR
AftertheiringestionintheI-REACTOR,thesedatabecomeprecious‘triggerevents’fortheI-REACTinfrastructure and can activate additional actions, on the basis of automatic flows, defined inadvanceandsuggestedbytheDecisionSupportSystemModule(DSS),ormanuallyscheduledbyemergencymanagers.
For example, the arrival of information about exceeding a warning threshold, in a hydrometermeasure,mayrequirea localmonitoringcampaign in theaffectedareabyUAVor technicians/expertswhocanreportthroughmobileapplications.
A floodalert codeona largearea, instead,may requireawidedisseminationof information toaffectedpopulationsand/ortheactivationofspecificinteractionswithsocialnetworks.
In this first phase we have selected and connected sources from countries where projectdemonstrations areplanned:UK, Finland, Italy, Spain and SavaRiverBasin.However, the LEMSmodulehasbeendesignedanddevelopedtoallow,inthefuture,furtherintegrationsofsimilardatasourcesfromotherEuropeancountries(Table4-1).
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:54of84
Thefollowingtablesummarizesinformationabouttheconnectedsources.
Table4-1:Connectedsources
Country Owner MainHazards Content Originalformat
Finland FMI Extremeweather/Flood/Fires
Warningbullettin Xmlcap
Italy ARPAPiedmont
Extremeweather/Flood
Meteo/idrogeologicalbullettin
Xmlcap
Italy ARPAPiedmont
Extremeweather Thunderstormbullettin Xmlcap
Italy ARPAPiedmont
Flood Measures from sensorsexceeding warningthreshold
Xmlcap
Italy AIPO Flood Raw measures fromsensors
Dbf
Spain CatalunyaCivilProtection
Extremeweather/Flood/Fires
Warningbullettin Webservice
UnitedKingdom
MetOffice Extremeweather/Flood
Meteo/idrogeologicalbullettin
Webservice
UnitedKingdom
EnvironmentalAgency
Flood LocalFloodwarnings Webservice
UnitedKingdom
EnvironmentalAgency
Flood Raw measures fromsensors
Webservice
Bosnia, Croatia,Montenegro,Slovenia,Serbia
Sava RiverBasinCommission
Flood Meteo/idrogeologicalbullettin
Dataset
NoteonXmlCapStandard
If alreadyavailable,wepreferred to integrateexternal sourcesexposed inXmlCap format. TheCommonAlertingProtocol(CAP) isastandardprotocolusedtofacilitateemergencyinformationsharinganddataexchangeacrosslocalandnationalorganizationsthatprovideemergencyresponseandmanagementservices[RD11].Itisareliablecandidatetobecomeaworldstandardandmanyorganizations have already adopted it. Its wide diffusionwill reduce the disadvantages of non-homogeneousinformationformats.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:55of84
4.2 SURVEYOFEXISTINGEMERGENCYMANAGEMENTSYSTEMS
Asmentioned,theanalysisoflocalEMShasbeenfocusedoncountrieswherepilottestsofI-REACTsystemwill be conducted. The comparison of those various systems has led to generalize dataintegrationservices.
4.2.1 FINLANDTheFinnishMeteorologicalInstitute(FMI)isaresearchandserviceagencyundertheMinistryofTransport andCommunications. Themainobjectiveof theFinnishMeteorological Institute is toprovide theFinnishnationwith thebestpossible informationabout theatmosphereaboveandaroundFinland, forensuringpublic safety relating toatmospheric andairbornehazardsand forsatisfyingrequirementsforspecializedmeteorologicalproducts[RD12].
TheFinnishMeteorologicalInstitutemonitorstheweatherinFinlandandelsewhereintheworldround the clock. Depending on the situation, it issues warnings on dangerous or hazardousphenomenainFinland.
TheLEMSModule integratesanXmlCapstreampublishedbytheFMIthat includes informationcontainedintheirweather-warningbulletin(Figure4.1).
Figure4-1:WarningsissuedbytheFinnishMeteorologicalInstitute
Theseverityofthewarninglevelisshownonthemapbymeansofathree-colourcodesystem.
§ green-nomajordanger
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:56of84
§ yellow-dangerousweathermayoccur.Youareadvisedtotaketheweatherconditionsintoaccountwhenyouareexposedtoweather.Youshouldalsokeepaneyeontheweatherandavoidrisks.
§ orange - dangerousweather. Theweathermaycause injuriesandmaterialdamage.Youshouldavoidrisksthatmaybecausedbyweather.Youareadvisedtokeepaneyeontheweatheronaregularbasisandfollowtheinstructionsissuedbytheauthorities.
§ red-verydangerousweather.Injuriesandmaterialdamagecanbeexpectedoverawidearea.Youshouldkeepaconstanteyeontheweatherandtheawarenesslevel.Youshouldalso follow the instructions issued by the authorities and be prepared for exceptionalmeasures.Theredcolourappearsonthewarningmapveryrarely.
Warningsareissuedseparatelyforeachdayoftheweek.Forexample,onMonday,warningswillbeissuedforMonday,Tuesday,Wednesday,ThursdayandFriday.Theywillindicatetheworstsituationduringthatday,suchasthestrongestwind.Thewarningthresholdishigherinwarningsfor2–5days,andallwarningsarenotgivenforperiodslongerthan24hours.
TheFinnishMeteorologicalInstituteisaninformationproviderforMeteoalarm.eu19,thewebsiteinformingpublicandauthoritiesaboutsevereweatherconditionsin36Europeancountries.
WeatherparametersincludedinMeteoalarmdifferfromcountrytocountry.FinnishMeteorologicalInstitute(FMI)deliverssevereweatherinformationtoMeteoalarmaboutfollowingparameters:
§ Wind
§ Rain
§ Thunderstorms
§ Extremehightemperature(Heatwave)
§ Extremelowtemperature(Coldweather)
§ Snow/ice(Roadweather)
§ Forestandgrassfires
4.2.2 ITALYItaly
EmergencyManagementsSystemsinItalyareco-ordinatedbytheDepartmentofCivilProtectionwiththeFunctionalCentresNetwork,havingaCentreineachRegionandautonomousProvince.InthisdocumentwearefocusingontheexistingEMSintwoareas,PiemonteregionandtheareaofthePoriverstretchnearFerrara:thesetwoareaswillhosttheItalianpilotsitesforI-REACT.
Piemonte:Amongotheractivities,ARPA(theRegionalAgencyforEnvironmentalProtection)istheFunctionalCentreinPiemonteandoperatesahydro-meteorologicalmonitoringsystemthatuses
19http://meteoalarm.eu/
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:57of84
automatic tools to determine soil and atmospheric conditions. The interpretation of the dataacquired and the results of data processing by modelling make it possible to carry out thecontinuoussurveillanceoftheenvironmentandtheterritory,tomanageprediction,alertingandmonitoringsystemsforsituationsrequiringspecialattention,andtoupdateonacontinuousbasistheclimatologicaldocumentationforuseinplanningandprogrammingactivities.
TheLEMSModuleintegratesthreeexperimentalXmlCapstreamspublishedbyARPA:
1. Meteo/Hydrogeologicalwarningbulletin(Figure4-2).
TheAgencyissuesahydrogeologicalwarningbulletineveryday,butithasavalidityof36hours.
Figure4-2:Meteo/HydrogeologicalwarningbulletinissuedbyARPAPiemonte
Thebulletinprovides informationon11homogeneousareas. In the first section, foreacharea,criticalsituationsarehighlightedforweatherevents(avvisimeteo).Inthesecondsection,criticalsituationsrelatedtotheeffectsofweathereventsarehighlighted,inthesameareas.Inthiscase,thecriticallevelisexpressedbyfourstandardcolourcodes.
2. Thunderstormbulletin
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:58of84
This service, through meteoradar observations, generates alerts for the imminent arrival of athunderstorm and displays nowcast data on the position and amplitude of thunderstorms inprogress(Figure4-3).
Figure4-3:LivestormserviceissuedbyARPAPiemonte
3. Measuresfromsensorsexceedingwarningthreshold(Figure4-4).
Theserviceprovides,foreachofthewaterrainsensorsmanagedbyARPA,awarningconcerningtheovercomingofawarningthreshold.Inthiscase,besidestheindicationofthesensorinvolved,andthelevelofcriticality,thesurroundingareapotentiallyaffectedishighlighted.
Figure4-4:WarningsonsensorsissuedbyARPAPiemonte.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:59of84
Poriverstretch(Figure4-5and4-6):thePoRiverInterregionalAgency(AIPO)acquiresmeasuresfromalargenetworkofwaterandrainlevelsensorsmanageddirectlybyitself,orbythe4regionsthebasinismadeupof:Piemonte,Lombardia,VenetoandEmiliaRomagna.InI-REACT,atthisstage,weintegrateroughmeasurementsfromsensors.Futuredevelopmentcouldincludetheintegrationofthelogicrequiredtocomparemeasureswithalertthresholdvalues,asinthecaseoftheARPAPiemonteservice.
Figure4-5:Monitoringhydrostations–RiverPo
Figure4-6:PoriverstretchbetweenFerraraandPanaro
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:60of84
4.2.3 SPAINForSpain(Catalunya)theLEMSModuleintegratesawarningbulletinpublishedtheMeteorologicalServiceofCatalunya(http://www.meteo.cat/wpweb/divulgacio/la-prediccio-meteorologica/avisos-smp/).
Thebulletinisissuedeachtimethereisthepossibilityofreachingaspecificthreshold(Table4-2).Thethresholdchangesdependingontheweatherevent(idmeteor).
Table4-2:Catalunya
Meteor Lowthreshold Thresholdhigh
Intensityofrain
Intensity>20mm/30minutes Intensity>40mm/30minutes
Accumulationofrain
Accumulated>100mm/24hours Accumulated>200mm/24hours
Snowaccumulatedin24hours
thickness≥0cm,altitudelessthan300meters thickness > 5 cm in height less than 300meters
thickness>2cminaltitudeover300metresupto600metres
thickness>15cminaltitudeover300metresupto600metres
thickness>5cminaltitudehigherthan600metresto800metres
thickness>20cminaltitudehigherthan600metresto800metres
thickness>10cminaltitudehigherthan800metresupto1000metres
thickness>30cminaltitudehigherthan800metresupto1000metres
thickness>20cminheightsabove1000metersupto1500meters
thickness > 50 cm in heights above 1000metersupto1500meters
Wind(Map)
Highest streak > 20 m/s to: Anoia, Alt Penedès,Bages, Baix Llobregat, Baix Penedès Barcelonès,Garraf, Gironès, Maresme, el Moianès, Vic, Plad'urgell, Segarra, Segrià, Selva, Tarragona, Urgell,VallèsOccidentalandVallèsOriental
Highest streak > 30 m/s to: Anoia, AltPenedès,Bages,BaixLlobregat,BaixPenedèsBarcelonès, Garraf, Gironès, Maresme, elMoianès, Vic, Pla d'urgell, Segarra, Segrià,Selva, Tarragona, Urgell, Vallès OccidentalandVallèsOriental
Higheststreak>25m/sin:AltCamp,AltUrgell,AltaRibagorça, Baix Camp, Baix Empordà, Berguedà,CerdanyaConcadeBarberà,Garrigues,laGarrotxa,Noguera,PallarsJussà,PallarsSobirà,Pladel'estany,Priorat,Riberad'ebre,elRipollès,El,TerraAltaandVald'aran
Higheststreak>35m/storestin:AltCamp,Alt Urgell, Alta Ribagorça, Baix Camp, BaixEmpordà, Berguedà, Cerdanya Conca deBarberà, Garrigues, la Garrotxa, Noguera,Pallars Jussà, Pallars Sobirà, Pla de l'estany,Priorat, Ribera d'ebre, el Ripollès, El, TerraAltaandVald'aran
Higheststreak>30m/sAltEmpordà,BaixEbreandMontsià
Highest streak > 40 m/s region of AltEmpordà,BaixEbreandMontsià
Stateofthesea
Waves>2.50metres(heavysea) Waves>4.00metres(marbrava)
ColdMinimumtemperatureextreme:temperaturelowerthanthepercentile2oftheminimumtemperaturedaily
Wave of cold: temperature below thepercentile 2 of the minimum temperaturedailyforthreeconsecutivedaysormore
HeatMaximum temperature extreme: a temperaturehigher than the percentile 98 of the temperaturemaximumdaily
Heatwave:temperatureabovethepercentile98 maximum temperature daily for threeconsecutivedaysormore
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:61of84
Thebulletin can contain 7 different possibilitiesweather event, that are: intensitat de pluja ( =meansrainfall),acumulaciódepluja(=accumulation),neu(=Snow),vent(=wind),estatdelamar(=stormsurges),fred(=coldwave)andcalor(=heatwave).Thesecondandthethirdcolumninthetablerepresentthetwothresholds,Llindarbaix=meansLowthresholdandLlindaralt=meansHighthreshold. When the meteorological service sends a bulletin, it is updated twice a day (called“evolucions”)untiltheendoftheepisode.
4.2.4 UNITEDKINGDOMFor UK the LEMSModule integrates three different web services, provided byMet Office andEnvironmentAgency:
Meteowarningbulletin(Figure4-7).
TheMetOfficeprovideswarnings forEnglandandWales,concerninghazardousweathereventswhich have the potential to cause damage, widespread disruption and/or danger to life(https://www.metoffice.gov.uk/public/weather/warnings).
Thisserviceincludeswarningson5daysaboutrain,snow,windfogandicerepresentedbyacolourdependingona combinationof both the likelihoodof theeventhappening and the impact theconditionsmayhave.
Thebasicmessageassociatedwitheachwarninglevelis:
High
• Youmayneedtotakeactionasweareexpecting...
• Therewillbe...
Medium• Weshouldbepreparedfor...• Thereislikelytobe...
Low• Beawareofthepotential/possibility...• Thereisthesmallchanceof...
VeryLow • Beawarethatthereisaverysmallriskof...
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:62of84
Figure4-7:MeteowarningbulletinissuedbyMetOffice
Floodwarnings(Figure4-8).
TheEnvironmentAgencyprovideslocalfloodriskwarningsandpublishesdetailedinformationonthepotentiallyaffectedareas(https://flood-warning-information.service.gov.uk/warnings).
Figure4-8:FloodriskwarningsissuedbyEnvironmentAgency.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:63of84
Rawmeasuresfromsensors
Another service, publishedby EnvironmentAgency, provides data from themonitoring stationscheckingregularlytheriverandsealevels,helpingtounderstandthecurrentandfuturefloodrisk(https://flood-warning-information.service.gov.uk/river-and-sea-levels)
4.3 DEFINITIONANDDEVELOPMENTOFINTEGRATIONSERVICES
The LEMSmodule has been designed to extract information coming from the local EmergencyManagementSystems,andtointegrateitintoI-REACT.
Every local service is sending its own set of information, therefore a first effort was aimed atextractingcommonsetofcoredata,basicallywhat,whenandwhere.
Forthebulletinsandalerts:
- informationonthehazardtypeandseverity- informationonthetemporalextent(dateofissue,dateofexpiry)- informationonthelocation(polygonrepresentingtheaffectedarea)
Forthesensorsdata:
- informationonthemeasure:whatkindofsensor,valuemeasured- temporalinformation:dateandtimeofthemeasurement- informationonthelocation(coordinatesofthemeasuringstation)
TheintegrationschemabelowsummarizestheinteractionsbetweenthemodulesoftheI-REACTsystem(Figure4-9).
Figure4-9:LEMSintegrationmoduleandtheI-REACTsystem.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:64of84
Forbulletinsandalerts,theLEMSmodulesendsthedatatothe I-REACTData Interfacemodule,followingastandardI-REACTtemplate.
TheLEMSmodule stores themetadataare stored in theData layer,andproducesoneormoreGeoJSONfilethatwillbeavailableviatheI-REACTfront-endormobileapp.Forthistaskadedicatedsetofdatabase tables isusedby theLEMStostoreboth retrieved information fromthesourceproviders,andbackgroundinformation(forexamplethegeographicaldatathatarenecessarytocreatetheGeoJSONfiles,Figure4-10)).
Figure4-10:LEMSdatabase
Table4-3:I-REACTDataInterfaceattributes
Requiredattributes Optionalattributes
identification_resourcetitle layerattributes
identification_resourceabstract identification_coupledresource
identification_resourcetype temporalreference_dateofcreation
identification_resourcelanguage qualityandvalidity_lineage
classification_topiccategory qualityandvalidity_spatialresolution_latitude
classification_spatialdataservicetype qualityandvalidity_spatialresolution_longitude
keyword_keywordvalue qualityandvalidity_spatialresolution_scale
keyword_originatingcontrolledvocabulary constraints_conditionsforaccessanduse
temporalreference_start constraints_limitationsonpublicaccess
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:65of84
Requiredattributes Optionalattributes
temporalreference_end metadataonmetadata_date
temporalreference_dateofpublication metadatafileuri
temporalreference_dateoflastrevision sourcenames
qualityandvalidity_spatialresolution_measureunit creationtime
conformity_specification lastmodificationtime
conformity_degree isdeleted
metadataonmetadata_language deletiontime
coordinatesystemreference_code geographicboundingboxes
coordinatesystemreference_codespace
ireacttask
acquisitiondate
Table4-4:GeoJSONattributes
Requiredattributes
service
hazardCode
hazardGLIDECode
hazardName
hazardLevel
hazardLevelDescription
areaName
dateStart
dateEnd
dateLastRevision
creationDate
organizationName
country
AnexampleforthehydrogeologicalbulletinissuedbyARPAissketchedinthefigurebelow.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:66of84
Figure4-11:ARPAHydrogeologicalbulletininI-REACT.
Alertbulletinsareoftenissuedwithdifferentleadtimes.Forexample,theUKFloodRiskbulletincontains5-dayfloodriskforEnglandandWales.TheLEMSmoduleprocessesthisinformationdaily,producingonefilecontainingmetadatainformationandfivedifferentGeoJSONfiles,oneforeachdayofforecast,thatcanbeservedasmaps,etc.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:67of84
Insomecases,forexamplewhenafloodalertisissuedforalocalizedarea,anditmightbeusefultonotifyfirstrespondersandcitizensactiveinthatarea,theLEMSmodulecansendthedatatotheI-REACTbackendmodule,andthedatawillbeusedintheI-REACTsystemtocreateareport.
An example is the UK service providing flood warnings for England (https://flood-warning-information.service.gov.uk/warnings).Thedataisstructuredasexplainedinthetablebelow,andtheintegrationtaskwilltakecareofmappingtheinputdatatothisstructure.
Table4.5:DatastructureofUKserviceprovidingfloodwarnings.
Requiredattributes
type
start
end
areaOfInterest
location
ruleId
level
hazard
receivers
description
targetAreaOfInterest
targetLocation
sourceOfInformation
communicationStatus
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:68of84
5 SENTINELDATAPROCESSINGINTEGRATION
TheCopernicusprogramisEurope'sEarthObservation(EO)programofferingaseriesofthematicservices.Copernicus'ownSentinelsatellitesprovideuniqueoperationalsensingcapabilitiesacrossthewholemeasurement spectrum. Thanks to their advanced sensing concepts andoutstandingspatio-temporalsamplingcharacteristics,theSentinelsatelliteswillcollectmoredatathananyEOprogrambefore.ThefirstoftheSentinelsatelliteseries,Sentinel-1(S-1)waslaunchedon3rdApril2014.TheS-1keyapplicationsoverlandincludesthemonitoringoftopographicmovements(landsubsidence,glacierflow,etc.)andhydrologicprocesses(floodmapping,soilmoisture,waterbodies,etc.). This section describes the utilization of space-borne SAR data for historical frequencycalculation and development of near real-time flood mapping service using Envisat AdvancedSyntheticApertureRadar(ASAR)andSentinel-1satelliteremotesensingdata.
5.1 SATELLITEDATAPROCUREMENT
Thedataacquiredbythesatellitesaredownlinkedtothecollaborativegroundsegment,fromwhereviaESAnetwork it isbeingdistributedtoothersources likeScientificHubandESAServerof the“ZentralanstaltfürMeteorologieundGeodynamik”(ZAMG).FromZAMG,dataispushedtonationalmirrorandthentotheEarthObservationDataCentre(EODC)datastorage.EODChasthecompletearchive of the Envisat ASAR mission, covering the timespan from 2005 to 2012, and acquiresregularly the Sentinel-1 (S-1) data. The S-1 data at the EODCwarehouse are currently availableapproximately2.5hoursaftertheinitialsignalprocessingbyESA(level-1product)and6.25hoursaftertheacquisition.TheS-1level-1dataarearchivedonfastdisksstorageandbackedupusingarobotictapelibraryonaregularbasis.Figure5-1-1showsthedataacquisitionchainfromsatellitetoEODCstorage.
Figure5-1:EODCdataacquisitionstrategy.SatellitedataaretransferredfromthesatellitetoCollaborativeGroundSegment,thentoarollingarchivesystematZAMGandstoredtoEODCinternalstoragesystem.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:69of84
5.1.1 DATAANDSENSORSPECIFICATIONSThe first of the six Sentinel satellite series, Sentinel-1Awas launchedon3rdApril 2014. S-1 is aSyntheticApertureRadar(SAR)missionforoceanandlandmonitoringandrepresentsthecontinuitymissiontotheSARinstrumentsflownonboardofEuropeanRemoteSensing(ERS)satellitesandENVISAT.TheS-1missionisimplementedthroughaconstellationoftwosatellites(units1Aand1B).TheSentinel-1Bwaslaunchedon25thApril2016.Inthefollowingashortoverviewaboutthemainapplications,datapolicies,accessandsatellitespecificationsisgiven.(a)Applicationdomains:TheS-1keyapplicationoverlandincludesthemonitoringoftopographicmovements(surfacesubsidence,glacierflow,etc.)andhydrologicprocesses(floodmapping,soilmoisture,waterbodies, etc.). Furthermore, S-1 canplay an important role in sustainable forestmanagementwithclear-cutandpartial-cutdetection,foresttypeclassification,biomassestimation,disturbancedetectionandprecisionfarming.
(b)Data source:Copernicus is a European system formonitoring the Earth,which consists of acomplexsetofsystemscollectingdatafrommultiplesources:EOsatellitesandin-situsensors,suchasgroundstations,airborneandsea-bornesensors.S-1dataarealsodistributedbytheCopernicusprogramme.
(c) Data policy and access: The free, full and open data policy adopted for the CopernicusprogrammeforeseesaccessavailabletoallusersfortheSentineldataproducts.Registrationisopento all users via simple on-line self-registration accessible via the Sentinels Scientific Data Hub(https://scihub.copernicus.eu/). Following registration, the user can immediately downloadSentinelproductsgeneratedsystematicallyfromallacquireddata.(Note:MemberStatesrequiringdatafornationalinitiativesintheframeoftheSentinelsCollaborativeGroundSegmentneednotregisteronthisservice;theyareservedviathededicatedaccesspoint.)
OriginalS-1data,distributedbyCopernicusprogramme,isfree.Theprocesseddata/parameters,(SAR backscatter, temporal/monthly/seasonal composites) and other operational products andservices are distributed by Earth Observation Data Center (EODC), for more details visithttps://www.eodc.eu/.
(d) Sensor and data specifications: Table 5.1 gives the overview of Sentinel-1 system and itsspecifications,andtheTable5.2givesthedetailedexplanationofSentinel-1dataacquisition.
Table5.1:Sentinel1sensorspecifications.
Satellites/Sensor AdinterimOperator Lifetime Orbit Band Revisittime Datadelivery
Sentinel-1/A-B ESA 7years Sun-synchronous
C-band5.405GHz(wavelength
18cm)
12days(withthe
constellationof1Aand1B~6days)
Withitscontinuousandconflict-freeoperations,Sentinel-1willprovideahighlevelofservicereliabilitywithnearreal-timedeliveryofdatawithinanhourafterreceptionbythegroundstation,andwithdatadeliveryfromarchivewithin24hours.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:70of84
Table5-2:Sentinel1:specificationsofdifferentdataacquisitionmodes.
5.1.2 DATASTORAGESYSTEMAttheEODCplatform,theS-1dataarestoredonbothdisksforfastdataaccess,andtapesforlong-term storage and backup. Currently, two Petabyte (PB) of disk and four PB of tape space areavailableandmore than2million individualSentineldata filesarearchivedon this system.Thestoragesystemiscurrentlygoingtobeexpandedonregularbasisaccordingtothepredefinedplans.
EODC'shigh-speeddiskstoragesystemisbasedonIBMElasticStorageServer(ESS).IBMGeneralParallelFileSystem(GPFS)clusteredfilesystemtechnologyisimplementedtoensuretheefficiencyandperformanceinfilestorage.Currently,theEODChigh-speeddiskstoragesystemiscapabletohosttwoPBofdataforhigh-speedprocessing.Alternatively,EODCuserscanalsouseanotherhigh-speed storage system which is operated by Vienna Scientific Cluster 3 (VSC-3), called VSC-3distributedvolume.ThisfilesystemiscurrentlyrunontheparallelclusteredfilesystemBeeGFS,whichconsistsof360spinningdisksconnectedthrough160Gb/secbandwidth[RD14].
Tosavethefast-accessspaceondisks,arollingarchivingplanissetuptocontinuouslystorethedatasetstotapesthatarerarelyusedbyEODCpartners.Afterarchiving,thefolderstructureshownonfront-endterminalwouldbethesame,butthephysicallocationofthedataisontapestorageinsteadofhigh-speeddisks.Thisstrategyhelpstosavethediskstorageforhigh-speeddataaccessneedswithoutlosingtheaccessibilityofarchiveddata.Furthermore,thedata,whicharephysicallyarchivedontapescanbere-calledtohigh-speeddisksinashortperiodoftime.
EODChas implementeda comprehensivebackupplan to continuouslybackup thedataon theirstoragesystem.Thisisaninfiniteincrementalrollingprocessedwhichprovidestheabilitytoensurethedataaccessanddataconsistencyinthecasesofdisasters(virusattacks,hardwareandsoftwarefailures).
Operationalmodes Polarization Spatialresolution(m) Swathwidth
StripMapMode
VV+VHorHH+HV
5x5 80km
InterferometricWideSwathMode 5x20 250km
Extra-wideSwathMode 25x100 400km
WaveMode VVorHH 5x20 20kmx20km
DataProcessing:
Level-0Compressedandunprocessedinstrumentsourcepackets
Level-1GroundRangeDetectedGeo-referencedProducts(forI-REACTprojectLevel-1datawillbeused)
Level-2Oceanproducts
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:71of84
5.1.3 DATACOVERAGEANDAVAILABILITYThefollowingSentinel-1CoverageMaps(seeFigure5-2and5-3)showthecurrentstatusofS-1(A&B)interferometricwideswath(IWGRD)dataasstoredontheEODCdatawarehouse.TheyareprovidedinV(ertical)/V,H(orizontal)/H,V/H,andH/Vpolarisation,aswellasinacombinedmap(i.e.compositeofV/V+H/Hpolarisations)andaredynamicallyre-createdfromourdatabaseeveryweek.ForfloodmappingSentinel-1IWGRDLevel-1productisbeingused.
Figure5-2:Sentinel-1AdatacoverageforIWGRDacquisitionmode.
Figure5-3:Sentinel-1AdatacoverageforIWGRDacquisitionmode.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:72of84
5.1.4 HISTORICALREMOTESENSINGDATAEODC data storage archives the complete ENVISAT ASAR mission, covering the lifespan of themissionfrom2005to2012.Figure5-4showsthecoveragemapforWideSwathacquisitionmode.
Figure5-4:ENVISATASARcoveragemapforWideSwathmode
5.1.5 EODCPROCESSINGENVIRONMENTANDPLATFORMThe flood mapping algorithm is developed and implemented by TUWien running on a virtualmachineatScienceIntegrationanddevelopmentPlatform(SIDP)hostedbyEODC.Theprocessingchainincludespre-processingoftheSARdata,dataqualitycontrols,floodmapping,andnecessarypost-processingsteps.TheSIDPisalsoconnectedtotheViennaScientificClusterwithmorethan2000 computing nodes, which might be used for heavy processing tasks that need parallelprocessing.Thefinalproductswillbefloodandfloodfrequencymaps.
TheEODCisacollaborationbetweenpublic-privatescientificorganizations,researchcentresandcloud providers. Themainmission of EODC (or EODC platform, EODC cloud environment) is toprovideaccessandrequiredcomponentsforbigdataprocessingandanalysis.KeycomponentsofEODCinfrastructure,whichwentoperationalinspring2015,are:
• Acloud-basedvirtualresearchanddevelopmentenvironment.
• TheaccesstotheViennaScientificCluster3(VSC-3).
• Acomputationclusterformid-termdataprocessing.
• APetabyte-scaledatastorageandarchive.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:73of84
• Anearreal-timeOperationsandRollingArchive(NORA),whichisahigh-availabilitystorageandprocessingclustertocomplementthedatasetsinnearrealtime.
Figure5-5providesanoverviewofEODCkeycomponents,whichareinoperationatthemoment(September2017).ThedetailsofeachcomponentandtheViennaScientificClusterarepresentedinthenextsections.
Figure5-5:EODCinfrastructurecomponents.
5.1.6 EODCDATASTORAGEANDSENTINEL-1DATAFLOWRawdatafromtheEODCdatawarehouse(Level-1product)arepre-processedtocalculateSyntheticAperture Radar (SAR) signal backscatter with different spatial resampling (resolution) in Equi7projection[moredetailsaboutEqui7projectioncanbefoundinRD15].Figure5-6showsthedetailedstepsinvolvedintheENVISATASAR/Sentinel-1datapre-processingchain,whichisimplementedintheSARGeophysicalParametersRetrievalToolbox(SGRT).
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:74of84
Figure5-6:ENVISATASAR/Sentinel-1datapre-processingworkflow.
5.2 SENTINEL-1WATERMAPPINGALGORITHMDESIGN
Afterthepre-processingstep,parametersretrievalmodules/workflowofSGRTisinvokedinordertocalculatedifferentparametersi.e.,slope,meanbackscatter.Parametersarethennormalizedtoreference incidenceangles fordataconsistency.After theparameterscalculation, thefinalstep,productgenerationworkflowofSGRTislaunched.Inthisstepautomaticthresholddetectionandnoiseremovalmodulesareusedforthegenerationoffloodandfloodfrequencymaps.Figure5-7showsthesystematicworkflowdesignforfloodandfloodfrequencymapgeneration.
Figure5-7:ENVISATASAR/Sentinel-1schemeforproductiongeneration.
ThegraphicaloverviewoftheSARdatapre-processingandproductgenerationschemeisshowninFigure5-8.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:75of84
Figure5-8:AnexampleoverviewoftheSyntheticApertureRadar(SAR)datapre-processingandproductgeneration(NeusiedlLake:Austrian–Hungarianborder).
5.2.1 USERINTERFACEANDFLOODMONITORINGACTIVATIONMETHODService on request, this service is established to provide a rapid post disaster flood status andinundationextentmaps,whichcanbeusedforpost-disastermanagementandreliefactivities.Inthis service the user/customerwould contact TUVienna through I-REACT coordination (e.g. viaAzure Service Bus) to trigger the product generation. The requested product can be delivereddirectlytotheIDI(I-REACTDataInterface).Inthesectionbelowadetaileddescriptionofnear-realtimefloodmappingserviceisexplained.
5.2.2 SENTINEL-1 BASED NEAR-REAL TIME FLOOD MAPPING SERVICE LOGIC
IMPLEMENTATION
5.2.2.1DEVELOPMENTTOOLS,SUPPORTINGPRODUCTSANDQUALITYCONTROL
A. TUWienSARToolbox
The SAR Geophysical Retrieval Toolbox (SGRT) is a software package developed by the ViennaUniversityofTechnology(TUWien)forextractinggeophysicalparametersfromSyntheticApertureRadar (SAR) data. The version 2.0 of the SGRT,written in Python programming language, is anadaptationtoSentinel-1 (S1)of theearlierSGRT1.0developedforENVISATAdvancedSyntheticApertureRadar (ASAR)data, incorporatingoptimizations intended forhandling theconsiderablyhigherspatialresolutionandresultingexplosionindatavolumesforeseenofSentinel-1relativetoENVISATASAR[RD16].SGRTconsistsoffourtypesofprocessingchainswheredifferentnumberofworkflowsaredefinedundereachtype:
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:76of84
• Pre-processing: calibration, radiometric correction, georeferencing andterraincorrection,resamplingandtiling,qualitycontrol
• Analytics:timeseriesanalysistoextractmodelparameters
• Production:generateslevel-2andhigher-levelproductsfrompre-processeddatausingmodelparameters.
• Near-Real-Time (NRT): this component is designed to integrate differentworkflowsandSGRTfunctionalitiesinafullyautomaticprocessingchainforproductgeneration.
Furthermore,SGRTisequippedwithseveralimageandsignalprocessingcomponents.SomeoftheSGRTmodulesareusedinexternalutilitiesandplug-ins.e.g.thePythonbasedTimeSeriesAnalyseris an in-house tool, developed and integratedwith the open sourceQGIS software to visualizeSentinel-1 timeseries.SGRT isundercontinuousdevelopmentandwitheverynewrelease,newfunctionalitiesandworkflowsareintroduced.
B. SupportingProductsandModules
In order to develop an operational service a reliable quality control mechanism is essential tominimizetheartefactsinthefinalproduct.Inmountainousareas,topographicnoiseisverycommonwhichisduetosteepslopesandshadoweffectcausedbytheSARsidelookingacquisitiongeometry.Inordertoremovetopographicerrors,theHAND(heightabovethenearestdrainage)index[RD17]wasused.Figure5-9showsanexampleoffloodmapbeforeandafterapplyingtheHandIndexmask.
Figure5-9:ApplicationofHandIndexmasktoremovethetopographicnoise.
BordernoiseinS-1A/BandENVISATASARisquiteconsistentwhichisamajorsourceoferrorintime series analysis. In order to handle this error source an independent border noise removalmodule[RD18]wasdevelopedandintegratedintotheSGRT.Figure5-10showsanexampleforS1bordernoiseremovalmask(bordernoiseremovalmaskshowninmagentacolour).
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:77of84
Figure5-10:AnexampleS1bordernoiseremovalmask.
C. Dataacquisitionandmanagement
TheS-1dataattheEODCwarehousearecurrentlyavailableapproximately2.5hoursaftertheinitialsignalprocessingbyESA(level-1product)and6.25hoursaftertheacquisition.TheS-1level-1dataarearchivedonfastdiscsstorageandbackedupusingarobotictapelibraryonaregularbasis.InordertomanagetherawandprocessedS-1datafilesadedicatedmeta-database(EOMDB:EarthObservationMeta-DataBase)hasbeenestablished,whichallowstrackingofdataavailabilityandprocessingstatus.
5.2.2.2CONCEPTS,REQUIREMENTSANDINFRASTRUCTURETheSentinel-1basedfloodmappingandmonitoringserviceisanexternalmodule,whichhasbeendevelopedandmaintainedbyTUWienfortheI-REACTproject.ItcanbetriggereduponrequestviaAzureServiceBus–acloudmessagingservicebetweenapplicationsandservices.
A. Conceptandbigpicture
The I-REACT project has a multi-tier architecture and is named as I-REACTOR. Based on thepresentation,applicationprocessanddatamanagementtasks I-REACTORsfunctionsare logically
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:78of84
separatedintodifferentlooselycoupledmodulesthatcanbeeasilymodifiedwithoutaffectingtheothermodules.However,incaseofS-1floodmappingservicethepointofcontacttoI-REACTORisIDI(I-REACTDataInterface),wherefinalproductsarepushedontothefront-endforfurtherdataharmonization, processing and visualization. Figure 5-ä11 shows the different components of I-REACTORandtheirinternalassociationsanddependencies.
Figure5-11:AnoverviewofI-REACTORframework.
B. REQUIREMENTS
Apartfromthesatellitedataandservicemoduletherearesomekeyrequirementsthatarecrucialforoperationalservices:
• RegularupdateofEOMDB,toinclude/updatetherecentlyacquireddata.• Internetconnection:whichisrequiredtoconnecttotheAzureServiceBusandreceive
the message to trigger the flood monitoring processor. Internet connection is alsorequiredtoconnect-and-pushtheprocesseddatafiles(floodmaps)andmetadatatotheIDI(I-REACTDataInterface)databaseremotely.
C. InfrastructureandservicelogicimplementationschemeWithintheframeworkofI-REACTproject,theSentinel-1dataprocessingchainforfloodmappingservice is implemented within a virtual machine hosted by the Earth Observation Data Centre(EODC).TheprocessingchainanddataflowincludesfollowingstepsasillustratedinFigure5-12.
• Flood mapping processor: The flood mapping algorithm is developed by TUWien andimplemented within a virtual machine at the Science Integration and developmentPlatform(SIDP)hostedbyEODC.Theprocessingchainincludespre-processingoftheSAR
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:79of84
data,dataqualitycontrols,floodmapping,andnecessarypost-processingsteps.TheSIDPisconnecttotheViennaScientificClusterwithmorethan2000computingnodeswhichmightbeusedforheavyprocessingtasksthatneedparallelprocessing.Thefinalproductsarefloodandfloodfrequencymaps.
• Serviceonrequestandtriggeringmechanism:thisserviceisestablishedtoproviderapidpost disaster flooding status and inundation extentmaps,which can be used for post-disastermanagementandreliefactivities.Inthisservicetheuser/customerwillcontactTUWien through I-REACT coordination (e.g. via Azure Service Bus) to trigger the floodmapping/monitoringservice.TherequestedproductwillbedeliveredtotheIDI.Thefloodmapping/monitoringserviceisfullyautomaticandcanbetriggeredoveradefinedlocationwiththemessagereceivedviaAzureServiceBus.
• Watchdog: after getting the triggeringmessage–with a boundingboxof an areaofinterestandmonitoringperiod(startandenddate)–viaAzureServiceBus,theprogramwill automatically connect to the EOMDB and get the available raw files. Then theprocessing chain will be activated for available files. If there is/are no file(s) then awatchdogwillbeactivatedanditwillupdateEOMDBaftereverytwohoursandlookfornew files for the area of interest. Every time when it will find new file(s) a newindependentprocessingchaininstancewillbelaunched.Thisprocesswillcontinueuntiltheimageacquisitiondateislessthanorequaltotheenddatedefinedinthemessage.
Figure5-12:Anoverviewofinfrastructureandservicelogicfornear-realtimefloodmapping/monitoring.
D. PROTOTYPE
Thestabilityoftheprocessingchainandtheimplementedservicelogicwastestedbytriggeringtheservice for three different requests simultaneously. The processing chain ran successfully and
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:80of84
processed21Sentinel-1scenes.Thefinalproductof2.5GBwasuploadedtotheIDIdatabase.Duetothelimitednumberofcoresonthetestmachine,theprocessingchainranformorethan24hours.In caseofmultiple requests theprocessing timecanbeminimizedby increasing the computingpower.
Sentinel-1basedfloodandfloodfrequencymapswereproducedforallfivetestsite.Figure6showsanexampleofSentinel-1basedfloodfrequencyproductforatestsite in Italy(Emilia–Romagna,Figure5-13)andUnitedKingdom(Figure5-14).
Figure5-13:Anexampleofmulti-temporalfloodfrequencyproduct(Italy).
Figure5-14:Anexampleofmulti-temporalfloodfrequencyproduct(UnitedKingdom).
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:81of84
InadditiontothefinalproductoffloodorfloodfrequencymapsthatwillbedeliveredtotheIDI,differentSentinel-1basedintermediateproductswillalsobeproduced.Theintermediateproductsinclude both pre-processed data and various biophysical and statistical parameters. In order toquerythedata/productapre-definedsetofminimumrequiredmetadataisalsopushedtotheIDIasshowninFigure5-15.
Figure5-15:ExampleofSentinel-1productuploadedtoI-REACTdatabase.
Afterdataharmonizationfloodmapsbecomeavailableforvisualizationonthefront-end(I-REACTmainwebpage)oftheI-REACTOR.Thedisplayofthefloodfrequencyproductonthefront-endoftheI-REACTemergencyservicemanagementwebsiteisshowninFigure5-16.
Figure5-16:Floodmonitoringframework(top)andprototypeoffinalproductdisplayedonthefront-endoftheI-
REACTwebpage(bottom).
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015 Page:82of84
Insummary,thissectionconcludesthedemonstrationofaframeworkandfeasibilityofsettingupanoperational floodmappingandmonitoringservicebyexploitingabig-data infrastructureandhighperformancecomputingfacility.Thisserviceisoperational(intestphase)undertheframeworkof I-REACTproject, andnewdevelopment in termsofalgorithm improvementandperformanceenhancementwillbeintegratedintotheSGRT.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreementNo:700256 CallID:H2020-DRS-1-2015H2020-DRS-1-2015 Page:83of84
6 CONCLUSIONS
Theworkdescribed in the frameofDeliverable3.1– “ReportonEMS, EuropeanEarlyWarningSystemsandSentineldataintegration”representstheimplementationofexistingdataandservicesof Copernicus Emergency Management Services and Early Warning Services, for data andinformation on flood and wildfire monitoring as part of the I-REACT downstream services.Furthermore,withinthisdeliverable,theintegrationofprocessedSentinel-1dataintermsofflooddelineation and frequency mapping is also a crucial task. Moreover, local EMS services areintegratedwiththefocusonobtaininginformationconcerningalertsandmeasurementsfromthesemonitoringsystems.
ThetechnicaldevelopmentandinnovationworkwithinWP3,includingfurthersystemprototypinganddataintegrationofexternalservicesanddatastreamsalreadyavailable,providesoperationaldatafordownstreamservicesasexpected.Although,issuescamealong,suchasthedirectuseofthe EWS, the expectedCopernicus EWS couldbe implemented and integrated into the I-REACTsystembycollaborationwiththeJointResearchCentreoftheEuropeanCommission(JRC)andthecooperation of other I-REACT partners, which have access to the required input data and theexpertisetoproducethewildfiremonitoringinformation.
The results achieved have laid the groundwork for the creation of an early warning system toachieveincreasedpreparednesstoemergencies.Furthermore,otherpossibleCopernicusservices,whichwillprovideinformationforthedownstreamoffurtherdisasterservicesinthefuture,couldbeimplementedquitewell.
Thedatastreamsandserviceshavebeenintegratedfollowingtheas-a-serviceapproach,sothatnewdataisingestedautomaticallyintoI-REACTwithouthavingtheusertoactivateadatastream.Hence, the overall objective of this deliverable, to integrate existing data sources and systemsrelatedtonaturalhazardsintoI-REACTandtofilltheoverallsystemwithusefulinformationsources,isregardedasachieved.
ImprovingResiliencetoEmergenciesthroughAdvancedCyberTechnologies
Project:I-REACT “ReportonEMS,EuropeanEarlyWarningSystemsandSentineldataintegration”
DeliverableID:D3.1
GrantAgreement:700256 CallID:H2020-DRS-1-2015H2020-DRS-1-2015 Page:84of84
ENDOFTHEDOCUMENT