Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
CEOS Future Data Access & Analysis Architectures Study
Interim ReportVersion 1.0 – October 2016
1
CEOSFutureDataAccess&AnalysisArchitecturesStudy
Interim Report Version 1.0, 13th October 2016
IssuesforPlenaryDiscussionandDecision
1. ApprovalfortheAd-hocTeamonFutureDataAccessandAnalysisArchitecturestocontinueforafurtheryeartocompletethemandate.AndconfirmationoftheCo-Chairs.
2. Agreementfortheproposedpilotprojecttobeprogressedinparallelwiththeongoingreportwork,withoversightbytheFDAteamandcontributionsfromLSI-VC,SEO,andSDCG.
3. InvitationforfurtherproposalsforpracticaldemonstrationsintheareaofFDAfor‘lessonslearnt’evaluationbyCEOSPrincipalsatCEOS-31.
4. ActionforCEOSandSITChairstoconferwiththeFDATeamtoensurenecessaryCEOSPrincipalengagementonthestrategicissuesarisingfromthe2017Report,insupportofidentifyingcommongroundasthebasisforalong-termCEOSstrategy.
ThisreportwasachievedthankstoallagenciesandtheirrepresentativeswhoparticipatedintheAd-hocTeamprocess,inparticularthewritingteam:RobertWoodcock(CSIRO),TomCecere(USGS),AndrewMitchell(NASA),BrianKillough(NASA/SEO),GeorgeDyke(CEOSChairTeam),JonathonRoss(GA),MirkoAlbani(ESA),StephenWard(CEOSChairTeam),andSteveLabahn(USGS).
2
1.Introduction
OverviewWitheachpassingyear,newgenerationsofEarthobservation(EO)satellitesarecreatingincreasinglysignificantvolumesofdatawithsuchcomprehensiveglobalcoveragethatformanyimportantapplications,a‘lackofdata’nolongerneedstobethelimitingfactor.
Extensiveresearchanddevelopmentactivityhasdeliverednewapplicationsthatoffersignificantpotentialtodelivergreatimpacttoimportantenvironmental,economicandsocialchallenges,includingattheregionalandglobalscalesnecessarytotackle‘thebigissues’.SuchapplicationshighlightthevalueofEOtoMinistersandotherswhoultimatelyadjudicateoninvestmentinprogrammesandmissions
ThechallengeisinprovidingtherightsettingssothepotentialcantranslatetorealitybothforindividualCEOSmembersthroughtoglobalinitiatives.
ForEO,thegapbetweendata,applicationanduserneedstobebridged.Currently,manyapplicationsfailtosuccessfullyscaleupfromsmall-scaleresearchtoglobalorregionaloperationsbecauseofalackofsuitabledatainfrastructure.Eventoday,mucharchivedEOsatellitedatasitunder-utilizedontapes.Despitemultipleexamplesofbigdataanalyticsacrossapplicationdomains,significantdevelopmentremainsconsignedtoprototypes,pilotprojects,exemplarsandtest-beds.
Addressingthischallengeisdifficultforadvancedeconomies.Itissimplynottechnicallyfeasibleorfinanciallyaffordabletoconsidertraditionalprocessing(e.g.localdesktopworkstation)anddatadistributionmethods(e.g.scenebasedfiledownload)toaddressthis‘scaling’challengeinmanyeconomies,asthesizeofthedataandcomplexitiesinpreparation,handling,storage,analysisandbasicprocessingremainsignificantobstacles.ThischallengeisalreadyholdingbackkeyGEO/CEOSinitiativessuchastheGlobalForestObservationsInitiative(GFOI),Disasters,WaterResourcesandtheGEOGlobalAgriculturalMonitoringinitiative(GEOGLAM).
Addressingthisproblembyindividualusersworkingontheirdesktopworkstationshasnotresultedinanoptimalsolutionandmissestheopportunitiesofferedthroughcollaborativeenvironmentsthatbridgedataproviders,intermediaryvalue-adders(suchasresearchersandindustry)anduserstoworktogetheracrossdomains,andacrossgeographicboundaries,toco-createsolutions.
Fortunately,justassatelliteEarthobservationtechnologyhasadvancedsignificantly,sotohasinformationandcommunicationtechnology.Thedatamanagementandanalysischallengesarisingfromtheexplosioninfreeandopendatavolumescanbeovercomewiththehigh-performanceICTinfrastructure,technologiesandarchitecturesnowavailable.Thesesolutionshavegreatpotentialtostreamlinedatadistributionandmanagementforproviderswhilesimultaneouslyloweringthetechnicalbarriersforuserstoexploitthedatatoitsfullpotential.
3
PurposeInresponsetothesechanges,theCEOSFutureDataAccessandAnalysisArchitecturesAd-hocteam(FDA-AHT)hasbeentaskedbytheCEOSChairteamtoassessthepotentialofnewtechnologiesandapproaches,identifykeyissuesandopportunities,andproposeaplanofactionforconsiderationbyCEOS.
Thisreporthas:
1. ReviewedrelevantinitiativesandplansbeingundertakenbyCEOSandrelatedagencies;
2. ReviewedlessonslearnedfromtheearlyCEOS-ledprototypescurrentlyunderwaywiththegovernmentsofKenyaandColombia;
3. IdentifiedkeyissuesandopportunitiesresultingfromthetrendtowardsBigData,AnalysisReadyData,EOapplicationplatforms,etc;
4. MaderecommendationsforthewayforwardforCEOSanditsagencies,includinginrelationtostandardisation,interoperability,andhowcurrentCEOSprioritiesmightbeadvancedthroughasetofproposedactivities.
ThisstudyisanticipatedtobeofvaluebothtoCEOSAgenciesasdataprovidersandtoexistingandprospectiveusersandbeneficiariesofEOsatellitedata.ThefullpotentialofEOsatellitedatawillnotberealisedwiththeobstaclesthatusersfaceincurrentdatahandlingandanalysisapproaches.GlobalinitiativessuchasGFOIandGEOGLAMexemplifythedifficultiesthatcountrieswithoutdevelopednationalspatialdatainfrastructuresfaceintermsoflackofcapacitytohandleEOsatellitedata.ThiscapacitygapisamajorhindrancetotheuptakeofEOdatainthetypesofglobalinitiativesandagendasCEOShasstatedasimportantinrecentyears.Moreover,evenmanydevelopedcountriesarestrugglingtodeterminehowbesttocapitaliseonlargeandrapidlygrowingEOdatacollectionsandwouldappreciateguidanceonbothbestpracticetheycanadoptthemselves,andapproachesthatcanbringCEOSAgenciestogethertosupportapproachesthatmaximisevaluefromtheircollectiveconstellationofover130Earthobservationsatellites.
StructureoftheReportInthecreationofthisreportsubmissionsweremadebyanumberofCEOSmembersregardingcurrenttrendsandtheirspecificdevelopmentresponsesinEOsystemsarchitecturesandapplications.Aseachagencyhasdifferentterminology,operationalmethods,language,policycontextandbusinessdriversthesubmissionsappeardifferentinthedetail.CarefulanalysisthoughshowscommontrendsandresponsesthatareparticularlyrelevanttothemissionofCEOS.Inaddition,thereporthasbeenlimitedtoonlythoseaspectsoftheEOsystemsarchitecturethatarehighpriorityorimpactdirectlyontheCEOSmission.
Thereportisstructuredasfollows:
4
Chapter2consolidatescontributionsandidentifiestrendsandprioritiesinEOsystemsarchitecturedevelopmentacrossCEOSagencies.Itservesasabaselineofcurrentarchitectureandnearfuturedevelopmentresponses.Chapter3discussesthechallengesfacedinEOsystemarchitecturedesignanddevelopmentforthemediumtolongtermfuture.ItservestoidentifythekeychallengesthatmustbeaddressedinfuturedataarchitecturesChapter4describeskeyarchitecturalresponsesthatseektoresolvethechallengesidentifiedinChapter3.ItisnotacompletearchitecturaldescriptionandfocusesontheessentialelementsnecessaryfortheCEOSmissionleavingdetailstofutureprojectsorAgencydevelopments.Chapter5summarisestheoutcomesofthereportandpresentsrecommendationsonFutureDataArchitecturesandactivitiesforCEOS.
5
2.CurrenttrendsanddevelopmentsinEOsystemsarchitectureandapplicationsEarthObservation(EO)programmesofspaceagenciesarefacinganumberoftrendswhich,takentogether,aredrivingtheneedforchangeinthewaysinwhichdataareprocessed,accessed,distributed,andanalysed.ThemagnitudeandspeedofthesechangesisdeterminingtheimportanceandurgencywithwhichchangeisrequiredinfutureEOsystemarchitectures.Thischapterwillattempttosummarisesomeofthosekeytrendsanddevelopanassessmentofthestateofthesesystemsarchitecturesandtheirabilitytomeettheseuserneedsanduserapplications.
MaximisingtheValueofEarthObservationsMaximisingthevalueofEOisafundamentaldriverforallCEOSagenciesandakeypartoftheCEOSStrategicGuidance.ThereisanexpectationthatpubliclyfundedEOagenciesshouldmaximisethevaluereturnedtothecountrythroughtheapplicationofnationaldataholdings.AsafundamentaldrivermostagencysystemsarchitectureshavebeendesignedtodelivercalibratedobservationsandproducevalueaddedproductsforusebyotherGovernmentagenciesonpredominantlynationalandglobalsocietal,environmental,andscientificproblems.InrecentyearstherehasbeenasteadytrendacrossallagenciestowardsgreaterintegrationofdiverseEOdataholdingsforland,inlandwater,coastal,climateandoceanpurposeswithotherdatatypesheldbymorediverseGovernmentagencies-“Comprehensivecollectionandintegrationof...informationindependentlycontrolledbygovernmentalagenciesshouldbepromotedandsuchinformationshouldbedisclosedappropriatelytoincreasetheconvenienceforuserstoaccessandhandlesuchinformation.”(JAXA,OceanPolicy,2013).Theincreasedintegrationproducesamorediverserangeofapplicationsandleadstocomplexityinthevaluechainasobservationsarecombinedwithanalyticstomeetmultipleuserrequirements(Figure2-1).
Figure2-1:Oneexampleofthecomplexinter-dependenciesthatexistinmeetinguserrequirementsfromdiverseEOdatasources.CourtesyofJAXAOceanData
Infrastructure.
6
IncreasinglyEOdataarevaluednotonlyforitsscientificandtechnologicalvaluebutasapotentialfieldforeconomicgrowththroughnewcommercialventuresandindustrydevelopment.AgenciesarebeingaskedtopromoteandstrengthenanEOindustrywhilstcontinuingtomaintaintheirstrongscientificandtechnologicalfoundation.
Additionally,arecentUSGSreport(Milleretal,2012)attemptedtoevaluatethebenefitsofLandsatdatatoitsusers.Thereportconcludedthatmorethan80%oftheuserssawenvironmentalbenefitsandmorethan90%sawimprovementsindecision-making.Theestimatedannualeconomicbenefitofthisfree/opendataisgreaterthanUS$2billionperyear.Thoughthisisoneexample,therearelikelymanymoresimilarexamplesamongCEOSmissionsanddatasets.Overall,thereisanincreasedrelevanceofEOmissionsforresourcemanagementanddecision-making.
OpenDataPoliciesBeginningwiththeInstitutoNacionaldePesquisasEspaciais(INPE’s)movetowardfreeandopendatapoliciesin2004,datapolicychangeshavebeencriticalandareinfluentialinleadingtoasignificanttrendacrossallCEOSagencies.AnotherimportantpolicychangewastheUSGSadoptionoffreeandopenLandsatdatain2008.ThisallowedinternationalLandsatcollaboratorstochangebusinessmodelsandtomovefrombeingimageresellerstobeingdatascientistsandproviders.OtheragenciesincludingESAandJAXAalsochampionedthesechanges,howevertheglobalreachofLandsatdatameantthattheUSdecisionhadthewidestimplications.NASAhasbeenoperatingunderanopendatapolicysince1990andotherorganizationsaremovingtowardssuchpoliciesinrecentyears.WehaveseentherecentmovementfromEurope(e.g.CopernicusSentineldata,ESAEarthExplorersandHeritagemissions’data)andJapan(e.g.midtocoarseresolutiondata)toprovidefreeandopendata.
WithinAustralia,changesingovernmentpolicyfurthersupportedthedirectionoffreeandopendata.AgenciessuchasGeoscienceAustraliawereabletosupportthedevelopmentofsimplebuteffectiveopenlicensestoapplytotheirEOdatadistribution.Resourcespreviouslycommittedtolicensemanagementandmanualdistributionofproductswereabletobere-focusedonscientificexploitationofLandsatdata.
Fewerhurdlestodataaccessimpliesbroaderuseofdata.Thisresultsinamuchhigherreturnoninvestmentbyorganizationsfromtheirspaceborneandgroundsystems’assets.However,thisalsomeanshigherworkloadsfordatasystems.Itcanalsobeverydifficulttotracktheresultingimpactonceitleavestheconfinesoftheagency.ThereisconsiderablebusinessmodelinnovationoccurringacrossmanyagenciesandusersofEOdataastheimpactandvalueofEOdatanowreadilyavailableisunderstood.
OpenSourceSoftwareInadditiontofreeandopendata,itisalsodesirabletohavefreeandopenaccesstoanalysissoftwareandtoolsthatfacilitatetheuseofdata.SomeexamplesofthisincludeQGIS(anopensourcegeographicinformationsystem),THREDDSandGeoserver(opensourceEOandothergeospatialdatadeliveryservices).Opensourcesoftwarepolicieshelpwiththisdataexploitation.Adraftopensourcepolicyhasjustbeenreleased(March2016)forpubliccommentbytheU.S.FederalChiefInformationOfficer(see
7
https://sourcecode.cio.gov/)emphasisingthatusingandcontributingbacktoopensourcesoftwarecanfuelinnovation,lowercosts,andbenefitthepublic.AspecificexamplewithinCEOSistheDataCubeinitiative.Thisprojectreliesonopensourcesoftwareforthecreationofdatacubes(ingestingsatellitedata)andtheinteractionwithdatacubes(applicationprogramminginterfaces).Itisbelievedthatopensourcesoftwarewillstimulateapplicationinnovationandtheincreaseduseofsatellitedatasincetheseadvancedtechnologiescanbeutilisedgloballyandevenbydevelopingcountriesthatarenottraditionalusersofsatellitedata.
EmergentEOAnalysisPlatformsintheCloudCloudcomputinghashadadramaticimpactontheavailabilityofhighlyscalableandaccessiblecomputationalanddatainfrastructure.WithrecentadvancesintheCloudtechnologytheuseofscientificcomputingintheCloudhasgrownsignificantlywithmanypreviouslyHPConlyapplicationsnowrunningeffectivelyintheCloudenvironmentamongstthereareseveralEOofferings.
Themostwell-knownofthesewouldtheGoogleEarthEngine,which“combinesamulti-petabytecatalogueofsatelliteimageryandgeospatialdatasetswithplanetary-scaleanalysiscapabilities...availableforscientists,researchers,anddeveloperstodetectchanges,maptrendsandquantifydifferencesontheEarth’ssurface”.
AmazonWebServicesCloudofferingsoperateviaadifferentbusinessmodel,providingaccesstotheCloudplatformbutnotdirectlyofferingapplications,preferringtoprovideabaseplatformuponwhichothersbuildtheirown.AmongsttheEOofferingsavailableareopensourceEOtechnologieslikeGeoTrelliswhichmakeheavyuseofwebservicesarchitecturestoprovidescalablerasteroperations,andcommercialofferingslikePlanetLabsprovidingfullyautomatedscalableimageprocessingpipelinesdownloadingdata,performingcorrectionsandanalyticsat5-10terabytesperday.
WhilstAWSandGEEarementionedsimilarinitiativesexistonMicrosoftAzure(Layerscape)andotherorganisations.EachoftheseplatformsacquireCEOSagencydataandplacesitintoamanagedenvironmentwithcloselycoupledandhighlyscalableanalyticalcapabilitiesintheCloud.Significantlythebusinessmodelsinusevaryfromfree,toopensource,throughtofullcommercialserviceofferings.Howwellthesemodelsworkbothfortheorganisationoperatingtheserviceandfortheircustomershasnotbeenexploredinthisreportbutthebusinessmodelinuseisclearlyanimportantconsideration.TherearealsooperationalissuesfortheseorganisationsinacquiringtheCEOSdata,asoneexampleNASAhasbeenworkingwithmultiplecloudproviders(Amazon,MicrosoftandGoogle)tobetterunderstandhowNASAdatasystemscanbettersupportbulkdatadownloadsbycloudproviders.Theobjectiveistoenableefficientdiscovery,accessandtransferoflargevolumesofdatafromNASAarchivestocommercialcloudsandtomakethetransitiontocommercialcloudinfrastructureeasier.Someoftheseareprovidingefficientmetadata,theuseofstandardfileformats,useofstandardstructureddirectoriestoholdfilesanduseofwell-definedmapprojections.
8
IncreasedCommercialandNon-GovernmentalInteractionsNASA,NOAA,andtheUSGSconductalargepartoftheirEOactivitiesthroughcontractswithcommercialentities.ESAandJAXAconducttheiractivitiesthroughcommercialentitiesaswell,eventhoughtherearedifferencesinthenatureofcontractsamongthedifferentcountries.Thereisadistinctionbetweencommercialentitiesworkingundercontractswithgovernmentagenciesfordevelopmentandoperationofobservingsystemsanddatasystems,andothercommercialentitiesthatapplytheresultinginformationtosomeself-sustainingprofit-generatingactivities.TheFederationofEarthScienceInformationPartners(ESIP)isanexampleofbothtypesofcommercialentitiescollaboratingwithgovernmentanduniversityorganizations.Fromthepointofviewofdataarchitecture,commercialinteractionshaveaninfluenceonstandardsforinteroperability,amongotherthings.Withtheincreaseinopendatapoliciesandopensourcesoftware(seeexamplesabove),therewillbeanincreasingneedtoworkcloserwithcommercialentitiestoexpandtheuseofsatellitedataanditsbenefits.Thoughtherearemanyexamplesinthecommercialworld,someofthegreatestimpactsonsatellitedataapplicationhavebeenmadebyGoogleandAmazon.
Inadditiontothetraditionalcommercialentities,onemustalsoconsiderthenon-traditionalornon-governmentalgroupsastheyalsoplayamajorroleinconnectingEOdatatousers.SomerecentexamplesinCEOShavebeenconnectionswiththeUN-FAO(supportingforestmanagement),WorldBank(highinterestinwatermanagement),SilvaCarbon(fundedbyUSAIDtosupportforestmanagement),andtheClintonFoundation(workingincentralAfrica).Thesegroups,andmanyothers,utilisesatellitedataandwillcontinuetoincreasetheirdemandforsuchdatatosupportregionalandlocalapplications.
Pre-processedAnalysisReadyDataCountriesandinternationalorganizationshaveexpressedadesireforsupportfromCEOSagenciestofacilitateaccesstoandprocessingofsatellitedataintoCEOSAnalysisReadyDataforLand(CARD4L)products.CARD4Laresatellitedatathathavebeenprocessedtoaminimumsetofrequirementsandorganizedintoaformthatallowsimmediateanalysiswithoutadditionalusereffort.ExistingCEOSagencyeffortsincludeNASA'sMODISmodelthatsetthestandardforARD,Dr.DavidRoy'sWebEnabledLandsatData(WELD)modelthathasstimulateddemandformorehighlyprocessedLandsatdata,GeoscienceAustralia’sefforts,aswellasUSGSLandChangeMonitoringAssessmentandPredictionARDefforts.
SystematicandregularprovisionofCARD4Lwillgreatlyreducethetimeandtechnicalburdenonglobalsatellitedatausers,whohaveuptothispointneededtoinvestsignificanteffortsinpreparingEOdataforfurtheranalysis.Theprovisionofthisdataispossiblethroughmanyoptionsincludingsystematicprocessinganddistribution,processingonhostedplatforms,andprocessingviatoolkitsprovidedtousers.TheCEOSLandSurfaceImagingVirtualConstellation(LSI-VC)teamisdevelopingCARD4LdefinitionandspecificationdocumentsthatwilldefinethedetailsofCARD4L.ThesedocumentsintendtoimprovethecurrentandfutureprovisionofEOdataandtomaximisethevalueofthedatatousersandaddresstheneedsofthemajorityofglobalusers.Inaddition,CEOSisdevelopingaDataCube(spatiallyalignedtimeseriesstackof
9
pixels)architecturethatdependsonCARD4LtoallowimmediatecreationofDataCubesandsubsequentanalyses.Theresultofthiseffortwillbeimprovedinteroperabilityamonglandbaseddatasets,facilitatingtimeseriesanalysesandenhancedglobaluseandscientificvalueofsatellitedata.ThereisasimilardesiretodevelopARDspecificationsforocean,inlandwater,andcoastalenvironmentsaswellfornon-opticallybasedinstrumentslikeSAR.
AdvancedStorageandDistributionArchitectureforGrowingDataVolumesGrowingdatavolumeswillcontinuetoplacenewrequirementsonadvancedstorageanddistributionarchitectures.WiththeincreaseinCEOSmissionsandsensorsandincreasedspatialandtemporalresolution,theamountofdataavailabletotheworldwillbeordersofmagnitudegreaterthaninthepast.InarecentCEOSAd-hocSpaceDataCoordinationGroup(SDCG)GlobalDataFlow(GDF)StudyfortheGlobalForestObservationsInitiative(GFOI),itwasfoundthatexpectationsmustbemanagedtoachievesustainablesolutionsthatcanbeadaptedascountrycapacityincreases.Significantrisksareassociatedwithmaintaininginfrastructureandexpertisethoughthereisalwaysadesiretoutilisethelatestmoderntechnologiesforstorageanddistributionofkeysatellitedata.
ThroughtheeffortsoftheWorkingGrouponInformationSystemsandServices(WGISS),CEOSisdevelopingnewapproachestoeasedatadiscoverabilityanddevelopingstandardsformetadataanddistributionformats(i.e.spatialortemporaltilesvs.standardscenefiles).TheCEOSWorkingGrouponCalibrationandValidation(WGCV)isworkingonnewapproachesforproductvalidationtoimprovedataquality.Finally,theCEOSDataCubeinitiativeisinvestigatingnewadvancedstorageformats(pixel-baseddatacubesversusscenes)thatwillallowforsubsettingofdata(spatialandtemporalrelevance),significantdatacompressionandfacilitatedistributiontonon-expertusers.
TimeSeriesAnalysesandChangeDetectionTheextendedlifetimeandsuccessofmanymissions(e.g.LandsatandMODIS)haveallowedanewabilitytoexploitinformationfromlongtimeseries.FollowingtheopenreleaseofLandsatdata,therehasbeenasignificantnumberoftheseanalysesfocusedonlandusechange(includinginlandwaterandcoastlines).Theseanalyseshaveutilizedavarietyofchangedetectiontools(e.g.,ContinuousChangeDetectionandClassification(CCDC),BreaksForAdditiveSeasonandTrend(BFAST),Hansenetal.'sglobalforestgainsandlosses)tofindtrendsonthedataoridentifyperiodsofsignificantchange.OneexampleistheAustralianWaterDetectionfromSpace(WOFS)algorithmthatcalculatestimeseriespixel-levelwaterobservations.Theseresultsprovidecriticalinformationforwatermanagementthatwillallowuserstoassesswatercycledynamics,historicalwaterextentandtheriskoffloodsanddroughts.
TheavailabilityoftimeseriesdatawillcontinueintothefutureasprogramssuchasSentineldevelopplansforsustainedlong-termmeasurements.Toefficientlyandeffectivelyusetheselargetimeseriesdatasets,therewillbeaneedtousenewtechnologiesanddataarchitecturessuchasAnalysisReadyData,DataCubes,Data
10
Provenance,andadvanceddatabases.Withsuchadvancements,wewillbeabletoassesstheimpactofclimateandlandchangeonpeopleandnaturalresourcesovertime.
AdvancedUserRequirementsAdvanceduserrequirementsarethoserequirementsthatextendbeyondthetypicalusecasesandplacesignificantdemandsonfuturedataarchitectures.Theseinclude,butarenotlimitedto:realtimeapplicationsincludingrapidmonitoringoflandandwaterchanges;diverseapplicationsandoutputneedsformonitoring,assessmentsandprojections;integrationofmultipledatasets(climate,insitu,economic,demographic)andsensortypes(e.g.opticalandradar);fusionofdatasets(e.g.combiningLandsatandSentinel-2);accesstolowerlevelproductsfor“powerusers”;theuseofhighperformancecomputingforcomplexanalyses;andClimateDataRecords(CDRs)andEssentialClimateVariables(ECVs).Asmorecomplexdatabecomesavailable,therewillbeanincreasingneedfornewtechnologiesanddataarchitecturestomeetthoseneeds.
AsanexampleofhighperformancecomputingneedsinAustralia,HighPerformanceComputing(HPC)becameavailablein2011forthemanagementandanalysisofAustralia’sLandsatdatacollectionsunderthe“UnlocktheLandsatArchive”project.ThisworkappliedtheautomatedproductionsystemstotheentireAustralianLandsatcollection,whichwasmovedtotheAustralianNationalUniversityNationalComputationalInfrastructure(NCI)tocompletethiswork.GeoscienceAustraliabecameapartnerintheNCIin2012.TheHPCenvironmentallowedthedatatobeheldondisc,ratherthantape,anddirectlyattachedtolargecomputingresources.
AnexampleofadvanceduserrequirementsinJapanisfoundintheWDTMi-Coreproject.Thegoalofthisprojectistoprovideintegrateddataofbothinternationalocean-observingsatellitesandin-situobservations.Theprojectsintendtodevelopacommoninfrastructurewhichintegrates,manages,andprovidesdata,models,andanalyticalresultsfromoceanrelatedsatellitesandIn-Situobservation.
IncreaseintheNumberandDiversityofUsersInthepast,satellitedatausersweretraditionalscientistsandresearcherswiththecapacitytoobtainandanalyzethedatafordecision-making.Duetotrendsinopendataandcloud-basedhosting(e.g.GoogleandAmazon),therehasbeenasignificantincreaseinthenumberanddiversityofglobalEOdatausers.Nolongercanwesaythattheusersareonlytechnical,directlymanipulatingEOimages.Wemustnowconsiderhundredsandthousandsofnon-expertusersthatrelyonEOderivedproductslikelandcover,vegetationcondition,etc.Examplesofthesenon-expertusersincludelocaldecision-makersutilizingGoogleEarth,“crowdsourcing”projectstocompileEOdata,andtheuseofcommonsmartphonestoaccessEOdata.Therapiddevelopmentoftheearthobservationindustryandthelimitedavailabilityofscientificexpertiseitledtohasfosteredaparadigmofstart-to-finishsciencedelivery,startingwithearthobservationdataselection,todatapreparation,toconductingearthobservationanalytics,andconcludingbyrelatingresultingearthobservationpatternsandtrendstoimplicationstospecificapplicationdomains.However,thegrowingawarenessofthepowerofadvancedearthobservationanalyticshassubstantially
11
increaseddemandfromtheend-usercommunityforaccessibleandusablevalue-addedproductsbuiltfromdata-cubetechnologieswhichremaininthedomainsolelyoftheearthobservationexpertcommunity.Thisincreaseindemandfortheearthobservationscientificcommunitytodeliveronneedsandrequirementsdomesticallyandinternationallycannotbemetunderthetraditionalparadigmofsciencedelivery.Thus,theparadigmofstart-to-finishsciencedeliveryisrapidlydiminishinginfavourofthedevelopmentofanewparadigmwithend-userstakingownershipofthefinalstagesandearthobservationscientistsworkingtodeveloprichanddiversifiedanalysisreadydatathatenablesscientistsanddecision-makersacrossmultipleapplicationdomains.
Asthenumberofusersandtheirdiversityincreases,therewillbeincreasedquestionsoverthecontrolofEOinformationanditsapplicationtodecision-making.AsstatedbyKarenLitfin(MITPress,1998),therelationshipbetweensatellitetechnologyandstatesovereigntycontinuestobecomemorecomplex.Today,usersincludemultinationalcorporations,scientists,policymakers,grassrootsenvironmentalgroups,andindigenouspeoples.”Withthewidespreaduseofcloudstorageandcomputingservicesthetraditionalgeopoliticalbarriersnolongerexist.Usersarenowfreetointeractanddownloaddatafromcloud-basedservers,thoughtheymust“trust”theseprovidersandadheretotheirservicelevelagreements.Formanyinternationalusers,thisisaverylarge“leapoffaith”andmanyarequitereluctanttousecloud-basedservicesandprefertostoreandanalyzethedatalocallytoensure“ownership”andcontroloftheinformation.Withincreasingdatavolumesandcomplexityofdata,thisapproachisnotsustainableforthefuture,andthisparadigmmustshifttoenabletheseuserstotakefulladvantageofEOdata.
Successfullyabstractingthecomplexityofsensordatawillfreeuserstofocusonthedevelopmentofalgorithmswhichcanrunacrosssensors.Thisisakeyareaforfurtherdevelopmentandisnowonlypossibleduetotheadventofdatacubetechnologieswhichcanapplytechniquessuchasmachinelearningtomodelthevariabilityintargetspectralresponsebetweensensors.Withtherangeofnewsensormissionsonthehorizon,removingtheneedfordeepunderstandingofeachsensorwillbeintegraltotheeffectiveutilisationofthesenewdata.
LimitedInternetinDevelopingCountriesThoughdevelopedcountriesdependontheinternetformostoftheirdataaccessandanalyses,thisisnotalwaysthecasefordevelopingcountries.InarecentCEOSGlobalDataFlow(GDF)StudyforGFOIitwasfoundthat50%ofthestudiedcountrieshadinternetspeedsbelow5Mbps,whichwouldrequire~19daystodownload1TBofdata.Evenwithspeedsimproving,thecostofdownloadsisoftenaprohibitivefactor(http://www.tandfonline.com/doi/full/10.1080/01431160903486693).Withoutconsistentandcosteffectiveinternet,theremustbeotheroptionsforuserstoobtaindataoratleastinteractwithdata.AspartoftheCEOSDataCubeproject,theuseofregionaldatahubs(e.g.SERVIR)thatareclosetousersandhaveimprovedinternetperformancearebeinginvestigated.Otherapproachesareconsideringhostingofdataonlargercloud-basedhubs,suchasGoogleorAmazonWebServices,totakeadvantageofwebmappingservices(WMS)thatallowuserstoworkwiththedataremotely,usetheadvancedcloudcomputingcapabilitiesforanalysisandthenonlydownloadsmallresultingproductsoverlimitedinternetbandwidth.Thissameapproachisalsobeing
12
implementedinEuropefortheCopernicusServicesto“bringuserstothedata”forinteractingwithSentineldataoverlimitedbandwidthinternetwhileavoidingdownloadoflargedatasets.
13
3.ThechallengeandopportunityofchanginguserexpectationsandincreasingEOdatavolume,varietyandvelocityonEOsystemsarchitecture
Theimpactofvolume,velocityandvariety
Withindividualmissionsproducing10’sofPetabytesperyearby2020,and100’sofCEOSmissions,thereisnoquestiondatavolumesareundergoingastepchangeingrowthrate.Ingeneral,individualCEOSagenciesfactorinoperationaldatarequirementsaspartofmissiondesignandthisisuniquetothatorganization'sneeds(sharingofdesigninformationandlessonslearnedclearlyoccurs).Whilstvolumeandgrowthrateareclearlymajorchallengesforamission,itisbothvelocityandvarietythatposeamajorchallengeforEOsystemsarchitectureswhenviewedfromaCEOScommunityperspective:
● Largedatavolumesimplyaneedforveryaccurateandstructuredsearchservicessothatusersarenotoverwhelmedwiththevolumestheyneedtoaccess.Filediscoveryisnolongersufficient.
● Higheracquisitionratesandthenewnear-realtimeapplicationsthatbenefitfromthem(e.g.Disastermonitoringforbushfires,Floodmonitoring;Agriculturalmonitoring,etc.)requiretheentireacquisition,calibrationthroughproductgenerationpipelineinitsentiretytobecompletedpriortonextacquisition.Automationisessentialatallstagesandincludesthird-parties.
● Thebenefitofincreasingvarietycanonlyberealizedifthebarrierstoaccessingmultiplecollectiontypessimultaneouslyandconsistentlyarereducedsignificantlysothattheburdenofdiscoveryandintegrationdoesnotscalealongwithvolume,velocityandvariety.
● Datavolumesandvelocityaresuchthatinanincreasingnumberofcases,thevolumeistoolargetomovedatatoalocalanalysisplatformontheavailablenetworks.
● Manylocalanalysisplatforms(e.g.PCs,mobileplatforms,departmentalclusters)arenotlargeenoughtobenefitfromtheincreasingvolumesofdataavailable
● Withincreasedvolumes,automationandthirdpartyapplicationdevelopmentconveyingdataqualityinformationbecomeincreasinglyimportant
EarthObservation(EO)datasystemstodayfacechallengesfromtwodirections.Onthe“push”side,newinstrumentsandmodelsareproducingevergreatervolume,velocityandvarietyofdata.Onthe“pull”side,userexpectationsofthedata,andthesystemsthatservethem,areexpanding.
ThepushsidechallengesarethoseofthewiderBigDatamovement.Volumegrowthstemsfromimprovementsinsuchfactorsassensorresolution,space-to-groundbandwidth,retrievalalgorithmsandthecomputingpowerforprocessingthem.
WithinNASA,forinstance,theEarthObservationdatavolumeof15PBasofJanuary,2016,isexpectedtoincreasebyafactoroftenby2024.Inadditiontofindingaffordable
14
spaceforallofthedata,thevolumeincreasealsomanifestseitherinmoredatafiles,orlargerdatafiles,orboth,compoundingthedatamanagementproblem.
TheamountofdataandinformationbeinggeneratedbytheCopernicusspaceandservicecomponentsarealsochallengingtraditionaldisseminationchannels.ForthespacecomponentalonethecombinedarchivesofunitsAandBofSentinels-1,-2,-3willamounttoapproximately6PBperyearfrom2018onwards.Extrapolatingfromcurrentusagepatterns,eachproductmaybedownloaded10timesonaverage,amountingto60PBdownloadedperyear,or160TBperday.
ThesourcesofthevarietygrowthlieinthedevelopmentofmoreprocessingalgorithmsextractingyetmoreinformationfromtherawdataandtheincreasingavailabilityofmodeldataalongsideEarthObservationdata.
Thechangeinuserexpectationsonthepullsidehastwosources:technologyadvancementcausesuserstoexpectmoremoderncapabilitiesandinterfaces,suchasbroadlysearchingfordataacrossmanyfederationsofsystemsandthenretrievingdatadirectlyfromthediskonwhichtheyarearchivedandretrievingonlythespecificpieceofthedataneeded,orhighlyinteractive,responsive-designuserinterfaces.However,anothersourceofchangeinuserexpectationsistheexpansionoftheusercommunitiestoincludemoredifferentkindsofusersinsuchdiverseareasasinterdisciplinaryresearch,businessandgovernmentapplicationsandeducation.Withthebroadeninganddiversificationoftheusercommunities,informationaboutdataqualityanddataprovenancewillincreaseinimportanceasusersaccessdatafrommanymoredataproviders.Increasingfreeandopendatapolicieswillreducethehurdlestodataaccessandfacilitatebroaderuseofdata.Thiswillresultinamuchhigherreturnoninvestmentbyorganizationsfortheirspace-borneandgroundsystemsassets.
Section2hasalreadyshownthecurrenttrendtowardsmorenon-EOspecialistusers,theubiquityanddiversityofgeospatialapplicationsandchangingroleofparticipantsinapplicationdevelopment.Inthesubsectionsbelow,welookathowthesechallengesmanifestinandarehandledbythevariousareasofEOdatasystems.
DataDiscovery
ThebiggestchallengeinDataDiscoveryispresentedbytheincreaseinVarietyofdata.Thechiefsourcesofvarietyare:
● Sensor/instrument● Platform● Spatialfootprint● Spatialaggregation● Temporalaggregation● Levelofprocessing● Retrievalalgorithm
Giventhelargevarietyofsourcesandtypesofdata,inevitably,similardataproductsmeetingagivenapplication'sneedbecomeavailablefromvariousdataarchives.Thismakesitdifficultforenduserstosiftthroughtofindthemostappropriatedataproductstomeettheirneeds.Ametadataclearinghousecansimplifythesearchtools’taskof
15
queryingthenecessarysourcesfordataproductinformation.However,suchaclearinghouserequiresacommonmetadatamodel,suchasISO19115,whichcanprovidesomelevellingtoallowfordataproductcomparisons.Manymetadataclearinghousesstandardizetheirmetadatatoasingle,interoperablemetadataformat,suchasISO19115.However,systemdesignersarenowbecomingawarethattheyneedtocontinuesupportingmultiplemetadatastandardsintheirclearinghouse.Thisismostlyinresponsetoconcernsexpressedbythedataprovidercommunityovertheexpenseinvolvedinconvertingexistingmetadatasystemstosystemscapableofgeneratinganewmetadataformat.Asanexample,inordertocontinuesupportingmultiplemetadatastandards,NASAdesignedamethodtoeasilytranslatefromonesupportedstandardtoanotherandconstructedtheUnifiedMetadataModel(UMM)tosupporttheprocess.
Likewise,itisalsohelpfultoprovideanApplicationProgramInterface(API)toallowthedevelopmentofavarietyofsearchclients,rangingfromsimpledatasearch-and-fetchscriptstofull-featuredwebuserinterfaces.TherearetwocommonstandardsforsuchAPIsusedinthecommunity:theCatalogServicesfortheWeb(CSW)isahighlystructuredAPI,whiletheOpenSearchAPIislightweightandbasedprimarilyonkeywordsearch.Notethatsupplyinga“flagship”searchclient,orreferenceimplementation,canprovidenotonlyausefulsearchtoolforthecommunityinitsownright,butalsoaplatformforaddingnewsearchenginefeaturesandastartingpointforprospectiveapplicationdevelopers.Onearearequiringmoreattentioninthefutureisdealingwiththegrowingdiversityofusercommunities.Forexample,thehighlytechnical,detaileddataproductdescriptionsdemandedbythescienceresearchcommunityareoftennotappropriateorusefulto,say,acitizenscientistorabusinessapplicationowner.
DataAccess
Tousethedataandinformation,usershavetobeabletoaccessthem.Twotypesofaccessareconsidered:
A. eitherthedataandinformationaremovedtotheusers'premises(download/pull/push/broadcastreception)sothattheuserscanprocessthemandobtainfromthemtheresultstheywant;
B. orthedatastaysinthedatacentreandtheprocessingoccursnexttothedatawiththeobtainedresultseithersenttotheusersorstoredonlineforfurtherusebyotherusersdowntheline("bringtheusertothedata"scheme).
Bothtypesofaccessarepossiblebutthecostsinvolvedaredifferent:eitherthecostsconsistofhighernetworkbandwidthtomovethedatafromthestoragefacilitytotheusers'ownprocessinginfrastructureorthecostsconsistofmoreprocessingcapacitiesnexttothedataandoflessnetworkbandwidthformovingtheresultstotheusers(iftheusersstillneedtodownloadthem,ifnot,theresultingvalueaddedproductsmaywellbestoredinthesamedatacentreandpublishedfromtheretobefurtherusedbyothers).
Toreachoveralloptimalefficiencyofmodel(b),itshouldbereinforcedbytheguaranteethatthedatawillalwaysbeavailable,otherwiseuserswillbetemptedtodownloadthe
16
dataandarchivethemasaninsuranceofpermanentavailabilitywithoutnecessarilyimmediatelyknowingwhattodowiththedata/informationdownloaded.Thisaspectofavailabilityalsoincorporateslatencywherebydataassetsthatrequiresignificantlagperiodsfordeliverywillalsoencourageuserstoduplicatethedataholdings.
Bothmodelsmayscale,howevermodel(a)onlywouldonlyscaleifthebandwidthsarescalable,whilemodel(b)scalabilitymayplayondifferentfacilitieswithmoreloadbalancingpossibilities(storage,processing,bandwidthavailability).Thehigherthevolumeofdataandinformationtobeaccessed,themoreadvantageousmodel(b)islikelytobecome.
TheclearestchallengetoDataAccessarisesfromdatavolumes.Ifstoringthevolumesondiskisunaffordable,accesstothedatathatmustbearchivedontapeissignificantlydegraded,bothinlatencyandoverallthroughput.Thelatencyinturngenerallyrequiresanasynchronousaccessmethod,withnotificationstotheuserondatareadiness.Accordingly,compressiontechniquesaretypicallyappliedwheneverpossible,ideallylosslessinternalcompressionsuchasthatavailablewiththeHierarchicalDataFormat(HDF)andNetworkCommonDataForm(NetCDF)version4.DataAccessisalsocomplicatedbylargerdatavolumeswhichpromptmoreuserstorequestdatasubsetsforjustthedataofinterest.Subsettingiscomplicatedbythefactthatdataproductspecificsubsettingtoolsdonotscalewellwithvariety.Preferablearestandardformats(e.g.,HDFandnetCDF)thataretractabletogeneraltools.Also,somedataservices,suchasthoseofferedbytheOpenGeospatialConsortium’s(OGC)WebCoverageService(WCS)ortheOpenSourceNetworkforaDataAccessProtocol(OPeNDAP)canprovidesubsettingontheflyfordatainstandardformats,overtheInternet,alongwithotherservicessuchasreformattingthatsmoothovertheVarietyaspect.
DataUsage
Mostofthedataarestoredtypicallyinformsconvenienttoproducers.Theseformsarenotnecessarilyconvenientforuserstoaccess.Differentusersneeddifferentaccessmechanisms.Somewantaccessviathetraditionalgranuledownloadaccess;somewantgranulelevelaccessaftersomesubsetting(time,space,bands);somewantpixellevelaccess,i.e.,withoutregardtogranuleboundaries,andsomewantpre-builtstandardizedvalue-addedproducts.Itischallengingtoprovideaccesstospatialsubsetsoflongtimeseriesofdata.Howcanweprovideenoughdescriptiveinformationtotheuserstoenablethetypesofaccesstheyneed?
Giventhevarietyofpotentialapplications,'rightformats'meanthatdatawouldbeneededinseveraldifferentformatsandprocessinglevelsasneededbythedifferentusercommunityserved.Thisincludeslongtimeseriesofhomogeneousdatatomonitorchangesandlongtermtrends(e.g.Climate),lowerleveldata(i.e.Level0)toallowscientiststocontributetoalgorithmsdevelopment,higherlevelproducts(i.e.Level1,Level2andhigher)forresearchandapplicationactivitiesandoperationalservices.Dataneedtobecontinuouslyupgradedandvalorisedtoensurecontinuityofobservationsandcomparabilitywithnewmissiondata(e.g.Sentinels)andfitnessforpurposeforaneffectiveutilisationandexploitation.Asindicatedinthe“EOScienceStrategyforESA”,“Long-term,carefullycalibratedanddocumenteddatasetsoftheEarthsystemderived
17
fromEOsatelliteswillbecomealegacyofthehighestimportanceforscience,policymakersandsociety”.
Traditionally,formatshavebeenthemostproblematic,butcustomASCIIandbinaryformatshavelargelygivenwaytostandard,self-describingformatssuchasHDFandNetCDF.Thistrendinturnhasgivenrisetothedevelopmentofanumberofversatiledatatoolsthatworkonlargenumbersofdatasets,suchasPanoply,IDV,GrADS,aswellasfindingtheirwayintosupportbycommercialtoolssuchasArcGIS,IDLandMatlab.
However,whilethedataformathasbecomelessofanissue,itisstillimportanttofollowconventionsondatastructuresandattributes(suchastheClimateForecastconvention)toenablethesetoolstobeeffective.Furthermore,therearesomeareas,suchasdiversemapprojections,thatremainchallenging;evenwheretoolssupporttransformationstootherprojections,naiveusersapplyingthesere-projectionsmayunknowinglyintroducesignificantartefactsintotheresult.
DataVolumerepresentsasecondchallengetodatausagebytheenduser,whomaybefacedwithfindingenoughspacetostorethedataorenoughprocessingpowertoanalyzeitinareasonableamountoftime.Toaddressthis,somesystemsofferacertainamountofprocessingatthearchive,whichmayrangefromsophisticatedsubsettingschemestorunningtheuser’salgorithmatthearchive.
Fig.3-1Useranalysisofdatarangesfromfairlysimplevariablesubsettingtobothmorecomplexcontentsubsetting(e.g.,qualityfiltering)andtouser-provided
algorithms.
18
DataSystemFunctions
Inadditiontotheuser-centricaspectsenumeratedabove,somechallengesfallonthedatasystemsservingthem.Perhapsthemostimportantisthestewardshipofthedata.Thisiscomplicatedbyboththedatavolumeandvariety.Largevolumesmayforceadatacentertoresorttotapebackup.Althoughmediacostsarecheaprelativetodisk,thetapesmustbecontinuallyinspected(read)toensuretheyarestillreadable.Furthermore,thevarietyofdataleviesaresponsibilityofstewardingasmuchoftheavailablecontextaspossible,fromdescriptionsoftheinstrumenttoalgorithmdocumentstoproductspecifications.
Inaddition,sciencedatacentersrequireanumberofbackofficecapabilitiestoplanandmanageevolutionofthesystemwithchangingrequirements,usercommunitiesandtechnologies.Theseincludereliableandthoroughmetricsofdataarchivedanddistributed.Ideally,thesciencedatacenteralsomaintainsarepositoryofdatacitations,inordertogaugetheresearchvalueofdatasets,particularlywhenmanyarebeingmanaged.Thedatavarietyalsoaffectscommunitiesofdiversesciencedatacentersthatcoordinatetheirdevelopment,whichbenefitsfromashareddevelopmentenvironmentwithsuchelementsaswikis,tickettrackingsystems,andcoderepositories.
Identifiedaspirations,constraintsandopenproblems
Whilstdatavolumes,varietyandvelocityareclearlyamajortechnicalchallenge,probablythegreatestchallengetomaximisingvaluefromEOdata,andonthesystem'sarchitectureisthechangingexpectationsofusers.
ThereareanumberofCEOScommunityrelatedissuesthatalsoneedtobeconsideredindevelopingFutureDataArchitectures:
● Costre-allocation-therearepotentialopportunities,particularlywithCloudbasedbusinessmodels,tore-allocatethecostofdatadistributionandcomputationintoamoreuser-paysorientedmodelratherthanhavingtheentireburdensupportedbyanagency.
● EconomicsandPerformanceofCloudcomputingforEOstorageandanalysis-thisremainsanopenproblemcurrently.WhilstCloudpotentiallyhasexcellentscalabilityforanalysisagainstlocaldata,theeconomicsofstoringthedata,sovereigntyandsoftwareperformancearestillbeingassessedforEOapplications.
● Capacitybuilding-withexpectationsofeconomicgrowththroughnewEObusinessesandtheexpandinguserbasetherewillbeincreasingdemandfortrainingandcapacitybuilding.UsersupportandcommunicationtoabroadergroupwillalsobenecessaryplacingpressureonCEOSspaceagencies.
● Standardisationandinteroperability-Standardsarekeytointeroperability.ItispreferabletohaveasmallnumberofstandardstofacilitatesearchandaccesstodatainaninteroperablemanneracrosstheCEOScommunity(searchstandards,controlledkeywords,metadata).Wealsoneedamorediversesetofaccess
19
standardstosupportactionssuchassubsetting,fileformatconversions,spatialprojectionconversions,tileaccess,andpixellevelaccess.
● Administrativereporting-essentialtomaintaininginvestmentinEOactivitiesisbeingabletomeasuretheROIthroughitsuseandvaluegeneration.Inafuturewheretherearemanymorethirdpartiesdevelopingapplicationsandbusiness,alongwithmassiveautomationandconsumeruseofopendata,itwillbeincreasinglydifficulttocollectmetricsnecessaryforadministrativereportingusingthingslikeuserloginsoragencyportalaccess.Withmachinetomachineconnectivityitwillbenecessarytousealternatemethodstogathersuchinformationwhilstrespectingprivacyissuesandremainingtruetotheprincipleofopendata.TherisktoEOdatabecomingananonymouscontributortomajorapplicationsoutcomesishighasincreasinguseseesitbecometakenforgranted.
20
4.TheFutureofEODataArchitectures
Notwithstandingtheprogressmadeinrecentyears,thedifficultyoffindingandusingEOdataisstillabarriertorealizingtheirfullpotentialandtoproperlyharness“BigEarthScienceData”forsocietalbenefit.ThisismainlyduetotheinabilityofcurrentparadigmsinarchitecturestokeeppacewiththerapidlychangingEOdatamanagementlandscapeandimpedimentstothefreeflowofdatathroughanalysisworkflowsonsuitablecomputationalhardwarethroughouttheentirelifecycle.Thesimultaneousavailabilityofcomplete,high-quality,trusted,ready-to-useandintegratabledatasetsandoftheenablinginfrastructureallowingtheireffectiveutilisationandexploitationbyanincreasinglydiverseusercommunityiskeytomaximizetheimpactofEOdataassets.
Anenticingpossibility,evidentinmanyofthedevelopmenttrends,ischangingtheparadigmofuseranalysisfromthecurrentoneinwhichusersdownloadtheirowncopiesfrommultipledatacentresinordertoperformlocalanalysisovermultipledataproducts.ProvisioningdataintheCloudmayleadtomoreprocessinginplace(i.e.,intheCloudwiththedata),thusreducingnetworktransfersanduserdatapreparationandmanagementheadaches.Thisinturnmayenablemorecross-sensorandinterdisciplinaryanalysis.Thischangeintheuserparadigmfordataanalysisisleadingtodevelopmentofprocessesthatshort-cutwhattheuserneedstodotostartthedataanalysis.CEOSAnalysisReadyDataforLand(CARD4L)reducestheneedfordeepEOcalibrationexpertiseandbroadenstheaccessibleusercommunity.DataCubetechnologydemonstratedintheCEOSandAustralianGeoscienceDataCubeprojectsshowshowsupportingpixel-basedaccesstoCARD4Lwillallowuserstoaccessthespecificspatial,temporalandsensorrangesofthesatellitedataneededforscienceorindustryapplicationsimprovingtheeaseofuseoftimeseriesandinteroperabledatasets.
Anotherexampleistheshiftincostofstoragevscomputevsdatatransfer.Withincreasingcomputecapacityavailabletheabilitytoprocessdataon-the-flywillbecomearealityandwilldrivedowntherequirementforpersistentstorageofderivedproducts.DataCubes,orderivedproductscouldthenbeconsideredtransientcachesandspunupasrequiredatsmallcostratherthanpersistingthem.
Table4-1summarizesthestateofseveraldataarchitectureoptionsandtheirabilitytoaccomplishkeyuserrequirementsanduserapplications.Thismatrixismeanttocomparetraditionalapproachestodatastorageanddistribution(scenebasedmethods)andnewerinnovativeapproaches(pixel-basedmethods).Itisassumedthatpixel-basedmethodscantakeadvantageofsubsetting(e.g.onlyacquirethespatialandtemporaldataneeded)andcompression.Asummaryoftheassessmentisfoundbelowthetable.
21
Table4-1:AssessmentofUserRequirementsandUserApplicationsvs.DataArchitectureOptions
22
Bringingtheusertothedata:EarthObservationVirtualLaboratoriesEOVirtualLaboratories(VL),accessiblethroughwebbrowsers,virtualmachines,mobileapplicationsandotherweb-basedormachine-to-machineinterfaces,areonemeanstoaddresstheobjectivesabove.Suchplatformsarevirtualenvironmentsinwhichtheusers-individuallyorcollaboratively-haveaccesstotherequiredanalysisreadydatasourcesandprocessingtools,asopposedtodownloadingandhandlingthedata‘athome’.Akeyqualityofsuchplatformsisthattheyareshapedbyandscalableaccordingtotheneedsandambitionsofusers,theyco-locatedatacollectionsandcomputationalcapacityandtheyarecomposedofarangeofflexiblyinterconnectedservices,oftenfederated,allowingsubstantialtailoringandre-use.TheseVLplatformsaretypicallyimplementedinthe“Cloud”connectinghundredstoseveralthousandcomputernodesacrossanetworkofdatacentresorinregionalhubswithHighPerformanceDatacapabilities.SuchEOVLplatformsareintendedtobringtogetherthefollowingmainfunctionalities:
● dataforbothEOandnon-EOapplications● powerfulcomputingresources● large-scalestoringandarchivingcapabilities● collaborativetoolsforprocessing,datamining,dataanalysis● concurrentdesignandtestbenchfunctionswithreferencedata● high-bandwidthweb-basedaccess● applicationshopsandmarketplacefunctionalities● communicationtools(socialnetwork)anddocumentation● accountingtoolstomanageresourceutilizationandflexibilityinfreeorpay-for-
usebusinessmodels● securityandprivacyenforcement
ThetermVirtualLaboratory,isn’tuniversalnorexclusiveandtherearemanyothervalidnamesforsuchsystems.Thephraseisusedheretosimplyconveythesetofcharacteristicsthatarecommontosuchenvironments,regardlessofwhatnametheyarereferredto.WiththisdefinitiontheAGDCwhencombinedwithsuitablewebservicesinterface,theCEOSSEODataCube,andtheEuropeanThematicExploitationPlatforms(Fig4-1)areallvalidexampleswithmany,ifnotall,ofthesecharacteristics.
23
FIG.4-1 - EO Community Platform Concept. Image courtesy of ESA.
Conceptually,thisapproachispavingthewayforan“InformationasaService”scenario(Fig.4-2).EOandnon-EOdataareflexiblyandintelligentlylinkedandcombinedbymeansofmodernICTservices(“InfrastructureasaService”,“SoftwareasaService”,“DataasaService”,etc.)thusincreasinglyintegratingtheEOsectorintotheoveralldigitaleconomy.Theinherentchallengeinsuchanapproachistheorchestrationofheterogeneoussystems,datasets,processingtools,anddistributionplatformsleadingtothecreationofinnovativeandhigh-qualityinformationservicesforabroadrangeofusers.
Fig.4-2 - Model for the generation of “Information as a Service”. Image courtesy of ESA.
24
Insuchenvironments,evolutionisdrivenbyusercommunitiesandtheirneeds.Ideally,usercommunitieswillhaveaccesstoafullyscalableIT-infrastructureenablingthemtodevelopnewbusinessmodelsandtointroducenewapplicationsandservices.CrowdsourcingPlatforms,whichinsomecasesarealreadyprovidingsignificant“CitizenScience”output,mayprovideanumberofrelevantpointersinthiscontext.
Europeismovinginthisdirectionthroughtheimplementationofthe“EOInnovationEurope”conceptincoordinationbetweentheEuropeanCommission,ESAandotherEuropeanSpaceAgencies,andindustry.The“EOInnovationEurope”concepthastheobjectivetoenablelargescaleexploitationofthecomprehensiveEuropeanEOdataassetsforstimulatinginnovationandtomaximizetheirimpact.SimilararchitecturesexistintheUS,Japan,andoutsidespaceagenciesinthebroaderCEOScommunitywiththeCEOSSEODataCubeprojectdevelopingsuchplatformsforKenyaandColombiaandtheAustralianGeoscienceDataCubecombiningmultipleagencycollectionsobservingAustraliafromtheUS,JapanandEuropeanmissionsintoasingleplatformforbroaderuse.Takentogethertheseactivities,whilststillindevelopment,illustratemuchofthefutureofEOdataarchitectureswithimprovedaccessibilityandgreaterdistributionofcomputationempoweringdiverseend-userapplications.
ArchitecturalchangeArchitecturallytheseEOVLsareanetworkofinteroperableinterconnectedplatformsbuiltaroundcore-enablingelements,opentomulti-sourcefundinginitiativesorimplementationbyindependentorganisationsandrelyingoncommonstandardsforintegration.Thecommoditizationandseparationofcore-enablingelementsbeyondasingleagencyboundarysupportsscalabilityandempowersend-usersandindustrytovalue-addandallowsSpaceAgenciestoutiliselargescalecommercialinfrastructures(e.g.Cloudcomputing)whenitismorecosteffectivetodoso.
Figure4-3illustratestheEOInnovationEuropeapproachwherediverseinstitutionalandcommercialplatformsalreadyexistorarecurrentlybeingimplemented.Basedonfunctionalanalysesandidentificationofbestpractices,EO-InnovationEuropehasbeenstructuredaroundthreeelements:anenablingelement(actingasabackoffice),astimulatingelementandanoutreachelement(actingasafrontoffice).
25
Fig.4-3 – EO Innovation Europe. Image courtesy of ESA.
Withintheenablingelement(backoffice),a“mutualisation”(i.e.sharing)ofeffortsandfundingbetweenpublicinstitutionsshouldpreventanunnecessaryduplicationofinvestmentsforenablinginfrastructuresandwillstimulatetheexistenceofmanyexploitationplatformsorvalue-addingadd-onsfundedbydifferentpublicandprivateentitiesintheoutreachelement(frontoffice).ComparedtoexistingEOarchitecturesratherthansupplyanendtoendproductorbasedata,theSpaceAgencyprovidestheenablingelementandsuitabledatapreparationandinterfacestoallowdynamicusebythird-partiesforresearchorindustryexploitationasrequired.Theabilityforthird-partiestoquicklyandflexiblycreatenewvaluechainsandprovideinnovativeservicesoverthegenericenablingelementsisexpectedtosignificantlyenlargetheuserbase.
Thechangetoamoredistributedarchitecture,bothintermsofcomponentsandparticipation,isaccompaniedbyseveralarchitecturalprinciplesmanyofwhicharealreadyevidentinthetrendsdiscussedinSection2:
DiscoveryandAccess
1. MachineLevelDiscoveryandAccess:Alldataareavailableforsearchandaccesswithmachine-callableAPIs
2. Cross-agencyDiscovery:Cross-agencydatadiscoveryisseamlessatapixelbasedaccesslevel(pixelandattributelevelsub-setting)
3. DatasetSelectionGuidance:Guidanceisavailableondataselectionbasedonfitnessforpurpose.
4. MetadataNamingConventions:KeymetadatafollowstandardnamingconventionsforVariables,Platforms,Instruments,SpatialResolution,TemporalResolution
5. VirtualCollections:Virtualcollectionscanbeorganized/orientedaroundascienceproblemorTheme(eg.hurricanes,agriculture,algalblooms,fires)containingmultiplesensorcollectionsratherthansinglesensorbasedcollection(eg.Landsat,
26
Sentinel2).TheNASAESDSWGVirtualCollectionsWorkingGrouphasexploredideasaboutsuch“VirtualCollections”
6. AnalysisReadyData:Preparationanddistributionoftrusted,calibrated,welldocumenteddatafrommultiplesensorsinananalysisreadyformforland,inlandwater,coastalandoceanapplications.
7. Changestometadatarelatedtodiscoveryandqualitytoenablepixel-basedretrievalandfinegrainedqueryofverylargedatacollections(e.g.%ofcloudcoverformyregionandsensorsofinterest,ratherthan%ofcloudcoverinascenefile).
8. Webbasedseamlessvisualisationandbrowsingofentirecollections
Usage
1. IntelligentToolCatalogs:Intelligenttoolcatalogsautomaticallysuggestdataanalytics/visualizationtoolstoworkwiththedata.
2. LiveDataCitation:Publicationsarelinkedtodataandtoolsthatallowinteractionswiththedata.
3. MobileDataandProcessing:Dataandprocessingmovetransparentlyasnecessarytoachieveoptimalperformance.
4. QuantitativeQuality:Alldatahavequantitativemeasuresfordataquality.5. Reproducibility:Scientistscanreproduceotherscientists’researchresultswithhigh
precision.6. High-QualityDocumentation:Concise,ComprehensiveandConsistent
documentationexistsforalldatavariables.7. CapacityBuilding:Arichsetofcapacity-buildingandtranslationmechanismsexists
tofacilitateleveragingdataforusebypeoplewithlimitedliteracyinscienceandadvancedtechnology,and/orEnglish.
8. DataAnalysisatScale:Usersareabletoanalyzetheentiredatarecordforanydatavariableoveranyarbitrarilydefinedarea.
9. DatasetUpgrading:High-valuedatasetsareupgradedasnecessarytofullysupportintherichcapabilitiesavailableinthedatasystems.
Integration
1. Data:EOdatacanbeeasilycompared,merged,fusedand/orassimilatedwith(dependingontheapplication)datafromotheragencies,nationsandotherentities.Thisnotonlyrequiresclarityandqualityofgeospatialandtemporalcharacteristicsbutcomparablecalibrationongeophysicalobservations(e.g.inusingtwodifferentopticalsatelliteswithsimilarsensordetectionwavelengths).
2. ToolsandServices:Toolsandserviceswithinthecommunityareeasytouseinconcertandco-hostedwithlargeEOdatacollections
3. Sharing:TheEOcommunityareabletoshareallscientificresources(data,tools,results,workflows,contextualknowledge)
4. Standardisationofprogramminginterfacesandvocabulariesacrosssensortypestobettersupportdataintegration,discovery,andanalysis.
27
Infrastructure
1. HostedprocessinginfrastructurewithEOtoolboxes(visualisation,analytics)anddataavailableforIndustryandresearchuse
2. Virtualisationofhardwareandsoftwareservicestosupport“payforwhatyouuse”scalability
3. UseofOpenSourcesoftwarelicensingasamechanismtosupportinnovationbythirdpartiesbuildingoutfromagencyandresearchsuppliedtools
4. Consolidationofcommonarchitecturalcomponentsacrosscollections(samedeliverymechanism/experienceforallsensortypes)
5. Automationandaccelerationofthedatapreparationlifecycle(calibration,atmosphericandterraincorrection)tosupportnearrealtimeanalysis
28
5.Conclusions
CSIRO,asCEOSChairfor2016,establishedanAd-hocTeamonFutureDataAccess&AnalysisArchitectures(FDA)tosurveythechallengesandopportunitiesaroundEOdataarchitecturesgiventheoperatingenvironmentinwhichgovernmentsponsoredEOprogrammesareworking.
ProgresstodatehasestablishedthatCEOSshouldintensifystrategiceffortsinrelationtoFDAifEOistorealiseitsfullpotentialinsupportofsociety,includingmakingbestuseofallavailableCEOSagencydatasothat:
• densetimeseriesanalysisandapplicationsaremadefeasibleforallusers,particularlyinthe‘landdomain’butnotrestrictedtoitgiventheinteractionsbetweendomains.;
• CEOSeffortsinrelationtothegrandchallengesofclimate,foodsecurity,theSustainableDevelopmentGoals(SDGs),anddisastermitigationcanbesupportedthroughapplicationofEOdataandproductsbyusersacrosssectorsandnations.
Thefirstconclusionoftheteam’s2016effortsistorecognisethatthisactivityisverymuchneededandindeedoverduewithinCEOS.AllCEOSagenciesrecognisetheneedtodomoretoremoveobstaclestodataaccess,analysis,uptake,applicationandrealisationofbenefitstosociety.ThereissignificantactivityacrossCEOSagenciesinthisareawithgreatdiversityinapproachesandcapacities.Thismeansthereisalottobuildon;italsomeansthatmovingforwardinacoordinatedwaywillbemoredifficult.
TheteamwouldliketohighlightkeytrendsthatarecompellingactionbyspaceagenciesintheareaofFDA(presentedinmoredetailinsection2ofthereport):
- Themovetofully‘on-line’datasystems,plustheincreasedsizeandcomplexityofthedatabeingproduced(referredtointhereportasvolume,velocityandvarietyofdata)isoverwhelmingtraditionalapproachestodataarchitectures;
- TheBigDataplayersandtheiradvancedplatforms,populatedwithCEOSagencydata(amongstothers),arechangingexpectationsastohoweasyitcouldandshouldbetoaccess,analyseandapplyEOsatellitedata;
- ThesenewcapabilitiesareprovidingawelcomebroadeningoftheuserbaseforEOsatellitedata,tomoresectorsandmoreusers,manyorwhomarenon-expert,and/ornotfromlargetechnicalinstitutions.Theseeffortsaredemonstratingthatup-frontefforttoremovetheobstaclestoEOdatahandlingandusearepayingdividendstothemainstreamingofEOdataandtoitssocietalimpact.CEOSagenciesmusttakenoteastotheimplicationsforeaseofdatahandlingforCEOSagencymissiondata;
29
- CEOShasbeenplacingmoreemphasisonsupportinguptakeandapplicationofdata,includingforgrandthemesliketheSDGs,climate,andfoodsecurity.Theseinitiatives,suchasGFOI,havereporteddifficultiesinuseruptakeduetocomplexityofdataaccessandhandling,andthechangesinuserexpectationsfromexposuretoadvancedplatformsofcommercialbigdatacompanies.UsersareseeingsolutionstotraditionalobstaclestoEOdataaccessandapplicationandexpectspaceagenciestoadoptthem.
- Evolutionintheapplicationspaceiscreatingmoredemandforcapabilitytobringtogetherdifferentdatasetstoanswercomplexquestions,overlargeareas,overlongperiods,andincombinationwithother(e.g.socio-economic)non-spacedata;intheterrestrialdomaininparticularthisisfarfromeasytodo.
CEOShas,overtheyears,investedconsiderableeffortintocooperationininteroperabilityofdatadiscoveryandaccess.Effectivefuturecooperationshouldextendthistoencompasscommonworkonuser-datainteraction,facilitationofdataintegration/interoperability,andcompatibleserviceinterfacesforanalysis.CEOSeffortsshouldreflectthetrendsaroundincreasinguseof‘in-place’AnalysisReadyDatatoreplacedatadiscoveryanddownloadasaresponsetothe‘volume,velocity,variety’challenge.
Individualagencystrategiesarequitediverseandinclude:
- bringingtheusertothedata,incontrasttotryingtotransmitlargeamountsof‘rawdata’withtheassociatedcommunications,storageanddatamanagementproblemsbecomingahuge‘barriertoentry’forusers;
- APIs/VirtualLaboratories,enabledbystandards(suchasThematicExploitationPlatforms)thatmakeiteasierfortheworkofsometobeintegratedandlinkedwiththeworkofothers,toaccelerateprogress;
- pre-processingdatatoapointwhereitisameasurementcomparableinspaceandtimewithmeasurementsfrombothothersatelliteinstrumentsandothersectors,helpingisolateapplications(andusers)fromnon-relevant(totheirapplication)changesinthespacesegment(i.e.sensoragnostic);
- novelservicemodels(includingnewopportunitiestointegratecommercialdata‘onthefly’toaugmentproductsprimarilyusingpublicsectordata);
- movingtheburdenofdatapreparationprocessing(calibration,location,atmosphericcorrectionetc)fortheextractionofapplicationinformationfromtheuserstothespaceagencies(suchasAnalysisReadyData);and
- flexibleapproachestocomputing(includingHPCandCloud)thatshowcasetheabilitytoprocessmultipleobservationdatasets,atfullresolution,atcontinentandglobalscales.
FutureDataArchitecturescanbeassumedtoconsistofmultipleapproachestocoverallcircumstancesanduses.TheAd-hocTeamhasnotsoughttojudgeindividualagencyapproachesbutinsteadtoidentifycommongroundforeffectivecooperationthatwill:
30
- deliverbenefitsbacktoAgencyactivities;
- laythefoundationforAgenciestoworktogether,throughCEOS,toofferamoreintegrated‘wayin’tosatelliteEarthobservationdataanalyticsforusersandglobalintiatives.
AnumberofactivitiesinvolvingcoretechnologiesandexamplesarealreadyunderwayaspilotsinitiatedbyGFOI,theCEOSSEOandLSI-VC,inrelationtoAnalysisReadyDataandtheCEOSDataCube.Inaddition,thereisongoingfundamentalworkwithinWGISSaroundmetadatastandards,dataprovenanceandpreservation,andongoingtechnologyexploration.
ThetechnologyandinfrastructurebehindFDAischangingrapidlyandthechallengeforCEOSwillbetoestablishaprogrammeofworktotakeadvantageoftheseopportunities–withanemphasisonapproachesthatde-riskandsimplifyEOdataforusers,allowinguserstomakeuseofALLavailableandrelevantCEOSagencydata,andthatwillsupportCEOSambitionsinrelationtoitschosengrandchallenges.FutureworkshouldhelpusersbenefitfromallrelevantCEOSdataatallstages–complementingpasteffortswhichhaveemphasisedunityof‘discovery’,withoutfullyaddressingthesignificantchallengesintheanalysisandexploitationofdisparateorincompatibledataafterdiscovery.
RecommendationsOneyeardidnotpermitsufficientcapacityortimetobebroughttobearonwhathasbeenfoundtobeoneofthebiggeststrategicissuesforCEOSAgenciesandforCEOSastheirforumforinternationalcoordination.Theissuesarecomplex,andmoretimeisrequiredtoestablishconsensusamongCEOSagenciesastothemostproductivewayforwardforCEOSasacoordinatingbody-reconcilingthevariousagencyapproachesandprioritieswhilstfindingcommongroundforco-operation.
Asaconsequence,theteamrecommendsthatCEOSadoptatwo-streamapproachtocontinueitsengagementwiththistopic:
- afurtheryearofworktocontinueexplorationofkeyareas,andforfacilitatedstrategicdiscussions;
- paralleleffortstoprogressestablishedCEOSpilotprojectstoensurestrategicdiscussionsaresupportedbyreal-worldevidence.
Thisreportthereforefocusesonlyonshort-termrecommendations.
1. USGSandCSIROhavebothagreedtocontinuetoprovideleadershipfortheTeam,whichwillseeka1-yearextensiontoitsoperationasaCEOSAd-hocTeamattheNovember2016BrisbaneCEOSPlenary.TheAHTrecommendsthatathirdco-chairbeidentifiedfromESA/ECtocontributetotheformulationofarecommendedwayforwardforCEOSin2017,sincethefreeandopenglobaldataprogrammesofboththeUSandEuropewillnecessarilybeattheheartofanystrategyultimatelyadoptedbyCEOS.ItisalsocriticalthatotherAgenciesactivelyengagealso,asthewayforwardmustprovideopportunitiesforallAgenciestocontributeandbenefit.The
31
AHTrecommendsthatCEOSPlenarydirecttheTeamtoconcludeitsworkintimefordebateanddecisionattheUSGS-hostedPlenaryinOctober2017.
2. InparallelwiththeconclusionoftheAHTstudyandreportin2017,CEOSshouldprogress,accelerate,andintegratethepilotactivitiesalreadyunderwaywithinitssubsidiarygroups,includingandinparticular:
- LSI-VCworkinrelationtodefinitionsforCEOSAnalysisReadyData(ARD)forLand(CARD4L)andguidelinesforitsuse;and
- SEO/SDCGworkinrelationtodemonstrationsofaCEOSDataCubeanditsbenefitsforbothdataprovidersanddatausers,andasawayofengagingwithdonorinstitutionsonpracticalcapacitydevelopmentprojects.
TheAHTrecommendsthat2017effortsbedesignedaroundthefollowingobjectives:
- Asmall-scaledemonstrationoftheproductionandapplicationofCEOSARDanditsvaluetobothdataprovidersanddatausers,drawingonkeyfreeandopenglobalprogrammes;thisshouldhelpallCEOSagenciesdevelopanunderstandingastothenatureandscaleoftheactivityofdataproductionandintegration–andimportantlyshouldincorporateafeedbackloopfromengageduserorganisationsandfundingagenciesastolessonslearnedandthebenefitsforthem;andshouldemphasisewhatspaceagenciesgetbackbywayofdatauptakeasthemotivationforexpansionoftheconceptofCEOSARD;
- ContinueddevelopmentoftheCEOSDataCubeasawidely-supported,opensourceexampleofthebenefitsofHPCapproachestosupporteaseofuseofEOdata.Significantworkisalreadyunderway,documentedinaninformal3-yearWorkPlanpreparedbytheSEO,andincludesapilotdemonstrationwiththeGovernmentofColombiaforGFOI(andother)purposes.ThispilotcouldbeemployedasthepracticalapplicationfocusfortheARDproductiontrial–combiningmultipleopticalandradardatasetsusingtheCEOSARDdefinitions,inaframeworkthatwouldguaranteeuserfeedbackandpracticallessonslearned.TheSEOandSDCGhavebothindicatedawillingnesstosupportsuchanintegrationofexistingCEOSactivitiesrelevanttoFDA.TheworkwoulddemonstratethebenefitsofdensetimeseriesdataforGFOIpurposes,amongstothers.
ThisapproachwouldintegratedifferentaspectsofCEOSFDAworkinanapplicationfocusedactivitythatwouldguaranteeuserfeedbackanddevelopCEOSexperiencearoundtheproductionof,andbenefitsfrom,ARD.TheAHTcouldprovidethenecessaryintegrationeffortfortheseeffortsin2017toinformthedesignoftheCEOSFDAstrategywhilstcontributingactivitieswouldcontinueunderleadershipofLSI-VC,SEO,andSDCG,withexpertinputfromgroupssuchasWGISSandWGCV.Suchademonstrationwouldlikelyrequire12-24months.ItcouldeffectivelybuildontheexistingfoundationsunderwaywithinGFOI/SDCGandcouldadapttothefinalFDAReportrecommendationsinlate2017asneeded.
TheSEO,asthetechnicalleadforthisinitiative,shouldbeassuredsupportfromacoregroupofAgencieswillingtocontributetoevolutionofthetechnicaldesignanddevelopmentofnewcapability.
32
3. CEOSshouldcontinuesupportforongoingWGISSworkinrelationto:discoverysearchengineoptimization(searchrelevancy,keywordsearch,persistentidentifiers);accesscommonstandardsforinteroperabilityofproductformats(metadata/data)andapplicationprograminterface(API)foranalyticsanddataaccessservices;explorationofemergingbigdataservicesincludingcloudcomputing.WGISSshouldcontinuetoprovideguidancenotesandbestpracticesthatagenciescantakeonboardwhenplanningfutureinvestments.
4. Inthecourseofthe2016workoftheAHTfurthersuggestionsforpilotactivitieshaveemerged,includingnotablyaproposalfromESAforaCEOSThematicExploitationPlatformforDisasters.TheleadershipoftheAHTshouldestablishtheavailablecapacityandleadershipforthisandanyotherrelevantproposalsemergingfromthework,andbringtoCEOS(atthekeymeetingsofSITorPlenary)suitableproposalsfordebateandfurtherdevelopmentasappropriate.Suchproposalsmightbeevolutionsofexistingworkandwithinexistinggroupsormightmeritestablishmentofnewinitiativesandthemeanstomanagethem.AnynewinitiativesshouldbedevelopedinaccordancewiththeCEOSProcessPaper.USGS,asincomingCEOSChairhasalreadyproposedtoundertake2017activitiesinrelationtointeroperabilityofmoderateresolutionopticaldataproductsandthesecancontributetotheaboveandemergingproposalsrelatedtotheFDAwork.
5. FutureCEOSdecisionsonastrategyaroundFDAissueswillbebothstrategicandsensitivesincethebroadercontextinvolvesactivitiesandcompetitivenessofnational/regionalindustriesandcompaniesinrelationtoEOdatauptakeandapplications.Yettheconsequencesaresocentraltothefuturesuccessandhealthofgovernment-sponsoredEOprogrammes,andtotheeffectivenessofCEOSasacoordinationbodywithsignificantuser-facingglobalinitiatives,thatCEOSmustidentifyaworkablecooperationpathofcommoninteresttoitsagencies.TheAHTrecommendsthatextensiveCEOSPrincipalinputbeassuredinthedevelopmentofthefinalreportonAHTmattersandthatCEOSChairandSITChaircooperatein2017toensurethatthewayforwardisformulatedinlightofinputsfrombothworking-levelCEOScontributors(liketheAHT)andthosefromCEOSPrincipalsandLeadership.