Embed Size (px)
Citation preview
Page 1
ESA-ESRIN
Earth Science Infrastructure requirements for Long Term Data Preservation and
Access : the ESA approachMirko Albani, Vincenzo Beruti, Eugenia Forcada, Graeme Mason
(European Space Agency)
Raffaele Guarino (Cap-Gemini)
[email protected], [email protected], [email protected], [email protected]
APA Conference
Helsinki, November 22nd 2010
Page 2
Presentation outline structure
The Scientific Data e-infrastructure in Europe: the High Level
Expert Group Vision
The Earth Science Data Preservation and Access challenges
ESA Earth Observation LTDP Framework: one step ahead for a tangible solution
The ESA EO LTDP Preparatory Programme
• FIRST (Definition of LTDP User Requirements and Preserved Data Set Composition) study
• LAST (Long term data Archive Study on new Technologies)
Conclusions
Page 3
The High Level Expert Groupon Scientific Data
• The High Level Expert Group on Scientific Data was created by the European Commission’s Directorate-General for Information Society and Media to prepare a “vision 2030” for the evolution of e-infrastructure to scientific data.
• The group was composed of specialists with competence in the digital libraries domain from the following categories:
– Memory organisations (libraries, archives, museums);– Authors, publishers and content providers;– ICT industry (e.g. search engines, technology providers);– Scientific and research organisations, academia including ESA
• HLEG delivered the “vision 2030” report to the European Commission in October 2010.
• The vision focus in a scientific e-infrastructure that supports seamless access, use, re-use, and trust of data.
Page 4
The Vision into the future
• Data themselves become “the infrastructure” – a valuable asset, on which science, technology, the economy and
society can advance.
• The availability and open accessibility of reliable scientific data and associated knowledge would enable:
– Researchers to find, access and process the data they need • being confident in their ability to use and understand data
• and evaluate the degree to which that data can be trusted.
– Producers of data: to benefit from opening it to broad access• thanks to the return they can have (e.g. public organizations may see a
rise in available funding as funding bodies have confidence that their investments in research are paying back extra dividends to society, through increased use and re-use of publicly generated data).
– Industry to have the appropriate returns.
– Policy makers to make decisions based on solid evidence and to monitor the impacts of these decisions.
Page 5
Open questions
• How can we make sure that data are collected together with the information necessary to reuse them and make them understandable in future ?
• How can we get researchers – or individuals – to contribute to the global data set?
• How do we overcome the problems of diversity – heterogeneity of data, but also of backgrounds and data-sharing cultures in the scientific community?
• How do we deal with the diversity of data repositories and access rules – within or between disciplines, and within or across national borders?
• How can we trust the data (e.g. authenticity) and guarantee their integrity?
• How can we decide what to preserve?
• How can we fund the preservation of our information resources in the long term?
Page 6
Presentation outline structure
The Scientific Data e-infrastructure in Europe: the High Level Expert Group Vision
The Earth Science Data Preservation and Access challenges
ESA Earth Observation LTDP Framework: one step ahead for a tangible solution
ESA EO LTDP Preparatory Programme
• FIRST (Definition of LTDP User Requirements and Preserved Data Set Composition) study
• LAST (Long term data Archive Study on new Technologies)
Conclusions
Page 7
Earth Science requirements breakdown Earth Science requirements breakdown
Page 8
Earth Science: long time series of observations
Page 9
• Diversity in:– communities that generate and use the data (space agencies, universities and
research centres, private users, etc..).
– data formats and types,
– which data and associated information is preserved.
– ways of analysing, sharing and handling data.
– methods and conditions for data discovery and access.
– Approaches for data integrity, quality, authenticity, etc.
• Sustainability of Earth Science data preservation is not guaranteed in Europe.
• Earth Science is a very scattered domain representing micro-cosmos within the wider Science domain but fully representative of its challenges and open points.
Earth Science can be pathfinder for the wider science domain.
Earth Science Data Preservation and Access: the challenges
Page 10
Earth Science domain: the needs
• Aiming to the creation of an infrastructure that allows:
– Interoperability and sharing of existing infrastructures for the preservation and access of Earth Science data.
– Sustainability of the data infrastructure through the availability of funding mechanisms (public good only?) that enable all to contribute as well as to benefit, through an increased return on investment.
– Availability of common policies and approaches for the preservation and access to the data.
• The solution requires joining forces: An international framework for collaboration and cooperation.
Page 11
What do we mean with Infrastructure ?
• Does the infrastructure include at least:
– Organizations mandates– Cooperation guidelines and policies, agreements, common plans– Legal and financial aspects– Standards and procedures– Cost commitments, sustainability– Data to be preserved and accessed (awareness, quality, reliability, trust,
knowledge) and related metadata– Operational Data Repositories, data preservation, migration, – Researchers requirements– Technical physical resources (i.e. hardware, software, networking, web
visibility access, etc.)– Maintenance, operation, evolution
• There are major differences between HLEG Vision 2030 and e-IRG Roadmap 2010, both referring to an e- infrastructure?
Page 12
• Considering the Earth Science LTDP Framework like a living entity, the following issues need to be addressed:
– What kind of environment is necessary for it to come to existence?
– How should it be nurtured?
– How will it be self-sustained?
– How could we know if its survival is at risk?
Earth Science Data Preservation and Access Infrastructure : the issues
Page 13
Starting point
Acceptance of new LTDP culture and
Ownership
TransitionContinuous support and
keep motivation
The living ecosystem and sustained business model.
Change Management
State-of-the-art Basic rules Basic infrastructures Minimal investments EO Data provider vision
• Extended rules• Training• Stakeholders participating in building• First evolution of infrastructure (tools and technologies)
from rules to structured processes Investment Infrastructure enhancements Auditing and validation of LTDP Continue improvement of LTDP Individual entity adopts (de-facto standard)
• Wide benefits• Economical and technical sustainability• Mutual data enrichment• More Value added services• Continuous improvement
BirthParenting
NurturingSelf-sustained
Poorly reactiveUse
r/
Stakeholder
approachReactive Interactive Proactive
Earth Science Infrastructure Vision for Data Preservation and Access
Page 14
Presentation outline structure
The Scientific Data e-infrastructure in Europe: the High Level Expert Group Vision
The Earth Science Data Preservation and Access challenges
European Earth Observation LTDP Framework: one step ahead for a tangible solution
ESA EO LTDP Preparatory Programme
• FIRST (Definition of LTDP User Requirements and Preserved Data Set Composition) study
• LAST (Long term data Archive Study on new Technologies)
Conclusions
Page 15
ESA Earth Observation data preservation and access: background and challenge
• ESA’s EO archives:
– Content of EO data archives is extending from a few years to several decades.
– Very valuable data representing scientific long time-series for a large number of applications.
– ~150 TB archived in early ‘90s, more than 4 PB archived today and over ~27 PB expected in the next 10 years (Level 0 data).
– Factor of 4 if also Level 1-2 products considered.
• Same trend is expected in other European EO Satellites owners archives.
Page 16
ESA EO Archives Data volume expected trend
0
2000
4000
6000
8000
10000
12000
14000
16000
18000
20000
22000
24000
26000
28000
30000
Tota
l Arc
hiv
e in
Terr
aByt
es
(TB)
1986 1989 1993 1995 1998 2000 2003 2005 2007 2009 2012 2020
Year
Evolution of ESA's EO data archives between 1986-2009 and future estimates
Projection
ALOS (2007-today)
Modis (Jun 01-today)
ENVISAT LR (Mar 02-today)
ENVISAT HR (Mar 02-today)
Others
LANDSAT (75-today)
JERS SAR/ OPS VNIR (92-Sep 98)ERS HR (Jul 91-today)
ERS LBR (Jul 91-today)
Kompsat 2 (2008-today)
NOAA 9-19 AVHRR (86-today)
Page 17
To support the huge amount of scientific data needed by Earth Sciences, ESA is coordinating an initiative to set-up an European Earth Observation LTDP framework considering possible extension to Earth Science data.
ESA has initially targeted the LTDP framework to the Earth Observation space sensed data and has launched the ESA Long Term Data Preservation (LTDP) Programme for an initial period of three years (2009-2011).
Eueropean Earth Observation LTDP Framework Birth: one step ahead for a tangible solution
Page 18
European Earth Observation LTDP Framework context
• ESA is working since years in setting up an international EO LTDP framework for collaboration and cooperation in the Earth Observation domain
• Interoperability among existing infrastructures is becoming a reality in the Global Monitoring Environment and Security (GMES) project context
• None of the European organizations can do this alone, cooperation is mandatory and beneficial for all.
– Value of the sum of entities’ value is higher
• To be done:– Involve the full Earth Observation data value chain (including
researchers, individuals, scientists, value adders).– Identify approach to guarantee the sustainability of the existing data
infrastructure.– Extend the framework to the wider Earth Science domain.– Define an architecture for a pan-European Earth Science data
infrastructure based on existing and new infrastructures.
Page 19
European EO LTDP Framework: Goal and Principles
Goal:
to preserve the European, and Canadian, EO space data sets for an unlimited time-span ensuring and facilitating their accessibility and usability respecting the individual entities data policies.
Principles:
• To preserve the huge amount of scientific data we need an LTDP community supported by interacting organizations
• This LTDP community have to preserve the data and provide access services of value to customers, who are themselves members
• Over time, LTDP members will evolve their capabilities and roles, and will tend to align themselves with the directions collectively defined.
– New perspectives and collective growth of the different actors
Page 20
European Agencies joining forces
• A leading group of European Agencies owning EO mission archives have joined forces in the LTDP Working Group to address the LTDP challenge together since 2008.
• Under the Ground Segment Coordination Body control.
• ESA consultation with its Member States.
• Consultations with all European EO mission and data archive owners (http://earth.esa.int/gscb/ltdp/LTDP_Agenda.html).
Page 21
European EO LTDP Framework parenting
• A collaborative framework open to all possible members: with distributed and heterogeneous entities and facilities:
– to reach an harmonized infrastructure for the preservation and access of the European EO Space Data.
– Sustained through a multilateral cooperation with multiple funding sources from at least the European EO data owners.
• Each European EO Space data owners contributes:– With their ideas and expertise and possibly their infrastructure.
– Progressively adhering to commonly defined and agreed Guidelines and Procedures
– using each one resources in a more efficient manner to achieve the end goal
• Progressive implementation based on a stepwise approach (short, mid, long-term activities)
Page 22
The LTDP international context
ESA membership
Participation to several projects
LTDP task
WGISS
Standards (e.g. OAIS)
A GEO subtask is under Approval for a long term Data Preservation strategyFrom space to in-situ data
Page 23
• Promoted the creation of the European LTDP Framework in Earth Observation,
defined the major cooperation areas and the related implementation plan
o Implementation of the basic rules, “The guidelines”, published reflecting the consensus of the European EO Data providers.
o Defined the initial data set to be preserved, including the related glossary
• Leading the coordination of LTDP in the Earth Observation domain with the European partners:
- Information flow (workshops, web site, conferences
- Watch up the EO user needs and context for EO
- Sharing technology development information
- Participation to several EC initiatives (Caspar, Genesis DR, Parse, AParse, etc.)
• Achieved the approval of an EO LTDP ESA funded preparatory
programme for ESA missions, during ESA C-MIN 2008, and in a broader
scope exploiting also cooperation with EC DG INFOSO
LTDP Framework ESA major achievements
Page 24
Presentation outline structure
The Scientific Data e-infrastructure in Europe: the High Level Expert Group Vision
The Earth Science Data Preservation and Access challenge
ESA LTDP Earth Observation Framework: one step ahead for a tangible solution
ESA EO LTDP Preparatory Programme
• FIRST (Definition of LTDP User Requirements and Preserved Data Set Composition) study
• LAST (Long term data Archive Study on new Technologies)
Conclusions
Page 25
LTDP programme approved at ESA for the period 2009-2011.
LTDP programme approved at ESA for the period 2009-2011.
ESA LTDP Preparatory Programme 2009-2011 Work Plan
LTDP Workplan 2009-2011
Implementation(ESA archives)
Analysis & Studies
Define data set to be archived beyond telemetry
Cost
Funding options
Technical Content (activities for period 2012 onwards)
European LTDP Framework
Integrity of archives
Data security
Interoperability and standardization
Archives exploitation
ESA archives
Schedule
ESA general and mandatory budgets
EC Contributions
National Budgets
European LTDP Framework
Coordination
Create a European collaborative technical framework
Complete, maintain & evolve the LTDP Common Guidelines (involving CEOS/GEO)
Knowledge preservation beyond data archiving
Evaluate impact of latest technologies
Consolidate user requirements
Analyse & coordinate with other international LTDP projects (e.g. funded by EC)
European LTDP ProgrammePreparation
Starting from Earth
Observation
To achieve Earth
Science
Page 26
• Technologies for data preservation
• Maturity level of such technologies
• Gap analysis of ESA practices
• Risks concerned with technologies
ESA LTDP Programme WorkplanESA LTDP Programme Workplan
LTDP Workplan 2009-2011
• Analysis and studieso FIRSTo LAST
• European LTDP framework coordination
• Participation to European initiatives
• LTDP programme preparation
• Implementation (ESA archives)
• Consolidate user requirements
• Define Earth Science datasets composition
• Assess European capabilities
• Review current LTDP guidelines
Page 27
Presentation outline structure
The Scientific Data e-infrastructure in Europe: the High Level Expert Group Vision
The Earth Science Data Preservation and Access challenge
ESA LTDP Earth Observation Framework: one step ahead for a tangible solution
ESA EO LTDP Preparatory Programme
• FIRST (Definition of LTDP User Requirements and Preserved Data Set Composition) study
• LAST (Long term data Archive Study on new Technologies)
Conclusions
Page 28
FIRST (Definition of LTDP User Requirements and Preserved Data Set Composition) study
• Objectives:
– Collection and definition of requirements and needs from Earth Science users in terms of Long Term Preservation of Earth Science data, products and related information.
– Definition of the composition of the Earth Science data set that should be preserved in the long term.
– Preliminary assessment of the current capabilities of European archives and provision of recommendations for the set-up of a European LTDP Framework.
• Kicked off in June 2010 with a duration of 8 months.
– User requirements collected through a questionnaire and interviews.
– Involvement of international experts, GEO Communities of practices and User Interface Committee.
Page 29
Study methodology Study methodology
LTDP/FIRST
Project’s headlines and streams
Consolidate user requirements
Define Earth Science datasets composition
Assess European capabilities
• Looking at source documents• Performing a survey• Analysing similar initiatives
• Identify domains and applications• Identify concerned needs• Perform a survey / cross check
Review current LTDP guidelines
• Define dimensions for assessment• Identify owners, providers and other roles• Capture and assess
• Analyse missing, TBD and uncompleted with respect to requirements / needs
• Propose updating• Propose way forward for the LTDP
Framework
Page 30
Sources of requirementsSources of requirements
• gcos-138.pdf
• GCOS Climate observation needs (PR 2004/2008, et. al.)
• GMES Services: Geoland2, MyOcean, SAFER, MACC
• GMES/GISC in-situ data
• Climate Modelling User Group, Deliverable 1.2, Requirement Baseline Document
• CCI EO Data requirements.
• ECV/GCOS 107
• European Strategy Forum on Research Infrastructure
• http://inspire.jrc.ec.europa.eu/index.cfm
RequirementsCCI
Inspire/JRC
Feedback analysis process
Comparison with similar initiatives
Page 31
LAST (Long term data Archive Study on new Technologies)
• Objectives:
• Study the different archiving technologies mature for operation in the short and mid term time-frame or available in the long-term and able to satisfy at best ESA and EO data providers requirements in terms of digital information preservation.
• Kicked off in May 2010 with a duration of 12 months.
• Activities:
– Requirements Definition and Due Diligence
– Archiving Technology Survey
– Testing and Benchmarking
• Technology workshop held in Madrid in July 2010 with several participants from international space agencies.
• Technical web site implemented: http://ltdpts.eo.esa.int/
– to be used as reference site for archiving technology and LTDP technical issues (e.g. forums, etc).
Page 32
Presentation structure
The Scientific Data e-infrastructure in Europe: the High Level Expert Group Vision
The Earth Science Data Preservation and Access challenge
ESA LTDP Earth Observation Framework: one step ahead for a tangible solution
ESA EO LTDP Programme
• FIRST (Definition of LTDP User Requirements and Preserved Data Set Composition) study
• LAST (Long term data Archive Study on new Technologies)
Conclusions
Page 33
ESA LTDP programme in brief
• Preservation of Earth Science data in the long term is a major challenge in Europe.
• LTDP cooperation activities aimed at guaranteeing the preservation of European data are ongoing in the Earth Observation domain under ESA coordination.
• ESA is already active:
– In applying the EO LTDP Common Guidelines to its own missions.
– In implementing essential enhancements in its facilities focusing on data preservation and enhancement of data access.
– In preparing an LTDP programme proposal for the period beyond 2012 to be presented to its Delegations for approval.
• ESA plans to extend the LTDP Initiative to the wider Earth Science domain, enlarging the current Earth Observation scenario with key stakeholders, to guarantee preservation, access and exploitation of Earth Science data and information for the benefit of European users and citizens.
Page 34
Users view major elementsUsers view major elements
Major generic considerations as output from several user surveys results (FIRST, PARSE, ESA User Service, etc.):
1. Preservation is a common and widely recognized need to allow performing today and future Earth Science research and application activities
2. Preserve all forever is the mantra for the users
3. A user centered vision should be applied rather than a provider centered one
4. Preservation and future re-use of data paves the way to further knowledge growth
5. Harmonization among different actors and providers should be a primary objective
6. Access to data and services should be simple and costless
7. Security issues are matter of providers/owners and should minimize the effort on user side
8. Time-series durations/windows needs are different for different applications and utilizations
9. Periodic reprocessing of data is a necessity
Page 35
Infrastructure availability
• The opportunity of having infrastructure for Earth Science data preservation is a need
• An infrastructure for data preservation and access would positively impact the daily work
• would solve some of the identified concerns and barriers
• could improve data preservation, access and data distribution systems
• would allow to try new data assimilation methods and apply models
• would allow the virtualisation of data location and maintenance • would allow to take advantage of computational resources not accessible
otherwise
• APA could play a fundamental role in building up the infrastructure: generation of a generic scalable model ?
Page 36
The operational challenge: from simple data repository to an operational set up
• The implementation of an harmonized infrastructure will allow progressively to reach in long term an “Operational” Earth Science Data preservation & access system that requires at least :
– Clear mandates for all partners
– Coordinated approach between all partners and adoption of common policies / guidelines
– Commonality and harmonization of services
– Standardization at different levels
– Adequate systems infrastructure implementation, availability and maintenance
– Sharing of interoperable resources, networking, computing power
– Long term sustainability
Page 37
Thank you
END
