EarthCube – Brokering A Way Forward for Global Multi-disciplinary Data Sharing

ESIP Summer Meeting 2012 Slide 1

EarthCube – BrokeringA Way Forward for Global Multi-disciplinary Data Sharing

Brokering Concept Award TeamSteven F. Browdy

http://earthcube.ning.com/groups/brokering


Agenda

• Introduction of Brokering Concept Award• Short history of EarthCube brokering activities• Current brokering approaches• EuroGEOSS Broker• Brokering Principles• EarthCube Particulars• Future brokering possibilities• Future EarthCube brokering activities• Q & A


Brokering Concept Award

• EarthCube’s goal is “To transform the conduct of research in geosciences by supporting community-based cyberinfrastructure to integrate data and information for knowledge management across the Geosciences.”

• What is needed?– Integration of data, tools and communities through a

cyberinfrastructure– Utilization of current and emerging technologies to create a

transparent infrastructure for the geosciences community

• The EarthCube approach:– “Build it and they will come” doesn’t work– What gets built has to be the result of a community process,

where the community is much more than the IT enthusiasts



• Our objective– Develop a strategy and roadmap for evaluating promising

brokering technologies in cooperation with a diverse set of US and international partners

• Our approach– Develop and communicate a long-term strategy and roadmap for

fully developing brokering as a core Earth Cube cyber-infrastructure component

– Demonstrate and assess several instances of brokers that mediate search and access to heterogeneous multidisciplinary data sources and services



• Expectation– Engage the long tail folks, who are part of the 85% of

geoscientists who will be training the next generation– Bring them onboard, understanding at some level what we’re

doing and what is involved, helping to change the way science is done

– Interact with the other EC Groups regarding the utilization of brokering

• Technical assessment responsibility– Pilot studies with different scientific communities– Hack-a-Thons in conjunction with ESIP


Brokering CA Program Plan

4/1/12 6


Agenda



Short History

• Roadmap development– An ongoing process

• Hack-a-Thon 1– Used EuroGEOSS Broker

• Pilots– NSIDC Arctic Polar Pilot

• June EarthCube Charrette• Cross-fertilization with other EarthCube groups• July EarthCube PI Meeting


Short History - Roadmap

• Informed by our partners and members, as well as other EarthCube groups and the broader community of geoscientists.


Short History – Roadmap Executive (Meta)Summary

• Developing infrastructure is a complex, dynamic process. (See the NSF report on Understanding Infrastructure: Dynamics, Tensions, and Design)– http://earthcube.ning.com/group/governance/forum/

topics/research-review-understanding-infrastructure-dynamics-tensions

• Brokers help mitigate the tensions by connecting disparate systems without imposing new burdens upon those systems



• Some of the benefits of a Brokering Framework include:– Lowers barriers to participation in distributed systems– Accelerates interconnection of disparate systems– Facilitates sustainability, reusability, extensibility, and

flexibility of the infrastructure– Enhances multi-disciplinary interoperability– Removes need to impose common (e.g. federal, “top-

down”) specifications enabling a more adaptive “bottom-up” evolution of the infrastructure



• Community engagement– It is necessary to very actively engage both the

science and informatics communities, and to employ use-case-driven requirements

– it is impossible to define all requirements in advance of development of an infrastructure

– In the short term we will be addressing small community problems through collaborative pilots

• limited initial capabilities targeted to specific communities• guided by use cases• ongoing user and community feedback

– Evolve into larger prototypes addressing larger intercommunity problems



• Brokering has been demonstrated to be an essential part of a robust, adaptive cyberinfrastructure

• Next Steps– Critical questions of governance and detailed

implementation will be addressed– Continued pilots and community engagement– Expansion of brokering pilots into fully operational

prototypes


Short History – Hack-a-Thon 1

• Featured the EuroGEOSS Broker• All hack-a-thon related material is on the EarthCube Brokering ning

site– http://earthcube.ning.com/group/brokering/page/brokering-hack-a-thon-1

• A webinar was given to introduce the featured broker on May 10• A period of 10 days was used to allow participants to practice with

the broker– Validate services (required assistance)– Validate tools

• A 2-hour WebEx-enabled Hack-a-Thon occurred, after the practice period, on May 24– 15 participants from various EarthCube groups and science disciplines– Resulted in lessons learned and take-a-ways


Short History – Hack-a-Thon 1

• Take-a-ways– Keep the featured broker available post hack-a-thon for

continued “playing”– Setup a wiki area for further discussion and collaboration– Maintain a knowledgebase for broker issues

• Lessons learned– Too narrowly focused

• Future hack-a-thons will include discussions from previous ones

– Need a broader audience– Allow a longer practice period with telecons to maintain

engagement– Poll participants for questions and comments to engage

everyone


Short History - Pilots

• NSIDC Arctic Polar Pilot– Using the EuroGEOSS Broker– Discovering and accessing multidisciplinary

data– Analyzing the broker

• Providing detailed requirements to developers at NSIDC, CNR, and NCAR to enhance capabilities


Short History – etc.

• Attended June EarthCube Charrette– Much interaction with other groups– Very productive in terms of working towards a shared vision, but

more work to do– Started the consensus process on priorities and timelines

• Cross-fertilization with other EarthCube groups– Gave presentations about brokering to other groups– Worked together on common use cases– Understanding areas of potential success for brokering

• July EarthCube PI Meeting– Consensus on reference architecture diagram– Consensus on development of common use cases


Agenda



Current Brokering ApproachesSingle Monolithic Instance - Standalone


Current Brokering ApproachesSingle Monolithic Instance - Cloud


Current Brokering ApproachesMultiple Instances – Cloud Farm


Current Brokering ApproachesCommunity Deployment


Agenda



Introduction to the EuroGEOSS Broker

• The EuroGEOSS Broker was developed by the Laboratory of Earth and Space Science Informatics (ESSI-Lab) of the Institute of Atmospheric Pollution Research (IIA) of the National Research Council of Italy (CNR)

• It has been deployed in many situations, such as:– GEOSS Common Infrastructure

– NSIDC

– UNEP-Live

– WIS (WMO Information System)

• It’s capabilities include discovery, access, and “light” semantic mediation

• New “accessors” can be added to interoperate with data providers using standards currently unknown to the broker



Introduction to the EuroGEOSS Broker Supported Sources

• OGC WCS 1.0, 1.1, 1.1.2 • OGC WMS 1.3.0, 1.1.1• OGC WFS 1.0.0 • OGC WPS 1.0.0 • OGC SOS 1.0.0 • OGC CSW 2.0.2 Core, AP ISO 1.0,

ebRIM/CIM, ebRIM/EO, CWIC• WAF Web Accessible Folders 1.0 • THREDDS 1.0.1, 1.0.2• THREDDS-NCISO 1.0.1, 1.0.2• THREDDS-NCISO-PLUS 1.0.1, 1.0.2• CDI 1.04, 1.3, 1.4, 1.6• GI-cat 6.x, 7.x• GBIF• GeoNetwork (versions 2.2.0 and 2.4.1)

catalog service• HYDRO• Deegree (version 2.2) catalog service

• ESRI ArcGIS Geoportal (version 10) catalog service

• OpenSearch 1.1 accessor• OAI-PMH 2.0 (support to ISO19139 and

dublin core formats)• NetCDF-CF 1.4 • NCML-CF• NCML-OD• ISO19115-2• GeoRSS 2.0• GDACS• DIF• File system• SITAD (Sistema Informativo Territoriale

Ambientale Diffuso) accessor• INPE


Introduction to the EuroGEOSS Broker Available Catalog Interfaces

• OGC CSW 2.0.2 AP ISO 1.0• OGC CSW 2.0.2 ebRIM EO• OGC CSW 2.0.2 ebRIM CIM• ESRI GEOPORTAL 10• OAI-PMH 2.0• OpenSearch 1.1• OpenSearch 1.1 ESIP• OpenSearch GENESI DR• GI-cat extended interface


Broker Use Cases

View the webinar for Hack-a-Thon 1 to see these use cases demonstrated:https://esipfed.webex.com/esipfed/ldr.php?AT=pb&SP=MC&rID=59212547&rKey=72865c52d24d8992


Use Case 1 – Traditional Query

Preview via Access Broker

Native SourceWCSGeoTiffEPSG:4326

Accessed asWMSPNG

EPSG:4326


Use Case 2 – Semantics-enabled Query

Preview via Access BrokerNative SourceWFSGMLEPSG:4326

Accessed asWMSPNG

EPSG:4326


Use Case 3 – Access on a Common Grid

• Definition of a Common Grid Environment for discovered datasets– Encoding Format– Spatial Resolution– Spatial Extension– Coordinate Reference System– Etc.

• Download datasets on the Common Grid• Visualize Downloaded Datasets on the

Common Grid (using the Unidata IDV tool)


Use Case 4 – Third-party Clients

• Discovery:– GI-go (CSW/ISO Client)– Firefox Search Bar (Opensearch Client)

• Access– Mapshup (WMS Client)

Preview via Access Broker

Native SourceWCSGeoTiffEPSG:4326

Accessed asWMSPNG

(Google CRS) EPSG:3857


Agenda



Brokering Framework


Brokering PrinciplesProblem Statement

• Whereas:– Earth system science data are extremely diverse (in

format, description, scale, precision, etc.)– Different data users discover, access, interpret, and

manipulate these data with a wide variety of tools and services that may be specific to the data and each user’s needs

– Interdisciplinary science requires people to access and use data from very different disciplines and communities

– Data providers are increasingly expected to serve communities outside their normal clientele


Brokering PrinciplesProblem Statement

• Whereas:– Both users and data providers experience

impediments in achieving multi-disciplinary interoperability

– Connectivity for multi-disciplinary interoperability should not restrict the autonomous nature of discipline-specific systems

– Technology evolution and maintainability is a great challenge for cyber-infrastructure sustainability and usability

– Brokers should be capable of improving and facilitating access to both “big data” and “long tail” resources, in real time or static environments


Brokering PrinciplesAssertions

• We Believe:– No one information technology or standard will serve

all user needs– Data services (discovery, access, processing,

semantics) should be openly accessible in well-defined, machine-interpretable ways

– Cyberinfrastructure should maximize data and service usability for both data providers and data users

– Flexibility is necessary for incremental expansion of data service mechanisms and to easily achieve scalability

– Brokers can be an effective way to achieve this flexibility


Brokering PrinciplesAssertions

• We Believe:– Brokering [or a brokering framework] provides the

cyberinfrastructure that allows providers and users to better take advantage of the open services

– A brokering framework is best managed and maintained by the cyberinfrastructure (rather than Data Users or Data Providers)

– A brokering framework should be transparent to Data Users and Data Providers

– Not all existing systems will continue, but it is more sustainable to supplement, not supplant, systems mandates and governance arrangements


Brokering Principles Working Definition of a Brokering Framework

• Definition 1:– Brokers are middleware interconnecting client and server

components in the EarthCube cyberinfrastructure– Brokers are services facilitating the run-time interconnection

(sharing of resources) among users and providers in a way that requires little effort on the part of either

– A brokering framework can consist of multiple brokering components to support different capabilities

– A brokering framework may include discovery, semantic enhancement and natural language, data access, processing and publishing


Brokering Principles Working Definition of a Brokering Framework

• Definition 2:– A broker enables at runtime service components to transparently

make and receive requests and responses in a heterogeneous and distributed environment. It is the foundation for building multi-disciplinary applications from distributed service components and for interoperability between applications in heterogeneous and homogeneous environments.

Read our Brokering Principles at the Brokering Concept Award site, and feel free to post comments:http://earthcube.ning.com/group/brokering/forum/topics/brokering-principles-continued


Agenda



EarthCube Particulars

• Proposed reference architecture

• Connection with governance

• Common use cases


EarthCube ParticularsReference Architecture


EarthCube ParticularsGovernance

• How should EarthCube CI brokers be deployed and maintained?– What are the requirements for deployment, who sets them, and

what processes are used?– What are the requirements for maintenance, who sets them, and

what processes are used?– What is the funding mechanism?

• What are the policies for brokering across the boundary between communities and the EarthCube CI?

• Standards for publishing data, information, and knowledge?

• What is the process for the evolution and innovation of brokering for the EarthCUbe CI?

• Other issues?


EarthCube ParticularsUse Cases

• Implementation Goals– Interdisciplinary demonstration for diverse providers and users– “Full” range of the EarthCube CI exercised– Collaboration between EarthCube groups– Provide a clear demonstration of benefits– Can be continued as a practical implementation

• Use cases being evaluated– Atmospheric ash– Gulf of Mexico hypoxia– Arctic polar environment


Atmospheric AshSolid Earth Use Case

• Impacts of volcanic emission on human activities

• (Pilot 1 – volcanic supersite)• Pilot 2 – Volcanic ash from Icelandic

eruption• Disciplines – seismology, geodesy, polar,

atmosphere, biodiversity, transportation, Potential EarthCube participants –

• Other


Gulf of Mexico HypoxiaWater Use Case

• Water and Ocean environment• Gulf of Mexico Hypoxic region• Disciplines – land cover, river water quality,

ocean chemistry, weather, biodiversity• Potential EarthCube participants –• Other?


Arctic EnvironmentPolar Use Case

• Arctic Environment

• Data exploration service for discovery of interdisciplinary Arctic data.

• Focus areas – arctic contaminant monitoring, seasonal sea ice predictions, biodiversity

• Potential EarthCube participants –

• Other?


Agenda



Future Brokering Possibilities

• Additional functionalities– Enhanced semantics (ontologies)– Multi-natural language processing– Enhanced data transformation– Interoperation with workflows– HPC deployment for modeling

• Architecture/framework alternatives– Interoperability between same brokers– Interoperability between different brokers


Same Broker Interoperability

• Known interoperability mechanisms

• Mutual understanding


Different Broker Interoperability

• Unknown interoperability mechanisms (possibly new interoperability standards)

• New coordination policies needed (publish capabilities)• Possibly more governance issues


Agenda



Future EarthCube Brokering Activities

• Hack-a-Thons 2 and 3– Currently being planned– Hack-a-Thon 2 by mid-August– Hack-a-Thon 3 by mid-September– Nominate a broker for a hack-a-thon

• Email Steve Browdy at [email protected]

• More pilots

• Roadmap evolution


Q & A

Documents

EarthCube – Brokering A Way Forward for Global Multi-disciplinary Data Sharing