System/SDWG Update

Management Council Face-to-FaceFlagstaff, AZ

August 22-23, 2011

Sean Hardman

Topics

• Component Overview• Design Progress• Development Progress• Build 1d Deployment• Build 2 Plans• Build 2 Deployment• PDS3 Support

August 22-23, 2011 System/SDWG Update 2

Component OverviewCore System Services/Tool

• Registry Service- Provides functionality for tracking, auditing, locating,

and maintaining artifacts (e.g., products, services, etc.) within the system.

• Harvest Tool- Provides functionality for capturing and registering

product metadata with the Registry Service.• Search Service

- Provides functionality for searching the metadata contained in the Registry Service instances.

August 22-23, 2011 System/SDWG Update 3

Component OverviewSupport System Services

• Security Service- Provides the authentication and authorization

functions for the system.• Report Service

- Provides functionality for capturing and reporting metrics.

• Monitor Service- Provides functionality for monitoring service status

across the system.

* These services are satisfied with COTS or Open Source software products.August 22-23, 2011 System/SDWG Update 4

Component OverviewTools

• Design Tool- Provides functionality for designing product label

schemas using the XML Schema standard.• Generate Tool

- Provides functionality for generating product labels from and conforming to the specific product schema.

• Validate Tool- Provides functionality for validating product labels

and product data.• Catalog Ingest Tool

- Provides functionality for registering PDS3 catalog files with the Registry Service.

August 22-23, 2011 System/SDWG Update 5

Design Progress

• Each system component has a corresponding requirements and design specification.- Level 4 and 5 requirements traced back to PDS Level 1, 2

and 3 requirements.• Specifications completed and reviewed:

- Registry, Harvest, Security and Report- Updated for the System Design Review to reflect actual

implementation.• Specifications in progress / planned:

- Preparation (tools), Search, Monitor and Operator Portal• Latest versions posted to Engineering Node site

- http://pds-engineering.jpl.nasa.gov/index.cfm?pid=145&cid=134

August 22-23, 2011 System/SDWG Update 6

Design ProgressSearch Service

• PDS search scenarios have been gathered from a number of sources.- These scenarios drive query model development.

• The architecture for the service supports:- Multiple protocols on the front-end allowing support for the

PDS and PDAP protocols at a minimum.- Multiple data sources on the back-end allowing integration

with existing Node services and future upgrades.• The default back-end data source is the Apache Solr search

engine.• PDS search protocol supports both simple and

advanced query scenarios as well as data access.- The protocol is based on the Solr/Lucene protocol.

August 22-23, 2011 System/SDWG Update 7

Development Progress• Finishing up Build 1d (August 2011)

- Completing development and integration of core services (Registry, Harvest and Security).

- Exercising the Report Service with Node web logs.- Introduction of the Generate Tool developed by the

Imaging Node.- Performing stress testing.

• Working towards Build 2 (October 2011)- Initial development of the Search Service and search-

related applications.- Building out the tool suite to support early adopters.- Adding support for PDS3 going forward.

August 22-23, 2011 System/SDWG Update 8

Development ProgressBuild 1d – New and Modified Features

• Registry Service- Modified registry configuration from the data model

to include slot definition and classification schemes.- Added support for registry replication.- Added support for package registration.

• Harvest Tool- Added support for associated file registration.- Added support for registration of PDS3 proxy

products from PDS3 data sets.- Modified support for association handling and

representation in the Registry Service.

August 22-23, 2011 System/SDWG Update 9

Development ProgressBuild 1d – New and Modified Features cont.

• Introducing the Generate Tool developed by the Imaging Node.

• Integrated support for passing credentials into the Registry Service client API.

• Updated the Validate Tool with the latest set of schemas for Build 1d.

• Updated the original Registry User Interface to keep pace with the Registry Service interface changes.

• Exercised the Report Service by populating with Node web logs and replicating existing reports.

August 22-23, 2011 System/SDWG Update 10

Build 1d Deployment

• Not entirely different from the deployment of Build 1c.

• The core and support components will be deployed to the EN target operational environment.

• Requesting Node participation in exercising this build.

August 22-23, 2011 System/SDWG Update 11

Build 1d DeploymentEngineering Node

August 22-23, 2011 System/SDWG Update 12

Build 1d DeploymentDiscipline Node

August 22-23, 2011 System/SDWG Update 13

• Requesting that each Node download, install and test the core components in their environment.- Core components include the Registry Service,

Harvest Tool and the Registry User Interface.• This will help to identify any system

requirement issues prior to Build 2.• Atmospheres and Imaging (JPL and USGS)

have exercised previous builds.

Build 1d DeploymentDiscipline Node cont.

August 22-23, 2011 System/SDWG Update 14

Build 2 Plans• Initial Distribution Subsystem

- This includes the Search component and an updated Data Search interface at the EN.

- Replicate EN functionality with PDS 2010 infrastructure.• Ingestion Subsystem

- Completed core components with support for PDS3 products and keeping pace with changes in the data model.

• Tools- A stable suite of tools for early adopters including Design,

Generate, Validate and initial Transformation tools.• Operations

- Configuration of the Report component, selection of an off-the-shelf product for Monitor, and development of Tracking, Subscription and Phonebook interfaces.

August 22-23, 2011 System/SDWG Update 15

Build 2 Deployment• Engineering Node

- Deployment of the core components replacing the central catalog.

- Existing search interfaces reworked to utilize the PDS4 infrastructure.

• Discipline Nodes- Deployment of the core components at the Nodes will be

phased once the ORR has been completed.- Once deployment is complete, Node should begin registration

of PDS3 products with the Harvest Tool.• This is not PDS3 product migration.• A generic proxy label is generated and registered to facilitate

product-level tracking.• This product will be replaced once the product is migrated to

PDS4.

August 22-23, 2011 System/SDWG Update 16

Build 2 DeploymentEngineering Node

August 22-23, 2011 System/SDWG Update 17

Build 2 DeploymentDiscipline Node

August 22-23, 2011 System/SDWG Update 18

PDS3 Support

• The Harvest Tool supports both PDS4 registration and PDS3 registration.- PDS3 support consists of converting PDS3 labels into

PDS4 proxy labels.- Registered for tracking and reporting purposes.- Will be replaced when the corresponding PDS3 data

set is migrated to PDS4.• The current Catalog Ingest Tool is updated to

convert the catalog files to context products and register them with the registry.

August 22-23, 2011 System/SDWG Update 19

PDS3 Support cont.

August 22-23, 2011 System/SDWG Update 20

Wrap Up

• Design and development are on track to support the Build 1d and Build 2 releases and to begin the transition to PDS4.

• As with Build 1, incremental builds will occur after Build 2 to continue building on core functionality and to support migration activities.

August 22-23, 2011 System/SDWG Update 21

Questions/Comments