Upload
others
View
23
Download
0
Embed Size (px)
Citation preview
PDS/IPDApresentationtotheIVOAStevenJoy,BaptisteCecconi
Outline• PlanetaryDataSystem(PDS)activities
• PDSOverview• PDSArchiveProcess• PDS4ImplementationanddifferencesfromPDS3• PDS4ImplementationofDOIs• PDS4registryandsearch
• InternationalPlanetaryDataAlliance(IPDA)activities• IPDAOverview• IPDAStructure• IPDAGoalsandChallenges• IPDAProgressandFutureDirections
• SolarSystemInterestGroup(SSIG)• Comments
• Summary
2
PlanetaryDataSystem(PDS)Overview• PDSis,andhasalwaysbeenforemostadataarchiveandsecondarilyadatasystem
• Asanarchive,PDSrequiresthatdataanddocumentationarestoredinstandard,non-proprietaryformats
• ThisoftenprecludestheuseofcommonuserformatssuchasMSWordfordocumentsorIDLsavesets
• PDSisanopenarchive,availabletoanyone,anywhere• PDSisorganizedontheprincipalthatplanetarydataarediverse,andthatthebestcuratorsofsuchadiversearchivearedomainexperts(scientistandtechnologists)inthevariousfieldsofplanetaryscience.Assuch,PDSisafederationofScienceDisciplineandSupportNodes:
• AtmosphericSciences(ATM)– NewMexicoStateUniv.– NancyChanover• Geosciences(GEO)–WashingtonUniv,St.Louis– RayArvidson• ImagingandCartography(IMG)– USGSFlagstaff– LisaGaddis• PlanetaryPlasmaInteractions(PPI)– UCLA – RayWalker• RingMoonSystem(RMS)– SETI –MarkShowalter• SmallBodies(SBN)– Univ ofMd.,CollegePark – James(Gerbs)Bauer
• NavigationandAncillaryInformation(NAIF) – JPL – ChuckActon• Engineering(ENG)– JPL – DanCrichton• Management(MNG)– GSFC – TomMorgan
• MostoftheScienceDisciplineNodeshavemultiplesub-nodesthatprovideexpertiseacrossmanyaspectsofthedisciplinethatarenotavailableattheNodelocation
3
PlanetaryDataSystem(PDS)– SomeHistory• PDS1wentoperationalin1989
• Highlevelcatalogforfindingdatasetsbymission,instrument,spacecraftandtarget• ObjectDescriptionLanguage(ODL)isinventedforproductlabelingandcataloginformation• Dataarchivedanddistributedon9-tracktape
• PDS2(1990)• Cataloginformationstreamlined,longertextdescriptions,fewerkeyword/values• CD-ROM(CD-R)becomestheprimaryarchiveanddistributionmedia
• PDS3(1992)• PDSsetsupandmaintainsawebpresence• Movementtoonlinedistribution(~2002)ofelectronicvolumes(PDS-D)• OnlinemassstorageanddatabricksreplaceCD/DVDasarchivedistributionmedia
• PDS4(2014)• Prototypebuild1in2010,operationalin2014withLADEEasfirstmissiontousenewstandard• ModeldrivenarchitecturewithXMLlabels• Distributedregistriesandsearchservicesreplacethecatalogandsupportdirectfilelevelaccess• Newdatastandardsandreducedsetofarchivedataformatsandstructures• Internationalcollaboration
4
PDSArchiveProcess
• ThePDSarchiveprocessshouldbeaniterativeprocessbetweentheNodeandthedataprovider(s)duringwhichthecontentandstructureofthearchivearenegotiated,delivered,reviewed,andrevised
1. ArchiveDesign– ProvidersproposeacollectionofdataanddocumentationforarchiveandthenworkwiththePDSNodetorefinethedesign.Documentationiswritten,sampleproductswithlabelsaredeliveredforinternalreview.
2. PeerReview– OncetheNodeanddataprovideragreethatthedataarereadytoarchive,aformalpeerreviewissetupwiththereviewpanelconsistingofatleasttwoindependentsciencereviewers,PDSstandardsreviewers,andthedataproviders.Thereviewresultsinoneofthreeoutcomes:• Archived:Dataanddocumentationareacceptedforarchive– noissuesfound,noliens.• CertifiedwithLiens:Dataarecertifiedasscientificallyusefulastheyarebuttherearesomeissuesthat
needtobefixedbeforetheycanbearchived.Datacanbereleasedtothepublicintheinterim.• Notcertified:Dataordocumentationhavedeficienciesthatpreventthemfrombeingscientificallyuseful
untiltheissuesarecorrected.Archivewillneedanotherreviewbeforebeingreleasedtothepublic.3. LienResolution– Thedataprovidercorrectsissuesreportedduringreviewandresubmitsthedatafor
archive.4. Archive – Allissuesreportedduringthereviewhavebeenaddressed.Onlineversionhasusagewarnings
removedandthearchiveissenttotheNSSDCAfordeeparchive.5
InitiateContact
DataProvider
ArchiveDesign
DataProvider
PDSNodeInternalReview
PDSNode
PeerReviewScientistsLiens
ArchiveLienResolution
Verified
Comments
PDS4:AnInternationalPlanetaryArchivingEffort
• PDS4 is an international planetary archive data model and standard that is being used to build compatible archives
• An explicit information architecture• All planetary data tied to a common model to improve validation and discovery
• Use of XML, a well-supported international standard, for data product labeling, validation, and searching.
• A hierarchy of data dictionaries built to the ISO 11179 standard, designed to increase flexibility, enable complex searches, and make it easier to share data internationally.
• An explicit software/technical architecture• Distributed services both within PDS and at international partners
• Consistent protocols for access to the data and services
• Deployment of an open source registry infrastructure to track and manage every product in PDS
• A distributed search infrastructure
6
PDS4DifferencesfromPDS3
SlidetakenfromD.CrichtonpresentationonOperationalReadiness– 2013-09-137
SlidetakenfromD.CrichtonpresentationonOperationalReadiness– 2013-09-138
PDS4DataModel
ThePDS4datamodelgrowsfromtheconceptofadataset,itscontents,andhowiscanbedescribedanddocumentedinasearchabledatasystem
Adatasetconsistofcollectionofdigitalobjects(data,documents,calibration,etc.)thatwhendescribedbymeta-data(labels)becomedigitalproductsAdditionalmetadataarerequiredthedescribetheconceptualobjects(missions,targets,etc.)inordertosupportsearches
CollectionProducts
BundleProducts
Collection ProductsCollection Products
BasicProducts
BasicProducts
BasicProducts
BasicProducts
PDS4ArchiveOrganization
• Thereare3primarytypesofproductsinPDS4:
• Basic Products are the smallest unit of a PDS4 archive. They consist of an individual label and the associated file or files.
• Related basic products of the same type may be grouped together into a Collection.
• Related collections may be grouped together into a Bundle.
10100001000100101001101101000100001
SlidetakenfromJ.Mafi PDS4trainingpresentation,April20189
PDS4LogicalandVersionIdentifiers• ALogicalIdentifier(LID)isauniqueIDthatmaybeusedtoidentifyandreferenceanyPDS4product.
• LIDsmustbeuniqueacrosstheentiredatasystem• LIDstaketheformofaUniformResourceName(URN)• LID:urn:nasa:pds:bundle:collection:product
• urn,agency,andorganizationarestatic,butmayvarybyarchivingorganization(e.g.“urn:esa:psa”,“urn:jaxa:darts”,etc.)
• bundle isabundleidentifier(e.g.“maven-swea-calibrated”)• collection isacollectionidentifier(e.g.“data-svy-pad”)• productisanidentifierfortheindividualproduct
• TheproductVersionIdentifier(VID)maybeappendedtotheLIDtoformaLIDVID• Adoublecolon(::)isthedelimitertoseparatetheVIDfromtheLID• LIDVID:urn:nasa:pds:bundle:collection:product::vid
• WhenaproductisrequestedusingonlyaLID,themostrecentversionoftheproductisreturned.SpecificversionsofproductscanberequestedbyspecifyingthefullLIDVID.
10
FundamentalDataStructures
PDS4 archive products must be describable using one of the following fundamental structures:
Array – homogenous binary structures of 1 to 16 dimensions in which all of the elements have the same data type.
Table – ASCII or binary data with a repeating record structure made up of fixed-width fields.
Parsable Byte Stream – ASCII data with a repeating record structure made up of variable width fields separated by a field delimiter (e.g. CSV).
Encoded Byte Stream – Files formatted according some established standard (e.g. PDF).
SlidetakenfromJ.Mafi PDS4trainingpresentation,April201811
PDS4Implementation
PDS4XMLSchema
PDS4XML
SchematronPDS4
Metadata
The structure and content of PDS4 metadata is defined by a formal Information Model.Information Model
PDS4 is implemented in XML and expressed in terms of XML Schema and Schematron files.
SchemaSchematron
• Schema define the metadata structure• Schematron provide rule-based constraints on
elements and content
SlidetakenfromJ.Mafi PDS4trainingpresentation,April201812
AnatomyofaPDS4Label
• XMLidentificationtag;Schematron identification(optional)XML Declaration XML Declaration
Product (Root) Tag
Identification Area
Observation/Context Area
File Area
Reference List
Product Tag• Root tag; Namespace declarations; Schema identificationIdentification Area• Product identifying information
Observation/Context Area• Product provenance/background
• Links to relevant products and publications
• File format and/or structural information
Reference List
File Area
SlidetakenfromJ.Mafi PDS4trainingpresentation,April201813
PDS4UseofDOIs• ThemostrecentversionofthePDS4datamodeldoesnotyetsupporttheuseofDOI’s
• Thechangerequestforincludingthisattributewasstillunderreviewatthefreezedateofthelastbuild.
• DOI’swillbeincludedinthenextbuildthatwillbecomepublicinOctober2018
• PDSiscurrentlygeneratingDOIsforallPDS4collections,andforselecteddocumentsanddataproducts
• DataarchiveSIS(softwareinterfacespecification)andcalibrationproceduredocumentswillbeassignedDOIs– listsofprojectpersonnelwillnotbe
• OnlyspecialdataproductsthatareexpectedtobesitedintheliteraturefrequentlyarelikelytobeassignedDOIs– nodataproductstodatetomyknowledge
• PDSisregisteringDOIswiththeU.S.Dept.ofEnergyOfficeofScientificandTechnicalInformation(OSTI)
• DOIscanonlybeassignedbyLIDVIDsincetheyrefertoaspecificversionofaproductordocumentthatmustalwaysbevalid
• ADOImaybereservedforacollectionordocumentsothatitcanbeincludedinalabelpriortorelease(publication) 14
MetadataHarvestingandRegistration
SlidetakenfromD.CrichtonpresentationonOperationalReadiness– 2013-09-1315
InternationalPlanetaryDataAlliance
InternationalPlanetaryDataAlliancehttp://planetarydata.org
16
MissionofIPDA*
• Facilitateglobalaccessto,andexchangeof,highqualityscientificdataproductsmanagedacrossinternationalboundaries
• Supportconstructionofcompatiblearchives• Supportsharingoftoolsandsoftwareservices• DefinedatastandardswithintheIPDA,includingthedata
modelsandderiveddictionaries,basedontheNASAPlanetaryDataSystem(PDS)thatisthede-factostandardforallplanetarydataatthetimeoftheIPDAfounding
*ExtractedfromIPDACharter,July2007
17
SteeringCommitteeMembers
Structure
28Members12Countries/InternationalInstitutionsChair:TomStein,WashingtonUniversityDeputy Chair:ChristopheArviest,ESA
Technicalexpertsgroup
~20Members
Generalactivities
Annualmeeting,usuallyinJulyorAugust,20-30participantsRegularteleconferencesevery3months10-20participantsParticipationinrelatedmeetings:COSPAR,EGU,AGU,EPSC,etc...IPDAWebsitehttps://planetarydata.org
18
IPDAGoalsandProgress1. Supportconstructionofcompatibleplanetarysciencedataarchives
• In2012,theIPDAendorsedPDS4asthearchivingstandardforplanetarydata• Implementation,oraplantodoso,ofthePDS4standardacrossagenciesformissionarchiveincludingESA,IKI,ISRO,JAXA,KARI,NASA,UAE.
2. Supportthesharingoftoolsandsoftwareservices• ThePDS4validationtoolususedbyseveralIPDAmemberswhichimprovesinteroperabilitybetweenagencies.
• TheToolRegistryisanotherexampleofavirtualclearinghouseofplanetarydatarelatedtoolscreatedbyavarietyofproducersacrosstheglobe.
3. Facilitateglobalaccessto,andexchangeof,highqualityscientificdataproductsmanagedacrossinternationalboundaries• Atpresent,thereareRESTbasedaccessservicesareinplacebetweenESA,ISRO,andNASA.• High-levelsearchbetweenESAandNASAhasbeendemonstratedandisinplaceforbothPDS3andPDS4archives.
• ProjectsareunderwaytodevelopcitationlinkagewithpublicationsandtocreatebridgesbetweentheIPDAandtheInternationalVirtualObservatoryAlliance(IVOA).
19
IPDAFutureChallenges1. Supportconstructionofcompatiblearchives
• Guidelinesonhowtobuildhighqualityarchives(contentandreview)• Improvedsupportanddocumentationincludingtutorialsforimplementation
2. Supportsharingoftoolsandsoftwareservices• ServiceregistryforGIStechnologiesandtoolsassuggestedatthePlanetaryGISmeeting• Coordinatedtooldevelopmentacrossinternationalboundaries
3. Facilitateglobalaccessto,andexchangeof,highqualityscientificdataproductsmanagedacrossinternationalboundaries• APIaccesstoarchives• Continuetodevelop,evolve,andpromotetheOpenPlanetaryDataAccessProtocol(PDAP)servicesatJAXAincludingwebmappingservices
• IntegrateISROdataintointernationalsearchandaccess• Productlevelsearchandaccessacrossagencies• Integrateaccesswithcomputation• Expandinteroperabilityacrossspacesciences
20
SolarSystemInterestGroup(SSIG)oftheIVOACharter/Goals
• IreviewedtheSSIGcharterandtheonlyrealsuggestionthatIhaveisthatthestatementbeamended,probablyinthe“Goals”area,toaddastatementaboutself-promotion
• Somethinglike:
Promoteawarenessinthescientificcommunityoftheongoingeffortsoftheastrophysics,heliophysics,andplanetarysciencesdisciplinedatasystems/archivestosupportsysteminteroperabilityanddataexchange.
• Withoutpublicawarenessandsupport,itwillbedifficulttohaveoureffortsfunded• Valuetothecommunityneedstobedemonstratedbyouractionsandcommunicatedtoourpeersthroughthecoordinatedeffortsofourmembers
21
SolarSystemInterestGroup(SSIG)oftheIVOAReferenceFrames
• I’dliketostronglyendorseaddingsupportforcommonplanetaryreferencesframes• Keyissuesarenamingconventionsandthecommunicationofdefinitions
• Currentlywithintheplanetarycommunitytherearemultiplenamesforthesameframe• Therearealsoframesthatappeartobethesamebuthavesubtledifferencesthatarenotwellcommunicatedinthenamingconventionsand/ordescriptions
• NAIF/SPICEiscurrentlyintheprocessofformallydefiningandnamingawiderangeofdynamicplanetary/satellitereferenceframesthathavebeenpreviouslyusedinarchivingordefinedintheliterature.
• Thiseffortimposesauniformityinnamingconventions• Formalmathematicalframedefinitions• Documentationoftheoriginalframereferencesandnames(i.e.aliaslistsformanyframes)
• IftheNAIF/SPICEframenames/IDsbecomerecognizedastheproperwaytouniquelyspecifyplanetaryframes,andtheyareadoptedbythearchivesandrecordedinthemetadata,thenwemayhaveapathforwardfortheexchangeofframeinformationbetweendatasystems
• I’mnotsuggestingthattheIVOAcommunityadoptoruseSPICE,onlythatweconsideringleveragingtheSPICEefforttoformalizeframenamingconventions
22
Summary• PDSisalongstandingNASAorganizationresponsiblefordataarchive• PDS4isanewinformationmodeldrivenapproachtomodernizingthearchivestandardsusedwithintheplanetarysciencecommunity
• PDS4isbeingdevelopedwithinternationalparticipation• ThroughtheIPDA,memberorganizationscanmakestandardschangerequeststothePDS4datamodelandhaveavotingmemberonthechangecontrolreviewboard
• Designedtosupportinteroperabilitybetweendistributedplanetarydataarchivesystems
• IPDAisaninternationalorganizationofplanetarydataarchiveswithmembersfromatleasttwelvecountries
• IPDApromotestheadoptionofcommonstandards,cross-systemsearchandretrieval,andthesharingofsoftware,tools,andmethodsofdataexchange
• TheSSIGisafairlyneworganizationwithintheIVOAthegoalofworkingwithotherIVOAgroupstoreview,assess,andproposelow-impactadjustmentstotheIVOAstandardstosupportsolarsystemsciences
• Standardizinglistsofcoordinatesystemsandreferenceframesisoneofseveralareaswherethisgroupcanhaveapositiveimpact 23