A Partnership for the Next Generation of ESE Web Services Development Best Practices March 16, 2010 Presenters: Min-Gu Lee, Scott Campbell National Computer

A Partnership for the Next Generation of ESE

Web Services Development Best Practices

Web Services Development Best Practices

March 16, 2010Presenters: Min-Gu Lee, Scott Campbell

National Computer Center Application Hosting Web

Conferences

slide 2

Copyright Lockheed Martin

04/18/23

Agenda

Introduction Web Services Architecture An Inventory of EPA Web Services Web Services Development Web Services Deployment Example of Web Services in an EPA Application Questions

Introduction: Web Services Definitions

slide 3

• A software system designed to support interoperable machine-to-machine interaction over a network. - W3C

• Web Services provide an interface for a Service-Oriented Architecture (SOA), in which Web-based applications dynamically interact with other Web applications using open standards that include Extensible Markup Language (XML) running over Hypertext Transfer Protocol (HTTP), Universal Description, Discovery, and Integration (UDDI) , and Apache Simple Object Access Protocol (SOAP). - State of Utah, Dept. of Tech. Services

• Web services are automated information services that are conducted over the Internet, using standardized technologies and formats/protocols that simplify the exchange and integration of large amounts of data over the Internet. - EPA, Environmental Information Exchange Network

04/18/23

Introduction: Benefits

slide 4

Application and data integration (interoperability)

Code Reusability

Cost Savings

04/18/23

General Web Service Architecture

slide 504/18/23

Web Services in Terms of Roles• Service Provider• Service Requestor

Types of Web Services• SOAP-based• REpresentational State Transfer (RESTful): using HTTP

methods (PUT, GET, DELETE, POST) Web services in Terms of Protocol Stack

• Service Transport (including HTTP, Simple Mail Transfer Protocol (SMTP), File Transfer Protocol (FTP)

• XML Messaging (XML-RPC, SOAP)• Service Description (Web Services Description Language

(WSDL))• Service Discovery (UDDI)

Conceptual Web Service / SOA Architecture

slide 604/18/23


slide 704/18/23

Web Services• Business Web Services

Business functions that are exposed as Web services

• Utility Web Services Web services that provide general helper utilities for

applications

Registry and Repository• Registry Supports the publication, classification, discovery, and

testing of enterprise Web Services

• Repository Stores the definitions and metadata of enterprise services and

provides an integrated modeling environment for defining enterprise services


slide 804/18/23

Enterprise Service Bus (ESB)

• An architectural pattern for implementing a standard based functionality for transporting, routing, mediating, and orchestrating Extensible Markup Language (XML) messages in a secure manner

Federated Enterprise Service Bus

• Multiple ESB domains working together to form a single logical ESB infrastructure

• Addresses service governance, security and management concerns


slide 904/18/23

Business Process Modeling (BPM)• BPEL (Business Process Execution Language) – A standard executable language for specifying

interactions with Web Services

• BPMN (Business Process Modeling Notation) – A notation that specifies business process in a workflow

and is understandable by all users.

Monitoring• BAM (Business Activity Monitoring) – An enterprise solution offering real-time summary of

business activity to managers and partners

• EM (Enterprise Monitoring) – Enterprise-wide monitoring of real-time operations and

activities

An Inventory of EPA Web Services ….

slide 1004/18/23

slide 11

Examples of EPA Web Services by Access Level

EPA Extranet Services

(through CDX)

EPA Internal Services

EPA Public Services

Open Services by Vendors

44 data flows currently receive data from EPA, States, Tribes, and Industry.

Refer to http://epa.gov/cdx/projects/production.htm for a complete list

Enterprise Content Management System (ECMS) Web services

Web Access Management (WAM) Web services

Oracle Collaboration (OCS) Web Services

Taxonomy Services

UVIndex STOrage and

RETrieval (STORET) Services

Microsoft Bing Map Web Services

Google Earth Service

04/18/23

EPA Extranet Services via CDX ….

slide 1204/18/23

Central Data Exchange (CDX)

slide 13

Point of entry on the Environmental Information Exchange Network (Exchange Network) for environmental data submissions to the Agency

Partner organizations include:• EPA Program Offices• State and Local governments• Industry• Tribes

Cross-Media Electronic Reporting Regulation (CROMERR)• Provides legal framework for electronic reporting to the EPA• Reports submitted electronically to EPA must be submitted to CDX

CDX offers a full set of services including:• Authentication and authorization• Submission• Validation• Data archiving• User support

04/18/23

Examples of EPA Systems Using CDX Web Services

slide 14

Air Quality System (AQS) eBeaches Electronic Inventory Update Rule (eIUR) Facility Registry System Grants.gov / Integrated Grants Management System

(IGMS) National Emissions Inventory (NEI) Safe Drinking Water Information System (SDWIS) Substance Registry System Toxics Release Inventory (TRI) State Data Exchange Water Quality Exchange (WQX)

04/18/23

http://www.epa.gov/cdx/projects/production.htm

The complete list of the CDX Production Data Flows can be found at:

Web Services InventoryCDX (Extranet)

slide 15

Web Service Group

Methods Function

Network Security Services

Authenticate Provides the capability for the node or user to authenticate with the Network Authentication and Authorization Server (NAAS) and use the obtained Security Token to access services on the Network including data exchanges.

Authorization Determines if the service requestor is entitled to perform the operation, which canrange from invoking the Web service to executing a certain part of its functionality (executing a stored procedure, for instance, using the Query method).

Source: CDX Web Services Fact Sheet, http://www.epa.gov/cdx/about/webservicesfactsheet.pdf

04/18/23

Web Services Inventory CDX (Extranet)

slide 16

Web Service Group

Methods Function

Database Services

Query Provides the capability to perform a database query and return the result in XML format tothe requestor. Queries are associated with specific schema/data flows supported by the Network.

Solicit Provides the capability to perform an asynchronous database query and return the result in XML format to the requestor.

Document Exchange Services

Submit Provides the capability to send one or multiple payloads (Documents) to the serviceprovider.

Download Provides the capability to retrieve one or multiple payloads from the service provider.


04/18/23

Web Services InventoryCDX (Extranet)

slide 17

Web Service Group

Methods Function

Transaction Status Services

Get Status Each data exchange service is associated with a unique transaction. This WebService provides the capability of querying the current state of the transaction.

Notification Services

Notify Enables the requestor to notify a node about:• Document availability• Status, such as current status of a submission or service request• Events such as security alerts, shutdown notices, and other network management notes.

Administration Services

Node Ping Provides information about the accessibility of the node.

Get Services Allows requesters to query services provided by a Network node.

04/18/23


Web Services InventoryWQX via CDX (Extranet)

slide 18

Web Service Group

Web Services Function

WQX Data Services (Query/Solicit)

WQX.GetActivityByParameters_v2.0 Returns a collection of Activities and a count of the number of Results on each Activity.

WQX.GetActivityGroupByParameters_v2.0 Returns a collection of Activity Groups.

WQX.GetBiologicalHabitatIndexByParameters_v2.0

Returns a collection of Biological or Habitat Indices.

WQX.GetDomainValueByElementName_v2.0

Returns a list of domain values for the elementName passed in.

04/18/23

Source: Water Quality Exchange, Flow Configuration Document, http://www.exchangenetwork.net/schema/WQX/2/WQX_FCD_v2.0.pdf

Web Services InventoryWQX via CDX (Extranet)

slide 19

Web Service Group

Web Services Function

WQX Data Services (Query/Solicit)

WQX.GetMonitoringLocationByParameters_v2.0

Returns a collection of Monitoring Locations.

WQX.GetProjectByParameters_v2.0 Returns a collection of Projects for the organization passed in.

WQX.GetResultByParameters_v2.0 Returns a collection of Activities and Results.

WQX.GetTransactionHistoryByParameters_v2.0

Returns a summary of the transactions processed by the WQX System.

04/18/23

Source: Water Quality Exchange, Flow Configuration Document, http://www.exchangenetwork.net/schema/WQX/2/WQX_FCD_v2.0.pdf

Web Services Inventory Substance Registry System via CDX (Extranet)

slide 20

Web Service Group

Methods Function

Data Processing Services (Submit)

ListSubstanceAddition Allows an authorized partner to add a substance that is already defined in SRS to a substance list that is also already defined in SRS.

Substance Retrieval Data Publishing (Query)

GetSubstanceByALTIDListAcronym

Allows partners to select substances based upon an alternate identifier for the substance and whether the substance is a member of a specified list within SRS.

GetSubstanceByCAS Allows partners to select substances based upon the Chemical Abstracts Service (CAS) registry number associated with the substance in SRS.

GetSubstanceByEPAID Allows partners to select substances based upon the EPA identifier associated with the substance in SRS.

GetSubstanceByInternalTrackingNumber

Allows partners to select substances based upon the SRS internal tracking number associated with the substance in SRS.

GetSubstanceByName Allows partners to select substances based upon a substance name for the substance. The substance name will match based upon a substance’s name, systematic name, or one of its synonym names.

04/18/23

Source: Substance Registry System, Flow Configuration Document, http://www.exchangenetwork.net/schema/SRS/3/SRS_FCD_v3%200.doc

Web Services Inventory Substance Registry System via CDX (Extranet)

slide 21

Web Service Group

Methods Function

Substance Retrieval Data Publishing (Query)

GetSubstanceByTSN Allows partners to select substances based upon the taxonomic serial number associated with the substance in SRS.

GetSubstanceMatchByName Allows partners to determine whether a substance exists within SRS based upon a substance name for the substance.

Substance Retrieval Data Publishing (Solicit)

GetSubstancesByCAS Allows partners to select substances based upon the CAS registry number associated with the substance in SRS.

GetSubstancesByCategoryClassifications Allows partners to select substances based upon the classifications associated with the substance in SRS.

GetSubstancesByCharacteristics Allows partners to select substances based upon the characteristics associated with the substance in SRS.

04/18/23


Web Services InventorySubstance Registry System via CDX (Extranet)

slide 22

Web Service Group

Methods Function

Substance Retrieval Data Publishing (Solicit)

GetSubstancesByEPAIDs Allows partners to select substances based upon the EPA identifier associated with the substance in SRS.

GetSubstancesByInternalTrackingNumber Allows partners to select substances based upon the SRS internal tracking number associated with the substance in SRS.

GetSubstancesByListAcronym Allows partners to select all substances that are a member of the specified list in SRS.

GetSubstancesByNamesExact Allows partners to select substances based upon a substance name for the substance.

GetSubstanceCASByListAcronym Allows partners to select only the CAS registry numbers of all substances that are a member of the specified list in SRS.

GetSubstanceListAcronymsByAcronym allows partners to determine substance list acronyms that exist in SRS.

04/18/23


Web Services InventoryFacility Registry System via CDX (Extranet)

slide 23

Web Service Group

Methods Function

FRS Data Services

GetFacilityCount_v3.0 (Query)

Partners may choose to implement this internally for costing purposes, so that over-large queries can be intercepted before they are run.

GetFacilityList_v3.0 (Query or Solicit)

Full set of parameters but returning on the lightest payload (FacilityIndex) to aid in performance.

GetFacility_v3.0 (Solicit) Full set of parameters, also including Change Date, and returning a payload based on the full schema.

GetFacilityInterest_v3.0 (Query)

Full set of parameters but returning a payload of summary facility and environmental interest data (FacilityInterest) rather than the full schema, to aid in performance.

04/18/23

Source: Facility Identification Flow Configuration Document, http://www.exchangenetwork.net/schema/FACID/3/FACID_FCD_v3.0_DRAFT.doc

Web Services InventoryFacility Registry System via CDX (Extranet)

slide 24

Web Service Group

Web Service Function

FRS Data Services

GetFacilityByID_v3.0 (Query)

Get detailed facility data for one facility.

GetFacilityByChangeDate_v3.0 (Solicit)

Used to support the creation and maintenance of a replica set of facility data across Partners (i.e., data synchronization).

GetDeletedFacilityByChangeDate_v3.0 (Solicit)

This uses the FacilityIndex schema to return of basic identification data about each facility that has been deleted. This would only be used if a Partner is maintaining a replica set of facility data.

04/18/23

Source: Facility Identification Flow Configuration Document, http://www.exchangenetwork.net/schema/FACID/3/FACID_FCD_v3.0_DRAFT.doc

EPA Internal Services ….

slide 2504/18/23

Web Services InventoryWeb Access Management (Internal)

slide 26

Web Service Group Methods Function

Authentication Manager Services

Authenticate, validate, getUser Provides user registration and resource access requests.

Self-Service Functions changePassword, forgotUser Allows users to manage account attributes (e.g., their own passwords) across managed resources.

Provisioning Service Functions

subscribeToGroup, unsubscribe from Group

Process of making applications ready for users to use once self-registration info has been entered.

Delegated Administration Service Functions

getSelfRegistrationPendingRequests, getSubscribeToGroupPendingRequests, approveSelfRegistrationPendingRequest, addGroupMember, removeGroupMember

Supports delegation of most administrative functions to any group and/or user.

04/18/23

Source: Web Access Management Framework Integration Guide, Version 1.2, https://badger.epa.gov/iamfw/

Web Services InventoryTaxonomy Services (Internal)

slide 27

Web Service Group

Methods Function

Synaptica Automated Services

Login Users may log in to obtain a session key. Valid session keys can then be passed to all Synaptica web methods as authentication with the exception of Login.

Logout This method expires a session key making it invalid. As a security measure, Logout should be called to release any session keys obtained by auser that will no longer be needed.

ListTaskViews70 This method returns a list of the taskview profiles a user has permission to. It has the exact same functionality as ListTaskviews, except for its input.

ListVocabularies70 This method returns a list of the vocabularies a user has permission to within a Taskview.

ImportTerms70 This method imports terms and relationships provided in valid Resource Description Framework / Simple Knowledge Organization System (RDF/SKOS) format as defined in "Synaptica_RDF_SKOS_specification_20061129.doc"

04/18/23Source: Synaptica Knowledge Management System, Synaptica Web Services API. 02/04/2008

Web Services InventoryTaxonomy Services (Internal)

slide 28

Web Service Group

Methods Function

Synaptica Automated Services

ExportVocabulary70

Export an RDF/SKOS representation of one or more vocabularies from Synaptica. This method can be used for all general extracts from Synaptica.

ValidateTerms70 The method receives a list of terms to be validated within Synaptica. For each term, a descriptor and vocabulary name is supplied.

ListRelatedTerms70

This method returns relationships for a given term. The method request can specify a single relationship type or all relationships.

GetSavedReport70

This method returns the contents of files saved using the scheduledreport functionality of the Synaptica Web Interface.

SearchTerms70 This method is used to search object classes for specific terms by various criteria exposed by the Alphabetical view report. This may be used to return entire object classes, or narrow results depending on the input parameters.

04/18/23

Source: Synaptica Knowledge Management System, Synaptica Web Services API. 02/04/2008

Web Services InventoryEnterprise Content Management Services (Internal)

slide 29

Web Service Group Methods Function

ERMAWebService GETSUMMARYDXLRequest Constructs summary XML header information and ECMS attributes.

GETDXLRequest Constructs IBM Domino Extended Markup Language (DXL).

GETMANUALDXLRequest Constructs Manual DXL.

GETDXLBASE64Request Processes e-mail attachments.

CONFIRMARCHIVEDRequest Creates lock icon after an e-mail is saved to E- mail Records Management Application (ERMA) repository.

Documentum 6.5 Foundation Web Services

Link to archive that contains EMC Documentum Foundation Services 6.5 SP1 Technical Publications:https://community.emc.com/servlet/JiveServlet/downloadBody/2820-102-5-10615/dfs_65sp1_pubs.zip;jsessionid=1FFB8433464B4BCAD6014CF374E80BD7.node0

04/18/23

EPA Public Services ….

slide 3004/18/23

Web Services InventorySTORET (Public)

slide 31

Web Service Methods Function

Watershed Summary Web Service

getOrganizationSummary Provides organization and characteristic summary information aggregated at the watershed level or aggregated for a given organization within the watershed.

getCharTypeCharacteristicSummary

getCharacteristicSummary

getOrganizationCharTypeSummary

getOrganizationCharTypeCharacterisiticSummary

getCharTypeSummary

getWebServiceCatalog

04/18/23

Source: STORET Web Services Consumption User Guide , http://www.epa.gov/storpubl/STORETWebServicesUserGuide2008_0205.pdf


slide 32


Station Catalog Web Service

getWatershedOrgStationCharTypeSummary Provides station summary information such as station name, period of record, and result counts by characteristic type.

getStationCharTypeSummaryUsingXMLInput

getStationCharacteristicSummary

getStationCharTypeSummaryUsingStringInput


getWatershedStationCharTypeSummary

04/18/23



slide 33


Project Catalog Web Service

getProjectCountByLatLong provides project summary information such as the period of record, characteristics monitored, and station names

getProjectInfoByParameters

getProjectCountByParameters

getProjectInfoByLatLong

04/18/23



slide 34


Station Web Service

getStationInfo Provides information for all monitoring locations within a given geographic area (geographic bounding box).

(Note: The Station Web Service has been revised (as of February 8, 2008) into multiple web methods that provide enhanced data return.)

getStationCharacteristicsForMap


getStationCount

getStationsForMap

04/18/23



slide 35


STORET Result Web Service

getResultCount Provides direct access to STORET results.

(Note: The Result Web Service has been revised (as of February 8, 2008) to provide for faster data retrieval. )

getActivityCount

getResults

04/18/23


Web Services InventoryUV Index (Public)

slide 36


UV Index Web Service

getUVIndexAlertByCityState Obtains the specified UV Index for the specified city and state.

getUVIndexAlertByZipCode Obtains the specified UV Index for the specified zip code.

04/18/23

Source: http://www.epa.gov/sunwise/uviresources.html#webservice

Web Services InventoryOpen Services from Vendors

slide 37

Microsoft Bing Map Services Google Earth Services

04/18/23

Web Services Development Process

slide 38

Define system requirements Search for existing services in EPA UDDI

Registry and EPA Reusable Component Services (RCS) EPA Pre-Production UDDI Registry (

https://uddi.epacdxnode.net) EPA Production UDDI Registry (

https://uddi.epa.gov/) Create an Interface Control Document

(ICD) and WSDL

04/18/23

Web Services Deployment Process

slide 39

Design / develop application using WSDL Test application, integrating with web

services Deploy at NCC Conduct EPA Application Deployment

Checklist (ADC) Deploy to NCC Staging environment Publish in EPA UDDI and update SoR

Reusable Component Services (RCS) Deploy application and web services to

NCC Production environment

04/18/23

Example: Use of WAM Web Services in an EPA Application

slide 40

Application: Emergency Management Portal (EMP)

User Community: Office of Emergency Management (OEM) and other authorized personnel

Application Profile: The EMP represents a single point of entry to site-specific information, asset management tools, and emergency management information

Problem: Upon authentication to EMP, determine the user’s access privileges to EMP functionality

Solution: WAMFW Web Service call to determine the group membership for a user and map the group to an internal EMP role

04/18/23

Example: Use of WAM Web Services in an EPA Application

slide 4104/18/23

Source: Emergency Management Portal Security Document – Role and Group Management, Version 1.2, 12/09/2009

Determination of EMP Equipment Module Groups and Roles

slide 4204/18/23

QUESTIONS ……