Upload
aure
View
26
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Achieving Information Sharing in Federal Agencies via Rapid Data Services Enablement and SOA. MetaMatrix and the SOA CoP Demo. Chuck Mosher & Tony Vachino MetaMatrix October 31, 2006. Agenda. Data Services Rationale & Best Practices MetaMatrix Products & Capabilities - PowerPoint PPT Presentation
Citation preview
Achieving Information Sharing in Federal Agencies via Rapid Data Services Enablement and SOA
Chuck Mosher & Tony Vachino
MetaMatrix
October 31, 2006
MetaMatrix and the SOA CoP Demo
2
Agenda
• Data Services Rationale & Best Practices
• MetaMatrix Products & Capabilities
• Achieving Information Sharing– Service Enabling Data Assets– Resolving Semantics
• Demo
• Summary/Q & A
3
Handling The Data Challenge
Resolving data semantic and structural mismatches
SOA-enabling legacy data systems (i.e., Net Centricity)
Mapping data sources to vocabularies like XDM, NIEM, C2IEDM, HL7, …
Handling multi-source requests (data aggregation, mediation, fusion, federation)
Minimizing development and maintenance cost of custom code by metadata-based MDA
MetaMatrix can Help
Getting the right information to the right person at the right time requires:
4
Program Challenges• Multiple sources
• Different interfaces/drivers• Different physical structures• Different semantics
• Single interface to data desired• Real-time access to data• Performance• Maintainability as data changes• Maintainability as apps change
Mission Challenges• Time-to-deploy• Agility - Responsiveness to change• Automation – Reduce cost of new development and operations• ROI of enterprise information
Agency Challenges• 100’s/1000’s of data sources• 100’s/1000’s of applications• Multiple access points/modes for apps• Understanding relationships/semantics• Data consistency• Data reuse – bridging data silos• Support for Web Services & SQL• Control & manageability, compliance• Security & auditing
Information Resources
Communities of Interest
Information Challenges
?
5
Information Virtualization
Unified Semantic Layer
Information Virtualization Layer
Data Federation Layer
Data Access/Connectivity Layer
Enterprise Data Sources
Unification of different concepts across systemsSingle-query access to heterogeneous systemsUniform, standardized access to any system
6
Facilities ImagesUnitsSensors
Persons ODS Parts Personnel Documents
Information Services
MetaMatrix Information Integration Platform
Information on demand
High performance
Minimal replication
Manageable
Secure
Data Interoperability Through Information Services
C2
Intel
Weapon Systems
Dynamically Created COIs
LogisticsSensors
Personnel
Finance
Etc.
7
What is a Data Service?
MasterData
OperationalData Store
AgencyApplication
Data Service
SQL SQL APICall
XML/SOAP
• Decouple data sources from application– Data implementation shielded
from application• Semantic/Format Mediation
– Standard vocabulary • Single access point
– Web Service/XML– SQL
• Federation– Single source or multi-source
• Scalability– Security, performance
Bridge theGap
SQL
8
Data Services: Designed for Agility
• Data Services Best Practices– Provide transparency across all sources
– Define known relationships today and accommodate future relationships
– Support independence of mission systems
– Support ownership of operational data sources at the source
– Provide accelerated mechanisms for integrating new sources
– Support existing security policy and add degrees of security
• The value of a managed metadata abstraction layer– "Future Proofing" (future standards, exchange models, platforms)
– Limited skill set requirements
– Fixed long term costs for integration middleware
• Building consensus– Assure data owners they will continue to have control, and …
– Vocabulary of existing production systems will not be impacted
– Offer an option where legacy data migration is not 'required' 1st
9
Data,ContentSources
Logical Data Model
Data Services Approaches
T
Org, Person, Image,
Location
MaterializedLogical Model
<X>
</X>
<X>
</X>
<X>
<X>
<X>
</X>
<X>
</X>
<X>
<X>
Data Services for Multiple Purposes:
• Simplified access to value-added (tagged) data in real-time• Value-added (tagged) data materialized & staged
• Phased-in migration from legacy to new• Managed archiving via classification, retention tags
• Enhanced search via consistent content tags
Model-Driven Integration LayerModel-Driven Integration Layer
Data,ContentSources
Logical Data ModelT
Organization, Customer, Imagery, Location
MaterializedLogical Model
<X>
</X>
<X>
</X>
<X>
<X>
<X>
</X>
<X>
</X>
<X>
<X>
AgileInformation
Services
<X>
</X>
<X>
</X>
<X><X>
<X>
</X>
<X>
</X>
<X><X>
<X>
</X>
<X>
</X>
<X><X>
Enriched Data/Content Store
10
Search Engine Index / Metadata Catalog
Master Data Person / Facility / Vehicle
Enterprise Data Services
Stage SOAApp’s
Federal Agencies
Data Access Services• SQL, Web Service/XML• Staged Data (optional)
OntologyMgmt /Reasoning
Enterprise Service Bus / Intranet / Extranet
Distributed Data Services
Land/Sea
State/Local• Security/Authentication• Operations Management • Error / Exception Management
• Orchestration• Encryption• High Availability
MediationXSLT, Multi-source
Information Exchange Topology
11
Agenda
• Data Services Rationale & Best Practices
• MetaMatrix Products & Capabilities
• Achieving Information Sharing– Service Enabling Data Assets– Resolving Semantics
• Demo
• Summary/Q & A
12
MetaMatrix I.P.
MetaMatrix has 2 distinct innovations that work in concert to yield significant business benefits:
Model-basedModel-based
ExtensibleExtensible
Sharable, reusableSharable, reusable
Standards-basedStandards-based
Information ModelingInformation Modeling
Federated QueryingFederated Querying
Cost-based optimizerCost-based optimizer
Read/write/transactionsRead/write/transactions
Uniform API, any sourceUniform API, any source
Battle-tested/hardenedBattle-tested/hardened
13
Designing data services
MetaMatrix Approach to Data Services
xml
databases
warehouses
spreadsheets
services
<sale/> <value/></ sale >
geo-spatial
rich media
…Enterprise Enterprise Information Information
Sources (EIS)Sources (EIS)
Information Information ConsumersConsumers
Reusable,Reusable,Integrated Data Integrated Data
ObjectsObjects
ExposedExposedDataData
ServicesServices
<WSDL><WSDL>(contract)
<WSDL><WSDL>(contract)
<WSDL><WSDL>(contract)
Custom Apps
Web Services,Business Processes
Packaged Apps
Reporting, Analytics
EAI, Data warehouses
OD
BC
JDB
CS
OA
P
Logistics
Intelligence
14
• Transformations from one or more sources
• Transformations defined with:– Joins/unions– Criteria– Functions
• Elements mapped to dictionary
• Business definitions captured
Data Service Abstraction Layers
15
MetaMatrix Integration Server
Information Consumers
Web Svc XML RDBMSPackaged Connectors
Siebel,SAP
OracleApps
CICSVSAM
MetaMatrix Catalog
MetaMatrix Designer- Design and deploy data services
MetaMatrix Products
JMSODBC JDBC SOAP
QueryProcessor
ProcessorProcessorOptimizerOptimizer
Integration ServerVirtualDataBases
VDBVDBVDBVDB
IntegratedSecurity
UsersUsers
RolesRoles
EntitlementsEntitlements
AccessModels
ViewsViews XMLDocsXMLDocs
<a>
</a>
<b>
</b>…
ServicesServices
in outproc
MetaMatrix Connector Framework
MetaMatrixServer
16
Secure Access – Accredited
MetaMatrixClient AppClient Appusernamepassword
Membership Provider
Membership Provider
usernamepassword
authenticates
Connector
Connector
Connector
Connector
Data Source
Optionally accessessource-specific information
source-specific
trustedpayload
MetaMatrixClient AppClient App
Membership Provider
Membership Provider
Authentication Service
Authentication Service
logoninfo
authenticates,generates payload
trustedpayload payload
trustedpayload
authenticates,optionally modifies payload
payload
Username/Password Logon • Connector connects with same ID for all queries• Optional: Integrated with existing authentication system
Trusted Payload Logon:• Connector uses different credentials per connection, per query • Optional: Integrated with existing authentication system
Data Source
17
Process X
Process Y
Processes[BPM/BPEL]
Ontologies[OWL/RDF]Taxonomies
ServiceA
ServiceB
Web Services [WSDL]
Classification Schemes
Taxonomy A
KeyWords B
Relational
XMLXML
XML
XMLTransformations
DatatypesXMLRel
Rel
Domain[UML/ER]
MetaMatrixDesigner
MetaMatrixCatalog
GenericTypedRelationships
Models & Files[versioned]
Models & Files[versioned]
Search Index
Search Index
Web Reporting
Web Reporting<X>
</X>
<X>
</X>
…
<X>
</X>
<X>
</X>
…
WSDL
Public Health
Justice
Environment
Geo-spatial
Recreation
Immunization
Warrant
Wildlife
…
Camping
Public Health
Justice
Environment
Geo-spatial
Recreation
Immunization
Warrant
Wildlife
…
Camping
Public Health
Justice
Environment
Geo-spatial
Recreation
Immunization
Warrant
Wildlife
…
Camping
Public Health
Justice
Environment
Geo-spatial
Recreation
Immunization
Warrant
Wildlife
…
Camping
Public Health
Justice
Environment
Geo-spatial
Recreation
Immunization
Warrant
Wildlife
…
Camping
Public Health
Justice
Environment
Geo-spatial
Recreation
Immunization
Warrant
Wildlife
…
Camping
Application/Configuration
Managing Data Service Metadata
18
MetaMatrixEnterprise
MetaMatrixDimension
MetaMatrixQuery
MetaMatrix Product Lines
MetaMatrix Enterprise • Web services & SQL• Modeling enterprise data• Scalable deployment server• Metadata management• Application/legacy connectors
MetaMatrix Dimension • Web service-enablement of data sources• Expose business views as XML• Lightweight modeling – rapid integration• Standard WAR-based deployment
MetaMatrix Query • Embeddable Java component • Federated query engine• Query optimization• Standard JDBC to all sources• Standard SQL to all sources
En
terp
rise
Pro
ject
, No
de
ISV
/ P
roje
ct
19
Agenda
• Data Services Rationale & Best Practices
• MetaMatrix Products & Capabilities
• Achieving Information Sharing– Service Enabling Data Assets– Resolving Semantics
• Demo
• Summary/Q & A
20
T
«Text File»
«Relational»
«Application»MetaMatrix:Mapping from Data to XML
Source: Data Sources containingInformation to integrate
Target: Fixed (potentially complex) XML SchemaNeed:Data complying to Schema
Mediation: XML From Non-XML Sources
«XML»
<person> <addresses> … </addresses> <accounts> <accountID=…> … </accountID> </accounts></person>
21
• Model XML Docs, Schemas
• Build XML Doc. models from XML Schemas
• Map XML Doc. models to other data models
• Enable data access via XML
Map Data Sources to XML & Deploy
MetaMatrix Designer – for XML-centric Data Services
22
Dimension – Choose your approach
• Rapid design & deployment of Web Services• Expose integrated data as XML-based business views• Deployment of Web Services as standard Web apps• Runtime execution optimized through use of MetaMatrix Query Engine
Dimension Models
Web Server
Data Sources
Business Views
<XML><XML><XML>
Web Service Operations
WSDLXSD
Source Models
DeployImport Map Model
WARastoto
Start Here?
Start Here?
23
Agenda
• Data Services Rationale & Best Practices
• MetaMatrix Products & Capabilities
• Achieving Information Sharing– Service Enabling Data Assets– Resolving Semantics
• Demo
• Summary/Q & A
24
T
Authoritative Sources:• Mapped to logical
Multiple Internal/External Information Sources
Application views of information:
• Relational, XML
T T
XML Document<a>
</a>
<b>
</b>…
T
TT
ODBC/JDBC JDBC SOAP
WebServices
WebServices
Search Applications
Search Applications
BusinessIntelligence
Applications
BusinessIntelligence
Applications
Logical Data Model:• Agency or COI-specific• Rationalize, harmonize,
mediate
C2, Logistics, Intelligence, …
COI Data Dictionary
bldg_id SITENUM Facility_ID
Location_ID
bldg_type Depot_Number
Location_Type
25
FBI CBP NYC NY NJ
SemanticData Services
Matched (Confidence of 90%)
Gender ID
Person Sex Code
Ontology
“Sex” semantically related to “Gender”
Semantic Matching - example
Data Sources
Semantic Data Services– key component of information sharing
and interoperability programs – automated semantic mapping to aid
domain experts in quickly reconciling disparate schemas and vocabularies
– more rapid deployment of a mediation solution
MatchIt – an extensible ontology-driven tool– variety of algorithms for determining
semantic equivalence– discovers similarities between
elements of heterogeneous data, automatically exposing potential semantic matches.
– matches elements of data sources to target schemas of Data Services, such as TWPDES, GJXDM, NIEM, C2IEDM, HL7
26
Automated Term Discovery (Interpret)
A comprehensive list of terms automatically discovered across all sources
All the available definitions found in the MatchIT knowledge-base
All the usage instances where each term was used in any of the sources
Results of the automated
tokenization
27
Contextualize (Interpret)
Automated term tokenization
Automated semantic linking using the default knowledge-base contained within MatchIT
ArticleAmount
Amount Article
Sum
Assets
Creation
Synonym
Type-of
28
Semantic Matching (Mediate)
• With relationships pre-established within the knowledge-base…
• Identify the Target and the Source(s) and run the match.
ArticleAmount
ProductShares
Automatically linked by a specific % distance
29
Facilitate Decision Making (Mediate)
Helps facilitate rapid decision making
Target element for matching
Automatically calculated semantic distance between terms
Source candidate for matching
30
J-8 Force Structure
J-7 Operational Plans
J-6 C4CS
TData Sources- Authoritative- Redundant
- Overlapping
Multiple Internal/External Information Sources
T T
ODBC/JDBC JDBC SOAP
WebServices
WebServices
Portal Applications
Portal Applications
BusinessIntelligence
Applications
BusinessIntelligence
Applications
Enterprise-wide or COI-driven Data Models
• Rationalization• Harmonization• Data Catalogs (DDMS)
Support Multiple Enterprise Semantic Models
J-5 Plans & Policy
J-4 Logistics (GCSS)
J-3 Operations
J-2 Intelligence
J-1 Manpower / Personnel
31
Agenda
• Data Services Rationale & Best Practices
• MetaMatrix Products & Capabilities
• Achieving Information Sharing– Service Enabling Data Assets– Resolving Semantics
• Demo
• Summary/Q & A
32
Agenda
• Data Services Rationale & Best Practices
• MetaMatrix Products & Capabilities
• Achieving Information Sharing– Service Enabling Data Assets– Resolving Semantics
• Demo
• Summary/Q & A
33
MetaMatrix – Quick Facts
Middle-ware, model-driven, data management
DoD proven (DISA, NSA, TRANSCOM, etc.)
Version 5 – Mature product which is still unique and ahead of the competition
NIAP certified and NSA-credentialed
Can handle the enterprise (or COI) perspective as well as the bottom-up perspective (data service enablement of legacy systems)
Can rapidly implement data integration strategies
34
Major US Federal Government Customers
NSA - Multiple Programs (NES Base-lined) In-Q-Tel/CIA TRANSCOM – Command Metadata
Management System Air Force - Command and Control Center DISA - Global Combat Support Systems
(GCSS) DISA – Anti Drug Network (ADNET) DLA – Integrated Data Environment (IDE) Mitre – Air Force ESC/DoD DDMS work UK – NSA Equivalent, CJIT
35
• On-demand information– Real time data integration– Information sharing between business units
• Enabling SOA in an evolving world– Consume and produce Web services– And still provide full support for ODBC, JDBC, and legacy
• Federation of disparate information– Rationalized to controlled vocabularies– Relational + XML + Web Services + Enterprise Apps + Legacy
• Faster time to market– Integrated information in days, weeks– Tight coupling of design & implementation phases– Leveraging the skill-set of the data architects for integration
• Costs across application lifecycle reduced– Model-driven abstraction layer eases development/maintenance– Better management of data assets across the enterprise
MetaMatrix Value PropositionMetaMatrix Value PropositionHighly cost-effective COTS tool for rapid enterprise
information integration and exchange