Enterprise Service Bus Lowering the cost of integration Date: 9/1/2009 Dan McCreary President Dan McCreary & Associates [email protected] (952) 931-9198

Enterprise Service BusLowering the cost of integration

Date: 9/1/2009

Dan McCrearyPresidentDan McCreary & [email protected](952) 931-9198

M

D

Metadata Solutions

Copyright 2008 Dan McCreary & Associates 2

M

D

Agenda/Objective

• What is an ESB?• What is enterprise messaging?• Calculating the ROI on enterprise service bus


M

D

Why Build an ESB?

• Three primary business benefits– Precision– Consistency– Agility

Precision Consistency Agility


M

D

Precision

• Requirements reference Business Capabilities and are defined in business approved terms and data elements.

• Business terms and data elements are stored in the metadata registry

• Metrics and business rules reference data elements that are in the registry

• Shared processes and services use data elements that are in the registry

Precision


M

D

Consistency

• There is a common compensation processing model and each system/product maps into that common model.

• The processing model references Thrivent Business Capabilities, business terms, data elements, rules and metrics that are in the registry.

• Everyone (architects, analysts, developers, testers) use the same definitions for business terms, data elements, rules and metrics – all stored in the registry.

Consistency


M

D

Agility

• A single change to elements in the registry allows all models to be updated

• A single change to models can more seamlessly be reflected in updates to applications and services

• Changing and testing the models, then transforming the models into executable code allows faster response to change, and higher quality of delivered changes.

Agility


M

D

Quality Metrics for an ESB

Flexibility

Consistency

Maintainability

Utility

Products

StandardizedCalculations

StandardizedInformation

Components

Interfaces

Orchestration

Rules

Plan Components

Rules and Policies

Participants

ATAMOutput


M

D

The Challenge

• N applications that need to communicate with each other

• 1st application needs N-1 connections

• 2nd application needs N-2 connections

• 100th system needs 99 new adapters created

• O(N*N)• 100 systems = O(10,000

interfaces)

A1

A2

A3

A4

A5A6

A7

AN


M

D

The Solution

• Build a central EAI information bus that has “adapters” to each of the applications

• O(N) adapters

A1

A2

A3

A4

A5A6

A7

AN

EAIBus


M

D

The Challenge

• How to define an Adapter for each system so that other applications can “subscribe” to its data and process services

• Make the transport reliable

Application

EAI Bus

Adapter

MessagingProtocol


M

D

Incoming!

XMLHas this webthing gone away yet?

web


M

D

Publish/Subscribe Securely

Subscriber 1

Publisher 1

Enterprise Service Bus

Publisher 2 Publisher 3

Subscriber 2 Subscriber 3

MetadataRegistry


M

D

3 EAI Architectures

Presentation

Database

MethodMethod

Presentation

Database

Presentation

Method

Data


M

D

The Heart of the Enterprise

The Metadata Registry

A metadata registry is a central location in an organization where metadata definitions are stored and maintained in a controlled method.

http://en.wikipedia.org/wiki/Metadata_registry


M

D

Quality

• How do you measure it?– Quality Metrics

• How can you put in “benchmarks” to make sure it is improving?


M

D

Reuse Rule #1

If you don’t know about it

…you can’t reuse it

• If we have any asset, how does a person know it exists?• What are the corporate repositories of reusable assts?• How easy is it to use and be notified if an asset changes?


M

D

Metadata Publishing Workflow Funnel

• Develop a simple workflow system for publishing data elements

• Include harvesting areas of simple glossary-of-terms found in documentation, web sites and by using metadata “scrapers” to inventory all columns in relational database systems

• Get stakeholder teams to “accept” a data elements, review them and take on the data stewardship role for these data elements

• Commit to change-control only after data elements are marked “approved for publication” by over 50% of the stewardship team

UnderReview

Approved forPublication

GlossaryOf Terms

InitialDraft

Metadata H

arvesters


M

D

The Garden Metaphor

• Beautiful gardens don’t just grow by themselves• They need to be designed, cultivated and

maintained• Getting out the weeds is a key part of all beautiful

gardens (removing duplicate elements)


M

D

Data Stewardship

• An organization needs to believe that data and the management of enterprise data is a strategic goal of the enterprise

• At it core the ongoing data stewardship process in not a function of IT, it is an enterprise value instilled in each business unit

• We need to find data steward champions in each business unit• IT can facilitate the maintenance of data but the business unit needs to

be empowered

I Love Precise Data Definitions

I Love Precise Data Definitions


M

D

Federated Search

• Search many sources using a single tool


M

D

Search ResultsInternal Data Standards External Data Standards


M

D

Rules

Process

Data

Rules

Can they be separated?


M

D

Rules and Data

Rules should always be expressed in terms that are consistent with corporate metadata standards

–Tae Sondreal, Business Rules Expert

But how do we show this in our POC?


M

D

Assume a Future SOA ESB

• Each system has adapters that send and receive data in standardized XML formats governed by an enterprise metadata registry

Adapter Adapter Adapter Adapter




M

D

What are SOA-Centric Rules?

• Rule are used by any business process that executes conditional logic on a fragment of XML data on the Enterprise Service Bus

XMLFile

Rule Rule Rule

Exception Exception Exception

Happy Path

BusinessEvent


M

D

Data Type Rules

A rule that only looks at a leaf on a and XML tree

• Example:– Is -1.O3 a valid number?– Is “2008-28-29” a valid date?

• Rules do not depend on any resources outside of that node

• Executes quickly using streaming memory

• “Leaf-level rules”• Not applied to branches• Use w3c XML Schema datatype

standards• Most code tables under 1,000 values• Patterns validation using regular

expressions• Enterprise SimpleTypes are stored in

the metadata registry with the data element definition

scope

See w3c XML Schema datatypes


M

D

Structural Rules

• Rules that express the structure of a document

• What branches are valid?• What branches are required and

how many repeating values (cardinality)?

• Can be expressed by XML Schema constraints

• Output is a valid/invalid Boolean value for the entire document

• Executed in a single pass even for gigabyte files

• No need to keep the entire document in working memory

XMLSchema.xsd


M

D

Transformation Rules

• Rules that change an XML document either by adding or changing structure

• Rules split up documents or combine documents

• Expressed as XQuery or XSLT

• Stored in transformation libraries that promote reuse

• Classified by the Enterprise Integration Patterns (Content Enrichment, Content Filter etc.)

A B C

C = A + B


M

D

Conditional Rules

• Rules that can be take every node in the tree into account

• Expressed as a Boolean (true/false) outcome

• Used to redirect a document in a workflow

• Expressed as XPath expressions using Schematron rules file and compiled into an XSLT file

• XForms application checks for valid data element definitions

A B C

If C does not equal A + B then reportan error.

Ruleregistry

Metadataregistry


M

D

Service Rules

• Rules that require sending a subset of the XML data to an external web service

• Depends on an enterprise resource on a remote server

• Required consideration of unreliable networks and services

WebService

database


M

D

Classification of Rules in the SOA Realm

• Simple Business Rules– Data type rules (XML Schema data types)– Structural rules (XML Schemas, ER relationships)– Done quickly using streaming technology (SAX)

• Medium Business Rules– Transformation and Conditional rules– Stated as XSLT or if-then-else expressions– Assume and in-memory model (DOM)

• Service Rules– Any check that requires data not easily stored in a simple XML

Schema or a rules files– Address validation– CIF lookup– Legacy services require web service adapters


M

D

Rules in the Registry

• We have datatype, structural and services in the registry but we do not yet have a registry of transformation and conditional rules

MetadataRegistry

Property DataElements

WantlistsXML Schema

Version Registry

SimpleTypesSubset

Generation

ServiceRegistry


M

D

Rule Registry

• The Rule Registry would contain rule sets• Each rule set would be controlled by a group• Rule sets contain many Rules• Each Rule contains many And (Or) expressions that are either of type assertion or

report• Expressions my only reference registered data elements or literal values

RuleRegistry

RuleSet

RuleAnd (Or)

Expression

contains many

contains many

contains many

reference many

DataElement

MetadataRegistry

registered in theGroup

Controlled by

assertions reportliteral


M

D

Scope of Proof-of-Concept

• We can do an architectural proof-of-concept today as long as we don’t have to deal with network reliability issues right away

• Network reliability issues need to be considered for production systems

• Can be simulated with “wrapper” web services


M

D

Rule as a Service

• Rule Engines can be implemented as services that hangs off of the ESB• They have XML/REST interfaces• The take XML data in and send XML data out• They can live within the eXist XQuery engine or DB2 v9 or MS-SQL 2005 or Oracle 11g• XML Schemas define their inputs and outputs


Adapter

Rule EngineAdapter


xml xml


M

D

Distributed Computing Issues

• What if a remote web service is down?

• Request will time out after timeout interval (by default – 15 seconds)

• Ideally we would have a “watcher” that would watch over these calls and keep trying until the remote service returns

• How can this be added to the XQuery environment?


M

D

Reliable Web Services Issues

• When you call any service on a remote computer it gets transparently put in a local message queue

• The XQuery waits till the result returns

• People are notified when messages are not returned with an acceptable timeframe

• Example WS-Reliable Messaging


M

D

What is Orchestration?

• The ability for any web service to call other web services

WebService

Call Web Service X Call Web Service Y

Call Web Service X

Request

ResponseAlso known as a “composite”Web service


M

D

XQuery

• XQuery is inherently a choreography language

xquery version "1.0";

let $y := doc(‘http://www.example1.com/service-y.xq’)/my-xpath-into-y

let $z := doc(‘http://www.example2.com/service-z.xq’)/my-xpath-into-z

return

<results>

{$y}

{$z}

</results>

x.xq


M

D

Strategy

• Continue to build an infrastructure that is consistent with the metadata registry– Selection list are context sensitive

• Use simple XML-centric standards (like XPath)– Rules expressed in terms of metadata

• Leverage “as-you-type” user interfaces (see suggest demo)


M

D

Thank You!

Please contact me for more information:• Native XML Databases• Metadata Management• Metadata Registries• Service Oriented Architectures• Business Intelligence and Data Warehouse• Semantic Web

Dan McCreary, PresidentDan McCreary & Associates

Metadata Strategy [email protected]

(952) 931-9198

Documents

Enterprise Service Bus Lowering the cost of integration Date: 9/1/2009 Dan McCreary President Dan McCreary & Associates [email protected] (952) 931-9198