Upload
vaibhav-gupta
View
227
Download
0
Embed Size (px)
Citation preview
7/27/2019 BI Lecture Number 8
1/100
Enterprise Application Integration
Dr. N. P. SinghProfessorManagement Development InstituteMehrauli Road, SukhraliGurgaon -122001E-mail: [email protected]
7/27/2019 BI Lecture Number 8
2/100
Contents
Part-I
What is EAI?
7/27/2019 BI Lecture Number 8
3/100
Definitions
Enterprise Application Integration
Definition: The process of integrat ing mult ip leappl icat ions that were independently developed,
may use incompat ib le technology, and remain
independent ly managed.
By this definition, EAI would include: Business Process Integration
Enterprise Information Integration
7/27/2019 BI Lecture Number 8
4/100
Data Propagation
It is the set of processes by which an
organization centralizes and optimizesapplication integration through bulk datamovement.
Underlying technologies are EAI.
The target of these technologies are theapplication themselves
7/27/2019 BI Lecture Number 8
5/100
Few More Definitions!!!!
From Wikipediathe integration of databetween applications in a company.
wMUserstechnology that connectsenterprise-wise systems evolved to refer to
technologies used to connect systemsanywhere
Hohpe/Woolfenterprise integration usingmessaging
7/27/2019 BI Lecture Number 8
6/100
EAI & Messaging
Enables data or commands to be sent
across the network using a send and forget approach
7/27/2019 BI Lecture Number 8
7/100
Messaging How? Message-oriented middleware (MOM) like that offered by
IBM WebSphere MQ Microsoft BizTalk
TIBCO WebMethods SeeBeyond (now Sun owned) Vitria and others
Using Java Message Service (JMS) Microsofts Message Queuing (MSMQ) and/or Messaging libraries in
Microsoft .NET Web services standards that support asynchronous Web services
WS-ReliableMessaging Suns Java API for XML Messaging (JAXM)
Microsofts Web Services Extensions (WSE).
7/27/2019 BI Lecture Number 8
8/100
7/27/2019 BI Lecture Number 8
9/100
Selection of vendor
The three short-listed vendors were: IBM,TIBCO and Microsoft. Several criteria were
identified as critical for S-Tels requirementsand each vendor was evaluated againstthem, in an attempt to ensure that the finalselection process will not be flawed becauseof inconsistent data. At the end of thisprocess, S-Tel decided to go for IBMsMQSeries because it had the greatest score
overall
7/27/2019 BI Lecture Number 8
10/100
Basic Architecture
7/27/2019 BI Lecture Number 8
11/100
Basic Architecture
7/27/2019 BI Lecture Number 8
12/100
Vendor Product
Vitria Technology BusinessWare
Tibco Software ActiveEnterprise
SeeBeyond Technology e*Xchange eBusiness Integration Suite
CrossWorlds Software Crossworlds
webMethods webMethods Enterprise
IBM WebSphere Business Integration.
Iona Orbix E2A Web Services Integration Platform
EAI Vendors
7/27/2019 BI Lecture Number 8
13/100
EAI Enterprise Application
Integration
Britannia Airways estimates savings of975,000 annually through a mobileapplication that enables over 2,000 crewmembers to access key enterprise systemsbefore a flight, including email, flight crewrosters, health and safety information and a
duty free point-of-sale application. Mobile middleware played a critical role in
both solutions.
7/27/2019 BI Lecture Number 8
14/100
Difference Between B2B and EAI
The purpose of EAI is:
application level command and control application level information propagation
The purpose of B2B is:
conductance of inter-business transactional
agreements and enforcements
7/27/2019 BI Lecture Number 8
15/100
EAI B2B
EAI typically deals with the integration of
applications and data sources within anenterprise to solve a local problem.
B2B application integration is the integrationof systems between organizations to support
any business requirement, such as sharinginformation with trading partners to support asupply chain or collaborating on a productdesign
7/27/2019 BI Lecture Number 8
16/100
B2B & EAI
The technology and approaches applied to
both types of solutions are similar. For example, both may employ middleware,
such as message brokers, to exchangeinformation between various systems.
Both may have similar approaches tosystems integration.
Which solution come before?
7/27/2019 BI Lecture Number 8
17/100
Drivers EAI
Companies has addressed the integrationproblem by focusing one single applicationprovider but success is limited.
Reasons for failure includes Software drivers: inability of software providers to
deliver 100% needs of a business organization to
combine legacy applications which has createdheterogeneous IS Architecture
Financial Drivers: 40 percent of the IT budget isspend on integration. It was $85 billion in 1998. The
cost of integrating & maintenance of packagedsoftware is very.
7/27/2019 BI Lecture Number 8
18/100
Drivers: Continued
Internal Divers: The support for fast re-
organization of business processes andintegration of legacy systems which aredeveloped internally.
External Drivers: Integration of business
partners such as customers & suppliers atboth data and process level. It necessitatetime & cost efficient integration.
7/27/2019 BI Lecture Number 8
19/100
Approaches of Information System
Integration-3 (1)
Homogeneous with one instances
One process is supported by one application andone database.
This model avoids the problems emerging fromredundant data storage and asynchronous dataexchange between separated applications.
7/27/2019 BI Lecture Number 8
20/100
Approaches of Information System
Integration -3(2)
Homogeneous with several instances
Several identical processes in different businessunits are supported by several identicalapplications that runs on different computers andrely on logically separated data bases.
Example : Application link enabling (ALE) fromSAP, which provide a mechanism for thecoordination of master and transaction data inphysically distributed SAP environment
7/27/2019 BI Lecture Number 8
21/100
Approaches of Information
Systems Integration -3(3)
Heterogeneous
Several different processes in different businessunits are supported by several differentapplications.
The applications are build on divergent datamodels
Which means that they provide differentsemantics (of the data to be interchanged).
7/27/2019 BI Lecture Number 8
22/100
Integration
What you need?
A common language
7/27/2019 BI Lecture Number 8
23/100
Example Inter Process
Integration
Example: EDI
7/27/2019 BI Lecture Number 8
24/100
Integration levels
Company A Company B
Pragmatics Process Levelintegration (Workflowetc.)
Pragmatics
Semantics Object level integration(EAN, D&B etc)
Semantics
Syntax Data level integration
(EDIFACT, ANSI etc)
Syntax
Communication
Services
Communication
Standards ( ISO/OSI-la ers 4 to 7
Communication
Services
7/27/2019 BI Lecture Number 8
25/100
Integration focus of research
approaches
Intra-organizational Inter-organizational
Distributed processing
Architecture
F NF
Integration Backbone F NFDistributed Architecture F NFB2B integration F F
Event drivenArchitecture
F F
EAI Architecture F NFEnterprise Integration
ArchitectureF F
Integration infrastructure F NF
7/27/2019 BI Lecture Number 8
26/100
Integration focus of research
approaches (Integration Objects)
Data Integration Object & process Integration
Distributed processing
Architecture
F NF & NF
Integration Backbone NF F & NFDistributed Architecture NF F & FB2B integration F F & F
Event drivenArchitecture
F NF & NF
EAI Architecture NF F & FEnterprise Integration
ArchitectureF NF& NF
Integration infrastructure F NF & NF
7/27/2019 BI Lecture Number 8
27/100
EAI Systems
7/27/2019 BI Lecture Number 8
28/100
EAI Systems
Traditionally known as Middleware.
These systems are known for data levelintegration
They do not provide any functionality atobject level or process integration
EAI systems fill this gap.
7/27/2019 BI Lecture Number 8
29/100
Sevices Offered by EAI systems
Interface services: Connectors & adaptorsease the burden of programming byproviding pre-built interface.
Transformation Services: ease thedevelopment burden of encoding message
formats & routing messages based on theircontents.
7/27/2019 BI Lecture Number 8
30/100
Continued
Process Management Services: Processmanagement services gather messages andexecute multiple transformations by ensuringthat the flow of information between a set ofresources follows the flow defined in an
established business process
7/27/2019 BI Lecture Number 8
31/100
Continued
Development Services: It helps programmersin the development & adoption of adaptorsfor integrating custom built systems.
Run Time Services: They ensureperformance, scalability, availability &
reliability for all the applications that areintegrated over an EAI systems
7/27/2019 BI Lecture Number 8
32/100
CASE STUDY
The Robert Bosch Group in an internationalcompany with 190000 employee in 132countries and annual revenue higher thanUS25 Billion.
7/27/2019 BI Lecture Number 8
33/100
EAI Laws
1) The whole is greater than the sum of its
parts.
2) There is no end-state.
3) There are no universal standards.
4) Information adapts to meet local needs.
5) All details are relevant.
7/27/2019 BI Lecture Number 8
34/100
EAI Principles
Align EAI plans with business strategy
Consolidate first, integrate second
Use a process-driven approach to develop end-to-end solutions Establish clear lines of ownership and accountability
Enforce an EAI architecture
Mandate integration requirements for new applications
Develop a common representation of data and process
Test early and often Re-factor interfaces constantly so they never become legacy
Evolve business processes through experimentation
7/27/2019 BI Lecture Number 8
35/100
P-1: Align EAI plans with business
strategy
7/27/2019 BI Lecture Number 8
36/100
Principle #2: Consolidate first,
integrate second
The first choice, whenever feasible, shouldbe to consolidate disparate systems thatperform similar functions. Try to removedelicacies
7/27/2019 BI Lecture Number 8
37/100
P-3: Use a process-driven approach to
develop end-to-end Solutions
Integration dimensions can include data,process, platform, network, organization,
location, employees, customers, andproducts among others. With so manyvariables, the potential solutions are infinite. data-centric approach
WWW Business Event Model is the common integration
element driving all others.
7/27/2019 BI Lecture Number 8
38/100
P-4: Establish clear lines of ownership
and accountability
Two critical aspects of ownership andaccountability are 1) Program managementfor the initial deployment of an integratedsolution, and 2) Ongoing management of theshared integration infrastructure.
In addition to C-level executive (CIO, CFO,CPO, even the CEO.
7/27/2019 BI Lecture Number 8
39/100
P-5: Enforce an EAI architecture
7/27/2019 BI Lecture Number 8
40/100
P-6: Mandate integration requirements
for new Application
Applications should be designed with theassumption that they will be part of a largerend-to-end process.
7/27/2019 BI Lecture Number 8
41/100
Principle #7: Develop a common
representation of data and process
Common representation of data
7/27/2019 BI Lecture Number 8
42/100
P-8: Test Early & Often
Use simulation games at the exploratorystage to predict outcome.
Use integrator simulators
Write software components with built intesting features.
7/27/2019 BI Lecture Number 8
43/100
P-09: Re-factor interfacesconstantly so they never become
legacy)
Stable interface is key to sustainability
But there is no stability in applications,protocols etc
7/27/2019 BI Lecture Number 8
44/100
P10: Evolve business processes
through experimenation
There is no end state: P-2.
7/27/2019 BI Lecture Number 8
45/100
EAI
EAI enables data propagation and business
process execution throughout the numerousdistinct networked applications as if it would be aunique global application.
It is a distributed transactional approach and its
focus is to support operational businessfunctions such as taking an order, generating aninvoice, and shipping a product
7/27/2019 BI Lecture Number 8
46/100
Company Product Price
Vitria Technology BusinessWare $500,000 to $700,000
Tibco Software ActiveEnterprise Starting at $100,000
SeeBeyond
Technologye*Xchange eBusiness
Integration Suite$400,000 to $500,000
CrossWorldsSoftware
Crossworlds $500,000
webMethods webMethodsEnterprise
$700,000
Top Vendors, Product average price of software &
deployment
7/27/2019 BI Lecture Number 8
47/100
Application and interfaces at Robert
Bosch
7/27/2019 BI Lecture Number 8
48/100
Common Layers of EAI Solutions
Business Intelligence
Business ProcessManagement
Messaging
Adapters
Provides real-time and historical data on performanceof processes and assists in making decisions.
Manages and tracks business transactions that mightspan multiple systems and last minutes to days.
Ensures the reliability of data delivery across theEnterprise or between systems.
Provides open connectivity into data sources whileallowing filtering and transformations of data.
7/27/2019 BI Lecture Number 8
49/100
7/27/2019 BI Lecture Number 8
50/100
Mitigating Risks Through Automation Validation
The audit process must:
1. Be viewed with prestige and in a positive way
2. Bring together all aspects of the company to do what is nearlyimpossible team building
3. Analyze all controls
4. Document rules and processes in the automation
5. Document the data lineage from reports
6. Have access to and check historical data to see what the peopleand automated systems are catching and what they are missing
7. Effectively and proactively handle external audits to minimizepenalties for infractions
7/27/2019 BI Lecture Number 8
51/100
Information Validation Methodology
Step One: Identify Critical Information End-items
Step Two: Trace Data Lineage Back to Origins
Step Three: Determine the Meaning and Validate the Quality of the
Original Data
Step Fou r: Validate Application Processes, Business Rules and
Related Controls and Verify Automation Security
Step Five: Follow Data Lineage Forward to Validate Mappings,
Transformations and Data Quality
Step Six: Verify Security at Data Consumption Points
Step One: Identify Critical Information
7/27/2019 BI Lecture Number 8
52/100
Applications
DataOrigination
Points
Code and
Data
Step One: Identify Critical InformationEnd-items
Query
Management
ReportsData Marts
ETL
Data Stores and
Warehouses
ETL
Website
EDI
Green
Screen
QueryETLETL
QueryETLETL
EAI
ET
L
Step Two: Trace Data Lineage Back to Origins
7/27/2019 BI Lecture Number 8
53/100
Applications
DataOrigination
Points
Code and
Data
Query
Management
ReportsData Marts
ETL
Data Stores and
Warehouses
ETL
Website
EDI
Green
Screen
QueryETLETL
QueryETLETL
EAI
ET
L
Step Two: Trace Data Lineage Back to Origins
Via Data Marts
Step Two: Trace Data Lineage Back to Origins
7/27/2019 BI Lecture Number 8
54/100
Applications
DataOrigination
Points
Code and
Data
Query
Management
ReportsData Marts
ETL
Data Stores and
Warehouses
ETL
Website
EDI
Green
Screen
QueryETLETL
QueryETLETL
EAI
ET
L
Step Two: Trace Data Lineage Back to Origins
Via Data Stores and Warehouses
Step Two: Trace Data Lineage Back to Origins
7/27/2019 BI Lecture Number 8
55/100
Query
Management
ReportsData Marts
ETL
Data Stores and
Warehouses
ETL
Applications
Website
EDIMessage
Green
Screen
DataOrigination
Points
QueryETLETL
QueryETLETL
EAI
ET
L
Step Two: Trace Data Lineage Back to Origins
Code and
Data
And Via Source Applications
Step Two: Trace Data Lineage Back to Origins
7/27/2019 BI Lecture Number 8
56/100
Query
Management
ReportsData Marts
ETL
Data Stores and
Warehouses
ETL
Applications
Website
EDIMessage
Green
Screen
DataOrigination
Points
QueryETLETL
QueryETLETL
EAI
ET
L
Step Two: Trace Data Lineage Back to Origins
Code and
Data
Until the Origins Are Reached
Step Three: Determine Meaning & Validate
7/27/2019 BI Lecture Number 8
57/100
Applications
DataOrigination
Points
Code and
Data
Query
Management
ReportsData Marts
ETL
Data Stores and
Warehouses
ETL
Website
EDI
Green
Screen
QueryETLETL
QueryETLETL
EAI
ET
L
Step Three: Determine Meaning & ValidateThe Quality of Original Data
Step Four: Validate Application Processes
7/27/2019 BI Lecture Number 8
58/100
Applications
DataOrigination
Points
Code and
Data
Query
Management
ReportsData Marts
ETL
Data Stores and
Warehouses
ETL
Website
EDI
Green
Screen
QueryETLETL
QueryETLETL
EAI
ET
L
Step Four: Validate Application Processes,With Related Business Rules and Controls
Step Five: Validate Mappings, Transformations & Data Quality
7/27/2019 BI Lecture Number 8
59/100
Applications
DataOrigination
Points
Code and
Data
Query
Management
ReportsData Marts
ETL
Data Stores and
Warehouses
ETL
Website
EDI
Green
Screen
QueryETLETL
QueryETLETL
EAI
ET
L
Via Data Warehouses
Step Five: Validate Mappings, Transformations and Data Quality
7/27/2019 BI Lecture Number 8
60/100
Applications
DataOrigination
Points
Code and
Data
Query
Management
ReportsData Marts
ETL
Data Stores and
Warehouses
ETL
Website
EDI
Green
Screen
QueryETLETL
QueryETLETL
EAI
ET
L
And Via Data Marts
Step Five: Validate Mappings, Transformations & Data Quality
7/27/2019 BI Lecture Number 8
61/100
Applications
DataOrigination
Points
Code and
Data
Query
Management
ReportsData Marts
ETL
Data Stores and
Warehouses
ETL
Website
EDI
Green
Screen
QueryETLETL
QueryETLETL
EAI
ET
L
Until Critical Information End-items Are Validated
Step Six: Validate End-to-end
7/27/2019 BI Lecture Number 8
62/100
Step Six: Validate End to endSystem Security
Applications
Data
Origination
Points
Code andData
Query
ManagementReportsData Marts
ETL
Data Stores andWarehouses
ETL
Website
EDI
Green
Screen
QueryETLETL
QueryETLETL
EAI
ETL
Authorized Unauthorized
7/27/2019 BI Lecture Number 8
63/100
Enterprise Integration
Taxonomy
7/27/2019 BI Lecture Number 8
64/100
The Case Study
A major retail chain has dozens of distribution centers Each distribution center or warehouse services hundreds of
stores (>1200 total stores). Each distribution center is moving thousands of
cartons (i.e. boxes) around the warehouse each day Receiving them from trucks through dock doors.
Moving them with fork lifts to storage areas in the warehouse
Conveying them to break down areas for distribution tostores.
Conveying them down belts to storage areas or outboundtrucks.
Moving them onto trucks that depart the warehouse.
7/27/2019 BI Lecture Number 8
65/100
The Case Study
A box is tracked via labels and bar codereaders. Some reads are manual and some are
automated.
Generating literally hundreds of events persecond per warehouse.
RFID was about to make create more events. More reads from more points in the warehouse.
Potentially adding store reads to the event list.
7/27/2019 BI Lecture Number 8
66/100
The System
Part I The retail chain wanted allthe data on events
regarding the movement of cartons sent to HQ Providing them with unparalleled real time information oninventory levels and product status.
Providing more accurate information for merchandiseanalyst and productivity monitoring for warehousemanagers.
Part II (Not germane to the discussion today) Providing a Java Web application to nearly 10,000
users to access the data company wide. Reports galore.
Some limited ad hoc query reporting.
7/27/2019 BI Lecture Number 8
67/100
Lets do the math
>25 warehouses Each generating ~15-20 carton events per
second Averaging 400 messages a second incoming at
HQ Peak around 1300 messages a second incoming
at HQ
Data around an event ~200bytes/msg 24x7x52 (31,449,600 seconds for those not
counting) = ~ 4-7GB a day
7/27/2019 BI Lecture Number 8
68/100
and the math isnt getting
better
During Christmas time things were worse muchworse.
The organization wants to double its current size by2010!
Oh yahdid I mention RFID was coming Tripling or quadrupling the number of events
7/27/2019 BI Lecture Number 8
69/100
Challenge
Design and implement a system to get the data fromthe warehouses to HQ
In near real time to support the reporting needs Use whatever makes sense (to some degree more later)
With a good size team (20-25 people in various roles)
7/27/2019 BI Lecture Number 8
70/100
The Solution Significant company resources and investment in SeeBeyond
EAI product. Put SeeBeyond at all the endpoints (warehouses and HQ).
All data would move through SeeBeyond. SeeBeyond is Java based (also a company technology direction). Write routing/minor processing code in Java in SeeBeyond.
Significant company resources and investment in OracleRDBMS. Oracle already at the warehouses Obtain a honking big Oracle DB at HQ. Use Oracle stored procedures for heavy lifting (data processing
report data preparation).
7/27/2019 BI Lecture Number 8
71/100
Solution Diagram Bundling 20-40 eventmessages at a time
Ex: move this carton there, but haveWe gotten the receive carton msg yet?
We have recd a carton do we have thereference data for the product yet?
7/27/2019 BI Lecture Number 8
72/100
Problem #1We werent ready
As an architect, I was not aware how different an EAImessaging system is. Asynchronous-everywhere nature Had no patterns to follow (No I had not read Hohpe/Woolf
EAI book) Did not have an awareness of the vendor landscape Was easily talked into solutions by others.
My organization didnt see how big it was Had only implemented smaller EAI solutions
Finding good help was hard and a critical step Internally lots of support but no experience Contractors lots of desire, but little implementation
experience to the scale/level of effort
7/27/2019 BI Lecture Number 8
73/100
Getting Yourself Ready
Get yourself ready Understand your optionsall the three letter Es
(EAI, ETL, EII, EDR, etc.) Read EAI patterns Know the products (WBI, Vitria, Tibco,
WebMethods, SeeBeyond, etc.)
Find people with real EAI experience Experienced with systems matching the size of
your app Find people with product expertise Find people with design/pattern expertise
7/27/2019 BI Lecture Number 8
74/100
EAI Patterns
Enterprise Integration Patterns:Hohpe/Woolf
Next Generation Application Integration:Linthicum
IT Architectures and Middleware:Britton
7/27/2019 BI Lecture Number 8
75/100
EAI Component Basics
A typical messaging system is comprised ofthe following parts.
Endpoints
Messages
Channels
Routers Translators
Monitors
7/27/2019 BI Lecture Number 8
76/100
EAI Component Analogy
7/27/2019 BI Lecture Number 8
77/100
EAI Patterns
As the GOF pointed out in generic software, thereare common behaviors in software systems.
They are powerful tools for communicating behavior. They represent naturally occurring processes.
Are generally repetitive in nature, and lend themselves toreuse.
Each of the message components also has severalpatterns that represent common behaviors in amessaging system and encourage reuse.
7/27/2019 BI Lecture Number 8
78/100
Getting Resources Ready
Let the network engineers know of yourplans
You are going to be using a significant amount ofpipe. Have you considered failover/load balancing?
(comm lines around warehouses get cut onoccasion)
Let the database engineers know of yourplans Terabytes of data to be stored and processed
where will it go? Consider backup/recovery systems Database logs/archiving
7/27/2019 BI Lecture Number 8
79/100
Getting Support Ready
Support staffs will be lost at turnover How many of your support shops really know
How to manage application servers? How to manage web applications effectively?
Can you expect them to be able to operate, maintainand support component based messaging systems? Do they know what a message server or bus is? Across a very distributed environment?
Get them trained early (in messaging infrastructure). Have them help you design the monitoring tools and alert
systems. Work together to develop proactive systems checks and
troubleshooting procedures.
7/27/2019 BI Lecture Number 8
80/100
Getting Others Ready
If your development team isnt ready, whatabout Testing/QA teams? Analyst?
Managers?
For example, finding experienced testers for
asynchronous messaging systems is difficult. They usually need intricate knowledge of the
messaging subsystem monitors and admincapabilities.
7/27/2019 BI Lecture Number 8
81/100
Problem #2 Proprietary EAI
EAI Products/Solutions are many.
EAI Standards are few.
EAI/ETL/EII/ market place is tumultuous Sun has purchases SeeBeyond
IBM bought Ascential
Everyone calling their product an ESB (example on nextpage)
Products/Solutions have scale limits Some they know about
Others they do not
Java alone does not make you platform
independent.
7/27/2019 BI Lecture Number 8
82/100
Can you identify this product??
provides an award-winning messagingbackbone for deploying your enterprise
service bus (ESB) today as the connectivitylayer of a service-orientated architecture(SOA).
7/27/2019 BI Lecture Number 8
83/100
Examine Your Solution Options See if what you already have would work.
There is a reason MQ has been around a long time. Where possible consider tried, true and already deployed
platforms But again do the math and see if they can support the extra
load. In house support is probably better equipped (more in a bit)
Not everything has to travel by message.
Consider multiple/alternate technologies for parts of yoursolution. ETL is great for certain parts of a large solution There is a reason why products like Oracle are expensive
(technologies like Oracle Replication more in a bit). Does, however, create more issues of timing.
7/27/2019 BI Lecture Number 8
84/100
What Travels by Message? Consider multiple/alternate technologies for parts of
your solution.
Replication of reference data Bulk/batch transfers
Non-real time needs
ETL is great for certain parts of a large solution
Examine features in your DB/App Servers There is a reason why products like Oracle are expensive
(technologies like Oracle Replication more in a bit).
How about those Message Beans in the app server?
This can, however, create more issues of timing.
7/27/2019 BI Lecture Number 8
85/100
Reference Data In many applications, you need reference data on
both ends of the messaging systems. You can build a replicating message engine to treat this
like other message data (not recommended). Referential integrity becomes a real problem. Consider issues of message timing (PR becomes the 51st
state but messages with PR references start to arrive beforethe new state data does)
Use simple replication technologies where possible ETL tools - if reference data changes only happen at certaintimes.
Technologies like Oracle Replication for real time (it canoperate over a WAN).
7/27/2019 BI Lecture Number 8
86/100
Interoperability
We used Java, but
Even when you use Java, how is it being applied?
Java running inside of proprietary components (likeSeeBeyond eWays) does not make you portable.
Write component code that can be used by orincorporated by proprietary systems.
Under the covers, is the vendor using JMS JMX/SNMP
Web services/WS-Reliable Messaging/JAX-RPC
Etc
7/27/2019 BI Lecture Number 8
87/100
Process outside the bus Process outside the message bus/subsystem if you
can Let the bus focus on delivering the goods.
Too much processing time in the bus will create Scalability problems Monitoring problems Possibly interoperability problems (especially when using
proprietary technology/components)
Process with components that are Flexible easy to get at (and change) interoperable (if possible) and contain reusable business logic (if possible)
7/27/2019 BI Lecture Number 8
88/100
Problem #3Math we didnt do
We didnt do enough math up front.
We didnt plan for failure/growth.
The messages moved slower than anticipated. The message processing took more time than
expected.
The amount of data was larger than expected.
7/27/2019 BI Lecture Number 8
89/100
Do the math and ask the tough?s
How much time its going to take to get a messagefrom A to B Test that estimate early.
Work with the business analysts to figure out howmany messages need to be moved. Make volume estimates part of the non-functional
requirements gathering process. Check that against the existing databases if possible.
How much data needs to be packaged, shipped,processed, stored? Design the messages and calculate the size of the overall
message (XML and all). Calculate the rate and add up the total volume.
7/27/2019 BI Lecture Number 8
90/100
and pad your answer!
Do you have room to spare?? Can the messaging system handle that (on both ends)?
Can the consuming database handle that? Can the hardware and network handle that?
Anticipate failure What happens if something/anything goes down for an
hour?
What happens if you go down for a day? What happens if you have unexpected growth?
7/27/2019 BI Lecture Number 8
91/100
Problem #4 Exception
handling wasnt More considerations for failover and redundancy
Versus Web application
We did not plan on downtime
Unplanned system issues Planned outages
We didnt build in enough redundancy Load balancing and Failover were both after thoughts
Allmessages always correct allthe time (NOT) At first, we had no proper dead letter queuing No proper exception processing
No means to properly see and react to issues Many more points of failure and potential issues More widely distributed
D i l d b l i &
7/27/2019 BI Lecture Number 8
92/100
Design load balancing &failover upfront
Load balancing and failover must be accommodated
Like security, you need a multi-layered approach
Hardware (like Big IP) Redundant message bus/message servers
Processing components
Database
EAI system throttling
How are you going to kick over to the failoversystems (and return to regular systems)? Without losing messages
Without causing timing problems in message deliver/receipt
7/27/2019 BI Lecture Number 8
93/100
Throttling
Throttling limits ("throttles") the number ofrequests it will respond to within a specified
period of time. Limits congestion.
Built into most good EAI solutions today.
Often overlooked and not used. Used in messaging systems to ensure that
no one part of the system is driven beyond itscapacity or performance efficiently.
Th ttli t th t ti l
7/27/2019 BI Lecture Number 8
94/100
Throttling at the potentialcongestion point
Distribution Systems
DBE
Message Poll
& DecorationQueue Message
bundler
E
Validation &
RequestProcessing
(VRP)
DBRecving
Queue
Problem
Message
!
WAN WAN
x 25
Dead Letter
Queue
DB Ready
Queues
Retry Logic
Throttle points.Potentially lots of messages,especially if the WAN goesdown
Congestion point
7/27/2019 BI Lecture Number 8
95/100
Space, space and more space
Plan on extra space for failure A place for queued messages to sit if something goes down
Space in the DB or space in the message channels orboth
Consider the time lags for getting additional hardwarebought, installed, and up and running
Plan on extra space for logs
You are going to want to keep log files around for a while. Some problems take time to manifest to a point of
awareness.
Devise an automated archive/clean up for logs.
Nonot all EAI systems provide log clean up utilities.
7/27/2019 BI Lecture Number 8
96/100
96
Process
Integration Level (2)
Application
Data
????? ????
Fundamental Integration Levels
Electronic Business =Zero-Latency Business {Gartner}
7/27/2019 BI Lecture Number 8
97/100
WarehouseMgmt.
ACCOUNTNO.
SHEETNO.
ACCOUNTSRECEIVABLE LEDGER
DATE INVOICENUMBER/DESCRIPTION CHARGES CREDITS BALANCE
BALANCEFORWARD
ACCOUNTNO.
SHEETNO.
ACCOUNTSRECEIVABLE LEDGER
DATE INVOICENUMBER/DESCRIPTION CHARGES CREDITS BALANCE
BALANCEFORWAR D
Logistics
Order Entry
Financials
Bank
Parcel Service
Call Center
E-Commerce Portal
Customer
The Enterprise
1) I want to buy it.
2) I dont have it yet!
3) When are you goingto get it delivered?
4) Please pay
for it.
B2B
B2B
B2CA2A
Zero-Latency Business {Gartner}
7/27/2019 BI Lecture Number 8
98/100
98
Zero or Low Data Latency
Just the old concept of timeliness with an additionaltwist
Refers to the real time (zero) or near-real time (low)transfer of information between applications
Zero data latency tends to imply transactionalsolutions, while low data latency or near-real time
allows for assured delivery (messaging) options Low data latency within the enterprise is no longer
enough; it must be enabled across the enterpriseperimeter
7/27/2019 BI Lecture Number 8
99/100
99
An Affirmation; Modeling theProblem
Adding EAI oriented solutions to the problem of data availabilitydoes not reduce the modeling effort required it intensifies it
Each application view of the data must be defined andunderstood
Additionally the dependencies involved in sharing andtransferring information must be supported
7/27/2019 BI Lecture Number 8
100/100
Modes of Integration? AFunctional View!
Data Consistency
Multistep Process
Composite Application
Batch or immediatetransfers
Multiple processes Multiple steps One-way, asynchronous
interactions (loosely
coupled) Systems are physically
and logically independent
Manual or straight-throughflows
Batch or immediate,
individual transfers One business process Multiple steps One-way, asynchronous
interactions (looselycoupled)
Systems are physically andlogically independent
Potential long transactions
Immediate interaction One business process One step Two-way, synchronous
interactions (Tightly Coupled) Systems are physically and
logically dependent Usually a client server
(request reply) interaction