BI Lecture Number 8

Embed Size (px)

Citation preview

  • 7/27/2019 BI Lecture Number 8

    1/100

    Enterprise Application Integration

    Dr. N. P. SinghProfessorManagement Development InstituteMehrauli Road, SukhraliGurgaon -122001E-mail: [email protected]

  • 7/27/2019 BI Lecture Number 8

    2/100

    Contents

    Part-I

    What is EAI?

  • 7/27/2019 BI Lecture Number 8

    3/100

    Definitions

    Enterprise Application Integration

    Definition: The process of integrat ing mult ip leappl icat ions that were independently developed,

    may use incompat ib le technology, and remain

    independent ly managed.

    By this definition, EAI would include: Business Process Integration

    Enterprise Information Integration

  • 7/27/2019 BI Lecture Number 8

    4/100

    Data Propagation

    It is the set of processes by which an

    organization centralizes and optimizesapplication integration through bulk datamovement.

    Underlying technologies are EAI.

    The target of these technologies are theapplication themselves

  • 7/27/2019 BI Lecture Number 8

    5/100

    Few More Definitions!!!!

    From Wikipediathe integration of databetween applications in a company.

    wMUserstechnology that connectsenterprise-wise systems evolved to refer to

    technologies used to connect systemsanywhere

    Hohpe/Woolfenterprise integration usingmessaging

  • 7/27/2019 BI Lecture Number 8

    6/100

    EAI & Messaging

    Enables data or commands to be sent

    across the network using a send and forget approach

  • 7/27/2019 BI Lecture Number 8

    7/100

    Messaging How? Message-oriented middleware (MOM) like that offered by

    IBM WebSphere MQ Microsoft BizTalk

    TIBCO WebMethods SeeBeyond (now Sun owned) Vitria and others

    Using Java Message Service (JMS) Microsofts Message Queuing (MSMQ) and/or Messaging libraries in

    Microsoft .NET Web services standards that support asynchronous Web services

    WS-ReliableMessaging Suns Java API for XML Messaging (JAXM)

    Microsofts Web Services Extensions (WSE).

  • 7/27/2019 BI Lecture Number 8

    8/100

  • 7/27/2019 BI Lecture Number 8

    9/100

    Selection of vendor

    The three short-listed vendors were: IBM,TIBCO and Microsoft. Several criteria were

    identified as critical for S-Tels requirementsand each vendor was evaluated againstthem, in an attempt to ensure that the finalselection process will not be flawed becauseof inconsistent data. At the end of thisprocess, S-Tel decided to go for IBMsMQSeries because it had the greatest score

    overall

  • 7/27/2019 BI Lecture Number 8

    10/100

    Basic Architecture

  • 7/27/2019 BI Lecture Number 8

    11/100

    Basic Architecture

  • 7/27/2019 BI Lecture Number 8

    12/100

    Vendor Product

    Vitria Technology BusinessWare

    Tibco Software ActiveEnterprise

    SeeBeyond Technology e*Xchange eBusiness Integration Suite

    CrossWorlds Software Crossworlds

    webMethods webMethods Enterprise

    IBM WebSphere Business Integration.

    Iona Orbix E2A Web Services Integration Platform

    EAI Vendors

  • 7/27/2019 BI Lecture Number 8

    13/100

    EAI Enterprise Application

    Integration

    Britannia Airways estimates savings of975,000 annually through a mobileapplication that enables over 2,000 crewmembers to access key enterprise systemsbefore a flight, including email, flight crewrosters, health and safety information and a

    duty free point-of-sale application. Mobile middleware played a critical role in

    both solutions.

  • 7/27/2019 BI Lecture Number 8

    14/100

    Difference Between B2B and EAI

    The purpose of EAI is:

    application level command and control application level information propagation

    The purpose of B2B is:

    conductance of inter-business transactional

    agreements and enforcements

  • 7/27/2019 BI Lecture Number 8

    15/100

    EAI B2B

    EAI typically deals with the integration of

    applications and data sources within anenterprise to solve a local problem.

    B2B application integration is the integrationof systems between organizations to support

    any business requirement, such as sharinginformation with trading partners to support asupply chain or collaborating on a productdesign

  • 7/27/2019 BI Lecture Number 8

    16/100

    B2B & EAI

    The technology and approaches applied to

    both types of solutions are similar. For example, both may employ middleware,

    such as message brokers, to exchangeinformation between various systems.

    Both may have similar approaches tosystems integration.

    Which solution come before?

  • 7/27/2019 BI Lecture Number 8

    17/100

    Drivers EAI

    Companies has addressed the integrationproblem by focusing one single applicationprovider but success is limited.

    Reasons for failure includes Software drivers: inability of software providers to

    deliver 100% needs of a business organization to

    combine legacy applications which has createdheterogeneous IS Architecture

    Financial Drivers: 40 percent of the IT budget isspend on integration. It was $85 billion in 1998. The

    cost of integrating & maintenance of packagedsoftware is very.

  • 7/27/2019 BI Lecture Number 8

    18/100

    Drivers: Continued

    Internal Divers: The support for fast re-

    organization of business processes andintegration of legacy systems which aredeveloped internally.

    External Drivers: Integration of business

    partners such as customers & suppliers atboth data and process level. It necessitatetime & cost efficient integration.

  • 7/27/2019 BI Lecture Number 8

    19/100

    Approaches of Information System

    Integration-3 (1)

    Homogeneous with one instances

    One process is supported by one application andone database.

    This model avoids the problems emerging fromredundant data storage and asynchronous dataexchange between separated applications.

  • 7/27/2019 BI Lecture Number 8

    20/100

    Approaches of Information System

    Integration -3(2)

    Homogeneous with several instances

    Several identical processes in different businessunits are supported by several identicalapplications that runs on different computers andrely on logically separated data bases.

    Example : Application link enabling (ALE) fromSAP, which provide a mechanism for thecoordination of master and transaction data inphysically distributed SAP environment

  • 7/27/2019 BI Lecture Number 8

    21/100

    Approaches of Information

    Systems Integration -3(3)

    Heterogeneous

    Several different processes in different businessunits are supported by several differentapplications.

    The applications are build on divergent datamodels

    Which means that they provide differentsemantics (of the data to be interchanged).

  • 7/27/2019 BI Lecture Number 8

    22/100

    Integration

    What you need?

    A common language

  • 7/27/2019 BI Lecture Number 8

    23/100

    Example Inter Process

    Integration

    Example: EDI

  • 7/27/2019 BI Lecture Number 8

    24/100

    Integration levels

    Company A Company B

    Pragmatics Process Levelintegration (Workflowetc.)

    Pragmatics

    Semantics Object level integration(EAN, D&B etc)

    Semantics

    Syntax Data level integration

    (EDIFACT, ANSI etc)

    Syntax

    Communication

    Services

    Communication

    Standards ( ISO/OSI-la ers 4 to 7

    Communication

    Services

  • 7/27/2019 BI Lecture Number 8

    25/100

    Integration focus of research

    approaches

    Intra-organizational Inter-organizational

    Distributed processing

    Architecture

    F NF

    Integration Backbone F NFDistributed Architecture F NFB2B integration F F

    Event drivenArchitecture

    F F

    EAI Architecture F NFEnterprise Integration

    ArchitectureF F

    Integration infrastructure F NF

  • 7/27/2019 BI Lecture Number 8

    26/100

    Integration focus of research

    approaches (Integration Objects)

    Data Integration Object & process Integration

    Distributed processing

    Architecture

    F NF & NF

    Integration Backbone NF F & NFDistributed Architecture NF F & FB2B integration F F & F

    Event drivenArchitecture

    F NF & NF

    EAI Architecture NF F & FEnterprise Integration

    ArchitectureF NF& NF

    Integration infrastructure F NF & NF

  • 7/27/2019 BI Lecture Number 8

    27/100

    EAI Systems

  • 7/27/2019 BI Lecture Number 8

    28/100

    EAI Systems

    Traditionally known as Middleware.

    These systems are known for data levelintegration

    They do not provide any functionality atobject level or process integration

    EAI systems fill this gap.

  • 7/27/2019 BI Lecture Number 8

    29/100

    Sevices Offered by EAI systems

    Interface services: Connectors & adaptorsease the burden of programming byproviding pre-built interface.

    Transformation Services: ease thedevelopment burden of encoding message

    formats & routing messages based on theircontents.

  • 7/27/2019 BI Lecture Number 8

    30/100

    Continued

    Process Management Services: Processmanagement services gather messages andexecute multiple transformations by ensuringthat the flow of information between a set ofresources follows the flow defined in an

    established business process

  • 7/27/2019 BI Lecture Number 8

    31/100

    Continued

    Development Services: It helps programmersin the development & adoption of adaptorsfor integrating custom built systems.

    Run Time Services: They ensureperformance, scalability, availability &

    reliability for all the applications that areintegrated over an EAI systems

  • 7/27/2019 BI Lecture Number 8

    32/100

    CASE STUDY

    The Robert Bosch Group in an internationalcompany with 190000 employee in 132countries and annual revenue higher thanUS25 Billion.

  • 7/27/2019 BI Lecture Number 8

    33/100

    EAI Laws

    1) The whole is greater than the sum of its

    parts.

    2) There is no end-state.

    3) There are no universal standards.

    4) Information adapts to meet local needs.

    5) All details are relevant.

  • 7/27/2019 BI Lecture Number 8

    34/100

    EAI Principles

    Align EAI plans with business strategy

    Consolidate first, integrate second

    Use a process-driven approach to develop end-to-end solutions Establish clear lines of ownership and accountability

    Enforce an EAI architecture

    Mandate integration requirements for new applications

    Develop a common representation of data and process

    Test early and often Re-factor interfaces constantly so they never become legacy

    Evolve business processes through experimentation

  • 7/27/2019 BI Lecture Number 8

    35/100

    P-1: Align EAI plans with business

    strategy

  • 7/27/2019 BI Lecture Number 8

    36/100

    Principle #2: Consolidate first,

    integrate second

    The first choice, whenever feasible, shouldbe to consolidate disparate systems thatperform similar functions. Try to removedelicacies

  • 7/27/2019 BI Lecture Number 8

    37/100

    P-3: Use a process-driven approach to

    develop end-to-end Solutions

    Integration dimensions can include data,process, platform, network, organization,

    location, employees, customers, andproducts among others. With so manyvariables, the potential solutions are infinite. data-centric approach

    WWW Business Event Model is the common integration

    element driving all others.

  • 7/27/2019 BI Lecture Number 8

    38/100

    P-4: Establish clear lines of ownership

    and accountability

    Two critical aspects of ownership andaccountability are 1) Program managementfor the initial deployment of an integratedsolution, and 2) Ongoing management of theshared integration infrastructure.

    In addition to C-level executive (CIO, CFO,CPO, even the CEO.

  • 7/27/2019 BI Lecture Number 8

    39/100

    P-5: Enforce an EAI architecture

  • 7/27/2019 BI Lecture Number 8

    40/100

    P-6: Mandate integration requirements

    for new Application

    Applications should be designed with theassumption that they will be part of a largerend-to-end process.

  • 7/27/2019 BI Lecture Number 8

    41/100

    Principle #7: Develop a common

    representation of data and process

    Common representation of data

  • 7/27/2019 BI Lecture Number 8

    42/100

    P-8: Test Early & Often

    Use simulation games at the exploratorystage to predict outcome.

    Use integrator simulators

    Write software components with built intesting features.

  • 7/27/2019 BI Lecture Number 8

    43/100

    P-09: Re-factor interfacesconstantly so they never become

    legacy)

    Stable interface is key to sustainability

    But there is no stability in applications,protocols etc

  • 7/27/2019 BI Lecture Number 8

    44/100

    P10: Evolve business processes

    through experimenation

    There is no end state: P-2.

  • 7/27/2019 BI Lecture Number 8

    45/100

    EAI

    EAI enables data propagation and business

    process execution throughout the numerousdistinct networked applications as if it would be aunique global application.

    It is a distributed transactional approach and its

    focus is to support operational businessfunctions such as taking an order, generating aninvoice, and shipping a product

  • 7/27/2019 BI Lecture Number 8

    46/100

    Company Product Price

    Vitria Technology BusinessWare $500,000 to $700,000

    Tibco Software ActiveEnterprise Starting at $100,000

    SeeBeyond

    Technologye*Xchange eBusiness

    Integration Suite$400,000 to $500,000

    CrossWorldsSoftware

    Crossworlds $500,000

    webMethods webMethodsEnterprise

    $700,000

    Top Vendors, Product average price of software &

    deployment

  • 7/27/2019 BI Lecture Number 8

    47/100

    Application and interfaces at Robert

    Bosch

  • 7/27/2019 BI Lecture Number 8

    48/100

    Common Layers of EAI Solutions

    Business Intelligence

    Business ProcessManagement

    Messaging

    Adapters

    Provides real-time and historical data on performanceof processes and assists in making decisions.

    Manages and tracks business transactions that mightspan multiple systems and last minutes to days.

    Ensures the reliability of data delivery across theEnterprise or between systems.

    Provides open connectivity into data sources whileallowing filtering and transformations of data.

  • 7/27/2019 BI Lecture Number 8

    49/100

  • 7/27/2019 BI Lecture Number 8

    50/100

    Mitigating Risks Through Automation Validation

    The audit process must:

    1. Be viewed with prestige and in a positive way

    2. Bring together all aspects of the company to do what is nearlyimpossible team building

    3. Analyze all controls

    4. Document rules and processes in the automation

    5. Document the data lineage from reports

    6. Have access to and check historical data to see what the peopleand automated systems are catching and what they are missing

    7. Effectively and proactively handle external audits to minimizepenalties for infractions

  • 7/27/2019 BI Lecture Number 8

    51/100

    Information Validation Methodology

    Step One: Identify Critical Information End-items

    Step Two: Trace Data Lineage Back to Origins

    Step Three: Determine the Meaning and Validate the Quality of the

    Original Data

    Step Fou r: Validate Application Processes, Business Rules and

    Related Controls and Verify Automation Security

    Step Five: Follow Data Lineage Forward to Validate Mappings,

    Transformations and Data Quality

    Step Six: Verify Security at Data Consumption Points

    Step One: Identify Critical Information

  • 7/27/2019 BI Lecture Number 8

    52/100

    Applications

    DataOrigination

    Points

    Code and

    Data

    Step One: Identify Critical InformationEnd-items

    Query

    Management

    ReportsData Marts

    ETL

    Data Stores and

    Warehouses

    ETL

    Website

    EDI

    Green

    Screen

    QueryETLETL

    QueryETLETL

    EAI

    ET

    L

    Step Two: Trace Data Lineage Back to Origins

  • 7/27/2019 BI Lecture Number 8

    53/100

    Applications

    DataOrigination

    Points

    Code and

    Data

    Query

    Management

    ReportsData Marts

    ETL

    Data Stores and

    Warehouses

    ETL

    Website

    EDI

    Green

    Screen

    QueryETLETL

    QueryETLETL

    EAI

    ET

    L

    Step Two: Trace Data Lineage Back to Origins

    Via Data Marts

    Step Two: Trace Data Lineage Back to Origins

  • 7/27/2019 BI Lecture Number 8

    54/100

    Applications

    DataOrigination

    Points

    Code and

    Data

    Query

    Management

    ReportsData Marts

    ETL

    Data Stores and

    Warehouses

    ETL

    Website

    EDI

    Green

    Screen

    QueryETLETL

    QueryETLETL

    EAI

    ET

    L

    Step Two: Trace Data Lineage Back to Origins

    Via Data Stores and Warehouses

    Step Two: Trace Data Lineage Back to Origins

  • 7/27/2019 BI Lecture Number 8

    55/100

    Query

    Management

    ReportsData Marts

    ETL

    Data Stores and

    Warehouses

    ETL

    Applications

    Website

    EDIMessage

    Green

    Screen

    DataOrigination

    Points

    QueryETLETL

    QueryETLETL

    EAI

    ET

    L

    Step Two: Trace Data Lineage Back to Origins

    Code and

    Data

    And Via Source Applications

    Step Two: Trace Data Lineage Back to Origins

  • 7/27/2019 BI Lecture Number 8

    56/100

    Query

    Management

    ReportsData Marts

    ETL

    Data Stores and

    Warehouses

    ETL

    Applications

    Website

    EDIMessage

    Green

    Screen

    DataOrigination

    Points

    QueryETLETL

    QueryETLETL

    EAI

    ET

    L

    Step Two: Trace Data Lineage Back to Origins

    Code and

    Data

    Until the Origins Are Reached

    Step Three: Determine Meaning & Validate

  • 7/27/2019 BI Lecture Number 8

    57/100

    Applications

    DataOrigination

    Points

    Code and

    Data

    Query

    Management

    ReportsData Marts

    ETL

    Data Stores and

    Warehouses

    ETL

    Website

    EDI

    Green

    Screen

    QueryETLETL

    QueryETLETL

    EAI

    ET

    L

    Step Three: Determine Meaning & ValidateThe Quality of Original Data

    Step Four: Validate Application Processes

  • 7/27/2019 BI Lecture Number 8

    58/100

    Applications

    DataOrigination

    Points

    Code and

    Data

    Query

    Management

    ReportsData Marts

    ETL

    Data Stores and

    Warehouses

    ETL

    Website

    EDI

    Green

    Screen

    QueryETLETL

    QueryETLETL

    EAI

    ET

    L

    Step Four: Validate Application Processes,With Related Business Rules and Controls

    Step Five: Validate Mappings, Transformations & Data Quality

  • 7/27/2019 BI Lecture Number 8

    59/100

    Applications

    DataOrigination

    Points

    Code and

    Data

    Query

    Management

    ReportsData Marts

    ETL

    Data Stores and

    Warehouses

    ETL

    Website

    EDI

    Green

    Screen

    QueryETLETL

    QueryETLETL

    EAI

    ET

    L

    Via Data Warehouses

    Step Five: Validate Mappings, Transformations and Data Quality

  • 7/27/2019 BI Lecture Number 8

    60/100

    Applications

    DataOrigination

    Points

    Code and

    Data

    Query

    Management

    ReportsData Marts

    ETL

    Data Stores and

    Warehouses

    ETL

    Website

    EDI

    Green

    Screen

    QueryETLETL

    QueryETLETL

    EAI

    ET

    L

    And Via Data Marts

    Step Five: Validate Mappings, Transformations & Data Quality

  • 7/27/2019 BI Lecture Number 8

    61/100

    Applications

    DataOrigination

    Points

    Code and

    Data

    Query

    Management

    ReportsData Marts

    ETL

    Data Stores and

    Warehouses

    ETL

    Website

    EDI

    Green

    Screen

    QueryETLETL

    QueryETLETL

    EAI

    ET

    L

    Until Critical Information End-items Are Validated

    Step Six: Validate End-to-end

  • 7/27/2019 BI Lecture Number 8

    62/100

    Step Six: Validate End to endSystem Security

    Applications

    Data

    Origination

    Points

    Code andData

    Query

    ManagementReportsData Marts

    ETL

    Data Stores andWarehouses

    ETL

    Website

    EDI

    Green

    Screen

    QueryETLETL

    QueryETLETL

    EAI

    ETL

    Authorized Unauthorized

  • 7/27/2019 BI Lecture Number 8

    63/100

    Enterprise Integration

    Taxonomy

  • 7/27/2019 BI Lecture Number 8

    64/100

    The Case Study

    A major retail chain has dozens of distribution centers Each distribution center or warehouse services hundreds of

    stores (>1200 total stores). Each distribution center is moving thousands of

    cartons (i.e. boxes) around the warehouse each day Receiving them from trucks through dock doors.

    Moving them with fork lifts to storage areas in the warehouse

    Conveying them to break down areas for distribution tostores.

    Conveying them down belts to storage areas or outboundtrucks.

    Moving them onto trucks that depart the warehouse.

  • 7/27/2019 BI Lecture Number 8

    65/100

    The Case Study

    A box is tracked via labels and bar codereaders. Some reads are manual and some are

    automated.

    Generating literally hundreds of events persecond per warehouse.

    RFID was about to make create more events. More reads from more points in the warehouse.

    Potentially adding store reads to the event list.

  • 7/27/2019 BI Lecture Number 8

    66/100

    The System

    Part I The retail chain wanted allthe data on events

    regarding the movement of cartons sent to HQ Providing them with unparalleled real time information oninventory levels and product status.

    Providing more accurate information for merchandiseanalyst and productivity monitoring for warehousemanagers.

    Part II (Not germane to the discussion today) Providing a Java Web application to nearly 10,000

    users to access the data company wide. Reports galore.

    Some limited ad hoc query reporting.

  • 7/27/2019 BI Lecture Number 8

    67/100

    Lets do the math

    >25 warehouses Each generating ~15-20 carton events per

    second Averaging 400 messages a second incoming at

    HQ Peak around 1300 messages a second incoming

    at HQ

    Data around an event ~200bytes/msg 24x7x52 (31,449,600 seconds for those not

    counting) = ~ 4-7GB a day

  • 7/27/2019 BI Lecture Number 8

    68/100

    and the math isnt getting

    better

    During Christmas time things were worse muchworse.

    The organization wants to double its current size by2010!

    Oh yahdid I mention RFID was coming Tripling or quadrupling the number of events

  • 7/27/2019 BI Lecture Number 8

    69/100

    Challenge

    Design and implement a system to get the data fromthe warehouses to HQ

    In near real time to support the reporting needs Use whatever makes sense (to some degree more later)

    With a good size team (20-25 people in various roles)

  • 7/27/2019 BI Lecture Number 8

    70/100

    The Solution Significant company resources and investment in SeeBeyond

    EAI product. Put SeeBeyond at all the endpoints (warehouses and HQ).

    All data would move through SeeBeyond. SeeBeyond is Java based (also a company technology direction). Write routing/minor processing code in Java in SeeBeyond.

    Significant company resources and investment in OracleRDBMS. Oracle already at the warehouses Obtain a honking big Oracle DB at HQ. Use Oracle stored procedures for heavy lifting (data processing

    report data preparation).

  • 7/27/2019 BI Lecture Number 8

    71/100

    Solution Diagram Bundling 20-40 eventmessages at a time

    Ex: move this carton there, but haveWe gotten the receive carton msg yet?

    We have recd a carton do we have thereference data for the product yet?

  • 7/27/2019 BI Lecture Number 8

    72/100

    Problem #1We werent ready

    As an architect, I was not aware how different an EAImessaging system is. Asynchronous-everywhere nature Had no patterns to follow (No I had not read Hohpe/Woolf

    EAI book) Did not have an awareness of the vendor landscape Was easily talked into solutions by others.

    My organization didnt see how big it was Had only implemented smaller EAI solutions

    Finding good help was hard and a critical step Internally lots of support but no experience Contractors lots of desire, but little implementation

    experience to the scale/level of effort

  • 7/27/2019 BI Lecture Number 8

    73/100

    Getting Yourself Ready

    Get yourself ready Understand your optionsall the three letter Es

    (EAI, ETL, EII, EDR, etc.) Read EAI patterns Know the products (WBI, Vitria, Tibco,

    WebMethods, SeeBeyond, etc.)

    Find people with real EAI experience Experienced with systems matching the size of

    your app Find people with product expertise Find people with design/pattern expertise

  • 7/27/2019 BI Lecture Number 8

    74/100

    EAI Patterns

    Enterprise Integration Patterns:Hohpe/Woolf

    Next Generation Application Integration:Linthicum

    IT Architectures and Middleware:Britton

  • 7/27/2019 BI Lecture Number 8

    75/100

    EAI Component Basics

    A typical messaging system is comprised ofthe following parts.

    Endpoints

    Messages

    Channels

    Routers Translators

    Monitors

  • 7/27/2019 BI Lecture Number 8

    76/100

    EAI Component Analogy

  • 7/27/2019 BI Lecture Number 8

    77/100

    EAI Patterns

    As the GOF pointed out in generic software, thereare common behaviors in software systems.

    They are powerful tools for communicating behavior. They represent naturally occurring processes.

    Are generally repetitive in nature, and lend themselves toreuse.

    Each of the message components also has severalpatterns that represent common behaviors in amessaging system and encourage reuse.

  • 7/27/2019 BI Lecture Number 8

    78/100

    Getting Resources Ready

    Let the network engineers know of yourplans

    You are going to be using a significant amount ofpipe. Have you considered failover/load balancing?

    (comm lines around warehouses get cut onoccasion)

    Let the database engineers know of yourplans Terabytes of data to be stored and processed

    where will it go? Consider backup/recovery systems Database logs/archiving

  • 7/27/2019 BI Lecture Number 8

    79/100

    Getting Support Ready

    Support staffs will be lost at turnover How many of your support shops really know

    How to manage application servers? How to manage web applications effectively?

    Can you expect them to be able to operate, maintainand support component based messaging systems? Do they know what a message server or bus is? Across a very distributed environment?

    Get them trained early (in messaging infrastructure). Have them help you design the monitoring tools and alert

    systems. Work together to develop proactive systems checks and

    troubleshooting procedures.

  • 7/27/2019 BI Lecture Number 8

    80/100

    Getting Others Ready

    If your development team isnt ready, whatabout Testing/QA teams? Analyst?

    Managers?

    For example, finding experienced testers for

    asynchronous messaging systems is difficult. They usually need intricate knowledge of the

    messaging subsystem monitors and admincapabilities.

  • 7/27/2019 BI Lecture Number 8

    81/100

    Problem #2 Proprietary EAI

    EAI Products/Solutions are many.

    EAI Standards are few.

    EAI/ETL/EII/ market place is tumultuous Sun has purchases SeeBeyond

    IBM bought Ascential

    Everyone calling their product an ESB (example on nextpage)

    Products/Solutions have scale limits Some they know about

    Others they do not

    Java alone does not make you platform

    independent.

  • 7/27/2019 BI Lecture Number 8

    82/100

    Can you identify this product??

    provides an award-winning messagingbackbone for deploying your enterprise

    service bus (ESB) today as the connectivitylayer of a service-orientated architecture(SOA).

  • 7/27/2019 BI Lecture Number 8

    83/100

    Examine Your Solution Options See if what you already have would work.

    There is a reason MQ has been around a long time. Where possible consider tried, true and already deployed

    platforms But again do the math and see if they can support the extra

    load. In house support is probably better equipped (more in a bit)

    Not everything has to travel by message.

    Consider multiple/alternate technologies for parts of yoursolution. ETL is great for certain parts of a large solution There is a reason why products like Oracle are expensive

    (technologies like Oracle Replication more in a bit). Does, however, create more issues of timing.

  • 7/27/2019 BI Lecture Number 8

    84/100

    What Travels by Message? Consider multiple/alternate technologies for parts of

    your solution.

    Replication of reference data Bulk/batch transfers

    Non-real time needs

    ETL is great for certain parts of a large solution

    Examine features in your DB/App Servers There is a reason why products like Oracle are expensive

    (technologies like Oracle Replication more in a bit).

    How about those Message Beans in the app server?

    This can, however, create more issues of timing.

  • 7/27/2019 BI Lecture Number 8

    85/100

    Reference Data In many applications, you need reference data on

    both ends of the messaging systems. You can build a replicating message engine to treat this

    like other message data (not recommended). Referential integrity becomes a real problem. Consider issues of message timing (PR becomes the 51st

    state but messages with PR references start to arrive beforethe new state data does)

    Use simple replication technologies where possible ETL tools - if reference data changes only happen at certaintimes.

    Technologies like Oracle Replication for real time (it canoperate over a WAN).

  • 7/27/2019 BI Lecture Number 8

    86/100

    Interoperability

    We used Java, but

    Even when you use Java, how is it being applied?

    Java running inside of proprietary components (likeSeeBeyond eWays) does not make you portable.

    Write component code that can be used by orincorporated by proprietary systems.

    Under the covers, is the vendor using JMS JMX/SNMP

    Web services/WS-Reliable Messaging/JAX-RPC

    Etc

  • 7/27/2019 BI Lecture Number 8

    87/100

    Process outside the bus Process outside the message bus/subsystem if you

    can Let the bus focus on delivering the goods.

    Too much processing time in the bus will create Scalability problems Monitoring problems Possibly interoperability problems (especially when using

    proprietary technology/components)

    Process with components that are Flexible easy to get at (and change) interoperable (if possible) and contain reusable business logic (if possible)

  • 7/27/2019 BI Lecture Number 8

    88/100

    Problem #3Math we didnt do

    We didnt do enough math up front.

    We didnt plan for failure/growth.

    The messages moved slower than anticipated. The message processing took more time than

    expected.

    The amount of data was larger than expected.

  • 7/27/2019 BI Lecture Number 8

    89/100

    Do the math and ask the tough?s

    How much time its going to take to get a messagefrom A to B Test that estimate early.

    Work with the business analysts to figure out howmany messages need to be moved. Make volume estimates part of the non-functional

    requirements gathering process. Check that against the existing databases if possible.

    How much data needs to be packaged, shipped,processed, stored? Design the messages and calculate the size of the overall

    message (XML and all). Calculate the rate and add up the total volume.

  • 7/27/2019 BI Lecture Number 8

    90/100

    and pad your answer!

    Do you have room to spare?? Can the messaging system handle that (on both ends)?

    Can the consuming database handle that? Can the hardware and network handle that?

    Anticipate failure What happens if something/anything goes down for an

    hour?

    What happens if you go down for a day? What happens if you have unexpected growth?

  • 7/27/2019 BI Lecture Number 8

    91/100

    Problem #4 Exception

    handling wasnt More considerations for failover and redundancy

    Versus Web application

    We did not plan on downtime

    Unplanned system issues Planned outages

    We didnt build in enough redundancy Load balancing and Failover were both after thoughts

    Allmessages always correct allthe time (NOT) At first, we had no proper dead letter queuing No proper exception processing

    No means to properly see and react to issues Many more points of failure and potential issues More widely distributed

    D i l d b l i &

  • 7/27/2019 BI Lecture Number 8

    92/100

    Design load balancing &failover upfront

    Load balancing and failover must be accommodated

    Like security, you need a multi-layered approach

    Hardware (like Big IP) Redundant message bus/message servers

    Processing components

    Database

    EAI system throttling

    How are you going to kick over to the failoversystems (and return to regular systems)? Without losing messages

    Without causing timing problems in message deliver/receipt

  • 7/27/2019 BI Lecture Number 8

    93/100

    Throttling

    Throttling limits ("throttles") the number ofrequests it will respond to within a specified

    period of time. Limits congestion.

    Built into most good EAI solutions today.

    Often overlooked and not used. Used in messaging systems to ensure that

    no one part of the system is driven beyond itscapacity or performance efficiently.

    Th ttli t th t ti l

  • 7/27/2019 BI Lecture Number 8

    94/100

    Throttling at the potentialcongestion point

    Distribution Systems

    DBE

    Message Poll

    & DecorationQueue Message

    bundler

    E

    Validation &

    RequestProcessing

    (VRP)

    DBRecving

    Queue

    Problem

    Message

    !

    WAN WAN

    x 25

    Dead Letter

    Queue

    DB Ready

    Queues

    Retry Logic

    Throttle points.Potentially lots of messages,especially if the WAN goesdown

    Congestion point

  • 7/27/2019 BI Lecture Number 8

    95/100

    Space, space and more space

    Plan on extra space for failure A place for queued messages to sit if something goes down

    Space in the DB or space in the message channels orboth

    Consider the time lags for getting additional hardwarebought, installed, and up and running

    Plan on extra space for logs

    You are going to want to keep log files around for a while. Some problems take time to manifest to a point of

    awareness.

    Devise an automated archive/clean up for logs.

    Nonot all EAI systems provide log clean up utilities.

  • 7/27/2019 BI Lecture Number 8

    96/100

    96

    Process

    Integration Level (2)

    Application

    Data

    ????? ????

    Fundamental Integration Levels

    Electronic Business =Zero-Latency Business {Gartner}

  • 7/27/2019 BI Lecture Number 8

    97/100

    WarehouseMgmt.

    ACCOUNTNO.

    SHEETNO.

    ACCOUNTSRECEIVABLE LEDGER

    DATE INVOICENUMBER/DESCRIPTION CHARGES CREDITS BALANCE

    BALANCEFORWARD

    ACCOUNTNO.

    SHEETNO.

    ACCOUNTSRECEIVABLE LEDGER

    DATE INVOICENUMBER/DESCRIPTION CHARGES CREDITS BALANCE

    BALANCEFORWAR D

    Logistics

    Order Entry

    Financials

    Bank

    Parcel Service

    Call Center

    E-Commerce Portal

    Customer

    The Enterprise

    1) I want to buy it.

    2) I dont have it yet!

    3) When are you goingto get it delivered?

    4) Please pay

    for it.

    B2B

    B2B

    B2CA2A

    Zero-Latency Business {Gartner}

  • 7/27/2019 BI Lecture Number 8

    98/100

    98

    Zero or Low Data Latency

    Just the old concept of timeliness with an additionaltwist

    Refers to the real time (zero) or near-real time (low)transfer of information between applications

    Zero data latency tends to imply transactionalsolutions, while low data latency or near-real time

    allows for assured delivery (messaging) options Low data latency within the enterprise is no longer

    enough; it must be enabled across the enterpriseperimeter

  • 7/27/2019 BI Lecture Number 8

    99/100

    99

    An Affirmation; Modeling theProblem

    Adding EAI oriented solutions to the problem of data availabilitydoes not reduce the modeling effort required it intensifies it

    Each application view of the data must be defined andunderstood

    Additionally the dependencies involved in sharing andtransferring information must be supported

  • 7/27/2019 BI Lecture Number 8

    100/100

    Modes of Integration? AFunctional View!

    Data Consistency

    Multistep Process

    Composite Application

    Batch or immediatetransfers

    Multiple processes Multiple steps One-way, asynchronous

    interactions (loosely

    coupled) Systems are physically

    and logically independent

    Manual or straight-throughflows

    Batch or immediate,

    individual transfers One business process Multiple steps One-way, asynchronous

    interactions (looselycoupled)

    Systems are physically andlogically independent

    Potential long transactions

    Immediate interaction One business process One step Two-way, synchronous

    interactions (Tightly Coupled) Systems are physically and

    logically dependent Usually a client server

    (request reply) interaction