Presentation Master thesis defense

Professionaliteit door kwaliteit

Towards Distributed Information AccessVictor de GraaffNovember 18th, 2009

Towards Distributed Information Access

Possibilities and Implementation

Victor de Graaff

November 18th 2009

Professionality through quality


Master thesis defense

Towards Distributed Information Access

Possibilities and Implementation

2

Victor de GraaffUniversity of TwenteComputer ScienceSoftware Engineeringdr. Luís Ferreira Piresdr. ir. Marten van Sinderening. Gerke Stam, TSi Solutions

Student:University:Master:Track:Graduation Committee:



Introduction

Introduction Motivation Objectives Approach Overview

Requirements

Comparison of Integration Technologies

Application Integration Architectures

Choosing an Implementation

Proof of Concept

Final Remarks

1. Motivation

2. Objectives

3. Approach

4. Overview

3



What is an information broker?


Requirements




Proof of Concept

Final Remarks

4





Requirements




Proof of Concept

Final Remarks

5



Current situation


Requirements




Proof of Concept

Final Remarks

6





Requirements




Proof of Concept

Final Remarks

7



Main objective


Requirements




Proof of Concept

Final Remarks

To determine the most suitable architecture and technology to realize distributed access on real-time to

information on products.

Architecture Structure of the system.

Technology Software to support that structure.

Distributed Information comes from different places.

Real-time Information comes at the time it is needed.

8



Sub-objectives


Requirements




Proof of Concept

Final Remarks

To reach our main objective:1. To provide an overview of existing integration

technologies;

2. To provide an overview of architectural options to use these integration technologies.

To validate our main objective:1. To provide an overview of existing implementations of

the integration technologies and determine the most suitable one;

2. To create and test a prototype which provides an example implementation for the proposed architecture.

9



Approach


Requirements




Proof of Concept

Final Remarks

Step-by-step:

1. Requirements analysis

2. Literature survey of existing technologies

3. Identification of possible architectures

4. Comparison of possible architectures

5. Literature survey of existing implementations

6. Selection of the best implementation for this purpose

7. Implementation and testing

10



Overview


Requirements




Proof of Concept

Final Remarks

Introduction

Requirements




Proof of Concept

Final Remarks

11



Requirements

Introduction

Requirements Approach Stakeholders Use cases Functional requirements Non-functional requirements Weighting factors




Proof of Concept

Final Remarks

1. Approach

2. Stakeholders

3. Use cases

4. Functional requirements

5. Non-functional requirements

6. Weighting factors

12



Approach (1)

Introduction





Proof of Concept

Final Remarks

Input from three sources:

1.Information broker analysis

2.Questionnairea.CTOb.System architectsc.Developers

3.Discussions

13



Approach (2)

Introduction





Proof of Concept

Final Remarks

Four steps:

1.Identification of stakeholders

2.Identification of use cases

3.Identification of requirementsa.Functionalb.Non-functional

4.Assignment of weight factors

14



Stakeholders

Introduction





Proof of Concept

Final Remarks

• Consumers– Travel Agencies– Website developers

• Information broker– Project leaders– System architects– Developers– System administrators

• Producers– Tour operators– Insurance companies– Payment service providers– etc.

15



Use cases

Introduction





Proof of Concept

Final Remarks

16

Producer

Consumerretrieve available productsfor one provider

Information Broker

retrieve available information on products for several providers

request info

request info

provide info purchase/cancel (combined) product

initiate

direct to

corresponding producer(s)

handle purchase/cancellation

configure which producers are available

configure

configure which consumers are allowed to use/offer products

configure

retrieve information on previous product purchase

request info

request info

provide/enrich info



Functional requirements

Introduction





Proof of Concept

Final Remarks

1. Generic transportation of messagesa. Support to multiple transportsb. Support to multiple protocolsc. Transparent message exchange

2. Add existing applications easilya. External applicationsb. In-house applications

3. Routing

4. Global configuration

5. Security

17



Non-functional requirements

Introduction





Proof of Concept

Final Remarks

1. Open source

2. Ability to scale out

3. Ability to upgrade the system without a complete shutdown

4. Reliable messaging

5. Independent of implementation language

6. Operation time

7. Message speed

8. Fail fast adequacy

18



Weighting factors (1)

Introduction





Proof of Concept

Final Remarks

19

1. Generic transportation of messagesa. Support to multiple transportsb. Support to multiple protocolsc. Transparent message exchange

2. Add existing applications easilya. External applicationsb. In-house applications

3. Routing

4. Global configuration

5. Security

444

53

4

1

5



Weighting factors (2)

Introduction





Proof of Concept

Final Remarks

20

1. Open source

2. Scale out

3. Upgrade without a complete shutdown

4. Reliable messaging

5. Independent of impl. language

6. Operation time

7. Message speed

8. Fail fast adequacy

4

5

4

3

5

4

4

3




Introduction

Requirements

Comparison of Integration Technologies Approach Point-to-Point Hub-and-Spoke EMB ESB Conclusion



Proof of Concept

Final Remarks

1. Approach

2. Point-to-Point

3. Hub-and-Spoke

4. Enterprise Message Bus

5. Enterprise Service Bus

6. Conclusion

2121



Introduction

Requirements




Proof of Concept

Final Remarks

Approach

22

C2C1

B2B1

A2A1



Introduction

Requirements




Proof of Concept

Final Remarks

Point-to-Point integration

23

C2C1

B2B1

A2A1



C2C1

B2B1

A2A1

Introduction

Requirements

Comparison of Integration Technologies Approach Point-to-Point Hub-and- Spoke EMB ESB Conclusion



Proof of Concept

Final Remarks

Single hub Hub-and-Spoke

24

N



Introduction

Requirements

Comparison of Integration Technologies Approach Point-to-Point Hub-and- Spoke EMB ESB Conclusion



Proof of Concept

Final Remarks

Multi hub Hub-and-Spoke

25

C2C1

B2B1

A2A1



C2C1

B2B1

A2A1

Introduction

Requirements




Proof of Concept

Final Remarks

Enterprise Message Bus

26

EMB

Adapter

A A

A

A A

A

A

A



ICIC

Introduction

Requirements




Proof of Concept

Final Remarks

Enterprise Service Bus

27

C2C1

B2B1

A2A1

ESB

IC

Intelligent Connector

ICIC

ICIC

ICIC ICIC

ICIC



Introduction

Requirements




Proof of Concept

Final Remarks

Conclusion

28

C2C1

B2B1

A2A1

C2C1

B2B1

A2A1

N

C2C1

B2B1

A2A1

EMBA A

AA

AA

C2C1

B2B1

A2A1

ESB

Enterprise Message Bus Enterprise Service Bus

Point-to-Point Hub-and-Spoke

IC

IC

ICIC

IC

IC




Introduction

Requirements


Application Integration Architectures Hub-and-Spoke EMB ESB Conclusion


Proof of Concept

Final Remarks

1. Hub-and-Spoke Architecture

2. EMB Architectures

3. ESB Architectures

4. Conclusion

29



Consumer B

Consumer C

Consumer A

Consumers

Producer A

Producer B

Producer C

Producers

Introduction

Requirements


Application Integration Architectures Hub-and- Spoke EMB ESB Conclusion


Proof of Concept

Final Remarks

Hub-and-Spoke Architecture

30

Information Broker

Services



Consumer B

Consumer C

Consumer A

Consumers

Introduction

Requirements




Proof of Concept

Final Remarks

EMB Architectures (1)

31

Load BalancerSe

rvic

es

Producer A

Producer B

Producer C

EMB in combination with a load balancer:

Ente

rpris

e M

essa

ge B

us

Producers Information Broker



Ente

rpris

e M

essa

ge B

usProducer A

Producer B

Producer C

Consumer B

Consumer C

Consumer A

ConsumersProducers Information Broker

Introduction

Requirements




Proof of Concept

Final Remarks


32

Service 3

Service 2

Service 1

Two EMBs:

Ente

rpris

e M

essa

ge B

us



Introduction

Requirements




Proof of Concept

Final Remarks


33

Producers

Information Broker

Producer A Producer B Producer C Consumer A Consumer B Consumer C

Consumers

Service 1 Service 2 Service 3

One EMB:

Enterprise Message Bus



Introduction

Requirements




Proof of Concept

Final Remarks

ESB Architectures (1)

34

Consumer B

Consumer C

Consumer A

Consumers

Load BalancerSe

rvic

es

Producer A

Producer B

Producer C

Ente

rpris

e Se

rvic

e Bu

s

Producers Information Broker

ESB in combination with a load balancer:



Ente

rpris

e Se

rvic

e Bu

s

Introduction

Requirements




Proof of Concept

Final Remarks


35

Producer A

Producer B

Producer C

Consumer B

Consumer C

Consumer A

ConsumersProducers Information Broker

Service 3

Service 2

Service 1

Ente

rpris

e Se

rvic

e Bu

s

Two ESBs:



Introduction

Requirements




Proof of Concept

Final Remarks


36

Producers

Information Broker

Producer A Producer B Producer C Consumer A Consumer B Consumer C

Consumers

Service 1 Service 2 Service 3

Enterprise Service Bus

One ESB:



Introduction

Requirements




Proof of Concept

Final Remarks

ESB with one bus.

Advantages over EMB: Transparent message exchange Easy addition of external applications Resilience to changes

Additional advantages over hub-and-spoke: Active open source projects Scaling out capabilities Upgrading of system without shutdown

Conclusion (1)

37



Introduction

Requirements




Proof of Concept

Final Remarks

Conclusion (2)

ESB with one bus.

Advantage over ESB with two buses:• Easy addition of internal applications

Advantage over ESB with a load balancer:• Easy addition of external applications

38




Introduction

Requirements



Choosing an Implementation Approach Mule ServiceMix OpenESB Comparison

Proof of Concept

Final Remarks

1. Approach

2. Mule

3. ServiceMix

4. OpenESB

5. Comparison

39



Approach

Introduction

Requirements




Proof of Concept

Final Remarks

Analysis of three ESB implementations:

1. Tool support2. Components3. Hello World example4. Open requirements:

a. Global configurationb. Message speedc. Fail fast adequacy

40



Mule

Introduction

Requirements




Proof of Concept

Final Remarks

Decent tool support Mule IDE 2.0 Use of XML Schemas Lack of visual tools

More than sufficient components

Simple to create Hello World example

Open requirements: Global configuration through Mule Galaxy Claimed to perform better than other ESBs Failure behaviour can be configured

41



ServiceMix

Introduction

Requirements




Proof of Concept

Final Remarks

42

No specific tool support Use of standards


Very hard to create Hello World example Extensive use of Maven Examples in standard distribution

Open requirements: No global configuration application Worst performance of the three Apache Camel for process orchestration



OpenESB

Introduction

Requirements




Proof of Concept

Final Remarks

43

Very good tool support NetBeans Use of standards Graphical editor for configuration files


Many steps to create Hello World example

Open requirements: GlassFish Admin Console Better performance than ServiceMix Apache Camel for process orchestration



Comparison

Introduction

Requirements




Proof of Concept

Final Remarks

44

Requirement Weight Mule ServiceMix OpenESB

Global configuration 1 2 0 2

Message speed 4 2 0 1

Fail fast adequacy 3 2 2 2

Total 16 6 12



Proof of Concept

Introduction

Requirements




Proof of Concept Problem situation Basics of Mule Configuring Mule Testing environment Conclusion

Final Remarks

1. Problem situation

2. Basics of Mule

3. Configuring Mule

4. Testing environment

5. Conclusion

45



Problem situation (1)

Introduction

Requirements





Final Remarks

TSi's TravelSearch:

Offers search functionality to consumers

Searches TSi's own database

Database is updated at steady intervals

46



Problem situation (2)

Introduction

Requirements





Final Remarks

Drill-down search functionality:

1. Possible search criteria

2. Collection of search results

3. Specific information of one result

4. Booking code

47



Basics of Mule (1)

Introduction

Requirements





Final Remarks

48

Service

Transport Inboundrouter

Component Outboundrouter Transport

Exception listener



Basics of Mule (2)

Introduction

Requirements





Final Remarks

49

<<Endpoint>>HTTP

<<Endpoint>>JMS

<<Component>>POJO

Applicationor

component

T

TT

Inbound router Outbound router

The component's response issent to the outbound router

for endpoint dispatching

An inbound event gets routedto the component's entry point

through transformers

The component's response canbe returned to the caller if

the endpoint is synchronous

Applicationor

component



Basics of Mule (3)

Introduction

Requirements





Final Remarks

50

<<Endpoint>>HTTP

<<Endpoint>>JMS

<<Component>>POJO

Applicationor

component

Applicationor

component

T

TT

Inbound router Outbound router

Service

Transport Inboundrouter

Component Outboundrouter Transport



Configuring Mule (1)

Introduction

Requirements





Final Remarks

51

<<Endpoint>>HTTP

<<Endpoint>>dtsRequests

VM

<<Component>>WSProxyService

POJO

Consumer

ExternalServices

<<Endpoint>>dtsResponses

VM

<<Component>>DTSResponseAgg

POJO

T

T

Main service:



Configuring Mule (2)

Introduction

Requirements





Final Remarks

52

<<Endpoint>>dtsRequests

VM

<<Endpoint>>externalWebservice

HTTP

VMMessage T

<<Endpoint>>dtsResponses

VM

T

External service:



Testing environment

Introduction

Requirements





Final Remarks

Main service supported by: Java transformation SOAP to XML Java response aggregator

External service supported by: XSLT transformations

XML to provider request Provider response to XML

53



Conclusion

Introduction

Requirements





Final Remarks

54

1 10 100 10000

2000

4000

6000

8000

10000

12000

14000

16000

18000

20000

Concurrent requests

Aver

age

resp

onse

tim

e (m

s)



Final Remarks

Introduction

Requirements




Proof of Concept

Final Remarks Conclusions Future research

1. Conclusions

2. Future research

55



Conclusions

Introduction

Requirements




Proof of Concept


To determine the most suitable architecture and technology to realize distributed access on real-time to

information on products.

Requirements Four integration technologies Seven possible architectures ESB with one bus

Mule Prototype

56



Future research

Introduction

Requirements




Proof of Concept


Validation under more realistic circumstances

Transformation optimization

ESB performance comparison

57



Questions

58

Documents

Presentation Master thesis defense