Enterprise Service BusInnovation Project for FY15

Presented by:

Mike Thomas – Sr. ArchitectAdministrative Technology Services

Enterprise Applications

HUIT Sr. Leadership Team Meeting - June 18, 2015

ESB Innovation Project

Team Members:

• Bill Brickman (Data Warehouse)

• Anita Cutting (FinProc)

• Leo Dai (Grants)

• Sreeni Gunnala (FinProc)

• Lisa Justiniano (Sponsor)

• John Marks (Grants)

• Shanti Muppirala (Peoplesoft)

• Joe Rickert (FinProc)

• John Shen (Peoplesoft)

• Brian Sullivan (Data Warehouse)

• Curt Springer (Student Financials)

• Karen Stelle (Proj Mgr)

• Mike Thomas (Sr. Architect)

Pilot Goals:

• Get experience with ESBs in general and with 2 or 3 specific ESB products

• Determine if an ESB would benefit ATS

• Evaluate ESBs to identify one that best fits our needs

• Get our feet wet with Amazon Web Services

Timeline:

POC: April 2014 - June 2014

Pilot: July 2014 - present

Pilot Budget: $200 per month

2

Enterprise Service Bus (ESB) - Wikipedia Definition

An enterprise service bus (ESB) is a software architecture model used for designing and implementing communication between mutually interacting software applications in a service-oriented architecture (SOA).

As a software architectural model for distributed computing, an ESB promotes agility and flexibility with regard to communication between applications.

Its primary use is in enterprise application integration (EAI) of heterogeneous and complex landscapes.

3

Problem Statement

• Data integration / data Interoperability between our applications has become a quagmire of technology and methodology.

• Each integration is hand-crafted and designed for a single purpose.

• Each development team selects the technology and approach that fits best with their products and skill sets, without consideration for standardization, consistency or scalability.

• The same pieces of data are being included in a variety of different integrations because there is no catalog or directory of available data integrations.

4

Enterprise Service Bus Software Components

Two Major Components:

1. ESB

– provides web services

– provides integrations using common protocols like SSH, JDBC, SMTP

2. Message Broker

– pass arbitrary data

– two models:

• point-to-point queues

• publish and subscribe topics

MessageBroker

Application A Application B

ESB

5

Advantages of an ESB

• Standards-based integration mechanism instead of one-off solutions

• Flexible integration options: anything that listens on a TCP port

• Faster development: configuration instead of code

• Improved discovery and transparency

• Centralized security

• Data / protocol transformations

• Data fan-out

• Decoupling systems

• Batch processing can become transactional

System A System B System C

System A

System B

System C

Publish / Subscribe Topic

VS

6

Software Stack

We built a HA stack:

• 2 x AZ

• 2 x ServiceMix ESB

• 2 x ActiveMQ Msg Broker

• 2 x (or 4x) ELB

• Not shown:

• 1 x VPC

• 4 x network subnets

• 3 x security groups

• 1 x direct connect RDS Oracle RDS Oraclehot standby

ESB Load Balancer

ServiceMix ServiceMix

ActiveMQ ActiveMQ

Messaging Load Balancer

us-east-1a availability zone us-east-1b availability zone

We chose:

• Apache ServiceMix & ActiveMQ

• Free, active community

7

We built four integrations

1. General Ledger Chart Of Account Validator:

• Provide standards-based mechanism to validate chart of accounts values

• Existing validator PL/SQL JDBC Web Service

2. Eureka (Online Training System) to PeopleSoft Feed:

• Push course completion data to PeopleSoft

• Standalone ServiceMix Queue PeopleSoft Web Service

3. FRAP Data Load:

• Move sponsored financial reporting data from Windows laptop to GMAS

• FRAP control laptop Queue JDBC GMAS DB

4. Staff Terminations:

• Make terminations feed available to interested parties

• DB JDBC Queue, using another Queue to maintain batch state

8

What Challenges Did We Encounter?

• Amazon OpenStack Amazon

• ServiceMix Fabric8 ServiceMix

• Amazon Direct Connect, asymmetric routing, and Elastic Load Balancers

• ActiveMQ authentication is all or nothing

9

100% ServiceMix and its components: Camel, CXF

90% ActiveMQ

80% Amazon Web Services (in many cases their first experience)

Every developer who hadn't already taken the 1 day AWS course took it.

What did the developers learn?

Selected Developer Survey Comments

Learning about the different services in ServiceMix and their functionality was a very positive experience. I feel it could come in handy when prototyping integration projects in the future.

I learned a good amount about ESB and how to jump out to custom code, and what a message broker is all about, good hands-on experience.

I learned about redundancy and failover in the cloud.

Hosting/consuming web services in Servicemix, using OSGi blueprint with Camel for routing data and XSL ... ActiveMQ for data transportation, AWS.

I learned to create Camel routes using blueprint to work with ActiveMQ ... I created a Java bundle ... performed the transformation using XSL.

I was assigned to work on installing SSL keys for the load balancers ... this is a great introduction to AWS for me. I had no prior experience with AWS.

Very easy to learn and start developing bundles, fewer lines of code to create an integration, more configuration rather than coding, loosely coupled systems

For me it was an intro to the new products/tools, infrastructure and terminology. This technology is in its infancy.

10

Developer Conclusions

1. ServiceMix is suitable when embedded for the exclusive use of a single application, but what developers want is an ESB deployed as a centralized service. ServiceMix lacks:

• security flexible enough to truly isolate integrations from one and other

• code isolation to avoid naming collisions and version collisions

• GUI for admin actions like adding users/groups, deploying code, browsing logs

• good debugging support

• registry / discovery of services

• rate limiting (to guarantee SLAs)

• analytics

2. Given choice of a single product, 2/3rds preferred the message broker.

3. These are powerful and useful tools and we want one.

11

What is Next?

• Wind down our infrastructure in AWS

• Continue to use the products:

• We can use standalone ServiceMix to ease integrations, e.g. COA Validator, maybe FRAP

• We can embed Camel directly in Java applications to get the nice terse integration language without the overhead of a service

• Data Interoperability Initiative led by Jon Saperia

• Jon joined ESB project at the tail end

• ATS Participants: Bill Brickman, Sreeni Gunnala, Curt Springer

• Much larger scope: includes data governance, master data mgt., etc.

• Becomes a request for an ITCRB Capital Project?

12

Thank You!

13