MuCon 2015 - Microservices in Integration Architecture

Where is Integration in a Microservices world?

Kim Clark 9th October

MuCon 2015

What do we mean by “Integration Architecture”? How data and function are shared between applications within and across enterprise boundaries

1

Application Application

Application

Application

Application

Application

Point to point Hub and

spoke

Service Oriented

Architecture Enterprise Service Bus

Messaging

APIs

Managed File Transfer

Business to Business

(B2B)

Extract Transform

Load

Connectors / Adapters

Mapping / Transformation

Enterprise Application Integration

Application

Microservice application

µService

µService µService

µService

Large enterprise landscapes are horrible! •  Mergers and acquisitions

•  Multiple systems of record for the same data types •  Non-correlating duplicates •  Account centric vs. Customer centric

•  Version locked systems •  There are always 3 versions of a system

–  The one you’re migrating from –  The one you’re on now –  The one you’re migrating too

•  Months to add a fix pack

•  Very long lived entities •  Mortgages, deeds, contracts, pensions, insurance policies

•  Evolved complexity •  Decades of business logic embedded alongside the data

•  Business Partner interfaces •  Exceptions are the rule •  Too important to deprecate

2

Integration architecture subsets

External (API) Exposure

Enterprise (Service/API)

Exposure

Batch-based Hub and Spoke

Batch Transfer Direct

Batch Transfer B2B

Event-based Hub and Spoke

Event-based Point to Point

Real-time Direct Access

External Event Management

Loca

l

Ent

erpr

ise

Pub

lic/P

artn

er

Batch-based Event-based Connected

What does a large scale integration landscape look like

Systems of record •  Older technology •  Harder to change •  Challenging integration points

Integration Hub

Integration Hub Adapter Adapter

“Systems of Engagement” Applications

Exposure Gateway (internal)

“Systems of Record”

Applications

Mature large enterprise (simplified)

Exposure Gateway (external)

Adapter Adapter “Hub and Spoke”

integration

“Enterprise Service Bus?”

“Web API Gateway”

Systems of engagement •  Modern languages/runtimes •  Agile updates •  Simple modern connectivity

Confusions that arise around microservices

5

Obligatory “What is a microservice” slide

Monolithic application

Microservices application

Silo

Microservice (component)



Agility Scalability Resilience

Microservice component

Common misconception resulting from the term “microservice”

Monolithic application Microservices application

Exposed services/APIs



Exposed services/APIs

Silo component

Microservices are fine grained WSDL operations

APIs are microservices

The “service” in “microservice” refers to the granularity of the components, not the granularity of the exposed interfaces

x 1 x 3

x 4 x 4

Is “microservices architecture” is really “micro-component architecture”?

Application

SOA relates to enterprise service exposure

Application Application Application

Service oriented architecture (SOA) and microservices architecture relate to different scopes


µService

µService µService

µService

Microservices relate to application architecture

However, even this simple distinction can be contentious depending on your definition of SOA

What was SOA really about? Integration or Components? “SOA is about how to achieve integration often to aging complex back end systems in order to expose services” In this case SOA is primarily a connectivity p rob lem w i th l i t t l e re la t i onsh ip to microservices architecture and certainly at a different scope.

“SOA is about re-factoring your IT landscape into components that better align with the business needs and expose the services that it needs” Here the connectivity problem is pushed down to the applications and the focus of SOA is on realignment of to the business needs. The service components look more like applications, and we might consider microservices as “more granular SOA”, or even “SOA done right”.

System of Record

Integration Hub

Adapter Adapter

Exposure Gateway

Exposed Services/APIs

System of Record

System of Record

Adapter

Exposed Services/APIs

System of Record

Adapter

Service Component

Exposure Gateway

9

Service Component

Service Component

Why such split opinions on microservices vs SOA?

10

Integration Hub


Enga

gem

ent

App

licat

ions

SaaS

A

pplic

atio

n

Bus

ines

s Pa

rtne

r

Sy

stem

s of

R

ecor

d

Bus

ines

s Pa

rtne

r


Mature large enterprise Microservices are just one style of application

Exposing services is an integration and data challenge

Green field online start-up Much of landscape could be microservice based

The landscape is as (micro)service oriented architecture



µService

µService µService

µService

Exposure Gateway (internal) µService

µService

µService

µService

µService

µService µService

µService

µService

µService

µService

Ente

rpris

e B

ound

ary

Ente

rpris

e B

ound

ary

Adapter Adapter

SOA principles get a second chance with microservices

11

Challenges of SOA era Advantages in microservices era

Funding Poor funding model. No project wanted to pay for re-usable services.

API Economy provides potential funding mechanisms. Costs of integration reducing.

Integration Connectivity was hard. Standards were immature. Tools were just emerging.

Integration standards and tools improved. Modern APIs often provided as out of the box.

Data Data landscape disjoined. Duplication common. Function embedded with data.

SOA/MDM initiatives rationalised some landscapes. New data techniques to handle big data more flexibly.

Requirements Hard to guess the perfect service model. Business needs constantly changing.

Microservices lend themselves to lightweight agile techniques enabling more iterative requirements discovery.

Result Integration led services. Poor re-use.

Business led API requirements. Creation on demand.

Levels of API/service exposure Just because it’s REST/HTTP, doesn’t mean it’s all the same

•  How different is the APIs exposure style at each level?

•  How strong should the application boundaries be?

12


Microservice

Microservice

Within-application

Enterprise-wide

Internet exposed

API/service exposure scope

Microservice

Microservice Microservice

Microservice

Microservices inter-communication

Aim is decoupling for robustness Messaging where possible •  Lightweight messaging

(e.g. AMQP, Kafka) •  Publish/subscribe •  Eventual consistency Direct calls where necessary Lightweight protocols (e.g. JSON/HTTP) •  Load balancing/scaling via service

discovery •  Circuit breaker •  Caching


Microservice

Subscribe

JSON /HTTP

Microservice

Microservice

Message Hub

API

Microservice Publish

API

JSON /HTTP Publish

Service Discovery

Application

Internal vs. External/Partner exposure – comparing at the level of functionality/capability

Firewall

Canonical Integration Data Translation

Routing/Distribution

Composition

Exposure

Traffic Management - Basic

Security - Basic

Decoupling

Visibility - Basic

Partner Management

Account based access

Accounting/Billing

Self Administration

Service level management

Discovery - Basic

Exposure

Traffic Management

Security

Decoupling

Visibility

Discovery

API/Service Gateway

API/Service Gateway

Canonical Integration Data Translation - Basic

Routing/Distribution - Basic

Composition - Basic

There may be only a handful of well understood internal

consumer applications

There may 10s of external consumer applications …and there could be hundreds of “experimenters”

Do todays systems of engagement become tomorrows systems of record?

Enga

gem

ent

App

licat

ion

Logic

Sy

stem

s of

R

ecor

d

Presentation

Data

Logic

Presentation

Data

Logic

1

2

3

1 New engagement application •  Provides new, engaging, agile presentation tier •  Cache/subset of data, for temporary and/or low

latency storage •  Integrates with enterprise services/APIs easily.

2 Aging engagement application •  Presentation less attractive and agile •  Data model broadened, with increased integrity

as now used by other applications •  Struggles to integrate with other systems

3 Becomes system of record •  Presentation tier no longer used. Logic/data

now accessed and/or replicated to new systems of engagement.

•  Data model stabilised, and hard to change, with increased integrity as now used by other applications

•  Requires external tools to integrate with other systems.

1.  How long before your microservice application becomes one of the “legacy” systems?

2.  How long before you have more than one mircoservices platform?

Exposure Gateway

Heterogeneous on the inside, homogeneous on the outside

Freedom to choose runtimes, languages, datastores etc. may be a birthright with microservices, but don’t let it become a millstone. •  Choose runtimes based on their properties,

not the preferences/skills of the team •  Look for natural convergence on runtimes,

then embrace it.

Seek to push maximum commonality into the framework surrounding the microservices: •  Scaling •  Workload distribution •  Resilience •  Service discovery •  Deployment orchestration •  Messaging •  Logging •  Testing

16


Microservice (Node.js)

Microservice (Java +

MongoDB)

Microservice (Go +

CloudantDB)

But what is an application boundary in an microservices world?

Similar to application boundaries today because •  Group related functionality •  Enable more coarse grained

ownership and responsibility (in addition to full ownership at the microservice level)

Different because •  Application sub-components

(microservices) are truly independent.

•  Application boundaries can be easily changed – it’s primarily about the right ownership

•  Application boundaries can spread across network boundaries (e.g. between clouds)…but should they!

17

SaaS

A

pplic

atio

n

Bus

ines

s Pa

rtne

r


µService µService

µService

µService

µService

µService µService

µService

µService

µService

µService

Ente

rpris

e B

ound

ary

Traditional Engagement Application

System of

record

System of

record

SaaS Application

Decentralising service/API exposure

In a perfect world…

•  Applications self-administer exposure of their own APIs

•  API monitoring/diagnostics would be gathered across the organisation

•  Security models could be implemented more consistently.

•  No additional team required to enable integration

•  Application logic is firmly seated with the application teams

18

Business Partner

Ent

erpr

ise

Bou

ndar

y

Public API

Private API API Gateway

Microservice Engagement Application

In reality, it will be a long haul for many organisations…

19

Syst

ems

of

Rec

ord

Integration Hub


Enga

gem

ent

App

licat

ions

Mic

rose

rvic

e ap

plic

atio

ns

SaaS

App

licat

ions

(e

xter

nal)

Adapter

Externally Exposed Services/APIs

Exposure Gateway (internal)


Bus

ines

s Pa

rtne

rs

Caution with comparisons (or “how to start an argument”!)

Comparisons that make some level of sense

Microservice architecture vs. Service oriented architecture Trying to achieve many of the same things (enable the creation of re-usable business functions as isolated components), but perhaps a different scale. SOA evolved to become focused on the enterprise scale (though some might say that wasn’t the original intent). Microservices, is primarily at the application scale.

Microservice component vs. Service component Both describe the actual implementation of a business function. They differ primarily in granularity, but also arguably in maintainability, agility etc.

Confusing comparisons

Microservices vs. SOA Microservices are components. SOA is an architecture. What you’re probably trying to compare is Microservices Architecture with SOA (see above).

Microservices vs. APIs APIs are a interface. A mechanism for exposing business function. Microservices are the component architecture used to implement those business functions. It makes no sense to directly compare them.

Microservice vs. Service “Service” means different things to different people. Needs to be qualified. Service Implementation? Service exposure? Service definition? The same could be said for microservice. Need to be specific.

20

Technology

MuCon 2015 - Microservices in Integration Architecture