When Architecture Meets Data

endava.com

QUALITY. PRODUCTIVITY. INNOVATION.

When Architecture Meets DataSelf-Aware Systems for an Imperfect World

Eoin WoodsIEEE Software Experts Summit

Beijing, October 2015

2

When Architecture Meets Data

Self-Aware Systems for an Imperfect World

Defining Architecture Architecture Today Where Architecture is Going Self-Adaptive Systems Examples Implications For Us All

QUALITY. PRODUCTIVITY. INNOVATION.

3

3

To begin: what is “architecture”?

Many uses of the term “architecture”For our purposes:

Software Architecture is the set of software related elements that comprise a software-intensive system, their relationships, and the set of design decisions that allow the system to meet its system quality requirements (“non-functionals”)

Software architecture tends to be the decisions that are difficult to change once implemented [Fowler]

QUALITY. PRODUCTIVITY. INNOVATION.

4

4

Architecture Today

Software architecture today can be characterised as:

• Static yet Ephemeral• Set in stone, difficult to discern in the running system

• Depends on “Up-Front” Platform Design• The least agile part is the runtime platform design

• Difficult to change• Architectural change is a manual, time-consuming affair

QUALITY. PRODUCTIVITY. INNOVATION.

5

5

Architecture – Static yet Ephemeral

QUALITY. PRODUCTIVITY. INNOVATION.

uml-diagrams.org

http://msdn.microsoft.com

6

6

Architecture – Big Platform Up Front

QUALITY. PRODUCTIVITY. INNOVATION.

http://www.clipartpanda.com/http://kenwoodrecords.com/http://www.aberdeeninc.com/http://www.tbngconsulting.com/

7

7

Architecture – Difficult to Change

QUALITY. PRODUCTIVITY. INNOVATION.

http://pubs.opengroup.org/

https://www.thehouseshop.com

8

8

Where Architecture is Going

Where do current trends point for architecture?

QUALITY. PRODUCTIVITY. INNOVATION.

Data Driven

Dynamic

http://daily.captaindash.com/

9

9

Learning from Infrastructure

Infrastructure is undergoing a revolution

• Static, device centric, manual change, request based… to dynamic, virtualised, automated, policy based

• Enabled by the “API for Infrastructure” – “SDx”QUALITY. PRODUCTIVITY. INNOVATION.

http://www.cxotoday.com/story/cisco-vmware-deliver-next-generation-cloud-infrastructure/http://www.bbc.co.uk/news/business-22879160

10

10

Democratisation of Data

Data and analytics are also undergoing major change

• Centralised very expensive warehouses… to scalable open source and SaaS analysis platforms

• Platform and algorithms becoming accessible commodities• Allow predictive analytics on large data sets

QUALITY. PRODUCTIVITY. INNOVATION.http://www.iconshut.com/http://www.dbtekpro.com/2014/10/21/scaling-data-apps-cloud/

11

11

Analytics Providing System Insight

Analytics possibilities• Descriptive – what has happened so far?• Predictive – what is likely in the future?• Prescriptive – how do we optimise in response?

Example using Predictive Analytics• Transaction profiling of system activity metrics• Machine learning to “teach” the system what

peaks in activity look like• Resulting model predicting future activity peaks

QUALITY. PRODUCTIVITY. INNOVATION.

https://azure.microsoft.com

12

12

Impact on Software Architecture?

What could the effect be on software architecture?

• SDx gives measurement and control of platform• Big data allows insight into system operation• SDx + Big Data == dynamic platform• Where does this leave the software architecture?• The weak link?

QUALITY. PRODUCTIVITY. INNOVATION.

Picture: infralayer – analytical controller – Software architecture static on top

13

13

Impact on Our Systems

• SDx + Analytics == dynamic platform• Software architecture is still static – left behind?

QUALITY. PRODUCTIVITY. INNOVATION.

DynamicInfrastructure

Platform

Analytics Based

Controller

Observations

Changes

http:

//m

sdn.

mic

roso

ft.co

m

14

14

• What if we had an API for our architecture?• Needs first class architecture in the code

• SDI + Analytics + SDA == dynamic system

ArchitectureAPI

Control

The Missing API - Architecture

QUALITY. PRODUCTIVITY. INNOVATION.

PlatformAPI

AnalyticsBased

Controller

Control

DynamicInfrastructure

Platform

Measurements

15

Not a new idea

1999 – IEEE Intelligent Systems Article• An Architecture Based Approach to Self-Adaptive Systems• Oreizy, Gorlick, Taylor, Heimbigner, Johnson, Medvidovic,

Quilici, Rosenblum and Wolf

2009 – QoSA Conference theme• Architectures for Self-Adaptive Software Systems

Research still going on today• Recent survey paper “A survey on engineering

approaches for self-adaptive systems” by Krupitzer, Roth, VanSyckel, Schiele, and Becker

Self-Adaptive Systems

QUALITY. PRODUCTIVITY. INNOVATION.

16

16

Self-Adaptive Systems

QUALITY. PRODUCTIVITY. INNOVATION.

Evolution

Maintain Integrity

Make changes; collect observations

Adaptation Evaluate Operation

Plan Change

Apply Change

Model

(Adapted fromOreizy 99)

System

17

17

Self-Adaptive Systems

QUALITY. PRODUCTIVITY. INNOVATION.

ConditionalStatements

OnlineAlgorithms

GenericAlgorithms

AlgorithmSelection

GeneticAlgorithms &

Machine Learning

(Adapted fromOreizy 99)

18

18

What problems might this solve?

We could have systems that …

• Auto-scale reactively• Add capacity when load increases (we have this today)

• Intelligently auto-scale predictively• Adjust capacity for predicted peaks and troughs

• Change their functional behaviour• Predictively replace processing components to meet

predicted requirements

QUALITY. PRODUCTIVITY. INNOVATION.

19

19

Risk Estimator

An Example

QUALITY. PRODUCTIVITY. INNOVATION.

Transaction Processor

Request Handler

Demand

Message Bus

Risk Calculator

Risk Dashboard

Performance Analytics

PredictiveControllerRisk Estimator

Risk Calculator

20

20

An API for Architecture

Controlling our architecture would involve:• Knowing the current structure (a model)• Adding & removing elements and connectors• Being able to “rewire” elements & connectors• Constraints to reject invalid structures• API implemented by the platform (PaaS)

Could extend the programming language or be an external language – but in the code & “compiled”• Maybe related to existing “ADLs”?

QUALITY. PRODUCTIVITY. INNOVATION.

21

21

An API for Architecture

Our example using an imaginary API

QUALITY. PRODUCTIVITY. INNOVATION.

arch = getArchitecture("MyArch")riskCalc = arch.getElement("RiskCalculator")msgBus = calc.getConnector("inboundMsg")dashBrd = calc.getConnector("dashboard")riskEst = arch.getElement("RiskEstimator")riskEst.setConnector("dashboard", dashBrd)riskEst.setConnector("inboundMsg", msgBus)riskCalc.setConnector("dashboard", null)riskCalc.setConnector("inboundMsg", null)riskCalc.deactivate()riskEst.activate()arch.commitChanges()

22

22

Implications

What does this mean for research and practice?

• More cross disciplinary research than is normal• Data, machine learning, adaptive systems, middleware, …

• More industry and academic cooperation• Industry has the systems, academia the models

• First class runtime software architecture• You need a model of the architecture to change it• No (general) runtime architecture models in use today• The challenge Software Architecture research needs?

QUALITY. PRODUCTIVITY. INNOVATION.

23

23

Our Future Systems Architectures

QUALITY. PRODUCTIVITY. INNOVATION.

Handler

Processor Calculator

Store

SDx Infrastructure System Analytics Architecture API Self-Adaptive Systems

Dynamic Data-Driven

Systems

24

Thank you

QUALITY. PRODUCTIVITY. INNOVATION.

Eoin WoodsEndava

eoin.woods@endava.com+44 207 367 1000en_ewoods