Upload
others
View
4
Download
0
Embed Size (px)
Citation preview
Complex event flows indistributed systems
@berndruecker With thoughts from http://flowing.io@berndruecker | @martinschimak
3 common hypotheses I check today:
# Events decrease coupling# Orchestration needs to be avoided# Workflow engines are painful
Berlin, Germany
[email protected]@berndruecker
Bernd RueckerCo-founder and Developer Advocate ofCamunda
Simplified example:dash button
Photo by 0xF2, available under Creative Commons BY-ND 2.0 license. https://www.flickr.com/photos/0xf2/29873149904/
Three steps…
Who is involved? Some bounded contexts…
Checkout
Payment
Inventory
Shipment
(Micro-)services
Checkout
Payment
Inventory
Shipment
Autonomous (micro-)services
Checkout
Payment
Inventory
Shipment
Dedicated Application Processes
Dedicated infrastructure
Dedicated Development Teams
Events decrease coupling
Example
Checkout
Payment
Inventory
Shipment
The button blinks if we canship within 24 hours
Request/response: temporal coupling
Checkout
Payment
Inventory
Shipment
RequestResponse
The button blinks if we canship within 24 hours
Temporal decoupling with events and read models
Checkout
Payment
Inventory
Shipment
GoodStored
Read Model
GoodFetched
The button blinks if we canship within 24 hours
*Events are facts about what happened (in the past)
Events can decrease coupling*
*e.g. decentral data-management, read models, extract cross-cutting aspects
Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Order placed
Payment received
Goodsshipped
Goodsfetched
Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Order placed
Payment received
Goodsshipped
Goodsfetched
The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years.
https://martinfowler.com/articles/201701-event-driven.html
The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years.
https://martinfowler.com/articles/201701-event-driven.html
The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years.
https://martinfowler.com/articles/201701-event-driven.html
Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Order placed
Payment received
Goodsshipped
Goodsfetched
Fetch the goodsbefore thepayment
Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Order placed
Payment received
Goodsshipped
Goodsfetched
Fetch the goodsbefore thepayment
Customers canpay via invoice
…
Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Order placed
Payment received
Goodsshipped
Goodsfetched
Fetch the goodsbefore thepayment
Peer-to-peer event chains
Checkout
Payment
Inventory
Shipment
Fetch the goodsbefore thepayment
Goodsfetched
Order placed
Payment received
Goodsshipped
Photo by born1945, available under Creative Commons BY 2.0 license.
Order
Extract the end-to-end responsibility
Checkout
Payment
Inventory
Shipment
*Commands have an intent about what needs to happen in the future
Order placed
Paymentreceived
Retrievepayment
Commands help to avoid (complex)peer-to-peer event chains
Orchestration needs to be avoided
Smart ESB-like middleware
Checkout
Payment
Inventory
Shipment
Order
Order placed
Payment received
Goodfetched
Goodshipped
Dumb pipes
Checkout
Payment
Inventory
Shipment
Order
Smart endpoints and dumb pipes
Martin Fowler
Danger of god services?
Checkout
Order
A fewsmart god services tellanemic CRUD services what to do
Sam Newmann
Payment
Inventory
Shipment
Danger of god services?
Checkout
Payment
Inventory
Shipment
Order
A fewsmart god services tellanemic CRUD services what to do
Sam Newmann
A god service is only createdby bad API design!
Example
Order Payment
RetrievePayment
Example
Order PaymentCreditCard
RetrievePayment
Example
Order PaymentCreditCard
RetrievePayment
Rejected
Example
Order Payment
Client of dumb endpoints easily become a god services.
If the creditcard was
rejected, thecustomer canprovide new
details
CreditCard
RetrievePayment
RejectedRejected
Paymentfailed
Who is responsible to deal with problems?
Order Payment
If the credit card was
rejected, the customer can provide new
details
CreditCard
RetrievePayment
Rejected
Paymentreceived
Paymentfailed
Long-running execution
Order Payment
Clients of smart endpoints remains lean.
If the credit card was
rejected, the customer can provide new
details
CreditCard
RetrievePayment
Rejected
Paymentreceived
Smart endpoints are potentially long-running
Persist thing(Entity, Document, Actor, …)
State machine orworkflow engine
Typicalconcerns
DIY = effort, accidentalcomplexity
Scheduling, Versioning, operating, visibility, scalability, …
Handling State
Workflow engines are painful
Complex, proprietary, heavyweight, central, developer adverse, …
Avoid the wrong tools!
Death by properties panel
Low-code is great!(You can get rid
of your developers!)
Complex, proprietary, heavyweight, central, developer adverse, …
Workflow engines, state machines
It is
relevantin modern
architectures
CADENCE
Silicon valleyhas recognized
Workflow engines, state machines
CADENCEWorkflow engines, state machines
CADENCE
also at scaleWorkflow engines, state machines
Zeebe.io
public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration().buildProcessEngine();
engine.getRepositoryService().createDeployment() //.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") //.startEvent().serviceTask("Step1").camundaClass(SysoutDelegate.class).serviceTask("Step2").camundaClass(SysoutDelegate.class).endEvent()
.done()).deploy();
engine.getRuntimeService().startProcessInstanceByKey("flow", Variables.putValue("city", "New York"));
}public class SysoutDelegate implements JavaDelegate {public void execute(DelegateExecution execution) throws Exception {System.out.println("Hello " + execution.getVariable("city"));
}}
What do I mean by„leightweight?“
public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration().buildProcessEngine();
engine.getRepositoryService().createDeployment() //.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") //.startEvent().serviceTask("Step1").camundaClass(SysoutDelegate.class).serviceTask("Step2").camundaClass(SysoutDelegate.class).endEvent()
.done()).deploy();
engine.getRuntimeService().startProcessInstanceByKey("flow", Variables.putValue("city", "New York"));
}public class SysoutDelegate implements JavaDelegate {public void execute(DelegateExecution execution) throws Exception {System.out.println("Hello " + execution.getVariable("city"));
}}
Build enginein one line ofcode(using in-memory H2)
public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration().buildProcessEngine();
engine.getRepositoryService().createDeployment() //.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent().serviceTask("Step1").camundaClass(SysoutDelegate.class).serviceTask("Step2").camundaClass(SysoutDelegate.class).endEvent()
.done()).deploy();
engine.getRuntimeService().startProcessInstanceByKey("flow", Variables.putValue("city", "New York"));
}public class SysoutDelegate implements JavaDelegate {public void execute(DelegateExecution execution) throws Exception {System.out.println("Hello " + execution.getVariable("city"));
}}
Define flowe.g. in Java DSL
public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration().buildProcessEngine();
engine.getRepositoryService().createDeployment() //.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent().serviceTask("Step1").camundaClass(SysoutDelegate.class).serviceTask("Step2").camundaClass(SysoutDelegate.class).endEvent()
.done()).deploy();
engine.getRuntimeService().startProcessInstanceByKey("flow", Variables.putValue("city", "New York"));
}public class SysoutDelegate implements JavaDelegate {public void execute(DelegateExecution execution) throws Exception {System.out.println("Hello " + execution.getVariable("city"));
}}
Define flowe.g. in Java DSL
BPMNBusiness Process
Model and Notation
ISO Standard
public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration().buildProcessEngine();
engine.getRepositoryService().createDeployment() //.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent().serviceTask("Step1").camundaClass(SysoutDelegate.class).serviceTask("Step2").camundaClass(SysoutDelegate.class).endEvent()
.done()).deploy();
engine.getRuntimeService().startProcessInstanceByKey("flow", Variables.putValue("city", "New York"));
}public class SysoutDelegate implements JavaDelegate {public void execute(DelegateExecution execution) throws Exception {System.out.println("Hello " + execution.getVariable("city"));
}}
We can attachcode…
public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration().buildProcessEngine();
engine.getRepositoryService().createDeployment() //.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent().serviceTask("Step1").camundaClass(SysoutDelegate.class).serviceTask("Step2").camundaClass(SysoutDelegate.class).endEvent()
.done()).deploy();
engine.getRuntimeService().startProcessInstanceByKey("flow", Variables.putValue("city", "New York"));
}public class SysoutDelegate implements JavaDelegate {public void execute(DelegateExecution execution) throws Exception {System.out.println("Hello " + execution.getVariable("city"));
}}
…that iscalled whenworkflowinstances pass through
public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration().buildProcessEngine();
engine.getRepositoryService().createDeployment() //.addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent().serviceTask("Step1").camundaClass(SysoutDelegate.class).serviceTask("Step2").camundaClass(SysoutDelegate.class).endEvent()
.done()).deploy();
engine.getRuntimeService().startProcessInstanceByKey("flow", Variables.putValue("city", "New York"));
}public class SysoutDelegate implements JavaDelegate {public void execute(DelegateExecution execution) throws Exception {System.out.println("Hello " + execution.getVariable("city"));
}}
Start instances
Payment
Now you have a state machine!
Payment
Easy to handle time
[email protected]@martinschimak
Synchronous communication
RESTOrder Payment
CreditCard
Synchronous communication
RESTOrder Payment
CreditCard
Synchronous communication
REST
Order PaymentCreditCard
Synchronous communication
RESTOrder Payment
CreditCard
StatefulRetry
Distributedsystems
It is impossible todifferentiate certain
failure scenarios.
Independant ofcommunication style!
Service Provider
Client
Distributed systems introduce complexity you have to tackle!
CreditCard
PaymentREST
Distributed systems introduce complexity you have to tackle!
CreditCard
PaymentREST
Distributedsystems
Distributed transactions using compensation *
Compensation
Relaxed consistency
Temporarily inconsistent
state
But eventually consistent
No Isolation (as in ACID)
https://thenewstack.io/5-workflow-automation-use-cases-you-might-not-have-considered/
<= milliseconds seconds minutes, weeks, …
Business
IT
Business processesautomation
Distributed Transactions
Orchestration
Communication in distributed systems
long runningalways short running short running, but potentially long running
Use cases for workflow automation
Biz Devimprove
communicationimprove
communication
Ops
Biz Dev
Leveragestate machine & workflow engine
Living documentation
Visibility in testing
improvecommunication
improvecommunication
Ops
Visual HTML reports for test cases
Biz Dev
Leveragestate machine & workflow engine
Living documentation
Visibility in testing
Understand and discussbusiness processes
Evaluate optimizationsin-sync with
implementation
improvecommunication
improvecommunication
Ops
Living documentation for long-running behaviour
Biz Dev
Leveragestate machine & workflow engine
Living documentation
Visibility in testing
Understand and discussbusiness processes
Evaluate optimizationsin-sync with
implementation
improvecommunication
improvecommunication
Ops
Biz Dev
Leveragestate machine & workflow engine
Living documentation
Visibility in testing
Operate with visibilityand context
Understand and discussbusiness processes
Evaluate optimizationsin-sync with
implementation
improvecommunication
improvecommunication
Ops
ProperOperations
Visibility + Context
Before mapping processes explicitly with BPMN, the truth was buried in the code and nobody knew what was going on.
Jimmy Floyd, 24 Hour Fitnesse
„
Workflows live inside service boundaries
Manigfold architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91
Manigfold architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91
Manigfold architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91
Manigfold architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91
…
Manigfold architecture options
https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91
Lightweight workflow engines aregreat* – don‘t DIY
*e.g. enabling potentially long-running services, solving harddeveloper problems, can run decentralized
Sales-Order & Order-Fulfillment
via Camunda
for every order worldwide
(Q2 2017: 22,2 Mio)
Code example & live demo
InventoryPaymentOrder ShippingCheckout Monitor
https://github.com/flowing/flowing-retail/
Human Tasks
H2 H2
# Events decrease coupling: sometimesread-models, but no complex peer-to-peer event chains!
# Orchestration needs to be avoided: sometimesno ESB, smart endpoints/dumb pipes, important capabilities need a home
# Workflow engines are painful: some of themlightweight engines are easy to use and can run decentralized,they solve hard developer problems, don‘t DIY
Thank you!
[email protected]@berndruecker
https://bernd-ruecker.com
https://blog.bernd-ruecker.com
https://github.com/flowing
https://www.infoq.com/articles/events-workflow-automation
With thoughts from http://flowing.io@berndruecker | @martinschimak
Contact:
Slides:
Blog:
Code:
https://www.infoworld.com/article/3254777/application-development/3-common-pitfalls-of-microservices-integrationand-how-to-avoid-them.html
https://thenewstack.io/5-workflow-automation-use-cases-you-might-not-have-considered/