Handling Failures with Messaging

Elton StonemanMicrosoft MVP & Pluralsight Author

http://particular.net

Handling Failures with Messaging

Introduction to ServiceInsight for NServiceBusHandling Failures with Messaging

•Types of failure

•Demo app

•Synchronous failure handling

•Asynchronous failure handling

•Messaging architecture

Outline

Introduction to ServiceInsight for NServiceBusHandling Failures with Messaging

Distributed architectureTimeout / process overloadedTemporary – should retry

Transient Failures

DatabaseNetwork timeoutPool exhaustion

REST APIConnection timeout503 'Service Unavailable'

Introduction to ServiceInsight for NServiceBusHandling Failures with Messaging

Contract changedAuthorization revokedPermanent – should not retry

Permanent Failures

DatabaseProcedure changePermission change

REST API400 'Bad Request'401 'Unauthorized'

Introduction to ServiceInsight for NServiceBusHandling Failures with Messaging

Demo App

Web App REST APIs

Reliable (200)

Unreliable (200 or 503)

Broken (400)

Introduction to ServiceInsight for NServiceBusHandling Failures with Messaging

Synchronous processingAPI calls with WebClientExceptions from non-OK result

App – V1

Introduction to ServiceInsight for NServiceBusHandling Failures with Messaging

Demo – V1

No failure handlingBubble up to userTransient & permanent

Introduction to ServiceInsight for NServiceBusHandling Failures with Messaging

Handling Failures

Type?

Audit

End

Retry

OK?

Failure

Transient

Permanent

Yes

No

Introduction to ServiceInsight for NServiceBusHandling Failures with Messaging

Handling Failures

End

OK?

Failure

Transient

No

Yes

Permanent

Transient or permanent?Retry policiesAudit process

Type?

Audit

Retry

Introduction to ServiceInsight for NServiceBusHandling Failures with Messaging

Simple failure handlingDetermine failure typeRetry transient; log permanent

App – V2

try {}catch {}

Introduction to ServiceInsight for NServiceBusHandling Failures with Messaging

Demo – V2

Always appears successfulRetry options limitedAudit process basic

Introduction to ServiceInsight for NServiceBusHandling Failures with Messaging

Business / technical fixFailure backlogReplay process

Replaying Failures

Introduction to ServiceInsight for NServiceBusHandling Failures with Messaging

Asynchronous messagingWeb App sends messageHandler has retry & audit policy

App – V3

Introduction to ServiceInsight for NServiceBusHandling Failures with Messaging

Integration happens offlineWider scope for retryFull details for replay

Demo – V3

Introduction to ServiceInsight for NServiceBusHandling Failures with Messaging

Messaging Architecture

MessageAll request dataProcessing data

QueueOrdered storeTransactions/ACKs

HandlerDecouples front-endCan be stopped

Introduction to ServiceInsight for NServiceBusHandling Failures with Messaging

•Expect failure

•Handle failure in the right place

•Retry & backoff damages front-end

•Decoupled handlers have more options

•Persisted messages can be replayed

Summary

Thank you

github.com/sixeyed/handling-failures

pluralsight.com/author/elton-stonemanq