66
System Reliability and Resilience and stuff

Reliability and Reslience

Embed Size (px)

DESCRIPTION

Code patterns that can be used to increase reliability and resilience in your applications

Citation preview

Page 1: Reliability and Reslience

SystemReliability and

Resilienceand stuff

Page 2: Reliability and Reslience

Some things need to be cleared up first

Page 3: Reliability and Reslience

http://en.wikipedia.org/wiki/Vedette_(cabaret)

Page 4: Reliability and Reslience

tuple

Page 5: Reliability and Reslience

//Initialize customer and invoiceInitialize(customer, invoice);

Page 6: Reliability and Reslience

public void Initialize(Customer customer, Invoice

invoice){

customer.Name = “asdf”;invoice.Date = DateTime.Now;

}

Page 7: Reliability and Reslience

Initialize(customer, invoice);//did something happen to customer// and/or invoice?

Page 8: Reliability and Reslience

customer.Name =

InitNameFrom(customer, invoice);invoice.Date =

InitDateFrom(customer, invoice);

Page 9: Reliability and Reslience

customer.Name =

GetNameFrom(customer, invoice);invoice.Date =

GetDateFrom(customer, invoice);

Page 10: Reliability and Reslience

var results = Initialize(customer,

invoice);

customer.Name = results.Item1;invoice.Date = results.Item2;

Page 11: Reliability and Reslience

public tuple<string, DateTime>Initialize(customer,

invoice){

return new Tuple<string, DateTime>

(“asdf”, DateTime.Now);}

Page 12: Reliability and Reslience

public static bool TryParse(string s, out DateTime result)

or

public static tuple<bool, DateTime?>

TryParse(string s)

Page 13: Reliability and Reslience

tuple• Avoid side effects• Avoid out parameters•multiple values without a specific type

Page 14: Reliability and Reslience

null object

Page 15: Reliability and Reslience

private ILogger _logger;public MyClass(ILogger logger) {

_logger = logger;}

if (_logger != null) {_logger.Debug(

“it worked on my machine!”);}

Page 16: Reliability and Reslience

null checks for everyone!

Page 17: Reliability and Reslience

forget one and…

Page 18: Reliability and Reslience

public class NullLogger : ILogger {

public void Debug(string text) {

//do sweet nothing}

}

Page 19: Reliability and Reslience

private ILogger _logger = new NullLogger();

public MyClass(ILogger logger) {_logger = logger;

}

_logger.Debug(“it worked on my machine!”);

Page 20: Reliability and Reslience

null object• Can eliminate null checks• Simple to implement

Page 21: Reliability and Reslience

Circuit Breaker

Page 22: Reliability and Reslience
Page 23: Reliability and Reslience

Retry

Page 24: Reliability and Reslience

You

r A

pp

licati

on O

ut o

f Pro

cess

Dep

en

dency

N times

Page 25: Reliability and Reslience

Ou

t of P

roce

ss D

ep

en

dency

N times*

Y clients

Page 26: Reliability and Reslience

= Denial of Service Attack

Page 27: Reliability and Reslience

Limit the # of retries

Page 28: Reliability and Reslience

N * Ybecomes

5 * Y

Page 29: Reliability and Reslience

Y isstill a

problem

Page 30: Reliability and Reslience
Page 31: Reliability and Reslience

Circuit Breaker

Page 32: Reliability and Reslience
Page 33: Reliability and Reslience

State Machine

On :: Off

Page 34: Reliability and Reslience

On Offwhen not healthy

Page 35: Reliability and Reslience

Off Onmanually

Page 36: Reliability and Reslience

Get to softwarebefore we ask you to dance

Page 37: Reliability and Reslience

Healthyor

Unhealthy

Ou

t of P

roce

ss D

ep

en

dency

Page 38: Reliability and Reslience

State is independent of requestor

Ou

t of P

roce

ss D

ep

en

dency

Page 39: Reliability and Reslience

You

r A

pp

licati

on

Has many independent external dependencies

Page 40: Reliability and Reslience

You

r A

pp

licati

on

Can throttle itself

Page 41: Reliability and Reslience

You

r A

pp

licati

on

Has a wait threshold

Page 42: Reliability and Reslience

Your Application

External Dependency

Circuit Breaker

Threshold = 2Pause = 10msTimeout = 30sState = ClosedRequest

Request

Failure (i.e. HTTP 500)Failure Count = 1Pause 10ms

Request

Failure (i.e. HTTP 500)Failure Count = 2State = Open

OperationFailedException

Page 43: Reliability and Reslience

Threshold = 2Pause = 10msTimeout = 30sState = OpenRequest

30s has not passed

CircuitBreakerOpenException

Request

30s has not passed

CircuitBreakerOpenException

System can try to

become healthyfor 30s

Your Application

External Dependency

Circuit Breaker

Page 44: Reliability and Reslience

Threshold = 2Pause = 10msTimeout = 30sState = ½ OpenRequest

Request

Failure (i.e. HTTP 500)

Failure Count = 2State = Open

OperationFailedException

30s has passed

Your Application

External Dependency

Circuit Breaker

Page 45: Reliability and Reslience

Threshold = 2Pause = 10msTimeout = 30sState = ½ OpenRequest

Request

Failure Count = 0State = Closed

Response

30s has passed

Response

Your Application

External Dependency

Circuit Breaker

Page 46: Reliability and Reslience

ClosedOpen

½ Open

Page 47: Reliability and Reslience

½ Open is like a

manual reset

Page 48: Reliability and Reslience

PauseTimeout

Page 49: Reliability and Reslience

Pausebetween calls

in the loop

Page 50: Reliability and Reslience

Timeoutbefore you

can call again

Page 51: Reliability and Reslience

Exceptions

Page 52: Reliability and Reslience

OperationFailed:

AggregateException

Page 53: Reliability and Reslience

CircuitBreakerOpen:

ApplicationException

Page 54: Reliability and Reslience

Don’t Loose Exception Info

Page 55: Reliability and Reslience

Always use InnerException(s)

Page 56: Reliability and Reslience

Threshold = 3State = ClosedRequest

Request

Failure (i.e. HTTP 500)Request

Failure (i.e. HTTP 500)Failure Count = 2

Failure Count = 0State = Closed

Response

Response

Request?

Your Application

External Dependency

Circuit Breaker

Failure Count = 1

Page 57: Reliability and Reslience

SegregateDependencies

Page 58: Reliability and Reslience

circuitBreaker(“database”)

circuitBreaker(“weatherservice”)

Page 59: Reliability and Reslience

Dependency type, endpoint svc,

endpoint

Page 60: Reliability and Reslience

Where?

Page 61: Reliability and Reslience

You

r A

pp

licati

on O

ut o

f Pro

cess

Dep

en

dency

Cir

cuit

Bre

ake

r

Pro

xy

Page 62: Reliability and Reslience

Watch forInception

Page 63: Reliability and Reslience

You

r A

pp

licati

on W

eb

Serv

iceC

ircu

it B

reake

r

Cir

cuit

Bre

ake

r

Pro

xy

Data

base

Reposi

tory

Page 64: Reliability and Reslience

circuit breaker• retry looping• slow down attempts• good neighbour

Page 65: Reliability and Reslience

¡Muchas gracias!

Page 66: Reliability and Reslience

gracias

Donald Belcham@dbelcham

[email protected]