32
a bug’s life How to avoid production catastrophes Tomer Shiri [email protected]

Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes

Embed Size (px)

DESCRIPTION

Lesson learned from production bugs that really happened to Tomer Shiri in his iOS apps. Presented at the TLV iOS Developer Meetup on 16/11/2014

Citation preview

Page 1: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes

a bug’s life

How to avoid production catastrophes

Tomer [email protected]

Page 2: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes

Remove before flight

Page 3: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes
Page 4: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes

Who’s to blame?

Page 5: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes
Page 6: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes

How to avoid?

Don’t release anything that has local changes

Avoid Pre release checklists completely

Avoid manual configurations

Use build configurations

Page 7: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes

avoid manual operations

Page 8: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes

Demo time

Page 9: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes

Highway 64

Page 10: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes
Page 11: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes
Page 12: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes

What when wrong?

Page 13: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes

All integers are MAX_INT

Page 14: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes

How these values age generated?

Configfile

Page 15: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes

@dynamic properties

Page 16: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes

Seems reasonable?

Page 17: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes

Whats ‘c’?

Page 18: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes
Page 19: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes

Whats ‘B’?

Page 20: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes

Solution?

Page 21: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes

That’s an edge case, man

Have you ever swizzled private methods?

Removed that annoying UIScrollView from Apple’s views?

Used KVM in order to access / change private ivars?

Page 22: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes

Don’t rely on internal implementation

Page 23: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes

The overachiever

Page 24: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes
Page 25: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes

High traffic!

Page 26: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes

Some web services stopped functioning.

Oath API getting hammered by the clients

Page 27: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes

When an API requests fails, it stops all other requests and tries to obtain a new auth_token.

When two threads tries to do that in the same time, they are failing each other.

Page 28: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes

Solution?

There is no good solution for that.

Threading bug are hard to hunt.

Page 29: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes

Use kill switches

Page 30: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes

Prepare for the worst

Page 31: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes

avoid manual operations

Everything that can compile - should compile

Don’t rely on internal implementation

Prepare for the worst

Page 32: Tomer Shiri - A Bug's Life - How to Avoid Production Catastrophes

Tomer Shiri

[email protected]

@tomertomershiri

github.com/tomersh