5 Steps for Identifying Deficiencies and Fixing Problems FAST

Confidential, Dynatrace LLC 1

5 Steps for Identifying Deficiencies and Fixing Problems FAST

Gary CarrSoftware ArchitectAmerican Fidelity

Asad AliProduct Specialist

Dynatrace

slide 2

1. Know what’s important to ALL key stakeholders• Who identifies deficiencies?• Who discovers problems?• Who wants resolution?

2. Create specific criteria for PoC• Review current systems, know your gaps• Stay open minded

3. Work on Deployment4. Always look for the metrics5. Share the results

Confidential, Dynatrace LLC

The 5 Steps

slide 3 Confidential, Dynatrace LLC

Nine times on Forbes “Top 100 Places to Work”

Focus on 24x7 service to Customers, Colleagues,

Community

Technology supports relationships, rather than replacing

Partner support is key to growth

1.9 million policies, 1,800 employees

Who is American Fidelity?

slide 4 Confidential, Dynatrace LLC

In-house tools provided the basics

Before Dynatrace

• API, ESB, F5 URL calls• DNS related

problems• No visibility into

exceptions

slide 5 Confidential, Dynatrace LLC

• Two kinds of problems• Visible, but lots of time and resources• Invisible

Before Dynatrace

“It was like being in a dark room”Traci Tabyanan, Applications Manager, American Fidelity

slide 6 Confidential, Dynatrace LLC

• Business• More accurately measure

user satisfaction• Provide excellent customer

experience• Release applications more

often• Reduce technical debt

Reasons for Enterprise-ready APM

“We’re heading toward being a technology company that supplies insurance”Diana Bittle, VP of Software Solutions, American Fidelity

43k Users

slide 7 Confidential, Dynatrace LLC

Reasons for Enterprise-ready APM• Technical

• Reproduce production problems accurately in dev

• Understand how 3rd party tools operate

• Excessive database calls• Visual topology of complex delivery

chain• Between tiers and technology

platforms• Inconsistent performance between

test environments for same build• Code-level diagnostics

slide 8 Confidential, Dynatrace LLC

• Use of guesswork to identify root cause• Lack of visibility into the end user response time• Visibility gaps between application tiers• Need to reproduce problems to identify root cause• Necessity for war rooms for major problems

How to identify deficiencies in your APM

slide 9 Confidential, Dynatrace LLC

Don’t be like Michael Scott…

“the office” is a trademark of NBC

slide 10 Confidential, Dynatrace LLC

Prepare for Successful PoC• Focused on enterprise wide applications• Invited developers, testers, DevOps, security, and project managers• Shared applications and issues with APM experts• Set up before PoC began with specialists from our company and Dynatrace• Very specific requirements for specific tests

• Found a need in our company and used the PoC to solve• Visibility from individual’s browser to deepest calls in the system• In-depth analysis of resource consumption per transaction• Dashboards specific to each stakeholder

• Showed enterprise level• AND provided easy drill-down

slide 11 Confidential, Dynatrace LLC

• Define concrete and specific PoC success criteria• Use most significant application(s)• Stress test your environment with the APM tool• Invite all stakeholders to the PoC• Look for value DURING• Ensure apples-to-apples

Lessons Learned from 500+

slide 12 Confidential, Dynatrace LLC

• Problem #1: Most of the execution time for main app was spent in .NET remoting and ADO• + ID where most of

time was spent• Focus on reducing

technical debt in those areas

AF’s PoC: Key Findings

slide 13 Confidential, Dynatrace LLC

• Problem #2: Able to isolate remoting issue to one particular server• Been looking for a month• Saw because capture all

transaction, including exceptions• Seamless fit: infrastructure

and processes• Helped id specific code

changes

AF’s PoC: Key Findings

slide 14 Confidential, Dynatrace LLC

• Simple view of complex delivery chain

• Automatically created

• Auto-discovery removes guesswork

PoC: Key Findings

Dynatrace

Art!

slide 15 Confidential, Dynatrace LLC

• Database connectivity issues• Configuration settings • Remoting calls => $40M savings• Architecture anti-patterns• Memory leaks• N+1 query patterns

Other Key Findings

slide 16 Confidential, Dynatrace LLC

• Involve DevOps early!• Share successes• Early example:

• 2-day MIRT right after deployment, so were not using Dynatrace much yet…

• Let’s check one thing easier than thought

• Solution in 20 minutes• Benefit grows exponentially

• Now Dynatrace involved early in MIRT

Deployment Success

slide 17 Confidential, Dynatrace LLC

Metrics help build better apps, faster

More enrollments. Yeah!

slide 18 Confidential, Dynatrace LLC

Metrics help build better apps, faster

More processing. Boo!

• From 20 minutes to 2 hours

• Errors and unexpected timeouts

• Tried increasing planned timeouts

slide 19 Confidential, Dynatrace LLC

Metrics help build better apps, faster

• Quickly saw exact problem• Database query

taking 7 minutes Called 443 times

• Down to 56 calls• Straightforward

code change

slide 20 Confidential, Dynatrace LLC

Gasps + Clapping = Performance HeroEven as enrollments continue to grow, processing time remains low

slide 21 Confidential, Dynatrace LLC

Take-aways from a developer• Look at System Profiles> YourApp> Diagnose

Events> Exceptions• Customize your timeframe (i.e. 7PM – 9PM)• Use PurePath™ = “Intense Tracing”

• Look for method names• Look for long execution times• Keep drilling down• Sampling would not have found this issue

• Don’t be afraid to ask for help

slide 22 Confidential, Dynatrace LLC

• Coincidences bring nightmares• Sometimes the state of the system is the problem• Gap-free APM provides

• All the data leading up to issue, from every tier• Plus inter-tier stitching• Prioritize and reduce technical debt

• Faster fixes• “Customer cannot log in, it happened at 8:45 a.m.”• Just grab all the data from 8:44 a.m. to 8:46 a.m.

It’s all about the visibility

slide 23 Confidential, Dynatrace LLC

• Ability to compare slow transactions with fast transactions• Reduces the need to

reproduce problems• Easily help identify root cause

of pool exhaustion issues• Enables conversion of

application data to business data• Eliminates guesswork

Benefits of gap-free data

slide 24 Confidential, Dynatrace LLC

User Satisfaction• Now developers have a

closer connection to customers• APM shows impact of coding

methods• Developers can see real-time

user satisfaction

slide 25 Confidential, Dynatrace LLC

• Promotes DevOps culture• Reduces Ops anxiety over newer

code release• Enables frequent code releases• Make Dev an equal partner in

problem resolution• Promotes innovation

Dev and Prod synch serves customers

slide 26 Confidential, Dynatrace LLC

• War room calls directly show the power• Town Hall meetings: App Development, Business, Marketing,

and more• TV and tablets with HTML5• Results• Challenges!

How AF spreads the use of APM

slide 27 Confidential, Dynatrace LLC

• Promote key improvements to upper management

• Use APM tool to make the developers the “hero”

• Translate APM success to business $$$

• Display APM dashboards in public areas

• Use APM tool to reduce schism between Dev and Ops

Insights from other customers on spreading APM

slide 28 Confidential, Dynatrace LLC

• Continue to improve relationship with the business• Using dashboards• See their applications

• Dynatrace Community• Sharing information

• Training• User Groups and Conferences• New release strategy

Extra notes

slide 29 Confidential, Dynatrace LLC

•Define concrete PoC criteria•Use significant application(s) for PoC• Look for value DURING• Share success of APM results• Promote DevOps culture•Make Dev an equal partner in problem resolution

Summary

slide 30 Confidential, Dynatrace LLC

1. Know what’s important to ALL key stakeholders• Who identifies deficiencies?• Who discovers problems?• Who wants resolution?

2. Create specific criteria for PoC• Review current systems, know your gaps• Stay open minded

3. Work on Deployment4. Always look for the metrics5. Share the results

In Review: The 5 Steps

Copyright 2016, Dynatrace LLCConfidential, Dynatrace LLC 31

THANK YOU!