Upload
chris-chew
View
568
Download
5
Tags:
Embed Size (px)
DESCRIPTION
Deploying an application and just waiting for the error logs to roll in is the surest way to be forgotten. Instead, applications should be launched with clear goals and ways to measure the application's level of achievement. This talk presents a systematic framework for launching successful applications that includes: Determining which indicators should be measured, ways of capturing the measurements, and finally how to gain momentum by advertising the achievements.
Citation preview
Successful Applications
A systematic framework for launching successful applications
Chris K. ChewEnterprise Infrastructure TeamPearson eCollege
@chrischew
Successful Applications
Successful Applications
Successful Applications
Successful Applications
Successful Applications
Successful Applications
Successful Applications
Successful Applications
Successful Applications
Successful Applications
Successful Applications
Successful Applications
Successful Applications
Successful Applications
Successful Applications
Successful Applications
Successful Applications
Successful Applications
Launching Successful Applications
1. What To Count– Key Performance Indicators– KPI’s in software– Choosing your indicators
2. How To Count– Finding the right places to measure– Frameworks that can help
3. Count Out Load– Transparency builds goodwill– Advertising success creates momentum
4. When To Evolve Your Measurements
Successful Applications
What To CountKey Performance Indicators
1
Successful Applications
Successful Applications
Successful Applications
Successful Applications
Successful Applications
Successful Applications
Successful Applications
Presentation Title runs here l 00/00/00
Key Performance Indicators
KPI’s are a small number of measurable objectives that will add the most value to the organization for the foreseeable future.
Successful Applications
Key Performance Indicators
• Targeted towards your strategic goals
• Must be objectively knowable
• Each measurement must clearly add value to the organization
• Articulate which scenarios are included in each measurement
• The fewer the better
Successful Applications
Key Performance Indicators
• Quantitative: Something that is a number
• Practical: Something that ties into existing business processes
• Directional: Specifying whether something is improving, trending
• Actionable: Identifying a problem that is within the organization's ability to change
• Comparative: Comparing yourself with a "peer universe", e.g. industry. Also referred to as “financial”
Successful Applications
Key Performance Indicators
Etsy’s Indicators
• Quantitative: How many items were sold last month?
• Practical: How many new members signed up?
• Directional: Comparison from January 2011 and February 2010
• Actionable: Number of complaints or returned items?
• Comparative: What are ebay’s trends for the same month?
Successful Applications
Key Performance Indicators In Software
KPI’s are a small number of measurable objectives that will add the most value to the organization your application for the foreseeable future.
Successful Applications
Key Performance Indicators In Software
• Quantitative: Measure the occurrences and durations of activities and transactions
• Practical: Measuring strategically-targeted usage patterns
• Directional: Did recent changes help or hurt performance?
• Actionable: Any error should be an actionable indicator
• Comparative: Compare your application to similar applications in your installation
Successful Applications
Key Performance Indicators In Software
• Functional: Measuring the frequency and duration which features are used.
– Examples:
○Minutes in class○ Click-throughs○ SSO hand-offs○ Page views
Successful Applications
Key Performance Indicators In Software
• Non-Functional: Measuring the transactional throughput of an application.
– Examples:
○ Connections per server○ Response times○ Page load times
Successful Applications
Key Performance Indicators In Software
• Transactional: Measuring the frequency and duration which business transactions are executed.
– Examples:
○Messages○ Enrollments processed○ Courses viewed○ Thread responses
Successful Applications
Key Performance Indicators In Software
How To Identify Your KPI’s
• What are the nearest-term goals of your application?
• How will you know when you have reached your goals?
Successful Applications
Key Performance Indicators In Software
Example: Prospero Message Bus
Strategic Goal: To provide an always-on message bus that guarantees delivery with very low message publication latency.
What To Measure:• Availability• Message loss• Delivery latency
Successful Applications
Key Performance Indicators In Software
Example: Prospero Message Bus
Key Performance Indicators:• Messages produced (per second and day)• Number of times it takes longer than 500ms to publish a message• Number of delivery problems• Production-to-Delivery latency• Linux “load factor” of servers
Successful Applications
Key Performance Indicators In Software
Example: Anton.io Browser-Channel Test
Strategic Goal: To feel comfortable recommending the use of web sockets and flash sockets for realistic event-driven web UI.
What To Measure:• Connection reliability per transport• Connection density and server behavior of Node.js
Successful Applications
Key Performance Indicators In Software
Example: Anton.io Browser-Channel Test
Key Performance Indicators• Connections succeeded and failed• Transport used (web sockets vs. flash vs. XHR polling)• Connection time in milliseconds
Successful Applications
How To CountCollecting and presenting your application’s statistics
2
Successful Applications
How To Count
The Golden Rules Of Audit Logging
• Log transactions, not errors and debug statements
• Find the right places to place the hooks
• Record the transaction as concisely as possible
• Leverage existing frameworks
Successful Applications
How To Count
The Golden Rules Of Audit Logging
• Log transactions, not errors and debug statements
• Find the right places to place the hooks
• Record the transaction as concisely as possible
• Leveraging existing frameworks
Successful Applications
How To Count
The Golden Rules Of Audit Logging
• Log transactions, not errors and debug statements
– Error logs are for identifying code and system problems
– Debug statements are for development and last-resort debugging
– Audit logs need to be noiseless and easily parsed
Successful Applications
How To Count
The Golden Rules Of Audit Logging
• Log transactions, not errors and debug statements
• Find the right places to place the hooks
• Record the transaction as concisely as possible
• Leveraging existing frameworks
Successful Applications
http://assets.en.oreilly.com/1/event/7/Improving%20Netflix%20Performance%20Presentation.pdf
Successful Applications
Successful Applications
Successful Applications
Successful Applications
How To Count
Finding The Right Places To Measure
...is all about knowing what pieces are included in the measurement
Successful Applications
Successful Applications
Successful Applications
How To Count
The Golden Rules Of Audit Logging
• Log transactions, not errors and debug statements
• Find the right places to place the hooks
• Record the transaction as concisely as possible
• Leveraging existing frameworks
Successful Applications
How To Count
Anatomy Of An Audit Log Statement
• Items:– Timestamp– Action– “key=value” pairs supplying context
• Format:– TIMESTAMP action=ACTION [key1=value1 ...]– Exclude anything the is deducible or implicit, e.g. server name – $64k Question: Can you parse it with a regular expression?
Successful Applications
How To Count
Anatomy Of An Audit Log Statement - Prospero Example
2011-04-11T16:13:53Z action=SUBSCRIPTION-CANCELLED sub=e0490bb8-1eb7-4221-a6f2-4cb1bc9c76d4
2011-04-11T16:15:56Z action=MESSAGE msg=c14fd86f-f448-4aa6-b662-87c7f27895fc principal=ONE length=11 realm=default content_type=text/plain tag=MessageType:messageType1 tag=SubSystem:default tag=System:default tag=ClientString:default tag=Client:default
2011-04-11T16:15:56Z action=ATTEMPTING-DELIVERY sub=fc9c7e50-542b-44cb-b48f-7c1f992d13a9 msg=c14fd86f-f448-4aa6-b662-87c7f27895fc del_attempt=d52054d8-1cbb-47f7-927e-ff901d6020ce
2011-04-11T16:15:56Z action=DELIVERED sub=fc9c7e50-542b-44cb-b48f-7c1f992d13a9 msg=c14fd86f-f448-4aa6-b662-87c7f27895fc del_attempt=d52054d8-1cbb-47f7-927e-ff901d6020ce del_ref=12345 duration_seconds=0 latency_seconds=1 ramp_churn=0
NOTE: This is word-wrapped...always do one statement per line for easier parsing!
Successful Applications
How To Count
Anatomy Of An Audit Log Statement - Prospero Example
2011-04-11T16:13:53Z action=SUBSCRIPTION-CANCELLED sub=e0490bb8-1eb7-4221-a6f2-4cb1bc9c76d4
2011-04-11T16:15:56Z action=MESSAGE msg=c14fd86f-f448-4aa6-b662-87c7f27895fc principal=ONE length=11 realm=default content_type=text/plain tag=MessageType:messageType1 tag=SubSystem:default tag=System:default tag=ClientString:default tag=Client:default
2011-04-11T16:15:56Z action=ATTEMPTING-DELIVERY sub=fc9c7e50-542b-44cb-b48f-7c1f992d13a9 msg=c14fd86f-f448-4aa6-b662-87c7f27895fc del_attempt=d52054d8-1cbb-47f7-927e-ff901d6020ce
2011-04-11T16:15:56Z action=DELIVERED sub=fc9c7e50-542b-44cb-b48f-7c1f992d13a9 msg=c14fd86f-f448-4aa6-b662-87c7f27895fc del_attempt=d52054d8-1cbb-47f7-927e-ff901d6020ce del_ref=12345 duration_seconds=0 latency_seconds=1 ramp_churn=0
NOTE: This is word-wrapped...always do one statement per line for easier parsing!
Successful Applications
How To Count
Anatomy Of An Audit Log Statement - Prospero Example
2011-04-11T16:13:53Z action=SUBSCRIPTION-CANCELLED sub=e0490bb8-1eb7-4221-a6f2-4cb1bc9c76d4
2011-04-11T16:15:56Z action=MESSAGE msg=c14fd86f-f448-4aa6-b662-87c7f27895fc principal=ONE length=11 realm=default content_type=text/plain tag=MessageType:messageType1 tag=SubSystem:default tag=System:default tag=ClientString:default tag=Client:default
2011-04-11T16:15:56Z action=ATTEMPTING-DELIVERY sub=fc9c7e50-542b-44cb-b48f-7c1f992d13a9 msg=c14fd86f-f448-4aa6-b662-87c7f27895fc del_attempt=d52054d8-1cbb-47f7-927e-ff901d6020ce
2011-04-11T16:15:56Z action=DELIVERED sub=fc9c7e50-542b-44cb-b48f-7c1f992d13a9 msg=c14fd86f-f448-4aa6-b662-87c7f27895fc del_attempt=d52054d8-1cbb-47f7-927e-ff901d6020ce del_ref=12345 duration_seconds=0 latency_seconds=1 ramp_churn=0
NOTE: This is word-wrapped...always do one statement per line for easier parsing!
Successful Applications
How To Count
Anatomy Of An Audit Log Statement - Prospero Example
2011-04-11T16:13:53Z action=SUBSCRIPTION-CANCELLED sub=e0490bb8-1eb7-4221-a6f2-4cb1bc9c76d4
2011-04-11T16:15:56Z action=MESSAGE msg=c14fd86f-f448-4aa6-b662-87c7f27895fc principal=ONE length=11 realm=default content_type=text/plain tag=MessageType:messageType1 tag=SubSystem:default tag=System:default tag=ClientString:default tag=Client:default
2011-04-11T16:15:56Z action=ATTEMPTING-DELIVERY sub=fc9c7e50-542b-44cb-b48f-7c1f992d13a9 msg=c14fd86f-f448-4aa6-b662-87c7f27895fc del_attempt=d52054d8-1cbb-47f7-927e-ff901d6020ce
2011-04-11T16:15:56Z action=DELIVERED sub=fc9c7e50-542b-44cb-b48f-7c1f992d13a9 msg=c14fd86f-f448-4aa6-b662-87c7f27895fc del_attempt=d52054d8-1cbb-47f7-927e-ff901d6020ce del_ref=12345 duration_seconds=0 latency_seconds=1 ramp_churn=0
NOTE: This is word-wrapped...always do one statement per line for easier parsing!
Successful Applications
How To Count
The Golden Rules Of Audit Logging
• Log transactions, not errors and debug statements
• Find the right places to place the hooks
• Record the transaction as concisely as possible
• Leveraging existing frameworks
Successful Applications
How To Count
Leverage Existing Frameworks
• Log files work well:
– Rotated with logrotate or CHOMP
– Many search utilities (grep, sed, awk)
– Easily processed by any language
– log4j and log4net write asynchronously
– They can always be latter parsed and inserted into a database
Successful Applications
How To Count
Leverage Existing Frameworks
– Try our audit logging library available on GitHub at○ https://github.com/PearsonLearningStudio/audit-logging
– Java
– Coming soon:○ .NET○Node.js○ Erlang
Successful Applications
Successful Applications
Successful Applications
Successful Applications
Successful Applications
Count Out LoudTransparency and advertisement
3
Successful Applications
Successful Applications
Successful Applications
Count Out Loud, Be Transparent
• Being transparent with successes and failures builds goodwill towards your application.
– Users are more likely to trust your application
– Users will be more patient when problems occur
Successful Applications
Successful Applications
Count Out Loud, Advertise Your Results
• Advertising your results will help keep users aware of your application after the excitement of the initial launch dies down.
– Product folks are more likely to prioritize improvements to the application
– Potential users of the application will feel more comfortable adopting it
– The data you share may help your peers
Successful Applications
Successful Applications
When To Evolve Your MeasurementsMetrics change as goals change
4
Successful Applications
Successful Applications
When To Evolve Your Measurements
• Goals will change either because strategies change or because the goal is reached and new ones must be set.
• KPI’s should be adjusted at the same time when goals change
Successful Applications
When To Evolve Your Measurements
• Goals will change either because strategies change or because the goal is reached and new ones must be set.
• KPI’s should be adjusted at the same time when goals change
• But it is also important to listen for negative feedback because you might be measuring the wrong things.
Successful Applications
Wrong KPI?
• Inadvertently incentivize undesirable behavior
• Start measuring the negative feedback to determine if the problem is real
• Either replace the old KPI with the new one or add it to the squad
Successful Applications
Buffer Bloat: Measuring The Wrong Thing
• Bufferbloat is the existence of excessively large (bloated) buffers in systems, particularly network communication systems
Successful Applications
Buffer Bloat: Measuring The Wrong Thing
• Bufferbloat is the existence of excessively large (bloated) buffers in systems, particularly network communication systems
• Some experts believe that bufferbloat is the cause of much of the poor performance and human pain experienced using today’s Internet
Successful Applications
Buffer Bloat: Measuring The Wrong Thing
• Bufferbloat is the existence of excessively large (bloated) buffers in systems, particularly network communication systems
• Some experts believe that bufferbloat is the cause of much of the poor performance and human pain experienced using today’s Internet
• If true, this is a quintessential example of picking the wrong industry KPI– Networks should probably use “Round Trip Latency” instead of
“Dropped Packets” because excessive work to not drop packets negates the effectiveness of TCP/IP
Successful Applications
Buffer Bloat: Measuring The Wrong Thing
• See http://www.bufferbloat.net for details
Successful Applications
Buffer Bloat: Measuring The Wrong Thing
• See http://www.bufferbloat.net for details
• Wager: Phở is on me if Buffer Bloat is not part of the general networking conversation by the end of 2012.
Successful Applications
Conclusion
5
Successful Applications
Launching Successful Applications
1. What To Count– Key Performance Indicators in software– Choosing your indicators
Successful Applications
Launching Successful Applications
1. What To Count– Key Performance Indicators in software– Choosing your indicators
2. How To Count– Finding the right places to measure– Frameworks that can help
Successful Applications
Launching Successful Applications
1. What To Count– Key Performance Indicators in software– Choosing your indicators
2. How To Count– Finding the right places to measure– Frameworks that can help
3. Count Out Load– Transparency builds goodwill– Advertising success creates momentum
Successful Applications
Launching Successful Applications
1. What To Count– Key Performance Indicators in software– Choosing your indicators
2. How To Count– Finding the right places to measure– Frameworks that can help
3. Count Out Load– Transparency builds goodwill– Advertising success creates momentum
4. Evolve Your Measurements– When goals change– Negative feedback
Thank you
Chris K. ChewEnterprise Infrastructure Team
@chrischew