25
Crittercism Mobile Experience Benchmark – Apple Edition September 2014

Apple Edition: iOS Mobile Experience Report

Embed Size (px)

Citation preview

Page 1: Apple Edition: iOS Mobile Experience Report

Crittercism Mobile Experience Benchmark – Apple Edition September 2014

Page 2: Apple Edition: iOS Mobile Experience Report

Mobile Experience Benchmark

2

Mobile app performance is challenging with over millions of permutations of variables to manage.

Mobile App Complexity

Key Performance Takeaways for iOS Apps

To be competitive, your app should respond to user requests within 1 second. The reality is most apps don’t because they rely on unpredictable services.

App Responsiveness

Your goal should be to have an app that crashes less than 1%. Most apps crash more.

App Uptime (Crash)

Geography Mobile apps should perform well in all geographies. However, location-based factors impact regional experience.

Page 3: Apple Edition: iOS Mobile Experience Report

Mobile Experience Benchmark

3

iOS apps live in a complex environment, making it difficult to manage performance and ensure superior user experience. The pillars that comprise mobile app performance are:

1. App uptime 2. App responsiveness

App uptime and responsiveness are also impacted by the Geography of the user. Crittercism is the leading mobile application performance management (mAPM) solution that processes over 30,000 requests per second from over 1B mobile users in real-time in over 120 countries. We’ve gathered rich iOS related performance data and are honored to share the first-ever Crittercism Mobile Experience Benchmark – Apple Edition report with the world. This report uses data from Crittercism customers actively optimizing their app performance. App performance across un-optimized applications is expected to be worse.

Introduction

Page 4: Apple Edition: iOS Mobile Experience Report

Mobile Experience Benchmark

•  At 97% uptime, the company would lose $2.5M per month

•  At 99% uptime, the company would lose $0.8M per month

•  At 99.9% uptime, the company would lose $82,192 per month

Let’s look at how poor uptime could impact a company with $1B in annual mobile revenue*:

Mobile Performance Impacts Your Bottom Line

4

*Examples of companies with significant mobile revenue include Starbucks which Business Insider estimates drives $1B through mobile and Domino’s which sees $700M in mobile sales according to Mobile Commerce Daily.

Page 5: Apple Edition: iOS Mobile Experience Report

5

iOS Mobile Apps Live in a Complex World

Page 6: Apple Edition: iOS Mobile Experience Report

Mobile Experience Benchmark

6

Fragmentation adds Complexity

Millions Devices OS

versions Carriers of permutations,

creating a complex ecosystem

X X = X App

versions

Page 7: Apple Edition: iOS Mobile Experience Report

Mobile Experience Benchmark

7

•  The average iOS app depends on 6+ cloud services . Examples include: Facebook for logins, Amazon Web Services for storage, and Flurry for analytics

•  52% use over 5 services

Findings

Average Number of Services Used By iOS App

Connecting to 6+ Cloud Services Increases Complexity

•  iOS apps depend on many underlying cloud services, which means they not only need to be free of crashes, but the services they depend on must also work and be available

Takeaway

Page 8: Apple Edition: iOS Mobile Experience Report

8

Responsiveness Performance

The pillars of mobile app performance are responsiveness, uptime and geography. Responsiveness of cloud services gives you insight into how much faster or slower your app is running than your benchmark or expectations. Responsiveness helps answer questions such as: •  Which services have the highest error rates? •  Which services are slowing down my app? This section will shed light on mobile app responsiveness and error rates by app and top services.

Why this matters: iOS apps depend on many cloud services and responsiveness impacts end-user experience

Page 9: Apple Edition: iOS Mobile Experience Report

Mobile Experience Benchmark

9

iOS apps depend on many services and service calls. A single app may make multiple services calls per end user interaction. Hence it is the aggregate latency and error rates across these service calls that determines the end user experience. The two metrics to focus on are error rate and latency. Users expect responses in under 1 second*, yet 5% of calls experience higher latency and 46% have over 500 ms latency. In addition, 17% of service calls have over 5% error rate and 13% have over 10% error rates.

Service Calls are Unpredictable

46%

17%

of cloud services that iOS apps use have over a half second latency

of service calls from iOS apps have error rates over 5%

*Based on data from Chimera Labs: http://chimera.labs.oreilly.com/books/1230000000545/ch10.html

Page 10: Apple Edition: iOS Mobile Experience Report

Mobile Experience Benchmark

10

•  Google Analytics has lowest error rate (0.1%); error rate is percent of errors per number of service calls made

•  Of top public services, Google Analytics has lowest average latency

Findings

Error Rates & Response Times by top iOS Services

Google Analytics is Best Performing Top Service

•  Even top services like Google Analytics and Facebook experience latency and error rates

•  Teams must manage every aspect of mobile app performance, including third-party services

Takeaway

Page 11: Apple Edition: iOS Mobile Experience Report

Mobile Experience Benchmark

11

•  T-Mobile is the fastest iOS carrier •  Sprint and US Cellular are the slowest

iOS carriers

Findings

iOS Latency Rankings By Major US Carriers

T-Mobile is the Fastest iOS Carrier

•  Responsiveness of iOS apps varies across the major carriers. Expect that your mobile team will need to adapt to performance issues driven by these carrier specific differences

Takeaway

Page 12: Apple Edition: iOS Mobile Experience Report

12

Geography

Responsiveness and uptime are affected by the location of the end user. Factors to manage include: device and OS adoption, regional carriers and networks. A geographic performance lens helps answer questions such as: •  Is my app slow in different geographies? •  What is causing a poor experience for users in

different countries, cities, or store locations? This section will summarize iOS app performance across countries and cities.

Why this matters: Expanding apps to new geographies introduces location factors that impact user experience

Page 13: Apple Edition: iOS Mobile Experience Report

Mobile Experience Benchmark

13

•  Virginia Beach has the fastest iOS WiFi responsiveness

•  Cleveland has the slowest iOS WiFi responsiveness

Findings

iOS WiFi Latency Rankings by Most Populous US City

Cleveland has Slowest iOS WiFi

Takeaway

•  iOS users across US cities experience app responsiveness differently. In particular, WiFi speeds vary by city

Page 14: Apple Edition: iOS Mobile Experience Report

Mobile Experience Benchmark

14

•  Raleigh has the fastest iOS Carrier responsiveness

•  Kansas City has the slowest iOS Carrier responsiveness

Findings

iOS Carrier Latency Rankings by Most Populous US City

Raleigh has Fastest iOS Carrier

Takeaway

•  App owners should optimize apps and architectures (i.e. Content Delivery Networks and regional data centers) to account for geographic variance in response times

Page 15: Apple Edition: iOS Mobile Experience Report

Mobile Experience Benchmark

15

•  Yokohama & Tokyo have the fastest iOS WiFi responsiveness

•  Ho Chi Minh City has the slowest iOS WiFi responsiveness

Findings

iOS WiFi Latency Rankings by Major World Cities

Yokohama and Tokyo are Fastest

•  Be prepared to deal with regional wireless networks and latencies

Takeaway

Page 16: Apple Edition: iOS Mobile Experience Report

Mobile Experience Benchmark

16

•  Tokyo and Yokihama have the fastest iOS Carrier responsiveness

•  Bangkok has the slowest iOS Carrier responsiveness

Findings

iOS Carrier Latency Rankings by Major World Cities

Tokyo has Fastest Carrier Responsiveness

•  iOS users across world cities experience app responsiveness differently. Carrier speeds vary by world city

•  Be prepared to deal with regional carrier networks and latencies

•  App owners should optimize apps and architectures (i.e. Content Delivery Networks and regional data centers) to account for geographic variance in response times

Takeaway

Page 17: Apple Edition: iOS Mobile Experience Report

Mobile Experience Benchmark

17

•  France has the fastest iOS WiFi responsiveness

•  India has the slowest iOS WiFi responsiveness

Findings

iOS WiFi Latency Rankings by G20 Countries

France has Fastest WiFi Responsiveness

•  iOS users in different countries experience app responsiveness differently. WiFi speeds vary by world city

Takeaway

Page 18: Apple Edition: iOS Mobile Experience Report

Mobile Experience Benchmark

18

•  Japan has the fastest iOS Carrier responsiveness

•  China has the slowest iOS Carrier responsiveness

Findings

iOS Carrier Latency Rankings by G20 Countries

Japan has Fastest Carrier Responsiveness

•  iOS users in different countries experience app responsiveness differently. Carrier speeds vary by world city

•  Be prepared to deal with regional carrier networks and latencies

•  App owners should optimize apps and architectures (i.e. Content Delivery Networks and regional data centers) to account for geographic variance in response times

Takeaway

Page 19: Apple Edition: iOS Mobile Experience Report

Uptime Performance

In addition to responsiveness and geography, the other main culprit of app performance issues is app uptime. Mobile app uptime is defined by Crittercism as the percent of app usage that did not experience a crash. Uptime helps answer questions such as: •  Which devices are causing my app to crash? •  Does my app crash more on a particular OS

version?

This section will reveal acceptable uptime performance of mobile apps. It will also provide data across app categories and industries.

Why this matters: Uptime performance is a fundamental measure of end-user experience

Page 20: Apple Edition: iOS Mobile Experience Report

Mobile Experience Benchmark

20

•  30% of apps have over a 2% crash rate •  46% of apps have over a 1% crash rate

Findings

Percent of iOS apps by crash uptime

46% of iOS Apps Have Over 1% Crash Rate

•  In the web world, the norm uptime is 99.9% •  In the mobile world, however, your app

should have a crash rate under 1% (uptime of 99%)

•  Apple lists Crashes and Bugs as the number one reason apps get rejected from the App Store

Takeaway

1% crash rate is based on customers actively using Crittercism. Apps in the wild can experience anywhere from 3-10% crash rate.

Page 21: Apple Edition: iOS Mobile Experience Report

Mobile Experience Benchmark

21

•  iOS 7.1 & iOS 7 have 2.1% crash rates •  iPhone 5s crashes least •  iPad Mini crashes the most •  Apps on iPhones are more stable than iPads

Findings

Apple OS and top device breakdown

Apple: 4S Crashes The Most Among Top iPhones

•  Prior to iOS 8’s release, app developers have stabilized crash rates across iOS versions

•  Expect iPad users to experience stability issues more often than those using iPhones

Takeaway

Page 22: Apple Edition: iOS Mobile Experience Report

Mobile Experience Benchmark

22

•  Games and Photo & Video have the highest crash rates

•  Health & Fitness and Business apps have lowest crash rates

•  The Lifestyle category (which includes many ecommerce apps) has a 1.5% crash rate

Findings

Uptime by iOS App Store Category

Games Crash the Most

•  Graphics intensive apps such as Games and Photo & Video crash most

•  Apps that enable transactions crash less, but the cost of downtime is significant. Pay close attention to the performance of these apps

Takeaway

Page 23: Apple Edition: iOS Mobile Experience Report

Improving mobile app performance is critical Enterprises will sharpen their focus from web to mobile, and leverage apps to increase revenue and productivity. In addition, users will demand better performance from mobile apps. Companies that have consistently delivered top user experience use a mobile APM solution to optimize their apps. To learn more about Crittercism’s mobile app performance management solution, visit WWW.CRITTERCISM.COM

Page 24: Apple Edition: iOS Mobile Experience Report

Mobile Experience Benchmark

24

ISSUE BEFORE AFTER IMPACT

Real Business ROI Crittercism has worked with thousands of customers across multiple industries to improve their mobile apps and experience. Below is a glimpse into the achievements of customers.

App performance issues impact $100K’s of mobile

revenue/hour

Poor visibility into app issues affecting users

App performance issues affecting user experience

Reactive approach to issues

Long time to resolution for service issues

2 weeks to resolve issues causing

downtime

6% downtime

<0.02% crash rate

Rapid troubleshooting of service issues

Resolve most issues in 2 days

<1% downtime

5 Star app ratings

Improved revenue

14X more efficient at resolving critical crashes

2X faster app releases

7x performance

improvement

6x revenue improvement

Monetize through mobile channel while creating best

user experience 3% downtime across all

platforms < 0.5% downtime

across all platforms 83% improvement in uptime

2 stars à 4+ stars

Page 25: Apple Edition: iOS Mobile Experience Report

Mobile Experience Benchmark

25

Glossary & Methodology

•  MAU: Monthly Active Users are the unique end-users on a mobile app during a given month •  DAU: Daily Active Users are the unique end-users on a mobile app on a given day •  Responsiveness: Time it takes for each service call to respond •  Mobile app uptime: Availability of the mobile app; percentage of app loads that do not result in a crash * •  Crash rate: Percentage of app loads that result in a crash * •  Error rate: Percentage of HTTP or network errors an app experiences while making cloud service API calls •  Service: One or more endpoints providing a function that can be accessed by clients over the web.

Example: s3.amazonaws.com •  Endpoint: Specific API on a cloud service. Examples include account creation, check-out, or in-app purchase

Methodology We’ve gathered data from 1B users in over 120 countries and analyzed 3B+ events/day over a month. *Mobile app uptime/crash metrics in this report are most accurate when used within the context of the platforms they reside on as there are differences in how platforms report app loads.