1

The Data Flywheel: Building momentum by putting your data to work

Take a comprehensive approach to getting the most value from your data, one project at a time

2

Gets the high elasticity, low latency, and real-time processing needed to power its live leaderboard and deliver customizable rider data to its community of virtual bike riders.

Can now manage up to 8X more riders during peak times. Gains customer insights that power its shared-ride product. Leverages scalability to store GPS coordinates for all rides.

Stores user profiles and serves up relevant ads through data, both growing from zero to hundreds of millions of active users in a short time.

Uses data to personalize search and recommended results. Says that AWS is helping it to prepare for more growth.

Data is your most valuable business asset

What do Airbnb, Lyft, Snapchat, Tinder, and Peloton all have in common?

Apart from being successful startups founded in the last 5-10 years, all of these companies have built their businesses on the foundation of maximizing the value of data. In other words, they have figured out how to “put their data to work.”

And that’s critical because data has transformed from a cumbersome, expensive-to-store asset into the lifeblood of many successful organizations. Some have gone so far as to say that data has replaced oil as the world’s most valuable resource.1

Migrating to AWS drives results

2

1 https://www.economist.com/leaders/2017/05/06/the-worlds-most-valuable-re-

source-is-no-longer-oil-but-data

INTRODUCTION

3

INTRODUCTION

Points two and three from this list work together to significantly increase the value of data. Your organization can now store every relevant piece of data, even as those stores grow to massive volumes. And it can use a variety of analytics approaches to derive better insights from that data, driving better decisions and, ultimately, giving you a competitive edge.

In many ways, connected devices and the cloud have democratized data, enabling companies of every size to use data in innovative ways that spark business growth. Organizations today are using data to determine when and how to develop and release new products, opportunities for new revenue streams, where to automate manual processes, how to earn customer trust, and more. All of these decisions fuel innovation and drive your business forward.

But what’s the best model—the best strategy—for analyzing this data and applying the resulting insights in a structured way? In this e-book, we’ll introduce you to the Data Flywheel, a comprehensive approach to managing your data strategy.

The flywheel—as popularized by Jim Collins—is a self-reinforcing loop made up of key initiatives that feed into and are driven by each other. Recently, AWS unveiled the Data Flywheel: a holistic framework that applies this strategy to help you derive the most value possible from your data.

Before we get into the specifics of the Data Flywheel, let’s take a look at some leading businesses that are using the philosophies behind the flywheel to achieve data-driven success today.

1 Connected devices, apps, and systems now generate more data than ever before

2 The cloud has reduced the cost of storage, with businesses now able to hold on to all of their relevant data

3 The cloud provides pay-as-you-go, on-demand compute, allowing organizations to more easily analyze and gain insights from data

How did we get here?

Over the last 5-7 years, three key trends have significantly upgraded the value of data:

4

Fortnite is a multiplayer, online, third-person shooter. The game has become a phenomenon, now attracting more than 250 million players.

Fortnite is free to play, with its entire revenue generated by in-game micro-transactions. That means the game’s parent company, Epic Games, can only make money from Fortnite by capturing the attention of gamers through player engagement. The higher the player’s level of engagement, the more likely they are to spend via micro-transactions. In order to do this effectively, Epic collects billions of records on a daily basis, tracking virtually everything happening within the game:

• How players interact

• How often they use certain weapons

• The strategies they use to navigate the game universe

Currently, Epic stores more than 14 petabytes of Fortnite data, and that number is growing by two petabytes per month. Epic is able to efficiently handle and derive accurate insights from this massive data volume through real-time and batch analytics.

These analytics help Epic decide how to design new maps, what weapons to distribute, where to re-spawn players—anything to make the game more engaging. And Epic never stops learning—when the company introduces new elements and updates, it uses analytics to learn how the Fortnite community is responding, and then adjusts the game design accordingly.

Thanks to real-time, data-driven decisions, Fortnite continues to drive higher levels of player engagement, continually proving the viability of its micro-transaction model and allowing it to maintain its position as one of the most popular and successful games on the market.

Watch the presentation >

How today’s companies are winning with data

CUSTOMER EXAMPLES

CUSTOMER EXAMPLES

5

Peloton sells stationary bikes with 22” touch-screen monitors where riders can watch and interact with live and on-demand classes. The company has built a community of over a million riders who are connected to each other and instructors—all in the comfort of their own homes.

Peloton’s business model is based on a subscription service. Apart from buying the equipment, users purchase a yearly or monthly subscription to access the live and on-demand classes. The renewal rate of that subscription is entirely dependent on one factor: user engagement.

One of the primary tools Peloton uses to drive user engagement is a capability it has built into its application called the “leaderboard.”

The leaderboard lets users see the other riders currently using the application, showing their overall rank, comparisons to their own personal bests, and total power output—all in real time. This helps create an environment where riders can encourage each other, compete with each other, and compete with themselves.

For the leaderboard to work, it must update tens of millions of data points —simultaneously and in real time. Peloton uses a specialized, purpose-built in-memory cache database to achieve this.

Peloton’s user base exceeded one million riders within its first five years. With the leaderboard and other data-driven innovations, enabling the user experiences that turn customers into loyal, long-time subscribers, Peloton’s rapid success offers further evidence of the transformative power of data.

Watch the video >

6

Get the most from your data with the Data Flywheel

GETTING STARTED

Hopefully, by now, you understand the value of data—and ready to realize self-sustaining business momentum through the Data Flywheel strategy. So how do you get started? Building your Data Flywheel consists of five basic steps. In the next sections, we’ll explore each of these steps individually.

4. Analyzedata in your data lake

5. Innovatewith ML and blockchain

1. Move dataand workloads to

the cloud

3. Builddata-driven apps

2. Runfully-managed databases,

store all your data

6

7

Move your data and workloads to the cloud

Data is growing exponentially, and on-premises solutions are becoming too cost prohibitive and complex to keep up with demands. That’s why hundreds of thousands of businesses are moving to the cloud. Unfortunately, you can’t snap your fingers and go straight from cloud zero to cloud heaven. You’ll need to undergo a cloud migration—and while migrating to the cloud is easier today than ever before, it’s not without its challenges. How you migrate to the cloud is as vital as where you migrate. AWS migration services can help you quickly and efficiently move data, workloads, databases, and applications to the cloud—all with minimal business disruption. AWS Database Migration Service (DMS) and CloudEndure Migration are available to address server, database, and application issues, respectively. AWS partners like CloudVelox, ATADATA, Racemi, and Attunity can also support your teams.

1

THE FIVE STEPS

8

4. Analyzedata in your data lake

5. Innovatewith ML and blockchain

3. Builddata-driven apps

Migrating to AWS drives results

THE FIVE STEPS

Innovates faster while saving millions of dollars. Decreased average network latency from 700 ms to less than 50 ms. Achieves 230% CPU consumption efficiency in data processing.

Reduced IT costs 6X. Spends 60-80% less time on maintenance, upgrades, and recovery processes. Set up a new service in one-fifth the time it would’ve taken on-premises.

Cut infrastructure costs by 50%. Collects and processes 100,000 messages/sec. Can focus more on its core product instead of infrastructure management.

1 Move your data and workloads to the cloud

a Migrate databasesa Migrate data warehousesa Streaming data

2. Runfully-managed databases,

store all your data

1. Move dataand workloads to

the cloud

9

The world of databases has long been a challenging place for most organizations. Old-guard database providers are expensive, proprietary, have high lock-in, and impose punitive licensing terms. Because of these issues and others, we’ve seen many companies trying to move as fast as they can to open source databases like MySQL, PostgreSQL, and MariaDB. However, getting high performance on open source databases is difficult, and the price of missteps is high—with poor quality data costing the U.S. $3.1 trillion annually.2

That's why AWS built Amazon Aurora, a MySQL and PostgreSQL compatible relational database built for the cloud that combines the performance and availability of high-end commercial databases with the simplicity and cost- effectiveness of open source databases. Aurora has 5x the performance of standard MySQL and 3x the performance of standard PostgreSQL, with the security, availability, and reliability of commercial-grade databases, at one-tenth the cost.

While Aurora is the best choice for relational databases, you might choose to lift-and-shift some databases to AWS first to immediately get the scalability, reliability, and ease of management that the cloud provides. This gives you the time you need to migrate any legacy code and exhaust the remaining term of your license agreements with other commercial database vendors.

2 https://www.ciodive.com/news/the-organizational-costs-of-data-thats-hidden-in-plain-sight/516405/

For these databases, we recommend Amazon RDS—a fully managed relational database service that can run your choice of database engines. Amazon RDS automates time-consuming administration tasks such as hardware provisioning, database setup, patching, and backups. It lets you scale with only a few clicks or an API call—often with no downtime. For high-availability, Amazon RDS can synchronously replicate your data to a standby instance in a different availability zone, automatically detecting any database failures and failing over to the standby instance within 30 seconds.

Run fully managed databases, store all your data2

THE FIVE STEPS

One reason getting high performance out of these newer database engines is difficult is the so-called “IT skills gap”—the expanding separation between the capabilities of emerging technologies and the ability of internal IT staffs to take full advantage of those new functionalities, particularly in the cloud. 31% of businesses cite “lack of the right skill sets to manage and derive maximum value from cloud investments” as one of their top three cloud challenges.3

But by switching to fully managed databases in the cloud, you can offload these tasks and others to your cloud provider—which means you no longer have to spend time on the undifferentiated heavy lifting of common database administrative tasks.

3 https://cdn2.hubspot.net/hubfs/1624046/2018%20Cloud%20Computing%20Executive%20Summary.pdf

Going the fully managed route will boost productivity and efficiency, but no matter who’s managing them, your databases, workloads, and applications will all benefit from the agility and elasticity of the cloud, which allows you to store and process data on-demand. No more figuring out what data you need and what data to throw away—in the cloud, you can seamlessly scale your workloads up and down, and pay what you consume.

Once you’re taking advantage of the scalability of the cloud and running fully-managed databases like Amazon Aurora, Amazon RDS, Amazon DynamoDB, and Amazon Redshift, you can start focusing your time on building applications that deliver the performance and high availability your business needs and your customers demand.

THE FIVE STEPS

Bridging the skills gap

10

2 Run fully managed databases, store all your data

Migrating to AWS drives results

THE FIVE STEPS

Moved from SQL Server to Amazon Aurora. Reduced its datacenter footprint by two-thirds. Gained better insight into the cost of its apps and systems.

Moved from Teradata to Amazon Redshift. Estimates tens of thousands of dollars a year savings in licensing costs. Can add capacity with a few clicks.

2 Run fully managed databases, store all your data

11

4. Analyzedata in your data lake

5. Innovatewith ML and blockchain

1. Move dataand workloads to

the cloud

3. Builddata-driven apps

2. Runfully-managed databases,

store all your dataa Save timea Save costsa Store all data

1212

Relational databases have been the cornerstone of every enterprise application for the past two decades. However, a “relational only” approach no longer works in today’s world. With the rapid growth of data—not just in volume and velocity but also in variety, complexity, and interconnectedness—the needs of databases have changed.

Many new applications that have social, mobile, Internet of Things (IoT), and global access requirements cannot function properly on a relational database alone. These applications need databases that can store TBs to PBs of new types of data, provide access to data with millisecond latency, process millions of requests per second, and scale to support millions of users anywhere in the world.

Purpose-built databases can more easily provide the real-time performance—with little to no downtime—that modern applications need when scaling to meet these new requirements. Like Airbnb, Lyft, and so many others, businesses of all sizes are using purpose-built databases today to build scalable and globally distributed applications.

Types of purpose-built databases include relational databases for transactional applications, key-value databases for internet-scale applications, in-memory data stores for caching and real-time workloads, graph databases for building applications with highly connected data, document databases to store semi-

structured data as documents, time-series databases for collecting, synthesizing, and deriving insights from time-series data, and ledger databases for when you need a centralized, trusted authority. A list and links to specific AWS solutions for each of these database types is available on the next page.

With these purpose-built databases, your business can start to innovate more rapidly, building what we call “data-driven applications.” These applications take full advantage of your data, using it to create new capabilities, improve existing functions, and enhance customer experiences.

Build data-driven applications3

THE FIVE STEPS

The process for building data-driven applications will differ greatly based on business needs and use cases. But in most instances, data-driven application design will involve these three fundamental considerations:

1 Choosing the appropriate purpose-built database type

2 Examining every application component to ensure proper architecture and scalability

3 Breaking down complex applications into highly distributed microservices

THE FIVE STEPS 3 Build data-driven applications

AWS purpose-built databases:

Key value: Amazon

DynamoDB

Graph: Amazon Neptune

Document: Amazon

DocumentDB

Time series: Amazon

Timestream

In-memory: Amazon

ElastiCache

Ledger: Amazon QLDB

Relational: Amazon

Aurora and Amazon RDS

13

14

THE FIVE STEPS

AWS purpose-built databases drive results

Achieves <10ms latency with Amazon DynamoDB and improves performance with Amazon ElastiCache. Complies with 400+ security regulations. Cuts data curation costs by as much as 30%.

Uses Amazon DynamoDB to store 31 billion items and enable 24k read units/sec and 3.3 write units/sec, Amazon ElastiCache to increase performance, and Amazon RDS for permanent storage.

Supports 600K+ trips for its rapidly growing rideshare app with 1M+ downloads, using a database architecture including Amazon Relational Database Service (RDS), Amazon DynamoDB, and Amazon ElastiCache.

14

4. Analyzedata in your data lake

5. Innovatewith ML and blockchain

3. Builddata-driven apps

2. Runfully-managed databases,

store all your data

a Agilitya Global distributiona Scale and performance

1. Move dataand workloads to

the cloud

3 Build data-driven applications

15

THE FIVE STEPS

Data is a difficult beast to tame. It’s growing exponentially, coming in from new sources, becoming more diverse, and is increasingly harder to process. If you’ve completed the last phase, building data-driven applications, those applications themselves are bringing in new data on customer patterns and preferences.

In today’s world, static data solutions are insufficient to produce the analytics your business needs. These stagnant data warehouses are usually:

• Siloed: With data trapped in silos, seeing the big picture is nearly impossible.

• Delayed: Analytics based on yesterday’s data arrive too late to gain competitive advantage.

• Expensive: When performing analytics costs more than the value of their insights, nobody wins.

• Cumbersome: You might get an analytics platform running, but few will likely be able to use it efficiently.

• Limited to specialists: Making more data-driven decisions means democratizing analytics.

Analyze data in your data lake4With all that in mind, how can your business possibly capture, store, and analyze its data at the near-real-time rate needed to remain competitive? To solve these issues and more, we recommend moving to a data lake architecture.

A data lake architecture allows you to store all your data once in a single place, in open formats that can then be analyzed by many types of analytics and machine learning services—faster and more efficiently than with traditional, siloed approaches. With a data lake architecture, multiple groups within your organization can benefit from analyzing a consistent pool of data than spans the entire business.

A data lake and analytics architecture breaks down walls and eliminates silos, allowing you to collect, process, store, and analyze all of your data more efficiently. With this architecture, you can derive more accurate and timely insights that drive smarter decisions, boost business outcomes, and give your Data Flywheel a further push toward self-sustaining momentum.

AWS provides the most comprehensive, secure, and cost-effective portfolio of services for every step of building a data lake and analytics architecture. These services include data migration, cloud infrastructure, management tools, analytics services, visualization tools, and machine learning. A list with links to specific AWS data lake and analytics solutions is available on the next page.

16

THE FIVE STEPS 4 Analyze data in your data lake

AWS data lake and analytics solutions

Data lake setup AWS Lake Formation: Build a secure data lake with just a few clicks. You define where your data resides and what policies you want to apply. AWS then collects, catalogs, and moves the data into your Amazon S3 data lake, cleans and classifies data using machine learning (ML) algorithms, and secures access to your sensitive data.

Interactive analytics Amazon Athena: Analyze data directly in Amazon S3 and Amazon Glacier using standard SQL queries. There’s no infrastructure to set up or manage —start querying data instantly, get results in seconds, and pay only for the queries you run.

Big data processing Amazon EMR: Process vast amounts of data using the Apache Spark and Apache Hadoop frameworks—easily, quickly, and cost effectively. This managed service supports 19 different open source projects, with managed EMR Notebooks for data engineering, data science development, and collaboration.

Data warehousing Amazon Redshift: Run complex analytic queries against petabytes of structured data—at less than a tenth the cost of traditional solutions. Includes Redshift Spectrum, which runs SQL queries directly against exabytes of structured or unstructured data in Amazon S3, without the need for unnecessary data movement.

Real-time analytics Amazon Kinesis: Collect, process, and analyze streaming data—such as IoT telemetry data, application logs, and website clickstreams—as it arrives in your data lake. Respond in real-time instead of waiting until all your data is collected.

Operational analytics Amazon Elasticsearch Service: Search, explore, filter, aggregate, and visualize your data in near real-time. Enable fast, simple application monitoring, log analytics, clickstream analytics, and more. Get easy-to-use APIs alongside the availability, scalability, and security that production workloads require.

Dashboards and visualizations Amazon QuickSight: Easily build stunning visualizations and rich dashboards that can be accessed from any browser or mobile device.

Customer recommendations Amazon Personalize: Improve customer engagement with ML-powered product and content recommendations, tailored search results, and targeted marketing promotions. No prior ML training necessary—you provide an activity stream from your applications and the solution will identify what is meaningful, select the right algorithms, and train and optimize a customized personalization model.

Predictive analytics Amazon SageMaker: Easily build, train, and deploy predictive ML models. Get everything you need to connect to your training data, select and optimize the best algorithm and framework, and deploy your model on auto-scaling clusters of Amazon EC2.

17

THE FIVE STEPS 4 Analyze data in your data lake

The AWS analytics portfolio

Data Movement

Database Migration Service | Snowball | Snowmobile | Kinesis Data Firehouse | Kinesis Data Streams | Managed Streaming for Kafta

Data Lake Infrastructure & Management

S3/Glacier Lake Formation AWS GlueNEW

Analytics

Redshift EMR (Spark& Hadoop)

AWS Glue(Spark & Python)

Athena ElastisearchService

Kinesis Data Analytics

Visualization, Engagement, & Machine Learning

Quicksight Pinpoint Comprehend Lex Polly Rekognition Translate TranscribeSageMaker

Visualization, Engagement, & Machine Learning

Quicksight Pinpoint Comprehend Lex Polly Rekognition Translate TranscribeSageMaker

11+ more11+ more

18

THE FIVE STEPS

AWS data lake and analytics solutions drive results

Built a data lake using Amazon S3 to store PBs of data. Uses analytics services from AWS—including Amazon EMR and Amazon Kinesis —to perform site personalization, advertising optimization, and build recommendations for over 100M homes.

Uses Amazon S3 as a data lake and Amazon EMR to look for fraud, market manipulation, insider trading, and abuse. Operates at 60% lower cost than with its previous on- premises solutions.

Data lakes allow Redfin to cost- effectively innovate and manage data on hundreds of millions of properties.

NASDAQ optimized performance, availability, and security while loading seven billion rows of data per day.

18

4. Analyzedata in your data lake

5. Innovatewith ML and blockchain

3. Builddata-driven apps

2. Runfully-managed databases,

store all your data

a New & faster insightsa Broader access to analytic

1. Move dataand workloads to

the cloud

4 Analyze data in your data lake

19

Now that your Data Flywheel has some momentum, it’s time to add in the automation and intelligence that power perpetual innovation. Machine learning (ML) can provide better customer experiences at a scale that was not possible before. With ML, customers of every size and industry can gain valuable insights into their businesses with ease and speed.

With blockchain, you can store your data with integrity and ensure that you have a complete, immutable, and verifiable history of changes of all your data. Additionally, blockchain enables you to perform transactions more efficiently and removes unnecessary intermediaries.

But ML and blockchain are so new that it’s hard to know how to implement them—particularly without costs and complexity growing out of control. Let’s take a look at some best practices for integrating ML and blockchain into your Data Flywheel.

AWS has the broadest and deepest set of machine learning and artificial intelligence (AI) services for your business. AWS is focused on solving

some of the toughest challenges that hold back machine learning from being in the hands of every developer. You can choose from pre-trained AI services for computer vision, language, recommendations, and forecasting; Amazon SageMaker to quickly build, train and deploy machine learning models at scale; or build custom models with support for all the popular open source frameworks.

Innovate with machine learning and blockchain5

THE FIVE STEPS

20

5 Innovate with machine learning and blockchain

Uses ML to improve personal connections and serve their customers better and faster by predicting their needs in real-time

Applies ML through AWS solutions—including Amazon SageMaker —to create smarter devices, design new products, and gain entry into new markets

Uses ML on AWS to automatically identify new tax deductions for their customers, saving each an average of $4,300

THE FIVE STEPS

Made with machine learning

Machine learning

Machine learning (ML) in the cloud has made it possible for businesses of any size to innovate rapidly to make the self-sustaining Data Flywheel possible. It is always recommended to start a ML project with a pilot. This will give you a path to learn, innovate, and expand. All ML pilots will be different, we suggest following these basic steps to begin:

1 Start with data strategy: A good starting point is to collect, prepare, and store relevant data.

2 Identify the business problem: Find your best use cases by considering how ML can improve customer experiences, increase efficiency and productivity, and drive innovation.

3 Choose your project: Evaluate possible pilot projects focused on your priority use cases, then determine what tools, skills, and budget are needed.

At AWS, our mission is to put machine learning in the hands of every developer and data scientist. To achieve this, AWS has the broadest and deepest capabilities for ML.

You can choose from pre-trained AI services for a variety of use cases including vision, language, recommendations, and forecasting. For developers familiar with ML, Amazon SageMaker is a fully-managed, modular service to build, train, and deploy ML models at scale. We also provide support for all the leading deep learning frameworks along with the most scalable infrastructure for custom ML models.

21

Blockchain

Blockchain and, more broadly, ledger solutions have the potential to provide substantial value for a number of industries. These blockchain networks and ledger applications are often used to solve two primary types of customer needs:

1 Decentralized Trust: Multiple parties transact in a decentralized manner without the need for a centralized, trusted authority. For example, a consortium of banks and export houses looking to perform cross-boundary transfer of assets amongst each other, without a centralized authority acting as a liaison. Amazon Managed Blockchain can help you build and manage blockchain networks that scale as usage grows while improving reliability and hardening durability.

2 Centralized Ledger: Multiple parties work with a centralized, trusted authority to maintain complete and verifiable record of transactions. For example, a retail customer looking to connect its suppliers with a verifiable history of product movement through its supply chain. With Amazon QLDB, you can put this use case to work with a purpose-built ledger database to achieve the transparency, security, immutability, and speed it takes for enterprise use.

THE FIVE STEPS

Using Amazon's blockchain technology, Singapore Exchange was able to identify new ways of trading financial assets across its global network.

Trading better with blockchain

21

5 Innovate with machine learning and blockchain

22

THE FIVE STEPS

22

4. Analyzedata in your data lake

5. Innovatewith ML and blockchain

1. Move dataand workloads to

the cloud

3. Builddata-driven apps

2. Runfully-managed databases,

store all your data

a Better experiencesa Deeper engagementa Efficient processes

As you learn how to effectively use emerging technologies such as ML and blockchain to get more value from your data, you’ll gain the insights you need to build more desirable products, create new business models, and discover untapped customer segments.

23

New data-driven apps, data lake architectures, products, and services create more data that can be stored and managed in the cloud, which allows organizations to develop new capabilities and apps, gain new insights, and deliver new products. This is a step-wise, repeatable process, which must be run project by project, like turning a flywheel, building momentum with each turn.

One of the most unique characteristics of the Data Flywheel is that no “one thing” powers it, and organizations that search for such a fundamentally basic solution will likely lose their way. The Data Flywheel moves by many components acting in concert, equating to a whole that’s greater than the sum of its parts.

Spinning your Data Flywheel

CLOSING SUMMARY

As you develop each of the components outlined in this ebook—implementing the most relevant technologies and procedures at every phase—your flywheel will begin to generate perpetual business momentum. The flywheel will help put your data to work, automating data management and analytics, and allowing you and your team the time needed to make smarter, data-driven decisions, while directing more resources toward the business objectives that matter most.

Want to learn more about how the Data Flywheel can help your organization achieve perpetual business momentum? Visit our interactive Data Flywheel site to dig deeper into each of its components.

Interact with the Data Flywheel >

4. Analyzedata in your data lake

5. Innovatewith ML and blockchain

1. Move dataand workloads to

the cloud

3. Builddata-driven apps

2. Runfully-managed databases,

store all your data