Upload
others
View
6
Download
0
Embed Size (px)
Citation preview
Success StoryYahoo Japan Adopts Clustered Data
ONTAP as a New OpenStack Infrastructure
for Service Developments. Deploying
Multiple VMs While Reducing Storage Costs
Adopting OpenStack as the cloud infrastructure
for development to support a rapid growth of
services
Yahoo Japan Corporation (hereinafter referred to
as Yahoo) is an information and communications
company operating Japan’s largest portal site
Yahoo! Japan. Yahoo! Japan, is a "problem-
solving engine" that solves people's and society's
problems with IT. It has developed a number
of user-focused services, such as search,
news, mail, auctions, and shopping. In recent
years, it has also set "detonation velocity,"
focusing on speed as the goal. They have
continued challenging themselves to develop
amazing services through a bold commitment
to "challenging 10-fold, failing 5-fold, and
succeeding 2-fold."
Since 2009, Yahoo has operated a private
cloud infrastructure, mainly used by application
developers, in order to consolidate physical
servers that increased rapidly in response to
growth. Utilizing an API that was developed
Industry
• Information and Communications Technology
The Challenge
• Build a cloud infrastructure with excellent
scalability that meets the needs for rapid
business growth
• Deploy an OpenStack infrastructure that can
respond flexibly to the diverse needs of each
service
• Adopt storage systems to support OpenStack
Block Storage
The Solution
• NetApp Clustered Data ONTAP® 8.2
- RAID-DP, FlexVol, Snapshot
• NetApp FAS system with active-active
configuration
• NetApp Flash Cache
• NetApp SnapMirror ® Software
• NetApp OnCommand™ management
software
Benefits
• Support non-stop operations of the cloud
infrastructure through high-availability storage
• Enable deployment of a large number of VMs
while maximizing disk utilization with FlexVol
• Possible development in disaster recovery
capacity between sites by utilizing NetApp
data protection feature
KEY HIGHLIGHTS
• Since 2009, Yahoo Japan Corporation has operated a private cloud infrastructure for
developing applications which support a variety of services.
• The number of vir tual machines increased to thousands along with a rapid growth
of services, causing the company to face major issues such as the scalability and
operational load of the cloud infrastructure.
• Yahoo decided to adopt open source OpenStack as a cloud infrastructure for the next
generation since it can respond flexibly to in-house feature improvements utilizing APIs.
• They adopted NetApp Clustered Data ONTAP, enabling non-stop operations of the
cloud infrastructure and supporting OpenStack Block Storage (development code
name: Cinder).
• Four nodes of NetApp FAS3220AE have been deployed, and a total of approx. 20,000 VMs
are in operation on the OpenStack infrastructure.
• Through advanced integration with FlexClone, they achieved a very high disk utilization while
deploying a large number of VMs.
• In the future, the OpenStack infrastructure will be deployed to more data centers, and
disaster recovery will be enhanced through storage integration between data centers with
SnapMirror.
ABSTRACT
in-house, the cloud infrastructure provides
an environment where resources such as
virtual machines, storage, and the network
can be controlled flexibly through a Web
browser. However, as the number of virtual
machines increased to one thousand units,
new challenges emerged: difficulty in efficient
expansion of hardware, increase in operational
management load, and so on.
The company has decided to adopt "OpenStack"
which can provide a more advanced and open
cloud infrastructure for the purpose of optimizing
data center resources and achieving more
efficient operational management. The company
has watched developments with OpenStack
since around 2010. By 2012 OpenStack had
significantly more components and reached a
level that has sufficient capacity for the company's
cloud infrastructure. Then, in the second half of
2012, the company started to consider introducing
OpenStack. In March 2013, they launched the
company's first OpenStack project.
"Although there are some solutions for commercial
server virtualization with which you can operate
a few thousands virtual machines, we decided
to deploy an open source cloud infrastructure
because we have made a variety of feature
improvements in-house for the convenience of
our users. In particular, we had the impression
that OpenStack has a very active community
and that its experts provide improvements and
bug fixes promptly. In addition, OpenStack's
rich components and APIs enable the cloud
infrastructure to be customized flexibly. For
example, OpenStack Identity (development
code name: Keystone) is used to unify account
information, which is managed separately for
each service using the account authentication
in our OpenStack infrastructure. We expect that
this will remove the need for traditional in-house
API developments and will promote efficiency for
infrastructure development." -- Daisuke Masaki
Non-stop operations and OpenStack integration
are the reasons for adopting Clustered Data
ONTAP
Possessing extensive operational experience
with Yahoo’s various service and development
platforms, NetApp products were selected
for storage systems that support OpenStack
infrastructure. Yahoo mainly used 7-Mode
for operations; however, they are currently
integrating scale-out Clustered Data ONTAP
to pursue a scalable storage infrastructure that
allows for 24/7 non-stop operations and that
can respond to the rapid growth of services. In
particular, the current Clustered Data ONTAP
8.2 supports a QoS (Quality of Service) feature
to optimize the storage performance for each
co-existing service and data synchronization
features between storage clusters (SnapMirror)
to help in creating a disaster recovery
environment. Another key development is a
NetApp storage dedicated driver (NetApp
Unified Driver for Clustered Data ONTAP),
which supports OpenStack Block Storage
(development code name: Cinder).
"When you use a dedicated driver provided
by NetApp, you can intelligently integrate
OpenStack Block Storage (hereinafter referred
to as Block Storage) processing into various
NetApp storage features. For example, when you
create a virtual machine on the Block Storage,
you can use the NetApp virtual cloning feature
(FlexClone) to deploy virtual machines. Without
FlexClone, disk space is physically consumed,
depending on the number of virtual machines.
FlexClone helps you minimize the disk usage
and lower the load on the disk drive at the same
time. NetApp also contributes significantly to the
OpenStack community, and this positive attitude
was also key to choosing NetApp products."
-- Noriyasu Atsuta
NetApp storage with 4 nodes total deployed in
data centers
Yahoo conducted a full-scale model testing
in conjunction with the vendors and system
integrators in October 2013. In the test, they
checked to see the differences between
7-Mode, commonly used in the company, and
Clustered Data ONTAP. They also checked
basic operations such as volume creation and
deletion for the Block Storage and snapshot
creation in the OpenStack environment--and
measured virtual machine performance. Based
on a number of these validation results, they
designed the overall system and established
an OpenStack infrastructure from February
to March 2014. The OpenStack infrastructure
adopted was OpenStack Havana Release, the
latest release at the time.
"We deployed Clustered Data ONTAP based NetApp storage
systems that quickly added support for integration with OpenStack
Block Storage."(Right)
Yahoo Japan Corporation
Systems Management Group
Site Operations Headquarters
Infrastructure Technology Department 3, Storage
Daisuke Masaki
(Left)
Yahoo Japan Corporation
Systems Management Group
Site Operations Headquarters
Infrastructure Technology Department 3, Storage
Noriyasu Atsuta
To support the OpenStack infrastructure, NetApp
FAS3220AE storage systems are deployed in
multiple node configurations with redundant
storage controllers. Storage controllers improve
overall storage read performance by incorporating
cache modules (NetApp Flash Cache) based on
flash technology. For access and management of
NetApp storage in the OpenStack environment,
a dedicated driver for Block Storage is used.
With this driver, you can choose iSCSI or NFS
as storage protocols, but the company has used
NFS for its convenient file-level storage access.
A virtual machine is started from its image file in
the volume at the moment, but they are planning
that it can also be started from its volume in the
future. The review phase is almost complete,
and a virtual machine is expected to be started
both from its image and from its volume soon.
"In our OpenStack infrastructure, a number
of virtual machines operate at the same time,
which produces a significant load on the storage
system. In addition, most of the virtual machines
are deployed by FlexClone, and local access
to the disk drive is likely to occur. Although our
NetApp storage is equipped with a SATA drive
due to its cost effectiveness, the system still
demonstrates high performance with Flash
Cache. In the future, access load is expected
to increase along with the diversification of
services. We will consider deploying an SSD-
based advanced caching technology, "Flash
Pool," if higher write performance is required. We
believe that we can use NetApp storage with
confidence, now and in the future, thanks to its
various options for further storage performance
enhancements." -- Daisuke Masaki
Data backup and restore for virtual machines are
performed using an OpenStack data protection
feature, but in case of an emergency, the
infrastructure operations team also has a backup
system in place that leverages NetApp storage
data protection. They are performing fine-tuned
multi-generational backup with Snapshot as well
as data synchronization to another volume with
SnapMirror. At present this is limited to the use
of SnapMirror within the same cluster, but they
plan to extend it to different clusters in the same
data center--or SnapMirror between different
clusters in the future. With this plan, they intend
to enhance the business continuity levels and
create a robust cloud environment that does
not stop services even if a large-scale disaster
would occur.
Planning to expand the OpenStack
infrastructure, assuming 50,000 virtual machines
The OpenStack infrastructures (in two locations)
are both operating with stability and have a high
degree of affinity with Clustered Data ONTAP.
These OpenStack infrastructure systems house
approximately 20,000 virtual machines in total
(as of April 2014), and their disk utilization is
maximized through advanced integration with
FlexClone. According to research by Yahoo, the
proportion of disk space that is actually used is
low compared to the sum of spaces allocated
to each virtual machine. These results show
that the company could reduce storage cost
effectively. If the OpenStack infrastructure will
be simultaneously used in many services in the
future, the storage I/O that occurs in one service
may affect other services. To address such
issues, the company plans to introduce the QoS
feature of Clustered Data ONTAP and to optimize
the storage I/O for each service.
The company also intends to expand the entire
infrastructure in stages to meet the expected
growth, assuming that 50,000 virtual machines
would start operations on the OpenStack
infrastructure by the end of 2014. In principle,
they plan to deploy OpenStack clusters with
approximately 4,000 operating virtual machines
in various data centers nationwide. Major version
upgrades for OpenStack are planned once
every six months. So the company intends to
always use the latest releases for any software
supporting the OpenStack infrastructure to
enhance the entire infrastructure. The company
is also expecting to link OpenStack infrastructure
systems between data centers to help disaster
recovery in the future.
"OpenStack has been upgraded almost every
half a year: the release was called "Icehouse"
in April 2014 and then "Juno" in October 2014.
In general, this six-month update cycle is
considered very short; however, Yahoo, with its
goal of "detonation velocity," needs to expand
OpenStack infrastructure rather quickly and
welcomes the speed of OpenStack upgrades.
We are carrying out technical validations of new
Approx. 20,000VMs total are in operation
Current production environment Future plan
Control
Provide Cinder Volume
Provide Cinder Volume
Approx. 20,000VMs total are in operation
SnapMirror synchronization between data centers
Control
www.netapp.com
COMPANY PROFILEOpenStack infrastructure in parallel with the
existing infrastructure operations so that we can
always deploy the latest version when building
an OpenStack infrastructure. In such cases,
we hope to integrate new NetApp storage
features and build a more attractive OpenStack
infrastructure." -- Noriyasu Atsuta
"As storage-related components, OpenStack
has Block Storage in addition to Object Storage
(development code name: Swift). Object
Storage is used to store unstructured files such
as images and videos attached to various
services as well as archiving data. We would
like to investigate how we can leverage Object
Storage in our services. A solution provided by
NetApp uses the NetApp E-Series as the Object
Storage data storage destination, and one of its
key benefits is the simple system configuration
compared with the existing storage combination.
If we decide to deploy Object Storage in the
future, we will definitely consider adopting
NetApp E-Series." --Daisuke Masaki
"Assuming virtual machines will increase up to 50,000 by the end of 2014,
we build OpenStack infrastructure in stages for our data centers"
NetApp, Inc.
NetApp provides effective storage and data management solutions
that enable dramatic cost reduction and enhance your company's
competitiveness. Discover our passion for helping companies around
the world go further, faster at www.netapp.com.
495 East Java Drive, Sunnyvale, CA 94089 USA TEL: +1 408 822 6000 URL: http://www.netapp.com E-mail: [email protected]
© 2014 NetApp, Inc. All rights reserved.
All specifications are subject to change without notice.
No portions of this document may be reproduced without prior written
consent of NetApp, Inc. NetApp, NetApp logo, Go further, faster,
Data ONTAP, Snapshot, FlexVol, MultiStore, FilerView are registered
trademarks of NetApp,Inc. in the United States and/or other countries.
All other brands or product names are trademarks or registered
trademarks of their respective companies. CSS-6712-0414
Yahoo Japan Corporation
http://www.yahoo.co.jp/
Headquarters Midtown Tower 9-7-1 Akasaka,
Minato-ku, Tokyo
Founded January 1996
Capital 8 billion 271 million yen
(as of March 31, 2014)
Number of Employees
Approximately 4,607
(as of March 31, 2014)
Businesses Internet Advertising business,
e -Commerce business,
Member Service business, etc.