EMC Hybrid Cloud Solution with VMware: Hadoop Applications Solution Guide 2.5

Solution Guide

EMC HYBRID CLOUD SOLUTION WITH VMWARE Hadoop Applications Solution Guide 2.5

EMC Solutions

Abstract

This document serves as a reference for planning and designing a Pivotal Hadoop solution that enables IT organizations to quickly deploy Hadoop as a service (HaaS) on an existing cloud.

August 2014

2 EMC Hybrid Cloud Solution with VMware Hadoop Applications Solution Guide 2.5

Copyright © 2014 EMC Corporation. All rights reserved. Published in the USA.

Published August 2014

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

EMC2, EMC, and the EMC logo are registered trademarks or trademarks of EMC Corporation in the United States and other countries. All other trademarks used herein are the property of their respective owners.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.

EMC Hybrid Cloud Solution with VMware Hadoop Applications Solution Guide 2.5

Part Number H13221

http://www.emc.com/legal/emc-corporation-trademarks.htm

http://www.emc.com/legal/emc-corporation-trademarks.htm

Contents


Contents

Chapter 1 Executive Summary 7

Document purpose ..................................................................................................... 8

Audience .................................................................................................................... 8

Solution purpose ........................................................................................................ 8

Business challenge .................................................................................................... 9

Technology solution ................................................................................................... 9

Chapter 2 EMC Hybrid Cloud Solution Overview 11

Introduction ............................................................................................................. 12

EMC Hybrid Cloud features and functionality ............................................................ 13

Automation and self-service provisioning ............................................................ 13

Multitenancy and secure separation .................................................................... 14

Workload-optimized storage ................................................................................ 14

Elasticity and service assurance .......................................................................... 14

Operational monitoring and management ............................................................ 15

Metering and chargeback .................................................................................... 15

Modular add-on components ............................................................................... 16

Chapter 3 EMC Hybrid Cloud Hadoop as a Service 19

Overview .................................................................................................................. 20

EMC Hybrid Cloud HaaS and IaaS ............................................................................. 20

Pivotal Hadoop ......................................................................................................... 21

Serengeti .................................................................................................................. 22

VMware vSphere Big Data Extensions ....................................................................... 22

Chapter 4 HaaS Component Integration 25

Overview .................................................................................................................. 26

Integrating Hadoop components with EMC Hybrid Cloud .......................................... 26

BDE Topology....................................................................................................... 26

Virtualized Hadoop .............................................................................................. 27

Configuring the platform ........................................................................................... 28

Installing and configuring BDE ............................................................................. 28

Installing and configuring PHD ............................................................................. 30

Installing and configuring EMC Hybrid Cloud IaaS ................................................ 33

Contents


Chapter 5 Creating vCO Workflows and vCAC Catalog Services for HaaS 35

Overview .................................................................................................................. 36

Importing and modifying custom vCO workflows ...................................................... 36

Modifying custom workflows ............................................................................... 36

Creating BDE Clusters ............................................................................................... 42

Creating new BDE clusters ................................................................................... 42

Configuring a Hadoop cluster ............................................................................... 42

Creating vCAC Catalog Services ................................................................................ 45

Accessing vCAC ................................................................................................... 45

Creating a new service blueprint .......................................................................... 45

Chapter 6 Use Cases: EMC Hybrid Cloud IaaS 49

Overview .................................................................................................................. 50

IaaS – storage services ............................................................................................. 50

Overview .............................................................................................................. 50

Use case 1: Storage provisioning ......................................................................... 50

Use case 2: Select virtual machine storage .......................................................... 54

Use case 3: Metering storage services ................................................................. 55

Summary ............................................................................................................. 56

Monitoring and capacity planning ............................................................................ 57

Monitoring ........................................................................................................... 57

Capacity planning ................................................................................................ 57

Capacity planning example .................................................................................. 60

Metering and chargeback ......................................................................................... 61

Chapter 7 Conclusion 65

Summary .................................................................................................................. 66

Appendix A References 67

VMware references ................................................................................................... 68

Contents


Figures Figure 1. EMC Hybrid Cloud key components ..................................................... 12

Figure 2. EMC Hybrid Cloud self-service portal ................................................... 14

Figure 3. EMC ViPR Analytics with VMware vCenter Operations Manager ............ 15

Figure 4. IT Business Management Suite overview dashboard for hybrid cloud .. 16

Figure 5. EMC Hybrid Cloud HaaS component overview ...................................... 21

Figure 6. Pivotal Hadoop (PHD) components ...................................................... 22

Figure 7. BDE and Serengeti stack ...................................................................... 23

Figure 8. BDE and vSphere deployment topology ............................................... 26

Figure 9. The evolution of virtual Hadoop ........................................................... 27

Figure 10. Configuring the SSO lookup service and management server IP addresses ........................................................................................... 29

Figure 11. Importing Hadoop binaries into BDE management server .................... 31

Figure 12. Removing the default Apache template from BDE ................................ 32

Figure 13. Importing custom workflows into vCO .................................................. 36

Figure 14. Using the validate workflows action .................................................... 37

Figure 15. How to edit the attributes .................................................................... 37

Figure 16. Editing and creating custom parameter passing .................................. 38

Figure 17. Launching scripts from the VCO ........................................................... 39

Figure 18. Launching of Micro Hadoop Cluster workflow ...................................... 40

Figure 19. Status of creation of Micro Hadoop cluster from BDE (vSphere web client) .................................................................................................. 41

Figure 20. Status of Micro Hadoop cluster creation from BDE vSphere Client ....... 41

Figure 21. Create and name a new Big Data Cluster ............................................. 42

Figure 22. Advance Service Designer ................................................................... 46

Figure 23. Edit Entitlement window ...................................................................... 46

Figure 24. vCAC Service Catalog showing Hadoop as a Service ............................ 47

Figure 25. Storage Services - Provision cloud storage .......................................... 51

Figure 26. Provision Cloud Storage – select vCenter cluster ................................. 52

Figure 27. Storage Provisioning – Select datastore type ....................................... 52

Figure 28. Storage provisioning – Choose ViPR storage pool ................................ 53

Figure 29. Storage provisioning – Enter storage size ............................................ 53

Figure 30. Provision Storage – Storage Reservation for vCAC Business Group ...... 53

Figure 31. Set storage reservation policy for virtual machine disks ...................... 54

Figure 32. Create new virtual machine storage profile for Tier 2 storage ............... 55

Figure 33. Automatic discovery of storage capabilities using EMC ViPR Storage Provider ............................................................................................... 55

Figure 34. VMware ITBM chargeback based on storage profile of datastore ......... 56

Figure 35. Choosing virtual machine consumption models and profiles ............... 58

Contents


Figure 36. Specifying configuration and projected capacity usage of new virtual machines ............................................................................................ 58

Figure 37. Capacity summary showing insufficient CPU and RAM resources ......... 59

Figure 38. Specifying number of hosts and amount of CPU and memory .............. 59

Figure 39. Specifying datastore size ..................................................................... 60

Figure 40. Compared scenarios ............................................................................ 60

Figure 41. Combined scenarios ............................................................................ 61

Figure 42. Categorized hybrid cloud environment cost overview .......................... 62

Figure 43. vSphere Cluster cost overview ............................................................. 63

Figure 44. Storage cost overview .......................................................................... 63

Chapter 1: Executive Summary


Chapter 1 Executive Summary

This chapter presents the following topics:

Document purpose ..................................................................................................... 8

Audience .................................................................................................................... 8

Solution purpose........................................................................................................ 8

Business challenge .................................................................................................... 9

Technology solution ................................................................................................... 9



Document purpose

This document serves as a reference for planning and designing a Pivotal Hadoop solution that enables IT organizations to quickly deploy Hadoop as a service (HaaS) on an existing cloud. The solution delivers infrastructure as-a-service (IaaS) capabilities to support big data application development. This document introduces the main features and functionality of the solution, the solution architecture and key components, and the validated hardware and software environment. It demonstrates the integration of Pivotal Hadoop Enterprise in the EMC® Hybrid Cloud solution.

The Pivotal Hadoop solution is a modular add-on to the EMC Hybrid Cloud solution. EMC Hybrid Cloud Solution with VMware: Foundation Infrastructure Reference Architecture 2.5 and EMC Hybrid Cloud Solution with VMware: Foundation Infrastructure Solution Guide 2.5 describe the reference architecture and the foundation solution upon which all the EMC Hybrid Cloud add-on solutions build.

The following documents provide further information about how to implement specific capabilities or enable specific use cases within the EMC Hybrid Cloud solution with VMware:

EMC Hybrid Cloud Solution with VMware: Data Protection Continuous Availability Solution Guide 2.5

EMC Hybrid Cloud Solution with VMware: Data Protection Disaster Recovery Solution Guide 2.5

EMC Hybrid Cloud Solution with VMware: Data Protection Backup Solution Guide 2.5

EMC Hybrid Cloud Solution with VMware: Security Solution Guide 2.5

EMC Hybrid Cloud Solution with VMware: Pivotal CF Platform as a Service Solution Guide 2.5

Audience

This document is intended for executives, managers, architects, cloud administrators, and technical administrators of IT environments who want to build a self-service Pivotal Hadoop-based Enterprise big data platform. Readers should be familiar with VMware vCloud Suite, Pivotal Hadoop, VMware Big Data Extensions (BDE), EMC ViPR®, general IaaS defined datacenter concepts, and how a hybrid cloud infrastructure accommodates these technologies and requirements.

Solution purpose

The EMC Hybrid Cloud solution enables EMC customers to build an enterprise-class, scalable, multitenant infrastructure that enables:

Complete management of the infrastructure and application service lifecycle

On-demand access to and control of network bandwidth, servers, storage, and security



Quick deployment of IaaS components to support HaaS-based services without IT administrator involvement

Scalable, elastic, flexible HaaS-based services for maximum asset utilization

Access to application services from a single platform for both business-critical and next-generation cloud applications

This solution provides the reference architecture and the best practice guidance necessary to integrate the key components and functionality of enterprise HaaS into an underlying EMC Hybrid Cloud infrastructure.

Business challenge

Today’s enterprise demands an agile development platform that can enable the continuous delivery, updating, and horizontal scalability of applications. The Pivotal Hadoop (PHD) platform enables developers to easily deploy, bind, and scale applications and data services. When integrated with VMware vCloud Automation Center, it delivers a self-service Pivotal Hadoop platform that facilitates rapid deployment and instant scaling or updating of Hadoop clusters.

HaaS interoperability with the underlying infrastructure needs to accommodate consumable new generation applications while maintaining existing end-to-end service delivery to provide:

Efficiency and flexibility

Fast, proactive responses for services requests

Easy as-a-service model of deployment

Adequate visibility into the cost of the infrastructure

Technology solution

This EMC Hybrid Cloud solution integrates the best of EMC, VMware, and Pivotal products and services, and empowers IT organizations to adopt an as-a-service implementation model of compute and storage infrastructure within the data center. Agile, elastic, on-demand, end-to-end IaaS provisioning is crucial to support a comprehensive, dynamic, and fast-growing big data environment.

The key solution components include:

EMC ViPR software-defined storage platform

VMware vCloud Suite cloud management and infrastructure

EMC and VMware integrated workflows

VMware NSX virtual networking technologies

VMware vSphere virtualization platform

VMware Big Data Extensions (BDE) with Project Serengeti

Pivotal Hadoop (PHD)



Chapter 2: EMC Hybrid Cloud Solution Overview


Chapter 2 EMC Hybrid Cloud Solution Overview


Introduction ............................................................................................................. 12

EMC Hybrid Cloud features and functionality ........................................................... 13



Introduction

The EMC Hybrid Cloud solution enables a well-run hybrid cloud by bringing new functionality not only to IT organizations, but also to developers, end users, and line-of-business owners. Beyond delivering baseline infrastructure as a service (IaaS), built on a software-defined data center (SDDC) architecture, the solution delivers feature-rich capabilities to expand from IaaS to business-enabling IT as a service (ITaaS). Backup as a service (BaaS) and disaster recovery as a service (DRaaS) are now policies that users can enable with just a few mouse clicks. End users and developers can quickly access a marketplace of resources for Microsoft, Oracle, SAP, EMC Syncplicity®, and Pivotal applications, and can add third-party packages as required. All of these resources can be deployed on private cloud or public cloud services, including VMware vCloud Air, from EMC-powered cloud service providers.

The EMC Hybrid Cloud solution uses the best of EMC and VMware products and services, and takes advantage of the strong integration between EMC and VMware technologies to provide the foundation for enabling IaaS on new and existing infrastructure for the hybrid cloud.

Figure 1 shows the key components of the EMC Hybrid Cloud solution. For detailed information, refer to EMC Hybrid Cloud Solution with VMware: Foundation Infrastructure Solution Guide 2.5. For information on EMC Hybrid Cloud modular add-on solutions, which provide functionality such as data protection, continuous availability, and application services, refer to Modular add-on components and to the individual Solution Guides for those add-ons.

Figure 1. EMC Hybrid Cloud key components



EMC Hybrid Cloud features and functionality

The EMC Hybrid Cloud solution incorporates the following features and functionality:

Automation and self-service provisioning

Multitenancy and secure separation

Workload-optimized storage

Elasticity and service assurance

Operational monitoring and management

Metering and chargeback

Modular add-on components

The solution provides self-service provisioning of automated cloud services to both users and infrastructure administrators. It uses VMware vCloud Automation Center (vCAC), integrated with EMC ViPR software-defined storage and VMware NSX, to provide the compute, storage, network, and security virtualization platforms for the SDDC.

Cloud users can request and manage their own applications and compute resources within established operational policies. This can reduce IT service delivery times from days or weeks to minutes. Automation and self-service provisioning features include:

Self-service portal—Provides a cross-cloud storefront that delivers a catalog of custom-defined services for provisioning workloads based on business and IT policies, as shown in Figure 2

Role-based entitlements—Ensure that the self-service portal presents only the virtual machine, application, or service blueprints appropriate to a user’s role within the business

Resource reservations—Allocate resources for use by a specific group and ensure that those resources are inaccessible to other groups

Service levels—Define the amount and types of resources that a particular service can receive during initial provisioning or as part of configuration changes

Blueprints—Contain the build specifications and automation policies that define the process for building or reconfiguring compute resources

Automation and self-service provisioning



Figure 2. EMC Hybrid Cloud self-service portal

The solution provides the ability to enforce physical and virtual separation for multitenancy, as strongly as the administrator requires. This separation can encompass network, compute, and storage resources to ensure appropriate security and performance for each tenant.

The solution supports secure multitenancy through vCAC role-based access control (RBAC), which enables vCAC roles to be mapped to Microsoft Active Directory groups. The self-service portal shows only the appropriate views, functions, and operations to cloud users, based on their role within the business.

The solution enables customers to take advantage of the proven benefits of EMC storage in a hybrid cloud environment. Using ViPR storage services, which leverage the capabilities of EMC VNX® and EMC VMAX® storage systems, the solution provides software-defined, policy-based management of block- and file-based virtual storage. ViPR abstracts the storage configuration and presents it as a single storage control point, enabling cloud administrators to access all heterogeneous storage resources within a data center as if the resources were a single large array.

The solution uses the capabilities of vCAC and various EMC tools to provide the intelligence and visibility required to proactively ensure service levels in virtual and cloud environments. Infrastructure administrators can add storage, compute, and network resources to their resource pools as needed. Cloud users can select from a range of service levels for compute, storage, and data protection for their applications and can expand the resources of their virtual machines on demand to achieve the service levels they expect for their application workloads.

Multitenancy and secure separation

Workload-optimized storage

Elasticity and service assurance



The solution features automated monitoring and management capabilities that provide IT administrators with a comprehensive view of the cloud environment to enable smart decision-making for resource provisioning and allocation. These automated capabilities are based on a combination of EMC ViPR Storage Resource Management (SRM), VMware vCenter Log Insight, and VMware vCenter Operations Manager (vC Ops), and use EMC plug-ins for ViPR, VNX, VMAX, and EMC Avamar® systems to provide extensive additional storage detail.

Cloud administrators can use ViPR SRM to understand and manage the impact that storage has on their applications and to view their storage topologies from application to disk, as shown in Figure 3.

Figure 3. EMC ViPR Analytics with VMware vCenter Operations Manager

Capacity analytics and what-if scenarios in vC Ops identify over-provisioned resources so they can be right-sized for the most efficient use of virtualized resources. In addition, for centralized logging, infrastructure components can be configured to forward their logs to vCenter Log Insight, which then aggregates the logs from all the disparate sources for analytics and reporting.

The solution uses VMware IT Business Management Suite (ITBM) to provide cloud administrators with comprehensive metering and cost information across all business groups in the enterprise. ITBM is integrated into the cloud administrator’s self-service portal and presents a dashboard overview of the hybrid cloud infrastructure, as shown in Figure 4.

Operational monitoring and management




Figure 4. IT Business Management Suite overview dashboard for hybrid cloud

The EMC Hybrid Cloud solution provides modular add-on components for the following services:

Application services

This add-on solution leverages VMware vCloud Application Director to optimize application deployment and release management through logical application blueprints in vCAC. Users can quickly and easily deploy blueprints for applications and databases such as Microsoft Exchange, Microsoft SQL Server, Microsoft SharePoint, Oracle, and SAP.

Data protection services

EMC Avamar and EMC Data Domain® systems provide a backup infrastructure that offers features such as deduplication, compression, and VMware integration. By using VMware vCenter Orchestrator (vCO) workflows customized by EMC, administrators can quickly and easily set up multitier data protection policies and enable users to select an appropriate policy when they provision their virtual machines.

Continuous availability

A combination of EMC VPLEX® virtual storage and VMware vSphere High Availability (HA) provides the ability to federate information across multiple data centers over synchronous distances. With virtual storage and virtual servers working together over distance, the infrastructure can transparently provide load balancing, real time remote data access, and improved application protection.

Modular add-on components



Disaster recovery

This add-on solution enables cloud administrators to select disaster recovery (DR) protection for their applications and virtual machines when they provision their hybrid cloud environment. ViPR automatically places these systems on storage that is protected remotely by EMC RecoverPoint® technology. VMware vCenter Site Recovery Manager automates the recovery of all virtual storage and virtual machines.

Platform as a service

The EMC Hybrid Cloud solution provides an elastic and scalable IaaS foundation for platform-as-a-service (PaaS) and software-as-a-service (SaaS) services. Pivotal CF provides a highly available platform that enables application owners to easily deliver and manage applications over the application lifecycle. The EMC Hybrid Cloud service offerings enable PaaS administrators to easily provision compute and storage resources on demand to support scalability and growth in their Pivotal CF enterprise PaaS environments.



Chapter 3: EMC Hybrid Cloud Hadoop as a Service


Chapter 3 EMC Hybrid Cloud Hadoop as a Service


Overview .................................................................................................................. 20

EMC Hybrid Cloud HaaS and IaaS ............................................................................. 20

Pivotal Hadoop ......................................................................................................... 21

Serengeti ................................................................................................................. 22

VMware Big Data Extensions .................................................................................... 22



Overview

This chapter identifies and briefly describes the major features and functionality required to support Pivotal Hadoop as a service and promote scalability in the EMC Hybrid Cloud environment.

EMC Hybrid Cloud HaaS and IaaS

Project Serengeti

VMware Big Data Extensions (BDE)

Pivotal Hadoop (PHD)

HaaS Self-Service Portal

EMC Hybrid Cloud HaaS and IaaS

EMC Hybrid Cloud HaaS is a solution stack made up of EHC IaaS, integrated with BDE and PHD. The self-service aspect of the portal is controlled by vCAC as shown in Figure 5.

Hadoop is an open-source software program that supports the processing of large data sets in a distributed computing environment. It is part of the Apache project sponsored by the Apache Software Foundation. PHD is an Apache Hadoop distribution.

Deploying a Hadoop cluster using traditional methods is complex and time-consuming. It typically involves setting up the infrastructure, installing and configuring the operating system, acquiring the respective Hadoop media, installing Hadoop components, and finally creating the Hadoop cluster.

This process typically takes weeks and requires a significant skillset. The EMC HaaS offering simplifies the process by using extensive workflow automation in the EHC IaaS backend. Through self-service automation, it is now possible to deploy or expand a Hadoop cluster in minutes using the vCloud Automation Center self-service portal.



Figure 5. EMC Hybrid Cloud HaaS component overview

Pivotal Hadoop

Pivotal Hadoop (PHD) is an open-source software program that supports the processing of large data sets in a distributed computing environment. It is part of the Apache project sponsored by the Apache Software Foundation. PHD is an Apache Hadoop distribution. The complete PHD platform contains a number of components that are not specifically used within this solution:

YARN (Yet Another Resource Negotiator)—a distributed processing framework that can schedule and execute resource requests from multiple applications

HBASE—a column database that runs on top of the Hadoop Distributed Files System (HDFS)

HAWQ—HAWQ is a parallel SQL query engine that combines the merits of the Greenplum Database Massively Parallel Processing (MPP) relational database engine and the Hadoop parallel processing framework

ZooKeeper—a centralized service for maintaining configuration information, naming services, providing distributed synchronization, and providing group services

Hive—a data warehouse infrastructure built on top of Hadoop infrastructure

Hadoop Map Reduce—Map Reduce is a programming model for processing and generating large data sets with a parallel, distributed algorithm on a cluster



Figure 6 shows the PHD components.

Figure 6. Pivotal Hadoop (PHD) components

Note: YARN, HBASE, HAWQ and HIVE are not referenced in this solution. HAWQ is not installed by default and must be installed separately. This can be automated through the use of vCO workflows if required.

Serengeti

Serengeti is an open source project initiated by VMware to enable the deployment and management of Hadoop and big data clusters in a vCenter Server managed environment. The key components are the Serengeti Management Server, which provides a framework for running big data clusters on vSphere, and a command line interface that provides tools and utilities that form an administrative interface for managing and monitoring the cluster environments.

VMware vSphere Big Data Extensions

VMware vSphere Big Data Extensions, or BDE, is a feature within vSphere to support big data and open source Hadoop distribution workloads. BDE provides an integrated set of management tools to help enterprises deploy, run, and manage Hadoop on a common virtual infrastructure. Figure 7 shows how BDE is an installable virtual appliance plug-in that controls and monitors Hadoop Services. The BDE virtual appliance runs on top of vSphere and uses the Serengeti Management Server to control cluster creation by cloning templates through the template server.

BDE is a commercial version of Serengeti, which is an open source project from VMware. BDE provides the features of Serengeti in an enterprise format, including:

An open source supported version of the Apache Hadoop Distribution



The big data extensions GUI which is integrated into vSphere Web Client to perform Hadoop infrastructure and cluster management tasks

Elastic-enabled clusters that optimize and provide scaling of physical compute resources in a vSphere environment

Figure 7. BDE and Serengeti stack



Chapter 4: HaaS Component Integration


Chapter 4 HaaS Component Integration


Overview .................................................................................................................. 26

Integrating Hadoop components with EMC Hybrid Cloud ......................................... 26

Configuring the platform .......................................................................................... 28



Overview

This section provides guidance on configuring the services required for Hadoop as a Service, specifically BDE and PHD, and integrating them with EMC Hybrid Cloud IaaS services.

Integrating Hadoop components with EMC Hybrid Cloud

To install and configure Hadoop-as-a-Service components, refer to the appropriate vendor documentation referenced in in the installing and configuring sections for the component in this chapter.

The steps discussed assume that the EMC Hybrid Cloud has been installed and configured as described in the EMC Hybrid Cloud Solution with VMware – Foundation Intrastructure Solution Guide 2.5, and that the IaaS, portal, catalog services, and tenant structure are all in place.

BDE runs on top of Serengeti. Figure 8 shows the virtual appliance that runs the Serengeti Management Server and Template Server. BDE provides the GUI for managing Hadoop clusters, communicating through the Serengeti Management Server.

Figure 8. BDE and vSphere deployment topology

With VMware’s vSphere Big Data Extensions, you can enable deployment of Hadoop inside your VMware vSphere environment. The Big Data Extensions are distributed as a downloadable OVA-based virtual appliance that is imported into an existing environment. The minimum requirements to support BDE are vSphere 5.0 or later and Enterprise or Enterprise plus vSphere licenses. By default, the basic Apache Foundation distribution of Hadoop is also included, but it is very easy to add in other commercial Hadoop distributions such as Pivotal Hadoop, Cloudera Hadoop, Hortonworks Hadoop, or MapR Hadoop. This solution uses the Pivotal Hadoop distribution integrated with the EMC Hybrid Cloud IaaS stack to create Hadoop as a Service.

BDE Topology



After BDE is installed, you can begin creating a virtual Hadoop cluster. You can specify a number of configuration options including distribution, topology (basic, compute/storage separation, HBase-only, or custom), and the number and size of the virtual machines for each of the Hadoop roles (for example, name node, client node, and data nodes). Note the options presented in the web interface are only a fraction of what can be invoked through the advanced command-line tools and API.

When you start to deploy a Hadoop cluster, BDE clones the appropriate virtual machines and automatically builds out the cluster. When you are satisfied with the cluster, you can scale up (increase the size of the virtual machine’s memory and CPU resources) or scale out (increase the number of virtual machines). You can configure the cluster to scale automatically as the load alters for additional flexibility and efficiency.

Some of the benefits of virtualizing Hadoop—for example, elasticity and multi-tenancy—arise from the increased number of deployment options that become available when Hadoop is virtualized. Figure 9 shows the evolution of virtual Hadoop, from self-contained to a tenant-based model.

Figure 9. The evolution of virtual Hadoop

The traditional Hadoop model combines compute and data. While this implementation is straightforward, representing how the physical Hadoop model can be directly translated into a virtual machine, the ability to scale up and down is limited because the lifecycle of this type of virtual machine is tightly coupled to the data it manages. Powering off a virtual machine with combined storage and computing means access to its data is lost. Scaling out by adding more nodes would necessitate rebalancing data across the expanded cluster, so this model is not particularly elastic.

Separating computing from storage in a virtual Hadoop cluster can achieve compute elasticity, enabling mixed workloads to run on the same virtualization platform and improving resource utilization. It is simple to configure using a HDFS data layer that is always available, along with a compute layer comprising a variable number of TaskTracker nodes, which can be expanded and contracted on demand.

Extending the concept of data-compute separation, multiple tenants can be accommodated on the virtualized Hadoop cluster by running multiple Hadoop

Virtualized Hadoop



compute clusters against the same data service. Using this model, each virtual compute cluster enjoys performance, security, and configuration isolation.

While Hadoop performance using the combined data-compute model on vSphere is similar to its performance on physical hardware, providing virtualized Hadoop increased topology awareness can enable the data locality needed to improve performance when data and compute layers are separated. Topology awareness allows Hadoop operators to realize elasticity and multi-tenancy benefits when data storage and computing are separated. Furthermore, topology awareness can improve reliability when multiple nodes of the same Hadoop cluster are colocated on the same physical host.

To optimize the data locality and failure group characteristics of virtualized Hadoop:

Group virtual Hadoop nodes on the same physical host into the same failure domain, and avoid multiple replicas.

Maximize usage of the virtual network between virtual nodes on the same physical host. The virtual network has higher throughput and lower latency than the physical network and does not consume any physical switch bandwidth

Configuring the platform

Refer to VMware vSphere Big Data Extensions Administrator's and User's Guide to install and configure the BDE components required for Hadoop as a Service.

Configuration task order

The following steps outline the high-level tasks you need to perform to install and configure BDE:

1. Ensure the environment meets the minimum vSphere requirements, correct licensing is in place, and compute, storage and networking pre-requisites are met.

2. Configure cluster settings, including vSphere HA, Distributed Resource Scheduling, host monitoring, and admission control.

3. Configure network settings using either vSwitch, vSphere Distributed Switch (vDS), or NSX. Ensure the required ports are configured as part of any firewall policy.

4. Deploy the BDE OVF file and assign the management network. When you deploy BDE the setup will ask for a destination port group; this is the network that the management network uses to communicate with the server so the port group should be the same as the VLAN ID. If vCenter or BDE are unable to communicate with each other, then the integration will fail.

Configuring SSO service

As part of the configuration process an important step is to configure the SSO service and management server IP addresses.

1. As shown in Figure 10, from the left pane in the Deploy OVF Template page select Customize template.

Installing and configuring BDE

http://www.vmware.com/files/pdf/vsphere/vsphere-big-data-extensions-administrators-users-guide-b1.pdf



2. In the VC SSO Lookup Service URL box, type the vCenter Server Fully Qualified Domain Name FQDN in the same format as shown (if the default server name has not been changed). If you do not specify the FQDN here, then the certificate will not be accepted and there will be a connection issue between BDE and the Serengeti server later.

3. Under Management Server Network Settings, enter the appropriate IP address settings.

Figure 10. Configuring the SSO lookup service and management server IP addresses

Starting BDE in vSphere

After successfully installing and configuring BDE within vSphere, power on the BDE management server and then register BDE within vSphere as the final part of configuration by performing the following steps:

1. Log in to the vSphere client with administrative privileges.

2. Within the vSphere client, locate the BDE management server. The management server is located under the datacenter resource pool in which it was deployed.

3. Select and record the management IP address.

4. Register the management server using the register plugin URL: https://management-server-ip-address:8443/register-plugin where management-server-ip-address is the IP address you recorded in step 3.

5. Complete the required registration information and then click Submit.

The BDE icon should now be available in the list of objects within the inventory.

https://management-server-ip-address:8443/register-plugin



Before installing and configuring PHD, download the following required components and make them available for the installation:

Cent OS 6.2 64 bit ISO

Pivotal Hadoop Tar files

Oracle JDK 7, 64 bit rpm for Cent-OS

Big Data Extension OVF

VMware BDE comes supplied with a default Hadoop distribution from Apache. The HaaS integration requires that Pivotal Hadoop be installed. Get the Pivotal Hadoop media and documentation from http://www.gopivotal.com/big-data/pivotal-hd, and register and obtain the necessary licenses. The following high level tasks outline the process to load the media and create a PHD template within the BDE configuration.

Installing PHD

To create the required installation configuration for BDE, use Yum repositories (as opposed to a TAR-ball). When you create a Hadoop cluster that is YUM-deployed, the Hadoop nodes within the cluster then download the Red Hat Package Manager (RPM) packages for the Pivotal Hadoop distribution from the official Yum repositories.

The Pivotal Hadoop distribution must be installed in a 64-bit version of the CentOS 6.x operating system. You must use either CentOS 6.2 or CentOS 6.4 to create the Hadoop template virtual machine . The template is used in the cloning process for creating a Hadoop cluster. After you have deployed the BDE OVF you must follow the steps to integrate YUM into PHD by creating a YUM repository as outlined below, and then create the template.

Creating a Yum repository for PHD The steps for configuring PHD with BDE are described in the VMware vSphere Big Data Extensions Administrator’s and User’s Administration Guide.

Creating a Hadoop template virtual machine You must use either CentOS 6.2 or CentOS 6.4 to create the Hadoop template virtual machine. To upgrade from a previous version, refer the chapter titled “Create a Hadoop Template Virtual Machine using RHEL Server 6.x” in the VMware vSphere Big Data Extensions Administrator’s and User’s Administration Guide.

The following steps outline the procedure for creating a Hadoop template virtual machine:

1. Import the PHD binaries and create PHD media by logging into the BDE management server and importing the PHD tar files into an appropriate directory structure on the server. Figure 11shows the binary import process.

Installing and configuring PHD

http://vault.centos.org/6.2/isos/x86_64/

https://network.gopivotal.com/products/pivotal-hd-service

http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html

https://my.vmware.com/web/vmware/details?downloadGroup=BDE_110_GA&productId

http://www.gopivotal.com/big-data/pivotal-hd



http://pubs.vmware.com/bde-2/topic/com.vmware.ICbase/PDF/vsphere-big-data-extensions-20-admin-user-guide.pdf




Figure 11. Importing Hadoop binaries into BDE management server

2. Test that the import was successful by accessing the URL path from a browser and ensuring that the expected folders are present.

3. After installing the media into the BDE management server, create a new Pivotal Hadoop template.

4. Make the new Pivotal Hadoop template the default template by removing the default Hadoop Apache template from the BDE management server, as shown in Figure 12.



Figure 12. Removing the default Apache template from BDE

Configuring custom resources for BDE

VMware BDE requires two resources types when automating Hadoop clusters: networking resources and storage resources.

Networking resources Networking is used to assign virtual machines IP addresses. BDE deploys all nodes of a Hadoop cluster from a single common CentOS template that comes preconfigured with the BDE vApp management server. As BDE deploys virtual machines into a cluster, it uses either an existing DHCP server or a statically created IP address pool. As part of the deployment process, hostnames are assigned by BDE. The hostnames are the same as the IP addresses. For example, if DHCP assigns 10.10.10.10 then the hostname of that virtual machine is 10.10.10.10. Hadoop then uses this hostname for the clusters.

Storage resources BDE defines two types of storage resources—local and shared. Shared storage is useful for management or client servers deployed by BDE as shared storage can be protected with technologies such as VMware HA.

Within Hadoop there are two types of nodes: master and worker nodes. Master nodes provide tracking functions whereas worker nodes provide job processing capabilities. Because worker nodes are disposable, they do not require top tier storage since Hadoop is designed to deal with node failure. There is also no reason to deploy worker nodes on shared storage. The choice of storage however must be capable of dealing with the required level of performance for the nodes. Allowing BDE



to use local VMFS storage for worker nodes is analogous to deploying physical worker nodes on commodity storage using direct attached storage.

The final stage of configuration is to assign storage resources to BDE. This defines how the Hadoop clusters are deployed, either using local or shared datastores. By default BDE defines data stores as local. If you need shared datastores, you must configure the datastores accordingly. Refer to Chapter 6 of the VMware vSphere Big Data Extensions Administrator’s and User’s Guide for details on how to add datastores and networks to a cluster from the vSphere client.

For details. refer to the EMC Hybrid Cloud Solution with VMware - Foundation Infrastructure Reference Architecture 2.5. Detailed installation and configuration information is available only to select EMC personnel and authorized partners.

Installing and configuring EMC Hybrid Cloud IaaS





Chapter 5: Creating vCO Workflows and vCAC Catalog Services for HaaS


Chapter 5 Creating vCO Workflows and vCAC Catalog Services for HaaS


Overview .................................................................................................................. 36

Importing and modifying custom vCO workflows ..................................................... 36

Creating vCAC Catalog Services ............................................................................... 45



Overview

The automation of Hadoop clusters is achieved by using custom workflows created with VMware vCloud Orchestrator (vCO). This chapter describes how these workflows are configured from within VMware Cloud Automation Center (vCAC) to present enterprise organizations with a self-service portal that includes a catalog of pre-configured Hadoop deployment scenarios.

Importing and modifying custom vCO workflows

To use HaaS within EMC Hybrid Cloud, the administrator must use custom vCO workflows for deploying HaaS. These workflows offer a choice of cluster sizes that can then be presented as catalog items from the vCloud Automation Center portal. The workflows are imported into VMware vCO using the vCO import function to be edited, tested, and packaged according the needs of the organization.

This section describes the process for importing the custom workflows into vCO, so that the Hadoop Administrator can alter them and link them with the big data cluster configurations created in the earlier stages of the process.

Importing custom workflows

From within the vCO client, as shown in Figure 13, select Run, click Workflows, and select Import workflow. Browse to the location where you have placed the workflow package and click Open. The imported workflow appears in the folder selected.

Figure 13. Importing custom workflows into vCO

Validating workflows

After importing the workflows into vCO, validate them by clicking the name of the folder containing the workflows and then selecting the Validate option from the context menu, as shown in Figure 14. The validation process ensures there are no

Modifying custom workflows



open ends, unreachable workflow elements, or unused attributes in the workflows, so that they will execute correctly.

Figure 14. Using the validate workflows action

Customizing HaaS workflows

The HaaS workflows provide a framework for deploying each Hadoop cluster configuration of a given size through an automated workflow. The Hadoop administrator should modify the attributes of these workflows to meet the specific needs of the organization. Figure 15 shows how to use the vCO client to edit the attributes within a workflow.

Figure 15. How to edit the attributes



Configuring custom parameters

To make the workflows dynamic, vCO uses a combination of attributes and parameters to transfer data when it is processing a workflow. Workflow parameters must receive an input to generate an output or action. An example of configuring a custom parameter is when an input is received from the user or system. The input can then be passed to a command or script that would create a username or password, This in turn can be passed to the Hadoop cluster for authentication.

Figure 16 shows how to create a custom username and password for the Hadoop Client node.

Figure 16. Editing and creating custom parameter passing

Launching a custom script

Scripts help to edit the schema, which is the main component of a workflow. Launching individual scripts lets you test the components of the workflow one element at a time, or execute a script at runtime to prepare the data set, for example.

Figure 17 shows how to launch scripting from within the workflow by using the Schema panel within the workflow itself.



Figure 17. Launching scripts from the VCO

Testing VCO HaaS custom workflows

The previous sections demonstrated how to import the HaaS sample workflows into EMC Hybrid Cloud, specifically the vCenter Orchestrator which is the main orchestration and automation engine for the solution. As shown, once imported, the default workflows can be altered to meet any modifications made to the Hadoop clusters. The workflows can also be modified to pass any additional parameters that may be required, for example, passing the username and password or executing parts of additional scripts components.

The final stage in importing and configuring the workflows is to test the workflows that have been imported and modified for each of the HaaS cluster sizes (micro cluster, small cluster, and large cluster). Figure 18 shows how to:

Select the specific workflow for a given cluster size

Execute the workflow from vCO

View the execution process

Verify the execution progress by checking the log files for any error messages



Figure 18. Launching of Micro Hadoop Cluster workflow

Viewing cluster creation

After the VCO workflow is launched, the cluster creation process starts within vSphere and BDE. The management server uses the template server to clone the nodes required to create the cluster in terms of the numbers and types of node that comprise the cluster. To view and verify the cluster creation process, follow these steps:

1. Login to the vSphere web client.

2. Go to the BDE and view the actual cluster being created.

Figure 19 shows the status of the creation of a micro Hadoop cluster in the BDE panel of the vSphere web client.



Figure 19. Status of creation of Micro Hadoop cluster from BDE (vSphere web client)

You can also log in to the vSphere Client Application and view the Hadoop cluster being created. Figure 20 shows the status of the creation of the Micro Hadoop cluster in the vSphere Client Application.

Figure 20. Status of Micro Hadoop cluster creation from BDE vSphere Client



Creating BDE Clusters

After the vCO workflows are imported they need to be customized for the different sized clusters according to the requirements of the enterprise. The examples provided describe micro, small, and large Hadoop clusters.

The custom workflows define the type of the cluster, including cluster configuration, in terms of the number of master nodes, client nodes, and data nodes for each size.

Creating a Hadoop cluster

These steps document the procedure for creating a Hadoop cluster within BDE, which can then be translated when building a VCO workflow:

1. In vCenter, under Objects > Data Extensions,click New Big Data Cluster.

2. Follow the steps in the wizard, specifying the appropriate parameters as required. More detail can be found in the VMware vSphere Big Data Extensions Administrator’s and User’s Guide.

The following sections outline the options and details required during the cluster configuration process.

Naming a Hadoop cluster

When prompted by the wizard, type a name to identify the cluster. Valid characters for cluster names are alphanumeric and underscores. When choosing a cluster name you should also consider the associated vApp name. Together the vApp and cluster name must be less than 80 characters.

Configuring the Hadoop distribution

When configuring a Hadoop cluster, you must select the correct Hadoop distribution from the Hadoop distribution list box Change the default from Apache to Pivotal HD, as shown in Figure 21. The distribution name matches the value of the name parameter that was passed to the config-distro.rb script when the Hadoop distribution was configured. For a Pivotal PHD 1.1 cluster, you must configure a valid DNS and FQDN for the cluster's HDFS and MapReduce traffic. Without valid DNS and FQDN settings, the cluster creation process might fail or the cluster is created but does not function.

Figure 21. Create and name a new Big Data Cluster

Specifying deployment type

When prompted by the wizard, select the deployment type for the cluster, either Basic Hadoop Cluster or Data/Compute Separation Cluster. The type of cluster you create determines the available node group selections.

Creating new BDE clusters

Configuring a Hadoop cluster





Identifying the DataMaster node group

The DataMaster node is a virtual machine that runs the Hadoop NameNode service. This node manages HDFS data and assigns tasks to Hadoop TaskTracker services deployed in the worker node group. To identify the group:

1. Select a resource template from the list box or select Customize to create a custom resource template.

2. For the master node, specify shared storage so that the virtual machine is protected with vSphere HA.

Identifying the ComputeMaster node group

The ComputeMaster node is a virtual machine that runs the Hadoop JobTracker service. This node assigns tasks to Hadoop TaskTracker services deployed in the worker node group. To identify the group:



Identifying the HBaseMaster node group (HBase cluster only)

The HBaseMaster node is a virtual machine that runs the HBase master service. This node orchestrates a cluster of one or more RegionServer slave nodes. To identify the group:



Identifying the Worker node group

Worker nodes are virtual machines that run the Hadoop DataNode, TaskTracker, and HBase HRegionServer services. These nodes store HDFS data and execute tasks. To identify the group:


2. For the worker nodes, use local storage.

Note: You can add nodes to the worker node group by using Scale Out Cluster, but you cannot reduce the number of nodes.

Identifying the Client node group

A client node is a virtual machine that contains Hadoop client components. From this virtual machine you can access HDFS, submit MapReduce jobs, run Pig scripts, run Hive queries, and run HBase commands. When configuring the cluster for use with HaaS, you do not configure the Client node group unless any of these configuration items are required outside of the HaaS solution.



To identify the group:


2. For the client nodes, use local storage.

Note: You can add nodes to the client node group by using Scale Out Cluster, but you cannot reduce the number of nodes.

Selecting the Hadoop topology configuration

When you create a cluster with BDE, BDE disables automatic migration for the cluster’s virtual machines. This prevents vSphere from migrating anything but does not prevent the administrator from migrating nodes unintentionally to other vCenter hosts. It is essential that migrating is not performed from within vCenter as this could break the cluster placement policy.

As part of the final cluster configuration you should select the topology configuration that you want the cluster to use: RACK_AS_RACK, HOST_AS_RACK , HVE, or NONE.

More information is available in the chapter “About Cluster Topology” in chapter 7 of the VMware vSphere Big Data Extensions Administrator’s and User’s Guide.




Creating vCAC Catalog Services

The focus of customization for this EMC Hybrid Cloud solution is the VMware vCAC user self-service portal, where additional functionality is included to enable additional services for cloud users. The final stage of integrating Hadoop as a Service is to present to vCAC the HaaS workflows that have been imported and modifiedso that they can be selected as catalog items.

VMware vCAC 6.0 provides the extensibility to enable IaaS functionality through Advanced Service blueprints. The IaaS functionality is achieved by exposing custom vCO workflows that the vCAC 6.0 portal can present as a catalog of services for cloud users.

You can create custom workflow definitions using vCAC Designer. The vCAC Designer console provides a visual workflow editor for customizing vCAC lifecycle workflows. The extensibility toolkits include a library of activities that serve as building blocks for custom workflows.

Using the Advanced Service Designer, you can define new service offerings and publish them to the common catalog as catalog items.

To create the service blueprints you must access vCAC from a browser and log in to vCAC.

Each tenant has a unique URL to the vCAC console:

The default tenant URL is in the following format: https://hostname/shell-ui-app where hostname is the Fully Qualified Domain Name (FQDN) of a vCAC host.

The URL for additional tenants is in the following format: https://hostname/shell-ui-app/org/tenantURL where tenantURL is the URL name specified when the tenant is being created. This is the workspace in which the customer creates catalog services.

The following steps demonstrate, at a high level, how to integrate the HaaS workflows into the vCAC self-service catalog by showing the creation of:

Catalog services

Blueprints

Custom resources and resource actions

For more information, refer to the vCloud Automation Center Extensibility Guide.

To integrate the HaaS workflows into the vCACA self-service catalog, follow these steps:

1. From the main vCAC portal page, click Advanced Services to list all of the current service blueprints defined.

2. Click the green “plus” symbol, shown in Figure 22, to create a new service blueprint.

Accessing vCAC

Creating a new service blueprint

https://hostname/shell-ui-app

https://hostname/shell-ui-app/org/tenantURL



Figure 22. Advance Service Designer

Follow these steps to create a new service blueprint:

1. Select one of the imported Hadoop Cluster Creation workflows from the list.

2. Name the new service and create a form to support user input for the required parameters. If required, delete the default form and create a new form.

3. Drag and drop any appropriate input fields onto the form.

4. Publish the new service to create the appropriate service definition in the catalog management.

5. Assign a catalog management service to the new advanced service, and create the appropriate entitlement definition in the catalog management, as shown in Figure 23.

Figure 23. Edit Entitlement window



When these tasks are completed, the new service is then available in the service catalog for the cloud administrator. It is possible to replace the default VMware logo icons in the service catalog with more suitable HaaS icons. The replacement of icons is the final stage of customization and ensures that the service catalog items are tailored to a specific function or application. This can be performed from the Catalog Management menu by selecting the Catalog Items list box, selecting the configure an icon option, and then browsing and selecting a new icon.

After the configuration stages have been performed within vCAC, the service catalog is available to provision HaaS items, as shown in Figure 24.

Figure 24. vCAC Service Catalog showing Hadoop as a Service



Chapter 6: Use Cases: EMC Hybrid Cloud IaaS


Chapter 6 Use Cases: EMC Hybrid Cloud IaaS


Overview .................................................................................................................. 50

IaaS – storage services ............................................................................................ 50

Monitoring and capacity planning ............................................................................ 57

Metering and chargeback ........................................................................................ 61



Overview

This chapter covers EMC Hybrid Cloud IaaS and other use cases that can be incorporated to extend the functionality beyond virtual machine provisioning to consume resources.

From time to time additional physical resources will be required to support the extension of a Hadoop environment. The following sections show how EHC storage provisioning workflows can be used to create additional resources on demand by provisioning additional storage as required, and how the VMware vC Ops tool set can be used to analyze consumed resources, provide capacity planning, increase resources using scenarios that increase physical resources, and increase VM and node capacity.

IaaS – storage services

Storage is provisioned, allocated, and consumed by different cloud users in this solution.

For vCAC IaaS users, the storage services provided in the vCAC service catalog provision storage resources that will be allocated to and consumed by other cloud users.

Once the storage resources are available, fabric group administrators can assign the resources to business groups. Creators of virtual machine blueprints (business group managers) can then configure their blueprints to use those particular storage resources for the list of virtual machine disks.

When they provision virtual machines, cloud users consume the storage and, depending on their entitlements, may choose the storage service for their virtual machines.

This use case demonstrates how ViPR software-defined storage is provisioned for the hybrid cloud from the VMware vCAC self-service catalog.

1. To provision block or file storage from the vCAC self-service portal, select the Provision Cloud Storage item from the vCAC service catalog, as shown in Figure 25.

Overview

Use case 1: Storage provisioning



Figure 25. Storage Services - Provision cloud storage

The storage service blueprint can be created using vCAC anything-as-a-service (XaaS) functionality in the vCAC Advanced Service Designer. EMC ViPR provisioning workflows, which are presented by vCO to the vCAC service catalog, support storage services.

The storage provisioned by the IaaS user enables the fabric group administrator to make storage resources available to their business group. The storage provisioning request requires very little input from the vCAC IaaS user.

The main inputs required are:

Datastore Type: VMFS or NFS

Datastore Size

vCenter Cluster

Storage Tier

Most of these inputs, except LUN size, are selected from pre-populated list boxes whose items are determined by the cluster resources available through vCenter and the virtual pools available in ViPR.

After entering a description and reason for the storage-provisioning request, enter your password. The vCenter Server will manage multiple ESXi clusters; therefore, you must choose the relevant vCenter cluster to tell the provisioning operation where to assign the storage device. Select a vCenter cluster from the next screen, as shown in Figure 26.



Figure 26. Provision Cloud Storage – select vCenter cluster

2. Select the type of datastore you require from the list of available storage types, as shown in Figure 27. A datastore type of VMFS requires block storage, while NFS requires file storage. Other data services such as disaster recovery and continuous availability are displayed as appropriate only if detected in the underlying infrastructure.

Figure 27. Storage Provisioning – Select datastore type

3. Select from which storage offering the new storage device should be provisioned. The list of available storage offerings is based on the datastore type selected, such as VMFS or NFS, and what matching virtual pools are available from the ViPR virtual array.

In this example, a single NFS-based ViPR virtual pool is available to provision storage from, with the available capacity of the virtual pool also displayed to the user, as shown in Figure 28.

The storage pools listed have been configured in the EMC ViPR virtual array and their storage capabilities are associated with storage profiles created in vCenter.



Figure 28. Storage provisioning – Choose ViPR storage pool

4. Enter the size required for the new storage, in GB, as shown in Figure 29.

Figure 29. Storage provisioning – Enter storage size

5. The fabric group administrator must reserve the new Storage Pool for use by the business group, as shown in Figure 30.

Figure 30. Provision Storage – Storage Reservation for vCAC Business Group

When the automated process sends an email notification to the fabric group administrator that the storage is ready and available in vCAC, the fabric group administrator can then assign capacity reservations on the device for use by the business group.



In this example, a number of required input values, such as LUN or datastore name, have been masked from the user during the storage provisioning request process. Some of these values are locked-in and managed by the orchestration process and logic to ensure consistency.

In addition to the initial provisioning of storage to the ESXi cluster at the vSphere layer, this solution provides further automation and integration of the new storage up into the vCAC layer. The ViPR storage provider automatically tags the storage device with the appropriate storage profile based on its storage capabilities.

The remaining automated steps in this solution are:

vCAC rediscovery of resources under vCenter endpoint

vCAC storage reservation policy assigned to new datastore

vCAC fabric group administrator notification of availability of new datastore

This use case demonstrates how cloud users can consume the available storage service offerings. This use case is part of the broader virtual machine deployment use case, but here it relates directly to how the business group manager and users can manage the storage service offerings available to them.

VMware vCAC business group managers and users can select the appropriate storage for their virtual machine through the VMware vCAC user portal.

For business group managers, the storage type for the virtual machine disks can be set during the creation of a virtual machine blueprint. As shown in Figure 31, the relevant storage reservation policy can be applied to each of the virtual disks.

Figure 31. Set storage reservation policy for virtual machine disks

After the storage reservation policy is set, the blueprint will always deploy this virtual machine and its virtual disks to that storage type. If more user control is required at deployment time, the business group manager can elect to allow business group users to reconfigure the storage reservation policies at deployment time by selecting the checkbox Allow user to see and change storage reservation policies.

Use case 2: Select virtual machine storage



This solution uses VMware IT Business Management Suite (ITBM) to provide chargeback information on the storage service offerings for the hybrid cloud. Through its integration with VMware vCenter and vCAC, ITBM enables the cloud administrator to automatically track utilization of storage resources provided by EMC ViPR.

The EMC ViPR VASA provider in vCenter automatically captures the underlying storage capabilities of LUNs provisioned from virtual pools on the EMC ViPR virtual array. Storage profiles are created based on these storage capabilities, which are aligned with the storage service offerings. This integration enables ITBM to automatically discover and group datastores based on predefined service levels of storage.

In this solution we created a separate virtual machine storage profile for each of the storage service offerings, as shown in Figure 32.

Figure 32. Create new virtual machine storage profile for Tier 2 storage

The storage capabilities are shown automatically in vSphere, as shown in Figure 33, where Tier 2 EMC ViPR storage is supporting a datastore.

Figure 33. Automatic discovery of storage capabilities using EMC ViPR Storage Provider

Note: Storage capabilities are only visible in the traditional vSphere client and not in the web client. Also, the web client uses virtual machine storage policies in place of virtual machine storage profiles.

After the EMC ViPR Storage Provider has automatically configured the datastores with the appropriate storage profiles, the data stores can be grouped and managed in ITBM in line with their storage profile. Figure 34 shows that the cost profiles created

Use case 3: Metering storage services



in vCenter are discovered by ITBM. This allows the business management administrator to group tiered datastores provisioned with ViPR and set the monthly cost per GB as needed.

Figure 34. VMware ITBM chargeback based on storage profile of datastore

VMware vCAC can provide a storefront for storage services to be used by cloud users. These service catalog items deploy EMC ViPR software-defined storage services based on the usage of multiple service offerings of block and file storage across EMC VNX and VMAX storage arrays. Each service offers varying levels of availability, capacity, and performance to satisfy the operational requirements of different lines of business.

This solution combines EMC ViPR with EMC array-based FAST-enabled storage service offerings across the EMC storage arrays with VMware vSphere to simplify storage operations for hybrid cloud consumers.

Summary



Monitoring and capacity planning

The vCenter Operations Management Suite has functions that can help HaaS administrators to achieve the following goals:

Eliminate or significantly reduce the manual problem-solving effort in the environment.

Proactively manage core service and cloud infrastructure performance, and utilize infrastructure resources optimally.

Provision proactive warnings regarding performance issues before problems affect the end user. Real-time performance dashboards enable service providers to meet their SLAs by highlighting potential performance issues before end users notice these issues.

Infrastructure maintenance and operations teams need the end-to-end visibility and intelligence to make fast, informed operational decisions to proactively ensure service levels in cloud environments. They need to get promptly to the root cause of performance problems, optimize capacity in real time, and maintain compliance in a dynamic environment of constant change.

The vCenter Operations Management Suite offers many features and functions to deliver quality of service, operational efficiency, and continuous compliance for your dynamic cloud infrastructure and business critical applications.

This section describes in detail the capacity planning functions that can help you to predict the impact on underlying infrastructure of new HaaS deployments or of upgrading current HaaS instances with new services.

Forecasting capacity risks in vCenter Operations Manager involves creating what-if scenarios to examine the demand and supply of resources in the cloud infrastructure.

A what-if scenario is a supposition about how capacity and load might change if certain conditions, influenced by an increased or decreased number of ESX hosts, storage resources, or virtual machines in environment, occur, without making actual changes to your virtual infrastructure. If you implement the scenario, you know in advance what your capacity requirements are.

To create a what-if scenario, you can use models and profiles based on current resource consumption in the existing environment. Alternatively, you can manually define amounts of virtual machine RAM, storage, CPU, and utilization in a new consumption profile, as shown in Figure 35, to predict the potential impact of growth.

Monitoring

Capacity planning



Figure 35. Choosing virtual machine consumption models and profiles

To define a new virtual machine profile, you can make detailed specifications that give you the option to include and predict specific resource utilizations, reservations, and limits in order to get as accurate a projection as possible, as shown in Figure 36.

Figure 36. Specifying configuration and projected capacity usage of new virtual machines

Figure 37 shows that there are insufficient resources for a planned deployment scenario consisting of either 50 or 85 new virtual machines. In this case, we can easily provision new vSphere hosts using vCAC services as described in previous sections.



Figure 37. Capacity summary showing insufficient CPU and RAM resources

Before you provision new hardware resources, you can create hardware change scenarios to determine the effect of adding, removing, or updating the hardware capacity in a vSphere cluster. You can create a scenario that models changes to hosts and datastores, as shown in Figure 38 and Figure 39.

Figure 38. Specifying number of hosts and amount of CPU and memory



Figure 39. Specifying datastore size

The what-if scenario capacity planning function allows you compare how adding different amounts of virtual machines and hardware will impact your actual environment, as shown in Figure 40.

Figure 40. Compared scenarios

In a planning exercise, assume that you:

Have a request to deploy an additional 45 Hadoop node instances in the existing HaaS.

Plan to purchase blade servers compliant with a certain specification.

Want to deploy an additional 25 Hadoop clusters.

Capacity planning example



In Figure 41, each column shows how an individual change affects resources in your environment. The Combined Scenarios column shows you the cumulative effect of hardware purchasing and an overall expansion of 70 virtual machines.

Figure 41. Combined scenarios


VMware ITBM provides cloud administrators with comprehensive metering and cost information across physical and virtual resources in the EMC Hybrid Cloud environment. Besides working out the cost of physical components such as storage, compute, and networking resources, you can also include and configure other factors that affect the overall cost of your cloud environment, such as operating system licensing, maintenance, labor, and environmental facilities costs, as shown in Figure 42.



Figure 42. Categorized hybrid cloud environment cost overview

ITBM is integrated into the vCAC portal for the Hadoop administrator and presents a dashboard overview of the hybrid cloud infrastructure.

VMware ITBM Standard Edition uses its own reference database, which has been preloaded with industry-standard data and vendor-specific data to generate the base price for virtual CPU (vCPU), RAM, and storage values. These prices, which default to the cost of CPU, RAM, and storage, are automatically consumed by vCAC, where they can be changed as appropriate by the cloud administrator. This eliminates the need to manually configure cost profiles in vCAC and assign them to compute resources.

ITBM is also integrated with vCenter and can import existing resource hierarchies, folder structures, and vCenter tags to associate EMC Hybrid Cloud resource usage with business units, departments, and projects.

Infrastructure resources consumed by HaaS instances and hosted applications are provided by dedicated vSphere clusters with associated vSphere hosts and datastores. ITBM provides you with detailed information about:

Number of vSphere hosts in the vSphere cluster and the number of virtual machines on each host

CPU and RAM capacity and utilization of the vSphere cluster

Overall cost of the compute resources provided by the dedicated vSphere cluster

Cluster cost by virtual machine



The Clusters tab provides you with insight into the cost of the vSphere cluster resources consumed by Hadoop cluster instances. You can monitor costs while provisioning new hosts, as shown in Figure 43.

Figure 43. vSphere Cluster cost overview

The Datastores tab provides insight into the cost of the storage resources consumed by an HaaS instance. The name of a datastore provisioned by vCAC storage services inherits a cluster name prefix as part of its published name. Performing a sort by datastore name gives you a list of the names and costs of the datastores provisioned and assigned to hosts in the vSphere cluster, as shown in Figure 44.

Figure 44. Storage cost overview



Chapter 7: Conclusion


Chapter 7 Conclusion


Summary .................................................................................................................. 66

Chapter 7: Conclusion


Summary

Pivotal Hadoop is designed to create an easy-to-scale big data framework. To achieve this kind of flexibility, HaaS is designed around the modular system components of Pivotal Hadoop. Using vCenter Orchestrator workflows, the administrator can provide fixed cluster configuration catalog items or create dynamic workflows that can be called from a catalog. The size of the nodes used is determined by the individual making the request.

Elastic provisioning refers to the ability to provision flexible computing resources when and where they are required and to easily scale resources up and down to match demand. Resource elasticity can relate to processing power, memory, storage, bandwidth, and so on. This document indicates the importance of having an elastic and scalable IaaS platform on which to support the hosting of dynamically changing and fast-growing big data platforms.

VMware vCenter Operations Manager enables you to deliver quality of service, attain operational efficiency, and gather current capacity capabilities while forecasting the effect of future HaaS deployments or upgrades in your cloud infrastructure.

HaaS clusters can grow to a large number of node instances. The limit can be changed by changing the BDE configuration parameters. It is crucial therefore to have proactive performance monitoring and capacity planning solutions in place.

To support comprehensive, dynamic, and fast-growing development environments such as Hadoop as a service, you must ensure the stability of the underlying cloud compute infrastructure, which must provide availability, scalability, flexibility, and performance to the big data platform and its services. As a solution to these challenges, this document has addressed simple provisioning from a self-service catalog and considerations for building scalable Hadoop as-a -ervice environments, with an elastic and easy-to-deploy underlying IaaS infrastructure provided by the EMC Hybrid Cloud solution.

Appendix A: References


Appendix A References

This appendix presents the following topic:

References ............................................................................................................... 68

Appendix A: References


VMware references

The following VMware documents provide additional and relevant information:

Advanced Service Design vCloud Automation Center 6.0

Installing and Configuring VMware vCenter Orchestrator

VMware Compatibility Guide

VMware vSphere Big Data Extensions Administrator’s and User’s Guide: vSphere Big Data Extensions 1.0

Installing and Configuring VMware vSphere Big Data Extensions (Video)

http://pubs.vmware.com/vCAC-60/topic/com.vmware.ICbase/PDF/vcloud-automation-center-60-advanced-service-design.pdf

http://pubs.vmware.com/vsphere-55/topic/com.vmware.ICbase/PDF/vcenter-orchestrator-551-install-config-guide.pdf

http://www.vmware.com/resources/compatibility/search.php



http://www.youtube.com/watch?v=KMG1QlS6yag