Cloud testing with synthetic workload generators

Cloud Testing with Synthetic Workload Generators

White Paper

Why cloud testing? Cloud Computing and Virtualization (NV and NFV) are adding incredible diversity in the

number of providers and services offered, but most of this ecosystem is running on a

handful of virtual platforms from vendors such as VMware, Citrix, Amazon, and Microsoft,

or from a purely open source solution such as KVM. Compared to traditional server

applications and networking products however, virtual platforms are comparatively new

and obviously less tested. This point can be derived from the lack of test tools targeted

to testing virtualization platforms. There is a burgeoning new class of cloud test tools

referred to as “Cloud Management Platforms or CMP tools”; but these tools focus more

on capacity planning, active monitoring, and basic network performance within a public,

private or hybrid cloud than they do actual benchmarking of the virtual platform at the

center of cloud or NFV infrastructure. CMP tools also derive the majority of their analytics

from native APIs provided by the hypervisor, duplicating information already offered by

hypervisor vendor tools. There are also cloud based traffic generators, SaaS offerings

that specifically aim to determine application traffic performance over the cloud, but

such tools are limited in their scope of test methodologies and even the analytics they

provide are only about network performance and not virtual infrastructure.

Introduction Cloud computing is redefining the meaning

of scale and complexity. It also demands the

ability to fuse the previously distinct roles

of application hosting and network traffic

processing over a common system. Until

recently, application testing and network device

testing are treated as two very distinct domains

starting from different hardware and software

to different testing tools and practices. As the

complexity of interaction between software

serving different purposes is evolving with

cloud computing and virtualization, so is the

need for validation solutions and supporting

methodologies. Conventional test tools were

not designed to test “hypervisors” – the

cornerstone of virtualization. Spirent intends to

introduce the concept of a complete resource

bearing “synthetic workload generator” as

the best testing solution for benchmarking

hypervisors – by validating both functionality

and performance of various applications and

services (IaaS, PaaS, SaaS) running over them.


2 | spirent.com

White Paper

Organic vs. Synthetic WorkloadsThe lack of adequate test tools targeting virtualization platforms can largely be attributed to the complexity of developing such

tools. The current approach is to use “organic” workloads where actual applications create the workloads and traffic running on

a virtual machine, e.g. web server, then generate client side requests via automation to scale up the workloads and traffic. This

approach is effective, but can be very costly to scale for software that is not open source. The “organic” approach also requires

that software and its dependencies such as drivers, packages etc., be installed which introduces considerable overhead testing

from one type of application to another. This significantly adds to the time and complexity to execute testing. Lastly, organic

workloads are not flexible enough. Workloads are confined to the natural state machine of the application under test. Complex

defects only surface when the test loads run for a lengthy period of time, or if the test loads can break out of the confinement of a

particular application’s state machine. Organic workloads cannot have a particular resource like CPU, memory, or storage, skewed

to change the load characteristic of the test. A better approach pioneered by Spirent is the concept of “synthetic workloads”.

Synthetic workloads will drive the utilization of vCPUs, memory, storage I/O, and network I/O using synthetically created processes

and threads of execution at the VM level for hypervisor scheduling. Using synthetic workloads is a far less expensive approach

that scales higher, automates easier, offers much greater workload flexibility, and ultimately exposes more bottlenecks in resource

scheduling by the hypervisor kernel – ultimately improving the performance of virtual platforms that SaaS and virtual service chains

run on.

Introduction to Synthetic Workload GeneratorsJust as the popularity and scale of the Internet mandated the need for new types of test tools that could generate and receive

hundreds of millions of IP flows, and emulate hundreds of thousands of IP switching and routing devices, tools Spirent introduced

to the world and has been a leader in ever since. So too, will Cloud Computing necessitate a new generation of tools be developed

to meet the sheer scale and diversity that complex deployments will demand of the virtual platforms they are built on. Synthetic

Workload Generators (SWGs) not only can provide the cost and scale benefits, but also control, flexibility, load isolation, and real-

time feedback that unaltered organic application workloads fall short of. SWGs are designed for live production environments and

deployed in actual live cloud or virtual infrastructure. Key features include:

� Ability to provide any type of modeled workload on a single resource, any combination of resources, or even all compute

resources simultaneously providing unparalleled control and flexibility of workload scheduling

� Cause adversity or havoc like sustained high CPU utilization or excessive memory consumption emulating memory leak

conditions

� Provide deep insights into a hypervisor’s scheduling algorithm performance and ability to respond to such problems

� Help preempt service deployment issues, isolate faulty hardware in the cloud, and expose complex defects with the

hypervisor platform that IaaS, PaaS, SaaS, or NFV is running over

� Ability to gauge or conduct dry runs of a particular service before deploying the actual cloud or NFV application and quickly

throttle back from workload thresholds that impact live production VMs.

These features add up to a complete divergence from conventional network testing where load generators are connected to lab

test beds or application performance is assessed on non-production servers.

spirent.com | 3

Use cases for Synthetic Workload GeneratorsBy utilizing VMs or containers to generate and precisely control the magnitude of, per resource synthetic workloads – vCPU,

memory, storage I/O, network I/O – and derive real-time feedback on the performance of each independent workload operation,

any type of test can be run cost effectively, in any environment whether live or not, and can provide a complete performance

assessment over any period of time for a given virtual infrastructure. For example, individual resource control allows for workload

isolation to just a specific resource operation, memory read for example. This load isolation can then be scaled across many

generators, also referred to as agents, to determine the overall scale capability of a given resource operation like memory reads

in a cloud or NFV deployment. Organic software loads cannot mimic this behavior. Cloud and Service Providers could use this

information to better gauge their VM memory capacity planning. SWGs can be used to validate the QoE or SLAs of virtualized

services and their performance under scale across all resources. Cloud customers, or even private cloud owners, could use such

a tool to determine the best provider or hypervisor platform to build their cloud on based on a given IaaS, PaaS, SaaS, or NFV

implementation.

Figure:

Bare Metal Server

HypervisorDeploymentService

AnalyticsCollection

SWG1 SWG2 SWG3 SWG4 SWGn

Synthetic Workload Generators deployed for auto-scale use case

Synthetic Workload Generators can model custom enterprise applications that are impractical to obtain or implement for testing or

benchmarking purposes. SWGs can also be used to accurately assess spare capacity, which is especially critical to smaller private

cloud implementations that need to maximize hardware resources, or Cloud/Service Providers trying to assess the impact of

adding another service chain to compute nodes already hosting other tenants. SWGs are perfectly suited to auto scale validations,

whether assessing boot time, policy enforcement or determining the maximum achievable scale of an elastic computing solution in

a large public cloud, providing pinpoint analytics of every operation across all resources per agent exposing resource contention

issues and bottlenecks at scales that are difficult to comprehend. These examples are just the tip of the iceberg; many more use

cases exist in areas like high availability and migration performance benchmarking.

© 2016 Spirent. All Rights Reserved.

All of the company names and/or brand names and/or product names referred to in this document, in particular, the name “Spirent”

and its logo device, are either registered trademarks or trademarks of Spirent plc and its subsidiaries, pending registration in

accordance with relevant national laws. All other registered trademarks or trademarks are the property of their respective owners.

The information contained in this document is subject to change without notice and does not represent a commitment on the part

of Spirent. The information in this document is believed to be accurate and reliable; however, Spirent assumes no responsibility or

liability for any errors or inaccuracies that may appear in the document. Rev A | 01/16


spirent.com

AMERICAS 1-800-SPIRENT +1-800-774-7368 | [email protected]

EUROPE AND THE MIDDLE EAST +44 (0) 1293 767979 | [email protected]

ASIA AND THE PACIFIC +86-10-8518-2539 | [email protected]

White Paper

SummaryCloud computing and all of its instantiations require a new type of tool to validate

the platforms they run over. A tool that is commercially available, economical, meets

a hyperscale requirement, and flexible enough to simulate any type of application

workload, one that allows precise control and provides key analytics plus offers real-

time feedback on compute resource performance and that is driven by methodologies

applicable to cloud and NFV applications. Synthetic Workload Generators meet all of

these requirements. Synthetic application workloads are not a new concept. It has also

been proven that application workloads can be accurately and realistically synthesized1.

And Spirent is pioneering efforts towards bringing such tools and supporting

methodologies to market. For more information about Spirent Cloud Solutions, please

visit www.spirent.com/Cloud.

About SpirentSpirent provides software solutions to high-tech

equipment manufacturers and service providers

that simplify and accelerate device and system

testing. Developers and testers create and share

automated tests that control and analyze results

from multiple devices, traffic generators, and

applications while automatically documenting

each test with pass-fail criteria. With Spirent

solutions, companies can move along the path

toward automation while accelerating QA cycles,

reducing time to market, and increasing the

quality of released products. Industries such

as communications, aerospace and defense,

consumer electronics, automotive, industrial,

and medical devices have benefited from

Spirent products.

1 Georgia Institute of Technology: Synthetic Workload Generation for Cloud Computing

Applications, Bahga and Krishna Madisetti