Software Testing in a Virtualized World

Anuradha Biswas

Prakat Solutions

Living upto the CIO’s Priorities

• Performance

• Security

• Reliability

• Cost

Viewpoints on Cloud Computing

“I don’t understand what we would do differently in the light of Cloud Computing other than change the wordings of some of our ads”

Larry Ellision, Oracle’s CEO

“I have not heard two people say the same thing about it [cloud]. There are multiple definitions out there of “the cloud”

Andy Isherwood, HP’s Vice President of European Software Sales

“It’s stupidity. It’s worse than stupidity: it’s a marketing hype campaign.”

Richard Stallman, Free Software Foundation founder

IDC’s Viewpoint

• Spending on IT cloud services is expected to grow nearly threefold, to $42 billion by 2012

• Cloud computing will also account for 25% of IT spending growth in 2012 and nearly a third of the IT spending growth in 2013

News …

Microsoft unveiled on Tuesday (16 Nov 2010) the results of its behind-the-scenes work to

bring a popular suite of supercomputing software tools to its Azure cloud platform.

The company collaborated with the Seattle Children's Hospital on a test job that it says would have cost an estimated $3m (£1.9m) if it had used traditional on-premises hardware, but cost little more than $18,000 using a hybrid approach.

Source: http://www.zdnet.co.uk/news/cloud/2010/11/17/microsoft-shows-off-azure-power-with-software-test-40090884/

Defining Cloud Computing

What is the Fuss all about?

Source: Madgeek’s Gallery on Picasa

Cost Savings Potential …

Source: http://www.testingthefuture.net/

Non-Cloud Testing Steps

Requirement Study Test Planning

Test scenario identification Test Case / Script Writing Coverage Review

Test Execution Environment Preparation Test Case / Automated suite execution

Defect Reporting and Tracking Record defects Report Defect

Non Functional Testing Performance Security/ Vulnerability Accessibility Testing

Cloud Testing Steps

Requirement Study Test Planning

Test scenario identification Test Case / Script Writing Coverage Review

Test Execution Environment Preparation Test Case / Automated suite execution

Defect Reporting and Tracking Record defects Report Defect

Non Functional Testing Performance Security/ Vulnerability Accessibility Testing

How Cloud helps traditional Testing challenges

• Securing multiple environments (OS/Service Pack/Browser etc) cost and time prohibitive

Cloud provides a way to quickly ramp up different environments on the fly without investing in physical environments.

• On premise qualified resource to execute testing activities

Cloud breaks the physical barrier and allows executing tasks from any place with secured connectivity.

• Standardization of toolsets and expectations

Testing on cloud requires usage of specific tools and procedures. This in turn enforces certain discipline and data mining and reporting becomes symmetric.

Critical Factor for Effective Cloud Testing

• A challenge in testing cloud-based applications is the difficulty of replicating the number, diversity, and geographic distribution of the clients.

• Testing the behavior and functionality is not sufficient.

• It’s difficult to quickly and efficiently generate enough client. A cloud of virtual test clients along with software to orchestrate, record, and analyze the testing.

Performance in the Cloud

Testing all the layers — from your application to the cloud service provider

Source: Madgeek’s Gallery on Picasa

Measuring Performance

• Cloud InfrastructureTest performance of the cloud provider(s) to ensure they can sustain the needed load.

• DataTest the system with real life data handling scenarios

• Test the flow of data from various endpoint, with varying payloads, spread over several hours

• You need to ensure that the cloud can sustain heavy loads, handle concurrency, and consistently deliver solid performance for all transactions.

• Analyze the impact of encryption, transformation, data replication, and the various ways that data is being manipulated to address security, compliance, reliability, and scalability requirements.

• This stage is critical and forms a basis for troubleshooting on any performance bottlenecks.

Measuring Performance

• Business LogicInteractions of services and processes.

• User ExperienceDoes the application meet user needs?

• First test each service individually.

• Test the flow of data through the various combinations of service calls that the system is expected to perform.

• Rigorous testing and measure performance for load, sustainability, concurrency, etc.

• Test the system holistically and in the eyes of the end user

• It would be very expensive to find performance issues from the lower levels of the architecture at this point.

• This approach is iterative and agile and aims at removing performance risks earlier in the lifecycle thus reducing the risk of project delays.

Mind the “Security” Gaps

Source: cloudsecurityalliance.org

Security in the Cloud

• Test the security of the VMs augmented by third party security tools which provides layered security

• Ensure security controls are in place internal to the VMs other than the built in hypervisor isolation — such as intrusion detection, anti-virus, vulnerability scanning, etc.

• Test the security controls which are in place external to the VMs to protect administrative interfaces (web-based, APIs, etc.) exposed to the customers.

• VM-specific security mechanisms embedded in hypervisor APIs must be utilized to provide granular monitoring of traffic crossing VM backplanes, which will be opaque to traditional network security controls.

Security in the Cloud

• Administrative access and control of virtualized operating systems is crucial, and should include strong authentication integrated with enterprise identity management, as well as tamper-proof logging and integrity monitoring tools.

• Ensure that VMs are segregated in separate security zones by type of usage (e.g., desktop vs. server), production stage (e.g., development, production, and testing) and sensitivity of data on separate physical hardware components such as servers, storage, etc.

• Be aware of multi-tenancy situations with the VMs where regulatory concerns may warrant segregation.

In conclusion - Why is it Different?

Shared “multi-tenant” test environments Security in context of the applications being off premise and

potentially sharing all resources – processor, storage, network

Integration of on/ off premise systems (Public/ Private Clouds)

Performance / Stress testing is challenging since often new instances get spawned off automatically at certain thresholds

Defect isolation is not always easy Cloud Platform is dynamic in terms of the actual versions of

software running and in terms of loads and other applications sharing the same resource pool

Thank You

anu@prakat.com