DEEPAK BHIMARAJU; EDWARD ALLEN

TEST CHALLENGES IN THE CLOUD

Speakers

Edward Allen18 years QA experienceQA Director - Oracle Service CloudCo-author of a chapter in the book “Experiences of Test Automation”

Speakers

Deepak BhimarajuWorked as a QA Engineer at Oracle IndiaRecently moved to a development role

The Cloud – Our Partly Sunny Version

Multi-tenant

Multi-version

Enterprise

The Cloud – Our Partly Sunny Version• Email• Chat• Collaboration• Case Management

Software as a Service

• Custom Data• Application Development• Middleware• Managed Frameworks

Platform as a Service

• Coming Soon?Infrastructure as a Service

Consume

Build

Host

On Premise Deployment: BenefitsKnown hardwareTried and Tested ConfigurationsPhysical proximity

Image Credits:SilverStartalk, http://en.wikipedia.org/wiki/Computer_network_diagram#/media/File:Sample-network-diagram.png

On Premise Deployment: BenefitsLegacy software providing stability over timeTest approach or process is a known quantity

Primary test focus is functionalCompatibility, Capacity, Scalability, Security testing

Image Credits:SilverStartalk, http://en.wikipedia.org/wiki/Computer_network_diagram#/media/File:Sample-network-diagram.png

On Premise Deployment: Challenges

ScalabilityMaintenanceSetup CostsLatency*

Image Credits:Erik Pitti, https://www.flickr.com/photos/epitti/2370848133/

Promises of The Cloud

ScalableMaintainableLow Latency*Secure

Image Credits:Christi Gain, https://www.flickr.com/photos/christigain/5636888777/

Challenges of The Cloud

Reliance on the Cloud Operations teamPinning responsibility can take time (environment or code or both)Finding bugs early as fixing takes more time.

Test Challenges: Multi-Tenant Saas

SecurityTransaction Volumes

Shared servicesSpillover from other tenants

Scale and CapacityTenant growthData volume per tenant

Test Challenges: Multi-Version SaaS

Rapid Scheduled Deliveries ZERO down-timeNo customer patience for break-fixes or post-upgrade work (promise of the Cloud)

Customers choose when to uptake featuresSingle-version Shared Services

Test Challenges: PaaS

Managed Frameworks

App Development

Custom DataReporting

Workflow and Business Logic

Test Challenges: IaaS

Functional Testing is minimalMust be reliable, secure, compatible

Image Credits:Torkild Retvedt, https://www.flickr.com/photos/torkildr/3462607995/

Test Challenges: Architecture

Reference: Li, et al. (2011). Characteristics of multiple component defects and architectural hotspots: A large system case study. Empirical Software Engineering, 16 (5), 667 702. ‐

• Multiple Component Defects (MCDs): Architectural defects which typically span more than one component encompassing multiple, interacting components.

• Cost of a single MCD can be amplified by the number of tenants.

• Cost of MCDs is as much as 20 times that of a Functional Defect

Test Challenges: Architecture

92%

8%

52%

48%Regular Defect

MCD

Reference: Li, et al. (2011). Characteristics of multiple component defects and architectural hotspots: A large system case study. Empirical Software Engineering, 16 (5), 667 702. ‐

% of Total App Defects

% of Total Repair Efforts

Solutions: General

Promote and Value Testing as a ProfessionGreater decision making authority

Technical LevelProduct Level

Popularize in UniversitiesFind new test perspectives and approachesFinding the weakest links in the cloud

Solutions: General

Utilize the “inter-regnum”Work for other teams on the cloudCreate a common Knowledge BaseOffer incentives to move developers to QAEqual Pay

Solutions: Functional TestingRapid Release

Automate quickly and in parallelManual scripted tests minimizedExploratory testing

Solutions: Cloud Testing Gap

Functional Analysis (Integration and System Test)

Structural Analysis (Reliability, Security, Scalability, Capacity,

Compatibility)

Static Analysis Unit Test

Tester

Developer

Attribution to Dr. Bill CurtisSVP and Chief Scientist, CAST Research Labs

Solutions: Structural Testing

Testing cannot be “after the fact”Much of the work isn’t submitting bug reportsTesters have greater product expertise than developers

Solutions: Structural Testing

Testers lead on capacity, scalability, performance, securityFinding best solution among many good options

Or least worse among many bad optionsContinual, rapid, iterative development

Solutions: Structural Testing

Need to be experts in Customer usageAvailabilitySystem administrationCloud app configurationSecurityScalabilityCapacityPerformanceNetworkingAPIs

Solutions: Structural TestingLearn craft and discipline of teaching and leading

Testers must leadTeach usage patterns, data volume modeling, exploitsUnique position to drive discussion

Testers train developersPrevent and RemoveFewer defects created in the first place

Secure codingApp configurationPerformance

Solutions: Structural Testing

Expertise demanded of Cloud Testing is ratcheted upWe must be up to the challenge

Cannot be satisfied to test boundary conditions and alpha-numeric fieldsCannot continue the exploratory/traditional manual testing debate

DEEPAK BHIMARAJU; EDWARD ALLEN

TEST CHALLENGES IN THE CLOUD

SPECIAL THANKS:

BRAD PASCOE, Oracle Corp.

PRASAD NAGARAJU, Oracle India

NILOY GUPTA, CMU, Pittsburgh