SaaS Security – Performance Validation

Challenge

Our client has developed a system offering their customers a SaaS solution for scanning and filtering of web traffic entering and leaving the customer premises in order to protect against web-borne viruses, spyware and other threats. To provide a seamless user experience it is imperative that internet connectivity remains reliable and performs optimally even at peak times of usage. The SaaS provider needs reliable benchmark figures in order to specify the architecture for each of their clients and therefore deliver a high quality service. The challenge for the project team was how to verify the scalability of the system without unintentionally impacting public websites which could have inadvertently been regarded as a denial of service attack. Initial testing by the project team had resulted in inconsistent results that did not reflect the reality for customers using the service and so EOH were engaged to review the approach and deliver an effective way to benchmark the offering.

Our Approach

Our approach was to identify what the customer really needed to achieve their objectives by challenging

their thinking and bringing EOH’s experience of solving these type of problems. The customer had

unsuccessfully been able to test effectively due to the existing test tools not providing an accurate

reflection of user behavior and uncontrolled elements of the test as public websites were being

accessed. By using our experience of testing web applications we challenged the assumptions on user

behavior and developed a more realistic model which could be trusted to provide real world

expectations.

EOH owned the architecture of the test environment and overall test design with the client project team

implementing the test environment configuration. We prototyped a self-contained world wide web

emulation system which allows total control over the testing environment and eliminated the issues

inherent in targeting public websites. This system consisted of servers to host content and simulate the

internet as well as using HP LoadRunner to execute a range of requests via the security platform. The

main challenge we found was being able to test against multiple domain name service (DNS) servers to

represent a real world environment.

Operating Model

We identified that the best model was a team made up of key networking and system experts from the

customer and one EOH Senior Performance Testing Consultant to drive the web emulation system

implementation. Initially the EOH team was based on the client premises to build relationships and

collaborate on the design before transitioning offshore to our test centre in South Africa for the test

execution phase.

The model enabled the client to make rapid progress through collaboration before benefiting from the

cost benefit of delivery using our South African Test Centre once the project was into a steady state. We

used a mentoring approach to enable the client team to gradually take ownership of performance test

execution under out supervision.

Value Add

EOH delivered a number of key benefits to our customer:

Supported client sales as we enabled a clear understanding of the solution’s capabilities

Provided a long-term solution for evaluating the solution’s performance and scalability

Eliminated risk from the project by delivering a solution without relying on external content

Provided a large cost benefit by utilising capacity from South Africa

EOH Key Differentiators

Proven performance testing methodology

Full ‘Managed Service’ approach

Collaborative working model

Ability to deliver using a combination of onshore and offshore resources

Summary Statement

EOH challenged the thinking of our customer and applied their automation methodology using a

collaborative operating model. This approach delivered rapid benefits by enabling the client to quickly

take their new offering to market safe in the knowledge of its capabilities and will provide a good

experience for their customers as a result.

Contact Details

Contact info@eoh-uk.com for more information or visit eoh-europe.com