Non-Functional Testing

www.kanoah.com

ABOUT US

Kanoah is an innovative company providing ground-breaking solutions to software testing professionals on the

Atlassian JIRA platform

About Kanoah Tests

Kanoah Tests is a full featured test management, integrated seamless into JIRA with the same look-n-feel. No need to learn or switch between different applications

Coordinate all test management activities including test planning, authoring, execution, tracking and reporting from a central location

Kanoah Tests enables you to track testing progress and quality to foster collaboration and visibility across traditional and agile teams

Get real-time insights into your testing progress with out of the box reports

Easily integrate your automated tests and submit test results with Kanoah Tests’ powerful REST API or use the API to automate many areas of the application

After looking for several years at plugins for test management we finally found Kanoah Tests. The other solutions were either too complex, didn't integrate well with Jira, or were focused on a single project. Kanoah Tests proved to be an elegant solution that allowed linking between any project. Kanoah has been very responsive to feedback, requests, suggestions as well bugs. The customer service is awesome. I'd highly recommend Kanoah Tests to teams of any size looking to simplify test management and consolidate tools.

Don Pierce

Robert Murhamer

Liked Kanoah from the moment I discovered it. Integrates nicely with JIRA and especially with Agile. Test cases can be authored right from the story level, but has all other functionality a Test Case Management solution would need to have, ranging from creating test plans, executing test cases, importing test cases, API for automation, a.s.o. Additionally the team at Kanoah is amazing and responding to any question very quickly. It wasn't hard to sell to my management to purchase Kanoah. Will highly recommend Kanoah to anybody.

Zour Brosh

I just start working with Kanoah and I am impressed how it's simple to manage tests without unlimited non-used features like in most of the test management tools and still to get the needed functionality and results. The integration with Jira is a great working solution that enable to share testing and development in simple way on one system. I recommend to use Kanoah for testing management. It will help to do Kanoah as much as possible customizable like Jira to match each group methodology

For more reviews, visit: https://marketplace.atlassian.com/plugins/com.kanoah.test-manager/server/reviews

Reviews

Key Features

Native seamless integration with JIRA

No need to learn or switch between different applications

Perfect for agile & traditional testing approaches

Manage, organize and track all your testing efforts in a central place

Reuse test cases across your projects

Powerful REST API

Establish clear traceability between requirements, test cases, and defects

Execute test cases and track results that matter

Get real-time insights into your testing progress with out of the box reports

Live statistics accessible to your entire team

Benefits for the testers

No need to lear or switch between different applications

Reuse test cases across projects for regression

Link test cases to requirements and defects

API support for automated efforts

Benefits for the teams

Informed decisions based on real-time insights

End-to-end traceability and impact analysis

Centralized Test Management

Save time and increase productivity

Why choose Kanoah Tests

Coordinate all test management activities right inside JIRA

1 Testing right inside JIRA

Take advantage of the built-in reports to track the results and measure progress

2Make informed decisions

Kanoah Tests users receive priority support, even during trials

3Stelar support

Non-Functional Testing

What is Non-Functional Testing?Non-functional Testing is the testing of a software application or system for its non-functional requirements: the way a system operates, rather than specific behaviours of that system. [1]

Non-functional Testing is concerned with the non-functional requirements and is designed specifically to evaluate the readiness of a system according to the various criteria which are not covered by functional testing. [2]

Non-functional requirements tend to be those that reflect the quality of the product, particularly in the context of the suitability perspective of its users. It can be started after the completion of Functional Testing. The non functional tests can be effective by using testing tools. [3]

What is Non-Functional Testing?

Non-functional Testing implies the testing of software attributes which are not related to any specific function or user action like performance, scalability, security or behavior of application under certain constraints.

Non-functional testing has a great influence on customer and user satisfaction with the product. Non functional testing should be expressed in a testable way, not like “the system should be fast” or “the system should be easy to operate” which is not testable. [3]

Types of Non-Functional TestingThe following types and definitions have been recompiled from these authors [1] [2] [3].

Reliability testing: Reliability Testing is about exercising an application so that failures are discovered and removed before the system is deployed. The purpose is to determine product reliability, and to determine whether the software meets the customer’s reliability requirements.

Efficiency testing: Efficiency testing test the amount of code and testing resources required by a program to perform a particular function. Software Test Efficiency is number of test cases executed divided by unit of time (generally per hour).

Usability testing: The testers tests the ease with which the user interfaces can be used. It tests that whether the application or the product built is user-friendly or not. It also includes five components:

1. Learnability: How easy is it for users to accomplish basic tasks the first time they encounter the design?

2. Efficiency: How fast can experienced users accomplish tasks?

3. Memorability: When users return to the design after a period of not using it, does the user remember enough to use it effectively the next time, or does the user have to start over again learning everything?

4. Errors: How many errors do users make, how severe are these errors and how easily can they recover from the errors?

5. Satisfaction: How much does the user like using the system?

Types of Non-Functional Testing

Types of Non-Functional TestingMaintainability testing: It basically defines that how easy it is to maintain the system. This means that how easy it is to analyze, change and test the application or product.

Portability testing: It refers to the process of testing the ease with which a computer software component or application can be moved from one environment to another, e.g. moving of any application from Windows 2000 to Windows XP. This is usually measured in terms of the maximum amount of effort permitted. Results are measured in terms of the time required to move the software and complete the and documentation updates.

Baseline testing: It refers to the validation of documents and specifications on which test cases would be designed. The requirement specification validation is baseline testing.

Types of Non-Functional TestingCompliance testing: It is related with the IT standards followed by the company and it is the testing done to find the deviations from the company prescribed standards.

Documentation testing: As per the IEEE Documentation describing plans for, or results of, the testing of a system or component, Types include test case specification, test incident report, test log, test plan, test procedure, test report. Hence the testing of all the above mentioned documents is known as documentation testing.

Endurance testing: Endurance testing involves testing a system with a significant load extended over a significant period of time, to discover how the system behaves under sustained use. For example, in software testing, a system may behave exactly as expected when tested for 1 hour but when the same system is tested for 3 hours, problems such as memory leaks cause the system to fail or behave randomly.

Types of Non-Functional TestingLoad testing: Load testing is performed to determine a system’s behavior under both normal and at peak conditions. It helps to identify the maximum operating capacity of an application as well as any bottlenecks and determine which element is causing degradation.

Activities of Load Testing might include:

Validate that the system performs as expected when concurrent users access the application and gets the expected response time.

This test is repeated with multiple users to get the response time and throughput.

At the time of testing, the data base should be realistic.

The test should be conducted on a dedicated server which stimulates the actual environment.

Types of Non-Functional TestingPerformance testing: Performance testing is performed to determine how fast some aspect of a system performs under a particular workload. It can serve different purposes like it can demonstrate that the system meets performance criteria. It can compare two systems to find which performs better. Or it can measure what part of the system or workload causes the system to perform badly.

Activities in Performance Testing might include:

Validate that the system meets the expected response time.

Evaluate that the significant elements of the application meets the desired response time.

It can also be conducted as a pert of integration testing.

It can also be conducted as a part of systems testing.

Types of Non-Functional TestingCompatibility testing: Compatibility testing is basically the testing of the application or the product built with the computing environment. It tests whether the application or the software product built is compatible with the hardware, operating system, database or other system software or not.

Activities in Compatibility Testing might include:

Test with each hardware with minimum and maximum configuration.

Test with different browsers.

Test cases are the same which were executed during functional testing.

In case the number of hardware and software are too many, we can use OATS techniques to arrive at the test cases to have maximum coverage.

Types of Non-Functional TestingSecurity testing: Security testing is basically to check that whether the application or the product is secured or not. Can anyone came tomorrow and hack the system or login the application without any authorization? It is a process to determine that an information system protects data and maintains functionality as intended.

Scalability testing: It is the testing of a software application for measuring its capability to scale up in terms of any of its non-functional capability like load supported, the number of transactions, the data volume etc.

Types of Non-Functional Testing

Volume testing: Volume testing refers to testing a software application or the product with a certain amount of data. E.g., if we want to volume test our application with a specific database size, we need to expand our database to that size and then test the application’s performance on it.

Some key points of Volume Testing:

Subject the software to large amounts of data and check the limit where the software fails.

Maximum database size is created and multiple client query the database or create larger report.

If the application is processing database to create a report, a volume test would be to use a large result set and check that the report is printed correctly.

Types of Non-Functional Testing

Stress Testing: It is a form of testing that is used to determine the stability of a given system. It puts greater emphasis on robustness, availability, and error handling under a heavy load, rather than on what would be considered correct behavior under normal circumstances. The goals of such tests may be to ensure the software does not crash in conditions of insufficient computational resources (such as memory or disk space).

Activities of Stress Testing might include:

Test on low memory or low disc space on clients or servers that reveals the defects which cannot be found under normal conditions.

Multiple user performance the same transactions on the same data.

Multiple clients connected to servers with different workloads.

Reduce the Think Time to “Zero” to stress the servers to their maximum stress.

Types of Non-Functional TestingRecovery testing: Recovery testing is done in order to check how fast and better the application can recover after it has gone through any type of crash or hardware failure (etc). Recovery testing is the forced failure of the software in a variety of ways to verify that recovery is properly performed.

Recovery Testing activities might include:

Power interruption, to the client while doing CURD activities.

Invalid database pointers and keys

Database process are aborted or prematurely terminated.

Database pointers, fields and keys are corrupted manually and directly within database.

Physically disconnect the communication wires, power turn off, turn down the routers and network servers.

Types of Non-Functional TestingInternationalization testing: Internationalization is a process of designing a software application so that it can be adapted to various languages and regions without any changes.

Localization Testing: Localization is a process of adapting internationalized software for a specific region or language by adding local specific components and translating text.

Estimating Non-Functional RequirementsOnce the team accepts a non-functional requirement into the project they need to remain in compliance with that non-functional requirement for the remainder of the project.

Doing performance testing (or staying in compliance with any non-functional requirement) creates some amount of overhead on the team. This overhead, or tax, must be paid regularly. In some cases, the team and product owner will decide the tax must be paid every sprint. In other cases the team and product owner may agree to pay the tax every few sprints.

Estimate the cost of initial compliance just like any other user story or product backlog item. The team and product owner will need to incorporate an estimate of when they'll do this. Adding performance testing after five sprints is different than adding it after 20 sprints. [5]

Distinction between Functional and Non Functional Testing

Functional testing ensures that your product meets customer and business requirements, and doesn’t have any major bugs.

Functional testing is designed to determine that the application’s features and operations perform the way they should.

Non-functional testing wants to see if the product stands up to customer expectations.

Non-functional testing wants to know that the product “behaves” correctly.

Failure to release a working product that meets the needs of consumer demands can damage your company’s reputation and reduce overall product sales. [6]

Functional Testing

The following list has been extracted from here [7].

Tests how well the system performs.

Based on client requirements.

Means testing the application against business requirements.

It is a part of System Testing.

Validates the behavior of the application.

Non-Functional Testing

The following list has been extracted from here [7].

Tests how well the system responds.

Based on client expectations.

Means testing the application against the client and performance requirements.

Also a part of System Testing.

Validates the performance of the application.

References

[1] https://en.wikipedia.org/wiki/Non-functional_testing

[2] http://reqtest.com/testing-blog/functional-vs-non-functional-testing/

[3] http://www.softwaretestinghelp.com/what-is-non-functional-testing/

[4] https://www.mountaingoatsoftware.com/blog/non-functional-requirements-as-user-stories

[5] https://www.mountaingoatsoftware.com/blog/estimating-non-functional-requirements

[6] http://www.ibeta.com/functional-vs-non-functional-testing-whats-difference/

[7] http://testingbasicinterviewquestions.blogspot.com.br/2015/02/functional-testing-vs-non-functional-testing.html

Next Steps

Getting Started Guide

GETTING STARTED RESOURCES FREE TRIAL

DocumentationSupport

TutorialsWhat we're reading this weekBlog

Start you 30-day free trial now