ERP Testing Failures - Questionnaire

University of Colombo School of Computing

ICT 3001

ENTERPRISE RESOURCE PLANNING

THIRD YEAR GROUP ASSIGNMENT

ERP TESTING FAILURES - QUESTIONNAIRE

GROUP 9

Lecturer: - Mr. Samantha Mathara Arachchi

Date of Submission

10th

of June 2013

Name Registration Index

Number Number

SANDARUWAN J.W.C. 2010/ICT/018 10020187

WICKRAMASINGHE W.M.M. 2010/ICT/038 10020381

MENDIS D.G.T. 2010/ICT/057 10020578

UPENDRA K.A.N. 2010/ICT/070 10020705

KARUNARATHNE H.B.K.A. 2010/ICT/082 10020829

WIJESUNDARA W.M.D.M. 2010/ICT/092 10020926

KARUNARATHNA S.A. 2010/ICT/102 10021027

ERP TESTING FAILURES | UCSC | GROUP 9 Page 1 of 30

We are a research group from University of Colombo School of Computing (UCSC). We

would like to invite you to participate in a research study which aims at collecting data for a

project as a part of Enterprise Resource Planning (ICT 3001).

Before you begin, take a few minutes to read why we are inviting you to participate and what

will be done with the information you provide. You will be asked to fill out a questionnaire

that aims to investigate the ERP testing failures which could be beneficial you all as you are

a part of the ERP community. Please feel free to stop at any time if you have questions about

the study. This project is conducted only on sites which are engaging in ERP development.

Your individual privacy and confidentiality of the information you provide will be maintained

in all published and written data analysis resulting from the study.

Your participation should take approximately 10-15 minutes for each phase mentioned in the

questionnaire. We ask you to fill it out with as much accuracy as possible. Please understand

your participation is entirely on a voluntary basis and you have the right to withdraw your

consent or discontinue participation at any time without penalty. At no instance you will be

asked to reveal any personal information. Participation in this study involves no major risks

whatsoever, be it physical or emotional. This study is directly beneficial to the participants,

engaging in ERP development. You will receive no incentive or payment for your

participation.

If at any time and for any reason, you would prefer not to answer any questions, please feel

free to skip those questions. If at any time you would like to stop participating, please tell us.

We can take a break, stop and continue at a later date, or stop altogether. You will not be

penalized for deciding to stop participation at any time. If you have any questions, you are

free to ask them now. If you have questions later, you may contact:

If you have any questions about your rights as a participant in this study, or you want to talk

to someone outside the research, please contact the:

University of Colombo School of Computing No. 35, Reid Avenue

Colombo 07.

Tel: +94 112- 581245-7

I have read and understood the above, and I agree to participate in

the study.

The consent form is to be detached and kept with the participant.

Name Phone Number Email address

Saman Jayantha 0771-362587 [email protected]

Dilruk Karunarathne 0715-897423 [email protected]

WELCOME TO THE ERP TESTING FAILURES SURVEY, 2013


Table of Contents

WELCOME TO THE ERP TESTING FAILURES SURVEY, 2013 ....................................... 1

1) Unit Testing ........................................................................................................................... 4

2) Integration Testing ................................................................................................................. 6

3) System Testing ...................................................................................................................... 9

3.1) Usability Testing ............................................................................................................. 9

3.2) Load testing and Stress testing ........................................................................................ 9

3.3) Parallel Testing ............................................................................................................. 10

3.4) Configuration testing and installation testing ............................................................... 10

3.5) Security Testing ............................................................................................................ 11

4) Documentation Testing........................................................................................................ 11

4.1) Master Plan ................................................................................................................... 11

4.1.1) Incomplete Documentation .................................................................................... 11

4.1.2) Unsuitable scope .................................................................................................... 11

4.1.3) Mess of testing ....................................................................................................... 11

4.1.4) Incorrect testing methods ....................................................................................... 12

4.2) Level test plan ............................................................................................................... 13

4.3) Test report ..................................................................................................................... 14

4.3.1) Ambiguity............................................................................................................... 14

5) Other Testing Failures ......................................................................................................... 15

Justification .............................................................................................................................. 17

1) Unit Test .............................................................................................................................. 17

2) Integration Test Failures ...................................................................................................... 18

2.1) Interface Testing Failures ............................................................................................. 18

2.2) Acceptance Testing Failures ......................................................................................... 18

2.3) Interoperability Testing Failures Question .................................................................. 18

2.4) Infrastructure Testing Issues ......................................................................................... 19

3) System Testing Failures....................................................................................................... 20

3.1) Failures with Usability Testing ..................................................................................... 20

3.1.1) Poor Task Design ................................................................................................... 20

3.1.2) Not well structured test cases ................................................................................. 20

3.2) Load testing and Stress testing ...................................................................................... 21

3.2.1) Bugs in the Application under test ......................................................................... 21


3.2.2) Software Interaction failures .................................................................................. 21

3.2.3) Problems with the Load Generation ....................................................................... 22

3.3) Parallel Testing Failures ............................................................................................... 22

3.3.1. Higher complexity of the system ............................................................................ 22

3.3.2. Less interaction with sub system component testing.............................................. 22

3.4) Configuration testing and installation testing failures .................................................. 22

3.4.1. Tests are getting too much time for their completion. ............................................ 22

3.4.2. Compatibility issues in the system. ........................................................................ 23

3.4.3. Failures of identifying configuration changes in the system. ................................. 23

3.5) Security testing failures ................................................................................................ 23

3.5.1 Core security concepts are not followed. ................................................................. 23

4) Documentation testing failures ............................................................................................ 24

4.1. Incomplete Documentation........................................................................................ 24

4.2. Unsuitable scope ........................................................................................................ 24

4.3. Mess of testing ........................................................................................................... 24

4.4. Incorrect testing methods........................................................................................... 24

4.5. Under and over estimations of resources ................................................................... 25

4.6) Budget allocation....................................................................................................... 25

4.7) Validation and verification problem ......................................................................... 25

4.8) Estimation failures..................................................................................................... 25

4.9) Incomplete report ...................................................................................................... 26

5) Other Testing Failures ......................................................................................................... 27

5.1) E-commerce and security compliance failures ............................................................. 27

5.2) Communication failures ................................................................................................ 27

5.3) Failures of third party tools ........................................................................................... 27

Mind Map................................................................................................................................. 28

Reference List .......................................................................................................................... 29


1) Unit Testing

1. Does unit test catch each and every bug in the code?

2. This question is regarding the test cases. Tick your opinion/s.

3. Do you agree or disagree with this statement. (1 being highly agree and 5 being highly

disagree)

“The test code also may have bugs same as the code it testing. Therefore if any error of bug

found it is difficult to identify whether it is an error with the code or test code”.

1 2 3 4 5

4. Do you believe before start unit testing is it necessary to remember as the test is behaved as

a part of the complete system to exercise the code in a realistic scenario?

5. What are the best practices you follow in unit testing?

6. What are the biggest challenges you will be facing in ERP unit testing?

Yes No

The test cases for unit testing are normally written by developers with the same

programming language, before implementation of each unit.

Test cases should be developed for every exception path and for every possible input.

Then programmers have to write 3 to 5 lines of test code for every written line of the code.

This required much time and effort.

This doesn’t require much time and effort.

Yes No

Following a rigorous way of working through development process.

Keep records of the entire test that have been performed.

No need of writing test cases for every exception path and for every possible input.

Make records of all the changes that have been made in the source code during the unit

testing.

If others, mention them briefly: - -----------------------------------------------------------------------------

---------------------------------------------------------------------------------------------------------------------

Writing test cases for every exception path and for every possible input.

Software is being developed and tested in one platform and should be deployed in to another

platform.

Make records of all the changes that have been made in the source code during the unit

testing.

Writing tests for legacy codes are not an easy task and it is necessary to refer documentation

of those legacy systems and may have to change the code to fit with the tests.


---------------------------------------------------------------------------------------------------------------------


7. Tick the choices that you feel more comfortable with system test failures?

8. Select most suitable choices regarding unit test automation?

Developing tests for each and every unit may not be possible with the time and budget

constraints of the project which in such situations developers may use same test cases to test

many units.

If there is an error with a test case there will be many testing failures with many units. And if

two or more units use same resources simultaneously test failure may increase.

Tests those are hard to understand cause failures.


---------------------------------------------------------------------------------------------------------------------

Unit test automation can solve most of unit testing failures.

Test automation cannot separate each component and validate its correctness well.

Using framework is a popular method of unit test automation. It simplifies the unit test process

and has been developed for wide variety of languages.

Unit test along cannot solve most of the failures and are only developed for few languages.


---------------------------------------------------------------------------------------------------------------------


2) Integration Testing

9. Have you been provided the scope of the integration testing?

10. How clear does your defined scope?

Very clear

Clear

Neither clear nor not clear

Not very clear

Not clear at all

11. How many number of “items” tested and not tested?

I. Number of items that are tested ………

II. Number of items that are not tested ………

12. How many number of “features” tested and not tested?

I. Number of features that are tested ………

II. Number of features that are not tested ………

13. Is the system tested on possible access platforms?

14. Did you identify data mismatches in the system?

a) If “Yes”, how many data mismatches did you correct? ………

15. Was the system tested by the client representatives and system users?

16. Do all the business functions work correctly?

If not, list the functions that fail to perform.

…………………………………………………………………………………………………

…………………………………………………………………………………………………

17. Did you achieve the enterprise wide compatibility?


…………………………………………………………………………………………………

…………………………………………………………………………………………………

Yes No

Yes No

Yes No

Yes No

Yes No

Yes No


18. Is each functional module tested with its corresponding interoperating external module?

If not, list the modules that were left untested.

…………………………………………………………………………………………………

…………………………………………………………………………………………………

19. What are the QA related reference documents referred? (tick “” appropriate documents)

IEEE Std. 1465-1998 // ISO/IEC 12119

(IEEE Standard for Quality requirements and testing)

IEEE Std. 829-2008

(IEEE Standard for Software Test Documentation)

Other (state name)

…………………………………………………………………………………………

…………………………………………………………………………………………

20. What are other applicable documents referred within the system? (tick “” appropriate

documents)

Architectural design document

Detailed design document

System requirement specification

Problem report

Modification request

Other (state name)

…………………………………………………………………………………………

…………………………………………………………………………………………

21. What are the planned deliverables before starting integration testing (tick “” if delivered)

Architectural design document

Detailed design document

Unit test pan and report

Integration test plan

Other (state name)

…………………………………………………………………………………………

…………………………………………………………………………………………

22. Did you check the pre-installation compatibility?

23. Did you carry out a contingency planning?

24. Does testing team have adequate qualified hardware testers?

Yes No

Yes No

Yes No

Yes No


25. What are the deliverables have to be submitted after finishing the integration test? (tick

“” if delivered)

Integration test report

Integration test output data

Problem report

Other (state name)

…………………………………………………………………………………………

…………………………………………………………………………………………

26. Environmental needs definition was,

Excellent

Good

Average

Poor

Not defined at all

27. Test case pass fail criteria definitions were,

Excellent

Good

Average

Poor

Not defined at all

28. Test procedure definitions were,

Excellent

Good

Average

Poor

Not defined at all


3) System Testing

3.1) Usability Testing

29. Did you use any usability testing tool/tools?

30. Did you consider the system that you are going to test, when selecting testing tools?

31. Did you find any changes required to improve user performance from those testing?

I. If YES, Did you notify those changes to the user?

32. Did you use an essential task designing component when designing tasks?

33. Following question is based on test cases.

I. Do you have following characteristics with your test cases?

a) It is accurate and tests what is intended to test.

b) No unnecessary steps included to it.

c) It is reusable.

d) It is traceable to requirements.

e) It is simple and clear, any tester is able to understand it by reading once.

3.2) Load testing and Stress testing

34. What is the loading test category that your system belongs to?

Quantification of risk

Determination of minimum configuration

Yes No

If Yes, mention them briefly: - -----------------------------------------------------------------------

------------------------------------------------------------------------------------------------------------

-

Yes No

Yes No

Yes No

Yes No

Yes No

Yes No

Yes No

Yes No

Yes No

If others, mention them briefly: - --------------------------------------------------------------------

------------------------------------------------------------------------------------------------------------


35.Did you test the system from both load and unload execution under load testing?

36.Can you certify that the load generator is working correctly?

37.Does the system interact with other applications, when execution?

I. If YES, did you check impact of other applications, with the load test?

3.3) Parallel Testing

38. Do you use complex methodologies in parallel testing?

39. Are you monitoring the tests in sub systems?

40. Is there a mechanism to measure that you test all the parts of the system?

3.4) Configuration testing and installation testing

41. Are you using business process based scenarios for testing?

42. Are you using different platforms, supported drivers and different computer

configurations?

43. Have you clearly identified where configuration changes might take place?

44. Have you defined test cases for each configuration tests?

45. Have you used an automated mechanism for configuration tests?

Yes No

Yes No

Yes No

Yes No

Yes No

If Yes, mention them briefly: - -----------------------------------------------------------------------

------------------------------------------------------------------------------------------------------------

Yes No

Yes No

Yes No

Yes No

Yes No

Yes No

Yes No


46. What are the system components that are tested for compatibility issues?

3.5) Security Testing

47. What are the types of security testing are you performing in the system?

4) Documentation Testing

4.1) Master Plan

4.1.1) Incomplete Documentation

48. What is the template do you use for test plan documentation?

IEEE 829 Standard

Organizational custom template

If any other standard (mention it here)……………………………………………………

4.1.2) Unsuitable scope

49. Have you go through the requirement specification report before determine the scope?

50. Have you identified what are the testable functions and non-testable functions?

4.1.3) Mess of testing

51. Have you prioritized the modules and functions to be tested?

52. Have you prepared a work breakdown schedule for testing?

Hardware platforms Software platforms

If any, mention in briefly: ----------------------------------------------------------------------------

------------------------------------------------------------------------------------------------------------

Confidentiality Integrity Authentication

Availability Authorization Non-repudiation

If any, mention in briefly: ----------------------------------------------------------------------------

------------------------------------------------------------------------------------------------------------

Yes No

Yes No

Yes No

Yes No


4.1.4) Incorrect testing methods

53. Have you determined what methods you are going to use for each test function?

4.1.5) Under and over estimations of resources

54. Have you identified all resources that you need to conduct the test?

55. What are the resources you have identified mentioned in last question?

Hardware resources (number of machines, performance, storages, etc)

Required Platforms (OS / versions )

Human resource (number of team members, skills, expertise, etc)

Budget

Mention any other resources you determine in documentation (if any)

…………………………………………………………………………………………………

56. How much do you allocate for testing from budget? (As percentage of whole budget)

…………………………………………………………………………………………………

57. Have you measured the time required for whole testing process?

.

58. Have you set buffer time to accommodate unexpected delays?

4.1.6) Ignore Risks

59. Have you determine critical functions to be tested?

60. Have you identified risks with the product?

4.1.7) Validation and verification problem

61. What is the test pass percentage of testing respect to all test cases?

……………………………………………………………………………………………

62. Have you validated the test plan with respect to the requirement specification?

63. Have you reviewed the test plan?

Yes No

Yes No

Yes No

Yes No

Yes No

Yes No

Yes No

Yes No


4.1.8) Ignore future requirement

64. What are the deliverables of test?

Test cases

Test scripts

Test report

Test log

Work load analysis

If any other deliverables mention them here

……………………………………………………………………………………………

4.2) Level test plan

These questions are regarding the testing documentation of ERP components testing.

4.2.1) Inadequate testing

65. What are the methods you are going to use for testing?

Black box testing

White box testing

Gray box testing

Manual testing

Automated testing

Mention any other methods you have determine in test plan documentation

………………………………………………………………………………………………

66. Have you determined the methods going to use for testing each phase?

67. Have you set test pass percentage of each testing units?

4.2.2) Incorrect estimations

68) Have you identified unique resources required in each test unit?

69) Have you estimated testing effort of each test unit?

Yes No

Yes No

Yes No

Yes No


70) How do you estimate the effort?

Based on previous experience

Based on standard effort estimate models

Based on expertise knowledge

Based on typical values like importance of functions, criticality of function or

number of test cases

4.3) Test report

This section is regarding the test reporting.

4.3.1) Ambiguity

71) Have you clearly defined what, when, who, where, how and to whom reporting?

If you said some are not, what are the things you defined?

……………………………………………………………………………………………

4.3.2) Inadequate Report

72) What are the charts and diagrams you are going to generate in tests?

Pareto chart

Cause and effect Diagrams (fishbone diagram, Ishikawa diagram, characteristics

diagram)

Check sheets

Histogram

Run chart

Control chart

Scatter plot diagrams

If any other diagram mention it here.

……………………………………………………………………………………………

73) What are the analysis tools used to report test results?

Cluster analysis

Regression analysis

Multivariate analysis

If any other analysis mention it here

…………………………………………………………………………………………..

Yes No Some are not


74) What are the reports you generate during the testing?

Function test matrix

Defect status report

Functional testing status report

Testing action report

If any other report mention it here

…………………………………………………………………………………………..

75) What are the reports you generate as final test reports?

Unit test report

Integration test report

System test report

If any other report mention it here

…………………………………………………………………………………………..

4.3.3) Incomplete report

76) Are you going to review test report before finalizing it?

5) Other Testing Failures

77) What parts of testing are you going to automate?

Management part of testing

Test specification part

Static testing part

Test execution and logging part

Performance and monitoring part

If any other parts mention them here

………………………………………………………………………………………....

78) Is testing team experienced with test automating tools?

Yes No

Yes No


79) Did you go through all rules and regulations to determine e-commerce compliance?

80) Are you going to use third party tools for testing?

a) If “Yes”, do you check the quality, reliability and the testability of that tools?

If your answer is “yes”, answer for these questions.

1. Can the selected tool handle the required functions?

81) How long the vendor stable in industry? _______________________________________

82) Are the selected tools upgraded continuously?

83) Are selected tools compatible with your environment?

84) Is the testing team familiar with the selected automating tools?

85) Have you obtained any CPI compliant certificate?

If “yes”, mention it here…………………………………………

86) Do developers and testers have peer communication method?

If yes, what is the present communicate system? ...................................................................

87) Are you satisfied about experience and knowledge of testing team members?

Extremely satisfied

Moderately satisfied

Neither satisfied nor dissatisfied

Dissatisfied

Yes No

Yes No

Yes No

Yes No

Yes No

Yes No

Yes No

Yes No

Yes No


Justification

1) Unit Test

1.1) Unit test failures

Question

Do u agree with this statement is correct or not. (1 being highly agree and 5 being highly

disagree)

“The test code also may have bugs same as the code it testing. Therefore if any error of bug

found it is difficult to identify whether it is an error with the code or test code”.

1 2 3 4 5

This question is asked to catch whether the tester have good knowledge on test code and

program code separately. From the response we can identify the tester knowledge on program

code and test code.

Question

Before start unit testing is it necessary to remember as the test is behaved as a part of

the complete system to exercise the code in a realistic scenario?

Tester should have a good knowledge in real world business process before writing the test

code. The aim of this question is to catch that answer. If not the system won’t be able to

match with the defined framework.

Question

What are the biggest challenges you will be facing in ERP unit testing?

From this multiple choice question we can measure the readiness of the system implementers.

Based on the answers to this question we can get an idea of their knowledge regarding ERP

challenges where it would be useful to build a better system.

Question

What are the best practices you follow in unit testing?

Best practices are mandatory in ERP implementation. Failures can be occur due to mal

practices followed by people. The aim of this question is to measure the level of following

best practices.

Yes No


Question

Select most suitable answers regarding unit test automation?

With this multiple choice question we are trying to find the unit test automation need for the

organization. Based on the answers we can come to a conclusion whether the development

clearly caught the test automation or not.

2) Integration Test Failures

2.1) Interface Testing Failures

Question

Is the system tested on possible access platforms?

The system is being accessed by users using various platforms. Thus it should be tested in

order to eliminate the platform issues. This question is used to identify those issues.

2.2) Acceptance Testing Failures

Question

Was the system tested by the client representatives and system users?

Acceptance of the system by its users should be verified. These question find out how the

accuracy of the system and performance are being verified.

2.3) Interoperability Testing Failures

Question

Is each functional module tested with its corresponding interoperating external module?

If not, list the modules that were left untested.

…………………………………………………………………………………………………

…………………………………………………………………………………………………

This question finds out all the scenarios where mismatches between modules and business

functions have been occurred. By this, mismatches between modules that interoperate in

different departments can be identified.

Yes No

Yes No

Yes No


Question

Did you achieve the enterprise wide compatibility?


…………………………………………………………………………………………………

…………………………………………………………………………………………………

This question finds out all the scenarios where mismatches between modules and business

functions have which can lead to the system being isolated from the whole network

(international) due to module mismatches and resulting in transaction failures.

2.4) Infrastructure Testing Issues

Question

Did you check the pre-installation compatibility?

This question finds out whether the compatibility of the hardware components and the

software components being used in the system are verified. This is known as device

compatibility testing.

Question

Did you carry out a contingency planning?

This question derives the necessary information regarding the performance of the system in

disaster situations and the stated guidelines to follow in a such situation and the drawbacks of

the guidelines.

Yes No

Yes No

Yes No


3) System Testing Failures

3.1) Failures with Usability Testing

Question

Did you consider the system that you are going to test, when selecting testing tools?

The testing tool should be based on the type of the testing method. So the appropriate testing

type should be chosen.

Question

Did you find any changes required to improve user performance from those tests?

In lots of situations, overall testing process will identify changes required to improve. It is

like a successful character in usability testing. Therefore, from this question analyst can

identify the results of modifications in system.

Question

Did you find any changes required to improve user performance from those testing?

I. If YES, Did you notify those changes to the user?

From this question, analyst can make sure that all component of the system meet the user

requirements. Sometime tester may not notice recently changed user requirements. Hence,

some components that do not meet the user requirements could be included in the final

product. This question is asked to track that kind of failures.

3.1.1) Poor Task Design

Question

Did you use essential task designing component when designing tasks?

In order to create effective design task, essential task designing component should be

included. As instances task scope, task support, task resource and task execution can be

mentioned. Ensuring that the tasks have been designed well or not is the core aim of this

question.

3.1.2) Not well structured test cases

Question

Do you have following characteristics with your test cases?

Yes No

Yes No

Yes No

Yes No

Yes No


a) It is accurate and tests what is intended to test.

b) No unnecessary steps included to it.

c) It is reusable.

d) It is traceable to requirements.

e) It is simple and clear, any tester is able to understand it by reading once.

Every tester writes test cases however many times the test cases are rejected by reviewers

because of bad quality, in order to write good test cases one should know what are the

characteristics of a good test case. Using this question, analyst can make sure that the test

cases are well structured and cover all what are intended to test.

3.2) Load testing and Stress testing

3.2.1) Bugs in the Application under test

Question

Did you test the system from both load and unload execution under load testing?

Most systems use load testing for identify sensitivity errors. And these sensitivity errors

appear only under the load or extended execution. Hence tester also tests under the load or

extended execution always. Sometime there can be errors without loading execution. Those

errors can also be identified from the above question.

3.2.2) Software Interaction failures

Question

Does the system interact with other applications, when execution?

This question is related to load testing. Load testing is showing all problems that are related

in handling load. It does not consider how many programs interact with the system while

showing result as a whole. Thus tester cannot certify that the problems are related to which

software. Hence from this question analysts can know whether system is interacted with other

application at the execution.

If YES, did you check impact of other application, with the load test?

If there is another application running same time, then first attention has to be paid for that

application. If that application works correctly then there is no impact for our load testing

result. Hence from this question, analyst can make sure that the load balancing is not showing

false results.

Yes No

Yes No

Yes No

Yes No

Yes No

Yes No

Yes No


3.2.3) Problems with the Load Generation

Question

Can you certify that the load generator that is used is working correctly?

Load generators are used for generating hundreds of concurrent request to access the

application. And then load the execution and check results. If load generator is not properly

working as intended, then end results can be false. Therefore the purpose of this question is to

identify whether load generating system works properly or not.

3.3) Parallel Testing Failures

3.3.1. Higher complexity of the system

Question

Do you use complex methodologies in parallel testing?

Based on the answer to this question, we can measure the complexity of the parallel testing.

There is a high possibility of parallel testing failure due to the complexity. Methodology

complexities can be simplified in order to minimize the potential failures.

3.3.2. Less interaction with sub system component testing

Question

Are you monitoring the tests in sub systems?

This question is asked to get an idea about sub system testing. If we get the answer as “No”,

we can assume there can be possible testing failures in sub systems. In parallel testing, sub

system test monitoring is a crucial factor for identifying the potential failures.

3.4) Configuration testing and installation testing failures

3.4.1. Tests are getting too much time for their completion.

Question

Have you used an automated mechanism for configuration tests?

This question is asked to catch the testing mechanism of the system. It means the manual

testing and the automated testing. Configuration and installation tests are taking much time in

manual testing as we have to look various kinds of test components. In order to get a

maximum throughput we can automate those tests.

Yes No

Yes No

Yes No


3.4.2. Compatibility issues in the system.

Question

What are the system components that are tested for compatibility issues?

Based on the answer for this question we can identify whether the system is properly tested

for compatibility issues.

3.4.3. Failures of identifying configuration changes in the system.

Question

Have you clearly identified where configuration changes might take place?

Due to the complexity of the system sometimes people couldn’t identify the places where

configuration changes should take place. From this question we can come to an idea, whether

configuration changes are correctly identified or not.

3.4.4. Test cases are not defined for every configuration test due to complexity

Question

Have you defined test cases for each configuration tests?

Whatever we have to test in front of us we have to do it. There are instances where people are

not defining test cases for small configuration changes that will cause serious problems in

system functionality.

3.5) Security testing failures

3.5.1 Core security concepts are not followed.

Question

What are the types of security testing are you performing in the system?

Based on the user answer we can get an idea of the system’s security. If the system is not

following the desired standards there will be drawbacks in security.

Hardware platforms Software platforms

If any, mention in briefly: --------------------------------------------------------------------------

------------------------------------------------------------------------------------------------------------

Yes No

Yes No

Confidentiality Integrity Authentication

Availability Authorization Non-repudiation

If any, mention in briefly: --------------------------------------------------------------------------

------------------------------------------------------------------------------------------------------------


4) Documentation testing failures

4.1. Incomplete Documentation

Question

What is the template you use for test plan documentation?

IEEE 829 Standard

Organizational custom template

Any other standard (mention it here)……………………………………………………

There should be a standard format to any business documentation. It is very important to

ensure they don’t miss any important thing to document with a standard. This question asks

to check whether they use proper standard template for documentation.

4.2. Unsuitable scope

Question

Have you go through the requirement specification report before determine the scope?

This question checks whether test plan cover all factors which are should test in the testing

phase. If testing team misses any important items to test, it will cause to fail the ERP system.

4.3. Mess of testing

Question Have you prioritized the modules and functions to be tested?

It is very important to correctly identify what are the most important and what are the primary

testing functions to test. If not it will raise two major problems.

I. Some interdependent function may lose to test.

II. Some very important and critical function may escape to test within the estimated

time.

Hence this question asks to ensure that they don’t miss any critical function to test.

4.4. Incorrect testing methods

Question

Have you determined what methods you are going to use for each test function?

It is very important to determine what method will be used to what functions. Some

functions should be automated; some require human inspection while some are both.

Yes No

Yes No

Yes No


Therefore this question ask check whether testing team clearly define the testing methods

which they going to use.

4.5. Under and over estimations of resources

Question

Have you identified all resources you want to do testing?

Testing phase requires lot of resources and it will deviate from ERP system to system. Some

ERP system tests require simulation software or very advance software having artificial

intelligent. Some system tests require install new platforms for testing environment. Hence it

is very important to verify all resource requirements and availability of current resources.

4.6) Budget allocation

Question

How much do you allocate for testing phase from budget? (As percentage of whole budget)

……………………………………………………………………………………………………………

It is important to allocate at least 40% of budget from whole to testing phase. Therefore this

question is check whether they allocate enough budgets for testing phase.

4.7) Validation and verification problem

Question

What is the test pass percentage of testing respect to all test cases?

…………………………………………………………………………………………………..

Some ERP systems require more reliability while some are not much. As well as with time

constraint user may like to accept system with less testing as first version of system in order

to continuously improve. Therefore it is very important to set proper percentage to pass the

test. Most of time the ERP system cannot achieve 100% bug free system. Therefore this

question check whether set percentage is accurate or not.

4.8) Estimation failures

Question

How do you estimate the effort?

Based on previous experience

Based on standard effort estimate models

Based on expertise knowledge

Based on typical values like importance of functions, criticality of function or

number of test cases

Yes No


These two questions are asked to ensure whether they do estimation using correct technique

or not. Different estimations use different methods to estimate effort.

4.9) Incomplete report

Question

Are you going to review test report before finalizing it?

This question is asked to check whether they do reviews on report before finalizing it. It is

very important to properly do the reporting, because it will use to make decision on next steps

of testing and maintenance processes.

Yes No


5) Other Testing Failures

5.1) E-commerce and security compliance failures

Questions

Did you go through all rules and regulations to determine e-commerce compliance?

Have you obtained any CPI compliant certificate?

If “yes”, mention it here…………………………………………

E-commerce involving ERP systems must be very secure. If not it will raise fraud through the

system. As well as confidence of user privacy is very important factor in ERP systems,

especially when e-commerce involve in the processes. If system lose to ensure confidence of

data whole ERP system is useless. Thus these questions are asked to check the security level

of ERP system.

5.2) Communication failures

Question

Do developers and testers have peer communication method?

If yes, what is the present communicate system? ...................................................................

Communication problems arise during the quality assurance (QA) process. This is the result

of using different tools and technologies, or working in disparate locations. But even when

development and testing take place in the same building, communication problems still exist.

To overcome these issues we can use communication systems with standardized frameworks.

5.3) Failures of third party tools

Question

Are you going to use third party tools for testing?

Most of all the time we have to use third party tools for testing. When we are choosing those

tools there should be a well-defined set of rules. That means we have to check them for their

quality, testability and the reliability. The aim of this question is to track the third party tools

usage of the testing process.

Yes No

Yes No

Yes No

Yes No


Mind Map


Reference List

1. Configuration Testing | James D. McCaffrey. 2013. Configuration Testing | James D.

McCaffrey. [ONLINE], Available at:

http://jamesmccaffrey.wordpress.com/2009/01/31/configuration-testing/. [Accessed 05th

June

2013].

2. ERP Testing | Testing ERP | Test ERP | ERP Test | ERP Validation. 2013. ERP Testing |

Testing ERP | Test ERP | ERP Test | ERP Validation. [ONLINE], Available at:

http://www.qualitestgroup.com/ERP-Testing. [Accessed 04th

June 2013].

3. James D. McCaffrey, 2009, Configuration Testing | James D. McCaffrey, [E-book],

Available at: http://jamesmccaffrey.wordpress.com/2009/01/31/configuration-testing/,

[Accessed 07th

June 2013].

4. Test driven guy: Common Unit Testing Problems. 2013. Test driven guy: Common Unit

Testing Problems. [ONLINE], Available

at:http://testdrivenguy.blogspot.com/2009/05/common-unit-testing-problems.html, [Accessed

03rd

June 2013].

5. Top 10 reasons why teams fail with Acceptance Testing . 2013. Top 10 reasons why teams

fail with Acceptance Testing . [ONLINE], Available at: http://gojko.net/2009/09/24/top-10-

reasons-why-teams-fail-with-acceptance-testing/, [Accessed 04th

June 2013].

6. Writing Great Unit Tests: Best and Worst Practices - Steve Sanderson’s blog - As seen on

YouTube™. 2013. Writing Great Unit Tests: Best and Worst Practices - Steve Sanderson’s

blog - As seen on YouTube™. [ONLINE], Available

at: http://blog.stevensanderson.com/2009/08/24/writing-great-unit-tests-best-and-worst-

practises/, [Accessed 03rd

June 2013].

7. Why do performance tests fail? @ 1202Performance. 2013. Why do performance tests fail?

@ 1202Performance. [ONLINE], Available at:

http://www.1202performance.com/2012/02/why-do-performance-tests-fail/, [Accessed 04th

June 2013].

8. What is Regression Testing?. 2013. What is Regression Testing?. [ONLINE], Available at:

http://www.guru99.com/regression-testing.html, [Accessed 04th

June 2013].

9. Usability testing mistakes | Information & Design. 2013. Usability testing mistakes |

Information & Design. [ONLINE], Available at:

http://infodesign.com.au/usabilityresources/usabilitytestingmistakes/, [Accessed 05th

June

2013].

10. Zhen Ming Jiang, Ahmed E. Hassan, Gilbert Hamann, Parminder Flora, 2006, Automatic

Identification of Load Testing Problems, [E-book], Available at:

http://sail.cs.queensu.ca/publications/pubs/ICSM2008Jiang.pdf, [Accessed 05th

June 2013].

Technology

ERP Testing Failures - Questionnaire