Effective Testing of Healthcare Simulation Software

Presenter: Shant Mandossian

EFFECTIVE TESTING OF HEALTHCARE SIMULATION SOFTWARE

SIMULATION SOFTWARE

• Modeled after real world systems.

• Allows users to observe an operation without actually performing it.

• Used in initial system designs in order to optimize component selection.

SIMULATION SOFTWARE

Two Different Types

• Discrete Event Simulation

• Continuous Simulation

SIMULATION SOFTWARE CONTINUOUS SIMULATION

SIMULATION SOFTWARE CONTINUOUS SIMULATION

SIMULATION SOFTWARE DISCRETE EVENT SIMULATION



• Input

• Agent Resources

• Non-Agent Resources

• Artifacts

• Oracle

• Resource Manager

• Output

Significant Components

SIMULATION SOFTWARE TESTINGCOMMON METHODS

• Research in simulation software focuses on validation

• Insufficient investigation into testing the software itself.

SIMULATION SOFTWARE TESTINGWHY IS IT DIFFICULT?

• Simulation software exists to find the answers in the first place.

• No test oracle exists because exact simulation results are simply not known.

• Subtle errors in implementation can greatly affect the accuracy of the simulation.

SIMULATION SOFTWARE TESTINGWHY IS IT DIFFICULT?

• Simulations consist of many interacting components

• Each step or event can take a random amount of time to complete.

SIMULATION SOFTWARE TESTINGN-VERSION PROGRAMMING

• Early approach to testing software without a test oracle

• Independent programming teams develop an application from the same specifications.

• Identical sets of input are fed into each version of the program.

• Results are compared in an effort to detect defects.

SIMULATION SOFTWARE TESTINGMETAMORPHIC TESTING

• Use of Metamorphic Testing instead of N-Version Programming

• Say that produces

• After identifying the metamorphic properties of , we alter them in some way to make a transformation function .

• Now we produce

• We should be able to predict the results of based on the alterations we made to .

• If the results are not as expected, there must be a defect somewhere.

SIMULATION SOFTWARE TESTINGMETAMORPHIC TESTING

• The Bad

• Like most tests, this method only shows us if defects exist.

• Not that the software is defect-free.

• The Good

• Provides a way of revealing potential defects without the help of a test oracle

SIMULATION SOFTWAREHEALTHCARE

• Healthcare professionals rely on simulation software


• Software is used to simulate elements of the human body for purposes of training and decision making.


• Simulations of a specific organ like the heart.

• Simulations of an entire physiological system.


• Emergency Department

SIMULATION SOFTWARE TESTINGEMERGENCY DEPARTMENT SIMULATOR

Process• Patient arrives

• Patient is seen by a triage-nurse

• Patient registers with the clerk

• Clerk collects information puts in the patient record

• If a bed is available, patient goes to treatment area

• If all beds are occupied, patients waits in the waiting room until a bed is cleared

• Patient is taken to a free bed

• Patient is seen by a nurse

• Doctor takes assessment of the patient

• Assessment may result in tests

• Once tests are done, doctor admits or discharges the patient

• Nurse fills out paperwork.


• Agent Resources

• Clerk

• Triage Nurse

• Nurse

• Doctor

• Non-Agent Resources

• Beds


Finding Metamorphic Properties• All resources used in simulation

• Doctors, Nurses, beds, etc…

• Input given to the simulator

• Simulation instructors

• Program source code


Potential Bottleneck• Patients wait until a bed is available.

• Run multiple simulations with varying numbers of beds,

but keep all other resources constant.

• Plot average Length of Stay (LOS) for each run-through of the simulator to see if results are as expected.


Results show that LOS improves with more beds, but the amount of improvement decays.


Plotting the utilization of doctors over the number of beds shows that on average, utilization increases but also with a decay


• Both Length of Stay (LOS) of patients vs. number of beds and Doctor utilization vs number of beds, displayed expected results.

• New test case with unexpected results.

• Five patients arrive, 6 minutes apart.

• Resources Available

• One nurse, doctor, clerk, and triage nurse.

• Three beds.

• Results

• Average LOS for each patient = 217.4





• With One Nurse

• Patient 4 was ready to be discharged by the doctor before patient 5 was ready to be seen by the same doctor.

• With Two Nurses

• Patient 5 was ready to be seen by the doctor before patient 4 was ready to be discharged.

This is unexpected behavior.

SIMULATION SOFTWARE TESTINGTESTING THE TEST METHOD

• Testing the effectiveness of Metamorphic Testing

• Mutation testing is used.• Defects purposely injected into the source code.

• 25 out of 104 generated mutants were used in the test

• Many mutants were excluded for various reasons.

• Caused obvious errors and crashes

• Were caught by exception handling.

SIMULATION SOFTWARE TESTINGTESTING THE TEST METHOD

Metamorphic testing successfully caught all mutation defects.

SIMULATION SOFTWARE TESTINGOTHER METHODS

• Other testing methods (instead of metamorphic)• Assertion checking.


• Assertion checking

• Wasn’t used here in comparison

• Expected to perform less successfully.

• Example

• Off by 1 error for range [a, b-1] instead of [a, b]

• Will never produce an out-of-range result, causing the assertion test to miss the defect.

• Metamorphic testing can take on the statistical mean and variance over many run-throughs and find an inconsistency.

SIMULATION SOFTWARE TESTINGCONCLUSION AND SUMMARY

• Simulation Software

• Discrete Event Simulation

• Testing Simulation Software

• Relation to healthcare software

• Difficult to test

• Metamorphic Testing

• Mutation testing on Metamorphic testing

• Assertion Checking

Documents

Effective Testing of Healthcare Simulation Software