Upload
others
View
10
Download
0
Embed Size (px)
Citation preview
January 28, 2020 Sam Siewert
SE420Software Quality Assurance
Lecture 3 – Unit Testing & Integration TestingPart-1
RemindersAssignment #1 Submission– Assigned on Thursdays– Dues Sunday 10 days later– Late due date 13 days later
Grading in progress
Assignment #2
Goal to practice code walk-throughs and start class on holding walk-throughs once SQA groups have formed
Sam Siewert 2
Reading This WeekSQA Text Chapt. 1&2 Complete, Start Chapter 3 as Planned, Chapter 4 with Assignment #2
http://softwaretestingfundamentals.com/
Skim SWEBOK, Chapter 4– Note 4.3 White-box, or “Glass-box” – tests based on information about how the
software has been designed and coded [synonyms in SWBOK and ISTQB]– Black-box – test cases relying only on input/output behavior– Gray-box - A test that has both Black-box and White-box qualities– Visibility and/or intrusiveness of a unit test is important consideration
Unit Testing – White-box, Black-box, Coverage CriteriaIntegration Testing – Find Defects in the Interfaces and InteractionBetween Modules and Components of a SubsystemSystem Testing – Does Implementation Meet Requirements?Acceptance Testing – Does Implementation Meet Customer Needs? Regression Testing – Did Anything Get Broken by Code or Configuration Changes?
Sam Siewert 3
Notes on Unit TestingWhite-box = Glass-box, but re-use distinction for COTS is noted in research
We will look at test harness intrusion and visibility
E.g. COTS is not necessarily full-source re-use (white vs. glass box reuse)
Gray-box implies a test with both White-box and Black-box
Sam Siewert 4
https://www.aviation-ia.com/product-categories/600-series,https://www.windriver.com/products/vxworks/certification-profiles/,https://www.windriver.com/universities/
When Does Testing Start?(Design, Development, Test of Test, Use)
Test Plan, Test Case, and Test Support and Policies for Lifecycle Concurrent with Analysis, Design, Development
Acceptance Test Design First (We’ll Look at that Next) – Top Down
For Purpose of Class, We’ll Go Bottom Up for Testing
Sam Siewert 5
Reg
ress
ion
Test
ing
Testing starts with SWE Kick-off and is ConcurrentCross-Validation During Each Phase of Test Design and Dev.Test Execution Starts with Units, With Design if Possible
https://insights.sei.cmu.edu/sei_blog/2013/11/using-v-models-for-testing.html
Software Unit - DecompositionModule, Component, CSU (Computer Software Unit - MIL-STD-498)
– Composes a Subsystem– Requires a Test Driver for Unit Test– Smallest Unit of Re-use– Could be an Object in OOA/OOD/OOP– Cohesion – functions (methods) that belong together– Loose Coupling – reliance on other modules
Subsystem, CSCI (Computer Software Configuration Item – MIL-STD-498)
– Specific Use Cases or Services– Can Stand Alone, Tested Alone Up to System/Subsystem Level
System, CSC (Computer Software Configuration)– A Hierarchy of Subsystems with Interfaces and Interaction Clearly Defined– Interface Control Document– Concurrent Engineering (Subsystem Teams)– Independent Testing (Validation & Verification)
System of Systems - http://sosengineering.org/2014/ (Research Area)
Sam Siewert 6
Goal this Week – Unit TestingSimple Unit Example
– Stick with the Crypto (Simple Substitution and Transposition)– Expand to include RAID-10 and RAID-50? (Simple Stripe and Mirror or Parity)– Option for Simple Image Processing – E.g. 3 transforms
Ideal– Requirements fully Validated (baseline)– System design fully validated– Architecture validated– Module Detailed Design validated and verified– Code Construction Complete According to Coding Standards– Start Testing Unit Test– Verify Module, Debug, Deliver to SQA
Reality– Proof-of-Concept or Exploratory Coding– Requirements, System, Architecture, Design will be Refined Over Time– Adaptation of Previously Developed Modules (Module Re-use)– Sustaining Engineering – Bug Fixes and Performance Improvements
Sam Siewert 7
Black-Box (Outside Code under Test)Expected Functionality of the Unit
– Focus on Input (test case) and Output (expected)– Can We test all Input Combinations? - Not Likely– Strategies for Test Cases (Negative, Positive)
Initial Conditions and Start-up– Test Empty or Minimum Inputs – No Input– Global State at start-up?
Boundary Value - Corner Cases and Pattern Tests– Extreme values (Max, Min, and Combinations)– Empty, Full, Max/Min Alternating, Etc.
Stress Testing and Soak Testing (Stability)– Random Inputs for 1000’s of Calls– Run Continuously Over night
Fitness Testing (Selection of Functional test cases)– Is the Output as Expected– E.g. Random Number Produces Uniform Distribution)– Does the Output meet Mathematical Requirements?
Performance Testing– Operations / second, with/without Compiler Optimization– On Reference Hardware– Matching Expected Algorithmic Complexity?
C or Script Driver for Each Case Sam Siewert 8
Fitness testing
Harman, Mark, and John Clark. "Metrics are fitness functions too." 10th International Symposium on Software Metrics, 2004. Proceedings.. Ieee, 2004.
Ahmed, Amr AbdelFatah, Mohamed Shaheen, and Essam Kosba. "Software testing suite prioritization using multi-criteria fitness function." 2012 22nd International Conference on Computer Theory and Applications (ICCTA). IEEE, 2012.
Boundary value analysis testing• Function with n parameters covered?• Ring buffer (full or empty?)• Tree is balanced?• Database is normalized?
Khan, Mohd Ehmer. "Different approaches to black box testing technique for finding errors." International Journal of Software Engineering & Applications 2.4 (2011): 31.
Nidhra, Srinivas, and Jagruthi Dondeti. "Black box and white box testing techniques-a literature review." International Journal of Embedded Systems and Applications (IJESA) 2.2 (2012): 29-50.
Black-Box Fitness Test [ rand(), srand() ]
Sam Siewert 9
C++ Test Driver
Is this Distribution Uniform?
How Close to Uniform?
Is it a Fair Test?
Should We Test more Iterations?
Is C++ rand() good?
Fit to Ship?
1) Good random number generators are (not so) easy to find, P. Hellekalek2) Random Number Generators: Good Ones are Hard to Find, S. Park, K. Miller
Random Number Generators – Improved?Why are they all suspect?
Not truly random, they emulate randomness –uniformity?
Must seed the sequence –with what?– Clock – same 2x per day or at
least 1x each day– Date and time?– Something else?
The point is to assess how good or how bad?
Repetition in pseudo-random sequence? For any seed? Sam Siewert 10
Randomness
Better left to nature
Groovy Baby!
Methods to seed pseudo-random
Cloudflare – Lava lamp wall
Tom Scott on Cloudflare method
What’s not to like?
Other options – Patent #1, #2, paper
Does Seed Affect Quality of rand()?
Sam Siewert 11
White-Box (Inside Code under Test)Single Step Debug and Examine State
Basic Block – Function Body or Entry to Exit Point– Instruction Path Length– Clock / Time Path Length– Does the Block Return or Terminate?
Coverage Criteria– Function (method)– Path– Statement– Instruction (Instructions Can Be Conditional)– Short-Circuit Logic Coverage
Sam Siewert 12
Automatic Static EvaluationsCode Complexity Based on Number of Paths [Structure]Principle of McCabe Metric (We will study in depth later)
Sam Siewert 13
Most complexDo you agree?
Short Circuit LogicIf (func_A() && func_B()) then do { funct_C() } else do { funct_D()};– If result of func_A() is FALSE (zero), do we need to call
func_B()?– Have we fully tested if we cover “do” and “else do”?– Why or why not?– Why do Compilers Implement Short-Circuit Logic?
If(func_A() or func_B()) then do { funct_C() } else do { funct_D()};
Any conjunctive logic
Sam Siewert 14
Short-Circuit Logic ExampleDo the Test Cases Provide Full Coverage of A, B, C, D?
Sam Siewert 15
#include <stdio.h>
int function_A(void){
static int toggle_A=0;
printf("function_A\n");if(toggle_A == 0)
toggle_A=1;else
toggle_A=0;
return toggle_A;}
int function_B(void){
static int toggle_B=0;
printf("function_B\n");if(toggle_B == 0)
toggle_B=1;else
toggle_B=0;
return toggle_B;}
void function_C(void){
printf("do function_C\n");}
void function_D(void){
printf("do function_D\n");}
int main(void){
int rc;
// Test Case #1 - Call all functionsrc=function_A();rc=function_B();function_C();function_D();
// Test Case #2, Test use in logicif((rc=(function_A() && function_B())))
function_C();else
function_D();
return(1);}
Potential short-circuit
Gcov Analysis of Previous Example(Code Under Test – A, B, C, D)
Sam Siewert 16
-: 0:Source:sclogic.c-: 0:Graph:sclogic.gcno-: 0:Data:sclogic.gcda-: 0:Runs:1-: 0:Programs:1-: 1:#include <stdio.h>-: 2:-: 3:2: 4:int function_A(void)-: 5:{-: 6: static int toggle_A=0;-: 7:2: 8: printf("function_A\n");2: 9: if(toggle_A == 0)1: 10: toggle_A=1;-: 11: else1: 12: toggle_A=0;-: 13:2: 14: return toggle_A;-: 15:}-: 16:
1: 17:int function_B(void)-: 18:{-: 19: static int toggle_B=0;-: 20:1: 21: printf("function_B\n");1: 22: if(toggle_B == 0)1: 23: toggle_B=1;-: 24: else
#####: 25: toggle_B=0;-: 26:1: 27: return toggle_B;-: 28:}-: 29:1: 30:void function_C(void)-: 31:{1: 32: printf("do function_C\n");1: 33:}-: 34:2: 35:void function_D(void)-: 36:{2: 37: printf("do function_D\n");2: 38:}
Never tested
Gcov Analysis of Previous Example(Test Driver Code Coverage)
Sam Siewert 17
-: 39:-: 40:1: 41:int main(void)-: 42:{-: 43: int rc;-: 44:-: 45: // Test Case #1 - Call all functions1: 46: rc=function_A();1: 47: rc=function_B();1: 48: function_C();1: 49: function_D();-: 50:-: 51: // Test Case #2, Test use in logic1: 52: if((rc=(function_A() && function_B())))
#####: 53: function_C();-: 54: else1: 55: function_D();-: 56:1: 57: return(1);-: 58:} Test call never made
Questions for Next TimeIdentification of Test Cases
Unit Test Driver Code Development
Automation?
Is the Test Valid, Correct?
How Do We Verify Coverage Criteria?
What are Unit Test Exit Criteria? - When are We Done?
Observation – Test Code May be More Voluminous and Complex than Code Under Test
Sam Siewert 18
January 21, 2016 Sam Siewert
Linux Skills
Building Unit Test Drivers(Strategy and Methods)
Walk-through and RAID DiscussionRead through Crypto Backgrounder
Browse code
Look at Gcov and Lcov for Crypto example – source coverage results
Discuss Quality– Defects?– Non-functional requirements improvements (readability,
portability, usability, etc.)– Strength of encryption (cryptanalysis durability)– Applications?
Sam Siewert 20
Assignment #2Use Code Example Unit– Crypto Transposition or Substitution Functions– RAID Parity or Mirroring with Striping– Image Processing– Other?
Debug to Determine Paths and To Drive Test Cases
Develop Test Drivers
Test Tests
Run Tests and Produce Test Report
Package for Delivery (Report)
Sam Siewert 21