Quality Assurance in Software Development ... · tugraz Institute for Software Technology QualityAssurance I Lecture: Tue11:15–12:00(i12) I Exercise: Tue12:00–12:45(i12) I Optional:

tugrazInstitute for Software Technology

Quality Assurance in Software DevelopmentQualitätssicherung in der Softwareentwicklung

A.o.Univ.-Prof. Dipl.-Ing. Dr. Bernhard Aichernig

Institute for Software TechnologyGraz University of Technology

Austria

Summer Term 2019

B. Aichernig Quality Assurance in Software Development1 / 20


Agenda

1 Organisation

2 Contents

3 Quality of SW



Quality Assurance

I Lecture: Tue 11:15 – 12:00 (i12)I Exercise: Tue 12:00 – 12:45 (i12)I Optional: Fri 14:15–15:00 (i12)I Registration via tugonline – 8.3. strict!I Student assistants:

I Dominik AugustinI Jorrit Stramer



Additional Resources

I WWW: http://www.ist.tugraz.at/qs.htmlI Newsgroup: tu-graz.lv.qsI Email: Subject: [QS19]

I [email protected]


http://www.ist.tugraz.at/qs.html


Marking

I Marking if registered on 8.3.I Written exam (Klausur) : 25.6. (50 %).I Three practical tasks (groups of 3):

I Task 1: 12.3.–2.4. (16 %)I Task 2: 2.4.–7.5. (16 %)I Task 3: 7.5–4.6. (18 %)

I Exam and exercises count each 50%I Positive if > 50% total points andI > 30% on exam andI > 30% on exercises in totalI Prüfungsabbruch: the registration to a group for the exercises

counts as item of work.



Marking (cont.)

Grading key:50,01% - 62,50%: genügend62,51% - 75,00%: befriedigend75,01% - 87,50%: gut87,51% - 100,0%: sehr gut

Nachklausur:I Date: 9.7., 11:00I After excused absence at exam

I e.g. due to illness with sick certificate (ärztlicher Bestätigung)I work is no excuse (take leave!)

I For all negative who have been excused at exam!



PlanDatum VO UETue 5.3. Organisation, IntroductionTue 12.3. Test Coverage Ausgabe Task 1Tue 19.3. Symbolic and Concolic ExecutionTue 26.3. Specifications 1Tue 2.4. Tool Demo Abgabe Task 1, Ausgabe Task 2Tue 9.4. Specifications 2

OsterferienTue 30.4. Property- & Model-based TestingTue 7.5. Tool Demo Abgabe Task 2, Ausgabe Task 3Tue 14.5.Tue 21.5. Input-Output ConformanceTue 28.5. (Model-based) Mutation TestingTue 4.6. Equivalence Classes and Test-

Sequence GenerationAbgabe Task 3

Di nach PfingstenTue 18.6. Decision Tables, White-box Test-

ing, ReviewsTue 25.6. Written exam



What we will not cover?

Quality management:I Process management, e.g. ScrumI Process improvement, e.g. ISO 9000I Test management, e.g. IBM Rational Quality ManagerI Version control, e.g. svn, gitI ...



General Aims of the Course

I Getting familiar with quality assurance techniquesI Raising the awareness for quality of softwareI Practising fault-based thinkingI Being able to distinguish process-oriented and technical quality

assuranceI Challenging common quality criteriaI Realising the relations between the foundations of software

engineering and quality assurance.I Learning the use and foundations of advanced testing tools.



Specific Learning Targets and Skills

I Getting familiar with a professional IDEI Writing good tests :(I Generating good tests :)I Specifying test oraclesI Using and understanding state-of-the-art test case generatorsI Test coverage metrics

"Don’t write test cases,

generate them!"

(John Hughes)



Specific Learning Targets and Skills

I Getting familiar with a professional IDEI Writing good tests :(I Generating good tests :)I Specifying test oraclesI Using and understanding state-of-the-art test case generatorsI Test coverage metrics

"Don’t write test cases,

generate them!"

(John Hughes)



Course ContentsI Introduction and motivationI Challenges of software testingI Coverage: Measuring the quality of test casesI Testing with Concolic ExecutionI Contracts as test oraclesI Model-based TestingI Mutation TestingI ReviewsI (Proof-based Development)I Tools (exercise):

I MS Visual Studio Enterprise 2015I MS IntelliTest (Pex)I .NET Code ContractsI FsCheck



Course ContentsI Introduction and motivationI Challenges of software testingI Coverage: Measuring the quality of test casesI Testing with Concolic ExecutionI Contracts as test oraclesI Model-based TestingI Mutation TestingI ReviewsI (Proof-based Development)I Tools (exercise):

I MS Visual Studio Enterprise 2015I MS IntelliTest (Pex)I .NET Code ContractsI FsCheck



Quality?

I Today, often a sufficient level ofI correctnessI efficiencyI costs

cannot be guaranteed.I Vision: Development methods for SW with warranty.



Quality?

I Today, often a sufficient level ofI correctnessI efficiencyI costs

cannot be guaranteed.I Vision: Development methods for SW with warranty.



Limited WarrantiesI Cisco: “Cisco does not warrant that the Software will

operate uninterrupted or error-free or that all errors willbe corrected. In addition, Cisco does not warrant thatthe Software or any equipment, system or network onwhich the Software is used will be free of vulnerability tointrusion or attack."Limited Warranty and Disclaimer, (c) Disclaimer; effective: May 22,

2017: (https://www.cisco.com/c/en/us/about/legal/

cloud-and-software/end_user_license_agreement.html)

I Skydrop Sprinkler Controller: “This warranty does notcover consumable parts, including batteries, unlessdamage is due to defects in materials or workmanship ofthe Product, or software (even if packaged or sold withthe product).”(https://www.skydrop.com/warranty/)

Marketing withApp


https://www.cisco.com/c/en/us/about/legal/cloud-and-software/end_user_license_agreement.html


https://www.skydrop.com/warranty/


Limited WarrantiesI Cisco: “Cisco does not warrant that the Software will

operate uninterrupted or error-free or that all errors willbe corrected. In addition, Cisco does not warrant thatthe Software or any equipment, system or network onwhich the Software is used will be free of vulnerability tointrusion or attack."Limited Warranty and Disclaimer, (c) Disclaimer; effective: May 22,

2017: (https://www.cisco.com/c/en/us/about/legal/

cloud-and-software/end_user_license_agreement.html)

I Skydrop Sprinkler Controller: “This warranty does notcover consumable parts, including batteries, unlessdamage is due to defects in materials or workmanship ofthe Product, or software (even if packaged or sold withthe product).”(https://www.skydrop.com/warranty/)

Marketing withApp




https://www.skydrop.com/warranty/


Bugs

Part of engineering jargon for manydecades:

I Moth trapped in relay of Mark II(Hopper 1946)

I Little faults and difficulties(Edison 1878):

I Software bugs

Relay #70 Panel F(moth) in relay.

First actual case of bug being found.



Bugs




I Software bugs

Relay #70 Panel F(moth) in relay.

First actual case of bug being found.



Bugs




I Software bugs

Definition

A software bug is the common termused to describe an

I error, flaw, mistake, failure, orfault in a computer program orsystem

I that produces an incorrect orunexpected result,

I or causes it to behave inunintended ways.(Wikipedia 2012)



Some Bugs Become Famous!

I Ariane 5 test flight (1996)I out of control due to

software failureI controlled destruction!

I Loss ofI money and timeI satellitesI research (TU Graz)

I Dijkstra (EWD 1036):I call it error, not bugI a programmer created it



Some Bugs Become Famous!

I Ariane 5 test flight (1996)I out of control due to

software failureI controlled destruction!

I Loss ofI money and timeI satellitesI research (TU Graz)

I Dijkstra (EWD 1036):I call it error, not bugI a programmer created it



Some Bugs Hide for a Long Time!Binary search bug in Java

I JDK 1.5 library (2006)

I out of boundary access oflarge arrays

I due to integer overflow

I 9 years undetected

Algorithm was proven correct!

I Programming Pearls[Bentley86, Bentley00]

I assuming infinite integers:(

1 public static2 int binarySearch(int[] a,int key)3 {4 int low = 0;5 int high = a.length - 1;67 while (low <= high) {8 int mid = (low + high) / 2;9 int midVal = a[mid];

1011 if (midVal < key)12 low = mid + 1;13 else if (midVal > key)14 high = mid - 1;15 else16 return mid; // key found17 }18 return -(low + 1); // key not found19 }

“Beware of bugs in the above code;I have only proved it correct, not tried it.”[Knuth77]





































1 public static2 int binarySearch(int[] a,int key)3 {4 int low = 0;5 int high = a.length - 1;67 while (low <= high) {8 int mid = (low + high) >>> 1;9 int midVal = a[mid];





Risks for the Public due to SW

The Risks Digest

http://catless.ncl.ac.uk/Risks/


http://catless.ncl.ac.uk/Risks/


Edsger W. Dijkstra

I In academia, in industry, and in the commercial world, there is awidespread belief that computing science as such has been all butcompleted and that,

I consequently, computing has matured from a theoretical topic forthe scientists to a practical issue for the engineers, the managers,and the entrepreneurs. [...]

I ... then he characterises the software crisis and concludes:I I would therefore like to posit that computing’s central challenge,

"How not to make a mess of it," has not been met.



Edsger W. Dijkstra (cont.)

I On the contrary, most of our systems are much more complicatedthan can be considered healthy, and are too messy and chaotic tobe used in comfort and confidence.

I The average customer of the computing industry has been servedso poorly that he expects his system to crash all the time, and wewitness a massive worldwide distribution of bug-ridden software forwhich we should be deeply ashamed.

(Communications of the ACM, Mar 2001)



Why?

I A possible explanation:

Quality assurance requires a certain degree of (healthy) redundancy

I → extra costs! (≥ 50% for critical systems)I Examples of redundancy:

I process managementI documentationI pair programmingI modellingI test casesI redundant components in fault-tolerant systems (accepted)



Why?

I A possible explanation:

Quality assurance requires a certain degree of (healthy) redundancy

I → extra costs! (≥ 50% for critical systems)I Examples of redundancy:

I process managementI documentationI pair programmingI modellingI test casesI redundant components in fault-tolerant systems (accepted)


Documents

Quality Assurance in Software Development ... · tugraz Institute for Software Technology QualityAssurance I Lecture: Tue11:15–12:00(i12) I Exercise: Tue12:00–12:45(i12) I Optional: