USING SOFTWARE METRICS IN EDUCATIONAL ENVIRONMENT

Ivan Pribela, Zoran Budimac, Gordana Rakić

Content

• Motivation• Assessment process• A case study• Conclusion

Motivation

• Automated assessment systems are very helpful– Reduce load for instructors– Quicker feedback for students

• But have limited scope– Black-box (input-output) tests, unit tests– Lately code style checks

• Software metrics are not used– Utilized only by plagiarism detection– Not to help students improve their skills

Motivation (cont.)

• Software metrics can tell a lot about a program– Have highly diverse methodologies and objectives– Difficult to define quality for various programs in uniform

way– What which metric means for particular assignment?– What metrics matter the most?

• We explore the usage of software metrics in automated assessment– To cover testing of aspects that demand instructor attention– Algorithm complexity– Number and size of programming units and functions

Assessment processStudent solution

Testreport

Student solutionMODULE Triplets1;...VAR x, y, z: INTEGER;BEGIN FOR x := 1 TO Gr DO FOR y := 1 TO Gr DO FOR z := 1 TO Gr DO IF x*x + y*y = z*z THEN WriteLn; WriteString('x = '); WriteCard(x,2); WriteString(', y = '); WriteCard(y,2); WriteString(', z = '); WriteCard(z,2) END END END ENDEND Triplets1.

Student solution

Parsing the solution• eCST generator

(the SMIILE tool)– Recognizes input language– Calls appropriate scanner & parser– Generates an eCST representation

• Enriched Concrete Syntax Tree– Modified Concrete Syntax Tree– Enriched with marker nodes

• Unit: class, module…• Loop Statement: for, while, repeat…• Branch Statement: if, case, switch…

– Independent of programming language

– Basis for calculation of software metrics

Student solution

eCSTParse

Measuring eCST• Metrics calculator

(the SMIILE tool)– Software metrics tool in the

development– Analyzes the eCST representation– Calculates software metrics– Exports results in XML file

• Final result is one XML file with values of all calculated metrics

• Current prototype of SMIILE tool supports several software metrics– Lines of Code– Cyclomatic Complexity– …

Student solution

Metric values

in xml

eCSTParse

Measure

Metric values in XML<?xml version="1.0"?><sourceElement endLine="24" firstLine="1" name="" type=""> <metrics loc="24" cc="4"/> <sourceElement endLine="24" firstLine="1" name="Triplets1" type="CONCRETE_UNIT_DECL"> <metrics loc="24" cc="4"/> <sourceElement endLine="23" firstLine="9" name="FOR" type="LOOP_STATEMENT"> <metrics loc="15" cc="4"/> <sourceElement endLine="22" firstLine="10" name="FOR" type="LOOP_STATEMENT"> <metrics loc="13" cc="3"/> <sourceElement endLine="21" firstLine="11" name="FOR" type="LOOP_STATEMENT"> <metrics loc="11" cc="2"/> ... </sourceElement> </sourceElement> </sourceElement> </sourceElement></sourceElement>

Student solution

Metric values

in xml

eCSTParse

Measure

Transform XML to properties

• XSL Transformator– Uses XSLT stylesheet

• Input XML file– Values of calculated

software metrics

• Output XML file– Can be manipulated

easily inside the testing system

Student solution

Metric values

in xml

eCST

Matric values

as properties

Parse

Measure

Transform

Metric values as properties<?xml version="1.0" encoding="UTF-8"?><metrics> <Triplets1> <loc>24</loc> <cc>4</cc> </Triplets1></metrics>

Student solution

Metric values

in xml

eCST

Matric values

as properties

Parse

Measure

Transform

Testing the values• Testing system (Testovid)

– Implemented as a framework for running domain specific testers

• Domain specific testers– Written as Apache Ant scripts

• Using software metrics– Script runs SMIILE tool– Transforms XML file with metrics

values– Loads calculated values– Freely uses them for grading,

intelligent advice generation…

• Final report– Contains advices– Success/failure information– Presented to the student

Student solution

Metric values

in xml

Script with

metric control values

eCST

Matric values

as properties

Testreport

Parse

Measure

Transform

Test

Test reportCourse: Data structures and algorithmsAssignment: Assignment 1 - Pythagorean tripletsStudent: John DoeTime: 05.09.2012. 11:15:00

Compilation100% (from 2 points)All is well, no errors.

Correctness33% (from 6 points)Not all triplets were found, check loop boundaries.

Optimality50% (from 2 points)Try using Euclid's formula.

-+-+-+-+-+-+-+-+-

Total: 5 out of 10 points.

Student solution

Metric values

in xml

Script with

metric control values

eCST

Matric values

as properties

Testreport

Parse

Measure

Transform

Test

A case study

• Data structures and algorithms course• Testing efficiency of a Modula 2 program• Using cyclomatic complexity metric– detect loop and branch statements

• Created a domain specific tester for Testovid– differentiate between typical student solutions– award points accordingly

• The assignment– Write a program which prints Pythagorean triplets,

positive integer numbers x, y and z for which x2+y2=z2

Solution 1: Naive solutionMODULE Triplets1;...VAR x, y, z: INTEGER; zreal: REAL;BEGIN FOR x := 1 TO Gr DO FOR y := 1 TO Gr DO zreal := REAL(Sqrt(LONGREAL(x*x + y*y))); z := TRUNC(zreal); IF zreal = FLOAT(z) THEN WriteLn; WriteString('x = '); WriteCard(x,2); WriteString(', y = '); WriteCard(y,2); WriteString(', z = '); WriteCard(z,2) END END ENDEND Triplets1.

• Cyclomatic complexity3

• EfficiencyAverage

• Points50%

Solution 2: Brute force solutionMODULE Triplets2;...VAR x, y, z: INTEGER;BEGIN FOR x := 1 TO Gr DO FOR y := 1 TO Gr DO FOR z := 1 TO Gr DO IF x*x + y*y = z*z THEN WriteLn; WriteString('x = '); WriteCard(x,2); WriteString(', y = '); WriteCard(y,2); WriteString(', z = '); WriteCard(z,2) END END END ENDEND Triplets2.

• Cyclomatic complexity4

• EfficiencyBad

• Points0%

Solution 3: Using Euclid’s formulaMODULE Triplets;...VAR x, y, z, m, n: CARDINAL;BEGIN FOR m := 1 TO Gr DO FOR n := 1 TO m-1 DO x := m*m - n*n; y := 2*m*n; z := m*m + n*n; WriteLn; WriteString('x = '); WriteCard(x,2); WriteString(', y = '); WriteCard(y,2); WriteString(', z = '); WriteCard(z,2) END ENDEND Triplets.

• Cyclomatic complexity2

• EfficiencyGood

• Points100%

Non solutionMODULE Triplets4;...VAR x, y, z, m, n, w, i, temp : CARDINAL;BEGIN w := 1; n := 0; FOR i := 1 TO Gr DO m := n + w; x := m*m - n*n; y := 2*m*n; z := m*m + n*n; WriteLn; WriteString('x = '); WriteCard(x,2); WriteString(', y = '); WriteCard(y,2); WriteString(', z = '); WriteCard(z,2); temp := w; w := 3*w + 4*n; n := 2*temp + 3*n ENDEND Triplets4.

• Cyclomatic complexity1

• EfficiencyExcellent

• Points100%

• Lose pointsfor correctness

Summary• The greater the cyclomatic complexity

the worst the solution efficiency• Awarded points should be reverse

proportional to the cyclomatic complexity– Inefficient solutions – no points– Average solutions – half the maximum

points– Efficient solutions – all the points

• Instructor should use knowledge and experience to– Choose metrics– Define minimum and maximum metric

values– Define awarded points for those cases

• Testing system can automatically– Classify student solutions– Grade them accordingly

Solution Efficiency Metric Points

Triplets1 Average 3 50%

Triplets2 Bad 4 0%

Triplets3 Good 2 100%

Triplets4 Excellent 1 !!! 100%

Se-ries1

Conclusion

• Utilized software metrics in the assessment process• Increased the scope of aspects that can be covered

by automatic tests• Platform and programming language independent• Support a wide range of metrics• Left great flexibility in selecting interesting metrics• Can provide hints and advices to students• Added intelligent assistance• Improved student learning experience

Thank you for your attention

Questions?