Assessing the Reliability of a Human Estimator

Assessing the Reliability of a Human Estimator

http://nas.cl.uh.edu/boetticher/publications.html The 3rd International Predictor Models in Software Engineering (PROMISE) Workshop

Gary D. Boetticher Nazim Lokhandwala Univ. of Houston - Clear Lake, Houston, TX, [email protected] [email protected]

http://nas.cl.uh.edu/boetticher/publications.html


mailto:[email protected]

mailto:[email protected]

Current Configuration of PROMISE Repository


Defect Prediction – 18

Others - 9

Effort Estimation - 9

http://nas.cl.uh.edu/boetticher/publications.html The 3rd International Predictor Models in Software

Engineering (PROMISE) Workshop

Research vs. Reality according to JörgensenTSE ’07: 300+ software est. papers,

76 journals, 15+ Years

5226197Misc.

4621223Human

7441321ML

2557013748Algorithm

Total00-0489-99-89

68% Algorithm

20% ML12% Human

72%Kitchenham 02100%Hill 0084%Jørgensen 9786%Paynter 9662%Heemstra 9189%Hihn 91

HumanPaper

JSS ’04: Compendium of expert estimation studies

82% Human

18% Formal

http://nas.cl.uh.edu/boetticher/publications.html The 3rd International Predictor Models in Software

Engineering (PROMISE) Workshop

Research vs. Reality

How to resolve?

• Researchers coerce/entice/exhort/nudge practitioners

• Practitioners ignore researchers

• Researchers meet practitioners where they are

COCOMO

Statement of Problem

How do human demographics affect human-based estimation?

Can predictive models be constructed using human demographics?


PROMISE 2006 Addressed the problem using Genetic Programs and non-linear

regression (up to 5th order) models Produced some accurate(77 – 93%) models, GP solutions got lengthy:

The 3rd International Predictor Models in Software Engineering (PROMISE) Workshop


((MgmtGCourses ^ (((Log (((TotLangExp / (TotLangExp / (TechGCourses * HWPMExp))) - (TechGCourses * HWPMExp)) - ((Sin (MgmtGCourses ^ (Sin ((TechGCourses * HWPMExp) - (MgmtGCourses ^ (((Log (HWPMExp ^ (TotLangExp / (TechGCourses * HWPMExp)))) - (Abs (Log ((TotLangExp / (TechGCourses * HWPMExp)) - ((Sin ((Sin (Abs (TechUGCourses / MgmtGCourses))) - (TotLangExp / (MgmtGCourses ^ (((Log (((TotLangExp / (HWPMExp / SWProjEstExp)) - (Sin (TotLangExp / (TotLangExp / ((MgmtGCourses ^ ((Log (TechGCourses * HWPMExp)) - (Sin (Abs (Log ((HWPMExp / SWProjEstExp) - (TechGCourses * HWPMExp))))))) + ((Sin (TechGCourses * HWPMExp)) - (Sin (TechUGCourses / MgmtGCourses)))))))) - (Sin (TechUGCourses / MgmtGCourses)))) - (TechGCourses * HWPMExp)) - (Sin (TechUGCourses / MgmtGCourses))))))) - (HWPMExp / SWProjEstExp)))))) - (Sin (TechUGCourses / MgmtGCourses)))))))) - ((Sin (Abs (Log ((TotLangExp / (TechGCourses * HWPMExp)) - ((Sin ((Sin (Abs (Log (HWPMExp ^ (TotLangExp / (TechGCourses * HWPMExp)))))) - (TechGCourses * HWPMExp))) - (HWPMExp / SWProjEstExp)))))) - (Sin (TechUGCourses / MgmtGCourses)))))) - (TotLangExp / (TechGCourses * HWPMExp))) - (Sin (TechUGCourses / MgmtGCourses)))) + (TotLangExp / (TechGCourses * HWPMExp)))

So for 2007…

The 3rd International Predictor Models in Software

Engineering (PROMISE) Workshophttp://nas.cl.uh.edu/boetticher/publications.html

PROMISE 2007

• Larger sample set.• 2006 PROMISE 122 samples• 2007 PROMISE 178 samples

• Many learners. • 51 classifiers, 4142 experimental trials

• Attribute analysis.

• Simpler models. • Focus is on classifiers Human readable models

Strategy

2. Create a Web-based survey

Users demographics

Users Estimate software components

Feedback Users

3. Build models: demographics estimates


The Survey (2001 -2005)

http://nas.cl.uh.edu/boetticher/EffortEstimationSurvey.html


Demographics Personal Academic Background Work Experience Domain Experience



Ecommerce: Competitive Procurement

Buyer Admin

Buyer1 Buyern...

Buyer Software

DistributionServer

Supplier1

Supplier2

Suppliern

:

SupplierSoftware


Sample Estimation Screenshots


Feedback to Users


User Demographics - 1


• Average age: 31.43

• 148 males, 30 females

• 1% Ph.D., 24% Master, 72% Bach., 5% High School

• 25 countries:• 42% India, 32% U.S., 6% Romania, 4% Vietnam.

User Demographics - 2


5.3856283.6629 Process Industry

4.4391251.4382 Procurement & Billing

Domain Experience

5.3856283.6692 Software Projects

4.4390251.4382 Hardware Projects

No. of Projects estimated

2.4757151.6967 Software Project Manager

3.0633251.0169 Hardware Project Manager

Years of Experience as a

Std. Dev.Max.Ave.

Years

Data preprocessing & Experiments


Remove outliers: Estimate > 10 * Actual or Estimate < 0.1*Actual

178 Samples163

Extract:

• 25 Worst under-estimators• 25 Best estimators• 25 Worst over-estimators

WEKA: 51 Classifiers, 4 seeds, 10-fold Attribute Reduction: 2 configs.

Results: Under vs. Best

64%VFI

64%ThresSel

64%Logistic

68%J48

76%PART

AccuracyClassifier


YYYTotal Lang Exp.

YYYTotal Workshops

YTotal Conferences

YTech Undergrad

Courses

YYSoftware Proj. Mgmt

Exp.

YYLevel of College

YYYY# of Hardware Proj. Est.

YYYMgmt Undergrad Crses

YMgmt Grad. Courses

YYYYHardware ProjectManagement Exp.

YYYDomain Exp.

VFIThresh.PARTLogisticJ48Demographic

Evaluator Classifier

Ave. Accuracy48.22%

68%Logistic/Logistic

70%VFI / VFI

74%PART/J48

74%J48/J48

74%LogitBoost/J48

74%Bagging/J48

76%ThresholdSel/ThresholdSel

78%ADTree/Part

AccuracyClass./Eval.

Under vs. Best: Attribute Reduction

YYTotal Lang Experience

YYTotal Workshops

YYTotal Conferences

YYTech Undergrad Crses

YYSoft. Proj. Mgmt Exp.

YYYLevel of College

Y# of Software Proj. Est.

YYYY# of Hardware Proj. Est.

YYYMgmt Undergrad Crses

YYYMgmt Grad. Courses

YYYYHardware Proj. Mgmt Exp.

YYYYDomain Experience

VFIThreshPARTLogisticJ48Demographic

Evaluator Classifier


68%Logistic / Logistic

70%VFI / VFI

74%PART / J48

74%ADTree / J48

74%PART/ PART

74%J48/ PART

76%ADTree/ThreshSel

AccuracyClass / Eval

Under vs. Best: Attribute Reduction


Domain Exp <= 3| No Of Hardware Proj Estimated <= 4| | Hardware Proj Mgmt Exp <= 1| | | MgmtUGCourses <= 0: BEST (23.0/8.0)| | | MgmtUGCourses > 0: UNDER (13.0/1.0)| | Hard. Proj Mgmt Exp > 1: BEST (5.0)| No Of Hard. Proj Est. > 4: UNDER (5.0)Domain Exp > 3: BEST (4.0)

J48 Rule: 74% Accuracy

BEST UNDER <-- classified as 21 4 | BEST 9 16 | UNDER

Results: Best vs. Over

60%Ridor

60%ThresholdSel

60%RandComm

62%Decorate

66%RndTree

AccuracyClassifier

YYTotal Lang Experience

YTotal Workshops

YYTotal Conferences

YTech Undergrad Courses

YYSoft. Proj. Mgmt Exp.

Y# of Software Proj. Est.

YMgmt Undergrad Crses

YYYMgmt Grad. Courses

YYYHard. Proj Mgmt Exp.

ThresholdSelector

RidorRnd

CommDemographic


Ave. Accuracy42.86%

62%ADTree /ThresholdSel

66%ThresholdSel /ThreshSel

72%Rand. Comm./RandComm

80%IB1 / Ridor

AccuracyClass/ Eval

Experiment: Best vs. Over

62%RidorRidor

62%ThresholdSelRidor

64%RidorThresholdSel

66%ThresholdSelNNge

72%DecoratePART

72%DecorateNNge

72%DecorateRndComm

74%DecorateRandomFores

t

74%DecorateIBk

74%DecorateIB1

80%RndCommRandomTree

80%RndCommRndComm

AccuracyEvaluatorClassifier


YYYTotal Lang Experience

YTotal Workshops

YTech Undergrad Courses

YYTech Grad Courses

YSoftware Proj. Mgmt Exp.

YYProcurement Industry Exp

YLevel of College

Y# of Hardware Proj. Est.

YYMgmt Undergrad Courses

YMgmt Grad. Courses

YYYHard. Proj Mgmt Exp

YDomain Experience

ThreshRidorRand

Comm.DecorateDemographic

Experiment:Best vs. Over

TechUGCourses < 45.5| Hardware Proj Mgmt Exp < 6| | No Of Hardware Proj Estimated < 4.5| | | No Of Hardware Proj Estimated < 3| | | | TechUGCourses < 23| | | | | Hardware Proj Mgmt Exp < 0.75| | | | | | TechUGCourses < 18| | | | | | | Hardware Proj Mgmt Exp < 0.13| | | | | | | | TechUGCourses < 0.5| | | | | | | | | TechUGCourses < -1 : F (1/0)| | | | | | | | | TechUGCourses >= -1| | | | | | | | | | Degree < 3.5 : A (4/0)| | | | | | | | | | Degree >= 3.5 : A (5/2)| | | | | | | | TechUGCourses >= 0.5| | | | | | | | | TechUGCourses < 5.5| | | | | | | | | | Degree < 3.5 : F (5/0)| | | | | | | | | | Degree >= 3.5| | | | | | | | | | | TechUGCrses < 2 : A (1/0)| | | | | | | | | | | TechUGCrses >= 2 : F (1/0)| | | | | | | | | TechUGCrses >= 5.5| | | | | | | | | | Degree < 3.5| | | | | | | | | | | TechUGCrs < 10.5 : A (3/0)| | | | | | | | | | | TechUGCrses >= 10.5| | | | | | | | | | | | TechUGCrs<12.5 : F (3/0)| | | | | | | | | | | | TechUGCrses >= 12.5| | | | | | | | | | | | | TechUGCrs<16: A (2/0)| | | | | | | | | | | | | TechUGCrs>15 : A (2/1)| | | | | | | | | | Degree >= 3.5 : F (1/0)| | | | | | | HardProjMgmt Exp >= 0.13 : A (2/0)| | | | | | TechUGCourses >= 18 : A (2/0)| | | | | Hard Proj Mgmt Exp >= 0.75 : F (1/0)| | | | TechUGCourses >= 23 : F (5/0)| | | No Of Hardware Proj Est >= 3 : F (1/0)| | No Of Hardware Proj Est >= 4.5 : A (5/0)| Hardware Proj Mgmt Exp >= 6 : F (4/0)TechUGCrses >= 45.5 : A (2/0)


BEST OVER <-- classified as 23 2 | BEST 8 17 | OVER

Conclusions

Very Good accuracy rates,

especially after attribute reduction

Bridges expert and model groups



Questions?



Thank You !


References

1) Jorgensen, M., “A review of studies on Expert Estimation of Software Development Effort,” Journal of Systems and Software, 2004.

2) Jørgensen, Shepperd, A Systematic Review of Software Development Cost Estimation Studies, IEEE Transactions on Software Engineering, 33, 1, January, 2007, Pp. 33-53.



Technology

Assessing the Reliability of a Human Estimator