7/28/2019 Exercise 5 Slides 2
1/27
ExerciseEmpirical Model Building and Methods(Empirische Modellbildung und Methoden)
Liliana Guzmn
SS 2012
Chapter 3.3 Design
7/28/2019 Exercise 5 Slides 2
2/27
Empirical Model Building &
Methods Exercise 5
Purpose
Gathering experience in
Operationalizing variables
Formulation of hypotheses
Sampling
Experimental design
Analysis of validity threats
Preparing the examination by
Reviewing most important aspects of the design phase
(using an example!)
Slide 2
7/28/2019 Exercise 5 Slides 2
3/27
Empirical Model Building &
Methods Overview of design phase
Purpose
Operationalizing variables and formalizing
Specifying experimental design: who? what? how? when?...
Steps
1. Operationalization
2. Formalization of hypotheses
3. Sampling
4. Selecting experimental design5. Selecting & designing data collection methods
6. Selection & designing material
7. Analyzing design validity/reliability
Specifying variables to make
them observable/measurable
Specifying what will be doneand observe/measure,
how, by whom,
when
Exercise
5
Specifying hypothesesto make them testable (challengeable)
Specifying who can participate
and how to get participants
How good is the design?
Can we draw conclusions? For which context?
Slide 3
7/28/2019 Exercise 5 Slides 2
4/27
Empirical Model Building &
Methods Experimental terminology (Example!!!!)
Research goal
Analyze notations for requirement specification to compare them w.r.t.
efficiency, effectiveness and acceptance from the perspective of
requirement engineers in the context ofIS development.
Object of study: Requirement notation
New graphical notation
Control: Structured natural languages (e.g. use cases) or graphical
notation (e.g. activity or sequence diagrams)
Population: Requirement engineers
Sample: Novice, e.g. students of the lecture of requirement
engineering.
Research purpose: comparison (quasi-) experimental design
Setting: IS material should be representative of this domain!
Variables: notations, efficiency, effectiveness and acceptance
SoP and SoA
support the
selection of
the control
Slide 4
7/28/2019 Exercise 5 Slides 2
5/27
Empirical Model Building &
Methods Experimental terminology (Example!!!!)
Research goal
Analyze notations for requirement specification to compare them w.r.t.
efficiency, effectiveness and acceptance from the perspective of
requirement engineers in the context of information system
development.
Variables: notations, efficiency, correctness and usability
Dependent variable (expected variation, response)
Efficiency, effectiveness and acceptance
Independent variable (expected cause(s))
Requirement notation
Control/Confounding variable
Characteristics of the sample, e.g. experience, skills, attitude
Characteristics of the setting and material, e.g. problems to be
modeled and tool support
Laboratory conditions, e.g. time, noise, light, fire alarmSlide 5
7/28/2019 Exercise 5 Slides 2
6/27
Empirical Model Building &
Methods Experimental terminology (Example!!!!)
Research goal
Analyze notations for requirement specification to compare them w.r.t.
efficiency, effectiveness and acceptance from the perspective of
requirement engineers in the context of information system
development.
Underlying hypotheses (expected relationship among variables!)
Requirements engineers using the new requirement notation are
more efficient than using activity diagrams.
Requirements engineers using the new requirement notation are
more effective than using (e.g.) activity diagrams.
Requirements engineers accepts the new requirement notation
more than (e.g.) activity diagrams.
Based on SoA and SoP we can specify the type of difference, i.e. more
than, less than, equal to, at least as ... as,
Slide 6
7/28/2019 Exercise 5 Slides 2
7/27
Empirical Model Building &
Methods Operationalization (Example!!!!)
Concept Variable(s) Instrument(s)
Efficiency Variable(s) Instrument(s)
Time required by a
requirement engineering
for modeling a setof requirements
Time: Minutes Time sheet
(Log file)
Effectiveness Variable(s) Instrument(s)
Degree to which arequirement engineering
correctly models a set
of requirements ,
Total N Defects:
N of defects per type:
(c, ) with c C
C:={missing, extraneous,
ambiguous, inconsistent,
correct & miscellaneous
information}
Defect report form
Slide 7
7/28/2019 Exercise 5 Slides 2
8/27
Empirical Model Building &
Methods Operationalization (Example!!!!)
Example of time sheets
In:
http://www.dummies.com/how-to/content/how-to-monitor-work-effort.html
http://www.corasystems.com/capabilities/project-time-sheet-tracking-software/
Slide 8
7/28/2019 Exercise 5 Slides 2
9/27
Empirical Model Building &
Methods Operationalization (Example!!!!)
Example of defect report form
In:
http://www.cs.umd.edu/projects/SoftEng/ESEG/manual/pbr_
package/download.html
In the above webpage you also find an example of
experimental design, including all material!
Slide 9
7/28/2019 Exercise 5 Slides 2
10/27
Empirical Model Building &
Methods Operationalization (Example!!!!)
Acceptance From SoA: Unified theory of acceptance and use of technology
Venkatesh, V.; Morris; Davis; Davis (2003), "User Acceptance of Information Technology: Toward a Unified View", MIS
Quarterly, 27, pp. 425478. For purpose of this exercise, we are interested in the final model! Nevertheless, this paper is
also a good example regarding Model Building.
Slide 10
7/28/2019 Exercise 5 Slides 2
11/27
Empirical Model Building &
Methods Operationalization (Example!!!!)
Variable(s) Performance expectancy Perceived of usefulness:= The degree to which a person believes
that using a particular system would enhance his or her job
performance.
Extrinsic motivation := The perception that users will want to performan activity because it is perceived to be instrumental in achieving
valued outcomes that are distinct from the activity itself, such as
improved job performance, pay, or promotions.
Job fit:= How the capabilities of a system enhance an individuals job
performance.
Relative advantage:= The degree to which using an innovation isperceived as being better than using its precursor.
.
Effort expectancy
Social influence
Facilitating conditions
Behavioral intention
Self efficiency
Anxiety
Attitude toward using technology
For remaining definitions See Venkatesh et al. 2003)
Slide 11
7/28/2019 Exercise 5 Slides 2
12/27
Empirical Model Building &
Methods Operationalization (Example!!!!)
Variable(s)
Slide 12
7/28/2019 Exercise 5 Slides 2
13/27
Empirical Model Building &
Methods Operationalization (Example!!!!)
Variable(s)
Slide 13
7/28/2019 Exercise 5 Slides 2
14/27
Empirical Model Building &
Methods Operationalization (Example!!!!)
Instruments Questionnaire
..
Question 10: Considering your performance expectancy with respect to , to what degree you agree or disagreewith the following statements:
a) I would find the system useful in my job.
1: Strongly agree
2: Agree
3: Neither agree nor disagree
4: Disagree
5: Strongly disagree
b) Using the system enables me to accomplish tasks more quickly.
Strongly agree
Agree
Neither agree nor disagree
Disagree
Strongly disagree
Interval scale!
It allows descriptive
and tendency analysis(H0: = 3 3
3; M:=median)
Ordinal scale!
It allows only descriptiveanalysis, e.g. frequency
analysis, mode, .
Slide 14
7/28/2019 Exercise 5 Slides 2
15/27
Empirical Model Building &
Methods Formalization of hypotheses (Example!!!!)
Requirements engineers using the new requirement notation are more
efficient than using activity diagram.
H1: , ,
H0: ,
,
Requirements engineers using the new requirement notation are more
effective than using (e.g.) activity diagram.
H1: , ,
H0: , ,
Requirements engineers accepts the new requirement notation more than
(e.g.) activity diagram.
H1: , ,
H0: , ,
But, e.g.
efficiency time!
more efficient less time!
Be aware that the formalization of hypotheses depends on the
operationalization!
Slide 15
7/28/2019 Exercise 5 Slides 2
16/27
Empirical Model Building &
Methods Formalization of hypotheses (Example!!!!)
Requirements engineers using the new requirement notation are more
efficient than using activity diagram.
Efficiency time!
H1: ,
,
H0: , ,
Requirements engineers using the new requirement notation are more
effective than using (e.g.) activity diagram.
Effectiveness total N defects and N of defects per type! H1: ,
,
H0: , ,
H1: ,
,
,
,
H0: , , , ,
with i:=missing, extraneous, ambiguous, inconsistent, correct and miscellaneous information
How do you formalize the hypothesis(es) concerning acceptance?
Slide 16
7/28/2019 Exercise 5 Slides 2
17/27
Empirical Model Building &
Methods Sampling (Example!!!!)
What sampling and sample type will be used?
Population: Requirement engineers
Sample: Novice, e.g. students of the lecture of requirement
engineering.Why? Avoiding bias because high experience in activity diagram.
x x x x
x x x x
x x x
x x xx x x x
x x
:
How do you select
subjects from thepopulation?
Probability sampling
- random, systematic, stratified
Non probability sampling
- quota, convenient
Be aware that we distinguish between randomization by :
1. Selecting subject from the population
2. Assigning subject to experimental treatments This
determined if the study is an experimental or quasi-
experimental design!!!Slide 17
7/28/2019 Exercise 5 Slides 2
18/27
Empirical Model Building &
Methods Sampling
What sample size is required?
Depends on:
Type of hypotheses: difference, change or causal
Expected effect size
Statistical test to be used, and
Slide 18
7/28/2019 Exercise 5 Slides 2
19/27
Empirical Model Building &
Methods Sampling (Example!!!!)
What characteristics of the subjects should be collected?
Individual attributes? e.g.
: Languages Its assumed that master and bachelor student
have an average English level.
; Education master and bachelor students from different
countries
; Highest education degree, major and the corresponding
University
; Experience
; In general, experience in software development, experience
in requirement elicitation , requirement documentation,
requirement inspection,
; In particular, experience in graphical notations, activity
diagrams,
What about gender, age, nationality, ?
Slide 19
7/28/2019 Exercise 5 Slides 2
20/27
Empirical Model Building &
Methods Sampling (Example!!!!)
What characteristics of the subjects should be collected?
Project attributes?
Type, size
Team structure
Development environment
Application domain
What about organizational attributes ?
Slide 20
7/28/2019 Exercise 5 Slides 2
21/27
Empirical Model Building &
Methods Sampling
What characteristics of the subjects should be collected?
How to Demographic test
1 or more questionnaires with open and closed questions
If information is required for assigning subjects to treatment, ask for
the corresponding information before or during the training.
Asked for remaining information, at the end of the study
When should we use open and closed question? Why?
Slide 21
7/28/2019 Exercise 5 Slides 2
22/27
Empirical Model Building &
Methods Sampling (Example!!!!)
What characteristics of the subjects may be considered as
confounding or control variables?
Confounding variables
e.g. Experience
When do you identify and analyze them? Why?
Design and analysis
What can you do if you identify a potential cofounding variable
during design?
Make it constant
Transform it in an independent variable (factor)
Use parallelization or matching sampling Take the risk
How do you explore cofounding variables during data
analysis?
Control
variables
Slide 22
7/28/2019 Exercise 5 Slides 2
23/27
Empirical Model Building &
Methods Research design (Example!!!!)
Comparison (Quasi-) experiment?
How many factors and groups?
How will you assign subjects to groups?
What is the experimental treatment? number and sequence of
steps, tasks and sessions; time,
What materials are required?
What instruments are required?
Slide 23
7/28/2019 Exercise 5 Slides 2
24/27
Empirical Model Building &
Methods Validity threats
Are observed relationships due to cause-effect
relationship?
Are correct conclusions drawn from (correct) statistical
analysis?
Do employed measures
appropriately reflect
constructs they represent?
Can findings of the study be generalized?
Slide 24
7/28/2019 Exercise 5 Slides 2
25/27
Empirical Model Building &
Methods Validity threats (Example!!!!)
Internal Validity?
Selection
Maturation
History
Instrumentation
Mortality
Testing
External Validity?
Interaction of selection and treatment
Interaction of setting and treatment
Interaction between history and treatment
Slide 25
7/28/2019 Exercise 5 Slides 2
26/27
Empirical Model Building &
Methods Validity threats (Example!!!!)
Conclusion Validity?
Low statistical power
Violated assumptions of statistical tests
Fishing for results and error rate
Reliability of measures
Reliability of treatment implementation
Construct Validity?
Inadequate operation
Mono-operation bias
Mono-method bias
Slide 26
7/28/2019 Exercise 5 Slides 2
27/27
Empirical Model Building &
Methods
What are the next steps?
Slide 27