Upload
others
View
12
Download
1
Embed Size (px)
Citation preview
Design Science Methodology
Roel WieringaRoel WieringaUniversity of TwenteThe Netherlands
2nd September 2010 Deutsche Telekom 1
What is design science?What is design science?
• Design science is technical science, engineering science• It validates proposed artefacts
– New jet propulsion technology– New information risk assessment method
• And studies implemented artefacts• And studies implemented artefacts – Steam machines– Smallpox vaccinationS a po acc at o– IS impact studies
• Natural science studies entities not constructed by people
2nd September 2010 Deutsche Telekom 2
OutlineOutline
1. A framework for design science2. A methodology for design science
2nd September 2010 Deutsche Telekom 3
Economy“Artefactbase”
Decisions about scarce
ISdesign
Goals,Budget Artifacts Goals,
BudgetScientific
knowledgeInstruments,
resources
design science
Artifacts
Budget Budget knowledge
Technology Science
subjects ofstudy
Goals,Budgets
Researchquestion
investigation
Practicalproblemsolving
Technology Science
Scientificknowledge
Scientificknowledge
Problem solving
knowledgeFlow of money,data material;
Knowledgebase
data, material;no control flow
2nd September 2010 Deutsche Telekom 4
Economy“Artefactbase”
Decisions about scarce
ISdesign
Goals,Budget Artifacts Goals,
BudgetScientific
knowledgeInstruments,
resources
design science
Artifacts
Budget Budget knowledge
Technology ScienceHevner
subjects ofstudy
Goals,Budgets
Researchquestion
investigation
Practicalproblemsolving
Technology ScienceHevner,March,Park,Ram Scientific
knowledge
Ram(2004)
Scientificknowledge
Problem solving
knowledgeFlow of money,data material;
Knowledgebase
data, material;no control flow
2nd September 2010 Deutsche Telekom 5
Iteration over science and technology• Practical problem: Confidentiality risk assessment in outsourcing
• Artefact proposal: New risk assessment methodTech
Science
• Validation question: Does it work?
• Validation research designs:Science
– Opinion poll– Lab experiment (cases solved by students)– Field experiment (cases solved by professionals)
Action research (researcher uses method in practice and then reflects on– Action research (researcher uses method in practice and then reflects on experience)
– …
B k h i l bl Diff bl d di ?• Back to the practical problem: Different problem understanding?
• Back to the artefact proposal: Improvemenents?Science
• Redo the validation
• ...
Tech
Science
2nd September 2010 Deutsche Telekom 6
Practical problem relevancePractical problem relevanceG l t b hi d• Goal to be achieved– Normal problems: Goal is stated by stakeholders; limited to what they can imagineto what they can imagine
• Achieve economic goal for the first time• Repair• Improve• Improve
– Radical problems: Goal not stated by stakeholders; limited b i i ti f i / i ti t/ tby imagination of engineer/scientist/entrepeneur
• Circumventing predicted performance limits• Meeting predicted demand
• Relevance is context‐dependent: people, technology, time, place, …
2nd September 2010 Deutsche Telekom 7
p ,
Research question relevanceResearch question relevance
Questions that come up in an engineering context:– Artifact & Context ~> Effect?
• Will it work?• Why does it fail?• Why does it work?• Why does it work?
– Effect satisfies stakeholder goals?• Will it satisfy goals?
Validation and evaluation questions
• Will it satisfy goals?• Why does it fail to satisfy goals?• Why does it satisfy goals?
– How to measure this?– How to compute this?
Conceptual questions
2nd September 2010 Deutsche Telekom 8
OutlineOutline
1. A framework for design science– The framework– Sources of relevance
2. A methodology for design science
2nd September 2010 Deutsche Telekom 9
Practical problems versus knowledge problems
i l bl• Practical problem– Difference between current state of the world and what a stakeholder would like it to be
• To solve it we need to change the world
• Knowledge problem– Difference between what current stakeholder knows and what the stakeholder wants to know
• To solve it stakeholder needs to change their knowledge of the world
2nd September 2010 Deutsche Telekom 10
What kind of problem?What kind of problem?• What is the architecture of the communication• What is the architecture of the communication infrastructure between A and B?– K Problem: infrastructure exists, stakeholder does not f ,know what its architecture is
Wh t i hit t f• What is an architecture of …– P Problem: A blueprint must be made
Misleading!• Design an architecture for …
– P Problem: A blueprint must be made
Misleading!– P Problem: A blueprint must be made
2nd September 2010 Deutsche Telekom 11
Heuristics• Practical problems
– Are solved by changing • Knowledge questions
– Are solved by changing the the state of the world
– Solution criterion is ili
knowledge of stakeholders.– Solution criterion is truth
utility• Problem‐dependent: stakeholders and goals
• Problem‐independent: no stakeholders
• One solution; butstakeholders and goals• Many solutions; but trade‐offs
One solution; but approximations
– Technology – Science
Doing ThinkingDoingChanging the worldFuture‐oriented
ThinkingChanging our mindPast‐oriented
2nd September 2010 Deutsche Telekom 12
OutlineOutline
1. A framework for design science– The framework– Sources of relevance
2. A methodology for design scienceTh i i l– The engineering cycle
– The research cycle
2nd September 2010 Deutsche Telekom 13
Solving practical problems rationally• The engineering cycle (a.k.a. The regulative cycle)
– problem investigation •K Stakeholders?•K Goals, criteria?•K Phenomena diagnosis?
– treatment design
•K Phenomena, diagnosis?
•P Specify a treatmentA k a sol tion
– design validation •K Will it work?K Will it ti f it i ?
A.k.a. solution
– implementation
•K Will it satisfy criteria?•K Trade‐offs?•K Sensitivity?
Design = deciding what to doSpecification = documenting that decision
implementation
i l t ti l ti K D it k?
Transfer to practice
2nd September 2010 Deutsche Telekom 14
– implementation evaluation •K Does it work?•K Does it satisfy criteria?
The engineering cycle elaboratedImplementation evaluation = Problem investigationProblem investigation
Implementation Stakeholders? Normal or radical goals?C it i ?Criteria?
Problematic phenomena?Impacts?Diagnosis?
Treatment designDesign validation
Diagnosis?Evaluation?
Treatment design
Expected impact: Context & Treatment → Effects?Expl n ti n?
Normal goals:Satisfy criteriaRepair failuresExplanation?
Evaluation: Effects satisfy goals?Trade-offs for different Solutions?Sensitivity for different Contexts?
Repair failuresImprove performance
Radical goals:Circumvent future failure
2nd September 2010 Deutsche Telekom 15
Sensitivity for different Contexts? Circumvent future failureSatisfy future demand
Mutual nestingMutual nesting• Doing i i l h i ti l bl f i l ki d• Doing empirical research is a practical problem of a special kind:
– Do something to acquire the knowledge!
• Special kind engineering cycle.– What is the research problem?– How to answer it (research design)?( g )– Is the research design valid?– Do the research– Evaluate the results
• Mutual nestingIn design science the top level cycle is to serve other stakeholders’ goals– In design science, the top level cycle is to serve other stakeholders goals
– In pure science, the top level cycle is to server the researchers’ knowledge goals
2nd September 2010 Deutsche Telekom 16
OutlineOutline
1. A framework for design science– The framework– Sources of relevance
2. A methodology for design scienceTh i i l– The engineering cycle
– The research cycle
2nd September 2010 Deutsche Telekom 17
Research cycleAnalysis of resultsAnalysis of results•Observations•Explanation•Generalization
Research probleminvestigation
•Generalization•Practical implications
investigation•Research goal•Problem owner•Unit of study
Researchexecution
•Unit of study•Research questions•Conceptual model•Current knowledgeCurrent knowledge
Research designU it f d t ll ti
Design validationC l i lidit •Unit of data collection
•Environment of data collection•Interaction with unit of data collection•Measurement instruments
•Conclusion validity•Construct validity•Internal validity•External validity
2nd September 2010 Deutsche Telekom 18
•Measurement instruments•External validity
Research cycleAnalysis of resultsAnalysis of results•Observations•Explanation•Generalization
Research probleminvestigation
•Generalization•Practical implications
investigation•Research goal•Problem owner•Unit of study
Researchexecution
•Unit of study•Research questions•Conceptual model•Current knowledgeCurrent knowledge
Research designU it f d t ll ti
Design validationC l i lidit •Unit of data collection
•Environment of data collection•Interaction with unit of data collection•Measurement instruments
•Conclusion validity•Construct validity•Internal validity•External validity
2nd September 2010 Deutsche Telekom 19
•Measurement instruments•External validity
Example 1Example 1
• Developing and validating a method for estimating effort of developing process‐aware g p g pinformation systems at Daimler
• See next slide• See next slide– Mutschler, B. (2008) Modeling and simulating causal dependencies on process‐
aware information systems from a cost perspective. PhD thesis, Univ. of Twente. ISBN 978‐90‐365‐2578‐7
2nd September 2010 Deutsche Telekom 20
How can we improve financial evaluation of process‐aware information systems?
bl ProblemProblemK Current problemswith evaluation?
K Current approaches to
St, Ph, Go, Cr
K Build taxonomy
Problemdecomposition
Problemsequence
financial evaluation? of approaches
K Classify approaches
K Criteria for taxonomies?K Collect taxonomies
K EvaluateK Evaluate them K Validate classification P Design new one
K Validate against criteria
K Evaluate them
P Develop new approach:Causal loop models
K Make causal loop modelsof cost factors of PAIS
K Collect modeling guidelines
P Acquire modelingtools
K V lid t it K Check designK Validate it K Check designargument P Experiment to test one model
P Pilot study using another model
2nd September 2010 Deutsche Telekom 21
P Pilot study using another modelK Reflection: lessons learned
Validation researchValidation research
• Characteristic of validation research:– The treatment has not been transferred to practice yet
– How to investigate something that does not exist ?How to investigate something that does not exist ?• Mathematical analysis, if possible• Modeling and simulation• Modeling and simulation
2nd September 2010 Deutsche Telekom 22
Technical validation researchProblem investigation
Treatment design
•Research problem investigationContext & Treatment produce Effects?Effects satisfy goals?
Treatment design
Treatment validation
Trade‐offs?Sensitivity?
•Research design
Treatment implementation
Implementation evaluation
•Research design validationWill this answer our questions?
•Research executionA l i d l i f lImplementation evaluation •Analysis and evaluation of results
Observations?Answers to research questions?Explanation?Explanation?Generalization?Practical impact?
Practical problem Research problem
2nd September 2010 Deutsche Telekom 23
Technical action researchProblem investigation
Problem classTreatment design
•Research problem investigationContext & Treatment produce Effects?Effects satisfy goals?
Treatment design
Treatment validation
Trade‐offs?Sensitivity?
•Research designProblem investigation
Client’s problemTreatment implementation
Implementation evaluation
•Research design validationWill this answer our questions?
•Research executionA l i d l i f l
Client s problemTreatment design
Customize treatmentTreatment validationImplementation evaluation •Analysis and evaluation of results
Observations?Answers to research questions?Explanation?
Check with clientTreatment implementation
Explanation?Generalization?Practical impact?
Implementation evaluationClient’s goals satisfied?
Practical problem Research problem Practical problem
2nd September 2010 Deutsche Telekom 24
Example 2Example 2
• Developing a confidentiality risk assessment method when IT is outsourced
• Researcher used method herself to help a client do this assessmentclient do this assessment
• Next slide– (Morali, A. and Wieringa, R.J. (2010) Risk‐Based Confidentiality Requirements
Specification for Outsourced IT Systems. In: Proceedings of the 18th IEEE International Requirements Engineering Conference (RE 2010), 27 Sept ‐ 1 Oct 2010, Sydney, Australia. IEEE Computer Society. )
2nd September 2010 Deutsche Telekom 25
2nd September 2010 Deutsche Telekom 26
Validation methods in design scienceCond. of pract.
Cntrl of cntxt
Unit of data collect.
Example User GoalsScaling up toConditions of practice
Illustration no yes model small designer illustration
Opinion imagined yes model any stakeh. support
Lab demo no yes model realistic designer knowledge
Lab expt. no yes model ! artificial subjects knowledge
Benchmark no yes model standard designer knowledge
Field trial yes yes model realistic designer knowledge
Field experiment
yes yes model realistic stakeh. knowledge
Action case yes no model real designer knowledge andAction case yes no model real designer knowledge and change
Pilot project yes no model realistic stakeh. knowledge and change
2nd September 2010 Deutsche Telekom 27
changeCase study yes no model real stakeh. knowledge and
change
Levels of knowledgeIdealized
N=∞ Universal theories Basic science
Idealizedconditions
k d h i
Di ti th i T t t th i Design science:
Background theories
N=kDiagnostic theories Treatment theories Design science:
Problem classes
Application Generali ation
Context Goal oriented
Application Generalization
N=1
Context X ――→ Effects, should satisfy Goals
Treatment
Goal‐orientedRequirementsEngineering
Trade-offs (effectiveness of other treatments?)Sensitivity (robust under future scenarios?)Conditions
of practice
2nd September 2010 Deutsche Telekom 28
Take home1. In design science research, we iterate over technology (practical
problem solving) and research (knowledge question answering)
2. Much of design science research is validation research– Treatment & Context produces effects?– Treatment & Context produces effects?– Effects satisfy criteria?– Trade‐offs?
S iti it ?– Sensitivity?
3. Validation research must simulate the treatment in practice– Opinion poll of practitioners– Experimenting with models– Action case
2nd September 2010 Deutsche Telekom 29
DiscussionDiscussion
2nd September 2010 Deutsche Telekom 30