ICT-DESIGN SCIENCE RESEACRHRaimo Hälinen (2012)

Lahti 11.10.2012

Dimensions of Innovation

The theory creation process

Source: Christensen and Sundahl (2001)

Paradigm

Theory

Classification

Observe, describe & measure phenomena

Dec

lara

tive

proc

ess

Pred

ict

Inductive process

Confirm

Theory is a statement of what causes why, what, and under what circumstances

Anomaly

An Activity Framework for DSR

Source: Venable (2006)

TechnologyInvention/DesignEnhancement creation of a method, product, system, practice or

techniques

Theory Building

Solution space and Problem theoriesUtility theories or

hypothesesTechnology Evaluation

Field studiesExperiments

Action researchSimulation

Problem DiagnosisProblem space understandingProblem causes

and consequences

Information Systems Design theory (ISDT) (Walls et al. (1992)1. Design theories must

deal with goals as contingencies

2. A design theory can never involve pure explanation or prediction

3. Desing theories are prescriptive

4. Design theories are composite theories with encompass kernel theories from natural, social science and mathematics.

Applying design theoryComponents Software development Managing risks

Tarkoitus ja alaPurpose and scope

Tavoitteena on kehittää vaiheittain rakentuvan tietosysteemin kehittämis- ja testauspolitiikka.

Sovellusten kehitystyöhön sisältyvien riskien hallinnan lähestymistavan kehittäminen.

Käytetyt ilmaisutConstructs

Vaiheittaisen kehittämisprosessin virheiden tunnistaminen ja testattavuus

Riskitekijöiden tunnistaminen, riskin aiheuttajien tunnistaminen ja riskin eliminoiminen

Muodon ja toiminnan periaatteetPrinsciples of form and function

Dynaamisten ohjeiden kehittäminen ja järjestelmäintegraation tunnistaminen

Riskien tunnistamisprosessin ja sen aktiviteettien kehittäminen

Artefaktin kehitysluonneArtefact mutability

Tiimioppisiseen liittyvien vaikutusten tunnistaminen kehitystyön erivaihessa.

Sovellusten ja tietosysteemien käytettävyys ja itse-ohjautuvuuden huomioon ottaminen

Testattavat väittämätTestable propositions

Sovelluksen tai tietosysteemien tuotosten (tulosten) arviointi ja simulointi perustuen suunnitteluteorian tosilauseisiin.

Suunniteltu riskien lähestymistapa on sovellettavissa yleisesti sovellusten kehittämistyöhön

Todistava tietämysJustifactory knowledge

Luonnon- ja sosiaalitieteiden antama tietämys, johon suunnittelutyö perustuu

Yleisestä riskienhallinnan johtamismalleista johdetut periaatteet.

ToteutusperiaatteetPrinciples of implementation

Sovelluksen tai tietosysteemin käyttöönottamisen periaatteet

Käyttöönottoon liittyvien riskien hallinta ja osaaminen

Konkreettinen toteutusExpository instantiation

Varsinaisen käyttöönottoprosessin vaiheistus ja ohjeistus

Riskitekijöiden tunnistaminen ja arviointi käyttöönottovaiheessa.

The General Methodology of DSR

Awareness of problem

Suggestion

Development

Evaluation

Conclusion

Proposal

Tentative desing

Artifact

Performance measures

Results

Knowledge Flows Process Steps Outputs

Circumscription

Operation and Goal knowledge

Source: Vaishnavi & Kuehler, (2004,2011)

Rigor CycleGrounding

Additions to KB

Relevance Cycle

RequirementsField testing

Design Science Research Cycle

Application Domain People Orgnizational Systems Technical Systems

Problems and Opportunities

Environment Design Science Research Knowledge Base

Build DesignArtifactsProcesses

Evaluate

FoundationsScientific Theoriesand Methods

ExperienceExpertise

Meta-ArtifactsDesign Products andDesign Processes

Design Cycle

Source: Hevner (2007)

Hevner: ” Design science research is essentially pragmatic in nature.” It emphasizes relevance.It makes clear contribution into the application environment.

Design science research cycle

Theory and models of ISIt-artefact

Outcomes(Reflection)

Hypotheses(Technological

rules)

Observations(Testing)

why, what, how, who,when

What might workfor whom and why

Context-mechanism-outcomes (CMO)

Source: Pawson and Tilley(1997) and Kazi (2003), Carlsson (2006)

Outputs Description

Constructs The conceptual vocabulary of a domain

Models A set of propositions or statement expressing relationships between constructs

Methods A set of steps to perform a task – how-to knowledge

Instantiation

The operational of constructs, models and methods.

Better theories

Artifact construction as analogous to experimental natural science, coupled with reflection and abstraction

Engaged Scholarship

Source: van de Ven (2006)

Engaged Scholarship (ES)...

Researchers

Consultants

Managers

Executives

Users

ES-Dia

mond

model b

y Van

de Ven

New theory

New practical solution

Design research guidelinesGuideline Descriptions

Design as an artifact

Tutkimuksen tavoitteena on sovelluksen, menetelmän tai tietosysteemin suunnittelu. Tarkoituksena ratkaista jokin tietosysteemiin tai sen käyttöön liittyvä ongelma.

Problem relevance Suunnittelu perustuu organisaation saamaan hyötyyn (taloudellinen ja toiminnallinen)

Desing evaluation Suunnittele, miten artifaktin hyöty ja tarkoitus voidaan arvioida suunnitteluprosessin aikana ja sen jälkeen.

Research contribution

Tutkimuksen tavoitteena on tuottaa suunittelutieteen käyttöön uutta tietoa: 1) uusi artifakti, 2) suunnittelutieteen perusteita, 3) arviointiin ja arviointimetodeihin liittyvää tietoa.

Research rigor Tutkimusote perustuu täsämälliseen metodiin, jota käytetään suunnittelussa ja /tai arvioinnissa. (Tutkimuksen toistettavuus).

Design as a search process

Suunnitteluprosessi on etsintää, missä pyritään löytämään parempia tapoja tuottaa artifakteja. (iteratiivinen prosessi)

Communication of research

Tutkimuksen tulokset julkaistaan ja välitetään sekä tutkijoille että käytännön toimijoille (johto, suunnittelijat, käyttäjät).

Source: Hevner et al. (2004)

Research Process ModelResearch Process

Focus Contribution

Problem identification

Relevant research problems

Formulation of research problem and questions

Research Proposal development

Context and scopeResearch stragegy

Answer the questions:why, what, how, who and when

Literature Review

Current state of theory and application

Analysis of focal theory, application and supporting technology

Conceptualization

Concept discovery and potential empirical generalization

Conceptual model, variables and relationships and constraints

Experimentation Design the approach and selecting methods and how to demonstrate

Confirmation of theoretical conjecture, propositions and hypothesesSource: Steenkamp and McCord (2007)

Stages of design processNeeds for

new artifact

Assump-tions

Initiative(First hunch)

Requirements and

assumptions

Structural specifications

Protype

Implementation

Evaluation

Evaluation

Verschuren and Hartog (2005, p.739Plan of evaluation1. Evaluation of

goals2. Evaluation of

means3. Evaluation of

relationship between goals and means.

Types of evaluation4. summative5. formative

Possible Research Entry Points

Design Science Research Process

Source: Peffers et al. (2008), Design Science Methodology Process Model (DSRM)

Identify problem and motivate

Show importance

Define objectives of a solutionWhat would be a better artifact to accomplish?

Design and development

Artifact

Demonstration

Find suitable context

Use artefact to solve problem

Evaluation

Obserce how

effective, efficient Iterate back to desing

Communication

Scholarly publications

Professional publications

Problem-centered initiation

Objective-

centered solution

Design &

development-

centered initiation

Client/ context initiated

Process iteerationNominal process sequence

Inference Theory How to Knowledge Metrics, Analysis Knowledge

Disiplinary knowledge

The design science research process

Design science research approach

(Educational) approach

Defining features of an application or

a process

Defining stakeholders’ preferences

and requirements

Designing research settings

and selecting methods

Data gathering

and analysis

Results and discussion

Feedback and possible contirubutions

An Incremental software process

First version project

Change projects

Requirementsand Features

Desining and programming

Requirements and Features

Implementation Usage of apps

New version

Use Maintenance

Source: modified from Järvinen P. (2011)

Design research and evaluation processes

The searech process

The design process

Ex-ante evaluatio

n

The construction process

The artefact

Ex-post evaluatio

n

Design science

Desing research

Source: Pries-Heje and Baskerville, (2008)

An innovation evaluation research process

Identify evaluation problem and

select evaluation method

Defining evaluation criteria

Develop and specify an evaluation

method

Data gathering and

demonstrating evaluation

ResultsNew methodImprovement

Failure

Time

Knowledge flows

Feedback and knowldge flows

User centred design activities

1. Plan the human centred

process

2. Specify the context of use

3. Specify user and

organizational requirements

4. Produce design

solutions

5. Evaluate design against

user requirements

Feedback

Research proposalsStages Questions of the research and -

paper

1. Title of research proposal What is the main results?

2. Problem area How precise are the claims?

3. Research question How could the outcomes be used?

4. Answer of the research question

What is evidence?

5. Reason for this answer How was the evidence gathered?

6. Research situation How were measurements taken?

7. Key concepts that are investigated

How carefylly are the algoritms and experiments described?

8. The research design Why is the paper (research) trustworthy?

9. The research results will address the research problem

Has the background literature been discussed?What would reproduction of the results involve?

Source: Van de Ven (2006)

Design research elements for product

Design product

Definition Description

Meta-requirement

Goals and main aims of the designed product

Describe the class of problems.

Meta-design A plan for system to meet meta-requirements

Describe the class of artefacts that will meet the meta-requirements.

Kernel theories Theories from natural or social sciences that provides a conceptual bridge between meta-requirements and meta-design

Are applied to describe conceptual connections between meta-requirements and meta-design. IT-artefacts are used IT-related work systems (Alter).

Testable design product hypotheses

Propositions to be examined regarding satisfaction of meta-requirements and meta-design

How developed or planned artefact can be tested to see if hypotheses can be verified.How meta-design satisfies the meta-requirements.

Design research elements for desiging processes

Design process

Definition Description

Design method Procedures to be used for construction of system

Design method defines the procedures, how artefact is constructed.

Kernel theories Theries that support the design method

Applied theories from natural or social sciences that used to develop the design process.

Testable design process hypotheses

Propositions to be examined regarding whether the design method produces an acceptable system

Defined hypotheses, which are used to verify, if the designed process is consistent with meta-desing.

Two research methodsVariance Method (Poole et al.(2000)

The Process Approach

1. The world is made up of fixed entities with varying attributes

1. The world is made up of entities that participate in events

2. Necessary and sufficient causality is requisite for explanation

2. Necessary causality is requisite for explanation

3. Efficient causalisity is the basis of explanation

3. Final and formal causality, supplemented by efficient causality is the basis for explanation

4. The generality of explanation depends on their ability to apply uniformly

4. The generality of explanation depends on their versality

5. The temporal sequince in which independent variables influence the dependent variables is immaterial to the outcome

5. The temporal sequince of event is critical

6. Explanations should emphasize immediate causation.

6. Explanation should incorporate layers of explantion ranging from immediate to distal

7. Attributes have one and only one causal meaning over the course of time.

7. An entity, attribute, or event may change in meaning over time.

Variance method as a research plan Stages Decision Description

1. The research question and perspective

What is the causal conditional proposal or question?

Try to take potential users, your own interest and organization’s goal into account. Reflective viewpoint may help to define research questions.

2. Unit of analysis

What individual or collective properties are being studied ?

Individual, group, organization, technology, it-artefact, process, data. Global, relational or analytical properties.

3. Causal model What is the variance research model?(see e.g. Whetten (2002, Modeling-as-theoryzing: A systematic methodology for theory development, Thousand Oaks, Sage)

List variables, draw arrows, list assumptions and boundary conditions. List alternative factors that may be rival explanations.

4. Experimental design

Is this a randomized, quasi or non-experimental design?

Randomized, quasi-experiment (survey), or non-experiment study (no control-group).

5. Sampe selection and size

What criteria are used to select units, constructs, observations & settings?

construct validity, external validity. Sample size must meet statistical significance.

6. Measurement How to measure variables? systematic and unsystematic measurement biases

7. Data analysis What data gathering technique will be used?

Select technique that fit the research guestions and model.

8. Validity What are the threats to validity of study findings?

Statistical conclusion, internal, construct and external validity.

Van de Ven: ”A variance research model represents the theory as a causal relationship among variables of units are sampled, measured and analyzed in accordance with expremimental design procedures.”

Process method as research planPhases Decision Research ideas and proposals

1. Meaning of process

A category of concepts or developmental sequence?

Are researchers interested in concepts or how a process itself can be described and analyzed (narrative, longitudinal).

2.Theories of process

Examine one or more models?

Select a process model or two models that can be compared.

3.Reflexity Whose viewpoint is featured?

Researchers, practitioner’s or user’s viewpoint to consider research outcomes.

4.Mode of inquiry

Deductive, inductive or retrodeductive?

Is the research based on the model of reality?Is the research based on how it-artefact process is going on at the real time?

5.Observational method

Real-time or historical observations?

Is a research process exploring active it-artefact development process?Is a research process observing how it-artefact process was carried out?

6.Source of change

Age, cohort or transient sources?

Try to develop parallel, synchronic research design.

7.Sample diversity

Homogenous or hetereogenous?

Try to gather different types of events.

8.Sample size

Number of events and cases?

Try to find out more than one case or more events inside the one case.

9.Process research designs

What data analysis methods to use?

Match data analysis methods to number of cases and events.

Ethics methodTask Description

Objective setting

Traditionally, system objectives have been set by top management and computer technologists. But increasingly other groups are asking or demanding to be involved in this process, in particular, line management, white and blue collar workers and the trade unions.

Adaptation The process of adaptation is concerned with moving from one kind of organizational structure and state to another and the means by which this change is assisted to take place smoothly and successfully. It is what normally happens during the implementation phase of a new system. Rapid adaptation does not easily come about of its own accord and there is a need for philosophies, facilities and strategies to assist the process.

IntegrationTechnologyPeopleTasks

Integration is the action taken, once the system has been designed and is being implemented, to ensure a new situation reaches a state of equilibrium.

Stabilization Once integration has been achieved it has to be maintained. To avoid "one man's job enrichment becoming another man's job impoverishment" imaginative solutions or even compromises may be required. The maintenance of a state of equilibrium into the future requires processes for socializing and educating new group members.

Source: Mumford (1983)

A Representational framework for generic IS artifact

Category Structure Behaviour Motivation Instantiation

Analytical Salient properties and features

Process supported

Major generic business/ organizational motivations

Mock-up prototypes

Synthetic Overall logical organization

The dynamic behaviour of IS; its working principles

Motivations for the type of IS solution as related to requirements

Working prototypes that illustrated the concept

Technological Refined structural design for the type of IS

Refined methods, algoritms, interactions employed by IS

Objectives for the design

Generig packages, frameworks, shells.

Implementation

Architecture design of implemented systems

Dynamic aspects of implemented systems

Specific business motivation

Implemented functionin systems

Source: Vahidov (2006)

The role of researchers and practitioners

Phases the researcher the practitioner

Agreement develop an agreement agree agreement

Start of research process

create a research plan agree research plan

During the real process

Develop a new artifact based on requirements and needed features of the artifact.

Participate in development process and agree the requirements and features.

Demonstration Carry out laboratory test for artifact simulating a reality.

Carry out simulation and lab-test for new artifact at the real environment.

Evaluation Develop evaluation plan and define needed criteria and combine results to the practitioner’s evaluation results.

Carry out evaluation process and give feedback to the researcher.

Conclusion Publish what is learned during the research process.

Discuss with the researcher concerning the research process.Source: Hevner et al. (2004), March and Smith (1995), Järvinen,(2012)

Selecting Design research approachResearch item

Questions Design science approach

The artefact

What is the role of the artefact?Is design of the artefact or improving organizational practice the primary goal of the intervetion?

The artefact is central of the research.Collaborators will focus on the artefacts, but success may need some changes to process and organizational practice

The process and research cycles

How the research process problem going to determined and agreed?Is a predetermined cycle of activity to be followed?Is softwarte development method a necessary part of the activities?

Problem must be identified and design criteria may be used to establish potential solutions.

The focus of evaluation

Can evaluation be a by-product of the research cycle, or must explicit evaluation activites be used?Does user acceptance testing and similar activities have a role to play?Does the artefact require the generation of use cases and some level of validation? (verification?)

Evaluation of the designed artefacts is crucial.Evaluation steps are included to the approach. Software testing is vital andis part of the research process.Design as a serch process suggest that use cases and testing are included to the research process.

Selecting Design research approach ...

Research item

Questions Design science approach

The role of knowledge

Is actionable knowledge a realistic goal of the research?To what extent is informing theory and objective?

To some extent knowledge emerges from the design.Explicit theory is less prevalent in DS. Design theory is employed as a framework and a reference point for outcomes.

The role of learning

What expectations are there with regards to learning?Will learning emerge from the design of the artefact or the actions undertaken by the researcher and collaborators?

Expectations for learning will exist.Expectations of learning from the artefact will perdominate.Researchers might be surprised by what they learn.

Source: Papas et al. (2012)

Design research evaluation framework

Semiotic level

Evaluation criteriaHelfert and Donnellan

Pragmatic Relevance, usability, completeness, timeliness, actuality, efficiency.

Semantic Precise definitions and terminology, easy to understand, interpreditability, accuracy (free-of error), consistent content.

Syntax Consistent and adequate syntax, syntactical correctness, consistent representation, accessability.

Source: Helfert M. and Donnellan B. (2012)

Ex-ante Ex-post

Naturalistic Design process/product

Design process / product

Artificial Design process / product

Design process / product

Source: Pries-Heje and Baskerville (2008)

Method Evaluation process

Observantional Case or field study

Analytical Static analysis, Architecture analysis,Optimization, Dynamic

Experimental Controlled experiment

Testing Functional (black box)Structural (white box)

Descriptive Informed argumentScenarious

Souce: Hevner et al. (2004)

Evaluation of Information systems

Evaluation type

Information system in use

Invormation system as such

Goal-based

Data source

Evaluator’s roleParticipators

Has the IT-system fulfilled the desired goals?IT-system, goal definitions, requirements, , interaction between users and the it-system.DeductiveEvaluator and users, managers

Has the IT-system fulfilled the desired goals? What is the contribution?IT-system, goal definitions, requirements.

DeductiveEvaluator expert

Goal-free

Data sourceEvaluator’s roleParticipators

Try to gain a deeper and broader understanding of the IT-systems.IT-system, description of IT-system.InductiveEvaluator expert

Try to gain a broader understanding of the IT-systems.IT-system, description of IT-system.InductiveEvaluator expert

Criteria-based

Data sourceEvaluator’s roleParticipators

Try to gain a deeper and broader understanding of the IT-system.IT-system, observation, users’ perceptions.InductiveEvaluator expert and user, manager

Try to evaluate quality of the IT-system. IT-system, description of IT-system, defined evaluation criteria.DeductiveEvaluator expert

Source: modified from Cronholm and Goldkuhl (2003)

Evaluation criteria for research papersEvaluation object

Relevant evalution questions

1. Design of an artefact

Produce the research an it-artefact, that includes a construct, a model or an instantiation.

2. Problem relevance

Is the research problem and objective of the it-artefact important? Is the it-artefact or solution relevant business problem?

3. Design evaluation

Could the results be verified?Are the evaluation method rigorous and well-defined?Is the utility, quality and efficacy rigorously demonstrated?

4. Research contribution

Is there a contribution?Is it significant?Is the contribution timely interest?

5. Research rigor

Are the results correct?Are the all technical detail correct? Are they sensible?

6. Design as a search process

Are the researchers utilized available means?Do the results satisfy laws on the problem environment?

7. Communication of research

Are the appropriate conclusions drawn from the results?Can the paper be understand?Is it clearly written?Is the results presented effectively both to the technology-oriented and management oriented audiences?

References Carlsson S.A. (2006), Towards and Information Systems Design Research Framework: A Critical Realist

Perspective, Destrist 2006, February 24-25 2006, Claremont CA. Crossan M.M. and M. Apaydin (2010), A Multi-Dimensional Framework of Organizational Innovation: A

Systematic Review of the Literature, Journal of Management Studies 47, No 6, 1154-1191. Hevner A.R. (2007), Three Cycle View of Design Science Research, Scandinavian Journal of Information Systems, Vol.

19, No.2, pp. 87-92. Hevner A.R., March S.T., Park J, and Ram S. (2004) Design science in information systems research, MIS Quarterly, Vol.

28, Issue 1, March 2004. Järvinen A & P. (2011) Tutkimustyön metodeista, Opinpajan kirja, Tampere. Järvinen P. (2012), On boundaries between field experiment, action research and design research, University of

Tampere, Shcool of Infomation Sciences, Reports in Information Sciences 14, Tampere. Iivari J. (2005), Information Systems as a design science, Information Systems Development: Adnvances in Theory,

Practice and Education, Edited bgy O. Vasilegas et atl. Springer. Papas N., O´Keefe R.M. and Seltsikas P. (2012), The action research vs design research science depate: reflections

from an intervention in eGovernment, European Journal of Information Systems, Vol. 21, pp- 147-159. Peffers K., Tuunanen T., Gengler C.E., Rossi M., Hui W., Virtanen V. and Bragge J.(2006), The Design Science Research

Process: A model for producing and presenting information systems research, Destrist 2006, February 24-25 2006, Claremont, CA.

Vahidov R. (2006), Design Researcher’s IS Artifact: a Representational Framework, Destrit 2006, February 24-25 2006 Clamenont CA (CGU2006).

Vaishnavi V. and Kuechler W. (2004), Design Science Research in Information Systems, http://desrist.org/design-research-in-information-systems/, last updated September 2011

van Aken Joan E. (2001), Management Research based on the paradigm of the Design Sciences; The Quest for tested and grounded technological rules.

Van de Ven (2007), Engaged Scolarship: A Guide for Organizational and Social Research, Oxword University Press Inc., New York

Venable J.R. (2006), The Role of Theory and Theorising in Design Science Research, Destrist 2006, Proceeding/2A_1pdf.

Verschuren P. and Hartog R. (2005), Evaluation in Design-Oriented Research, Quality & Quantitity, Vol. 39, pp. 733-762.

Zobel J. (2004), Writing for computer science second edition, Springer,-Verlag, London Limited.