PhD pre-defense September 2015

1/43www.janclaes.info

INTRODUCTION VISUALIZATION EXPLORATION THEORIZATION CONCLUSION

Jan ClaesSupervisors UGent : Geert Poels & Frederik Gailly

Supervisors TU/e : Paul Grefen & Irene Vanderfeesten

Investigating the process of process modeling and its relation to modeling quality

The Role of Structure Serialization



Structure of the PhD

CHAPTER 5CONCLUSION

CHAPTER 4THEORIZATION

CHAPTER 3EXPLORATION

CHAPTER 2VISUALIZATION

CHAPTER 1INTRODUCTION



CHAPTER 1 – INTRODUCTIONResearch objectives



Context

Business Process Management

Conceptual ModelingPhD



Context

Article availableOrder received

Check availability

No

Yes

late deliveryundeliverable

Payment received

Procurement

Remove articleFrom catalogue

Inform customer

Financial settlement

Ship article

Customer informed

Article removed

Inform customer

+

+

(From BPMN Quick Guide, OMG, 2015)



Definitions Definition 1: Business process“A business process consists of a set of activities that are performed in coordination in an organizational and technical environment. These activities jointly realize a business goal.” (Weske, 2007, p. 5)

Definition 2: Business process model“A business process model is a mostly graphical representation that documents the different steps that are or that have to be performed in the execution of a particular business process under study, together with their execution constraints such as the allowed sequence or the potential responsible actors for these steps.”

Definition 3: Process of process modeling“the sequence of steps a modeler performs in order to translate his mental image of the process into a formal, explicit and mostly graphical process specification: the process model.”



Study 1: visualization

Research Objective 1Build knowledge about

how people create models

Overall objectiveCuriosity-driven

Build knowledge about PPM


relation with quality


structured modeling

Study 3: theorizationStudy 2: exploration

Research Objectives



CHAPTER 2 – VISUALIZATIONPPMChart



Data collection

Pnina SofferMatthias Weidlich

Barbara WeberJakob Pinggera

Stefan ZugalJan Mendling

Hajo ReijersIrene Vanderfeesten

Dirk Fahland

Observational data

Cheetah Experimental

Platform

‘Experiment’ design



CREATE_ACTIVITYCREATE_START_EVENTCREATE_END_EVENTCREATE_ANDCREATE_XORCREATE_EDGE

Data collection

MOVE_ACTIVITYMOVE_START_EVENTMOVE_END_EVENTMOVE_ANDMOVE_XOR

DELETE_ACTIVITYDELETE_START_EVENTDELETE-END_EVENTDELETE_ANDDELETE_XORDELETE_EDGE

NAME_ACTIVITYRENAME_ACTIVITYNAME_EDGERENAME_EDGE



Study 1 – Visualization

(From Pinggera et al., 2014)

Not enough detail



Study 1 – VisualizationPoor visual expressivenessPoor perceptual discriminabilityPoor graphic economyPoor semiotic clarityPoor semantic transparencyPoor complexity managementPoor cognitive integration



Study 1 – VisualizationPPMChart

CREATE_ACTIVITY CREATE_START_EVENT CREATE_END_EVENT CREATE_AND CREATE_XOR CREATE_EDGE MOVE_ACTIVITY MOVE_START_EVENT MOVE_END_EVENT MOVE_AND MOVE_XOR DELETE_ACTIVITYDELETE_START_EVENT DELETE-END_EVENT DELETE_AND DELETE_XOR DELETE_EDGE NAME_ACTIVITY RENAME_ACTIVITY NAME_EDGE RENAME_EDGE

Start event Edge Activity

Gateway

Edge

Activity

Edge

Edge

Activity

Edge Gateway

Edge

7298

9

32

14

30

31

10

3356

34

time

mod

el e

lem

ents



FastmodelingSlow

modelingInitialdelayMany

pauzesFewelementsMany

elements No(separate)lay-outing

Quicklay-outingDedicated

lay-outingphase

Continuouslay-outingUnpaired

eventcreation

Pairedevent

creationNo pauzes

SerializationPaired

gatewaycreation

Delayededge

creationChunked

modeling

Study 1 – Visualization

Based on dataset of 357 unique modeling executions



Study 1 – VisualizationEvaluation

Sample of intended users (6 academic researchers) Five extreme examples in PPMChart or Dotted Chart Observe and measure amount, quality, and timing of

insights gained through the visualization Observe and ask about perceived usefulness

Results Perceived as useful More cognitive effective than Dotted Chart



CHAPTER 3 – EXPLORATIONRelation with quality



Study 2 – Exploration


Fastmodeling

Slowmodeling

Initialdelay

Manypauzes

Fewelements

Manyelements

No(separate)lay-outing

Quicklay-outing

Dedicatedlay-outing

phase

Continuouslay-outing

Unpairedevent

creation

Pairedevent

creation

No pauzesSerializationPairedgatewaycreation

Delayededge

creation

Chunkedmodeling



Study 2 – ExplorationFast

modelingSlow

modelingInitialdelay

Manypauzes

Fewelements

Manyelements


Quicklay-outing

Dedicatedlay-outing

phase


Unpairedevent

creation

Pairedevent

creation

No pauzes Pairedgatewaycreation

Delayededge

creation

Chunkedmodeling


Serialization




modelingSlow


Manypauzes

Fewelements

Manyelements


Quicklay-outingContinuous

lay-outingUnpaired

eventcreation

Pairedevent

creation


Delayededge

creation

Chunkedmodeling


Dedicatedlay-outing

phase




modelingSlow


Manypauzes

Fewelements

Manyelements


Quicklay-outing

Dedicatedlay-outing

phase

Unpairedevent

creation

Pairedevent

creation


Delayededge

creation

Chunkedmodeling





Study 2 – ExplorationSlow


Manypauzes

Fewelements

Manyelements


Quicklay-outing

Dedicatedlay-outing

phase


Unpairedevent

creation

Pairedevent

creation


Delayededge

creation

Chunkedmodeling


Fastmodeling





Manypauzes

Fewelements

Manyelements


Quicklay-outing

Dedicatedlay-outing

phase


Unpairedevent

creation

Pairedevent

creation


Delayededge

creation

Chunkedmodeling

Slowmodeling




Study 2 – ExplorationStructuredness Movement Speed


Fastmodeling

Slowmodeling

Quicklay-outing

Dedicatedlay-outing

phase


Serialization

Chunkedmodeling

CONJECTURE 1

CONJECTURE 2

CONJECTURE 3



Study 2 – ExplorationConjecture 1: Structured modeling

results in understandable models

Conjecture 2: A high number of move operations results in less understandable models

Conjecture 3: Slow modeling results in less understandable models



Structuredness• MaxSimulBlock• PercNumBlockAsAWhole

Speed• TotTime• TotCreateTime

Movement• AvgMoveOnMovedElements• PercNumElementsWithMoves


Model quality• Perspicuity

a model that is unambiguously interpretable and can be made sound with only small adaptations based on minimal assumptions on the modeler’s intentions with the model

CONJ

ECTU

RE 1

CONJECTURE 2

CONJECTURE 3




T-testt=-2,231 (p=0,028)

T-testt=2,199 (p=0,030)


CONJ

ECTU

RE 1




T-testt=-1,984 (p=0,049)

T-testt=0,457 (p=0,648)


CONJ

ECTU

RE 2




T-testt=-2,183 (p=0,031)

T-testt=2,505 (p=0,014)


CONJ

ECTU

RE 3



CHAPTER 4 – THEORISATIONStructured Process Modeling Theory

(SPMT)



Study 3 – Theorization


T-testt=-2,231

(p=0,028)

T-testt=2,199

(p=0,030)

CONJECTURE 1structuredness

T-testt=-1,984

(p=0,049)

T-testt=0,457

(p=0,648)

T-testt=-2,183

(p=0,031)

T-testt=2,505

(p=0,014)

CONJECTURE 2movement

CONJECTURE 3speed

WHY?




Combined

Flow-oriented Aspect-oriented

Undirected




Study 3 – Theorization Observation 1. Almost all modelers paused frequently

during the modeling process Observation 2. A large group can be categorized as

“flow-oriented process modeling” Observation 3. A smaller group can be categorized as

“aspect-oriented process modeling” Observation 4. Another large group used a combination

of both former styles Observation 5. Another small group can be categorized

as “undirected process modeling” Observation 6. The “undirected” sessions lasted longer

than the other approaches




Study 3 – Theorization Impression 1. Modelers need serialization of the modeling

process to deal with its complexity Impression 2. Structured serializing of the modeling

process helps avoiding ‘mistakes’

Impression 3. Structured serializing does not support every

modeler to avoid ‘mistakes’ to the same extent




Study 3 – TheorizationCognitive Load Theory

Working memory capacity is limited Working memory overload causes decrease in

• Effectiveness (i.e., more mistakes)• Efficiency (i.e., more time and effort)• Learning

Cognitive Fit Theory Load is lower when there is a fit

• Between representation, tool or strategy on the one hand• And task or modeler on the other hand




A B A determines BA B The more A, the more B+ A B The more A, the less B– A B A translates into B

learning style

degree of serialization

adopted serialization style

field-dependency need for structure

– +

course of intrinsic cognitive load for process modeling phases

course of intrinsic cognitive load for aggregation phases

course of cognitive overload

course of intrinsic cognitive load for strategy building phases

+ + +

serialization style fitstructuredness of serialization– – – –

1 2 3




Evaluation of utilityNovelty

(uses existing theories in fundamental new way)Parsimony

(11 constructs, 15 associations)Consistency

(can explain additional observations)Plausibility

(accurate and profound explanation)Credibility

(building blocks are established theories)Transferability

(problem solving in general)

Consistency based on dataset of 143 unique modeling executions



CHAPTER 5 – CONCLUSIONSummary & Future work



FUTURE WORK

Knowledge gaps

Knowledge contribution Research instrument

Knowledge gap 4How to change one’s modeling strategy?

Knowledge gap 3How should one model

in a specific context?

Study 1: visualizationContributions

A – PPMChartB – 22 patternsC – 13 observations

Study 2: explorationContributionsD – 8 patternsE – 3 conjecturesF – 1 metric

Knowledge gap 1How do people

currently model?

Knowledge gap 2Which strategy is

intrinsically better?

A BC

D E F

H HG I

Study 3: theorizationContributions

G – 5 stylesH1 – 6 observationsH2 – 3 impressionsI – SPMT

I



Future Work

Use developed knowledge toDevelop prescriptive theory

Cognitive profile best modeling approachDevelop method (SPMM)

(1) Determine (2) learn (3) apply best approachDevelop tool support

Cognitive tests Interactive digital approach tutorial



Reflection – Deduction, induction, abduction

Rule Case

Result

Deduction

Reveals effectResults in certainty

InductionCase Result

Rule

Reveals mechanismResults in probability

AbductionResult Case

Rule

Reveals causeResults in possibility

Process modeling as application domain> ordering of data facilitates pattern recognition <



Reflection – Student observationAssumption: overload has same causes

for students and practitionersAssumption: overload has same consequences

for students and practitioners

STUDENTS…… are representative participants

… don’t suffer from Expert-Reversal Effect… form a homogeneous group

… provide heterogeneous set of observations… reach point of overload faster than practitioners



Reflection – Empirical behavioral researchProcess modeling = complex and dynamic taskIdentify/measure/control confounding variables Ignore/assume constant/assume minimal effect

Include multiple variables in modelMix techniques (de/in/abduction, quanti/qualitative)

Open-world assumption> conclusions are incomplete unless proven otherwise <

> no conclusions from insignificant results <> more accurate, but slower progress <



Thanks for your attention! Do you have any questions?

Jan [email protected]

http://www.janclaes.infoTwitter: @janclaesbelgium