Alignment-based Precision CheckingA. Adriansyah1, J. Munoz Gamma2,

J. Carmona2, B.F. van Dongen1,

W.M.P. van der Aalst1

Tallinn, 3 September 2012

1) Department of Mathematics and Computer Science, Eindhoven University of Technology, The Netherlands

2) Software Department, Universitat Politècnica de Catalunya, Spain

2

Model needs to be precise

Event Log

Process Models

a da b

bc

end

c

a b

d

end

aabdabdacd

aabdc

ad

Unfitting

How to measure precision for unfitting event logs?

d

3

Overview

aabbdadabd

a d…

Unfitting Log

Process Model

Alignment Automaton

Experiments

aabdaabdacd…

Fitting TracesAlignment

Precision Metrics

4

Optimal alignment: least #deviations

a da b

bc

end

Trace adab

aAlignment

d ba a b

ad

Process Model

Fitting trace, closest to the original

Deviation

Deviation

Log Moves

Model Moves

aabd Reference: W.M.P. van der Aalst, A. Adriansyah, B.F. van Dongen (2012). Replaying history on process models for conformance checking and performance analysis. WIREs Data Mining and Knowledge Discovery, 2(2), 182-192.

5

Optimal alignments can be many

a da b

bc

end

Trace ad

Alignment 1 Alignment 2

Process Model

baa d

d caa d

d

Both alignments are optimal

abd or acd

6

Overview

aabbdadabd

a d…

Unfitting Log

Process Model

Alignment Automaton

Experiments Precision Metrics

aabdaabdacd…

Fitting TracesAlignment

7

a

b

b

a

d

24 4

2 2 2

d 2

aabdabdabd or acdaabd

1-Alignment automaton (Prefix-based)

Fitting Traces

a da b

bc

end

Process Model

1-Align Automaton

aabdabdabdaabd

8

b d

b

a

ad

4 4

2 2 2

1-Alignment automaton (Prefix-based)

0c

Fitting Traces

a da b

bc

end

Process Model

1-Align Automaton

aabdabdabdaabd

Imprecision

2 2

…

…

9

a22 3

db1.51.5 11

b1a

d

3

All-Alignment Automaton (Prefix-based)

c

Fitting Traces

a da b

bc

end

Process Model

All-Alignments Automaton

aabdabdabd or acd aabd

1 1

0.5 d 0.5

4 4

2 2 2

10

Comparison of log automata

aabdabdadadab

Event Log

a da b

bc

end

Process Model

Without alignment

1-Align

All-AlignPre

fix-

bas

ed L

og

Au

tom

ata

aabdabdabd/acd aabd

Fitting Traces

11

Overview

aabbdadabd

a d…

Unfitting Log

Process Model

Alignment Automaton

Experiments

aabdaabdacd…

Fitting TracesAlignment

Precision Metrics

Reference: J. Muñoz-Gama, J. Carmona (2010). A Fresh Look at Precision in Process Conformance. 8th International Conference of Business Process Management (BPM): 211-226.

Use calculation

12

Overview

aabbdadabd

a d…

Unfitting Log

Process Model

Alignment Automaton

Experiments

aabdaabdacd…

Fitting Traces

Precision Metrics

Alignment

13

Implementation in ProM 6

• Package: “ETConformance”

14

Experiment settings

Event Log

Flower (F)

Precise (P)

Flower’ (F’)

Precise’ (P’)

Duplicate Log Combined Log

Precise-Precise (PP’)

Precise-Flower (PF’)

Flower -Precise(FP’)

Flower-Flower(FF’)

15

Experiment 1: Measuring precision stability

Perfectly fitting logs and models Non-fitting logs and models

P FP’ PF’ PP’F P FF’ FP’ PF’ PP’

Values of change significantly, but not and

16

Experiment 2: Sensitivity to unfitting trace

• fluctuates when traces are unfitting

Model P

0.80

0.85

0.90

0.95

1.00

0 1 2 3 4

0.00

0.20

0.40

0.60

0.80

1.00

0 1 2 3 4

Prec

isio

n

Number of Removed Events

Noise Sensitivity - P

ETC

1-Align

All-Align𝑎𝑝❑𝑎𝑝1

𝑒𝑡𝑐𝑝❑

Pre

cisi

on

#Removed events

0.000.200.400.600.801.00

0 1 2 3 4

Model PF’

Pre

cisi

on

#Removed events

0.00

0.10

0.20

0.30

0.40

0 1 2 3 4

Pre

cisi

on

#Removed events

Model FP’

0.90

0.93

0.95

0.98

1.00

0 1 2 3 4

Pre

cisi

on

#Removed events

Model PP’

17

Experiment 3: Real life cases

• Using 5 logs and models from municipalities in the Netherlands

• Computation time of is much shorter than , yet provide close estimation

18

Overview

aabbdadabd

a d…

Unfitting Log

Process Model

Alignment Automaton

Experiments

aabdaabdacd…

Fitting Traces

Precision Metrics

Alignment

19

Conclusion and Future Work

• Pre-alignment provides more accurate precision measurements in cases where logs are non-fitting

• Applicable to models with duplicate tasks/unlogged activities

• gives close estimation to with lower computation time• Log-Model evaluation needs to consider multiple

dimension of conformance• Potential use of the technique: auditing, conformance

diagnosis, process repair

20

Questions

21

Thanks!