CoCoMoT with Uncertainty: Experiments

This website contains experimental data supporting the manuscript Multi-Perspective Conformance Checking of Uncertain Process Traces: an SMT-Based Approach by Paolo Felli, Alessandro Gianola, Marco Montali, Andrey Rivkin, and Sarah Winkler, submitted to EAAI.

Data sets

The following three data sets were considered in the experiments below:
(a) road fines (log, DPN)
  This log is a subset of this original log, containing one trace per cluster using the clustering technique of [1].
(b) sepsis (log, DPN)
  This log is a subset of this original log.
(c) hospital billing (log, DPN)
  This log is a subset of this original log.

(1) Comparison to behavior-net-based approach

The data in following table and the file linked below forms the basis of Figure 3 in the manuscript mentioned above, comparing cocomot with PROVED. The table lists random-generated logs from three DPNs from this experiments folder, using the script provided there to add a varying degree of uncertainty of the given kind. Every table cell contains a .xes log as well as the output of cocomot. On the bottom of each cocomot output file, statistics about encoding/solving time and distance are provided.
confidence values/indeterminate events
    0.1 0.3 0.5 0.7 0.9
  net10-1 xes, output xes, output xes, output xes, output xes, output
  net10-10 xes, output xes, output xes, output xes, output xes, output
  net10-2 xes, output xes, output xes, output xes, output xes, output
uncertain activity labels
    0.1 0.3 0.5 0.7 0.9
  net10-1 xes, output xes, output xes, output xes, output xes, output
  net10-10 xes, output xes, output xes, output xes, output xes, output
  net10-2 xes, output xes, output xes, output xes, output xes, output
uncertain timestamps
    0.1 0.3 0.5 0.7 0.9
  net10-1 xes, output xes, output xes, output xes, output xes, output
  net10-10 xes, output xes, output xes, output xes, output xes, output
  net10-2 xes, output xes, output xes, output xes, output xes, output
combination
    0.1 0.3 0.5 0.7 0.9
  net10-1 xes, output xes, output xes, output xes, output xes, output
  net10-10 xes, output xes, output xes, output xes, output xes, output
  net10-2 xes, output xes, output xes, output xes, output xes, output

The conformance checking time of PROVED on these logs is documented in this output file.

(2) Impact of uncertainty on performance

The data in following table forms the basis of Figure 4 in the manuscript mentioned above. The table lists logs augmented with a certain degree of uncertainty of a certain kind, as well as the respective output of cocomot.
(a) road fines data set
    0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
(C) confidence values logs xes xes xes xes xes xes xes xes xes xes
  best realization output output output output output output output output output output
  likelihood output output output output output output output output output output
(A) uncertain activities logs xes xes xes xes xes xes xes xes xes xes
  best realization output output output output output output output output output output
  likelihood output output output output output output output output output output
(D) uncertain data logs xes xes xes xes xes xes xes xes xes xes
  best realization output output output output output output output output output output
  likelihood output output output output output output output output output output
(T) uncertain timestamps logs xes xes xes xes xes xes xes xes xes xes
  best realization output output output output output output output output output output
  likelihood output output output output output output output output output output
(M) mixed uncertainty logs xes xes xes xes xes xes xes xes xes xes
  best realization output output output output output output output output output output
  likelihood output output output output output output output output output output
(b) sepsis data set
    0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
(C) confidence values logs xes xes xes xes xes xes xes xes xes xes
  best realization output output output output output output output output output output
  likelihood output output output output output output output output output output
(A) uncertain activities logs xes xes xes xes xes xes xes xes xes xes
  best realization output output output output output output output output output output
  likelihood output output output output output output output output output output
(D) uncertain data logs xes xes xes xes xes xes xes xes xes xes
  best realization output output output output output output output output output output
  likelihood output output output output output output output output output output
(T) uncertain timestamps logs xes xes xes xes xes xes xes xes xes xes
  best realization output output output output output output output output output output
  likelihood output output output output output output output output output output
(M) mixed uncertainty logs xes xes xes xes xes xes xes xes xes xes
  best realization output output output output output output output output output output
  likelihood output output output output output output output output output output
(c) hospital billing data set
    0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
(C) confidence values logs xes xes xes xes xes xes xes xes xes xes
  best realization output output output output output output output output output output
  likelihood output output output output output output output output output output
(A) uncertain activities logs xes xes xes xes xes xes xes xes xes xes
  best realization output output output output output output output output output output
  likelihood output output output output output output output output output output
(D) uncertain data logs xes xes xes xes xes xes xes xes xes xes
  best realization output output output output output output output output output output
  likelihood output output output output output output output output output output
(T) uncertain timestamps logs xes xes xes xes xes xes xes xes xes xes
  best realization output output output output output output output output output output
  likelihood output output output output output output output output output output
(M) mixed uncertainty logs xes xes xes xes xes xes xes xes xes xes
  best realization output output output output output output output output output output
  likelihood output output output output output output output output output output

(3) Comparison to explicit unfolding

The following table corresponds to Table 2 in the manuscript linked above. The columns refer to the type of uncertainty added to the log, the number of realizations, the conformance checking time in seconds, and the number of SMT solver timeouts (600 seconds). An ∞ symbol indicates that cocomot ran for more than 24h without completing the conformance check of the log. The number of realizations are hyperlinked to a xes containing all realizations, while the required time is a link to the output of cocomot and cocomot-u, respectively.
  cocomot cocomot-u
  uncertainty # realizations time # timeouts time # timeouts
(a) road fines (C) 0.1 7360 9886 166 454 0
  (A) 0.1 6505 5115 184 464 0
  (T) 0.1 6505 5115 184 464 0
(b) sepsis (C) 0.1 185 5122 1 273 0
  (A) 0.1 392 44646 1 631 0
  (D) 0.1 295 1062 1 611 0
  (T) 0.1 1076 ∞   1140 0
(c) hospital billing (C) 0.1 1346 1638 0 615 0
  (A) 0.1 2797 6099 0 614 0
  (D) 0.1 675 1629 0 685 0
  (T) 0.1 38191 ∞   1818 0

(4) Solver backends

The following two tables correspond to Tables 3 and 4 in the manuscript linked above; they list encoding and solving times as well as memory consumption of cocomot-u using different solvers. First, this comparison is done for the best-realization cost. Clicking on a cell containing the solving time for some setting leads to the output of cocomot-u in the respective configuration.
  yices (incremental) OptiMathsat z3 OptiMathsat (incremental) z3 (incremental)
  encode solve memory encode solve memory encode solve memory encode solve memory encode solve memory
(a) baseline 142 295 2.7 130 1673 16 1269 1530 1.5 133 1105 1.5 1325 1292 1.5
(b) baseline 9 608 2 12 1846 2.5 143 1628 1.5 11 1682 1.5 152 6713 1.5
(c) baseline 40 538 2 54 3177 4.8 595 2886 1.5 53 3634 1.6 709 6381 1.5
Second, this comparison is done for the likelihood cost:
  OptiMathsat z3
  encode solve memory timeouts encode solve memory timeouts
(a) baseline - - 30 - 1757 10459 1.6 5
(b) baseline 6270 23980 2.5 18 185 254 1.5 0
(c) baseline 1227 173363 5.6 214 770 6335 1.5 0