This directory contains examples, log files, and scripts from the
experimental evaluation in [1].

Note: Implementation and experimental evaluation data use the term
"partitioned propositions" rather than "grouped propositions".

The columns of the tables of results (all.raw.{csv,txt}) are as follows:
t           run time [seconds]
m           memory [MB]
sti         size of syntax tree of input formula [# nodes]
stj         size of syntax tree of input formula after possible nnf-ization
            [# nodes]
sts         size of syntax tree of simplified input formula [# nodes]
stc         size of syntax tree of uc with algorithm proof [# nodes]
stp         size of syntax tree of uc with algorithm proof [# nodes]
stpd        size of syntax tree of uc with algorithm proofdeletion [# nodes]
std         size of syntax tree of uc with algorithm deletion [# nodes]
clj         # SNF clauses obtained from input formula
cls         # SNF clauses obtained from input formula, simplified
clc         # SNF clauses of uc
pv          peak number of vertices in resolution graph
pe          peak number of edges in resolution graph
pve         peak number of (vertices + edges) in resolution graph
it          # iterations in deletion-based uc minimization
itu         # iterations in deletion-based uc minimization with unsat result
its         # iterations in deletion-based uc minimization with sat result
p           # of propositions in uc
po          # of proposition occurrences in uc
ppp         peak number of partitions per proposition in uc
>=2         # of propositions with 2 or more partitions in uc
c           # formulas of input formula set (PLTL-MUP) / # top-level conjuncts
            of input formula (procmine)
cc          # formulas of uc (PLTL-MUP) / # top-level conjuncts of uc
            (procmine)

Entries have the following meaning:
to          time out
mo          memory out
na          not available
si          simplified to false during translation from LTL to SNF
---         example not run (Only for subfamilies of increasing
            difficulty. For non-random families at least two less
            difficult examples of subfamily timed or memed out. For
            random families all instances from at least three less
            difficult levels of difficulty timed or memed out.)

File (type)                     Content
--------------------------------------------------------------------------------
all.raw.csv                     table of results as csv
all.raw.txt                     table of results as txt
config_examples.bash            list of examples for run_benchmarks_*.bash
Config_examples.pm              list of examples for generate_tab.pl
generate_tab.pl                 script used to generate tables of results
README                          this file
run_benchmarks_*.bash           scripts used to run the examples (note: the
                                version of run used is customized to allow
                                configuring the rate of reporting)

For the following files we have for all solvers:
- ${e}  is the name of an example

For TRP++UC:
- ${p}  is "simple" if partitioned propositions were disabled and
        "partitioned" if partitioned propositions were enabled
- ${m1} is whether uc extraction was enabled and which optimizations
        were not used:
        - none: no uc
        - proof: (non-minimal) uc extracted from resolution graph
        - proofdeletion: uc extracted from resolution graph +
          deletion-based minimization
        - deletion: uc by deletion-based minimization
        - proof_aug2: include premise of aug2 in resolution graph
        - proof_bfsloopitinitcg: include premise 1 of
          BFS-loop-it-init-c in resolution graph
        - proof_bfsloopitinitce: include premise 2 of
          BFS-loop-it-init-c in resolution graph
        - proof_bfsloopconclusion2e: include premise 2 of
          BFS-loop-conclusion2 in resolution graph
        - proof_nopruneloopits: don't immediately prune failed loop
          search iterations from resolution graph
        - proof_noprunemain: don't prune vertices and edges unreachable
          from the main partition between loop searches from resolution
          graph
        - proof_all: all above optimizations disabled
- ${m2} is one of ${m1} or "proof_partitionedpropositions" if a
        partitioned propositions uc was extracted from a resolution graph
- ${s}  is "same" if in the translation from LTL into SNF subformulas
        of the same polarity were shared and "none" otherwise
- ${f}  is the format of the uc: internal, snf, ltlc, ltl

For procmine:
- ${i} is "dk", "encoding", or "unsatcore"

Files for TRP++UC:
${e}.trp                        example
${e}.trp.${p}.${m1}.${s}.${f}   core file
${e}.${m2}.${s}.log             log file

Files for PLTL-MUP:
${e}.pltl                       example
${e}.pltlmup.log                log file

Files for procmine:
${e}.procmine                   example
${e}.procmine.log               log file
${e}.procminenomus.log          log file with uc minimization disabled
${e}.${i}.procmine              intermediate files
${e}.nomus.${i}.procmine        intermediate files with uc minimization disabled

Other files:
${e}.sizenonnf.log              log file for determining size of syntax tree of
                                input formula

Viktor Schuppan
Füssen, 24 June 2015

[1] V. Schuppan: Extracting Unsatisfiable Cores for LTL via Temporal
    Resolution. To appear in a special issue of Acta Informatica
    dedicated to extended versions of selected papers from the 20th
    International Symposium on Temporal Representation and Reasoning,
    TIME'13, Pensacola, FL, USA, September 26-28,
    2013. doi:10.1007/s00236-015-0242-1
