Testing¶
The benchmark system is covered by acceptance tests. These can be
found in the directory test
. To execute all acceptance tests,
type:
nosetests test
Regression tests¶
The testing setup will check all workflows stored in the benchmark
library (daisy-chains
) for compatibility with the current
benchmark system. It does this by checking if the “show all” command
will return without failure.
In most cases, a configuration file should be able to pass the test-suite without issues. However, occasionally glob and regex expressions might cause errors as they assume files or directories that are not present during the tests. To circumvent these issues, add a test directive to the configuration file specifying explicit filenames:
fastq: ../data/HG*.fastq.gz
test:
fastq:
../data/HG001-R1.fastq.gz
../data/HG001-R2.fastq.gz
../data/HG002-R1.fastq.gz
../data/HG002-R2.fastq.gz
group_regex: /([^/]+)-R.*.fastq.gz
The test directive is evaluated when --is-test
is specified
at the command line and replaces any fields of the same name in the
input section. Here, the glob expression is replaced by four
dummy filenames that ensure that the group_regex directive
will match.
The testing system will also apply regression tests to make sure that
the workflows have not changed. It does this by comparing the current
output to previous workflow output. Previous workflow outputs are
stored in the directory test/regression/daisy-chains
,
which mirrors in its layout the directory daisy-chains
.
If a workflow changes, simply delete the existing file in
test/regression/daisy-chains
and rerun the tests, a new
copy reflecting the latest workflow with be created automatically.
The directory test/regression/daisy-chains is under version
control and any changes need to commited. To re-create all regression
test data, type:
rm -rf test/regression/daisy-chains
nosetests test/regression