Testing

The benchmark system is covered by acceptance tests. These can be found in the directory test. To execute all acceptance tests, type:

nosetests test

Acceptance tests

To run the acceptance tests, type:

nosetests test/acceptance

Regression tests

The testing setup will check all workflows stored in the benchmark library (daisy-chains) for compatibility with the current benchmark system. It does this by checking if the “show all” command will return without failure.

In most cases, a configuration file should be able to pass the test-suite without issues. However, occasionally glob and regex expressions might cause errors as they assume files or directories that are not present during the tests. To circumvent these issues, add a test directive to the configuration file specifying explicit filenames:

fastq: ../data/HG*.fastq.gz

test:
  fastq:
    ../data/HG001-R1.fastq.gz
    ../data/HG001-R2.fastq.gz
    ../data/HG002-R1.fastq.gz
    ../data/HG002-R2.fastq.gz

group_regex: /([^/]+)-R.*.fastq.gz

The test directive is evaluated when --is-test is specified at the command line and replaces any fields of the same name in the input section. Here, the glob expression is replaced by four dummy filenames that ensure that the group_regex directive will match.

The testing system will also apply regression tests to make sure that the workflows have not changed. It does this by comparing the current output to previous workflow output. Previous workflow outputs are stored in the directory test/regression/daisy-chains, which mirrors in its layout the directory daisy-chains.

If a workflow changes, simply delete the existing file in test/regression/daisy-chains and rerun the tests, a new copy reflecting the latest workflow with be created automatically. The directory test/regression/daisy-chains is under version control and any changes need to commited. To re-create all regression test data, type:

rm -rf test/regression/daisy-chains
nosetests test/regression