Skip to content

Testing evalresp

Jump to bottom
Ilya Dricker edited this page Jun 27, 2017 · 2 revisions

Testing Evalresp

Andrew Cooke (First written on 2017-05-03)

Overview

Evalresp can be tested using unit tests (calling C functions) and black-box tests (calling the command itself). The unit tests are implemented using the ‘check’ library. The black-box tests use the ‘Robot Test’ framework’. On Unix machines, both can be run the command line or within Jenkins, but the black-box tests require the use of bash scripts which complicates their use on Windows.

Unit Tests

The unit tests are in the directory tests/c and are written using the ‘check’ library. They can be run either by executing:

	make check

Or

	tests/jenkins/run-c-tests.sh

Black-Box Tests

Base and Extended Tests

There are two kinds of black-box tests: base and extended. The base tests are a hand-written set of tests. The extended tests are automatically generated from a zip file of data and can include many thousands of files.

Directory Structure

Because the black-box tests require input and output files, and evalresp can process all files in a directory, it is simplest to run each test in a separate directory. We can use a similar structure to store the target files (the known correct files that the Robot Test framework uses as a comparison) and the tests themselves. In this way it is clear how tests, output, and target files are all related.

This is the directory structure used:

tests/robot/	# the root robot tests directory
  all/		# all tests (the actual robot test scripts)
    base/		# base tests (given explicitly)
      <name>/	# additional grouping of base tests
  	    extended/	# extended tests (generated by script)
    	      YYYY-DDD/	# extended tests for the given date
        data/		# input files (tarballs also cached here)
          base/   	# input files used in base tests
          extended/	# files used in extended tests (unpacked tarball)
        lib/		# support library for robot
        run/		# where tests are run (and output written)
    base/   	# where base tests are run
            <name>/ 	# additional grouping of base tests
          extended/	# where extended tests are run
            YYYY-DDD/   # extended tests for the given date
        target/		# target files (tarballs also cached here)
          base/   	# target files for base tests
            <name>/ 	# additional grouping of base tests
          extended/	# target files for extended tests (unpacked tarballs)
            YYYY-DDD/   # extended tests for the given date

As was described above: all, run and target all contain the same structure and so link the tests, output and target files, respectively.

Generating Extended Tests

Extended tests are generated from a set of response files available at http://ds.iris.edu/files/staff/chad/REST-testset.zip which is downloaded automatically. Data files are extracted, and tests generated for a given year, day and SNCL pattern using the command

	tests/jenkins/build-extended-robot-tests.sh

The target files for two dates are available at http://isti.com/~andrew/RESP-targets-YEAR-DAY.zip for 2010-365 and 2017-1. Again, these are downloaded automatically (if those dates are tested). Other target files can be generated by running the tests and then using the command

	tests/jenkins/collect-extended-targets.sh

See below for examples.

Jenkins Configuration

Jenkins can be configured as follows:

  • Add a ‘New Item’ as a “Freestyle Project’
  • Download from git as appropriate
  • Build using a script similar to:
   ./tests/jenkins/build-evalresp.sh
   ./tests/jenkins/clean-test-dirs.sh
   ./tests/jenkins/build-extended-robot-tests.sh 2017 1 '*.WMQ.*'
   ./tests/jenkins/build-extended-robot-tests.sh 2010 365 '*'
   ./tests/jenkins/run-c-tests.sh
   ./tests/jenkins/run-robot-tests.sh
  • Collect results from the tests in a post-build step.

Download Cache

The recipe above will download the (large) data files on every run. To avoid this, the data files can be downloaded elsewhere and then ROBOT_ARCHIVE_CACHE defined appropriately. So the build-extended commands would change to ROBOT_ARCHIVE_CACHE=/some/dir ./tests/jenkins/build-extended-robot...

APPENDIX

Setting up black-box testing on CentOS 6.4 (i386)

From here follow the instructions in Jenkins Configuration section above

Clone this wiki locally