DLXS Simple Regression Testing System


Functional overview

The DLXS Simple Regression Testing System consists of script to drive the testing process and a collection of files representing the test case database. Database entries are URL strings. The system can test for assertion failures and runtime errors in the DLXS middleware. It is not a system for verifying the correctness of search results or of the visual appearance of web pages.

The system has the following general capabilities:

While not designed to verify correctness of search results or the visual appearance of web pages the system does provide a powerful check on the basic operation of the DLXS middleware. By supporting the handling of many hundreds of URLs, the system can perform a more comprehensive test than actual users are able to do by random interaction through a web browser.

The system can grow in completeness over time as URLs that have generated errors are added to the database to be checked against subsequent coding changes. The fact that the tests are automated means that they are more likely to be run, resulting a more stable system.

Technical overview

The test system script and associated files that make up the database are located in DLXSROOT/bin/regression. Within this directory are the following components:

The test case database has a primitive schema consisting of the following fields:

The test system supports cookies. DLXS stores the session ID in the cookie. Therefore, during the execution of a sequence of test cases, the state of the DLXS middleware, as maintained in the session, is preserved. This allows testing of search history, bookbag functions, portfolios and various cache mechanisms.


The operation of the system consists of running the system script from the shell command line. There are four basic functions supported by the system. These are:

Executing Test Cases

The regression test script (simple) has the following command line arguments related to executing test cases. Square braces indicate optional arguments. Note that for test execution, all arguments are optional. In that case, all existing test cases are executed on the default virtual host, using the default database (testcases/default.tcd).

Test execution progress and results are written to the terminal. They can also be captured by redirecting the output of the test script to a file. Here is an example of a results report.

               Overall Test Result Summary
                       Total test cases executed: 4
                               Total elapsed time: 8 seconds
                               Total successful tests: 4
                               Total failed test cases: 0
                       Class="text" had:
                               4 successes
                               0 failures
                                       Setid="set:default" had:
                                               1 successes
                                               0 failures
                                       Setid="set:groups" had:
                                               1 successes
                                               0 failures
                                       Setid="set:tcgroups" had:
                                               2 successes
                                               0 failures

               Failed Test Result Summary
                       No failures

Executing Failed Test Cases

A test case originates from a test case database identified by some filename testcases/TESTCASEFILE. If that test case fails, it is appended to a database file in the failures/ directory also named TESTCASEFILE. The failures/TESTCASEFILE is initialized before the test cases from a given database are executed, wiping out any cases saved in the failures/TESTCASEFILE database from previous runs.

The test system can be run against these failed test case databases. Only one database at a time can be used as a source of failed test cases to be re-executed. These database files can be renamed to preserve them so that they are not overwritten, if desired. The following arguments to the test system are supported.

Importing Test Cases to the Database

The test system supports the importation of new test cases. The new test cases may be imported into a new database or incorporated into an existing database. The following command line arguments control the import function.

The format of the test case definition file is exhibited in this example:




Comments begin with '#'. Blank lines are skipped. The set specifer is optional and defaults to [set:default]. Use __VHOST__ to be replaced by the virtual host argument or the default virtual host when the test is executed.

The name of the import file determines the name of the database file that will be the destination of the import. So, for example, if the import file is imports/specialcases.txt, those test cases will be created in the testcases/specialcases.txt database. Omitting the import file name means to use imports/default.txt as source and testcases/default.tcd as destination.

This convention supports the addition of cases to the default database and the creation of additional databases for special purposes. Note that the run options support the execution of test from all databases in one run or just cases from a single selected database.

The import process will not add a test case that duplicates an existing case for a given class and set within that class. Duplicates are allowed in different sets and in different databases.

The import/ directory is basically a scratch directory to hold test case definition files. Once in the database, there is no reason to keep them around. Similarly, the error database files saved in the failures/ directory do not need to be kept after the problem has been resolved since the case is also present in the test case database. The test case databases should be maintained under source control so that should any corruption occur the last know good database can be recovered.

Listing Test Cases

The test system supports the ability to dump a list of test cases for a given setid or class and from a given database. When both CLASS and SETID are specified, CLASS takes precedence. The following command line arguments control the listing function. The format of the CLASS dump is identical to the format of the test case definition file. Therefore the dump can be saved to a file, edited and used to import testcases to a database. The format of the SETID dump is importable as well if [class] specifiers are added.

Test Case and Database Design and Management

Following are some points to consider when designing test cases and organizing them into one or more databases