Troubleshooting DLXS Middleware

Here are some tools, facilities and techniques useful for troubleshooting problems with the middleware after installation.

Server errors

If you see a page similar to this page it may due to a syntax error in your code. If you've made changes, check them for correct syntax. An easy way to do this is to run from the command line. User data is in bold, output in italics:

cd $DLXSROOT/cgi/t/text
perl ./text-idx
syntax error at ./text-idx line 105, near "foobar my"
Global symbol "%attributes" requires explicit package name at ./text-idx line 105.
Execution of ./text-idx aborted due to compilation errors.


Another possibility is that the location of Perl in your system has changed since the middleware was installed. This means the symlink to Perl in $DLXSROOT/bin/symlinks is now bad. You can change the symlink manually or restore Perl to its original location.

Assertion failures

If the CGI program runs but encounters a condition that makes it impossible to perform a specific function it will exit and generate an assertion failure page. An example might be a missing Terminology Mapper map file for one of your collections.

Note that this page displays an optional message about the nature of the failure, the URL that led to the assertion failure and a traceback of the stack up to the point the assertion failed and .

By examining each part you may be able to determine the reason for the failure. Sometimes it is useful to look at the line number in the code mentioned in the traceback to determine why the error occurred especially if no message appears.

Error messages

There are a few error messages that crop up frequently.

Could not start XPAT for only requested collection sampletc ... terminating. Error: Could not fork XPAT process or start remote process or child had exec error. Connection refused at /l1/dev/pfarber/lib/ line 147.

This means that value of host for this collection in the database is different from the machine or virtual host running the CGI program. In this event, the middleware attempts to start XPAT on that remote machine. Unless the remote access daemon (dlxsd) is running on that remote machine (and the collection data and indexes are there too) this error will occur. If your data is never remote, setting host to the value localhost is the simplest solution.

Debug flags

There are several useful debug flags which aid in determining the source of errors. Add ;debug=switch to the URL that caused the error and re-submit the URL in you browser. The complete documentation describes all the debug flags. The most useful values for switch are:

Turn on all debugging switches. This generates a lot of data.
Dump the contents of the CollsInfo database object. This lets you check that the database values for each collection, which you think are set, are actually those being used by the program.
dump the environment variables and their values. You can check for the correct values of critical environment variables such as REMOTE_USER, HTTP_HOST, AUTHZD_COLL, DLXSROOT and DOCUMENT_ROOT. This is a quick sanity check on the values of these environment variables set for CGI program by the web server / virtual host.
show the path where files subject to fallback resolution (.xml, .xsl, .css, .js) are found.
emit the raw xml from the middleware not subjected to XSLT transformation to HTML.
emit the virtual stylesheet constructed by the middleware.
dump the raw XPAT queries the middleware generated for a given search

Tools and techniques

You may be able to get additional information about what is going wrong by trying a few of these:

Running at the command line

Given a URL that causes a server error, try running the CGI program at the command line with the same input. For example given;page=simple try running the CGI program at the command line like this:

          cd $DLXSROOT/cgi/t/text/text-idx
          ./text-idx ';page=simple' 

Running the Perl debugger at the command line

You can also run the Perl debugger at the command line with the -d switch and step through the code line by line by typing 'n' at the Perl debugger prompt DB<1>. This assumes the code compiles and can be executed:

   cd $DLXSROOT/cgi/t/text/text-idx
   perl -d  ./text-idx ';page=simple' 
   Default die handler restored.

   Loading DB routines from version 1.07
   Editor support available.

   Enter h or `h h' for help, or `man perldebug' for more help.

   main::(./text-idx:87):  $::VERSION = 3.0000;

If you run at the command line you must set the AUTHZD_COLL environment variable to include the collections you request in your command line input. For the above example, in you shell:

          setenv AUTHZD_COLL :moa:moajrnl:sampletc:workshoptc:

Use newsid=1 URL parameter

In the browser you can add the newsid=1 URL parameter to cause the middleware to use a new session and re-read the collection database in case it has changed.

Set the DLPS_DEV environment variable

Set DLPS_DEV to 1 in your shell (debugging at the command line) or in your virtual host (developing under the browser) to get a full stack trace when a compilation error or runtime assertion failure occurs.

Miscellaneous checks

Here are a few additional areas you can check. Some apply to specific problems, some general.

Pageviewer error log file

The Pageviewer CGI program logs errors to a log file usually kept in /tmp/pageviewergifdir.log configured in $DLXSROOT/cgi/t/text/PageView.cfg by the $gMakeDirOutputLog Perl variable. It logs failures of the command used to create gif and pdf files from the tiff archival file and failures creating cache subdrictories of $DLXSROOT/web/t/text/gifcvtdir. This log file is maintained only for operations on the local machine not for caches on remote machines.

Apache error log

It may be useful to examine the last few entries in the Apache web server's error log in APACHE/logs/error_log. You can monitor the log in real time as you send requests to the server from your browser at your shell command line via:

          tail -f APACHE/logs/error_log

Perl module versions

The Installer prints out a report on missing Perl modules and Perl modules you have installed whose versions differ from the recommended. Double-check this list and make sure all modules are installed and that the versions match. We have found this to be especially important when dealing with the DBI/DBD database interface modules and with Apache::Session related modules.

You can check both existence and version for a given module at the command line. For example, to check DBI and Apache::Session, respectively:

perl -e 'use DBI; print "$DBI::VERSION\n"'

perl -e 'use Apache::Session; print "$Apache::Session::VERSION\n"'

Configured vs. actual file locations

Double check the values in the database (via collmgr) that specify locations for files to verify that they correspond to the actual file locations in your file system.

Move "release" rows to "production"

If the database edits you make to a collection are not taking effect it may be because you need to "Make a release to production" in collmgr. Recall that when you edit rows as the user dlxsadm, you are changing the "release" rows, i.e., those keyed by dlxsadm. When you run the DLXS middleware (using the suggested virtual host configuration), it is reading the rows keyed by production. Your edits are not visible to the running DLXS middleware unless you "Make a release to production".