DBupdate Test Suite HOWTO
Version 2.1
created: 2002-11-29
"Program testing can be used
to show the presence of errors,
but never to show their absence!"
Dijkstra
Table of Contents
1. Configuration
1.1 whois_rip configuration for testing
1.2 Testing system configuration
1.3 Other scripts
2. Usage
2.1 Script usage
2.2 Composing a filter
2.2.1 Objects and filters
2.2.2 Regular expressions and modifiers
2.2.3 Filters with parameters
2.2.4 Checking the presence of object in DB
2.3 Running external program
2.4 Log files
2.5 DBupdate options
2.6 Using testing program to test any other program
1. Configuration
1.1 whois_rip configuration for testing
Database description
You need to point the update source record to the database you are going to use. For this, modify this variable:
UPDSOURCE
Multiple sources are supported. However currently 'loader' program doesn't work properly if all sources are in the same database. So they need to be in separate databases, 1 source per database.
For every UPDSOURCE specified in rip.config the testing program will recreate the database and empty it before running every test. It takes quite some time. It is advised to comment out all extra sources if you're not testing them at the moment. This considerably speeds up a test.
E-mail addresses
During the test, dbupdate will be run with -r flag to redirect notifications. For more information about dbupdate flags used, check DBUPDATE_* valiables in testing configuration file, as well as corresponding section of this document.
Set these variables appropriately to make sure no mails are getting through during the testing:
HUMAILBOX
AUTOBOX
DEFMAIL
Check whoisd_start script in $WHOIS_ROOT/bin directory and edit e-mail addresses which are supposed to be notified if the server crashes.
Sources definition
In sources.config file, put the definition of your database(s).
Access control
To enable testing script to update the Database, you need to create an entry in your RIPADMIN database (see rip.config variable RIPADMIN) for the host where you're running the script.
1.2 Testing system configuration
The test root can be found in tests/dbupdate/testing. The datasets for testing are at: tests/dbupdate/test-data. These tests are suitable for new dbupdate ONLY.
The configuration file is dataset.config, you can supply any name. The most important bits are:
- root dir for testing script
- root dir for whois_rip
- names for logfiles
- path to dbupdate
- path to whois client
- path to make_db_test script
- path to rip.config
1.3 Other scripts
The scripts especially tailored for testing system are:
make_db_test
- load_file.sh
- extract-keycerts-from-text-dump.pl
- resync-gnupg-ring-from-text.sh
make_db_test differs from standard make_db. It does extra actions:
- re-creates the database
- runs load_file.sh
load_file.sh loads a single file with specified path.
Please note that testing script relies on whois_rip directory layout as much as all infrastructure scripts and utilities for whois_rip. You may need to modify them in order to point in the proper places if you have non-standard setup. The scripts can be found in the test root dir under 'bin' directory.
2. Usage
2.1 Script usage
The command line options for test.pl are:
usage:
--trace_test, -t turn on test tracing
--trace_dbupdate, -d turn on dbupdate tracing
--output_report, -o print report to stdout
--config, -c configuration file
--limit, -l N limit of mismatches (failed tests)
--stderr, -e print dbupdate stderr into stderrlog file
--stdout, -u print dbupdate stdout into stdoutlog file
--errors, -z N limit of program errors (misconfiguration, missing files, etc)
--rundir, -r [comma-separated list of directories]
- -t shows testing script configuration when starting;
- -d make testing script to run dbupdate in debugging mode, prints all debugging information;
- -o will duplicate all output to report file into stdout;
- -c [filename] the name of the configuration file for testing script
- -l N after this number of failures the testing script will exit.
- -e indicates that stderr from dbupdate will be written to [stderrlog] file
- -u indicates that stdout from dbupdate will be written to [stdoutlog] file
- -z N after this number of program errors, the testing script will exit.
- -r [dir list] run tests recursively in these dirs
Before starting the test, make sure whois_rip is running, otherwise the exception will be generated. For example, the options can be:
./test.pl -c [confdir]/dataset.config -o -r [dirlist]
Note that names in the dirlist are absolute paths. If you omit "-r" option, all the tests under $DATADIR will be run recursively.
2.2 Composing a filter
2.2.1 Objects and filters
In general, filter file should contain the declaration of objects to be checked and filters to be used. Every object or filter is RPSL-like parapraph, a list of "attribute: value" pairs. This file can also contain comments ('%' or '#' at the beginning of the line).
Objects may require several filters to be matched. If the object is specified with type and name, this instructs the program to find exact matching acknowlegement/notification etc for this object and check regular expressions only in this part of a logfile.
object: type pkey
body: filter_name
lines: [multiple|single] - default is "multiple".
...
Example:
object: person TEST1-DB-TEST
body: my_custom_filter
lines: single
Also the object can be specified without any particular name, using reserved name "all". This instructs the program to match regular expressions against the whole logfile.
object: all
body: filter_name
lines: [multiple|single] - default is "multiple".
...
Example:
object: all
body: my_custom_filter_for_entire_file
lines: multiple
"lines: multiple" instructs the program to check the output line by line, applying usual regular expression, where "." matches everything except from beginning and end of line. "lines: single" instructs the program to check the output as single line, applying multiline patterns with "/ms" modifier. For more information about multiline pattern matching see Perl documentation. For example, the filter using multiline matching can be the following:
object: person TEST1-DB-TEST
body: my_filter
lines: single
filter: my_filter
ack: AAAA\nBBBB\nCCCC\n
This test will be successful if in acknowlegement log file there is an acknowlegement for this object, and it contains the block of text:
AAAA
BBBB
CCCC
Filter can contain several attributes, indicating where to match the output. The syntax for filter is:
filter: [name] [mandatory]
ack: [!] [perl regexp] [optional]
notif: [!] [perl regexp] [optional]
forw: [!] [perl regexp] [optional]
upd: [!] [perl regexp] [optional]
whois: [!] [DATE_ON | DATE_OFF] [optional]
stdout: [!] [perl regexp] [optional]
stderr: [!] [perl regexp] [optional]
report: [!] [perl regexp] [optional]
summary: [!] [perl regexp] [optional]
2.2.2 Regular expressions and modifiers
The testing script will look for a string (first taken) matching every regular expression in the corresponding log file/block. If a regexp is preceeded by "!" sign separated by space, this means you want to make sure this regexp is _not_ matched. If a regexp contains only "!" it means you want to make sure that output in this particular log file for this object is empty. For example, to make sure no notifications about personal object TEST1-DB-TEST are send, use this syntax:
object: person TEST1-DB-TEST
body: my_filter_no_notif
filter: my_filter_no_notif
notif: !
2.2.3 Filters with parameters
The filter can contain a list of parameters. That is, the filter name is similar to function name and arguments are substituted in the filter's regular expressions.
Example:
object: person TEST1-DB-TEST
body: my_filter(person,TEST1-DB-TEST)
filter: my_filter
ack: \[$0\][\s]+$1
This effectively will mean:
filter: my_filter
ack: \[person\][\s]+TEST1-DB-TEST
When matching filter my_filter, $0 is substituted with 1st argument (i.e. "person"), and $1 is substituted with 2ndargument ("TEST1-DB-TEST"). The placeholder number can be only a digit ($0, $1, ... $9). Number of available placeholders is not matched with the number of arguments. I.e. if there are more placeholders than arguments available, those placeholders which don't have matching argument are not substituted and treated as usual text. The filter will most probably fail to be matched then. If there are less placeholders than arguments, all possible placeholders will be substituted with arguments, but no error will be reported about unused arguments.
2.2.4 Checking the presence of object in DB
In order to check if the object is in the database and looks as intended, whois search can be used.
"whois" attrubute allows to check whois output for the results. For named objects, type and primary key of the object will be taken to form the query like: "-r -T [type] [primary_key]. This query is issued to the server. The object received will be matched with corresponding object in 'whois' file. 'whois' file can contain more than one object.
If the query modifier is DATE_OFF, then dates in "changed:" attribute are filtered out and not considered during the comparison. If the query modifier is DATE_ON, objects are matched "as is". If 'whois' attribute of the filter contains '!' sign, the non-existence of the object in the database will be checked.
2.3 Running external program
There's a possibility to run external program before running the test. In order to do this, put special formatted string into the filters_local.config file for this test:
@ /path/to/my/script [args]
For now, testing program attempts to run this script if such a string is specified. However, please note that exit code from the script is not checked (yet). The testing script will simply wait until the program returns, and report error if it doesn't exit before timeout has expired.
2.4 Log files
The test program analyses several types of logfiles produced by dbupdate. All they are described in dataset.config file, but dbupdate-specific log files are declared in rip.config.
- acklog - dbupdate logging, contains acknowlegements
- updlog - dbupdate logging, contains updates
- notiflog - dbupdate logging, contains notifications
- forwlog - dbupdate logging, contains forwarded update request which failed due to auth. failure.
- stderrlog - test logging, contains stderr received from dbupdate
- stdoutlog - test logging, contains stdout received from dbupdate
- reportlog - test logging, contains more detailed information about test results, errors and (optionally) tracing.
- summarylog - test logging, contains less detailed information about test results, in short form with statistics of failed/successful/skipped/errors.
2.5 DBupdate options
The test program usually runs 'dbupdate' program to process 'dbupdate' and 'test' files while running a particular test. Any command-line options for dbupdate can be specified in 'dbupdate' and 'test' file. Every option has it's variable name in the configuration file. The available options are:
# dbupdate flags: -r to supress notifications sending, -T to allow mntner creation
# generic flags (although -f is added later to point to filename
DBUPDATE_GEN = -r -c $RIP_CONFIG
# points to a filename containing input
DBUPDATE_TEXT = -f
# input is a mail message
DBUPDATE_MAIL = -M
# ** TBD ** input comes from networkupdate
DBUPDATE_NET = -n
# switch on dbupdate tracing
DBUPDATE_TRACE = -t
# input comes from syncupdates
DBUPDATE_SYNC = -w
These flags can be specified as a string in 'dbupdate' or 'test' file. For example:
$DBUPDATE_GEN $DBUPDATE_MAIL $DBUPDATE_TEXT
When the testing program runs dbupdate, it takes this string and expands variable names into actual dbupdate flags taken from configuration file. Then filename containing input is added and dbupdate is run:
dbupdate -r -c rip.config -M -f filename
If the command-line string is not found in the file containing update, the default one is taken:
$DBUPDATE_GEN $DBUPDATE_TEXT
which effectively means
-r -c rip.config -f
And then the filename is added.
2.6 Using testing program to test any other program
Any other program giving an output can be tested using the testing system. How to do this: