02: Running the test suite

Now that we've compiled PHP from source, we need to learn how to run the tests using PHP's black-box testing tool called run-tests.php.

Where the test files live

The test files live in various places throughout the PHP source code.

We looked at the basic test at tests/basic/001.phpt and saw it was a normal PHP file so we ran it with our compiled binary.

$ sapi/cli/php tests/basic/001.phpt

The two primary ways of running the test suites are 1) directly through run-tests.php and 2) the make target test.

We first invoked the help screen for run-tests.

$ sapi/cli/php run-tests.php --help

The run-tests tool won't run unless we specify the PHP binary that will run the tests. There are three primary ways of specifying a PHP binary.

$ export TEST_PHP_EXECUTABLE=`pwd`/sapi/cli/php
$ sapi/cli/php run-tests.php

$ sapi/cli/php run-tests.php -p `pwd`/sapi/cli/php

$ sapi/cli/php run-tests.php -P

By default run-tests will run the entire test suite so we specified a single test by passing it as an argument.

$ sapi/cli/php run-tests.php -P tests/basic/001.phpt

You can also specify a directory or list other tests or directories.

$ sapi/cli/php run-tests.php -P tests/basic/ ext/standard/tests/random/

The easiest way to run run-tests is with the make target test.

$ make test

We specify the tests we want to run with TESTS.

$ make test TESTS=ext/standard/tests/random/
$ make test TESTS="ext/standard/tests/random/ ext/standard/tests/password"

After running the tests, run-tests will mark the test as one of the following statuses:

Passed The test passed
Failed The test failed
Borked Invalid test file (this shouldn't happen out of the box)
Warned A warning was raised from the --SKIPIF-- section (more on this later)
Leaked A memory leak was detected (Valgrind must be enabled with the -m flag)
XFailed The test failed, but it was expected to (weird)
Slow Test took longer than timeout (defaults to 60 seconds or 300 seconds if testing for memory leaks; timeout can be customized with --set-timeout)

If you found this guide helpful, say, "Hi" on twitter! I'd love to hear from you. :)