Handling Incomplete and Skipped Tests in PHPUnit

Introduction

#Incomplete Tests

When we about to write a new test class, we might want to start off by writing empty test methods such as this:

public function testSomething()
{
}

with the intention to complete it later, or to keep track of the tests we have to write. These test methods are usually empty and thus are referred to as incomplete tests in PHPUnit.

One major problem with this type of test methods in PHPUnit is that they are interpreted as a success, a false interpretation which renders the test reports useless. This is because PHPUnit by default does not have any mechanism of saying that a test is not yet fully implemented.

In this situation, even calling `$this->fail()` in the unimplemented test method does not help either, since the test would be interpreted as a failure. This would be just as wrong as interpreting an unimplemented test as a success.

We can think of it this way, taking a successful test as a green light and a failed test as a red light, thus, we need additional yellow light to mark a test as being incomplete or not yet implemented. 

Thanks to `PHPUnit\Framework\IncompleteTest`, a marker interface for marking an exception to be raised by a test method as the result of the test is incomplete or not fully implemented. 

The standard interface for implementing `PHPUnit\Framework\IncompleteTest` is the PHPUnit\Framework\IncompleteTestError`.

The example below shows a test case class, `SampleTest`, which contains one test method, `testSomething()`. In this test case class, the `PHPUnit\Framework\IncompleteTestError` exception was triggered by calling the convenience method `markTestIncomplete()` (which automatically raises an PHPUnit\Framework\IncompleteTestError exception) in the test method, thus marking the test as being incomplete.

<?php
use PHPUnit\Framework\TestCase;
class SampleTest extends TestCase
{
    public function testSomething()
    {
        // Optional: Test anything here, if you want.
        $this->assertTrue(true, 'This should already work.');

        // Stop here and mark this test as incomplete.
        $this->markTestIncomplete(
          'This test has not been implemented yet.'
        );
    }
}
?>

An incomplete test is denoted by an `I` in the output of the PHPUnit command-line test runner, as shown in the following example:

$ phpunit --verbose SampleTest
PHPUnit |version|.0 by Sebastian Bergmann and contributors.

I

Time: 0 seconds, Memory: 3.95Mb

There was 1 incomplete test:

1) SampleTest::testSomething
This test has not been implemented yet.

/home/sb/SampleTest.php:12
OK, but incomplete or skipped tests!
Tests: 1, Assertions: 1, Incomplete: 1.

In PHPUnit the APIs helps for marking tests as incomplete.

#Skipping Tests

In a real-life environment, some tests cannot run in some environment and some test mutually exclusive. In such situations, a test may be skipped in order to give way for another to run.

Consider, for instance, a database abstraction layer that has several drivers for the different database systems it supports. The tests for the MySQL driver can of course only be run if a MySQL server is available.

The example below shows a test case class, `DatabaseTest`, that contains one test method, `testConnection()`. In the test case class’ `setUp()` template method we check whether the MySQLi extension is available and use the markTestSkipped() method to skip the test if it is not.

<?php
use PHPUnit\Framework\TestCase;

class DatabaseTest extends TestCase
{
    protected function setUp(): void
    {
        if (!extension_loaded('mysqli')) {
            $this->markTestSkipped(
              'The MySQLi extension is not available.'
            );
        }
    }

    public function testConnection()
    {
        // ...
    }
}
?>

PHPUnit denotes a skipped test by an `S` in the output of the PHPUnit command-line test runner, as shown in the following example:

$ phpunit --verbose DatabaseTest
PHPUnit |version|.0 by Sebastian Bergmann and contributors.

S

Time: 0 seconds, Memory: 3.95Mb

There was 1 skipped test:

1) DatabaseTest::testConnection
The MySQLi extension is not available.

/home/sb/DatabaseTest.php:9
OK, but incomplete or skipped tests!
Tests: 1, Assertions: 0, Skipped: 1.

#Skipping Tests using @requires

In addition to the above methods, In PHPUnit it is also possible to use the `@requires` annotation to express common preconditions for a test case as shown on the table

An example illustrating a skipped test using @requires

<?php
use PHPUnit\Framework\TestCase;

/**
 * @requires extension mysqli
 */
class DatabaseTest extends TestCase
{
    /**
     * @requires PHP 5.3
     */
    public function testConnection()
    {
        // Test requires the mysqli extension and PHP >= 5.3
    }

    // ... All other tests require the mysqli extension
}
?>

If you are using syntax that doesn’t compile with a certain PHP Version look into the xml configuration for version dependent includes in Test Suites

Previous: Managing Risky Tests in PHPUnit.
Next: Understanding Test Doubles in PHPUnit.