How to optimize test coverage generation in PHP? A real-world case.

Marcos Marcolin • November 24, 2022 • 4 min read

php testes coverage gitlab pcov xdebug phpdbg phpunit

The problem

We were experiencing slowness generating PHP test coverage in our GitLab pipelines, and even locally using Docker. We had always used Xdebug to do that job, but as the project grows and the tests pile up, the work starts to slow down, and that's a cost to the company (computing resources and developer time).

To optimize this, we did some research to try to reduce execution time, and found several sources, which are cited at the end of this article.

Today, the most well-known driver for this job is inevitably Xdebug, by Derick Rethans. There are others, not as new, like PHPDBG and PCov, which are also worth considering.

Xdebug – Debugger and Profiler Tool for PHP

Xdebug is an extension for PHP and provides a range of features to improve the PHP development experience.

Before showing results, I need to mention that Xdebug doesn't just generate code coverage, it's a high-level debugger, among the best of any programming language, unquestionably. So we're not questioning its effectiveness here, but its performance, that's a different matter.

PHPDBG – The Interactive PHP Debugger

phpdbg aims to be a lightweight, powerful, and easy-to-use debugging platform.

This driver proved very effective and fast to run, though we hit a few issues related to some characteristics of our codebase. I suspect it's a conflict with another extension we use, but that's beside the point.

Running the test suites separately, it turned out to be much faster than Xdebug.

Also, this driver doesn't appear to be maintained anymore, its last release was in 2013, so keep that in mind before using it.

PCOV – CodeCoverage compatible driver for PHP

A standalone PHP code coverage compatible driver for PHP.

With PCov it was different, we had full compatibility with one of our projects, which is this article's real-world case. I won't get into the installation and setup details, you can easily find those elsewhere, but feel free to reach out if you have questions.

Joe Watkins and Remi Collet's Pcov was built exactly for this, code coverage. So it doesn't worry about other jobs, unlike Xdebug.

This driver is still maintained, and its latest version was released in 12/2021, so keep that in mind before using it.

PHPUnit – The PHP Testing Framework

PHPUnit is a programmer-oriented testing framework for PHP. It's an instance of the xUnit architecture for unit testing frameworks.

Most companies use this framework to run their tests, so we could say it's basically PHP's official testing framework.

The framework is compatible with every driver mentioned in this article.

Results

We ran several tests in a local environment with the stack: Docker + Debian Buster + PHP 7.3.x + MariaDB 10.3.x + PHPUnit 9.5.x, and also on GitLab. Running unit and integration tests across a large test suite.

I'll show the comparative execution results on GitLab, the CI/CD tool we use to manage and ship most of our projects.

Xdebug v2.7.0 x PCov 1.0.11

Xdebug v2.7.0 x PCov 1.0.11 Test execution only

In the image above, you can see there's a 1-day gap between the runs, but nothing that affects the final result.

The result is already exceptional for us, cutting nearly 10 minutes off the execution time.

Xdebug x PCov

Xdebug v2.7.0 x PCov 1.0.11 Full pipeline run

Above, the result of the full pipeline run.

Points worth highlighting from the image above:

Conclusion

So, our result was that, for our use case, switching the coverage driver pays off significantly, while barely affecting the final coverage result. But your case might be different, since there's no magic solution.

I should also stress that I'm not comparing the tools themselves, but their final results.

Personally, I'd recommend sticking with Xdebug if you're running a small test suite, without integration tests or slow tests. But if you have a large project, with a good chunk of legacy code for example, using PCov or PHPDBG will be more advantageous, though as I mentioned at the start, how much does that actually cost your business?

Cheers, and see you next time!

References

Share: