Running WordPress Cron via PHP-CLI

WP-Cron, the WordPress task scheduler, is a common source of problems, from missed publish schedules and failed auto-updates, to broken garbage collection and cache flushing.

There are plenty of good tutorials on working with the scheduler, so in this post we’d like to focus more on performance, and why it’s a better idea to trigger WP-Cron exclusively via a CLI process.

Running wp-cron.php via php-cli

The Problems

By default, WordPress spawns the cron runner using an HTTP request to itself. Such a request is quite a bit of overhead for the original user-facing request, usually about 1 second. This means that every once in a while (when cron needs to run), regardless of how fast your application is, the response time for an unlucky user will suck.

For high-traffic sites it gets even worse, because WordPress options are not thread-safe, you may easily end up with multiple concurrent requests attempting to spawn the scheduler, resulting in multiple unlucky users per cron run.

On the contrary, with an aggressive page caching setup, you might never be able to get that cron to spawn, because all requests would be served from cache. Although a high cache hit-rate is a good thing for performance, it does cause certain unpredictability in this case. Similarly, having very low traffic will cause the same effect.

Spawning via crond

The problems outlined above are the main reasons why so many WP-Cron tutorials focus around spawning the scheduler with cURL or wget using crond — the Linux cron daemon, like so:

*/10 * * * * user wget https://example.org/wp-cron.php > /dev/null

As well as disabling spawning WP-Cron from WordPress itself:

// wp-config.php
define( 'DISABLE_WP_CRON', true );

This approach is better than the default, but there’s a problem.

Every time you spawn the WordPress Cron via an HTTP request, you’re locking up a php-fpm worker (assuming you’re running php-fpm) for the duration of the request. It means that the PHP worker will be busy carrying out your cron tasks, instead of serving real HTTP requests for real people… Or robots.

If your cron tasks are very fast, it’s usually not a problem, but many plugins tend to offload long-running tasks to WP-Cron, such as garbage collection, XML sitemap generation, performing a backup, a security scan and whatnot.

Such long-running tasks can lock up your PHP worker for minutes, even hours. And sometimes a new cron process will be spawned before the previous one has finished, which means you now have two workers locked up.

With WordPress Multisite it gets even worse, we’ve seen crond configurations such as this:

*/10 * * * * user wget https://example-1.org/wp-cron.php > /dev/null
*/10 * * * * user wget https://example-2.org/wp-cron.php > /dev/null
*/10 * * * * user wget https://example-3.org/wp-cron.php > /dev/null

Or a PHP wrapper spawning multiple “background” requests with cURL. This is a real performance nightmare, blocking multiple PHP workers at once every 10 minutes.

Using a server configuration with up to 8 php-fpm workers and at least 8 WordPress sites in a Multisite network, you’re essentially putting all websites on pause until one of these spawned cron requests is finished. Every ten minutes.

Triggering WP-Cron from CLI

The solution to the above problem is simple — do not put cron requests in the same php-fpm pool that’s used to handle real user requests. In fact, don’t put them in a php-fpm pool at all. Use php-cli instead.

A php-cli process will skip going to nginx (or Apache), getting assigned a php-fpm child process, and blocking that child process for the duration of the run. It will simply execute whatever is in wp-cron.php, right away. It doesn’t mean you should go all crazy with CLI though, because php-cli processes will still consume your memory, disk IO and everything else.

Launching WP-Cron via the PHP CLI is similar to doing it with wget or cURL, you’ll just need the absolute path to your wp-cron.php file:

*/10 * * * * user php /path/to/wp-cron.php > /dev/null

If you run Multisite, you’ll want to specify the HTTP_HOST for each individual site, like so:

*/10 * * * * user HTTP_HOST=example-1.org php /path/to/wp-cron.php > /dev/null
*/10 * * * * user HTTP_HOST=example-2.org php /path/to/wp-cron.php > /dev/null
*/10 * * * * user HTTP_HOST=example-3.org php /path/to/wp-cron.php > /dev/null

That said, it’s not a very good idea to run many of these in parallel, since they can easily eat up your server resources, impacting your php-fpm workers. Also note that you’ll still be relying on the internal WordPress cron transient-based locking mechanism to prevent multiple concurrent runs, so long-running tasks (> 10 minutes in this case) can still potentially cause problems.

A Python Wrapper

For all the reasons above we wrote our own wp-cron.py wrapper, which is open source and licensed under the GPL. It’s a simple Python utility which:

  • Uses file-based locking to prevent concurrent runs
  • Is Multisite-friendly, uses WP-CLI to obtain a list of sites
  • Sequentially runs wp-cron.php via php-cli on each site

Usage

Using it is just as simple. Make sure the default cron spawning is turned off in your wp-config.php file, and the paths to your WordPress installation, the WP-CLI binary and the PHP binary are correct in the script itself.

*/10 * * * * user /usr/bin/python /path/to/wp-cron.py

You can perform a test-run by executing it with python:

$ python wp-cron.py
Running wp-cron.php for example-1.org
Running wp-cron.php for example-2.org
Running wp-cron.php for example-3.org

Furthermore, you can completely disable running wp-cron.php through an HTTP request with a simple WordPress plugin:

add_action( 'plugins_loaded', function() {
    if ( defined( 'DOING_CRON' ) && DOING_CRON && php_sapi_name() != 'cli' )
        die();
});

Keep monitoring the execution time of every cron run and make sure it doesn’t get out of hand, i.e. a long-running backup job (which you shouldn’t run with WordPress anyway) can cause delays for scheduled posts and other tasks. If you’re running a multiple-server environment, make sure you run wp-cron.py on a single web server and not on all of them.

You can download wp-cron.py from GitHub and contribute a patch if you’re into that kind of stuff. If you have any questions or need assistance setting this up on your server, feel free to get in touch.