Overview
WordPress can be a great platform to build your website on, with almost unlimited extensibility and feature sets.
One area of WordPress where it’s less than ideal for high traffic websites is the handling of 404 pages (ie a request which isn’t found). When a web request is made, WordPress runs through its standard process of parsing the query, then performing database requests to find where that permalink is located.
For high-performance sites, we always recommend page caching (such as W3 Total Cache) which instantly speeds up and bypasses this process. This allows pages to be served directly by the webserver, bypassing the need to load WordPress nor query the database.
However, 404’s mean that the page doesn’t exist so therefore a cached copy won’t exist either. For every 404 request, your website must load the full WordPress stack, then perform database queries to find the post / page / custom post type. WordPress may also attempt to find an alternative post to return as well, which while beneficial to the user, adds further resource usage to the overall request.
If you have restructured your website, deleted posts, pages, categories or tags then it’s critical to ensure these items are now redirected somewhere (or permanently served a 404 status). If not, this can generate tens of thousands of requests a day which may all result in the requests all having to load the WordPress stack.
This also affects Search Engine Optimisation (SEO) and how your website is indexed by Google and similar. Ensuring this content is cleanly redirected to other relevant content affects how your website ranks. Even if you didn’t care about the content on that page, all search engines may indefinitely crawl URLs which have previously existed. From Google’s Search Console advice on 404’s:
Once Google has successfully crawled a URL, it can try to crawl that URL forever.
It’s important to note that this behaviour isn’t unique to Google and other crawlers follow a similar pattern. The only permanent fix (for Google) is to submit a removal request to remove this page from Google’s search data. Other crawlers and search engines may have similar methods to request removal.
How to Reduce 404 Resource Usage
For a small number of redirects or changes which are simple (eg a category rename), you can use a simple redirect via the .htaccess file.
If there are complex changes or you’re making frequent changes, we highly recommend using a plugin to automatically track these changes and configure the redirects automatically for you. Some SEO plugins (such as Yoast or Rank Math) already include this functionality, so it’s best to review the features within your existing plugins to avoid duplication.
If not, one of the most popular plugins is simply called Redirection:
This plugin can automatically track when you make changes as well as log 404’s to therefore fix previous changes. This allows you to keep on top of the 404’s with minimal interaction.
By default, Redirection processes redirects through WordPress. However, it can also be configured to write redirects directly to your website’s .htaccess file. This allows Apache to process matching redirects before WordPress loads, significantly reducing PHP and database resource usage. For best performance, we recommend configuring server-level redirects where possible.