How We Automate WordPress Development Steps

Development WordPress
Coby Tamayo Coby Tamayo
11/15/19

This blog post is about how and why we automate WordPress development steps for new projects at SiteCrafting.

As we’ve transitioned toward hosting our WordPress websites on Pantheon‘s cloud platform, the net complexity we have to manage directly has gone down. Other technical choices, such as basing all our custom WordPress themes on the Timber library plugin and investing in Conifer, has drastically improved our efficiency. But, with the additional scale comes additional demands. Routine plugin and core updates become unsustainable to perform manually. The starting point for new WordPress projects requires more dependencies and setting these up manually for each new project is a lot of grunt work.

We found that for long-term use, we needed to invest in infrastructure to automate these beginning steps for each new WordPress project. Here’s that process as it stands today.

Introduction

We set out to make the WordPress project bootstrap process painless and future-proof. To do this, we use a combination of Jenkins, some custom WP-CLI commands, and a custom Pantheon upstream. At the beginning of every project, we run a Jenkins job that talks to Pantheon and creates the website from our custom upstream and sets up a bunch of project boilerplate.

The end result is a repo that a developer can clone and start working on right away. It includes a working dev environment, infrastructure for writing and running tests, and the beginnings of a project-specific doc website, all wrapped up nicely in a containerized local dev environment.

The Upstream

We host a custom upstream, sitecrafting/wordpress, at Bitbucket. This upstream contains everything we use on every new WordPress website:

  • The WP core
  • Our standard plugins
  • Custom WP-CLI commands and other tooling
  • The default Twenty Seventeen theme

By keeping a standard set of plugins installed in the upstream, we can keep them all up to date by updating them in a single place and making them available to all downstream website repos.

Why Not Groot?

Why doesn’t the upstream include the SiteCrafting starter theme, Groot? In short, because of merge conflicts. This is something that can happen when you try to merge two different changes that affect the same lines of code. Including a starter theme implies that you’re starting at a certain point and branching off from there, editing the original files. When an update to Groot affects the same lines of code as you just changed, Pantheon will report this as a merge conflict and will force you to fix them manually (not fun).

So, if that’s the case, why not include Groot and simply move or copy it at bootstrap time? We explored that option but found moving the folder is no good because Git is smart enough to track that as a move and will still complain about conflicting changes in files, even though those files live in a different place.

Why not make a copy, then? We considered the pros and cons of this strategy:

  • Pro: Less complexity bootstrapping new websites from the upstream
  • Con: Potentially confusing to newcomers to the codebase
  • Con: Dead weight in production

Considering that the only pro is essentially only saving us a call to wp theme install groot, we decided to simply not include it.

Standard Plugins

The standard plugins we install on every website are:

  • ACF Content Analysis for Yoast SEO
  • Advanced Custom Fields Pro
  • Classic Editor
  • Conifer
  • Enhanced Media Library
  • The Events Calendar (plus the Pro addon)
  • Google Analytics
  • Imsanity
  • Pantheon Advanced Page Cache
  • SendGrid
  • Timber
  • WordPress SEO Premium
  • WP Native PHP Sessions
  • WP Redis
  • WP Retina 2x
  • WP Updates Notifier
  • YoImages

These are plugins we find ourselves using on almost every single client project. If we do not end up using them, we deactivate them but keep them installed to avoid merge conflicts when there are upstream updates.

This strategy is obviously a little at odds with the reasoning behind not including the Groot starter theme, but we find the trade-off here is acceptable. As a starter theme, Groot is an area where merge conflicts would almost certainly pop up if we included it in the upstream. Premium plugins, on the other hand, take more time to install because they cannot be downloaded automatically. Cases where they need to be updated downstream directly are exceptional, and the whole point of tracking them in the upstream is to make them easier to update, making those cases even rarer.

As a precaution, when permanently disabling a plugin in production, we add the plugin directory to the list of protected paths in our pantheon.yml file. Here is another trade-off between having “dead” plugin code that will never be deactivated (not great) and having a mass-update process that scales (awesome). With the added security that Pantheon’s protected paths provide, we find this trade-off acceptable.

Plugin and Core Updates

A cron job checks nightly for any updates available to the WordPress core and to our standard plugins. It pulls any available updates in and pushes them to the master branch of our upstream.

Downstream, a similar cron job pulls these updates into each client website on a multidev branch. End-to-end and unit tests are performed against this branch. If the tests pass, the changes are typically deployed to production automatically (some projects may require additional manual checks). If the tests catch regressions, we create dev tasks to fix them and alert our quality assurance team.

One advantage to tracking all of our standard plugins in an upstream repo is that premium plugin updates can all happen in one place. Some require manual downloads, but once committed to the upstream repo, premium plugin updates flow downstream just like regular plugin and core updates!

Custom Tooling

Included in the upstream is some custom tooling for things like code scaffolding, reporting, and testing. Most of this tooling is implemented as custom WP-CLI commands.

WP YAML Fixtures help us bootstrap WordPress installs with data described in a standard, human-readable format. Because it is in YAML, it is easily scannable, and we can treat the default database as just another piece of infrastructure-as-code.

SiteCrafting WordPress Tools is a set of tools specific to our setup, reporting structure, billing, and other stuff.

The custom wp scaffold groot command is responsible for downloading a fresh copy of the latest stable Groot starter theme, initializing it with each project’s project name, theme path, custom namespaces, and other general info.

For example, a typical custom stylesheet header in WordPress looks like:

/**
 * Theme Name: This Site Is Bananas
 * Description: B-A-N-A-N-A-S
 * (More fields here.)
 */

This data gets parsed by WordPress for display in the admin area. The wp scaffold groot command sets it all up for us.

Development Dependencies

We want to be able to pull down a newly bootstrapped site, run a command to start the dev environment, and start hacking on it right away. We want the upstream repo to serve as the Source of Truth when it comes to which dependencies we are pulling in. But, declaring dependencies in the repo opens up a similar can of worms as does Groot. Like a starter theme, a composer.json or package.json file is something that might branch off from the upstream state, which could lead to a merge conflict down the line.

To get around these conflicting interests, inside the special /upstream/ directory we track a canonical “upstream version” of the files we want to have in every new project:

  • package.json
  • composer.json
  • .lando.yml
  • book.json
  • docs/ – A foundation for test-driven development is available in a few flavors on every new project. We include boilerplate code for PHPUnit, Mocha/Chai, and Cypress tests.
  • .gitignore

When bootstrapping a new website, we simply cp each upstream file to its new location at the root or, in some cases, we actually treat the upstream files as templates. Let’s look at how that works next.

Upstream File Templates

We use Lando for almost all of our local development. It’s the best local dev tool in the galaxy!

Setting up a Lando configuration requires some extra TLC, because it is meant to house data that is not available until the website has been initialized on Pantheon. To that end, on the Jenkins server, we clone the new website’s repo once it is ready and then compile a simple template to the .lando.yml file.

That template is the /upstream/.lando.yml file itself, which is actually a Twig template in disguise. It looks like this initially:

name: {{ SITE }}
recipe: pantheon

config:
  framework: wordpress
  site: {{ SITE }}
  id: {{ ID }}

# ...more hard-coded config stuff down here...

We simply pass SITE and ID to the Twig engine and replace the contents of the file.

Like the Groot theme scaffolding, this is implemented as a custom WP-CLI command: wp scaffold lando.

Because they contain project-specific info like website name and repo URL, we also treat /upstream/package.json and /upstream/composer.json as templates and compile them to their counterparts in a similar way.

Tests

A foundation for test-driven development is available in a few flavors on every new project. We include boilerplate code for PHPUnit and Cypress tests. The idea here is to be able to run lando unit to run all unit tests or lando e2e to run end-to-end tests written in Cypress as soon as you boot up the project.

The Jenkins Job

The Jenkins job we run at the beginning takes the project-specific info and churns it through this entire process.

First, it creates a new website at Pantheon, our WordPress hosting environment, from the custom upstream. Next, it clones the new repo and starts making changes. It generates a custom theme with wp scaffold groot. It bootstraps new development dependencies like package.json and composer.json. It commits its changes incrementally, so that each change is easily digestible. Finally, it pushes the committed back up to Pantheon, which automatically publishes the changes to the development website.

What’s Next?

As we’ve seen, running your own automation infrastructure is not without its drawbacks and trade-offs. It is more complex and requires more maintenance. But, for an agency like us, serving many, many different clients, investing the time in automating common setup tasks and tooling can mean a huge gain in productivity. It has for us.

That said, we are constantly trying to refine our process. The next major effort for us in this area is going to be streamlining premium plugin updates to require as little manual work as possible. Routinely updating (and even testing!) the free and open-source plugins we use is almost completely automated at this point. But due to their proprietary nature, premium plugins are inherently more difficult to keep up-to-date.

Another aspect we would like to improve is scaffolding for custom code. Thanks to Conifer, registering a custom post type in an extensible, object-oriented way is standardized and really, really easy. But we have an opportunity to make it even easier: At the time we generate a WordPress codebase, we typically know what custom post types we want to declare, so generating code for them would be relatively trivial.

Thanks for reading and happy hacking! If you have any questions or comments, please contact us at hello@sitecrafting.com.