Testing multiple Puppet versions with TravicCI (and allowing failures)

When it comes to running automated tests of my public Puppet code TravisCI has long been my favourite solution. It’s essentially a zero infrastructure, second pair of eyes, on all my changes. It also doesn’t have any of my local environment oddities and so provides a more realistic view of how my changes will impact users. I’ve had two Puppet testing scenarios pop up recently that were actually the same technical issue once you start exploring them, running tests against the Puppet version I use and support, and others I’m not so worried about.

This use case came up as I have code written for Puppet 3 that I need to start migrating to Puppet 4 (and probably to Puppet 5 soon) and on the other hand I have code on Puppet 4 that I’d like to continue supporting on Puppet 3 until it becomes too much of burden. While I can do the testing locally with overrides, rvm and gemfiles, I wanted the same behaviour on TravisCI.

It’s very easy to get started with TravisCI. Once you’ve signed up (probably with github auth) it only requires two quick steps to get going. The first step is to enable your repo on the TravisCI site.

You should then add a .travis.yml file to the repo itself. This contains the what and how of building and testing your code. You can see a very minimal example, that just runs rake spec with a specific ruby version, below:

---
language: ruby
rvm:
  - 2.1.0
script: "bundle exec rake spec"

This provides our basic safety net, but now we want to allow multiple versions of puppet to be specified for testing. First we’ll modify our Gemfile to install a specific version of the puppet gem if an environment variable is passed in via the TravisCI build config. If this is missing we’ll just install the newest and run our tests using that. The lines that implement this, the last five in our sample file, are the important ones to note.

To support testing under multiple versions of Puppet we’ll modify our Gemfile to install a specific version of the puppet gem if an environment variable is passed in, otherwise we’ll just install the newest and run our tests using that. The code that implements this, last five lines in our sample, are the important ones to note.

#!ruby
source 'https://rubygems.org'

group :development, :test do
  gem 'json'
  gem 'puppetlabs_spec_helper', '~> 1.1.1'
  gem 'rake', '~> 11.2.0'
  gem 'rspec', '~> 3.5.0'
  gem 'rubocop', '~> 0.47.1', require: false
end

if puppetversion = ENV['PUPPET_GEM_VERSION']
  gem 'puppet', puppetversion, :require => false
else
  gem 'puppet', :require => false
end

Now we’ve added this capability to the Gemfile we’ll modify our .travis.yml file to take advantage of it. Add an env array, with a version from each of the two major versions we want to test under, with the same variable name as we use in our Gemfile.

---
language: ruby
rvm:
  - 2.1.0
bundler_args: --without development
script: "bundle exec rake spec SPEC_OPTS='--format documentation'"
env:
  - PUPPET_GEM_VERSION="~> 3.8.0"
  - PUPPET_GEM_VERSION="~> 4.10.0"
notifications:
  email: dean.wilson@gmail.com

Now our .travis.yml is getting a little mode complicated you might want to lint it to confirm it’s valid. You can use the online TravisCI linter or install the TravisCI YAML gem and work offline. The example file above will trigger two separate builds when TravisCI receives the trigger from our change. If you want to explicitly test under two versions of Puppet, and fail the tests if anything breaks under either version, you are done. Congratulations!

If however you’d like to test against an older, best effort but unsupported version or start testing a newer version that you’re willing to accept failures from, assuming the main other version still passes, while you migrate you’ll need to add another config option to your .travis.yml file - matrix.

matrix:
  allow_failures:
    - env: PUPPET_GEM_VERSION="~> 3.8.0"

In this case (in combination with the config file above) failures under Puppet 4 fail the build, but we allow, and essentially ignore, failures against Puppet 3 as we no longer explicitly support it. If we were planning a move to Puppet 5 we’d add its version here and even on builds that failed we’d start to collect information on what needs to be investigated and fixed while still ensuring our code passes tests under Puppet 4.

I’d also recommend adding an explicit fast_finish to your matrix config if you allow_failures. This allows TravisCI to signal when required tests are finished, even if the results of those allowed to fail are not yet known, as you don’t need them to know if a run has been successful or not.

matrix:
  fast_finish: true
  allow_failures:
    - env: PUPPET_GEM_VERSION="~> 3.8.0"

Here’s an example of a build with Allowed Failures in the UI: