With the exception of children, puppies and medical compliance frameworks managing one of something is normally much easier than managing a lot of them. If you have a lot of puppet modules, and you’ll eventually always have a lot of puppet modules, you’ll get bitten by this and find yourself spending as much time managing supporting functionality as the puppet code itself. Luckily you’re not the first person to have a horde of puppet modules that share a lot of common scaffolding. Read on →

When it comes to running automated tests of my public Puppet code TravisCI has long been my favourite solution. It’s essentially a zero infrastructure, second pair of eyes, on all my changes. It also doesn’t have any of my local environment oddities and so provides a more realistic view of how my changes will impact users. I’ve had two Puppet testing scenarios pop up recently that were actually the same technical issue once you start exploring them, running tests against the Puppet version I use and support, and others I’m not so worried about. Read on →

As part of refreshing my old puppet modules I’ve started to convert some of my Puppet templates from the older ERB format to the newer, and hopefully safer, Embedded Puppet (EPP). While it’s been a simple conversion in most cases, I did quickly find myself lacking the ability to select a template based on a hierarchy of facts, which I’ve previously used multitemplate to address. So I wrote a Puppet 4 version of multitemplate that wraps the native EPP function, adds matching lookup logic and then imaginatively called it multi_epp. Read on →

In a large Puppet code base you’ll eventually end up with a scattering of time based ‘magic numbers’ such as cache expiry numbers, zone file ttls and recurring job schedules. You’ll typically find these dealt with in one of a few ways. The easiest is to ignore it and leave a hopefully guessable literal value (such as 3600). The other path often taken is the dreaded heavily linked and often missed comments that start off as 86400 # seconds in a day and over time become 3600 # seconds in a day. Read on →

After the recent puppet-lint 2.0 release and the success of our puppet-lint 2.0 upgrade at work it felt like the right moment to claw some time back and update my own (11!) puppet-lint plugins to allow them to run on either puppet-lint 1 or 2. I’ve now completed this and pushed new versions of the gems to rubygems so if you’ve been waiting for version 2 compatible gems please feel free to test away. Read on →

With the recent puppet-lint 2.0 release it seemed a good time to bump the version we use at $WORK and see what’d changed. In theory it was as simple as changing the version in our Gemfile and ideally everything should continue as normal, but in practise it was a little more work than that and in this post I’m going to explain what we found. Firstly let’s cover a lovely, free, bonus. Read on →

Once your puppet code base reaches a certain size you’ll often have a number of validate_ functions testing parameters and configuration values for compliance with local rules and requirements. These invocations often look like this: validate_re($private_gpg_key_fingerprint, '^[[:alnum:]]{40}$', 'Must supply full GPG fingerprint') Once you’ve spent a minute or two reading that you’ll probably be able to understand it; but wouldn’t it be nice to not have to care about the exact details and focus on what you’re actually testing? Read on →

While puppet-lint and rspec-puppet (thanks to Tim Sharpe) will help ensure your Puppet code is both clean and produces what you’d expect in the compiled catalog there are times when you’ll want to go further than unit testing with rspec-puppet and do some basic integration tests to ensure the system ends up in the desired state. In this post, with the assumption that you have Docker installed, I’ll show a simple way to run basic integration tests against a Puppet module. Read on →

On a *nix system a world writable file is one that anyone can write to. This is often undesirable, especially in production, where who can write to certain files should be limited and enabled with deliberation, not by accident. Ideally you should not be deploying files with those permissions, especially not across all your machines using puppet and so I wrote this plugin to provide a small safety net. class locked_down_file { file { '/tmp/open_octal': ensure => 'file', mode => '0666', } } files should not be created with world writable permissions The world_writable_files puppet-lint check is one possible solution to this. Read on →

The most recent in my recent series of puppet-lint plugins, the yumrepo gpgcheck enabled check, will mostly be of interest to security conscious Linux users who use a yum or dnf based package manager. In this case we’re checking the gpgcheck attribute, which indicates if yum should perform a GPG signature check on packages. Having this disabled means you’ll accept any packages from your configured repo, not just those signed by the packagers. Read on →

Sometimes there are certain puppet resource types that you don’t want to include in your code base. In my case it was cron but in yours it could be the more line originated augeas or the horribly named computer. The no cron resources check puppet-lint check will display a warning each time it finds a resource of that type in your manifests. class cron_resource { cron { 'logrotate': command => '/usr/sbin/logrotate', user => root, hour => 2, minute => 0, } }# and the lint check will show: # 'cron resources should not be used' While installing the plugin is done in the usual way - Read on →

In versions of Puppet under 3.8.5 it’s been possible to have the same parameter name specified multiple times in a class definition without error. Although allowed it was a little misleading as only the last value assigned to that parameter was taken and so in the name of simplicity it was decided in the No error on duplicate parameters ticket that this behaviour should change and now return an error. Puppet itself will start to throw an error in 3. Read on →

Modern versions of Puppet allow you to specify the mode of a file resource in one of two ways, either as a traditional octal value or the (newer addition) symbolic file modes. Although these may seem equivalent there is a minor difference that recently caused an issue on a project I do a little bit of puppet work for. Below are two example resources that each set the files permissions so the user can read and write it. Read on →

A fair while ago I wrote a Deprecation Warnings From Puppet Resources blog post and metaparameter for adding expiry information to your manifests - file { '/ec/cron.d/remove_foos': ensure => 'file', source => 'puppet:///modules/foo/foo.cron',# our custom metaparameter deprecation => '20130425:Release 6 removes the need for the foo cronjob', } We were happy users of this for a while but it had a high cost, we had to maintain our own puppet fork due to puppet being unable to load metaparams from modules. Read on →

A few projects ago we had a JSON app with quite a fiddly config file that was undergoing rapid iteration. Although we never deployed an invalid JSON config we hit a couple of snags with config files that didn’t quite match up to the applications expectations. A proposed solution was to produce a JSON Schema document we could use in both integration tests and to ensure the JSON we generated in Puppet was both well formed and, more importantly, valid for that version of the application. Read on →

I’ve recently begun to look at replacing as much of my custom puppet tooling as possible with third-party, open source, code. As part of this I’m planning to update my old libvirt testing infrastructure with more modern tools, and this seems to be leading me heavily down the docker path. One of the simpler, but less known, solutions in this space seems to be Dockunit, which bills itself as “Containerized unit testing across any platform and programming language” and is remarkably simple to get started with. Read on →

Have you ever needed to access Ansible facts from inside Puppet? well, if you ever need to, you can use the basic ansible_facts custom fact. # make sure you have ansible installed $ sudo puppet resource package ansible ensure=present # clone my experimental puppet fact repo $ git clone https://github.com/deanwilson/unixdaemon-puppet_facts.git # Try running the fact $ ruby -I unixdaemon-puppet_facts/lib/ `which facter` ansible_facts -j { "ansible_facts": { "ansible_architecture": "x86_64", "ansible_bios_date": "04/25/2013", "ansible_bios_version": "RKPPT%$DSFH. Read on →

Puppet’s always had a couple of little inconsistencies when it comes to the file and template functions. The file function has always been able to search for multiple files and return the contents of the first file found but it required absolute paths. The template function accepts module based paths but doesn’t allow for matching on the first found file. Although this can be fixed with the Puppet Multiple Template Source Function. Read on →

In the past if you wanted to run your own puppet-lint checks there was no official, really clean way to distribute them outside of the core code. Now, with the 1.0 release of puppet-lint you can write your own, external, puppet-lint checks and make them easily distributable. I spent a little bit of time this morning reading through the existing 3rd party community plugins and after porting a private absolute template path check over to the new system I have to say that rodjek has done an excellent job with both the ease of writing your own checks and the quality of the developer tutorial. Read on →

A little while ago in a twitter conversation, many hops away a few of us discussed the Puppet Certified Professional exam and topic coverage. Specifically how much of it was focused on Puppet Enterprise (PE) and if it would either dissuade users of purely FOSS Puppet or heavily impact their chance of passing if they’d never used PE. While I stand by my views I began to worry that my knowledge of the syllabus was only based on hearsay, the practice exam questions and that I was being overly harsh and possibly spreading misinformation through my own ignorance. Read on →

Structured facts in facter had become the Puppet communities version of ‘Duke Nukem Forever’, something that’s always been just around the next corner. Now that the facter 2.0.1 release candidate is out you can finally get your hands on an early version and do some experimentation. First we grab a version of facter 2 that supports structured facts from puppetlabs - # our play ground mkdir /tmp/facter && cd /tmp/facter # grab the code wget https://downloads. Read on →

One of the new features released in Puppet 3.4.0 is the ability to add options to rpm package installs. This is a feature that’s been discussed in a couple of tickets over the years and now we’ve got an official, in core, approach I’ve copied the code to the yum and apt providers github branch. class pkgoptions { package { 'strace': ensure => 'installed', provider => 'yum', install_options => [ '--noplugins', '--enablerepo=fedora' ],# or install_options => [ '-t', 'squeeze-backports' ], for Debian backports } } The approach in this patch is not the final one I’d like to take so I’ve not submitted it upstream. Read on →

"a simple resource that blocks transactions until a check passes, theoretically indicating that a remote resource is in a desired state.“ – Puppet Remote Resource Documentation I stumbled over the Puppet Remote Resource module while looking around the Puppetlabs github account for something completely different and was surprised to find that I’d never seen this little gem mentioned anywhere else. A pre-built way to have a puppet resource skipped based on the result of an external command is a very flexible tool, especially when you couple it with all the available nagios checks. Read on →

Over the years Puppet has handled resources ordering without explicit dependencies in different ways, with the release of Puppet 3.3.0 they’ve exposed this ordering logic to the admin with three interesting options. To test these options out we’ll use the ‘ordering’ test module shown below. We include three classes, ordering::beta, ordering::alpha and ordering::gamma (note that the includes are not in alphabetically sorted order). Each of these classes has three notify statements that show a number and the class they are contained in. Read on →

Even though I don’t spend as much time writing puppet code as I used to I try to stay relevant and as part of that I like to read all the Puppet books that come out. Below are the ones I’ve read this year, brief thoughts on them and the reading path I’d give to a new junior. As the name implies the Puppet 3 Beginner’s Guide is a decent place to start learning Puppet. Read on →

The kind people at Apress provided me with an alpha review copy of Pro Puppet and while it’s not the finished product you can already get a good feel for the books tone and coverage. I quite liked the first edition of Pro Puppet and this update is more evolutionary than revolutionary. All chapters from the previous edition are present and the biggest addition is the very welcome chapter on using Hiera in your modules; even if it’s oddly placed at the end of the book. Read on →

Puppet has always supported templating via ERB and while it’s a powerful, flexible templating engine the ability to use any arbitrary ruby code inside a template that’s run on the puppet master sometimes raises some eyebrows. As part of a security architecture review the concept of replacing the templating engine with something that still allows looping and text manipulation without allowing too much else was discussed and led to the idea of allowing templates to be written in Liquid. Read on →

While doing some experiments with Ansible I came across a little snippet of code that I really liked - - name: manage /etc/sudoers template: src=sudoers.j2 dest=/etc/sudoers validate='visudo -cf %s' Ansible runs the command specified by ‘validate’ against the expanded templates contents and only copies the newly generated file in to place if it’s valid. This is a wonderful feature that will help stop you from making some potentially time consuming errors. Read on →

Facter 1.7 introduced support for external facts, and I gave some external fact examples, but it left a couple of small issues unresolved. One of the larger ones is the subject of syncing the external facts down to the clients. At the moment most people are managing the external facts as file resources which creates one important difference between an internal and external fact. Internal facts are synced down at the start of the run and so are available to the puppet agent within a single run. Read on →

When writing puppet modules sometimes you need to ensure that certain classes are only used within your module itself. For example a class that implements functionality based on the local operating system that should only be used by your public class. While reading though the new puppetlabs-mcollective modules source I came across a new pattern I’d not seen used in puppet before that achieves this in a very elegant way and i thought it was worth a second look. Read on →

I’ve recently had the need to create a handful of small file based providers in puppet and while trundling uphill against the ParsedFile provider I decided to have a look at how custom providers are written using the Puppet augeas providers. My sample provider is a simple one, it manages entries in /etc/shells and was quite quick to write. While augeasproviders assumes you know how Puppet types and providers are structured it does present a very standardised way of using the augeas library in order to manipulate your target. Read on →

While Puppet may get all the glory, Facter, the hard working information gathering library that can, seldom gets much exciting new functionality. However with the release of Facter 1.7 Puppetlabs have standardised and included a couple of useful facter enhancements that make it easier than ever to add custom facts to your puppet runs. These two improvements come under the banner of ‘External Facts’. The first allows you to surface your own facts from a static file, either plain text key value pairs or a specific YAML / JSON format. Read on →

Over time parts of your puppet manifests will become unneeded. You might move a cronjob or a users in to a package or no longer need a service to be enabled after a given release. I’ve recently had this use case and had two options - either rely on comments in the Puppet code and write an out of band tool to scan the code base and present a report or add them to the puppet resources themselves. Read on →

It’s been a while since I’ve attended a Puppet Camp but considering the quality of the last one (organised by Patrick Debois) and the fact it was being held in the lovely city of Ghent again I thought it’d be a wise investment to scrape together the time off. The quality of the talks seemed quite high and considering the number of newer users present the content level was well pitched. Read on →

Back in October Nan Liu announced “pocco - a puppet manifest documentation experiment" as a way of generating much nicer looking documentation for puppet classes (you can see an example and reducing the amount of boilerplate needed to document your classes. After some issues with the ruby libraries it depends on, I ran it over a couple of my smaller manifests and I have to say the output is very readable and quite presentable. Read on →

<tl;dr> Search for puppet resources values using puppet, not just plain text</tl;dr> One of the ideas that has been sitting on my todo list is having a command that lets me grep a puppet manifest for certain properties, values or even just resources in a smarter way than just running a raw grep over files. While a simple grep works in some cases it is annoyingly fragile when you’re trying to ignore literal strings in resource types that you’re not interested in or narrow your search down to resources that have a property that can also appear in other types. Read on →

While most people know you can use puppet to ensure a service is running the mechanism it uses to determine if a service is actually running is often unexplored. By default (at least up to Puppet 2.6) puppet assumes that a service doesn’t supply a working status option and so will look up the services name in the process table to check if it’s running. If your service does support the status argument you can set ‘hasstatus => true’ and the platforms service provider will be used to interrogate the services current status. Read on →

Sometimes it’s the little niggles that annoy people the most. As my team progress in to puppet they have an annoying habit of asking very good questions; which can sometimes be a struggle to answer. Todays best question was - “How do I tell if this file is under puppets control?” While there are a couple of different ways to check (grepping through your git checkout or modifying the file and running puppet were the immediate winners) the best way is probably to look inside the catalog and check against the title of the File resources it contains. Read on →

<tl;dr>Log nrpe-runner state changes when puppet runs to see what broke or was fixed.</tl;dr> While people most often use puppet to configure and repair their infrastructures sometimes they also inadvertently use it to damage and cripple them. As part of my attempt to reduce the mean time to spot a mistake across my systems I’ve come up with a handful of small scripts that let me wrap a puppet run in a Nagios NRPE powered safety net. Read on →

At work we try, and sometimes even succeed, in using Test Driven Deployment so as one of my background projects I’ve been wrapping certaintools in to cucumber friendly forms. Over the last couple of days I’ve been grabbing ten minutes here and there to incorporate Puppet 2.6 in to the pile. Feature:Puppetwrappers Puppet Provider Examples Scenario:Confirming package installation When a machine has been puppeted Then the bash package should be installed Scenario:Confirm doodoodoo package is absent When a machine has been puppeted Then the doodoodoo package should not be installed Scenario:Confirm cron service is running When a machine has been puppeted Then the cron service should be running Scenario:Confirm tomcat6 service is not running When a machine has been puppeted Then the tomcat6service should not be running Scenario:Confirm dwilson is in libvirtd group When a machine has been puppeted Then dwilson should be a member of libvirtd Scenario:Confirm dwilson has a uid of 1000 When a machine has been puppeted Then dwilson should have a uid of 1000Scenario:Confirm dwilson has a given shell When a machine has been puppeted Then dwilson should have the /bin/bash shell I really like using the puppet providers for this because of the abstraction benefits they provide. Read on →

It all started with one of those annoying little items on the todo list find all the unpuppeted ssh authorized_keys files on a machine and alert on them. On first impressions it was going to be quite manual (always a bad sign), involve digging in to legacy installs and would be something we’d need to re-verify occasionally. It couldn’t be that bad though could it? After all how many places can an unmanaged-by- puppet sshkey live? Read on →

One of puppets more under-appreciated features is its ability to abstract and smooth the edges of certain operating system tasks and behaviours. Even something as trivial as installing a package can actually become a portability nightmare once you consider the number of different systems in the wild - rpm, yum, dpkg, pkgsrc etc. - and the varied commands needed to use them. You end up either hard coding commands, and sacrificing portability, or writing your own detection, lookup and invocation logic. Read on →

While adopting a configuration management tool like Chef and Puppet will have a large, nearly immediate, effect on your work flow even after using the tools for a while you’ll still get a little smile at all the little niceties you continuously discover. One recent small win we had recently was bringing some apache configs files under Puppet command. When we started we had the following block of config: RewriteCond %{REMOTE_ADDR} ! Read on →

Between Xmas and New Year I had some spare time to invest on a side project I’ve been looking forward to working on for quite a while. I’m pleased to announce the opening of the Puppet CookBook. I’ve introduced Puppet to quite a few companies, sysadmins and development teams over the years and a lot of the same issues, concepts and needs repeatedly crop up. By explaining how puppet works in terms of tasks and desired outcomes rather than in raw feature descriptions I hope to show some of its power and flexibility in easy to use examples in a different way to most of the existing documentation. Read on →

I’ve been watching the Marionette Collective for a while, and even gave it a small trial in a couple of testing environments, but this weekend was the first time I’ve experimented with it at a slightly larger scale (just over a hundred small VM nodes - you have to love EC2) and I’m still impressed. I can see how it’s going to make parts of my work flow easier, and in an attempt to learn a little more about how the plugin system works under the hood I decided to write a small agent, FileMD5er. Read on →

I’ve been a user of Puppet for about three years now and while on a recent dig in to some of my older classes it was a little embarrassing to see lots of file types used like this: file { '/srv/whi/maps': ensure => present, source => "puppet://$servername/whi/maps.conf", owner => 'whi', group => 'whi', mode => 0644, } file { '/srv/whi/elocs': ensure => present, source => "puppet://$servername/whi/eloc.conf", owner => 'whi', group => 'whi', mode => 0644, } Luckily as we get more experienced with a tool we can often go back and improve on the first steps. Read on →

While Puppet can be used to manage large, complex environments it’s also a useful tool at the lower end of the spectrum. Using just the puppet executable and a small inline class or two you can write very useful manifests in only a handful of lines. class build_host { package { 'build-essential': ensure => installed } package { 'subversion': ensure => installed } file { "/home/dwilson/repos/": ensure => directory, owner => dwilson, group => dwilson, }}node default { include build_host} To invoke the class you just run puppet -v build-host. Read on →