Managing CloudFormation Stacks with Ansible

Constructing a large, multiple application, virtual datacenter with CloudFormation can quickly lead to a sprawl of different stacks. The desire to split things sensibly, delegate control of separate tiers and loosely couple as many components as possible can lead to a large number of stacks, lots of which need values from stacks created earlier in the run order. While it’s possible to do this with the native AWS CloudFormation command line tools, or even some clever bash

(or Cumulus), having a strong, higher level tool can make life a lot easier and reproducible. In this post I’ll show one possible way to manage interrelated stacks using Ansible.

We won’t be delving into the individual templates used in this example. If you’re having this kind of issue with CloudFormation then you probably have more than enough of your own to use as examples. Instead, I’ll show a basic Ansible playbook for managing three related stacks.

---
- hosts: localhost
  connection: local
  gather_facts: False
  vars:
    stack_name: dswtest
    region: eu-west-1
    owner: dwilson
    ami_id: ami-n0tr34l
    keyname: key-64
    snsdest: test@example.org

The first part of our playbook should be familiar to most Ansible users. We set up where to run the playbook, how to connect and ensure we don’t spend time gathering facts. We then define the variables that we’ll be using as parameters to a number of stacks. The ability to specify literals in a single place was the first benefit I saw when converting a project to Ansible. This may not sound like a major win but being able to change the AMI ID in a single place, or even store it in an external file that our build system can automatically update, is something I’d find difficult to give up.

Now we’ll move to the first of our Ansible tasks, a CloudFormation stack represented as a single Ansible resource. The underlying template creates a basic SNS resource we’ll later use in all our auto-scaling groups.

  tasks:
  - name: Add SNS topic
    action: cloudformation
      stack_name={{ stack_name }}-sns-email-topic
      state=present
      region="{{region}}"
      template=sns-email-topic.json
    args:
      template_parameters:
        AutoScaleSNSTopic: "{{snsdest}}"
    register: asgsns

The ‘args:’ section contains the values we want to pass in to the template. Here we’re only passing a single value that we defined earlier in the ‘vars:’ section. We’ll see more complicated examples of this later. We also register the output from the CloudFormation action. This includes any values we specify as “Outputs” in the template and provides a nice way to deliberately define what we’re exposing from our template. The alternative is to pull out arbitrary values from a given resource created in a previous stack but that’s a hefty breach of encapsulation and will often bite you later when the templates change.

The Create Security Groups CloudFormation task doesn’t really have anything interesting from an Ansible perspective, we run it, create the repos and gather the outputs using ‘register’ for use in our next template.

  - name: Create Security Groups
    action: cloudformation
      stack_name={{ stack_name }}-security-groups
      state=present
      region="{{region}}"
      template=security-groups.json
    register: secgrp

The ‘Create Webapp’ example below shows most of the basic CloudFormation resource features in a single task. We use variables defined at the start of the playbook to reduce duplication of literal strings. We prefix the stack names to allow multiple developers to each build full sets of stacks without duplicate stack name conflicts while keeping grouping simple in the AWS web dashboard.

  - name: Create Webapp
    action: cloudformation
      stack_name={{ stack_name }}-webapp
      state=present
      region="{{region}}"
      template=-webapp.json
    args:
      template_parameters:
        Owner: "{{ owner }}"
        AMIId: "{{ ami_id }}"
        KeyName: "{{ keyname }}"
        AppServerFleetSize: 1
        ASGSNSArn:             "{{ asgsns['stack_outputs']['EmailSNSTopicARN']      }}"
        WebappSGID:            "{{ secgrp['stack_outputs']['WebappSGID']            }}"
        ElasticacheClientSGID: "{{ secgrp['stack_outputs']['ElasticacheClientSGID'] }}"

In the args section we also use the return values from our previous stacks. The nested value access is a little verbose but it’s easy to pickup and being able to see all the possible values when running Ansible under debug mode makes things a lot easier. We also had the need to pull down output values from stacks created outside of Ansible, so I wrote a simple Ansible CloudFormation lookup plugin.

So what does Ansible gain us as a stack management tool? In terms of raw CloudFormation it provides a nice way to remove boilerplate literals from each stack and define them once in the 'vars' section. The ability to register the output from a stack and then use it later on is an essential one for this kind of stack building and retrieving existing values as a pythonish hash is much easier than doing it on the command line. As for added power, it should be easier to implement AWS functionality that’s currently missing from CloudFormation as an Ansible module than a CloudFormation external resource (although more on that when I actually write one) and performing other out of band tasks, letting your ticketing system know about a new stack for example, is a lot easier to integrate into Ansible than trying to wrap the cli tools manually.

I’ve been using Ansible for stack management in a project that involves over a dozen separate moving parts for the last month and so far it’s been working fine with minimal pain.