Puppet Acceptance Testing With Beaker

Test-driven development can bring a number of benefits. An approach where you write tests first makes you think about what you need to develop. Subsequent refactoring generally leads to designs that emerge in the code with high levels of test coverage.

Recently PuppetLabs introduced a new tool for acceptance testing called Beaker. A Puppet module is applied to a system under test (SUT) which in turn is acceptance tested. These tests would normally complement a unit test-driven approach to writing modules with rspec-puppet.

Acceptance Testing with Beaker

Beaker can manage the temporary creation of a SUT via a hypervisor. The Puppet module under test is installed and then the tests are executed.

Install Beaker

Beaker can be installed via a Gemfile

group :development, :test do
...
  gem 'beaker', :require => false
  gem 'beaker-rspec', :require => false
...
end

Use bundler to install the gems

gem install bundler
bundle install

Beaker Configuration

Typically Beaker acceptance tests are held in a spec/acceptance subdirectory.

A spec/acceptance/nodesets subdirectory defines the machines that we want to test against. Nodesets are yaml files that define the machine’s name, location and platform. The hypervisor type is also specified. Examples for Vagrant can be found here.

module
  spec      
      acceptance
          <acceptance-test>_spec.rb
          nodesets
              default.yml
              ubuntu-server-1404-x64.yml
              ...
      defines
      fixtures
      spec_helper.rb
      spec_helper_acceptance.rb
  tests

The spec_helper_acceptance.rb file provides the basic configuration. The module under test is copied on to each host. UNSUPPORTED_PLATFORMS provides an easy way to de-scope tests for a particular platform.

require 'beaker-rspec/spec_helper'
require 'beaker-rspec/helpers/serverspec'

...
hosts.each do |host|
  install_puppet
end
...

UNSUPPORTED_PLATFORMS = ['windows','AIX','Solaris']
RSpec.configure do |c|
  proj_root = File.expand_path(File.join(File.dirname(__FILE__), '..'))
  c.formatter = :documentation
  c.before :suite do
    hosts.each do |host|
      copy_module_to(host, :source => proj_root, :module_name => 'module')
      
      ...
      
      on host, puppet('module','install','puppetlabs-stdlib'), { :acceptable_exit_codes => [0,1] }
      on host, puppet('module','install','puppetlabs-concat'), { :acceptable_exit_codes => [0,1] }
    end
  end
end

Acceptance Tests

The acceptance tests describe the expected behaviour and state of the SUT and are written in RSpec. In this example a test context is defined for user (and group) creation.

require ‘spec_helper_acceptance'

describe 'module defintion', :unless => UNSUPPORTED_PLATFORMS.include?(fact('osfamily')) do

  context 'user is created with a valid input parameter’ do
    it 'should work with without errors’ do
      pp = <<-EOS
        $theuser = {
        'auser' => { 'shell' => '/bin/bash' }
      }
      module { 'auser':
        user => $theuser
      }
      EOS
      apply_manifest(pp, :catch_failures => true)
      expect(apply_manifest(pp, :catch_failures => true).exit_code).to be_zero
   end
   
   describe user('auser') do
    it { is_expected.to exist }
    it { is_expected.to belong_to_group 'auser' }
   end
   
   describe group('auser') do
      it { is_expected.to exist }
   end
    
 end
end

This test context applies the manifest, expects it to execute without errors and then validates that a user is created in the correct group.

Tests can be run with

bundle exec rspec spec/acceptance

Test Execution

Beaker will initially use the specified hypervisor to bring up a SUT.

Hypervisor for ubuntu-server-1404-x64 is vagrant
Beaker::Hypervisor, found some vagrant boxes to create
created Vagrantfile for VagrantHost ubuntu-server-1404-x64
...

Puppet will then be installed and the acceptance tests within spec/acceptance will be performed.

Generally each test applies manifests within the module for a particular context.

ubuntu-server-1404-x64 $  mktemp -t apply_manifest.pp.XXXXXX  
/tmp/apply_manifest.pp.ae1dqd

ubuntu-server-1404-x64 executed in 0.01 seconds
localhost $ scp /var/folders/b4/whdj5tqn1_n_2783hl5cnvch0001bn/T/beaker20141123-1076-b25en7 
ubuntu-server-1404-x64:/tmp/apply_manifest.pp.ae1dqd {:ignore => }

ubuntu-server-1404-x64 $  env PATH="/usr/bin:/opt/puppet-git-repos/hiera/bin:${PATH}" 
RUBYLIB="/opt/puppet-git-repos/hiera/lib:/opt/puppet-git-repos/hiera-puppet/lib:${RUBYLIB}" 
puppet apply --verbose --detailed-exitcodes /tmp/apply_manifest.pp.ae1dqd  

[...Puppet Logging...]

ubuntu-server-1404-x64 executed in 6.22 seconds
Exited: 2

The tests are then performed. Beaker resolves them to an underlying operating system command in order to validate the assertion

User "auser"

ubuntu-server-1404-x64 $  id auser  
uid=1001(auser) gid=1001(auser) groups=1001(auser)

ubuntu-server-1404-x64 executed in 0.01 seconds
      should exist []
      
ubuntu-server-1404-x64 $  id auser | awk '{print $3}' | grep -- auser  
groups=1001(auser)

ubuntu-server-1404-x64 executed in 0.01 seconds
      should belong to group "auser"
    Group "auser"

ubuntu-server-1404-x64 $  getent group auser  
auser:x:1001:

ubuntu-server-1404-x64 executed in 0.01 seconds
      should exist []

The hypervisor will then shutdown and destroy the SUT. Beaker reports on the results when all the tests have completed.

Destroying vagrant boxes
==> ubuntu-server-1404-x64: Forcing shutdown of VM...
==> ubuntu-server-1404-x64: Destroying VM and associated drives...

Finished in 3 minutes 8.1 seconds
20 examples, 0 failures

Summary