Ton Kersten

Puppet environments

For my job I do a lot of Puppet and I thought it was about time to write some tips and tricks down.

First part of this post is about my environment setup. In my test setup I use a lot of environments. They are not at all useful, but that’s not the point. It’s my lab environment so things need to break once in a while. But with multiple environments Puppetlabs says that you should switch to directory environments (PuppetDoc) but some way or another I cannot get that to work in a good way with my PE version (3.4.3 (Puppet Enterprise 3.2.3)). So I started implementing dynamic environments, which is a simple way of specifying the directories for your environments.

Part of my puppet.conf looks like

[master]
    environment = production
    manifest    = $confdir/environments/$environment/manifests/site.pp
    manifestdir = $confdir/environments/$environment/manifests
    modulepath  = $confdir/environments/$environment/modules:/usr/share/puppet/modules
    templatedir = $confdir/environments/$environment/templates

So, my default environment is production and a client can specify another environment to be in. The command

puppet agent --environment=test

would place this node in the test environment. A simple module places a new puppet.conf file on the client stating this new environment. Couldn’t be more simple.

Well, that’s what you think. But what if you need to deploy 10.000+ hosts of which there are about a third in environment test and about a 1000 in environment development? It would take a lot of time to ssh into all these servers and run Puppet with the correct environment.

There has to be a way around that. And, of course, there is. In Puppet version 3 and up Hiera is integrated into Puppet and we already use that a lot. Why not integrate the environment in Hiera? Well, our hiera.yaml is now:

---
:hierarchy:
    - "%{environment}/hiera/%{::fqdn}"
    - "%{environment}/hiera/%{::hostname}"
    - "%{environment}/hiera/%{::domainname}"
    - "%{environment}/hiera/%{::systemtype}"
    - "%{environment}/hiera/%{::osfamily}"
    - "%{environment}/hiera/common"

:backends:
    - yaml

:yaml:
    :datadir:
        /etc/puppetlabs/puppet/environments

This challenges me with a chicken and egg problem. To get the environment I need to know the environment. But what if I make Hiera into an ENC and let this one deliver the environment? Can this be done? Yes, it can.

This is how I did it:

First create a part of the Hiera structure that’s not in the current environment, for example like this:

---
:hierarchy:
    - "hiera/%{::fqdn}"
    - "hiera/default"
    - "%{environment}/hiera/%{::fqdn}"
    - "%{environment}/hiera/%{::hostname}"
    - "%{environment}/hiera/%{::domainname}"
    - "%{environment}/hiera/%{::systemtype}"
    - "%{environment}/hiera/%{::osfamily}"
    - "%{environment}/hiera/common"

:backends:
    - yaml

:yaml:
    :datadir:
        /etc/puppetlabs/puppet/environments

And in the directory /etc/puppetlabs/puppet/environments/hiera I place a very small file, called default.yaml, which contains:

---
environment: 'production'

This makes sure that any node without a specific file, will get the production environment. This is the default for Puppet as well, so nothing changes for that.

To test this, run:

hiera environment ::fqdn=$(hostname -f)

This will give you something like environment: production. For every host in another environment as the production one, create a small file named the FQDN of the host with the contents stating the wanted environment.

(Watch for the :: in front of the fqdn. This means that the fqdn variable is a top scope variable, as all facter variables are.

Now integrate this into Puppet. First create a little script that executes the command above and returns the wanted output.

My script is called getenv and placed in /etc/puppetlabs/puppet/bin

#!/bin/bash

penv="$(/opt/puppet/bin/hiera                       \
            -c /etc/puppetlabs/puppet/hiera.yaml    \
            environment ::fqdn="${1}")"

echo "environment: ${penv}"

This returns a string like environment: production.

And last, but not least, place this settings in the [master] of your puppet.conf

node_terminus  = exec
external_nodes = /etc/puppetlabs/puppet/bin/getenv

It took some work to get things started, but a small shell thingy read the file with all 10.000+ hosts and required environments, that created all the Hiera files for all nodes that are not in the production environment.

Just one thing to do: When I have a lot of host-files in a single directory, this could become slow. I could place all definitions in a simple database, but things would get complicated again, and that’s not what I want. I also could split things up per letter, but I’m not sure yet if I really want that.

When I have resolved this, this entry will be continued.

Puppet Facter Fact

Look at me, I made a Puppet Facter Fact!!!

With a lot of thanks to Andrew Beresford who started the initial code. I just tweaked it.

What it does is rather simple, it finds the expiration date of the SSL certificate of this host and returns the expiration date and time when there are less than 30 days left. Otherwise it just returns a --sign. In the Puppet manifest I check if it’s this --sign and if not I generate a warning.

This is it:

#
# Set the Facter-Fact "certificate_expiry" to the SSL certificate
# expiration date and time.
#
# Usage example:
# --------
#   if "${::certificate_expiry}" != "-" {
#       notify { 'CertExp' :
#           message  => "Certificate expire date for ${::fqdn}: ${::certificate_expiry}",
#           withpath => false,
#       }
#   }

#
# $Id$%
# $URL$
#
Facter.add("certificate_expiry") do
    setcode do
        warndays = 30
        time = Puppet::SSL::Host.localhost.certificate.expiration
        warn = time - ( warndays * 60 * 24 )

        if ( warn - Time.now ) < 0
            time = time.strftime("%Y-%m-%d %H:%M:%S")
        else
            time = "-"
        end
        time
    end
end

Me proud, I am

Puppet User Group

Yesterday I attended the first meet up of the Dutch Puppet User Group and I gave a talk about how to start with Puppet.

It was called: “Puppet deployment, an introduction” and the PDF slideshow can be viewed or downloaded from speakerdeck.

If you have any comment, please send me an email.

Puppet updates

When working with Puppet and a VCS (like git and SVN) it's nice to have a simple way of updating the Puppet tree.

My tree is always in /etc/puppet and owned by user and group puppet. User puppet is allowed to checkout the complete tree from git or subversion.

I have created two one-liners to update the complete tree and make sure all rights are still correct.

update_svn

#!/bin/bash
# update_svn
su - puppet -c 'cd /etc/puppet; svn up; cd doc; ../bin/gendoc'

update_git

#!/bin/bash
# update_git
su - puppet -c 'cd /etc/puppet; git pull; cd doc; ../bin/gendoc'

But, of course, it's not handy to type update_git today and update_svn tomorrow. And I also don't want a path to /etc/puppet/bin.

The solution is a very simple one, as always:

cd /usr/local/bin
ln -s /etc/puppet/bin/update_git pupdate

and now I only have to type pupdate and things work out.

Updated Pygments

I'm using Pygments for quite some time now and I just noticed there was a new version available (1.5). I installed that and I was wondering if there would be a lexer included for Puppet. Well, it wasn't, but a short Google action directed me to the Pygments lexer for the Puppet DSL.

Of course my old CentOS 5 system with Python 2.6 doesn't want to install this, so I hacked the Puppet lexer into Pygments.

Here's an example of the result:

class generic::ssh {
    $ssh_service       = hiera("ssh_service")
    $ssh_packages      = hiera("ssh_packages")
    $ssh_debug         = hiera("ssh_debug",         "undef")
    $permit_root_login = hiera("permit_root_login", "no")
    $ssh_users         = hiera_array("ssh_users",   "undef")
    $ssh_groups        = hiera_array("ssh_groups",  "undef")

    package { $ssh_packages:
        ensure => present,
        before => File["/etc/ssh/sshd_config"],
    }

    file { "/etc/ssh/sshd_config":
        ensure  => present,
        content => template("generic/sshd_config.erb"),
        notify  => Service["${ssh_service}"],
    }

    service { $ssh_service:
        ensure     => running,
        enable     => true,
        hasrestart => true,
        hasstatus  => true,
    }
}

and an example of the Hiera Yaml file:

---

# SSH Settings
permit_root_login    : 'no'
ssh_service          :
                       - 'sshd'
ssh_users            :
                       - 'root'
                       - 'tonk'
ssh_groups           :
                       - 'wheel'
ssh_packages         :
                       - 'openssh'
                       - 'openssh-clients'
                       - 'openssh-server'

Nice

FreeBSD PXE boot Part 2

Some posts ago I wrote that I was busy to find out how a FreeBSD machine can be PXE-ed from a Linux server. Well, I found that some time ago, but I didn't have the time to type it here, yet. Well, as always, once you know how it's done, it's quite simple. But because a lot of the FreeBSD documentation is very old (talking about FreeBSD 4, 5 and 6) it takes some time to find it all.

Read more »

Why does Puppet keep breaking?????

In my previous post I stipulated that I was PXE booting FreeBSD. Well this works and I will come back on that. But for the configuration I want to run Puppet. Nice and easy config management.

On my server I run Puppet from source. This because the server is a CentOS box with a very old Ruby and Puppet. So I decided to run the Puppet client from source as well. Getting the git repo is easy enough and installing Puppet should not be to hard.

Well, well, how wrong have I been. Every time I update the Puppet client or server something breaks. And I do mean every time.

First it started with not parsing templates correctly. A couple of hours of debugging solved that, but then Puppet started crying with " Error 400 on SERVER: No support for http method POST". W.T.F. does it mean. This somehow got solved, but then the templates broke again. The Puppetlabs site stated (as always) to update to the newest version. So I did. And the template error was back again, but now it was a different one: "Failed to parse template issue/issue.erb: undefined method 'first' for "/etc/puppet/modules/issue/templates/issue.erb:19:in". Tinkering around for 4 (yes four!) hours solved this one (I can hardly remember what I tried, but I can assure you that I have seen all sites about Puppet that exist on the Inernet. Including the ones about handpuppets). And then I got the 400 error again. Running in debug mode doesn't help either, so I'm rather stuck. Man, do I hate this type of behaviour. Be stable or go away! I now completely had it with diving into Puppet sources to find the culpritt. If they are still seeking for a miracle for Pope John Paul II, maybe a stable Puppet client would be a good idea.

I'm getting rather fed up with this stuff. Ths way I'll never be able to update a server and be sure it will work. Maybe CFEngine3 is a better option!

umask per directory

Some users insist on using bash. This is a good shell, but not as good as zsh. But, I do want them to be able to use the per directory umask as well as all the zsh users.

So I started digging, as the bash shell does not support a chpwd hook.

This is what I came up with:

chpwd()
{   # Set the initial umask
    case "${PWD}/"
    in
        /etc/puppet/*)
            um=$(umask)
            umask 007
        ;;
        *)
            [[ x"${um}" != x"" ]] && umask ${um}
        ;;
    esac
}
function cd()
{
    builtin cd "${@}"
    chpwd
}

Now, when I change to the directory /etc/puppet I do get a umask of 007 and when I cd somewhere else, I do get the original umask.

I do redefine the intercal cd command to run the chpwd hook. There must be a more elegant way to do this, but this does the job.

umask per directory

I've been working with Puppet some time now, and we are configuring our way through a lot of hosts, with 6 persons, all working in the same Puppet master directory.

This should work fine with all UNIX/Linux groups and setgid directories. But simple problem arose with the git version control stuff.

Once in a while the complete git repo was destroyed and quite a lot of searching revealed the reason why.

We are all working as non-root and we are all members of the Puppet group. But: When I edit a file and commit it, the corresponding files in the git repo are made by me and the rights are set according to my umask. When someone else tries to edit the same file or something else which results in the same hash files, writing is not permitted, because of my ownership. A chown in a script will not work, as a chown is not honored as a non-root user.

This problem can simply be solved by setting the umask to something like 007 (or u=gwx,g=gwx,o=). But when I do edit stuff in my home-directory I do not want an open umask like that. So what to do, as ext[234] do not support per directory umasks.

I use zsh as a shell and I found a nice function in the man-page. There is a standard function, called chpwd() that gets executed every time a directory change is made. So I only had to fill in the blanks.

This is what I came up with:

chpwd()
{
    case "${PWD}/"
    in
        /etc/puppet/*)
            [[ ${UMSAVE} = 0 ]] &&
            {   um=$(umask)
                UMSAVE=1
            }
            umask 007
        ;;
        *)
            [[ x"${um}" != x"" ]] && umask ${um}
            UMSAVE=0
        ;;
    esac
}

Now, when I change to the directory /etc/puppet I do get a umask of 007 and when I cd somewhere else, I do get the original umask.

How much fun can it be