Puppet & Mcollective

How-To: Install Mcollective & Puppet on EC2

by nathanmc@

=======

EC2 Prep

1. Create security groups with the following rules:

   jump - tcp/22 - 0.0.0.0/0
   puppet-cli
   mcoll-cli
   rabbit
   mcoll-queue - tcp/6163 - mcoll-cli
   puppet-master - tcp/8140 - puppet-cli
   

   Set Up RabbitMQ

2. Instantiate a 64-bit m1.large of Ubuntu 12.04 LTS

   Security Groups:
   * rabbit
   * mcoll-queue
   * puppet-master
   * jump
   

3. SSH to the rabbitmq host and sudo su -

4. Execute:

   apt-get install -y erlang-base erlang-nox
   wget http://www.rabbitmq.com/releases/rabbitmq-server/v2.8.1/rabbitmq-server_2.8.1-1_all.deb
   dpkg -i rabbitmq-server_2.8.1-1_all.deb
   rabbitmq-plugins enable amqp_client
   rabbitmq-plugins enable rabbitmq_stomp
   

5. Edit /etc/rabbitmq/rabbitmq.config:

   [
       {rabbitmq_stomp, [{tcp_listeners, [{"0.0.0.0", 6163},
                                          {"::1",       6163}]}]}
   ].
   

6. Execute:

   rabbitmqctl add_user mcollective PASSWORD
   rabbitmqctl set_user_tags mcollective administrator
   rabbitmqctl set_permissions -p / mcollective ".*" ".*" ".*"
   /etc/init.d/rabbitmq-server restart
   

7. Sanity check - Ensure it's now listening on 6163:

   netstat -anp | grep -v unix
       ...
           tcp    0  0 0.0.0.0:6163       0.0.0.0:*           LISTEN      1240/beam 
           tcp6   0  0 ::1:6163           :::*                LISTEN      1240/beam
   

   If you don't, check your history to ensure you enabled the rabbitmq_stomp plugin

   Also look at /var/log/rabbitmq/startup*_log files for clues as to why it may not have started

8. Leave your terminal open and run

   tail -f /var/log/rabbitmq/rabbit@*hostname*.log
   

   This will show you the connections as they come in from the clients

   Set Up Mcollective

9. Instantiate a 64-bit client host of either Ubuntu 12.04 LTS or Amazon Linux AMI 2012.03

   Security Groups:
   * jump
   * puppet-cli
   * mcoll-cli
   

10. SSH in and apply all updates

    Amazon:
        sudo su -
        yum update
        shutdown -r now
    
    Ubuntu
        sudo su -
        apt-get update
        apt-get upgrade
        shutdown -r now
    

11. SSH in and execute:

    Amazon:
        sudo su -
        edit /etc/yum.repos.d/epel.repo
            set all to enabled=0
        yum install rubygem-stomp
        rpm -ivh http://yum.puppetlabs.com/el/6/products/i386/puppetlabs-release-6-5.noarch.rpm
        yum install mcollective mcollective-client
    

12. Make the following edits:

    /etc/hosts - add the line:
            <internal ip of your rabbit host>   puppet
    
    /etc/mcollective/server.cfg:
        plugin.psk = 01234    
            (NOTE:  Pre-Shared Key - Can be any value consistent across all)
        plugin.stomp.host = puppet
        plugin.stomp.port = 6163
        plugin.stomp.user = mcollective
        plugin.stomp.password = PASSWORD
    
    /etc/mcollective/client.cfg:
        plugin.psk = 01234    
            (NOTE:  Pre-Shared Key - Can be any value consistent across all)
        plugin.stomp.host = puppet
        plugin.stomp.port = 6163
        plugin.stomp.user = mcollective
        plugin.stomp.password = PASSWORD
    

13. Restart mcollective

    /etc/init.d/mcollective start
    

14. Test mcollective via the following command:

    mco find hosts
    

    Should see your localhost's name

    If not, double-check that all the passwords and psk's are the name in client and server cfg's

    Logs are written to /var/log/mcollective.log

    You should see connections listed in the log on rabbitmq

    Setup Puppet

    NOTE: This guide works for puppet 2.6.x CLIENTS. Be careful about what version you're installing because most repos have more than one.

15. Back on the rabbitmq server, execute:

    apt-cache policy puppetmaster
    

    You should see output similar to the following

    puppetmaster:
      Installed: (none)
      Candidate: 2.7.11-1ubuntu2.1
      Version table:
         2.7.11-1ubuntu2.1 0
            500 http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ precise-updates/main amd64 Packages
            500 http://security.ubuntu.com/ubuntu/ precise-security/main amd64 Packages
         2.7.11-1ubuntu2 0
            500 http://us-west-2.ec2.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages
    

    For our purposes, a 2.7 server will work fine, so install it with:

    apt-get install puppetmaster
    cd /etc/puppet
    mkdir files
    cd manifests
    

    Edit site.pp

        #/etc/puppet/manifests/site.pp
        import "nodes"
        filebucket { main: server => "<resolvable host name of self>" }
    
        #defaults
        File { backup => main }
        Exec { path => "/usr/bin:/usr/sbin:/bin:/sbin" }
    

    Edit nodes.pp

        # /etc/puppet/manifests/nodes.pp
    
        # We're just going to put a simple example
        node default {
          exec { "touch_file":
          command => "touch /tmp/stamped.txt",
          path    => "/usr/local/bin/:/bin/",        }
        }
    

    Edit /etc/puppet/autosign.conf (just an asterisk)

        *
    

    Restart the puppet daemon:

    /etc/init.d/puppetmaster restart
    

16. On each puppet client:

    Amazon Linux:

        yum info puppet
            ...
                Available Packages
                Name        : puppet
                Arch        : x86_64
                Version     : 2.6.16
                Release     : 1.6.amzn1
                Size        : 843 k
                Repo        : amzn-updates
                Summary     : A network tool for managing many disparate systems
                URL         : http://puppetlabs.com
                License     : GPLv2
        yum -y install puppet
        wget https://s3.amazonaws.com/trnsfr/nsmc-mco-puppetd-1.0.0-1.x86_64.rpm
        rpm -ivh nsmc-mco-puppetd-1.0.0-1.x86_64.rpm
        wget https://s3.amazonaws.com/trnsfr/nsmc-mco-facter-facts-1.0.0-1.x86_64.rpm
        rpm -ivh nsmc-mco-facter-facts-1.0.0-1.x86_64.rpm
    

    Edit the mcollective configs:

    /etc/mcollective/client.cfg:
        # Facts
        factsource = facter
    
    /etc/mcollective/server.cfg:
        # Facts
        factsource = facter
    

    Restart mcollective:

    /etc/init.d/mcollective restart
    

Symfony Deployment Gotcha

So I've been working with Symfony in the PHP arena and I found a really obnoxious gotcha that lives at the intersection of Symfony-based tutorials and Redhad-based distros. (In this case, CentOS.)

Namely, Symfony creates a rather expansive directory structure that - for security reasons - can't all live under your web root. The solution all the tutorials have you do is to set up the actual project in your home directory and then create an Apache virtual server that points to it. Seems simple enough. And you can add it to your config without any griping from Apache. Symfony has all the global perms set, so you're good, right?

Wrong, unfortunately. I was getting "403 Forbidden" over and over. I tried a wide variety of fixes, none of which worked. Eventually, I found this post on google groups:

http://bit.ly/mNRrlV

Sure enough, I checked the root perms of my home directory and it was all tightened down:

drwx------ 17 user user 4096 May 30 06:16 user

Just goes to show you the danger of returning to "childlike innocence" with new projects.

In reality, the right way to approach this is to create a project-specific directory somewhere under /usr/local and consciously set perms for the whole thing.

Following that approach, you get the right results straight out of the gate:

[root@localhost local]# ls -ld projects/
drwxr-xr-x 3 root root 4096 May 30 07:17 projects/

[root@localhost local]# ls -ld projects/milkshake/
drwxr-xr-x 11 root root 4096 May 30 07:18 projects/milkshake/

Now, you create your Symfony project under that:

symfony generate:project --orm=Propel milkshake

And your app:

symfony generate:app frontend

Finally, you add the following to your httpd.conf:


# Be sure to only have this line once in your configuration

NameVirtualHost *:8080



# This is the configuration for your project

Listen *:8080

<VirtualHost *:8080>

DocumentRoot "/usr/local/projects/milkshake/web"

DirectoryIndex index.php

<Directory "/usr/local/projects/milkshake/web">

AllowOverride All

Allow from All

</Directory>


</VirtualHost>


This is a good lesson for everybody: please check what you write and all your other assumptions, as well.

swatch queues messages

Swatch is a fantastic tool for tailing logs for errors. But it has a surprising built-in behavior: it forcefully queues the outbound mails it generates for 30 minutes. Obviously, this stuff can be time-sensitive. I bugged some enlightened folks on the IRC channel and I got this excellent suggestion:

Create a job in root's cron to have exim flush the queue for the destionation mail addy at whatever interval you want. Here's one that does it every five minutes:

*/5 * * * * /usr/sbin/exim -qR

This solution is particularly elegant because you don't need to dig around inside swatch's guts. (Something you should never do with a package for a whole host of reasons.)

Make your own daemon

A fantastic article on swatch (http://www.linux-mag.com/id/7807) included this delightfully simple framework for rolling your own daemons. If you take a moment to read it, you'll see a lot of unix design principles relating in perfect harmony:

#!/bin/sh 

# Simple Log Watcher Program  

case "$1" in 'start')   

/usr/bin/swatch --daemon --config-file=/etc/swatch.conf --tail-file=/var/log/auth.log --pid-file=/var/run/swatch.pid   

;; 

'stop')   PID=`cat /var/run/swatch.pid`   kill $PID   

;; 

*)   echo "Usage: $0 { start | stop }   

;; 

esac 

exit 0 

Unattended Ubuntu Apt Installs

This will be short and sweet.

When you want to

apt-get install package

and you don't want to get prompted for values, etc, just add the following the front of your shell:

export DEBIAN_FRONTEND=noninteractive

Poof. Magic.

Grepping out comments and blank lines

One thing we commonly see in open-source configs is an abundance of helpful comments. Often, they dwarf the number of active parameters being set and that can make things very hard to read.

In the blessed world of unix, there is an easy solution to this: you simply use grep to remove those lines from a listing of the contents of the file.

First, grep out the comments:

grep -v \# file

Usually, that will leave you with a lot of unnecessary whitespace, as well. So just pipe that output through an enhanced grep and filter for blank lines:

grep -v \# file | egrep -v "^$"

To clarify the above, here's a break down of each component:

grep - self explanatory

-v - tells grep to remove what matches rather than show it

\ - tells the shell the special character # is to be passed to the process without interpretation

# - the near-universal symbol for "comment" in config-ese

file - a stand in for whatever conf you're digging through

| - a pipe. Tells the shell to send the output of what's to the left of the pipe to what's on its right

egrep - grep with extended regular expressions (a sophisticated text manipulation language)

-v - telling egrep to remove the matches

"" - passing what's inside to grep without shell interpretation

^ - Means "the start of the line"

$ - Means "the end of the line"

So what you're telling it, in essence, is the following:

1. Read the contents of file
2. Remove all lines with "#"
3. Pass the result to egrep
4. Egrep: remove all lines which begin and immediately end (ie, a blank line) and output the result

This is one of those little tricks that goes a long way to simplifying interactions with Unix hosts and it's actually the first thing I do when I go to read any new service's default config.

Hope it helps.

Subversion Set Up

In a nutshell, ssh to the host and run

su

apt-get install subversion

exit

Then, as your own user, run

svnadmin create [repository name]

Note that if you want to share this repository with other users, you should create a group you will all share and create a special directory that you will give the group full RW access to.

For instance:

groupadd svn

[add your users to the svn group in /etc/groups]

su - [yourself]

groups

You should see "svn" listed.

It's important to remember that even though the group has RW access to your new directory, they won't have access to the repository by default. So give it to them explicitly:

chown -R .svn [repo]

chmod -R g+rw [repo]

Now have your users do a checkout with their ssh accounts:

svn co svn+ssh://[user]@[ip]/path/to/repo

(They'll be prompted for their password twice.)

or

svn co file:///path/to/repo

on the same host.

And a commit

svn commit

(Make sure everyone's commenting their commits)

If you see this:

svn: Can't open file '/home/svn/res/db/txn-current-lock': Permission denied

Make sure they're logged in as the appropriate group and that the perms are correct.