Subversion Set Up

In a nutshell, ssh to the host and run

su

apt-get install subversion

exit

Then, as your own user, run

svnadmin create [repository name]

Note that if you want to share this repository with other users, you should create a group you will all share and create a special directory that you will give the group full RW access to.

For instance:

groupadd svn

[add your users to the svn group in /etc/groups]

su - [yourself]

groups

You should see "svn" listed.

It's important to remember that even though the group has RW access to your new directory, they won't have access to the repository by default. So give it to them explicitly:

chown -R .svn [repo]

chmod -R g+rw [repo]

Now have your users do a checkout with their ssh accounts:

svn co svn+ssh://[user]@[ip]/path/to/repo

(They'll be prompted for their password twice.)

or

svn co file:///path/to/repo

on the same host.

And a commit

svn commit

(Make sure everyone's commenting their commits)

If you see this:

svn: Can't open file '/home/svn/res/db/txn-current-lock': Permission denied

Make sure they're logged in as the appropriate group and that the perms are correct.

File Encryption with OpenSSL

If you're like me, you're occasionally forced to mail some pretty sensitive things around. When that happens, it's not enough to say "Well, I deleted the message." The message is still in your trash, on a server's disk, etc, etc. You need to assume someone WILL get ahold of it and take measures to prevent them from doing anything with it.

My personal favorite (since it can be used in any situation) is the openssl file encryption capability. The openssl package is almost universally installed by default in distros these days, so you should just be able to jump straight to the commands.

In this case, I've got a file called "spreadsheet.xls" that I want to password encrypt. The syntax is simply this:

openssl aes-256-cbc -a -salt -in spreadsheet.xls -out spreadsheet.xls.enc

Then just supply the password you want to use.

And when you want to decrypt it, just add the '-d' switch and flip the filenames:

openssl aes-256-cbc -d -a -in spreadsheet.xls.enc -out spreadsheet.xls

And use your password.

Just like that, you've got munition-grade encryption from the command-line.

Enjoy

Zenoss Quirks

So I ran into two really obnoxious issues that took a considerable amount of digging to resolve:

1. It was marking my apache process monitor as failed/recovered randomly and often.
2. It would not let go of misapplied process monitors that had been picked-up by overly liberal regexes. It would include:

tail -f /var/log/nginx/access.log
nginx: worker

in the nginx process monitor if I was tailing the log when I modeled the host. The problem was, after I killed the tail, the process was alerting as "Process Not Running". FOREVER. Even if I deleted and recreated the process monitor, the host, the events, everything.

In the first case, it turns out that since apache marks its process as "apache defunct" when it's shutting down a child process, Zenoss would occasionally pick this up as a live apache process. It would then mark it as "Down" after the proc terminated. The solution for this was to make my regex more specific:

apache2 \-k start

The second case was much more obnoxious because not only would the events not clear, they would return and begin alerting every time the device was recreated.

After some digging online, it seemed that the best course was to restart the zenprocess daemon.

This is best done under Settings > Daemons. You can also view the logs there (which showed the bad checks prior to the restart and nothing after).

When that's complete, re-add your device and you should be rid of the baggage.

Tethering your iPhone in Ubuntu

This is a delightfully easy one. First, add this repository and update your apt:

sudo add-apt-repository ppa:pmcenery/ppa

sudo apt-get update

Now install the following three packages (you may have to go dig for them):

libimobiledevice-utils
ipheth-dkms
ipheth-utils

Restart your machine.

Turn on tethering inside your iPhone's General > Network menu.

Cable your iPhone to a USB port.

It should automatically connect. You'll see it in Network Manager.

Poof. Magic.

Enabling SSH Agent Forwarding in OS X

If you use EC2, you use SSH keys.

If you're sane, you keep the private keys on your workstation and forward them through the chain of public keys throughout your hosts.

I recently started working through OS X again and found some pretty obnoxious behavior: out of the box, the ssh keys don't forward. After some digging, I found the following discussion of the subject:

http://data.agaric.com/node/3061#comment-1604

Long story short, it appears that the location of the user's home directory isn't communicated when the keys are forwarded so the agent looks in the wrong place.

The fix? Run these two on your OS X machine:

ssh-add

ssh-add -l

et voila

Try again.

Removing packages from Debian/Ubuntu

Just a quick note so I don't waste time revisiting this topic again.

A coworker of mine forced a package install without repositories so all the dependencies obviously failed. Trying to remove it, I did a

dpkg --remove

But running an apt-get said it was still installed. Running

dpkg --list

Gave me a status of "rc" - which left a lot of files laying around. (Configs, etc.)

Finally, I just did a

dpkg --purge

And it cleared out all the junk.

From there, I was able to do a normal apt-get against the repositories.

Using Exim as a Mail Relay

One service that always seems required by any platform I've ever worked on is an outbound mail transfer agent. Essentially, everybody wants to send their customers notifications and needs a clearinghouse to pass it through.

This is all well and good, but there are a few inevitable problems one is likely to run into when putting one up.

By far, I've had the most success with Exim running on Debian since both are very lightweight, incredibly reliable and closely intertwined. (Exim is Debian's default MTA.)

Step 1 is to provide the basic setup. Log in, su to root and run the following:

dpkg-reconfigure exim4-config

You'll walk through some basic setup info. Essentially, you want to specify the host as an internet mail server and provide the ip addresses of your internal hosts are valid hosts to relay for.

When that's complete, you need to cd into /etc/exim4 and modify the exim4.conf.template file. Find the rewrite section by looking for

begin rewrite

Then add your rewrite rule under it. This will make it apply to all messages that pass thorugh Exim.

The generic rule I usually use looks like this:

*@* $1@mycompanysdomain Ffrs

What it essentially says is "match everthing (*@*) and replace what's in front of the @ with what was to the left of the original and everything after the @ with with explicit domain name - also, apply it to the following types of addresses: F - envelope From field, f - From header, r - Reply-To header, s - Sender header"

With that rule in place, restart Exim:

invoke-rc.d exim4 restart

And the config will be written out to /var/lib/exim4/configuration.autogenerated.

The best way to test is to use the Exim command line to see what rewrites will get applied:

# exim -brw root@localhost
sender: root@mycompanysdomain
from: root@mycompanysdomain
to: root@localhost
cc: root@localhost
bcc: root@localhost
reply-to: root@mycompanysdomain
env-from: root@mycompanysdomain
env-to: root@localhost

With that done, you need to take care of the blacklists. First, add an A record to your domain's authoritative DNS WITH A PTR. The reverse lookup is used as a method to ensure the mail server is a legitimate member of your domain. NOTE: A lot of people think MX records are required but that's incorrect. MX records are for inbound mail only.

Now fire off some messages to big mail destinations like gmail and yahoo. Tail the /var/log/exim4/mainlog to see what results you get. Yahoo especially is good about telling you where your IP is blacklisted for some reason. If it is, you'll get a lot of dropped messages till you contact the Blacklister and petition to have it removed.

On that same note, be careful about sending out a bunch of unsoliticed test messages. I've seen hosts get black-listed for that reason alone and it's a pain to get delisted.

Checking for Root

Good programmers know to check for assumptions their code is making before executing it. For an operator, assuming that you've su'd to root before running an installer you've written is a bad idea.

The easiest way to resolve that is to simply use the EUID bash variable to check the effective user-id you're executing under and then exit if it's not zero.

I cribbed the following

if [[ $EUID -ne 0 ]]; then
echo "This script must be run as root"
exit 1
fi

From this excellent article:

http://www.cyberciti.biz/tips/shell-root-user-check-script.html

Awk Alternative Delimiters

As it's always best to automate, it's often useful to fill the values of shell variables with the returned values of system commands.

For instance, if you wanted the ip of eth0, you could run

/sbin/ifconfig eth0

And you'd get back something like

[nsmc@nsmc-dt automation]$ /sbin/ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:26:F2:AC:C2:FE
inet addr:10.1.1.99 Bcast:10.1.3.255 Mask:255.255.252.0
inet6 addr: fe80::226:f2ff:feac:c2fe/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1142141 errors:0 dropped:0 overruns:0 frame:0
TX packets:223927 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:212209959 (202.3 MiB) TX bytes:205106885 (195.6 MiB)
Interrupt:18 Base address:0x2000

Then you could vi into your config script and export the value to a var manually.

-OR-

You could set the variable with the returned value of a system command after clearing away some noise.

Using an iterative approach, we could peel back each layer till we get the piece we want:

/sbin/ifconfig eth0 | grep "inet "

That gives us the "inet" line and not the "inet6" line.

[nsmc@nsmc-dt automation]$ /sbin/ifconfig eth0 | grep "inet "
inet addr:10.1.1.99 Bcast:10.1.3.255 Mask:255.255.252.0

But we only want the piece behind the first colon. Here we need to use a little awk trickery. awk uses spaces for it's default field delimiters. Let's change that to a colon (:) and see what we get.

[nsmc@nsmc-dt automation]$ /sbin/ifconfig eth0 \
| grep "inet " | awk -F \: '{print $2}'



10.1.1.99 Bcast

Close. Now we can pass it through a standard awk filter and just get the piece we want:

[nsmc@nsmc-dt automation]$ /sbin/ifconfig eth0 | grep "inet " \
| awk -F \: '{print $2}' | awk '{print $1}'



10.1.1.99

Now we just need to assign to our variable at runtime using the backtic's:

export PRIVATE_HOST_IP=`/sbin/ifconfig eth0 | grep "inet " \
| awk -F \: '{print $2}' | awk '{print $1}'`

Escaping Single-Quotes in sed

This is a just a re-post of this excellent article by Stuart Coleville:

Using single quotes in BASH ensures that the shell doesn’t expand the contents of the quoted string and this is useful most of the time. However if you want to use single quotes within a single quoted string things don’t work out as you might expect.

If I want to use sed to match some text with a single quotes in it, I will run into trouble if I run:

sed 's/user \= 'root/user \= 'moi/g'

Alternatively if I run (on the same snippet):

sed 's/user \= \'root/user \= \'moi/g'

I will get:

/bin/bash: -c: line 1: unexpected EOF while looking for matching `''
/bin/bash: -c: line 2: syntax error: unexpected end of file

This doesn’t work because the escaped single-quotes (\') are not expanded and are therefore treated literally.

To single quotes work you need to break out of the single quoted string then escape your single quote. Like so:

sed 's/user \= '\''root/user \= '\''moi/g'

Because \' is not inside of single quotes the single-quote is properly escaped and the output is as we’d expect:

user = 'root' ->  user = 'moi'

In conclusion, the title of this post is a bit of a misnomer. You actually can’t put single quotes inside of a single-quoted string. However breaking out allows us to get to where we want to be.

Installing Amazon EC2 tools to Debian Lenny

I was reading through Bill Childers excellent article in Linux Journal on running Ubuntu under EC2 and I thought I add my notes on how to make the client tools run on Debian.

First, install the Sun JDK (see my previous post on how to do that).

With that done, download the tools from this page:

AWS Developer Community

Or simply by using this link:

ec2-api-tools.zip

Next, unzip them.

Then create a configuration file to export the proper environment variables when your shell fires up:

$ vi ~/.ec2.conf
Add the lines:

export EC2_CERT=whereveryourx509lives/cert-whatever-x509.pem
export EC2_PRIVATE_KEY=whereveryourx509pklives/pk-whatever-x509.pem
export JAVA_HOME=/usr/lib/jvm/java-6-sun/jre
export EC2_HOME=whereveryourec2toolswererunzipped

Save it.

Now add this to the bottom of your .bashrc:

. ~/.ec2.conf
Start a new shell and test for the var's using:

$ export | grep EC2
if they're there, you're all set.

Now test by cd'ing into your ec2 tools bin folder and running:

ec2-describe-images -o self -o amazon
If you get a huge java stack trace, changes are your jdk isn't installed properly.

Otherwise, you should get a very long list of all the amazon AMI's available to run from.

Installing the JDK to Debian Lenny

The first thing to keep in mind is that Debian tries with all its might to remain free and open. In that regard, it resists including proprietary packages - even popular ones - in its main repositories.

So to give debian access to everything, you need to edit your

/etc/apt/source.list

use the following command in vi to include the contrib and non-free packages in your configured repositories:

:%s/main/main contrib non-free/g

:wq

For the sake of being thorough, run

# apt-get update

then

# apt-get install sun-java6-jdk

Answer in the affirmative to the various prompts.

When you're complete, adjust your alternatives like so:

#update-java-alternatives -s java-6-sun

et voila

Finding Installed Ruby Lib's

Since I can't seem to escape ruby, it would seem prudent to start documenting how to pick it apart.

Here's a little one-liner I just picked up apropos of puppet that tells you whether a lib is present:

# ruby -r -e "puts :installed"

If it's there, you'll see "installed". If not, something like "no such package".

Setting Linux Timezone from the Shell

Assuming you've got NTP, etc set up, your issue shouldn't be any more complicated than

1. Setting it properly in /etc/localtime:

cp /etc/localtime /etc/localtime.ORIG; 

cat /usr/share/zoneinfo/America/New_York > /etc/localtime 

2. Change /etc/sysconfig/clock to

ZONE="America/New_York" 

UTC=true 

ARC=false 

No OS restart is required, but it's probably a good idea. Any process with a JVM almost certainly needs one, though.

Note that the original localtime file is backed up in the line above, in case you need it.

Forgot to screen?

Sometimes even the best of us forget to invoke screen prior to running a long job.

I found an excellent workaround on serverfault (http://serverfault.com/questions/55880/moving-an-already-running-process-to-screen) that I'll repost in case it vanishes.

Simply put, you detach the process from your login shell using "disown".

So,

cntl-z

bg

jobs

disown -h %jobspec-id-from-jobs

And just like that, you've bullet-proofed it from a sighup. Kill your shell and pack up for the night because it'll run to completion.