Escaping Single-Quotes in sed

This is a just a re-post of this excellent article by Stuart Coleville:

Using single quotes in BASH ensures that the shell doesn’t expand the contents of the quoted string and this is useful most of the time. However if you want to use single quotes within a single quoted string things don’t work out as you might expect.

If I want to use sed to match some text with a single quotes in it, I will run into trouble if I run:

sed 's/user \= 'root/user \= 'moi/g'

Alternatively if I run (on the same snippet):

sed 's/user \= \'root/user \= \'moi/g'

I will get:

/bin/bash: -c: line 1: unexpected EOF while looking for matching `''
/bin/bash: -c: line 2: syntax error: unexpected end of file

This doesn’t work because the escaped single-quotes (\') are not expanded and are therefore treated literally.

To single quotes work you need to break out of the single quoted string then escape your single quote. Like so:

sed 's/user \= '\''root/user \= '\''moi/g'

Because \' is not inside of single quotes the single-quote is properly escaped and the output is as we’d expect:

user = 'root' ->  user = 'moi'

In conclusion, the title of this post is a bit of a misnomer. You actually can’t put single quotes inside of a single-quoted string. However breaking out allows us to get to where we want to be.

Installing Amazon EC2 tools to Debian Lenny

I was reading through Bill Childers excellent article in Linux Journal on running Ubuntu under EC2 and I thought I add my notes on how to make the client tools run on Debian.

First, install the Sun JDK (see my previous post on how to do that).

With that done, download the tools from this page:

AWS Developer Community

Or simply by using this link:

ec2-api-tools.zip

Next, unzip them.

Then create a configuration file to export the proper environment variables when your shell fires up:

$ vi ~/.ec2.conf
Add the lines:

export EC2_CERT=whereveryourx509lives/cert-whatever-x509.pem
export EC2_PRIVATE_KEY=whereveryourx509pklives/pk-whatever-x509.pem
export JAVA_HOME=/usr/lib/jvm/java-6-sun/jre
export EC2_HOME=whereveryourec2toolswererunzipped

Save it.

Now add this to the bottom of your .bashrc:

. ~/.ec2.conf
Start a new shell and test for the var's using:

$ export | grep EC2
if they're there, you're all set.

Now test by cd'ing into your ec2 tools bin folder and running:

ec2-describe-images -o self -o amazon
If you get a huge java stack trace, changes are your jdk isn't installed properly.

Otherwise, you should get a very long list of all the amazon AMI's available to run from.

Installing the JDK to Debian Lenny

The first thing to keep in mind is that Debian tries with all its might to remain free and open. In that regard, it resists including proprietary packages - even popular ones - in its main repositories.

So to give debian access to everything, you need to edit your

/etc/apt/source.list

use the following command in vi to include the contrib and non-free packages in your configured repositories:

:%s/main/main contrib non-free/g

:wq

For the sake of being thorough, run

# apt-get update

then

# apt-get install sun-java6-jdk

Answer in the affirmative to the various prompts.

When you're complete, adjust your alternatives like so:

#update-java-alternatives -s java-6-sun

et voila

Finding Installed Ruby Lib's

Since I can't seem to escape ruby, it would seem prudent to start documenting how to pick it apart.

Here's a little one-liner I just picked up apropos of puppet that tells you whether a lib is present:

# ruby -r -e "puts :installed"

If it's there, you'll see "installed". If not, something like "no such package".

Setting Linux Timezone from the Shell

Assuming you've got NTP, etc set up, your issue shouldn't be any more complicated than

1. Setting it properly in /etc/localtime:

cp /etc/localtime /etc/localtime.ORIG; 

cat /usr/share/zoneinfo/America/New_York > /etc/localtime 

2. Change /etc/sysconfig/clock to

ZONE="America/New_York" 

UTC=true 

ARC=false 

No OS restart is required, but it's probably a good idea. Any process with a JVM almost certainly needs one, though.

Note that the original localtime file is backed up in the line above, in case you need it.

Forgot to screen?

Sometimes even the best of us forget to invoke screen prior to running a long job.

I found an excellent workaround on serverfault (http://serverfault.com/questions/55880/moving-an-already-running-process-to-screen) that I'll repost in case it vanishes.

Simply put, you detach the process from your login shell using "disown".

So,

cntl-z

bg

jobs

disown -h %jobspec-id-from-jobs

And just like that, you've bullet-proofed it from a sighup. Kill your shell and pack up for the night because it'll run to completion.

Formatting Tool Output for Reports

If you're like me, you use cron to mail you a variety of stats generated during off-peak ours.

You may, at some point, want to start extrapolating patterns from the data you've collected. In my experience, nothing really gets the job done in this scenario quite like a spreadsheet.

That being said, the data generated by most tools is reader-friendly, not import-friendly. Therefore, they must be munged in order to add them to your data set.

Most of them use tabs to do their spacing/formatting. I have written a quick sed hack that removes one or more instances of a tab from a line and converts it to a comma. The end result is essentially a CSV.

So, to format the output of a

du -sm /home/*

you could pipe it through sed and into a mail command.

Ie:

du -sm /home/* | sed -e '/txt/d;s/\t\t*/,/' | mail -s "Usage" nathan@mccourtney.com

the '-e' lets it know that there is more than one command being passed - separated by a ';'.

In this case, I'm using the first part of the command to delete every line with 'txt' in it, since I'm really not worried about the text files. The second part matches at least one tab and all the tabs following it, then replaces them with a single ','. The output it then piped to a mail command and sent to my account.

When it shows up, the output goes from

nsmc@bangkok:~$ du -sm /home/*

1786 /home/nathan

to

1786,/home/nsmc

It's not terribly pretty, but you can easily import it into a spreadsheet. And really, I use a slightly more complex du script that generates a more readable listing before it even gets to sed. This was intended just to show the sed side of it.