Better Living Through Unix

Symfony Deployment Gotcha

2011-05-30T07:07:00.000-07:00

So I've been working with Symfony in the PHP arena and I found a really obnoxious gotcha that lives at the intersection of Symfony-based tutorials and Redhad-based distros. (In this case, CentOS.)

Namely, Symfony creates a rather expansive directory structure that - for security reasons - can't all live under your web root. The solution all the tutorials have you do is to set up the actual project in your home directory and then create an Apache virtual server that points to it. Seems simple enough. And you can add it to your config without any griping from Apache. Symfony has all the global perms set, so you're good, right?

Wrong, unfortunately. I was getting "403 Forbidden" over and over. I tried a wide variety of fixes, none of which worked. Eventually, I found this post on google groups:

http://bit.ly/mNRrlV

Sure enough, I checked the root perms of my home directory and it was all tightened down:

drwx------ 17 user user 4096 May 30 06:16 user

Just goes to show you the danger of returning to "childlike innonence" with new projects.

In reality, the right way to approach this is to create a project-specific directory somewhere under /usr/local and consciously set perms for the whole thing.

Following that approach, you get the right results straight out of the gate:

[root@localhost local]# ls -ld projects/
drwxr-xr-x 3 root root 4096 May 30 07:17 projects/

[root@localhost local]# ls -ld projects/milkshake/
drwxr-xr-x 11 root root 4096 May 30 07:18 projects/milkshake/

Now, you create your Symfony project under that:

symfony generate:project --orm=Propel milkshake

And your app:

symfony generate:app frontend

Finally, you add the following to your httpd.conf:

# Be sure to only have this line once in your configuration

NameVirtualHost *:8080

# This is the configuration for your project

Listen *:8080

<VirtualHost *:8080>

DocumentRoot "/usr/local/projects/milkshake/web"

DirectoryIndex index.php

<Directory "/usr/local/projects/milkshake/web">

AllowOverride All

Allow from All

</Directory>

</VirtualHost>

This is a good lesson for everybody: please check what you write and all your other assumptions, as well.

swatch queues messages

2011-02-25T09:53:00.001-08:00

Swatch is a fantastic tool for tailing logs for errors. But it has a surprising built-in behavior: it forcefully queues the outbound mails it generates for 30 minutes. Obviously, this stuff can be time-sensitive. I bugged some enlightened folks on the IRC channel and I got this excellent suggestion:

Create a job in root's cron to have exim flush the queue for the destionation mail addy at whatever interval you want. Here's one that does it every five minutes:

*/5 * * * * /usr/sbin/exim -qR

This solution is particularly elegant because you don't need to dig around inside swatch's guts. (Something you should never do with a package for a whole host of reasons.)

Make your own daemon

2011-01-28T11:19:00.000-08:00

A fantastic article on swatch (http://www.linux-mag.com/id/7807) included this delightfully simple framework for rolling your own daemons. If you take a moment to read it, you'll see a lot of unix design principles relating in perfect harmony:

#!/bin/sh

# Simple Log Watcher Program

case "$1" in 'start')

/usr/bin/swatch --daemon --config-file=/etc/swatch.conf --tail-file=/var/log/auth.log --pid-file=/var/run/swatch.pid

;;

'stop')   PID=`cat /var/run/swatch.pid`   kill $PID

;;

*)   echo "Usage: $0 { start | stop }

;;

esac

exit 0

Unattended Ubuntu Apt Installs

2011-01-27T18:06:00.000-08:00

This will be short and sweet.

When you want to

apt-get install package

and you don't want to get prompted for values, etc, just add the following the front of your shell:

export DEBIAN_FRONTEND=noninteractive

Poof. Magic.

Grepping out comments and blank lines

2011-01-02T10:54:00.000-08:00

One thing we commonly see in open-source configs is an abundance of helpful comments. Often, they dwarf the number of active parameters being set and that can make things very hard to read.

In the blessed world of unix, there is an easy solution to this: you simply use grep to remove those lines from a listing of the contents of the file.

First, grep out the comments:

grep -v \# file

Usually, that will leave you with a lot of unnecessary whitespace, as well. So just pipe that output through an enhanced grep and filter for blank lines:

grep -v \# file | egrep -v "^$"

To clarify the above, here's a break down of each component:

grep - self explanatory

-v - tells grep to remove what matches rather than show it

\ - tells the shell the special character # is to be passed to the process without interpretation

# - the near-universal symbol for "comment" in config-ese

file - a stand in for whatever conf you're digging through

| - a pipe. Tells the shell to send the output of what's to the left of the pipe to what's on its right

egrep - grep with extended regular expressions (a sophisticated text manipulation language)

-v - telling egrep to remove the matches

"" - passing what's inside to grep without shell interpretation

^ - Means "the start of the line"

$ - Means "the end of the line"

So what you're telling it, in essence, is the following:

Read the contents of file
Remove all lines with "#"
Pass the result to egrep
Egrep: remove all lines which begin and immediately end (ie, a blank line) and output the result

This is one of those little tricks that goes a long way to simplifying interactions with Unix hosts and it's actually the first thing I do when I go to read any new service's default config.

Hope it helps.

Subversion Set Up

2010-10-25T03:58:00.001-07:00

In a nutshell, ssh to the host and run

apt-get install subversion

exit

Then, as your own user, run

svnadmin create [repository name]

Note that if you want to share this repository with other users, you should create a group you will all share and create a special directory that you will give the group full RW access to.

For instance:

groupadd svn

[add your users to the svn group in /etc/groups]

su - [yourself]

groups

You should see "svn" listed.

It's important to remember that even though the group has RW access to your new directory, they won't have access to the repository by default. So give it to them explicitly:

chown -R .svn [repo]

chmod -R g+rw [repo]

Now have your users do a checkout with their ssh accounts:

svn co svn+ssh://[user]@[ip]/path/to/repo

(They'll be prompted for their password twice.)

svn co file:///path/to/repo

on the same host.

And a commit

svn commit

(Make sure everyone's commenting their commits)

If you see this:

svn: Can't open file '/home/svn/res/db/txn-current-lock': Permission denied

Make sure they're logged in as the appropriate group and that the perms are correct.

File Encryption with OpenSSL

2010-09-02T08:41:00.001-07:00

If you're like me, you're occasionally forced to mail some pretty sensitive things around. When that happens, it's not enough to say "Well, I deleted the message." The message is still in your trash, on a server's disk, etc, etc. You need to assume someone WILL get ahold of it and take measures to prevent them from doing anything with it.

My personal favorite (since it can be used in any situation) is the openssl file encryption capability. The openssl package is almost universally installed by default in distros these days, so you should just be able to jump straight to the commands.

In this case, I've got a file called "spreadsheet.xls" that I want to password encrypt. The syntax is simply this:

openssl aes-256-cbc -a -salt -in spreadsheet.xls -out spreadsheet.xls.enc

Then just supply the password you want to use.

And when you want to decrypt it, just add the '-d' switch and flip the filenames:

openssl aes-256-cbc -d -a -in spreadsheet.xls.enc -out spreadsheet.xls

And use your password.

Just like that, you've got munition-grade encryption from the command-line.

Enjoy

Zenoss Quirks

2010-08-14T10:35:00.000-07:00

So I ran into two really obnoxious issues that took a considerable amount of digging to resolve:

1. It was marking my apache process monitor as failed/recovered randomly and often.
2. It would not let go of misapplied process monitors that had been picked-up by overly liberal regexes. It would include:

tail -f /var/log/nginx/access.log
nginx: worker

in the nginx process monitor if I was tailing the log when I modeled the host. The problem was, after I killed the tail, the process was alerting as "Process Not Running". FOREVER. Even if I deleted and recreated the process monitor, the host, the events, everything.

In the first case, it turns out that since apache marks its process as "apache defunct" when it's shutting down a child process, Zenoss would occasionally pick this up as a live apache process. It would then mark it as "Down" after the proc terminated. The solution for this was to make my regex more specific:

apache2 \-k start

The second case was much more obnoxious because not only would the events not clear, they would return and begin alerting every time the device was recreated.

After some digging online, it seemed that the best course was to restart the zenprocess daemon.

This is best done under Settings > Daemons. You can also view the logs there (which showed the bad checks prior to the restart and nothing after).

When that's complete, re-add your device and you should be rid of the baggage.

Tethering your iPhone in Ubuntu

2010-07-28T07:16:00.000-07:00

This is a delightfully easy one. First, add this repository and update your apt:

sudo add-apt-repository ppa:pmcenery/ppa

sudo apt-get update

Now install the following three packages (you may have to go dig for them):

libimobiledevice-utils
ipheth-dkms
ipheth-utils

Restart your machine.

Turn on tethering inside your iPhone's General > Network menu.

Cable your iPhone to a USB port.

It should automatically connect. You'll see it in Network Manager.

Poof. Magic.

Enabling SSH Agent Forwarding in OS X

2010-07-25T16:53:00.000-07:00

If you use EC2, you use SSH keys.

If you're sane, you keep the private keys on your workstation and forward them through the chain of public keys throughout your hosts.

I recently started working through OS X again and found some pretty obnoxious behavior: out of the box, the ssh keys don't forward. After some digging, I found the following discussion of the subject:

http://data.agaric.com/node/3061#comment-1604

Long story short, it appears that the location of the user's home directory isn't communicated when the keys are forwarded so the agent looks in the wrong place.

The fix? Run these two on your OS X machine:

ssh-add

ssh-add -l

et voila

Try again.

Removing packages from Debian/Ubuntu

2010-07-08T13:13:00.001-07:00

Just a quick note so I don't waste time revisiting this topic again.

A coworker of mine forced a package install without repositories so all the dependencies obviously failed. Trying to remove it, I did a

dpkg --remove

But running an apt-get said it was still installed. Running

dpkg --list

Gave me a status of "rc" - which left a lot of files laying around. (Configs, etc.)

Finally, I just did a

dpkg --purge

And it cleared out all the junk.

From there, I was able to do a normal apt-get against the repositories.

Using Exim as a Mail Relay

2010-06-30T10:44:00.000-07:00

One service that always seems required by any platform I've ever worked on is an outbound mail transfer agent. Essentially, everybody wants to send their customers notifications and needs a clearinghouse to pass it through.

This is all well and good, but there are a few inevitable problems one is likely to run into when putting one up.

By far, I've had the most success with Exim running on Debian since both are very lightweight, incredibly reliable and closely intertwined. (Exim is Debian's default MTA.)

Step 1 is to provide the basic setup. Log in, su to root and run the following:

dpkg-reconfigure exim4-config

You'll walk through some basic setup info. Essentially, you want to specify the host as an internet mail server and provide the ip addresses of your internal hosts are valid hosts to relay for.

When that's complete, you need to cd into /etc/exim4 and modify the exim4.conf.template file. Find the rewrite section by looking for

begin rewrite

Then add your rewrite rule under it. This will make it apply to all messages that pass thorugh Exim.

The generic rule I usually use looks like this:

*@* $1@mycompanysdomain Ffrs

What it essentially says is "match everthing (*@*) and replace what's in front of the @ with what was to the left of the original and everything after the @ with with explicit domain name - also, apply it to the following types of addresses: F - envelope From field, f - From header, r - Reply-To header, s - Sender header"

With that rule in place, restart Exim:

invoke-rc.d exim4 restart

And the config will be written out to /var/lib/exim4/configuration.autogenerated.

The best way to test is to use the Exim command line to see what rewrites will get applied:

# exim -brw root@localhost
sender: root@mycompanysdomain
from: root@mycompanysdomain
to: root@localhost
cc: root@localhost
bcc: root@localhost
reply-to: root@mycompanysdomain
env-from: root@mycompanysdomain
env-to: root@localhost

With that done, you need to take care of the blacklists. First, add an A record to your domain's authoritative DNS WITH A PTR. The reverse lookup is used as a method to ensure the mail server is a legitimate member of your domain. NOTE: A lot of people think MX records are required but that's incorrect. MX records are for inbound mail only.

Now fire off some messages to big mail destinations like gmail and yahoo. Tail the /var/log/exim4/mainlog to see what results you get. Yahoo especially is good about telling you where your IP is blacklisted for some reason. If it is, you'll get a lot of dropped messages till you contact the Blacklister and petition to have it removed.

On that same note, be careful about sending out a bunch of unsoliticed test messages. I've seen hosts get black-listed for that reason alone and it's a pain to get delisted.

Checking for Root

2010-05-21T07:10:00.001-07:00

Good programmers know to check for assumptions their code is making before executing it. For an operator, assuming that you've su'd to root before running an installer you've written is a bad idea.

The easiest way to resolve that is to simply use the EUID bash variable to check the effective user-id you're executing under and then exit if it's not zero.

I cribbed the following

if [[ $EUID -ne 0 ]]; then
echo "This script must be run as root"
exit 1
fi

From this excellent article:

http://www.cyberciti.biz/tips/shell-root-user-check-script.html

Awk Alternative Delimiters

2010-05-20T07:57:00.000-07:00

As it's always best to automate, it's often useful to fill the values of shell variables with the returned values of system commands.

For instance, if you wanted the ip of eth0, you could run

/sbin/ifconfig eth0

And you'd get back something like

[nsmc@nsmc-dt automation]$ /sbin/ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:26:F2:AC:C2:FE
inet addr:10.1.1.99 Bcast:10.1.3.255 Mask:255.255.252.0
inet6 addr: fe80::226:f2ff:feac:c2fe/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1142141 errors:0 dropped:0 overruns:0 frame:0
TX packets:223927 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:212209959 (202.3 MiB) TX bytes:205106885 (195.6 MiB)
Interrupt:18 Base address:0x2000

Then you could vi into your config script and export the value to a var manually.

-OR-

You could set the variable with the returned value of a system command after clearing away some noise.

Using an iterative approach, we could peel back each layer till we get the piece we want:

/sbin/ifconfig eth0 | grep "inet "

That gives us the "inet" line and not the "inet6" line.

[nsmc@nsmc-dt automation]$ /sbin/ifconfig eth0 | grep "inet "
inet addr:10.1.1.99 Bcast:10.1.3.255 Mask:255.255.252.0

But we only want the piece behind the first colon. Here we need to use a little awk trickery. awk uses spaces for it's default field delimiters. Let's change that to a colon (:) and see what we get.

[nsmc@nsmc-dt automation]$ /sbin/ifconfig eth0 \
| grep "inet " | awk -F \: '{print $2}'

10.1.1.99 Bcast

Close. Now we can pass it through a standard awk filter and just get the piece we want:

[nsmc@nsmc-dt automation]$ /sbin/ifconfig eth0 | grep "inet " \
| awk -F \: '{print $2}' | awk '{print $1}'

10.1.1.99

Now we just need to assign to our variable at runtime using the backtic's:

export PRIVATE_HOST_IP=`/sbin/ifconfig eth0 | grep "inet " \
| awk -F \: '{print $2}' | awk '{print $1}'`

Escaping Single-Quotes in sed

2010-04-27T15:48:00.001-07:00

This is a just a re-post of this excellent article by Stuart Coleville:

Using single quotes in BASH ensures that the shell doesn’t expand the contents of the quoted string and this is useful most of the time. However if you want to use single quotes within a single quoted string things don’t work out as you might expect.

If I want to use sed to match some text with a single quotes in it, I will run into trouble if I run:

sed 's/user \= 'root/user \= 'moi/g'

Alternatively if I run (on the same snippet):

sed 's/user \= \'root/user \= \'moi/g'

I will get:

/bin/bash: -c: line 1: unexpected EOF while looking for matching `'' /bin/bash: -c: line 2: syntax error: unexpected end of file

This doesn’t work because the escaped single-quotes (\') are not expanded and are therefore treated literally.

To single quotes work you need to break out of the single quoted string then escape your single quote. Like so:

sed 's/user \= '\''root/user \= '\''moi/g'

Because \' is not inside of single quotes the single-quote is properly escaped and the output is as we’d expect:

user = 'root' ->  user = 'moi'

In conclusion, the title of this post is a bit of a misnomer. You actually can’t put single quotes inside of a single-quoted string. However breaking out allows us to get to where we want to be.

Installing Amazon EC2 tools to Debian Lenny

2010-04-18T08:04:00.001-07:00

I was reading through Bill Childers excellent article in Linux Journal on running Ubuntu under EC2 and I thought I add my notes on how to make the client tools run on Debian.

First, install the Sun JDK (see my previous post on how to do that).

With that done, download the tools from this page:

AWS Developer Community

Or simply by using this link:

ec2-api-tools.zip

Next, unzip them.

Then create a configuration file to export the proper environment variables when your shell fires up:

$ vi ~/.ec2.conf
Add the lines:

export EC2_CERT=whereveryourx509lives/cert-whatever-x509.pem
export EC2_PRIVATE_KEY=whereveryourx509pklives/pk-whatever-x509.pem
export JAVA_HOME=/usr/lib/jvm/java-6-sun/jre
export EC2_HOME=whereveryourec2toolswererunzipped

Save it.

Now add this to the bottom of your .bashrc:

. ~/.ec2.conf
Start a new shell and test for the var's using:

$ export | grep EC2
if they're there, you're all set.

Now test by cd'ing into your ec2 tools bin folder and running:

ec2-describe-images -o self -o amazon
If you get a huge java stack trace, changes are your jdk isn't installed properly.

Otherwise, you should get a very long list of all the amazon AMI's available to run from.

Installing the JDK to Debian Lenny

2010-04-18T07:53:00.000-07:00

The first thing to keep in mind is that Debian tries with all its might to remain free and open. In that regard, it resists including proprietary packages - even popular ones - in its main repositories.

So to give debian access to everything, you need to edit your

/etc/apt/source.list

use the following command in vi to include the contrib and non-free packages in your configured repositories:

:%s/main/main contrib non-free/g

:wq

For the sake of being thorough, run

# apt-get update

then

# apt-get install sun-java6-jdk

Answer in the affirmative to the various prompts.

When you're complete, adjust your alternatives like so:

#update-java-alternatives -s java-6-sun

et voila

Finding Installed Ruby Lib's

2010-04-13T09:28:00.000-07:00

Since I can't seem to escape ruby, it would seem prudent to start documenting how to pick it apart.

Here's a little one-liner I just picked up apropos of puppet that tells you whether a lib is present:

# ruby -r -e "puts :installed"

If it's there, you'll see "installed". If not, something like "no such package".

Setting Linux Timezone from the Shell

2010-03-24T12:16:00.000-07:00

Assuming you've got NTP, etc set up, your issue shouldn't be any more complicated than

1. Setting it properly in /etc/localtime:

cp /etc/localtime /etc/localtime.ORIG;

cat /usr/share/zoneinfo/America/New_York > /etc/localtime

2. Change /etc/sysconfig/clock to

ZONE="America/New_York"

UTC=true

ARC=false

No OS restart is required, but it's probably a good idea. Any process with a JVM almost certainly needs one, though.

Note that the original localtime file is backed up in the line above, in case you need it.

Forgot to screen?

2010-03-19T15:25:00.000-07:00

Sometimes even the best of us forget to invoke screen prior to running a long job.

I found an excellent workaround on serverfault (http://serverfault.com/questions/55880/moving-an-already-running-process-to-screen) that I'll repost in case it vanishes.

Simply put, you detach the process from your login shell using "disown".

So,

cntl-z

jobs

disown -h %jobspec-id-from-jobs

And just like that, you've bullet-proofed it from a sighup. Kill your shell and pack up for the night because it'll run to completion.

Formatting Tool Output for Reports

2009-11-02T11:24:00.000-08:00

If you're like me, you use cron to mail you a variety of stats generated during off-peak ours.

You may, at some point, want to start extrapolating patterns from the data you've collected. In my experience, nothing really gets the job done in this scenario quite like a spreadsheet.

That being said, the data generated by most tools is reader-friendly, not import-friendly. Therefore, they must be munged in order to add them to your data set.

Most of them use tabs to do their spacing/formatting. I have written a quick sed hack that removes one or more instances of a tab from a line and converts it to a comma. The end result is essentially a CSV.

So, to format the output of a

du -sm /home/*

you could pipe it through sed and into a mail command.

Ie:

du -sm /home/* | sed -e '/txt/d;s/\t\t*/,/' | mail -s "Usage" nathan@mccourtney.com

the '-e' lets it know that there is more than one command being passed - separated by a ';'.

In this case, I'm using the first part of the command to delete every line with 'txt' in it, since I'm really not worried about the text files. The second part matches at least one tab and all the tabs following it, then replaces them with a single ','. The output it then piped to a mail command and sent to my account.

When it shows up, the output goes from

nsmc@bangkok:~$ du -sm /home/*

1786 /home/nathan

1786,/home/nsmc

It's not terribly pretty, but you can easily import it into a spreadsheet. And really, I use a slightly more complex du script that generates a more readable listing before it even gets to sed. This was intended just to show the sed side of it.

ulimit - Runtime system resource tuning

2009-10-20T10:15:00.000-07:00

When you're dealing with daemons, you're likely to run into the system's process-level constraints. In additional to the kernel params that are set in the setup scripts, there are process limits you need to be aware of to keep from bumping your head.

The shell has a built-in command to query and set those values: ulimit.

ulimit: usage: ulimit [-SHacdfilmnpqstuvx] [limit]

brutus:~ nsmc$ ulimit -a

core file size (blocks, -c) 0

data seg size (kbytes, -d) unlimited

file size (blocks, -f) unlimited

max locked memory (kbytes, -l) unlimited

max memory size (kbytes, -m) unlimited

open files (-n) 256

pipe size (512 bytes, -p) 1

stack size (kbytes, -s) 8192

cpu time (seconds, -t) unlimited

max user processes (-u) 266

virtual memory (kbytes, -v) unlimited

As you can see, there are very tight limits on this system for the number of processes a user can run, as well as the number of files any of those processes can have open.

These values can be set using a basic syntax:

#ulimit -n 9162

However, they won't survive a reboot. Since these limits are tied to the shell process that invokes them, you need to set them in /etc/bashrc (or whatever your shell takes its initialization parameters from) in order to have them come up automatically.

egrep - LOVE it

2009-09-23T08:00:00.001-07:00

'egrep' is an alias for grep invoked with the '-e' switch to enable extended regular expressions.

The reason I especially love it is its ability to use inclusive "or's". In other words, give me all the lines with x, y or z.

So if you want to see a count of all the mails sent to your exim daemon from say, 5 to 8 pm on 9/22, you can simply execute the following:

egrep '(2009-09-22 17|2009-09-22 18|2009-09-22 19|2009-09-22 20)' mainlog.1 | grep dnslookup | wc -l

And voila! You will have your answer.

Reading Top on Multi-core Systems

2009-09-21T16:34:00.000-07:00

Take this sample output:

top - 11:24:56 up 2 days, 11:19, 2 users, load average: 0.45, 0.34, 0.32
Tasks: 176 total, 1 running, 175 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.1%us, 0.4%sy, 0.0%ni, 96.3%id, 0.1%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 8174036k total, 7693556k used, 480480k free, 260924k buffers
Swap: 2031608k total, 4k used, 2031604k free, 5043372k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26229 root 25 0 2181m 1.5g 96m S 28 18.7 133:33.65 java
3388 root 18 0 67692 2240 1732 S 0 0.0 0:01.95 rotatelogs
16890 nsmc 15 0 12736 1144 816 R 0 0.0 0:00.51 top

What we’re seeing seems completely contradictory - the java process is consuming 28 percent of the cpu's resources, but the total user process consumption on the system is 3 percent?

Basically, a rollup of CPU time on a multicore system is going to be an over-simplification. It’s essentially the total cpu of all processes divided by the number of cores. (In this case, eight).

Using the run-time command of ‘1’ (that’s the number one), you get a different output:

top - 11:28:25 up 2 days, 11:23, 2 users, load average: 0.14, 0.21, 0.27
Tasks: 174 total, 1 running, 173 sleeping, 0 stopped, 0 zombie
Cpu0 : 13.3%us, 1.3%sy, 0.0%ni, 84.1%id, 0.7%wa, 0.0%hi, 0.7%si, 0.0%st
Cpu1 : 0.3%us, 0.0%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu2 : 1.0%us, 0.0%sy, 0.0%ni, 99.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 0.7%us, 0.0%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu4 : 1.0%us, 0.0%sy, 0.0%ni, 99.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu5 : 0.7%us, 0.0%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu6 : 0.7%us, 0.0%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu7 : 0.7%us, 0.3%sy, 0.0%ni, 99.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 8174036k total, 7754924k used, 419112k free, 261028k buffers
Swap: 2031608k total, 4k used, 2031604k free, 5104464k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26229 root 25 0 2181m 1.5g 96m S 20 18.7 134:23.33 java

That makes a LOT more sense. So the default output is showing the cpu consumption of the java process at the individual core level, but the rollup as an average across all the cores. This new output shows how it’s getting distributed across the cores.

By the same token, if you go back to the default output and hit ‘I’ (that’s a capital i), you will get a cpu consumption number for the java process divided by the number of cores:

top - 11:31:31 up 2 days, 11:26, 2 users, load average: 0.14, 0.22, 0.26
Tasks: 174 total, 1 running, 173 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.0%us, 0.5%sy, 0.0%ni, 96.0%id, 0.3%wa, 0.0%hi, 0.2%si, 0.0%st
Mem: 8174036k total, 7808624k used, 365412k free, 261136k buffers
Swap: 2031608k total, 4k used, 2031604k free, 5156848k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26229 root 25 0 2181m 1.5g 96m S 3.5 18.7 135:04.92 java
16890 nsmc 15 0 12736 1144 816 R 0.0 0.0 0:01.32 top
1 root 15 0 10344 680 572 S 0.0 0.0 0:02.47 init

So now you’re consistent again.

I’m not sure why these two are aren’t viewed from the same perspective by default.

IOPS

2009-08-25T07:33:00.000-07:00

Input/Output Per Second (aka "IOPS") is a performance metric for hard disks that represents how many storage operations (reads/writes) the disk can execute in a given second. This, combined with how much actual data can be passed per operation, determines the throughput of the disk. The difference between the two is important since a 1 byte operation has the same IOP hit as a 1KB operation. (Assuming your drive can handle at least 1KB per operation.)

You can find the number of iops being used by your system via vmstat:

HOST::~$ vmstat 5

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----

r b swpd free buff cache si so bi bo in cs us sy id wa

0 0 0 50432 388 1083600 0 0 407 290 1 0 1 5 88 6

0 0 0 49892 388 1084348 0 0 119 39 1295 1182 0 2 92 6

0 0 0 56856 388 1078296 0 0 324 99 1799 1972 0 4 86 10

0 0 0 56440 388 1078704 0 0 87 7 1673 1885 0 4 93 2

0 0 0 54880 388 1080472 0 0 320 32 1573 1518 0 2 89 8

0 0 0 53020 388 1082512 0 0 384 43 1556 1536 0 2 91 6

1 0 0 51568 388 1083736 0 0 232 45 1460 1453 0 1 90 8

Note that with vmstat, you can ignore the first line - it's a rollup average of stats since the last restart.

The stats you want to look at are io: bi and bo and cpu's wa. bi stands for "Blocks In" and bo for "Blocks Out". It's a bit counter-intuitive, though, in that they represent blocks moving into and out of the kernel's memory space, NOT into and out of the device. In fact, because of that distinction, writes to devices are actually represented by bo and reads from are represented by bi.

wa stands for "wait time" and tells you how much of your cpu time is spent waiting for I/O to complete before moving to the next instruction. If your wait time is high, your device I/O is getting choked - probably from insufficient IOPS capacity. You can track down which device it is using iostat, though figuring out which process is responsible is still a bit of an art. (I'm pretty sure the hooks for doing it are only just starting to make their way into the kernel.)

In a RAID environment, the number of IOPS is an aggregation of all the disks that are active within the RAID group. (It doesn't include any online spares that are sitting around idle.) This is essentially the speed limit for a group of disks. If you attempt to perform more I/O operations than your RAID can support, the device or host will have to start queueing them which will lead to dramatically increased CPU wait time.

The performance numbers for IOPS take a little digging to find so I wanted to drop them here for future reference:

SAS - Serial Attached SCSI - 175 for 15K, 125 10K

SATA - Serial ATA - 100 +/-

FC - Fibre Channel - 200 for 15K, 150 for 10K

Note that THESE NUMBERS WILL CHANGE. It represents the current state of things based on my interaction with vendors.

Multiply these numbers by the number of drives that are part of the active RAID and you'll know the theoretical upper boundary of your performance. (Which you can then compare to your vmstat to see where you're landing.)

A great article for more information on how to calculate IOPS/etc:

http://storagearchitect.blogspot.com/2008/09/how-many-iops.html

How to trace end-to-end connections on the Netscaler Load-balancer

2009-07-31T16:07:00.000-07:00

HOW-TO: View Active Sessions End-To-End on the Netscalers

hide

First, the easiest way to see what the source IP's for traffic are is via the ASA firewall logs in syslog.

So to see where ftp connections are coming from, you could use something like

grep [ftp cluster vip] syslog | grep -v ICMP | grep -v [monitoring host's ip] | grep -v local-host

Which greps for the netscaler virtual ip of the ftp-cluster in the syslog file and filters out ICMP and monitoring-host traffic. (As well as connections to itself.)

Additionally, you can see all the active connections on a netscaler by ssh'ing to the CLI and running:

> show connectiontable | grep [ftp cluster vip]

[client ip] 45534 [ftp cluster vip] 21 FTP 7 TIME_WAIT

[client ip] 32570 [ftp cluster vip] 21 FTP 9 ESTABLISHED

That shows you where they're coming from. To find out where they're going to, also, you need to check the persistent connections:

> show persistence

Type SRC-IP DST-IP PORT VSNAME TIMEOUT REF_CNT

SOURCEIP [client ip] [ftp server ip] 21 ftp_cluster 103 1

SOURCEIP [client ip] [ftp server ip] 21 ftp_cluster 0 1

SOURCEIP [client ip] [ftp server ip] 21 ftp_cluster 75 0

SOURCEIP [client ip] [ftp server ip] 21 ftp_cluster 91 0

NOTE: This only works on vservers where persistence is handled by source-ip.

In the case of HTTP traffic, you can add a header to ip with the original ip, in the http traffic that hits the backend services.

There are also a number of Netscaler products allow you to do extensive log analysis.

Enabling Persistent Routes on a Debian Host

2009-07-27T09:46:00.001-07:00

1. su to root

2. cd to /etc/network/

3. Copy off the interfaces file to interfaces.DATE (or what have you)

4. Add lines of the following form under the primary network interface definition:

up route add -net 10.1.1.0 netmask 255.255.255.0 gw 10.2.1.1

down route del -net 10.1.1.0 netmask 255.255.255.0

So you should end up with something like this:

iface bond0 inet static

address 10.2.1.5

netmask 255.255.255.0

network 10.2.1.0

gateway 10.2.1.1

up /sbin/ifenslave bond0 eth0

up /sbin/ifenslave bond0 eth2

up route add -net 10.1.1.0 netmask 255.255.255.0 gw 10.2.1.1

down route del -net 10.1.1.0 netmask 255.255.255.0

That creates a route to the 10.1.1.x network for the host with the ip 10.2.1.5 through the 10.2.1.1 router whenever the interface goes up. (It also removes it whenever the interface goes down.)

How to Disable and Clean Netapp Snapshots

2009-07-27T09:30:00.000-07:00

To disable snapshots on a netapp volume, you need to disable snapshots on the volume:

vol options volume_name nosnap on

and disable the automatically scheduled snaps

snap sched volume_name 0 0 0

If you need to clear space from the snapshot volume, you can delete the old snaps.

Run

snap list volume_name

to find them, then

snap delete volume_name snap_name

to delete them.

How To Use Netapp SnapMirror

2009-07-17T10:09:00.000-07:00

I. To Create a Snapmirror Relationship:

Create source and destination volumes of the same size that have a same-sized aggregate. (This is critical for being able to change the direction of the sync.)

Go into FilerView > Volumes on the DESTINATION and mark the volume OFFLINE.

Go into FilerView > SnapMirror > Add on the DESTINATION and proceed through accepting all the defaults except, obviously, the volume names.

On the SnapMirror > Manage screen, click the Advanced properties of the new job. Inside the job, click "Initialize". It will clean the target volume and begin the first sync. The sync will begin automatically on schedule which, if you used the defaults, is every minute.

II. To mark a Snapmirror RW:

End the SnapMirror relationship with the

snapmirror break

command. This command changes the destination's status from

snapmirrored

broken-off

thus making it writable.

When you're ready to resync them, run the

snapmirror resync

command on the DESTINATION. This will change a former destination's status back to snapmirrored and will resynchronize its contents with the source.

(NOTE: When applied to a former source, snapmirror resync can turn it into a mirror of the former destination. In this way, the roles of source and destination can be reversed.)

At any time, you can see the status of all the snapmirrors by running the

snapmirror list

command.

Bookpool is gone!

2009-07-02T06:29:00.000-07:00

How terribly, terribly sad! I noticed their inventory had been pretty lean for a while there, but this was definitely the go-to site for at least half the books in my collection.

I guess the web giveth and the web taketh away.

Here's hoping a new store rises from the ashes . . .

Prune - Find's tricky switch

2009-06-26T16:10:00.000-07:00

After many thousands of uses, I finally got around to investigating find's exclusion parameters. Showing its age, it basically consists of an arcane string of -path options combined with an -o for "or" followed by a -prune to indicate the need to exclude it from the search.

I was always sure there was some way to do it. (With a regex, if nothing else.) I've read the man page about 100 times and I've never understood how it worked till last week. It works like a charm, in fact.

That being said, surely something closer to the du --exclude parameter is in order. Features aren't very useful if people don't know they're there.

ctime vs mtime

2009-06-18T11:55:00.000-07:00

So about that huge file cleanup . . . it seems that Netapp's ndmpcopy resets all the files' ctimes. (Meaning there were changes to the inodes somewhere in the process). Fortunately, the mtimes were preserved so I was able to generate a proper distribution-over-time stat.

I can only assume (since the guy who ran the ndmpcopy swears he didn't do anything else), that that means ndmpcopy is actually creating the inodes on the fly rather copying the existing ones over. Good to know. (rsync preserves those stamps the way I usually run it, so I've never run into this issue before.)

Multi-million-file Cleanup

2009-06-16T07:13:00.001-07:00

I'm queuing up a series of scripts for tonight to go through the customer filesystems and clean out files older than a certain interval. Under most circumstances, this would be a simple find and exec. Once you get past about 400,000 files, though, you're forced to break the operation down into atomic tasks.

Essentially, I do a while-loop that descends to the level of customer directories and the performs the find within each one of them. Normally, I would move the files to a trash directory at the root of the filesystem, but this is on the new netapps so a) this is mounted over NFS and moves would be extremely expensive and b) there's a built-in snapshotting capability in WAFL. So I'll snap it first, then run the cleaner. (On the plus side, I've been running this cleaner against our existing storage for years, so the basic algorithm is rock-solid.)

New Hosting Provider

2009-06-15T10:39:00.000-07:00

I lost my last blog/troubleshooting matrix to a break-in at my last provider, so I'm giving JustHost a try. It will be interesting to see how the domain transfer process has evolved over the last few years . . .