I’m an idiot. I was programming on the aforementioned PT3K when I deleted 2 solid days of coding. Needless to say I was about to cry, but not all was lost!

I found a little program calld recover, which is available in the main debian apt tree. I installed it, crossed my fingers, read the man pages, and, for the first time in years, said a little prayer.

Basically, recover looks for recently deleted inodes, which, until they are written with new data, still contain your lost data. Once it has found those inodes it dumps the data to a specified directory. After that you just need to start grep’ing around for your files. Needless to say it was a lifesave and the *very* first thing I did was back up a tarball to trusty zebulon (up almost 300 days!).

Ask any geek what he fears most and the answer will most likely have to do with losing their email in one form or another. I’ve had a few scares regarding email servers going up/down as well as email servers migrating IP’s, but today is a new one for me.

You see I’m migrating not only IP’s, but I’m migrating to a totally new server as well as a totally new server setup. My post yesterday went over the new email setup. I got so excited this morning I had to switch. Wish me luck and until everything settles down email me at jstump at aa dot acinc dot com.

We have been switching Jerum’s colocation facility over the last few days and I took the opportunity to install a new mail system. The new system, to put it mildly, totally rocks. It uses qmail for SMTP and POP3, vpopmail for virtual hosting and quotas, qmailadmin for domain administration, MySQL as the everyone’s favorite backend, EZMLM for mailing lists, SquirrelMail for web based email, SpamAssassin to stop SPAM in its tracks, and BlackHole+ClamAV for virus protection (which I hacked to make work with STDOUT).

Needless to say this took a lot of caressing to get it working. A few notes for those wishing to go down this path.

• qmailadmin has a bug in it that messes up the quotas used by vdelivermail.
• I was able to install everything but courier-imap, vpopmail, and qmailadmin as Debian packages.
• Despite popular belief you don’t need the maildir++ qmail patch or vqmaillocal to get this setup working.

This is a huge upgrade over my last setup which was lacking in a few ways. It didn’t have qmailadmin (something no qmail administrator should be without). The old version also didn’t support POP3 nor did it support roaming SMTP-AUTH. To boot the new version uses MySQL for everything (including EZMLM lists).

I finally figured out what happened to the RAID array at work. As I had suspected a high I/O script was causing the array to barf. I turned off the script for now and plan on optimizing it in the near future (along with the bajillion other things I need to get done, but don’t have enough time in the day for).

With the site going up and down frequently over the last few days due to this rogue script we haven’t really been able to get a sense of whether the new site is working out well or not. We’ve taken a few orders and hopefully, after we mail our old customers about the new site, we will take a few more. The good news is that the people who are buy are buying laptops (our bread and butter at ACI). The bad news is that we need to do a lot of search engine optimization or we face losing a lot of search engine traffic.

One thing that I’ve really noticed since I started working at ACI is that being a sysadmin and being a programmer are very much TWO fulltime jobs. I’ve focused on launching the new site, which has meant ignoring my sysadmin duties for the last month or two, which means my sysadmin TODO is about as tall as me (I’m 6’5”). Since I’m graduating I’ll have plenty of free time to figure things out though.

We took our new site live at work yesterday. The new site is a ton nicer than the old site. I was all happy that the new site was up and running until I went to check it at about midnite to find that it was down.

After some investigation I realized my worst nightmare had happend: the FILES were missing. Not a down DB server, not a runaway Apache thread, but actual data loss. This is any sysadmin’s worst nightmare, especially when all of your files sit on a central file system. We run nightly tape backups, but you can’t ever be sure of their integrity until you get onsite.

Upon further investigation this morning, after having turned off every server in a panic last night, we found that our RAID array went nuts. We rebooted the array, checked the drives, and rebooted the NFS server. All seems to be well now. The very first thing I did? Back up to tape.

I’m over at Paul‘s house swapping files and geek tales. Of course, no trip to Paul’s house would be complete without having to do major upgrades to zebulon. In the end we had to reinstall qmail, dist-upgrade, and recompile PHP/Apache. If I wasn’t so tired from that I’d upgrade to MySQL 4.0. All of this because the Debian maintainers decided to throw in a buggy version of libc6 the other day.

In a stunning example of what system administrators should not do, a former PaineWebber system administrator set up a logic bomb within PaineWebber’s computer network. Essentially when the bomb was “triggered” the network would go down. The sys admin even had the bright idea to purchase over $20k in put options.

Systems administrators should follow a strict code of ethics at all times. Having access to a mountain of personal information can be tempting at times. The above story also brings up an interesting question: Could he be charged with cyber-terrorism?

In other hacking news a student in California gets an A in hacking for hacking into his school’s computer system. What I find funny is that the student changed his 4.0 GPA to a 1.9. What’s interesting is that this was not only school sponsored, but the school is actively working with the student to close the security holes in their system.

I recently installed AWStats to parse my logs for this site. I was quite amazed to see the numbers. I average about 1,000 unique visitors a month. My only complaint thus far is that Apache was storing my logs in “common” format and not “combined” format. The “combined” format offers more information, such as browser and referral information. AWStats should also let me know what people are searching for when they find my site, which should be quite interesting. I shall know more after a few months of parsed “combined” logs. If you are into stats you can read all of mine on the output page. I’d also like to know who, other than friends and family, read this site. Email me at joe at joe stump dot net and tell me who you are and why you like my site.

Many of you know me as “Joe the Programmer.” At my new job I’m “Joe the Systems Administrator” right now. My first task here is to build a robust, fault tolerant, scalable server solution. This involves many fun things that I now get to play with more than I had before. The list includes LVS, the High Availability package for Linux, NFS, serious RAID solutions, a Sun E450, iptables, etc. The first few days I worked here there seemed to be a server down every day. With this new architecture I hope that will be a thing of the past. Now all I need is a fancy pager …

I’ve recently been working on a project for client that includes incorporating virus scanning and spam blocking. Read on to learn how I’m ridding myself of spam and virii.

The first step is to get the necessary software. Here is a short list of the programs that I have used (how I tied them together is something I won’t say since the client IS selling this as an ASP).

• Spam Bouncer: A complex set of procmail scripts created to do high level spam checking and even some virii checking.
• Amavis: A virus checker that replaces the qmail-queue program to check all incoming and outgoing mail. It relies on a some sort of external virus checker (I’m using a commercial one) to actually check the mail, but works great. It quaranteens the mail and then sends notices to recipient and sender.
• Qmail: Wouldn’t be much of an email system without a MTA. DJB, despite being a raging asshole, has created what I believe to be the best MTA out there.

This is a good start to ridding yourself of spam and virii. Spam Bouncer works fairly well, however it requires some detailed setup and is VERY restrictive in its default setup. Amavis is just beautiful and I highly recommend it. If you have any questions feel free to email me questions.