gtuhl: startup IT and software development

I had to wrangle with a bunch of overloaded email servers recently and wanted to share my new best friend when this sort of thing happens:

postsuper

This guy is available if your email server is running postfix (at /usr/sbin/postsuper) or if you are running Zimbra which uses postfix under the hood (at /opt/zimbra/postfix/sbin/postsuper in that case). You should be able to run it and access the man page for it as root. This is pretty basic stuff for a postfix admin but I had a lot of trouble finding even basic descriptions of how to go about doing the things that postsuper can do.

Postfix files messages into several queues. The main ones are the following:

• Incoming/Active - These are the common queues where incoming/outgoing messages live until they finish being delivered or received.
• Deferred - If messages cannot be delivered they go here and delivery is reattempted until the messages expire.
• Hold - A queue that you the administrator can move messages to. Messages placed here are not processed, no attempt is made to redeliver them, and they do not expire. As far as I can tell this queue only exists to make life easy on the administrator and gives you a safe place to store things temporarily.

Sometimes an email server will get crushed with volume from an attack, a misconfiguration, a mistake, or something similar and your machine will peg itself at full load while it tries to move the messages. This is when postsuper shines. There is a lot more to postfix and postsuper but these three commands can help you save your server and the people that depend on it for email in certain situations.

• postsuper -d (deletes messages)
• postsuper -h (move messages to Hold queue)
• postsuper -r (requeue messages, can requeue messages in Hold to Incoming/Active)

Each of the above commands take a third argument ‘queue_id’. This value can be ALL (must be all caps) to tell it to apply the operation to all messages in all queues, a dash (-) to tell it to apply the operation to queue_ids provided at stdin, or a specific message’s queue_id. The stdin option is especially nice as you can create a file with a queue_id on each line and then have postsuper process all of those messages in one run. You can also do ALL [queue name in lowercase] to apply the operation to all messages in a specific queue. For example, postsuper -d ALL hold would delete all messages in the Hold queue. Do NOT leave the queue name off if you are trying to do this else the ALL will snag everything in all queues.

Using the commands above, this is how you could deal with a single host, or a small set of hosts that is slamming your server with messages. Say these messages are not important and do not need to be read but they are backing up the postfix queues and preventing delivery of real email to your users.

(1) postsuper -h ALL

• Pushes all messages (the junk and the valid ones) to the Hold queue so you can breathe and think.
• Empties out the Incoming/Active queues so that new messages can be effectively delivered.
• At this point you need to stop the hosts sending the volume you don’t want, either by blocking the hosts if it is an attack or by fixing the issues if it is a misconfiguration or issue with a server you have control over.

(2) Next you would want to delete all of the messages you don’t want from the Hold queue. There are several approaches here, but this is one:

• Create a file containing the information on the bad messages:
  mailq | grep theEnemyHostName > bad_messages.txt
• Write a script to generate a new file from bad_messages.txt that contains just the queue id on each line, nothing else. Use perl or whatever you are comfortable with. The queue id is generally the first chunk of output for each message printed by mailq so you likely just need to grab the first eleven characters of each line.
• postsuper -d - < list_of_queue_ids.txt. This will delete all of the messages you identified.

(3) Requeue all of the remaining, valid messages with postsuper -r ALL.

All of the above applies to Zimbra as well. Even though it has that fancy web-based administrative interface and a real nice way to view and filter your mail queues that interface always blows up and pukes errors when I try to do an operation involving any kind of heavy message volume. So if you get in a similar jam the CLI is what you have to use. The postsuper command is also very fast so even if you have 10’s or 100’s of thousands of messages jamming things up it should not be a problem.

I posted a short while back about excitement surrounding a migration from Exchange 2003 to Zimbra for our company. The migration has had its ups and downs and now that it has happened and I have had a couple weeks to dig in as both a user and administrator I would like to share our experience.

The general takeaways are that Zimbra isn’t perfect. It does some things worse than Exchange and some things better but the balance, in my opinion, slants heavily in Zimbra’s favor. I’ll break it up into migration and then administration/usage.

The Migration

The migration was a bruiser. It involved a couple nights of failed attempts and then a brutal 6pm - 4am effort to get everything finished well enough to go to sleep. I had a sysadmin helping me that knew his stuff so the details of how to complete it aren’t here (he handled most of the work), just the headaches I saw. The issues included:

• The bulk migration tool was not able to migrate calendars.
• The individual .pst importing tool also was not able to migrate the calendars. It would just fail like crazy and then give up because the error count was too high. For users with 2k+ appointments the migration would fail after only a few dozen events. I eventually got these calendars over by doing .pst exports/imports with Outlook itself rather than trying to use server-side migration tools.
• We had to run the bulk migration over 2 nights because it took a long time. This isn’t a huge surprise because we had 100’s of 1000’s of emails, events, and contacts to migrate but the issue is that the second run re-imported everything imported in the first batch despite settings to the contrary. This essentially created duplicates of all emails and contacts.

To remove the duplicates of emails I used a perl script found at this page (this script actually worked fantastic). For contacts I used the Zimbra CLI to bulk clear the applicable address books and used client apps to re-import cleanly.

Administration/Usage

Zimbra started to shine after the migration ordeal. We immediately had all of our OSX users sync’ing their iCal, Apple Mail, and Address Book apps with the server, I had most of the Outlook users on the Zimbra Outlook connector without much effort, and most things worked well. There were a few issues I encountered.

• The Outlook connector worked flawlessly in XP Pro but was very difficult to install in Vista. You need to follow the tip here and then just keep trying until it works. If it doesn’t work remove the program and try again. I really hate Vista and the fact that it makes things so hard.
• The activesync with Windows Mobile is pretty flaky. It fails often for no apparent reason. I settled on using IMAP for email and just sync’ing my contacts and calendar and this seems to work consistently. It was as if it was stumbling over the greater volume of items to sync when the email was part of it.
• I’m not real happy with the calendar sharing. Without admin intervention a user must share their calendar with each individual user and each of those individuals must login to the web interface to accept the share and see it. These notifications cannot be accepted in Mail/Outlook/Entourage or whatever else. Once these calendars are accepted though you can use almost any app you want as your calendar and that is nice.
• There are connector apps for almost everything, but many of them are not updated to the latest versions of their target apps and none of them are completely polished and perfect. The Outlook and OSX ones seem to be the best but those also are not without issues.

In general though Zimbra works pretty well. I have calendar and contacts sync’d with my laptop using the OSX sync services and also sync’d to my Windows Mobile phone using activesync - a setup that never would have been possible with Exchange (without Entourage, but Entourage sucks in my opinion).

There are shortcomings but as I have worked through various user issues I have discovered what I believe is Zimbra’s biggest strength - its openness and open source underpinnings. It is a huge, powerful piece of code and between the CLI and the REST API you can do almost anything as an admin. Now that I am getting the hang of it I have created a set of quick scripts to interact with the CLI for doing things like auto-mounting calendars shared with distribution groups (getting around the email acceptance bummer mentioned above). The REST API is great and documented a bit here. It is completely trivial to export people’s contacts or calendars and to constrain what is exported using different parameters using the REST API.

Another big advantage in Zimbra’s favor is the community is quite strong and helpful. They have a wiki, forums, and bugzilla all very active and open.

So this is a bit of a ramble, but overall I am exceptionally happy that we made this switch. Zimbra is not perfect but it is powerful and utterly open making it possible to find workarounds for almost anything and it helps that it runs on Linux as well.

I have mentioned it in passing before, but almost every server associated with our company is running Linux (and Mac has managed to take over the workstations surprisingly fast - only 2 windows machines being used now). The last hold out on the server side was the Exchange server we setup when we first got an office that for obvious reasons had to be running Windows Server 2003. This was the same server I had the raid fun with.

Finally, after a year+ now of me hating Exchange solo, the requests and general feeling of the office has shifted against it across development AND sales and the migration to Zimbra is scheduled to be completed next week. I couldn’t be happier about it. Among many things I am most looking forward to administering a Linux machine, having a better web client (that doesn’t change by browser), and Apple iSync support. I am also looking forward to the Blackberry support which despite my unwanted but nontrivial experience with Windows servers I absolutely could not make work with Exchange.

To properly send Exchange on its way I thought I would enumerate some of the many reasons I hate it

• It runs on Windows. Windows is decent for a workstation but makes for an awful server in my opinion. Perhaps it comes down to experience, but I feel that the Windows approach to server administration (meaning hundreds of obscure windows, tabs, and buttons) requires more effort to learn, involves completely unnecessary abstractions over known technology, and makes everything you need to do take longer. They are unstable, require reboots to update (wtf?), and you have to use remote desktop to administer them. Enough about Windows as a server in general, back to Exchange.
• There is no reasonable method for setting up a catch all. Read this page if you need to do it and prepare to be disgusted.
• There is no reasonable method for forwarding email. How did they mess this one up so badly? It seems that the ability to setup an email forward would be a core feature of server software designed to send and receive email. To do this you have to create a dummy contact with the forward email, then create an exchange user account (with a different name and username else there is a collision), then configure that exchange user account to forward its mail to the dummy contact record, which will then cause the email to be forwarded to the final destination. Completely ridiculous as it bloats the active directory listing with loads of dead entries and takes too many steps to setup.
• I have had a lot of trouble with Exchange’s SMTP connectors where HTML emails headed towards external email accounts (via forwarding hack mentioned in previous bullet) back up in the queues for absolutely no reason and prevent messages from being delivered for hours sometimes.
• It doesn’t support iSync as far as I know.
• It doesn’t have spam filtering built in (it kind of does but it does an awful job in our experience).
• The web client is pretty terrible, and if you try to access it with anything other than IE is takes a severe dive to awful. In the non-IE mode you can’t search, can’t create folders, can’t create rules, and it is genuinely unusable.

The only big strength Exchange offered, and the reason we used it to begin with, was the calendar synchronization and thankfully Zimbra has arrived to offer an alternative to the mess that is Exchange. Zimbra is now feature rich, stable, and validated by huge installations such as the one at Georgia Tech. The feedback and reviews are glowing and the documentation makes it clear that all the little things I hate about Exchange because they take too long or are too cludgy are quick command line or file editing steps.

I’ll post again once I have put some hours of usage in with some content that doesn’t mention Exchange once and instead talks about Zimbra. I think it is safe to say though that if you are starting a company just skip Exchange from the start. You can get hosted Zimbra just as you can hosted Exchange if you don’t want to manage your own server.