april 12, 2009

For about three years, Dreamhost has been a great hosting company.  Then, a month ago, I got a cheery email from the support team about a server move, it starts out:

This is just a note to let you know that we're about to move your account
to a new server in as soon as 72 hours (it may be later, but be prepared)!

We apologize for the lack of further notice, but this was an emergency
move as a perfomance and stability measure.

The move was anything but a "performance and stability measure."  At Dreamhost, mail and webhosting were always hosted on the same server -- and if you had a shell account, it was all done in your account directory.  This was great because I could run a custom install of SpamAssassin on my incoming mail using procmail.  

This also was a cause for performance issues for Dreamhost since everything usually was on one box using one filer.  So, their master plan was to split off mail and webhosting.  Mail would go to one custer of servers and webhosting would go to another -- shell accounts going with webhosting.  This is a great plan, it would decentralize things and allow webhosting to be faster as those webhosting boxes would concentrate on serving content -- and mail would be delivered quicker since those boxes would only take care of mail.

While on paper, this looks great, Dreamhost's execution of their plan was horrible -- the moving of webhosting went without a hitch -- though the performance was pretty subpar.  The webhosting box was consistently overloaded (with uptime loads of 15 or greater).  Response times on my websites were a bit more than average (which, while not critical for me, was a bit annoying).  

Email was absolutely horrible.  I got moved to the "homie" cluster and that is a horrible cluster to be on.  The first sign that things were bad was that after my accounts were moved (and I got an email to my secondary external email that the move was completed), I was without email for a day.  Not just me, but everyone on my domains.  Why?  They did not configure the accounts correctly.  Every account had incoming mail bounced all day long.  Support, which was great before, was now glacially slow.  After hours of bounced mails, Dreamhost finally got the mail issue resolved.

Not a week after that cleared up, this issue came up.  Basically, they lost the configuration for most all of the accounts on homie.  When I logged in, my email had been rolled back a week and nothing new was in my inbox anymore.  This outage took more than four days to clear up.  While it was happening, yes, you guessed it.  Mail was bouncing.  

So, that brings us to last week, when things were running OK, but really slow.  In the mornings, incoming email would be delayed between 15 to 60 minutes before hitting my inbox.  The email took a long time going through their ineffectual SpamAssassin boxes.  Spam would not get caught either because Dreamhost does not turn on Bayesian filtering.  So, while things were slow, they were still working -- until early Saturday morning when the SMTP servers complained that I couldn't send email because the disk was full.  The IMAP servers complained that I couldn't access my folders because they weren't available.  And, of course:  All incoming mail was bouncing.

It took Dreamhost support eight hours to get back to me with a "everything is fine, we tested it" reply.  Which was not true because when I went to send email:  Disk full.  When I went to check mail:  Folders not available.  After I replied and told them what happened, they did not respond, all the while my email was bouncing.

I was tired of what looked like a hacked together mail system that seemed to be buckling under the weight of a webhosting company which had oversold its capacity.  I was tired of my mail delivery botched everytime Dreamhost screwed up their configuration.

Many have suggested that I go with Google Apps for my mail.  Which would have been fine.  I use Google Apps for, which is where I sync my phone's calendar and contacts.  But, I don't use the email on  I still have trust issues with Google (both on the privacy front and the reports of Google shutting off accounts at random).  Also, no matter what IMAP syncing tool I tried, none of them worked very well with Gmail's funky IMAP implementation -- Well, Larch worked great, but it doesn't yet support syncing of folders recursively and I have lots of folders.  Lastly, Google takes control of the 'postmaster' account, I actually use the postmaster account for a lot of stuff -- since postmaster is allowed through SpamAssassin.  There was a lot of work to get things switched out of postmaster.

So, I got a VPS at Linode and got started building my own mail and web server.  It was a toss up between Linode and Slicehost.  Both got equally fervent support from people that I talked with and that replied to me on Twitter.  I went with Linode mainly because they gave be a little more for my money (12GB storage vs 10GB; 360MB RAM vs 256MB; and 200GB bandwidth vs 100GB).

I got really lucky because I activated a "full backup" of my account on Dreamhost about 8 hours before email went to hell.  A "full backup" means that Dreamhost uses an automated script and zips up each user's shell directory and mail directory into nice files.  I had started the download (around 4GB) and it was done when I woke up on Saturday.  The mail backup was just that, a snapshot of each user's Maildir.  Awesome!  That meant that I could unzip that into the new mailhost and bam!  Instant recovery -- no need to mess around with IMAP sync tools!

It took me all day yesterday to figure out how to build a mail and web server again -- mainly because I have not built a proper production one from scratch in such a long time.  But, once I got things figured out, it was pretty quick -- I documented it all, tore down what I had done, and redid everything from scratch.  That didn't take very long.  

I have come full circle now.  Shortly after I joined Yahoo! and after my home mail and web server blew up, I signed up with Dreamhost.  My rational was that for $7/mo, I could have someone else take care of all the SA work for me.  Now, I am back at doing my own SA work on my own machine -- mainly because I want to keep learning and I am a control freak (and it is fun).  Now, I can have my own site-wide SpamAssassin install with my own tweaked rules.  I can run PHP my way.  I can set things up my way.  The only drawback is that now I have two kiddos and my time is limited to working on this stuff late at night when they are sleeping.  Oh well, who needs sleep anyways?

I still have some tweaking and setup to do (like setting up some certs for https), but right now the server is running.  It is taking mail.  It is taking web requests.  And that is more than I can say about Dreamhost.  I just got an automated email from them apologizing that they haven't looked at my mail issue in more than 24 hours since our last correspondence (which was 12 hours since my mail disappeared).  So, I guess if I had stuck with Dreamhost, I still would not have any mail.

Thank you Dreamhost for three good years, it is too bad we have to go separate ways because of such circumstances -- in other words, you were good, but you suck now.  I still have about a year of hosting with them (and most of that was paid via referral bonuses -- for those people that I referred, I'm sorry that Dreamhost has let us both down.  I will leave some webhosting there for the time being -- especially for the one free domain that I got via Dreamhost.  But, all my critical and good stuff, I have already moved over to my Linode VPS.

<< back || ultramookie >>