Jeremy's Explanation and FAQ

[CyberSaint]

So, how did everyone feel about Jeremy's explanation of the extended downtime and the measures taken to correct such a problem in the future?

Personally, I see this whole incident as a freak occurrence that would very rarely happen. This last weekend however, was one of those "very rarely" times. I now feel that my data is secure, much more so than when everything first began last week. Actually given everything that happened, I am even MORE confident it is secure after it is all now back in place!

Fastmail is a great service, and I impressed with the leaders of the outfit. I'm also impressed with the willingness to offer some users an extra month's subscription free of charge.

To provide more redundancy on this end, I now have the ability to forward email away from FastMail and towards a secondary service if a problem arises.

In short, I'm comfortable and pleased with FastMail, perhaps even more so after this string of freak occurrences. I hope others stick with the service as well.

CS

[JeffK]

I think Jeremy's FAQ is a good one, pitched at a good level for a non IT technician like me.

I can inderstand the frustration of those the outage affected substantially. Although I would say if one is relying on email for income then a contingency plan is very important.

I think, for those who can afford it, it is best not to tie our email address or the email istself to one provider for transmission, display or storage. I changed my address away from my ISP about 3.5 years ago and then away from FM after the NY power blackout and adjacent outage. My system is still failure prone but that is because I haven't got it set up properly.

When the recent outage started I couldn't access any email because it was all forwarded to my FM account. However I then used my "upstream" storage (Tuffmail). If Tuffmail was to go down I would have to update my MX records.

However many on this forum do not have this luxury and cannot afford more than one paid account, if that.

FWIW, Jeff

[Akin]

Well, I wasn't one of the customers affected by the recent events. If I were affected, I guess I would be fairly unhappy with FM. However, as far as I could tell, there has been constant update on what was happening, and it seems that this event was just a freak coincidence. The FAQ that Jeremy posted explained what happened in detail, and most importantly, there is an outline of the steps being taken to prevent such event from occurring again.

I don't think I'll leave FM just because of this one (albeit major) incidence. To quote the tagline from the Mutt e-mail client, "All mail clients suck. This one just sucks less." Similarly, all e-mail providers suck. This one sucks less.

[JRobert]

That 3 drives with 15 months' run time on them failed within a couple of hours of each other seems so unlikely - if we assume the failures are independent events - as to challenge that assumption, independence being the major assumption supporting redundancy as a measure to improve reliability. Even two simultaneous failures would give reason to question it.

Two things stand out - the failed drives are the same age, and Jeremy called RAID 6 "one of the newest and most advanced".

Are these drives the same make and model? Manufactured in the same batch or at about the same time? Housed in a common enclosure, powered by the same power supply? Have they received any other treatment in common (shipping to the site, power cycling) that might affect all three in some way? How much industry experience is there with RAID 6? Could the technology itself have un-discovered wrinkle?

In any case, some assumptions need close examining. I believe Jeremy addressed this on the blog in Q10. I'm still comfortable with FM as my mail service provider. They seem to be examining their infrastructure design and open changing it if need be, they appear to have put in a major effort to mitigate the damage and repair it as quickly as could be done, kept us informed of what they were doing and why, and they've offered some compensation to the customers affected. I don't know what more one could expect from a service provider.

Some people have taken a hard look (or not) at whether this provider is the appropriate one for their needs and in some cases have decided to look elsewhere. That examination is reasonable too, necessary, in the case of those who consider their e-mail continuity to be, in some sense, "critical".

The bottom line for me (and note that I was not affected by the outage, only by the delivery suspension) is that, given the occurrence, my expectations were met. I would not expect such occurences to be common, or even repeated, if the design (of the business and the infrastructure) and the assumptions behind them are valid.

-jeff-

[Chipper]

I was also satisfied with Jeremy's explanation. While, I was not one of the people affected, it made me realize that I have a "tombstone mentality."
Do I ever backup my PC? No.
Do I have any personal redundancy plan for e-mail in place? No.

Even more distressing: Will I change my behavior based on this event? Probably not. I will probably think to myself, "Boy, I really need to start backing up my PC and develop a personal redundancy plan for my e-mail." However, after a few days, it will be out of sight out of mind.

[davy51]

Jerwemy did fine with his explanation

Fastmail is a great service and if the past is any indication it always will be

Any service can have problems and can fail thats why everyone must backup important information that way any freak accident like this wont create such a dissaster

[perianwyr]

To survive a problem like this, the easiest way is to handle it like DNS. The idea behind DNS is that you can have a domain name point to any IP, thus allowing the infrastructure to exist separately from the logical addressing system.

What you do is get a domain, any domain. Get it through a registrar that will do email forwarding (I use dreamhost.) Then, set up an address at your domain that forwards to Fastmail. Start using an IMAP client locally and download all your message contents nightly. Don't give people your Fastmail address, give them the domain address that you control.

This way, if Fastmail goes down, you can change the forwarding to another email account (such as gmail) for the duration of the failure. The downloaded message contents will allow you to get at archived messages.

With a system like this, failure is significantly less likely.

[Sard]

If they had to resort to a nightly backup, will the emails delivered after the backup up until the failure have been permanently lost?

[mpetersen]

That is a partial solution perianwyr. Typically you are not aware of the failure until the time when some or many of your inbound messages have already been lost, or been queued up for subsequent processing.

A fuller solution is redundancy. Have all of your inbound messages sent to fastmail and to gmail at the same time. Or fastmail and a second email provider of your choice.

I say 'fuller solution' only because there are probabilities of failure with ANY solution.

Martin

[Jeremy Howard]

Originally posted by Sard
If they had to resort to a nightly backup, will the emails delivered after the backup up until the failure have been permanently lost?

No. Emails receiving during this time were queued, and delivered after the restore was complete.

[haebby]

Originally posted by Jeremy Howard
No. Emails receiving during this time were queued, and delivered after the restore was complete.

Jeremy - just to make sure I got that right...

The question was whether "emails delivered after the backup up until the failure" were lost, i.e. what happened to emails that were successfully delivered *before* the system crashed, but *after* the daily backup was made.

Obviously, these emails were not backed up. Since the system was still working, I assume they were delivered succesfully, thus not being queued. So I would have assumed these messages were lost.

Likewise, what about changes I made on my account *after* the daily backup wasmade and *before* the system crashed? E.g. what if I moved messages?

Thanks.

[Jeremy Howard]

Originally posted by haebby
Obviously, these emails were not backed up. Since the system was still working, I assume they were delivered succesfully, thus not being queued. So I would have assumed these messages were lost.

I apologise - I did not read your message carefully enough.

Emails delivered (or moved, or copied) after the backup, but before the failure, were not lost.

After the 3rd drive failed, we were able to bring the RAID array online again in a degraded, read-only mode. Because the drive hadn't catastrophically failed, we were able to bring it back online. From there, we ran another incremental backup run, which was able to create a backup of users' emails since the previous nightly backup (which had finished a few hours earlier). After that, we deleted the entire array and data, replaced the 2 remaining failed drives (one had already been replaced), created a newly formatted volume, and began the restore from backup.

[Shelded]

i think you are asking a hypothetical and Jeremy was answering an actual.

The solution perianwyr described is good, could be improved to address Sard's point by forking the mail forwarding to two mail accounts. One Gmail/Runbox/GMX and one Fastmail would do nicely.

[registered_user]

Originally posted by Jeremy Howard
No. Emails receiving during this time were queued, and delivered after the restore was complete.

See, here I'm impressed. I think it's a testament to the robustness of the current architecture, that even when the account data for certain domain names was inaccessible, the mail for those accounts was still queued and not just rejected by the SMTP servers to avoid relaying. The SMTP servers must have to frequently update information from the main account servers, but when one of those servers died, they dealt with its absence gracefully.

I know when I break it down in my head, that it is not the most impressive feat of network engineering in the history of time, but I've seen far more expensive services and far larger services fail to do as well during their own "freak occurences". So far, in spite of this incident, FM is easily the most reliable mail service I've encountered. All of my mail was preserved and all of my incoming mail kept coming in. That's reliability.

[haebby]

Originally posted by Jeremy Howard
After the 3rd drive failed, we were able to bring the RAID array online again in a degraded, read-only mode. Because the drive hadn't catastrophically failed, we were able to bring it back online. From there, we ran another incremental backup run, which was able to create a backup of users' emails since the previous nightly backup (which had finished a few hours earlier). After that, we deleted the entire array and data, replaced the 2 remaining failed drives (one had already been replaced), created a newly formatted volume, and began the restore from backup.

Jeremy - thanks much. That's the info I was interested in.