(Why) is FastMail down? I want better answers.

elvey · 15 Aug 2003, 07:41 AM

Well, yes, there is a region-wide power outage. BUT the new york times website is up, and getting updated, for example.

Fastmail's backup systems aren't looking good, at all!

http://admin.fastem.com/uptime.html shows no downtime. It claims the servers are fine. Not professional.

The european backup servers are down. They're not accepting mail, that is. Not professional.

OTOH:

http://average.xaffire.com/ shows a lot of sites are down.

Also, I can't even get out of California when I try to traceroute to fastmail.

I can't even resolve www.nyi.net!

Major backbones and colo centers are supposed to have generator backup power, not just battery backup power.

jgoldberg · 15 Aug 2003, 08:00 AM

First of all, even if the fastmail servers have appropriate back-up power, much of the network equipment near by also need to be up.

I don't know FM's details, but I've been in plenty of machine rooms during power failures. It takes a lot of power to run the air-conditioners needed to keep a server farm running. Emergency generators may or may not be able to cut it for an extended amount of time.

As for the European MX giving "4xx" responses. That is exactly what it should be doing. SMTP was designed when the network was far less reliable than today. And a 4xx response is the cleanest thing to do.

I'm sure there are lessons here, but they all have costs. I don't know what FMs budget is and spreading around a few more MX backups on different nets would almost certainly be on a wish list, but there may well be other things that money should be spent on.

At the risk of incurring the wrath of the moderators, when you talk about "unprofessional" may I ask how many machine rooms you've worked in?

Terry · 15 Aug 2003, 08:00 AM

They are saving up for the batteries.......

elvey · 15 Aug 2003, 08:09 AM

Quote:

Originally posted by jgoldberg
First of all, even if the fastmail servers have appropriate back-up power, much of the network equipment near by also need to be up.

Correct, but FM/NYI is supposed ot make sure this is the case.

Quote:

I don't know FM's details, but I've been in plenty of machine rooms during power failures. It takes a lot of power to run the air-conditioners needed to keep a server farm running. Emergency generators may or may not be able to cut it for an extended amount of time.

Then why pay a premium for hosting your machines in a data center? What you're supposed to be getting is super-reliable connectivity, cooling and power! (and security, fire, water protection...)

Quote:

As for the European MX giving "4xx" responses. That is exactly what it should be doing. SMTP was designed when the network was far less reliable than today. And a 4xx response is the cleanest thing to do.

That's not true. Not when the primaries are down for an extended period of time.

Quote:

I'm sure there are lessons here, but they all have costs. I don't know what FMs budget is and spreading around a few more MX backups on different nets would almost certainly be on a wish list, but there may well be other things that money should be spent on.

I'm not asking for more. I'm asking for the one that exists to work as it should!

Quote:

At the risk of incurring the wrath of the moderators, when you talk about "unprofessional" may I ask how many machine rooms you've worked in?

About 7.

BTW:2 minutes ago:
http://www.dnsreport.com/tools/dnsre...n=fastmail.fm:

ERROR: I could not connect to any of your mailservers!

ERROR: smtp.eu.messagingengine.com Greeting indicates server is down: 421 smtp.eu.messagingengine.com: Too many concurrent SMTP connections; please try again later. .
smtp.eu.messagingengine.com: Could not receive data: Winsock error #10053 (state 2).
smtp.us2.messagingengine.com: Timed out [Last data sent: HELO www.DNSreport.com ]
smtp.us.messagingengine.com: Timed out [Last data sent: HELO www.DNSreport.com ]

BBM · 15 Aug 2003, 08:13 AM

I think Edwin made a pretty good point earlier in the other thread. This outage may be too big for any reasonable contingency plan to handle. If EVERYTHING around you is out, how does backup power for your server help?

A couple of people have reported that the NY Times website is still running. How do we know that their site is hosted in NY? It could be in Omaha for all we know. Or maybe they have a backup server is in Omaha. Just something to think about...

As for http://admin.fastem.com/uptime.html, its failure is nothing new. If Jeremy and Rob or the FM docs regularly pointed to this resource as an accurate reflection of FM's status, then its lack of accuracy would be upsetting. However, they don't and I think they would be happy if everyone just ignored it. I do.

**JeffK** · 15 Aug 2003, 08:17 AM

Oh No. Not Omaha again.

fmfan · 15 Aug 2003, 08:19 AM

Same as in another post - but looks like two threads about same thing:

Can presently get my uReach.com accounts email.

Can presently get my MailSnare.net account email.

Cannot presently get my FastMail.fm accounts email.

So - seems to me - must be FastMail.fm servers - for me.

***Edwin*** · 15 Aug 2003, 08:23 AM

Since the blackout is being called "the biggest power blackout in history" I believe it's a little unfair to automatically expect contingency plans to be 100% effective against such an unprecedentedly large problem.

Presumably, the datacenter conducted various exercises in preparing its disaster recovery plans, but "total failure of power to most of the Eastern US" was almost certainly outside the scope of those exercises.

Can we please cool down a little and wait until the power is restored? If Fastmail stays down after that, there's much more cause for concern IMO!

The really big issue in any case is not "why are they down" but "how can they ensure they won't go down again in future under similar circumstances?"

Jacobl · 15 Aug 2003, 08:24 AM

Quote:

Originally posted by fmfan
Same as in another post - but looks like two threads about same thing:

Can presently get my uReach.com accounts email.

Can presently get my MailSnare.net account email.

Cannot presently get my FastMail.fm accounts email.

So - seems to me - must be FastMail.fm servers - for me.

And all of those are hosted in NY?

I don't see your point why fastmail should be up just because other mail services are up. HOtmail is up, so why isn't Fastmail??

Edwin's point still makes sense even if uReach.com, MailSnare.net are both up.

EDIT: Edwin posted right before I did - I do agree with him. I mean, yes ask for improvements if possible later. Yeesh, I remember a glorious time where email didn't exist... I wonder how we survived

***Edwin*** · 15 Aug 2003, 08:27 AM

Mailsnare.net is hosted in Houston, Texas, so it's hardly surprising it's unaffected.

You need to find:-

A) Companies hosted in NYC
B) Companies hosted in the SAME datacenter as Fastmail

If you find companies up and running in situation A) it means they were luckier in their choice of datacenter than Fastmail

If you find companies up and running in situation B) it indicates that it is probably a server, not a datacenter issue

Absent data on point B), no useful information can be obtained from comparisons with any other sites, beyond the obvious "hey, they're up and Fastmail isn't"...

BBM · 15 Aug 2003, 08:27 AM

Quote:

Originally posted by fmfan
Can presently get my uReach.com accounts email.

Can presently get my MailSnare.net account email.

Do you know where these services locate their servers? If they're not somewhere between Detroit and New York, the fact that they're still up doesn't really tell us much about their robustness compared to FastMail. They could be running out of someone's basement (not saying they are, just "what if") and as long as they were out of the reach of the blackout, they would be fine.

Edit: Wow, TWO people beat me to it.

elvey · 15 Aug 2003, 08:27 AM

Quote:

Originally posted by BBM
I think Edwin made a pretty good point earlier in the other thread. This outage may be too big for any reasonable contingency plan to handle. If EVERYTHING around you is out, how does backup power for your server help?

A couple of people have reported that the NY Times website is still running. How do we know that their site is hosted in NY? It could be in Omaha for all we know. Or maybe they have a backup server is in Omaha. Just something to think about...

Quote:

I did a traceroute. It's clearly in New York, or at least the blackout area.

As for http://admin.fastem.com/uptime.html, its failure is nothing new. If Jeremy and Rob or the FM docs regularly pointed to this resource as an accurate reflection of FM's status, then its lack of accuracy would be upsetting. However, they don't and I think they would be happy if everyone just ignored it. I do.

Someone recently posted that it does work, just not perfectly. So that's why I'm bringing it up again. This outage makes pretty clear that it isn't working at all.

Jacobl · 15 Aug 2003, 08:32 AM

Quote:

Originally posted by fmfan
1. Have *no* idea.
2. Did not (mean) to say that - just stating facts re my situation - in case FastMail.fm admins - Jeremy, Rob - interested.
3. Of course, definitely agree

Cheers - I'm cool - not in NY! ...

Okay... I'm sorry, the whole thing about text etc not being able to express tone etc..

bitequator · 15 Aug 2003, 08:32 AM

Quote:

Originally posted by Edwin
Mailsnare.net is hosted in Houston, Texas, so it's hardly surprising it's unaffected.

Hey I didn't know that! Where exactly, maybe I can drive down for a visit...

Yeah our local news keep reassuring us that all power grids for Houston and all of TX are not connected to other grids around the country -- kinda like an isolated island -- and how we don't have to worry about this at all!

fmfan · 15 Aug 2003, 08:32 AM

Quote:

Originally posted by Jacobl
And all of those are hosted in NY?

I don't see your point why fastmail should be up just because other mail services are up. HOtmail is up, so why isn't Fastmail??

Edwin's point still makes sense even if uReach.com, MailSnare.net are both up.

1. Have *no* idea.
2. Did not (mean to) say that - just stating facts re my situation - in case FastMail.fm admins - Jeremy, Rob - interested.
3. Of course, definitely agree

Cheers - I'm cool - not in NY! ...

edit: Sorry - got this post out-of-sync when did edit from
(mean) to (mean to) - & OK - yeah I'm in Hou, TX vicinity - had *no idea* where MailSnare.net servers hosted - *now* know - hey - FastMail.fm servers used to hosted SA, TX - no?
Have no idea re uReach.com servers - really don't care as long as work - though *would* find *interesting*.

Quote:

Originally posted by bitequator
Hey I didn't know that! Where exactly, maybe I can drive down for a visit...
... -- and how we don't have to worry about this at all!

Now I'm confused - thought had you figured out as a Canadian

Drive down from where - North TX?

15 Aug 2003, 07:41 AM	#1
elvey The "e" in e-mail Join Date: Jan 2002 Location: San Francisco Posts: 2,458	(Why) is FastMail down? I want better answers. Well, yes, there is a region-wide power outage. BUT the new york times website is up, and getting updated, for example. Fastmail's backup systems aren't looking good, at all! http://admin.fastem.com/uptime.html shows no downtime. It claims the servers are fine. Not professional. The european backup servers are down. They're not accepting mail, that is. Not professional. OTOH: http://average.xaffire.com/ shows a lot of sites are down. Also, I can't even get out of California when I try to traceroute to fastmail. I can't even resolve www.nyi.net! Major backbones and colo centers are supposed to have generator backup power, not just battery backup power.

15 Aug 2003, 08:00 AM	#2
jgoldberg Member Join Date: Jul 2003 Location: Plano, Texas Posts: 88	First of all, even if the fastmail servers have appropriate back-up power, much of the network equipment near by also need to be up. I don't know FM's details, but I've been in plenty of machine rooms during power failures. It takes a lot of power to run the air-conditioners needed to keep a server farm running. Emergency generators may or may not be able to cut it for an extended amount of time. As for the European MX giving "4xx" responses. That is exactly what it should be doing. SMTP was designed when the network was far less reliable than today. And a 4xx response is the cleanest thing to do. I'm sure there are lessons here, but they all have costs. I don't know what FMs budget is and spreading around a few more MX backups on different nets would almost certainly be on a wish list, but there may well be other things that money should be spent on. At the risk of incurring the wrath of the moderators, when you talk about "unprofessional" may I ask how many machine rooms you've worked in?

15 Aug 2003, 08:00 AM	#3
Terry The "e" in e-mail Join Date: Jul 2002 Location: VK4 Posts: 3,029	They are saving up for the batteries.......

15 Aug 2003, 08:13 AM	#5
BBM Essential Contributor Join Date: Dec 2002 Location: Chicago, IL Posts: 265	I think Edwin made a pretty good point earlier in the other thread. This outage may be too big for any reasonable contingency plan to handle. If EVERYTHING around you is out, how does backup power for your server help? A couple of people have reported that the NY Times website is still running. How do we know that their site is hosted in NY? It could be in Omaha for all we know. Or maybe they have a backup server is in Omaha. Just something to think about... As for http://admin.fastem.com/uptime.html, its failure is nothing new. If Jeremy and Rob or the FM docs regularly pointed to this resource as an accurate reflection of FM's status, then its lack of accuracy would be upsetting. However, they don't and I think they would be happy if everyone just ignored it. I do.

15 Aug 2003, 08:17 AM	#6
JeffK Moderator Join Date: Feb 2002 Location: Kingaroy, AU Posts: 3,179	Oh No. Not Omaha again.

15 Aug 2003, 08:19 AM	#7
fmfan Master of the @ Join Date: Jul 2002 Location: TX US Posts: 1,298	Same as in another post - but looks like two threads about same thing: Can presently get my uReach.com accounts email. Can presently get my MailSnare.net account email. Cannot presently get my FastMail.fm accounts email. So - seems to me - must be FastMail.fm servers - for me.

15 Aug 2003, 08:23 AM	#8
*Edwin* Administrator Join Date: Aug 2001 Location: UK Posts: 3,118	Since the blackout is being called "the biggest power blackout in history" I believe it's a little unfair to automatically expect contingency plans to be 100% effective against such an unprecedentedly large problem. Presumably, the datacenter conducted various exercises in preparing its disaster recovery plans, but "total failure of power to most of the Eastern US" was almost certainly outside the scope of those exercises. Can we please cool down a little and wait until the power is restored? If Fastmail stays down after that, there's much more cause for concern IMO! The really big issue in any case is not "why are they down" but "how can they ensure they won't go down again in future under similar circumstances?"

15 Aug 2003, 08:27 AM	#10
*Edwin* Administrator Join Date: Aug 2001 Location: UK Posts: 3,118	Mailsnare.net is hosted in Houston, Texas, so it's hardly surprising it's unaffected. You need to find:- A) Companies hosted in NYC B) Companies hosted in the SAME datacenter as Fastmail If you find companies up and running in situation A) it means they were luckier in their choice of datacenter than Fastmail If you find companies up and running in situation B) it indicates that it is probably a server, not a datacenter issue Absent data on point B), no useful information can be obtained from comparisons with any other sites, beyond the obvious "hey, they're up and Fastmail isn't"...