server4 outage - Page 6

fmfan · 11 Nov 2005, 05:03 PM

Quote:

Originally posted by DavidJX023
I feel bad for the guy with the job interview.

I don't want to be too insulting towards Fastmail because I have enjoyed so many years of excellent service; but, I would just like to compare Fastmail to FEMA's response to Katrina right now.

You mean: "Is there something I should tweak?"

anachron · 11 Nov 2005, 05:04 PM

To Jeremy H., thank you for the up-front status update. I don't expect any direct reply from you as doing that would get you involved in a long thread you don't have time for right now.

It is a little unclear from the update what the timeline is.

All the mail currently queued since the beginning of the outage -- will that appear on server4 very shortly after it's brought back online? I would hope so. That should be A1 top priority.

I want to politely but strongly echo the concern others are raising -- Fastmail.fm's first and by far most imporant service is basic fast, reliable email. I would gladly sacrifice the last year's interface & feature add-ons - SMS messaginge, WebDAV access, etc. - ALL of them, if it could have avoided this one single outage. (I.e. the time spent on those new features could have been dedicated instead to building in more redundancy.)

Again thanks for the up-front update, I am happier to start the process of redirecting my domain MX records now, late at night (California time), than at 8:00am PST this morning.

Good luck with the rebuild.

janniklindquist · 11 Nov 2005, 05:11 PM

What do people consider the best alternative to FastMail? Bluebottle? Mailsnare?

djoyal · 11 Nov 2005, 05:11 PM

I can't belive what I am reading on the fastmail banner - No email for 2 days??? I am in the middle of job interviews and am using my fastmail address for everything. Employers are trying to send me directions for interviews and job information as I type.

I have an enhanced account and find this completely unacceptable. Previous outages were frustrating but not catastrophic. I am done with fastmail and want my money back.

jolycu · 11 Nov 2005, 05:18 PM

Quote:

Originally posted by FMRocks
I don't think Server 4 has any free or member level accounts. It was brought online mostly to serve Enhanced users.

There must be some Member level accounts on that server because we have 2 Member accounts (one under myfastmail.com and the other under mailcan.com) that we can't access. Also, my Full level account under myfastmail.com is inaccessible too.

A friend has a free account under myfastmail.com, and she can still access her account!

AlexR · 11 Nov 2005, 05:19 PM

Time to review server set-up strategy methinks. Anything like a RAID server exist? So if one goes down, the other kicks in?

2 days - which is the timescale showing up when you try to login from the web interface does seem to be a mighty long time.

These unpredicable outages are worrying - I use my account as a work tool - without it I have problems.

I have not even thought about finding an alternative to FM - til now that is. I pay for this service because I expect greater reliability - but maybe $20 a year is not enough to ensure this (but then neither is 40 bucks).

I want concrete assurances that this will never happen again (although we have already had such assurances...), otherwise bye bye Fastmail - although this will make me sad because it has been very good - but the current situation is not good for people such as myself who rely on email for getting their jobs done.

earl7 · 11 Nov 2005, 05:22 PM

Quote:

Originally posted by Sherry
I know you may be to busy to answer a lot of questions now but can't resist asking why you would restore the Junk and Trash first? Also, will new mail be coming in during this time of restoring the backups or will the new and queued mail wait until after the backups are restored?

Sherry

Good point Sherry.
Sounds like the restore is from tapes.
Can't help but ask
1) How a (assuming RAID 5) redundant system can suffer this kind of outage.
But more to the point:
2) How a 9 server system is not setup with redundancy to avoid this situation
That was what I assumed when I was sold on this and recommended to clients and colleagues as per this link:
http://wiki.fastmail.fm/index.php/InfraStructure

Hope the fix goes quick and smooth.
Hope the answers come soon after too...

Earl from SF via Melbourne

kander · 11 Nov 2005, 05:23 PM

Hmm. This is not good... not good at all.

I'll definitely be looking into a more secure mail setup again, having tossed it all out recently for simplicity. Woe on me, and us.

You know... I really envy those people that use IMAP clients now. Perhaps I'll look into that as a means of backup archiving now... does have some merits to it, I guess.

--K

fmfan · 11 Nov 2005, 05:23 PM

Quote:

Originally posted by djoyal
I can't belive what I am reading on the fastmail banner - No email for 2 days??? I am in the middle of job interviews and am using my fastmail address for everything. Employers are trying to send me directions for interviews and job information as I type.

I have an enhanced account and find this completely unacceptable. Previous outages were frustrating but not catastrophic. I am done with fastmail and want my money back.

Totally agree with all and very sympathetic - seriously.

Hard lesson to learn, but don't be dependent on one email address in such important circumstance: Always *request* communications to 2 email addresses.

Any system can go down at anytime - continually lose power at work - boondocks of central Houston, TX, and very few UPS in the building.

For me, job interviews are few and long time between.

chrisjam · 11 Nov 2005, 05:26 PM

If you run internet servers, you will plan for redundancy against the common problems that you get, and then you will have disaster management plans against the more unlikely plans. Obviously, a corrupt filesystem is not something that happens every day, especially as seems likely here, the corruption traverses a RAID array.

(Note, if the corruption is being caused by faulty memory etc then no amount of RAID or SAN is going to stop that faulty memory changing the data and then writing it to disk wrong!)

This is extremely irritating, but it's just one of those things. Sometimes servers just stop working. I'm grateful that fastmail seem to take good backups and so we should get our service back even if we have to have a weekend without email.

(I really hope there are backups! When it comes back online I'll be thinking about a better email backup strategy for me).

DavidJX023 · 11 Nov 2005, 05:28 PM

Word of mouth is great advertisement; but, I'm glad that I stopped recommending quality products to friends and family a long time ago. I have long since learned my lesson, and so I can sleep in peace knowing that, although it doesn't look like I'll have email in the morning, at least I won't have any voicemails from irate friends.

I'm heading to bed where I'll be praying for a miracle.

anachron · 11 Nov 2005, 05:29 PM

Somehow I was hoping for the best, that maybe basic incoming email functionality would be restored within maybe 10 more hours, but...

This outage is very disappointing. I have lived through quite a few fastmail.fm outages, but this is probably the next-to-last one for me. If this ever happens again (and it's not World War III or some sort of global Internet traffic stand-still) I'll have to take my account (and those of my clients) elsewhere.

I don't see how this obvious point of failure (a single server) wasn't addressed long ago: even if the server catches on fire or is stolen in broad daylight, getting basic incoming email flowing again should be as simple as A) restoring the *account* info (just the basic configuration: username, password, domains, forwarding settings, etc.) from a hot-backup server (or two!) exclusively dedicated to storing just that info, to a brand-new or hot-spare standby IMAP/POP server and then B) turning on DNS so that the fastmail SMTP servers start unloading queued mail onto it. Boom, done.

I don't care as much about my stored IMAP or POP mail from the last X days, weeks, or months -- what I really want is not to have to redirect my domain (a significant PITA) and call / email a bunch of clients saying "sorry, any mail you sent me in the last 12-18 hours I won't receive until Saturday. Please re-send it."

xlr9 · 11 Nov 2005, 05:29 PM

Quote:

Originally posted by kander
Hmm. This is not good... not good at all.

I'll definitely be looking into a more secure mail setup again, having tossed it all out recently for simplicity. Woe on me, and us.

You know... I really envy those people that use IMAP clients now. Perhaps I'll look into that as a means of backup archiving now... does have some merits to it, I guess.

--K

“rely on 2 email addresses” Mod: translated "hogwash!". I pay good money for my enhanced FM account and I DEMAND that it works when I need it. I have never experienced such a severe outage on ANY email service: work, personal, paid, unpaid! An outage of this magnitude is OUTRAGEOUS. I rely on my email every day.

WHAT DO YOU INTEND TO DO TO MAKE UR CUSTOMERS HAPPY AGAIN!!!!!?????

fmfan · 11 Nov 2005, 05:30 PM

Quote:

Originally posted by AlexR
Time to review server set-up strategy methinks. Anything like a RAID server exist? So if one goes down, the other kicks in?

But, RAID only backups whatever there is to backup.

And if what is being backed-up is corrupted, not of much use until file system, or database, rebuilt?

Need a more-secure, stable, file system or database?

Either virus, trojan, or file system, database, not stable enough given operating environment, one would think.

larryjhs · 11 Nov 2005, 05:32 PM

Like many of us, I'm furious because of the lack of 'customer service' ie updates. Pity the poor bunnies who don't know how to find the forum or service status. I've worked at an ISP and they let people know of outages -- and in this case, it should have been in big print on the Fastmail front page with regular updates (Well, now it is, a workday -- that's what it is, late). It's so cheap now to call internationally, an up-to-the-minute recorded message might be the way to go.

I'm really surprised about this problem, since Jeremy is the owner of Fastmail and a management consultant based in Melbourne Australia where I am -- where's the customer service? Australia needs the export income from businesses like Fastmail!

If I knew where Jeremy lived, I'd personally get on my bike and shake my fist!

See http://www.theage.com.au/articles/20...538891792.html

for a story about Fastmail, and other Age articles have been very generous and consistently recommended Fastmail. I hope they don't get a wind of today's episode.

Not happy Jeremy, to paraphrase a well-known advertisement here.:

11 Nov 2005, 05:04 PM	#77
anachron Member Join Date: Jul 2003 Location: San Francisco, CA Posts: 35	Status update To Jeremy H., thank you for the up-front status update. I don't expect any direct reply from you as doing that would get you involved in a long thread you don't have time for right now. It is a little unclear from the update what the timeline is. All the mail currently queued since the beginning of the outage -- will that appear on server4 very shortly after it's brought back online? I would hope so. That should be A1 top priority. I want to politely but strongly echo the concern others are raising -- Fastmail.fm's first and by far most imporant service is basic fast, reliable email. I would gladly sacrifice the last year's interface & feature add-ons - SMS messaginge, WebDAV access, etc. - ALL of them, if it could have avoided this one single outage. (I.e. the time spent on those new features could have been dedicated instead to building in more redundancy.) Again thanks for the up-front update, I am happier to start the process of redirecting my domain MX records now, late at night (California time), than at 8:00am PST this morning. Good luck with the rebuild.

11 Nov 2005, 05:11 PM	#79
djoyal Junior Member Join Date: Jul 2004 Posts: 28	Friggin Ridiculous!! I can't belive what I am reading on the fastmail banner - No email for 2 days??? I am in the middle of job interviews and am using my fastmail address for everything. Employers are trying to send me directions for interviews and job information as I type. I have an enhanced account and find this completely unacceptable. Previous outages were frustrating but not catastrophic. I am done with fastmail and want my money back.

11 Nov 2005, 05:29 PM	#87
anachron Member Join Date: Jul 2003 Location: San Francisco, CA Posts: 35	Wow. Somehow I was hoping for the best, that maybe basic incoming email functionality would be restored within maybe 10 more hours, but... This outage is very disappointing. I have lived through quite a few fastmail.fm outages, but this is probably the next-to-last one for me. If this ever happens again (and it's not World War III or some sort of global Internet traffic stand-still) I'll have to take my account (and those of my clients) elsewhere. I don't see how this obvious point of failure (a single server) wasn't addressed long ago: even if the server catches on fire or is stolen in broad daylight, getting basic incoming email flowing again should be as simple as A) restoring the account info (just the basic configuration: username, password, domains, forwarding settings, etc.) from a hot-backup server (or two!) exclusively dedicated to storing just that info, to a brand-new or hot-spare standby IMAP/POP server and then B) turning on DNS so that the fastmail SMTP servers start unloading queued mail onto it. Boom, done. I don't care as much about my stored IMAP or POP mail from the last X days, weeks, or months -- what I really want is not to have to redirect my domain (a significant PITA) and call / email a bunch of clients saying "sorry, any mail you sent me in the last 12-18 hours I won't receive until Saturday. Please re-send it."

11 Nov 2005, 05:32 PM	#90
larryjhs Junior Member Join Date: Dec 2004 Posts: 29	So what is Fastmail? Like many of us, I'm furious because of the lack of 'customer service' ie updates. Pity the poor bunnies who don't know how to find the forum or service status. I've worked at an ISP and they let people know of outages -- and in this case, it should have been in big print on the Fastmail front page with regular updates (Well, now it is, a workday -- that's what it is, late). It's so cheap now to call internationally, an up-to-the-minute recorded message might be the way to go. I'm really surprised about this problem, since Jeremy is the owner of Fastmail and a management consultant based in Melbourne Australia where I am -- where's the customer service? Australia needs the export income from businesses like Fastmail! If I knew where Jeremy lived, I'd personally get on my bike and shake my fist! See http://www.theage.com.au/articles/20...538891792.html for a story about Fastmail, and other Age articles have been very generous and consistently recommended Fastmail. I hope they don't get a wind of today's episode. Not happy Jeremy, to paraphrase a well-known advertisement here.:

11 Nov 2005, 05:11 PM	#78
janniklindquist Essential Contributor Join Date: Jan 2002 Location: Copenhagen, Denmark Posts: 300	What do people consider the best alternative to FastMail? Bluebottle? Mailsnare?

11 Nov 2005, 05:19 PM	#81
AlexR Senior Member Join Date: Aug 2002 Location: Milan, Italy Posts: 142	Time to review server set-up strategy methinks. Anything like a RAID server exist? So if one goes down, the other kicks in? 2 days - which is the timescale showing up when you try to login from the web interface does seem to be a mighty long time. These unpredicable outages are worrying - I use my account as a work tool - without it I have problems. I have not even thought about finding an alternative to FM - til now that is. I pay for this service because I expect greater reliability - but maybe $20 a year is not enough to ensure this (but then neither is 40 bucks). I want concrete assurances that this will never happen again (although we have already had such assurances...), otherwise bye bye Fastmail - although this will make me sad because it has been very good - but the current situation is not good for people such as myself who rely on email for getting their jobs done.

11 Nov 2005, 05:23 PM	#83
kander The "e" in e-mail Join Date: Jan 2002 Location: The Netherlands Posts: 4,112	Hmm. This is not good... not good at all. I'll definitely be looking into a more secure mail setup again, having tossed it all out recently for simplicity. Woe on me, and us. You know... I really envy those people that use IMAP clients now. Perhaps I'll look into that as a means of backup archiving now... does have some merits to it, I guess. --K

11 Nov 2005, 05:26 PM	#85
chrisjam Senior Member Join Date: Jan 2004 Location: United Kingdom Posts: 104	If you run internet servers, you will plan for redundancy against the common problems that you get, and then you will have disaster management plans against the more unlikely plans. Obviously, a corrupt filesystem is not something that happens every day, especially as seems likely here, the corruption traverses a RAID array. (Note, if the corruption is being caused by faulty memory etc then no amount of RAID or SAN is going to stop that faulty memory changing the data and then writing it to disk wrong!) This is extremely irritating, but it's just one of those things. Sometimes servers just stop working. I'm grateful that fastmail seem to take good backups and so we should get our service back even if we have to have a weekend without email. (I really hope there are backups! When it comes back online I'll be thinking about a better email backup strategy for me).

11 Nov 2005, 05:28 PM	#86
DavidJX023 Senior Member Join Date: Apr 2002 Posts: 120	Word of mouth is great advertisement; but, I'm glad that I stopped recommending quality products to friends and family a long time ago. I have long since learned my lesson, and so I can sleep in peace knowing that, although it doesn't look like I'll have email in the morning, at least I won't have any voicemails from irate friends. I'm heading to bed where I'll be praying for a miracle.