Fastmail Down? - Page 3

brong · 13 May 2011, 10:54 PM

It's getting close to 1 hour, yes. We're hopefully not far off - but we had full disks followed by corrupted mailboxes databases - it's essential we get it right BEFORE we bring it back online. Recovering from backups of the DB file plus replicas to make sure it's all good.

(the small config error that got rolled out to a few machines is responsible for the runaway disk usage, and the new design where there's a single shared SSD for metadata means that it impacted all slots on the same machines.)

ChinaLamb · 13 May 2011, 10:55 PM

Quote:

Originally Posted by bblackmoor

Still down. Really surprised there are not more complaints here.

Complaint Complaint Complaint.

Doesn't do anything.

Really hope it gets back online soon...

Temmokan · 13 May 2011, 10:55 PM

The support form, as expected, isn't available as well, reporting "MessagingEngine.com Server Outage".

This the first time the outage is that long and there's no actual information on when it's expected to be over.

I wonder what was the misconfiguration with such consequences. Update it was explained, and it looks like this day proved the reputation of May the 13-th.

Good luck to the sysadms, I would be quite sorry to see my email lost.

ChinaLamb · 13 May 2011, 10:56 PM

Does anyone know whether mail is bouncing?

alka · 13 May 2011, 10:58 PM

Thank you Brong for continuing to give us updates. Silence is the killer. I can deal with a few hours downtime if you keep giving us juicy technical details. Makes me feel like I'm a part of fixing it.

johanborg · 13 May 2011, 10:59 PM

My own work mail is down here at Opera, yes

The status blog will be updated again with more info.

jerrycable · 13 May 2011, 11:00 PM

Good point. I just sent an e-mail from my Gmail account to my Fastmail account. We'll see if it bounces or not.

alka · 13 May 2011, 11:02 PM

Quote:

Originally Posted by jerrycable

Good point. I just sent an e-mail from my Gmail account to my Fastmail account. We'll see if it bounces or not.

Same here. No bounces after several minutes.

Doesnt FM have separate redundant servers for receiving and caching? In past outages, mail was simply held until the servers were back online.

ktheking · 13 May 2011, 11:04 PM

No access from webinterface. Access pops up ,but login not possible.
Neither for me or my customers.
Need solution quickly ,since it's affecting business of my customers which highly rely on mail.
Think I'd switch to own mailserver soon ,if this happens again ,and again.

kutsyy · 13 May 2011, 11:05 PM

Quote:

Originally Posted by brong

It's getting close to 1 hour, yes. We're hopefully not far off - but we had full disks followed by corrupted mailboxes databases - it's essential we get it right BEFORE we bring it back online. Recovering from backups of the DB file plus replicas to make sure it's all good.

Given that you are restoring from the backup (based on http://status.fastmail.fm/) how much data loss should we expect?

Temmokan · 13 May 2011, 11:09 PM

Also, do all the data are restored from backups, or only the corrupted mailboxes'?

And yes, how old are the backups?

Dimple · 13 May 2011, 11:12 PM

No bounces for me either when I send an test email to my FM account. However, the message hasn't been automatically forwarded into my gmail account either...

Hope we don't lose any data/email!

CyberSaint · 13 May 2011, 11:20 PM

Although there have been one or two sporadic outages over the past couple of years with FM, I've never lost or missed any emails due to the downtime.

Yes, FM being down is a pain, but in my opinion overall FM is still a bargain and a great service.

akorvemaker · 13 May 2011, 11:21 PM

Based on FM's infrastructure, I'd be extremely surprised if messages were bouncing or there is a single message lost. This just isn't the sort of outage that would impact them in that way.

Andy

EvilPeppard · 13 May 2011, 11:23 PM

In case it was missed earlier in this thread, here is where you can check for the latest status updates:

http://status.fastmail.fm/

13 May 2011, 11:04 PM	#39
ktheking Junior Member Join Date: Apr 2009 Posts: 15	Still down :( No access from webinterface. Access pops up ,but login not possible. Neither for me or my customers. Need solution quickly ,since it's affecting business of my customers which highly rely on mail. Think I'd switch to own mailserver soon ,if this happens again ,and again.

13 May 2011, 10:54 PM	#31
brong The "e" in e-mail Join Date: Jul 2004 Location: Melbourne, Australia Posts: 2,696 Representative of: Fastmail.fm	It's getting close to 1 hour, yes. We're hopefully not far off - but we had full disks followed by corrupted mailboxes databases - it's essential we get it right BEFORE we bring it back online. Recovering from backups of the DB file plus replicas to make sure it's all good. (the small config error that got rolled out to a few machines is responsible for the runaway disk usage, and the new design where there's a single shared SSD for metadata means that it impacted all slots on the same machines.)

13 May 2011, 10:55 PM	#33
Temmokan Member Join Date: Apr 2008 Location: Novosibirsk Posts: 31	The support form, as expected, isn't available as well, reporting "MessagingEngine.com Server Outage". This the first time the outage is that long and there's no actual information on when it's expected to be over. I wonder what was the misconfiguration with such consequences. Update it was explained, and it looks like this day proved the reputation of May the 13-th. Good luck to the sysadms, I would be quite sorry to see my email lost.

13 May 2011, 10:56 PM	#34
ChinaLamb The "e" in e-mail Join Date: Dec 2004 Location: a virtually impossible but finitely improbable position Posts: 2,320	Does anyone know whether mail is bouncing?

13 May 2011, 10:58 PM	#35
alka Essential Contributor Join Date: Apr 2003 Location: PA, USA Posts: 349	Thank you Brong for continuing to give us updates. Silence is the killer. I can deal with a few hours downtime if you keep giving us juicy technical details. Makes me feel like I'm a part of fixing it.

13 May 2011, 10:59 PM	#36
johanborg Essential Contributor Join Date: Apr 2010 Location: Oslo, Norway Posts: 213 Representative of: Opera.com	My own work mail is down here at Opera, yes The status blog will be updated again with more info.

13 May 2011, 11:00 PM	#37
jerrycable Member Join Date: Jan 2004 Location: Allen, TX Posts: 35	Good point. I just sent an e-mail from my Gmail account to my Fastmail account. We'll see if it bounces or not.

13 May 2011, 11:09 PM	#41
Temmokan Member Join Date: Apr 2008 Location: Novosibirsk Posts: 31	Also, do all the data are restored from backups, or only the corrupted mailboxes'? And yes, how old are the backups?

13 May 2011, 11:12 PM	#42
Dimple Junior Member Join Date: Aug 2003 Location: DC Posts: 17	No bounces for me either when I send an test email to my FM account. However, the message hasn't been automatically forwarded into my gmail account either... Hope we don't lose any data/email!

13 May 2011, 11:20 PM	#43
CyberSaint Essential Contributor Join Date: Apr 2004 Location: Hurricane, WV Posts: 206	Although there have been one or two sporadic outages over the past couple of years with FM, I've never lost or missed any emails due to the downtime. Yes, FM being down is a pain, but in my opinion overall FM is still a bargain and a great service.

13 May 2011, 11:21 PM	#44
akorvemaker Master of the @ Join Date: Nov 2002 Location: Canada Posts: 1,015	Based on FM's infrastructure, I'd be extremely surprised if messages were bouncing or there is a single message lost. This just isn't the sort of outage that would impact them in that way. Andy

13 May 2011, 11:23 PM	#45
EvilPeppard Essential Contributor Join Date: Aug 2006 Location: Centennial, CO USA Posts: 204	In case it was missed earlier in this thread, here is where you can check for the latest status updates: http://status.fastmail.fm/