All users now replicated

Jeremy Howard · 31 Oct 2006, 10:41 AM

We've reached a very exciting point this week - we now have all users on fully-replicated servers. This means that, in effect, every email is saved 6 times: on master server, plus RAID redundant copy; on slave server, plus RAID redundant copy; and in backups (also with a RAID redundant copy)!

The server3 problem over the weekend showed how non-replicated servers can create substantial outages - this was the last non-replicated server, and now all server3 users have been moved to replicated servers.

This would have been completed earlier, but our order of IBM hardware that was required to get this extra space, was delayed by two months (they told us every 2 weeks that it was just about to be delivered!)

For those of you who were on server3, I'm really sorry that this happened too late to avoid the enormous frustration that you would have experienced if you tried to access your email during the outage.

In the future, if there is a file-system corruption (as happened on server3), we will switch to the replica, which will take around 15-30 minutes. It would probably take 15-30 minutes to diagnose the problem as well, so in the end this would result in an outage of 30-60 minutes.

Hlao-roo · 31 Oct 2006, 11:24 AM

I wasn't among the affected users, but I was curious about how those who were will be compensated.

**Sherry** · 31 Oct 2006, 12:49 PM

Thanks for the info Jeremy. I don't remember ever reading that "every email is saved 6 times". Wow, sounds great.

As for the timing if there's a problem. Some time back I was moved from IMAP6 to IMAP5 when there was some problem. It took about the amount of time you mentioned but I think something was said about the "auto" move wasn't in place then. Is that something still coming in the future so the move is even less time? (not that the timing you're saying is any big thing)

Sherry

robmueller · 31 Oct 2006, 02:03 PM

We're not planning any failovers to be "automatic". I've added to Brons post on the status blog some more details in the update at the bottom.

http://status.fastmail.fm/?p=647

Rob

**Sherry** · 31 Oct 2006, 02:41 PM

Quote:

Originally posted by robmueller
We're not planning any failovers to be "automatic". I've added to Brons post on the status blog some more details in the update at the bottom.

Thanks Rob. Guess I read the automatic thing as something that was coming in the future. Bron may have been talking about something different than what I was thinking. I was going by the post Here. (It was some time back so it took me a bit to find it)

Quote:

---Again, things aren't in place for automatic failover - but it takes the recovery time for total hardware failure on the primary database server back from approximately 6 hours to more like 10 minutes -> half an hour.

Sherry

NJSS · 31 Oct 2006, 04:15 PM

Jeremy

I, and I anticipate many others, really appreciate seeing you posting here.

I hope we see you more frequently in the future, than of late.

I appreciate the problems with RSI, but am sure you are working round them.

It's really good news that everyone is now on fully-replicated servers - hopefully this will mean fewer long nights for the FM team.

hadaso · 31 Oct 2006, 06:11 PM

Quote:

Originally posted by Sherry
... "every email is saved 6 times". Wow, sounds great. ...

I think this is "oversimplifying" things. There are 3 "sets of copies" (master, slave, backup). I think we know that at least the master is on a RAID-6 array. This means that data is "broken up" and components are spread among the different disks in the array so that even if two pieces are missing (two disks are removed) the system can recunstruct the data using the remaining pieces. So this means that removing two copies you still get a copy. So essentially it seems like its equivalent to having at least 3 copies on each server (2 copies if older RAID technology that allows only one disk to be removed is used).

We did have the master+backup multiple copies in the past. The real news in Jeremy's post is that the new setup avoids the bottlenecks in the old setupthat resulted in slow data recovery while users cannot access their mail.

Edit: now that I've seen the status blog I see it's RAID-5 and not RAID-6. So each of the three servers has an equivalent of at least two copies (but it's not really two copies. It's something more exciting that let's you have a second copy with less than twice the storage space. Sometimes mathematicians produce useful things).

dzh · 31 Oct 2006, 11:49 PM

Quote:

Originally posted by Jeremy Howard
In the future, if there is a file-system corruption (as happened on server3), we will switch to the replica, which will take around 15-30 minutes. It would probably take 15-30 minutes to diagnose the problem as well, so in the end this would result in an outage of 30-60 minutes.

Is there not a faster way of switching? Future outages of 30 - 60 minutes, while preferable to 3 days, would still generate plenty of complaints on these fora! While I am reassured that future outages will be much shorter as a result of these new measures, will they make outages less frequent?

JasonWard · 31 Oct 2006, 11:58 PM

Quote:

Originally posted by dzh
Is there not a faster way of switching? Future outages of 30 - 60 minutes, while preferable to 3 days, would still generate plenty of complaints on these fora! While I am reassured that future outages will be much shorter as a result of these new measures, will they make outages less frequent?

Fastmail taking 15 to 30 minutes to be notified of problem, look into it and decide to do a failover seems pretty quick to me. A failover itself taking abother 15 to 30 minutes seems quite slow to me, and I would imagine that in future Fastmail will be able to improve on this.

As for frequency of outages, well as the system now has more components and is also more complex then it is almost certain that problems that could lead to outages occur more often. But the increase in complexity is aimed at reducing outages.

So I guess we have to suck it and see, but my prediction is that outages will be more common, last a lot less time and effect a lot less users, so from the view of an ordinary user outages will appear to decrease.

Jason

darens · 1 Nov 2006, 12:17 AM

Thanks for the update Jeremy. It's good to know progress is being made.

To me, the occasional ~ 1 hour outage is certainly within the bounds of acceptability.

Si1 · 1 Nov 2006, 01:38 AM

Quote:

Originally posted by Jeremy Howard
We've reached a very exciting point this week - we now have all users on fully-replicated servers.

Jeremy, Rob, Bron and all:

Thanks very much for all your efforts - I really appreciate it. I tried playing around with Google's hosted service at the weekend, but it's just not the same. Fastmail's a brilliant email system and I don't want to switch to anything else.

Cheers

Simon

njlieckfeldt · 1 Nov 2006, 05:11 AM

Too little, too late. Fool me once etc

Sitting out my paid time and then switching.

N.

Jeremy Howard · 1 Nov 2006, 08:09 AM

Quote:

Originally posted by hadaso
I think this is "oversimplifying" things. There are 3 "sets of copies" (master, slave, backup).

Well, 'simplifying', but hopefully not 'oversimplifying'.

You are exactly right on the details, Hadaso - I was trying to get the message across without making it too complicated...

DrStrabismus · 1 Nov 2006, 09:07 AM

The oversimplication was the thread title. No users have actually been replicated.

sflorack · 1 Nov 2006, 10:32 AM

Quote:

Originally posted by DrStrabismus
The oversimplication was the thread title. No users have actually been replicated.

Too bad.. I would have sent my clones into work for me today.

31 Oct 2006, 10:41 AM	#1
Jeremy Howard Ultimate Contributor Join Date: Sep 2001 Location: Australia Posts: 11,501	All users now replicated We've reached a very exciting point this week - we now have all users on fully-replicated servers. This means that, in effect, every email is saved 6 times: on master server, plus RAID redundant copy; on slave server, plus RAID redundant copy; and in backups (also with a RAID redundant copy)! The server3 problem over the weekend showed how non-replicated servers can create substantial outages - this was the last non-replicated server, and now all server3 users have been moved to replicated servers. This would have been completed earlier, but our order of IBM hardware that was required to get this extra space, was delayed by two months (they told us every 2 weeks that it was just about to be delivered!) For those of you who were on server3, I'm really sorry that this happened too late to avoid the enormous frustration that you would have experienced if you tried to access your email during the outage. In the future, if there is a file-system corruption (as happened on server3), we will switch to the replica, which will take around 15-30 minutes. It would probably take 15-30 minutes to diagnose the problem as well, so in the end this would result in an outage of 30-60 minutes.

31 Oct 2006, 11:24 AM	#2
Hlao-roo Member Join Date: Aug 2002 Posts: 83	I wasn't among the affected users, but I was curious about how those who were will be compensated.

31 Oct 2006, 12:49 PM	#3
Sherry Moderator Join Date: Dec 2002 Location: USA Posts: 8,687	Thanks for the info Jeremy. I don't remember ever reading that "every email is saved 6 times". Wow, sounds great. As for the timing if there's a problem. Some time back I was moved from IMAP6 to IMAP5 when there was some problem. It took about the amount of time you mentioned but I think something was said about the "auto" move wasn't in place then. Is that something still coming in the future so the move is even less time? (not that the timing you're saying is any big thing) Sherry

31 Oct 2006, 02:03 PM	#4
robmueller Intergalactic Postmaster Join Date: Oct 2001 Location: Melbourne, Australia Posts: 6,102 Representative of: Fastmail.FM	We're not planning any failovers to be "automatic". I've added to Brons post on the status blog some more details in the update at the bottom. http://status.fastmail.fm/?p=647 Rob

31 Oct 2006, 04:15 PM	#6
NJSS Master of the @ Join Date: Jul 2002 Location: Hampshire, UK Posts: 1,238	Jeremy I, and I anticipate many others, really appreciate seeing you posting here. I hope we see you more frequently in the future, than of late. I appreciate the problems with RSI, but am sure you are working round them. It's really good news that everyone is now on fully-replicated servers - hopefully this will mean fewer long nights for the FM team.

1 Nov 2006, 12:17 AM	#10
darens Member Join Date: Jul 2006 Posts: 46	Thanks for the update Jeremy. It's good to know progress is being made. To me, the occasional ~ 1 hour outage is certainly within the bounds of acceptability.

1 Nov 2006, 05:11 AM	#12
njlieckfeldt Member Join Date: Oct 2004 Posts: 78	Too little, too late. Fool me once etc Sitting out my paid time and then switching. N.

1 Nov 2006, 09:07 AM	#14
DrStrabismus The "e" in e-mail Join Date: May 2002 Posts: 2,804	The oversimplication was the thread title. No users have actually been replicated.