New servers ordered

robmueller · 5th December 2005, 03:59 PM

Just an FYI, last week I ordered our new servers for our replicated setup. Basically they're a fairly custom configuration to trade off our high IO requirements while maintaining a reasonable price point. They've said that they should ship around mid-December, and we'll need a few weeks to test and setup out our end, so hopefully they'll be online late December or early next year. We'll obviously let people know...

With these new servers we'll have:

Intraserver redundancy (within each server)

* Two small (4 drive) RAID-5 arrays within each each server for email data
* Two RAID-1 arrays within each server for meta data

So if any one drive in a server, or even several drives in separate arrays fails, the server will be fine

Interserver redundancy (between each server)

* The servers will run in pairs, replicating email and meta data between pairs. You can setup the replication "tick" interval, and it seems 1/10th of a second is quite feasible, so no replica is more than 1/10th of a second behind the master

So if any one server fails, the replica will be able to take over as the new master. Once we've fixed the server up, we can then swap it back as the new slave, and it will catch up to the new master again, or we can just bring online a whole new spare server that we'll have.

Emergency redundancy

We'll still be doing our nightly incremental backups to ensure even if the servers die somehow, we have the data. Also, we keep the backup data for 1 week, so we can retrieve any accidentally deleted data for up to 7 days after the deletion event.

About the only thing missing would be inter-data center redundancy, but unfortunately this just is not feasible at this point with our current scale... maybe one day

As it is, we're pretty happy that NYI have dealt with their issues post the NYC power outage and so we have power and network redundancy "intranyi", if not "internyi-someoneelse" yet...

Rob

edit: add "not" in there...

sjk · 5th December 2005, 05:13 PM

Thanks for the news and details, Rob.

luminosity · 6th December 2005, 04:17 AM

People in these forums seem pretty uptight about the recent outage. I admit being a bit ticked off, but I got nowhere near the insanity level of others on this forum. Nothing is perfect, not even the electricity powering our home computers. For a single service to go down like this is not even as bad as a supermarket's employees going on strike. WE JUST HAVE TO MAKE DO. Eventualy things turn around. I stuck it out and have upgraded my account to Full from Member. My dad was born in Eastern Europe right at the end of World War II. All of the food on the island was burned by the Nazis and the local population had to figure out a way to survive until foreign aid arrived. For the first few years of his life food was scarce and yet as a baby he survived and lived through today and is now in this 60's. If we can't survive a weekend email outage, then we are surely hopeless!! Past generations survived much more hardship!

I think effert deserves more attention that error. New effort in the infrastructure is very nice to see and even more reason to stay as a customer of FastMail.

sflorack · 6th December 2005, 01:54 PM

I'm by no means siding with those who rant and rave about outages, but I have a hard time drawing a correlation between the struggles of people living in Nazi Germany and a paid email service.

Paying users have more reason to complain if their provider doesn't live up to their promises, than those being forced to starve by a deranged leader. The fact is, Fastmail is a business, and as unfair as it may be, will probably lose customers due to an occasional outage from those who have come to expect more.

luminosity · 6th December 2005, 02:24 PM

I wasn't talking about Nazi Germany, but another island occupied by Nazis. My statement is that we are too spoiled. We need to learn to live with these little bumps in our lives, after all other generations have experienced much bigger bumps in their lives and they lived on. Thats what I meant.

luminosity · 6th December 2005, 02:27 PM

Fastmail has lived up to their promises. In their Terms of Service they clearly state that there is no guarantee for the service. Nowhere do they state that their systems cannot go down.

beq · 6th December 2005, 02:54 PM

Quote:

Originally posted by robmueller
About the only thing missing would be inter-data center redundancy, but unfortunately this just is NOT feasible at this point with our current scale...

Methinks you forgot the word "NOT"

Anyways, the real thing missing is Rob living in the data center

William9 · 6th December 2005, 03:41 PM

Quote:

Originally posted by sflorack
I'm by no means siding with those who rant and rave about outages, but I have a hard time drawing a correlation between the struggles of people living in Nazi Germany and a paid email service.

Paying users have more reason to complain if their provider doesn't live up to their promises, than those being forced to starve by a deranged leader. The fact is, Fastmail is a business, and as unfair as it may be, will probably lose customers due to an occasional outage from those who have come to expect more.

Ditto.
I add that it's not due to an occasional outage -- It is one extraordinarily long outage. Or for this or any other provider, it could be multiple outages that add up to poor performance compared to its competitors.
Yes, some people have come to expect a lot from email service providers.
I'm really pleased to hear about the purchase of new equipment. That is very reassuring.

eggman · 6th December 2005, 05:39 PM

great to see the new servers are coming online - i still trust fastmail with my email.

I say this because the same event doesn't cause failure twice in Fastmail - its normally something new. This is because the team look at the point of failure and then come up with a sensible solution for the problem to prevent it occuring again.

Keep up the good work.

sflorack · 6th December 2005, 10:30 PM

Quote:

Originally posted by luminosity
Fastmail has lived up to their promises. In their Terms of Service they clearly state that there is no guarantee for the service. Nowhere do they state that their systems cannot go down.

Terms of Services are legal jargon to prevent lawsuits. The cold, hard truth is that if people feel that they are wasting their money, they will not continue to pay. When I said "promises", I meant in the eyes of the user (perceived), not in actual writing (truth).

William9 · 6th December 2005, 11:51 PM

Quote:

Originally posted by sflorack
Terms of Services are legal jargon to prevent lawsuits. The cold, hard truth is that if people feel that they are wasting their money, they will not continue to pay. When I said "promises", I meant in the eyes of the user (perceived), not in actual writing (truth).

I agree: if their TOS Disclaimer is a promise of level of service, that would be absurd. It's all about customer expectations and competition. What is out there, how much does it cost, and how well do they perform (ie. do they satisfy their customers and compare well or do better than the competition)?

elvey · 10th January 2006, 03:11 AM

Any news on testing/implementation/rolllout?

Rolling out these kinds of redundant systems is in itself a high risk endeavour prone to CAUSING outages. I hope you're proceeding slowly.

http://en.wikipedia.org/wiki/Fault_t...#Disadvantages and all that...

BritTim · 12th January 2006, 01:17 AM

Quote:

Rolling out these kinds of redundant systems is in itself a high risk endeavour

Absolutely! If they can really do realtime replication with complete safety, I am all for it. That said, I would actually be less nervous with a system based on offline replication using update logs. Such an approach allows the existing system to proceed essentially without change, while providing an additional mechanism for rapid recovery should the need arise.

robmueller · 19th January 2006, 03:10 PM

The servers have arrived, and Brons being doing setup work and testing on them. Basically we've done some soft testing first by getting them to run as another server role for a while (eg mx server).

Unfortunately Bron's taking a 2 week break, so nothing is going to happen with these till he gets back.

Roll out will definitely be taken with care. Our usual testing procedure with new servers is to move our accounts over to them for a few weeks, so if anything goes wrong, we suffer, not customers.

We'll also be testing things like hard reboots, pulled power cords, etc to check that there's no unexpected corruption problems.

Rob

5th December 2005, 03:59 PM	#1
robmueller Intergalactic Postmaster Join Date: Oct 2001 Location: Melbourne, Australia Posts: 6,098 Representative of: Fastmail.FM	New servers ordered Just an FYI, last week I ordered our new servers for our replicated setup. Basically they're a fairly custom configuration to trade off our high IO requirements while maintaining a reasonable price point. They've said that they should ship around mid-December, and we'll need a few weeks to test and setup out our end, so hopefully they'll be online late December or early next year. We'll obviously let people know... With these new servers we'll have: Intraserver redundancy (within each server) * Two small (4 drive) RAID-5 arrays within each each server for email data * Two RAID-1 arrays within each server for meta data So if any one drive in a server, or even several drives in separate arrays fails, the server will be fine Interserver redundancy (between each server) * The servers will run in pairs, replicating email and meta data between pairs. You can setup the replication "tick" interval, and it seems 1/10th of a second is quite feasible, so no replica is more than 1/10th of a second behind the master So if any one server fails, the replica will be able to take over as the new master. Once we've fixed the server up, we can then swap it back as the new slave, and it will catch up to the new master again, or we can just bring online a whole new spare server that we'll have. Emergency redundancy We'll still be doing our nightly incremental backups to ensure even if the servers die somehow, we have the data. Also, we keep the backup data for 1 week, so we can retrieve any accidentally deleted data for up to 7 days after the deletion event. About the only thing missing would be inter-data center redundancy, but unfortunately this just is not feasible at this point with our current scale... maybe one day As it is, we're pretty happy that NYI have dealt with their issues post the NYC power outage and so we have power and network redundancy "intranyi", if not "internyi-someoneelse" yet... Rob edit: add "not" in there... Last edited by robmueller : 6th December 2005 at 07:26 PM.

5th December 2005, 05:13 PM	#2
sjk Master of the @ Join Date: May 2002 Location: Hawaii Posts: 1,975	Thanks for the news and details, Rob.

6th December 2005, 04:17 AM	#3
luminosity Member Join Date: Jan 2005 Posts: 82	People in these forums seem pretty uptight about the recent outage. I admit being a bit ticked off, but I got nowhere near the insanity level of others on this forum. Nothing is perfect, not even the electricity powering our home computers. For a single service to go down like this is not even as bad as a supermarket's employees going on strike. WE JUST HAVE TO MAKE DO. Eventualy things turn around. I stuck it out and have upgraded my account to Full from Member. My dad was born in Eastern Europe right at the end of World War II. All of the food on the island was burned by the Nazis and the local population had to figure out a way to survive until foreign aid arrived. For the first few years of his life food was scarce and yet as a baby he survived and lived through today and is now in this 60's. If we can't survive a weekend email outage, then we are surely hopeless!! Past generations survived much more hardship! I think effert deserves more attention that error. New effort in the infrastructure is very nice to see and even more reason to stay as a customer of FastMail.

6th December 2005, 01:54 PM	#4
sflorack The "e" in e-mail Join Date: Feb 2002 Posts: 2,644	I'm by no means siding with those who rant and rave about outages, but I have a hard time drawing a correlation between the struggles of people living in Nazi Germany and a paid email service. Paying users have more reason to complain if their provider doesn't live up to their promises, than those being forced to starve by a deranged leader. The fact is, Fastmail is a business, and as unfair as it may be, will probably lose customers due to an occasional outage from those who have come to expect more.

6th December 2005, 02:24 PM	#5
luminosity Member Join Date: Jan 2005 Posts: 82	I wasn't talking about Nazi Germany, but another island occupied by Nazis. My statement is that we are too spoiled. We need to learn to live with these little bumps in our lives, after all other generations have experienced much bigger bumps in their lives and they lived on. Thats what I meant.

6th December 2005, 02:27 PM	#6
luminosity Member Join Date: Jan 2005 Posts: 82	Fastmail has lived up to their promises. In their Terms of Service they clearly state that there is no guarantee for the service. Nowhere do they state that their systems cannot go down.

6th December 2005, 05:39 PM	#9
eggman Essential Contributor Join Date: Jun 2002 Location: AU Posts: 468	great to see the new servers are coming online - i still trust fastmail with my email. I say this because the same event doesn't cause failure twice in Fastmail - its normally something new. This is because the team look at the point of failure and then come up with a sensible solution for the problem to prevent it occuring again. Keep up the good work.

10th January 2006, 03:11 AM	#12
elvey The "e" in e-mail Join Date: Jan 2002 Location: San Francisco Posts: 2,328	Any news on testing/implementation/rolllout? Rolling out these kinds of redundant systems is in itself a high risk endeavour prone to CAUSING outages. I hope you're proceeding slowly. http://en.wikipedia.org/wiki/Fault_t...#Disadvantages and all that...

19th January 2006, 03:10 PM	#14
robmueller Intergalactic Postmaster Join Date: Oct 2001 Location: Melbourne, Australia Posts: 6,098 Representative of: Fastmail.FM	The servers have arrived, and Brons being doing setup work and testing on them. Basically we've done some soft testing first by getting them to run as another server role for a while (eg mx server). Unfortunately Bron's taking a 2 week break, so nothing is going to happen with these till he gets back. Roll out will definitely be taken with care. Our usual testing procedure with new servers is to move our accounts over to them for a few weeks, so if anything goes wrong, we suffer, not customers. We'll also be testing things like hard reboots, pulled power cords, etc to check that there's no unexpected corruption problems. Rob