Server reboot

[psalzer]

Originally posted by CML209
I started this new thread because the other has dwindled into techspeak.

mod: Please do not start new threads on the same subject. This is especially true regarding downtimes and similar tech support issues. People need to be able to find the thread with the answers easily and service reps need to be able to post their answers to one thread.

[jkibbe]

Was this a planned outage as mentioned earlier in this thread? If so, it goes back to notification as mentioned during a previous outage. Many of us are on "chatty notifcation," but only get 1 or 2 emails per year. FM should know through previous experience by now that a 30 min fix can easily take over 2 hours. I assume that most of us came to Fastmail (and didn't mind paying for it) because we demand a reliable, feature-rich service. I'm convinced that outages wouldn't be so painful if we knew they were coming (by email, not forum or blog!)...

[Jeremy Howard]

Originally posted by jkibbe
Was this a planned outage as mentioned earlier in this thread?

No. We had to reboot because of a stuck proc. We took the opportunity to switch to the new kernel while we did so. Unfortunately because of the stuck proc, the shutdown did not complete cleanly, requiring the database recovery process.

[mlevin]

Originally posted by anachron
What happened to the old fastmail? The reliable one?

Once this current batch of issues is resolved, PLEASE don't implement any new features until you have some kind of OPT-IN beta server setup whereby interested users can have their accounts migrated to the beta test server.

I could not possibly agree more. I have been saying this for a while. It is always something -- sometimes a quick patch, and invariably it takes longer than expected, causes some weird side effect, something is missed, etc., and each time the good folks at FM (honestly, I do think they are good people) promise, quite earnestly, it seems, that this should never happen again and that procedures are in place to monitor this new thing now, but then something else happens a little while down the line.

FM is great, but only when it is up and running! Quit tinkering with it on the production servers! Trying to make it better by adding 1 new feature and the result being that 0% of the features are available is not really an improvement

[Jeremy Howard]

The outages are unrelated to adding features. They are related to a bug in the Linux kernel, which should now be resolved.

[mlevin]

Originally posted by kander
it sounds to me like this database server is a very Single Point Of Failure.

Isn't the cornerstone of any good system architecture having no single point of failure?

We have redundant network connections, redundant power supplies, backup smtp servers, blah blah blah, and a SINGLE database? Is the DB usually FM's downfall?

Maybe it is time to invest in a database that is capable of clustering or replication (I don't think MySQL can do that, can it?).

[mlevin]

Originally posted by Jeremy Howard
No. We had to reboot because of a stuck proc. We took the opportunity to switch to the new kernel while we did so. Unfortunately because of the stuck proc, the shutdown did not complete cleanly, requiring the database recovery process.

In the future, I would like to ask that if you are just rebooting to fix problem A, please don't try to slip in fix B "while you are at it" -- obviously, this can have unintended consequences. Please fix issue B during the PLANNED outage (and please INFORM people set to "chatty" via EMAIL, not the blog). I'd much rather have had a quick reboot, even if it left the potential for another stuck process, and then a PLANNED outage for the thing you tried to slip in while you were at it. That way I could have PLANNED for the potentially longer outage.

Just my $.02

Thanks

[mlevin]

Originally posted by Jeremy Howard
The outages are unrelated to adding features. They are related to a bug in the Linux kernel, which should now be resolved.

My apologies. I still think, as I just mentioned in another post, that patches should be done as planned outages, obviously scheduled as quickly as possible but with a little warning, even if it means a few quick reboots as a temporary measure before you can get to the patch.

Why are so many patches required so often? Isn't linux pretty stable if you just leave it alone?

[rakhesh]

Originally posted by oysterquartz
You didn't address the question as to why File Space should be compromised, and ones virtual domain rendered useless for email and web/file access.

IMO this is an inherent flaw in architecture.

I think this is coz the database is what contains info on where your files are stored (I assume there are multiple servers, the same way as in mail, and so those details are with the database) ... and that's why file storage was compromised during the outage. Just what I think eh ... have no real idea to make a knowledgeable statement.

Like some pointed above ... the problem is the SINGLE database then! Multiple everything, but a single database. Not too great eh!

[rakhesh]

Originally posted by rakhesh
Hmm, I see. So when the database is down, the smtp.us server doesn't know where to deliver the mail to etc, and so it doesn't accept mail ... But the smtp.eu server doesn't need to know all these details -- it simply accepts whatever mails are given to it (its a store and forward) -- coz all the actual mail filing is being done by smtp.us anyways. Right?

Where does smtp.us2 fit into this picture? If the database is down, does that too not accept mails? (I know the answer is yes, coz if it had been accepting mails, then they would have gone to bitequator's backup servers ... still, thought I'd confirm).

Just thought I'd bump this post again so that Jeremy can answer. I already know the answer (based on what I've seen), but I'd like a confirmation nevertheless.

[Starion]

Originally posted by zaurus
No, it's not lucky at all for asian users. Here is 2 p.m.
The smtp, and pop die too, so I cannot send any mail.
Anyway, try your best , FM.

Hi Zaurus, I'm sorry that you aren't able to access your account. Fastmail seems to be ok now. I didn't need to access my mail at 0000 local time or 4 GMT. Take care.

[severach]

Originally posted by mlevin
Maybe it is time to invest in a database that is capable of clustering or replication (I don't think MySQL can do that, can it?).

Anthing done with transactions can be duplicated to 2 or more, and all those servers can be used live. Any server that goes down simply finds out what transactions were missed. Mistakes can be caught by servers comparing notes.

[oysterquartz]

The single database was my point. Why wouldn't you split databases and have customer data in one (additionally for increased security of customer's personal data) and handle the file storage/website side of the equation so that a single point of failure won't compromise everything.

Email down for several hours is bad enough, but having ones website down at the same time is even worse.

I seem to remember some commercial webmail applications advising this approach ...

[severach]

The portion of the web that is obtained from the Files menu requires your customer information to know what to display. If there is some non personalized web or business hosting then that may well be integrated even though it may not be necessary.

At the very least, send the transactions to Texas and have a functioning mail reading system should that be the only remaining working server. I think that server outages due to NYI or FM would be much less severe if we could be assured of handing new mail. Those needing high reliability mail would only need to ensure they have an easy to get offline copy rather than go to other services.

[Jeremy Howard]

Originally posted by rakhesh
Just thought I'd bump this post again so that Jeremy can answer. I already know the answer (based on what I've seen), but I'd like a confirmation nevertheless.

smtp.eu just queues mail, whereas smtp.us2 attempts to deliver them. So smtp.us2 relies on the DB.