Scheduled maintenance 13.06.2007

Geir · 12 Jun 2007, 04:17 AM

We will have approximately 1 hour downtime Wednesday 13.06.07 starting around 10 AM CET (4 AM EST, 1 AM PST) to complete the upgrade of the main database server.

We apologize for any inconvenience caused.

By the way, we are planning to establish failover redundancy for the database server so we can avoid similar downtime in the future.

- Geir

goofy · 13 Jun 2007, 03:49 AM

Geir,

Curious, why do you perform server updates during CET daytime?

carverrn · 13 Jun 2007, 05:28 AM

Because that's when the service people are available to do it.

Regards,
Rich

Rizlam · 13 Jun 2007, 05:36 PM

Ouch. This has come at a really bad time for me. I hope it is just an hour, otherwise I'm in serious trouble this morning.

Geir · 13 Jun 2007, 05:44 PM

Rizlam: If you're expecting a specific message I might be able to find it in the queue, provided the sender and recipient addresses.

- Geir

Rizlam · 13 Jun 2007, 06:19 PM

It will be anything with BSAVA in the subject.

Do I gather it's going to be well over an hour then

Geir · 13 Jun 2007, 06:26 PM

There doesn't appear to be any such message in the queue at the moment.

The reason for the extended downtime is likely to be a bug in the Linux kernel on the main database server. We're working intensively to have it corrected and bring the system back up.

- Geir

GarethP · 13 Jun 2007, 06:40 PM

Quote:

Originally Posted by Geir

The reason for the extended downtime is likely to be a bug in the Linux kernel on the main database server. We're working intensively to have it corrected and bring the system back up.

There's nothing worse than being told an outage will take 1 hour and then, after having had your business crippled because of no email, you find out after 1hr 26min minutes that its still down and no ETA has been mentioned.

I'm going to be recommending to senior management in my organisation that we move away from runbox as you are quite clearly costing us money with these outages, not to mention the "extended downtime" after the planned outages.

Any chance of knowing when you expect your "service" to be back up?

Rizlam · 13 Jun 2007, 06:46 PM

I'm not as annoyed as Gareth, but I do challenge Runbox to point to a time when they've given us an estimate for the duration of a planned outage and it has turned out to be accurate... sometimes it's 1, 2, even 3 or 4 hours longer than they say. It's just not good enough.

petrs · 13 Jun 2007, 06:49 PM

agree w/ rizlam - it's second time that the downtime is 'taking longer than expected'.

arguably, most of the people will get p****d off not by the downtime but by not keeping up w/ the comittments

Geir · 13 Jun 2007, 06:51 PM

It's exceedingly difficult to provide an estimation for recovery when the root cause of the problem is an error in the operating system, which is beyond our control.

This problem now appears to be resolved, however, and we should be running again in just a few minutes.

- Geir

petrs · 13 Jun 2007, 06:51 PM

release management - rings a bell?

Geir · 13 Jun 2007, 06:53 PM

Operations restored.

- Geir

Geir · 13 Jun 2007, 07:01 PM

The incoming email that accumulated while the service was inaccessible is now being processed and the remainder should be delivered within a few minutes.

- Geir

GarethP · 13 Jun 2007, 07:02 PM

Quote:

Originally Posted by Geir

Operations restored.

Thank you.

Are there any future scheduled downtimes that you are currently aware of which you could pass on to us now?

12 Jun 2007, 04:17 AM	#1
Geir The "e" in e-mail Join Date: Sep 2001 Location: Oslo, Norway Posts: 2,938 Representative of: Runbox.com	Scheduled maintenance 13.06.2007 We will have approximately 1 hour downtime Wednesday 13.06.07 starting around 10 AM CET (4 AM EST, 1 AM PST) to complete the upgrade of the main database server. We apologize for any inconvenience caused. By the way, we are planning to establish failover redundancy for the database server so we can avoid similar downtime in the future. - Geir

13 Jun 2007, 03:49 AM	#2
goofy Essential Contributor Join Date: Apr 2002 Posts: 226	Geir, Curious, why do you perform server updates during CET daytime?

13 Jun 2007, 05:28 AM	#3
carverrn Intergalactic Postmaster Join Date: Jan 2002 Location: Chicago, IL Posts: 5,606 Representative of: Runbox.com	Because that's when the service people are available to do it. Regards, Rich

13 Jun 2007, 05:36 PM	#4
Rizlam Member Join Date: Oct 2003 Posts: 48	Ouch. This has come at a really bad time for me. I hope it is just an hour, otherwise I'm in serious trouble this morning.

13 Jun 2007, 05:44 PM	#5
Geir The "e" in e-mail Join Date: Sep 2001 Location: Oslo, Norway Posts: 2,938 Representative of: Runbox.com	Rizlam: If you're expecting a specific message I might be able to find it in the queue, provided the sender and recipient addresses. - Geir

13 Jun 2007, 06:19 PM	#6
Rizlam Member Join Date: Oct 2003 Posts: 48	It will be anything with BSAVA in the subject. Do I gather it's going to be well over an hour then

13 Jun 2007, 06:26 PM	#7
Geir The "e" in e-mail Join Date: Sep 2001 Location: Oslo, Norway Posts: 2,938 Representative of: Runbox.com	There doesn't appear to be any such message in the queue at the moment. The reason for the extended downtime is likely to be a bug in the Linux kernel on the main database server. We're working intensively to have it corrected and bring the system back up. - Geir

13 Jun 2007, 06:46 PM	#9
Rizlam Member Join Date: Oct 2003 Posts: 48	I'm not as annoyed as Gareth, but I do challenge Runbox to point to a time when they've given us an estimate for the duration of a planned outage and it has turned out to be accurate... sometimes it's 1, 2, even 3 or 4 hours longer than they say. It's just not good enough.

13 Jun 2007, 06:49 PM	#10
petrs Senior Member Join Date: Dec 2003 Posts: 134	agree w/ rizlam - it's second time that the downtime is 'taking longer than expected'. arguably, most of the people will get p****d off not by the downtime but by not keeping up w/ the comittments

13 Jun 2007, 06:51 PM	#11
Geir The "e" in e-mail Join Date: Sep 2001 Location: Oslo, Norway Posts: 2,938 Representative of: Runbox.com	It's exceedingly difficult to provide an estimation for recovery when the root cause of the problem is an error in the operating system, which is beyond our control. This problem now appears to be resolved, however, and we should be running again in just a few minutes. - Geir

13 Jun 2007, 06:53 PM	#13
Geir The "e" in e-mail Join Date: Sep 2001 Location: Oslo, Norway Posts: 2,938 Representative of: Runbox.com	Operations restored. - Geir

13 Jun 2007, 07:01 PM	#14
Geir The "e" in e-mail Join Date: Sep 2001 Location: Oslo, Norway Posts: 2,938 Representative of: Runbox.com	The incoming email that accumulated while the service was inaccessible is now being processed and the remainder should be delivered within a few minutes. - Geir