Crashing servers

brong · 1st July 2012, 01:37 AM

We were hit by the leap second bugs in the Linux Kernel. I've written up my experience on our technical blog.

http://blog.fastmail.fm/2012/07/03/a...aping-seconds/

Original post here below:

We're having a really bad day of crashing spam scanning servers. Mostly users won't see any issues, apart from slight delays (5-10 minutes) on incoming emails when one of the servers crashes.

Interestingly, it seems to not just be us - other departments in Opera are having failures too. At first we suspected the specific hardware, but we've had crashes on other equipment too. It might be a rough day.

On the plus side, our robust failover systems are working fine

The process for dealing with a failed server is now so easy I can do it from my phone in ~30sec.

brong · 1st July 2012, 06:38 AM

Getting some press!

http://serverfault.com/q/403732/8453
http://news.ycombinator.com/item?id=4182642

And I think we're all fixed now

n5bb · 1st July 2012, 11:48 AM

Bron, Amazon.com has been getting bad press today due to the combination of record high temperatures and high winds taking out utilities in the eastern US. Maybe their attempts at redirecting traffic were thwarted by server bugs.

Did this indeed turn out to be a time rollover bug?

Bill

brong · 4th July 2012, 07:02 AM

Yes. it did.

I've updated the original post of this thread to link to the technical blog, and pointed the technical blog here. Nice and cirular!

rabarberski · 6th July 2012, 06:04 PM

I liked the long blog post you wrote about. Very entertaining...
(and thanks for handling this issue so well)

janusz · 6th July 2012, 06:40 PM

Quote:

Originally Posted by brong

Nice and cirular!

Nice and what??

rabarberski · 6th July 2012, 06:54 PM

Quote:

Originally Posted by janusz

Nice and what??

Circular. The missing c was probably typed during a leap second and got lost...

brong · 6th July 2012, 09:28 PM

I'm relaxing by the C right now...

n5bb · 6th July 2012, 09:49 PM

Bron must have been listening to the crashing servers:

Broken Cirular Logic

Bill

1st July 2012, 01:37 AM	#1
brong The "e" in e-mail Join Date: Jul 2004 Location: Oslo, Norway Posts: 2,380 Representative of: Fastmail.fm	Crashing servers We were hit by the leap second bugs in the Linux Kernel. I've written up my experience on our technical blog. http://blog.fastmail.fm/2012/07/03/a...aping-seconds/ Original post here below: We're having a really bad day of crashing spam scanning servers. Mostly users won't see any issues, apart from slight delays (5-10 minutes) on incoming emails when one of the servers crashes. Interestingly, it seems to not just be us - other departments in Opera are having failures too. At first we suspected the specific hardware, but we've had crashes on other equipment too. It might be a rough day. On the plus side, our robust failover systems are working fine The process for dealing with a failed server is now so easy I can do it from my phone in ~30sec. Last edited by brong : 4th July 2012 at 07:04 AM. Reason: Add blog entry

4th July 2012, 07:02 AM	#4
brong The "e" in e-mail Join Date: Jul 2004 Location: Oslo, Norway Posts: 2,380 Representative of: Fastmail.fm	Yes. it did. I've updated the original post of this thread to link to the technical blog, and pointed the technical blog here. Nice and cirular! Last edited by brong : 4th July 2012 at 07:08 AM.

1st July 2012, 06:38 AM	#2
brong The "e" in e-mail Join Date: Jul 2004 Location: Oslo, Norway Posts: 2,380 Representative of: Fastmail.fm	Getting some press! http://serverfault.com/q/403732/8453 http://news.ycombinator.com/item?id=4182642 And I think we're all fixed now

1st July 2012, 11:48 AM	#3
n5bb Intergalactic Postmaster Join Date: May 2004 Location: Irving, Texas Posts: 5,978	Bron, Amazon.com has been getting bad press today due to the combination of record high temperatures and high winds taking out utilities in the eastern US. Maybe their attempts at redirecting traffic were thwarted by server bugs. Did this indeed turn out to be a time rollover bug? Bill

6th July 2012, 06:04 PM	#5
rabarberski Cornerstone of the Community Join Date: Nov 2006 Location: Ghent, Belgium Posts: 935	I liked the long blog post you wrote about. Very entertaining... (and thanks for handling this issue so well)

6th July 2012, 09:28 PM	#8
brong The "e" in e-mail Join Date: Jul 2004 Location: Oslo, Norway Posts: 2,380 Representative of: Fastmail.fm	I'm relaxing by the C right now...

6th July 2012, 09:49 PM	#9
n5bb Intergalactic Postmaster Join Date: May 2004 Location: Irving, Texas Posts: 5,978	Bron must have been listening to the crashing servers: Broken Cirular Logic Bill