Greylisting of incoming email

robmueller · 21st July 2006, 05:26 PM

I've implemented a new policy server on all incoming email. This policy server currently only implements two main policies:

1. It counts the number of attempts to deliver to unique invalid recipients at FastMail from a particular IP and from a particular IP/24 subnet. If it reaches more than a certain threshold, then more attempts from that IP or subnet receive a response of:

"453 Too many recent unknown recipients"

The threshold depends on the type of IP (dialup, no reverse DNS, definte MTA) and decays reasonably quickly over time. I won't give the exact details, but so far this has worked very well at quickly and automatically blocking attempts to enumerate email account addresses

2. We now greylist attempts to deliver from any host that is a known dialup IP. In these cases, we return the response string:

453 Temporary deferral, try again soon

I explicitly don't use "greylist" in the return string, in case spam zombies start getting more clever and start looking for that string in the response. The delay time for successful resending varies slightly randomly depending on certain factors about the IP.

If a particular IP sucessfully passes greylisting twice in 24 hours (a "pass" means that the same ip/sender/recipient combination is seen from the same /24 subnet after the greylisting delay time has passed), then that IP is whitelisted for the next 24 hours at least, and if it continues to pass greylisting, it will continue to be whitelisted.

From my analysis of the logs so far, this has been extremely sucessful in stopping a large number of zombie machines, and the conservative nature of what we're greylisting (dialups only) and the auto-whitelisting policy means that email hosts on dialup connections are very quickly effectively permanently whitelisted and there is no further delay on their email.

Because of this, there currently is no option to turn of greylisting, and I'm not planning to add one. I think most people who would turn it off would be because of experience with bad greylisting systems (eg I just got an email from a user because their domain forwarder recently implemented greylisting, but are only whitelisting hosts by support request, not automatically, so all email from FM is currently delayed until our servers are listed).

I believe the current conservative policy used has lots of upside with no downside that I can see.

Anyway, greylisting isn't meant to be a full spam protection method, it's just another adjunct. So now we have the follow stages of protection:

1. RBLs (dsbl/xbl)
2. greylisting
3. spamassassin
4. backscatter detection

And probably the main last piece to come will be the per-user bayes db...

Rob

JasonWard · 21st July 2006, 06:28 PM

Thanks Rob,

If you implemented this in the last 2 to 3 days I would say its made a noticible impact for me,as the number of SPAM seems to have halved in that time.

Jason

paleolith · 22nd July 2006, 06:06 AM

Well, I got up this morning and went to do my daily cleanout of my junk mailbox. Hmm, only 29 junked emails!! Something's gotta be wrong!!!! A normal day is 100-150.

Finally came here and found this. So from my first-day experience, it's looking like a 2/3 to 3/4 decrease for me. Extremely impressive, even considering that my numbers will be better than most because I have an alias on *@paleo.org (in other words, I choose to receive all email addressed to my domain even when it's not a normally valid user name).

Quote:

I think most people who would turn it off would be because of experience with bad greylisting systems

For the past several years, I've tried to collect all the spam I get, mostly for statistical purposes. However, I've already opted to eliminate certain categories -- for example, I enabled FM's backscatter detection. And I have no desire for an option related to this category -- I am very happy to let FM block it. I find Robs's description convincing, and I agree that this is a very safe measure.

I would find it of some use if Rob published figures about the number of connections blocked. The fight against spam is far from over; measures such as these just emphasize the critical nature of the moat we've built to protect us and the fragility of our defenses. We still need to monitor the enemy's strength.

Edward

DrStrabismus · 23rd July 2006, 02:05 AM

Quote:

Originally posted by robmueller

2. We now greylist attempts to deliver from any host that is a known dialup IP.

I'd be inclined to go a bit further, and greylist any invalid EHLO hostnames, or hostnames where the DNS doesn't lead back to the last-hop IP address. Having validated the hostname you could then strip-off the domain part, and look for encoded IP addresses, long runs of digits, or strings like pool, dynamic, dsl, modem, cable etc.

There seems to be a fair amount of spam that doesn't hit any SA dial-up lists, but is clearly coming through DSL/Cable providers.

David · 23rd July 2006, 03:21 AM

Quote:

Originally posted by robmueller

2. We now greylist attempts to deliver from any host that is a known dialup IP. In these cases, we return the response string:

I admit to not understanding all of the implications here, but I do know that many (legitimate) mailservers in poorer nations use dialup as a method of getting a maiiserver online (and into the internet) - Often they will go online and deliver (and receive) mail and then hang up the connection. Are all these mailservers now automatically suspect? or am I misunderstanding exactly what is happening here?

It would be unfortunate if those who are the least able to buy connectivity are punished.

paleolith · 23rd July 2006, 04:27 AM

David,

First, Rob didn't say how FM is identifying "known dialups". My guess would be that the identification comes from specific lists, and that such legitimate mail servers are not in those lists. (If they are in those lists, then they are already being punished in much more severe ways.)

Second, in the algorithm Rob described, the email from a greylisted server will be temporarily rejected on the first try. When the server retries, on its next dial-out, it will succeed. All mail servers include this retry logic, which is a basic and necessary part of mail transfer. Once it succeeds twice, it will be able to send without restriction. If a single mail server is sending to FM customers regularly, it will essentially never be slowed. If it only sends occasionally, it will be slowed by the time needed to retry.

The key here is that spambot zombies don't retry, and thus will pass to blacklisted rather then to whitelisted. Waiting to see if the mail server retries is an excellent discriminator between valid mail servers and spam zombies.

Of course, those dial-up mail servers are also probably among the ones most hurt by spam. FM is clearly working diligently to avoid collateral damage in the war on spam, and any dent in the spam scourge will help those dial-up mail servers.

Edward

DrStrabismus · 23rd July 2006, 05:03 AM

Quote:

Originally posted by David
I admit to not understanding all of the implications here, but I do know that many (legitimate) mailservers in poorer nations use dialup as a method of getting a maiiserver online (and into the internet) - Often they will go online and deliver (and receive) mail and then hang up the connection. Are all these mailservers now automatically suspect? or am I misunderstanding exactly what is happening here?

It would be unfortunate if those who are the least able to buy connectivity are punished.

Firstly the penalty isn't that great, AOL and many other services, simply rejects such email out-of-hand.

Secondly, these days the prefered way to setup such a server is to have it relay through an external mail server. So there would be no problem if the dial-up ISP provides smtp access.

David · 23rd July 2006, 11:13 AM

Thanks DrS and paleolith for the explanation.

msulloway · 26th July 2006, 10:11 PM

This seems to have stopped all the dictionary attacks to my member account. Thanks.

paleolith · 27th July 2006, 07:27 AM

Quote:

1. It counts the number of attempts to deliver to unique invalid recipients at FastMail from a particular IP and from a particular IP/24 subnet. If it reaches more than a certain threshold, then more attempts from that IP or subnet receive a response of:

Hmm. I was thinking, Rob. Do you just count total number of invalid recipients? Or do you track a ratio of valid to invalid? If it's the latter, you should exclude domains (such as mine) which have an * alias, since that makes all recipient names in that domain valid from your point of view.

(I understand your reasons for not giving details, so I don't expect to to actually answer my question.)

Edward

Ruhe · 27th July 2006, 05:57 PM

What is "backscatter detection"?

JasonWard · 27th July 2006, 06:01 PM

Quote:

Originally posted by Ruhe
What is "backscatter detection"?

When a spammer is using your email address as the "sender" you get whats called backscatter, which is lots and lots of "Unknown user", "Mail not deliverable", "I'm on holiday", etc and occasionally abuse from people who think the SPAM really did come from you.

The detection part just means that Fastmail can tell (mostly) what is genuine messages you need to see and what is garbage you can discard.

Jason

Ruhe · 27th July 2006, 07:26 PM

Thanks Jason.

And this can be en-/disabled in FM.

JasonWard · 27th July 2006, 08:10 PM

Hmmm, I guess, I use Sieve scripting and currently ignore the backscatter stuff (so in effect it's turned off for me).

As I never see the rules screen, and I don't ever recall having seen it I am far from a good guide as to how you configure it when not using Sieve.

Jason

BobSchneider · 27th July 2006, 09:16 PM

I am not happy with your greylisting because there is a chance that it could result in mail that I need to receive being rejected....and I can't turn it off. The chance may be very very small I admit, but there is no way of telling what impact a rejected e-mail might have on me or my business.

I believe that if an e-mail is addressed to me it should be my choice whether or not it passses through filters that could cause non-delivery. I also believe that if mail gets filtered into non-delivery, I should have an option to be informed about its rejection so that I can take remedial action if necessary (eg contact the sender to inform them that they have a virus). It is not acceptable to me for mail addressed to me to be silently rejected by a process which I cannot avoid.

I can see why you have taken these steps, but in my book you have changed from a reliable e-mail service to an unreliable one, purely because you have taken steps that could lose e-mail addressed to me.

Bob

21st July 2006, 05:26 PM	#1
robmueller Intergalactic Postmaster Join Date: Oct 2001 Location: Melbourne, Australia Posts: 6,098 Representative of: Fastmail.FM	Greylisting of incoming email I've implemented a new policy server on all incoming email. This policy server currently only implements two main policies: 1. It counts the number of attempts to deliver to unique invalid recipients at FastMail from a particular IP and from a particular IP/24 subnet. If it reaches more than a certain threshold, then more attempts from that IP or subnet receive a response of: "453 Too many recent unknown recipients" The threshold depends on the type of IP (dialup, no reverse DNS, definte MTA) and decays reasonably quickly over time. I won't give the exact details, but so far this has worked very well at quickly and automatically blocking attempts to enumerate email account addresses 2. We now greylist attempts to deliver from any host that is a known dialup IP. In these cases, we return the response string: 453 Temporary deferral, try again soon I explicitly don't use "greylist" in the return string, in case spam zombies start getting more clever and start looking for that string in the response. The delay time for successful resending varies slightly randomly depending on certain factors about the IP. If a particular IP sucessfully passes greylisting twice in 24 hours (a "pass" means that the same ip/sender/recipient combination is seen from the same /24 subnet after the greylisting delay time has passed), then that IP is whitelisted for the next 24 hours at least, and if it continues to pass greylisting, it will continue to be whitelisted. From my analysis of the logs so far, this has been extremely sucessful in stopping a large number of zombie machines, and the conservative nature of what we're greylisting (dialups only) and the auto-whitelisting policy means that email hosts on dialup connections are very quickly effectively permanently whitelisted and there is no further delay on their email. Because of this, there currently is no option to turn of greylisting, and I'm not planning to add one. I think most people who would turn it off would be because of experience with bad greylisting systems (eg I just got an email from a user because their domain forwarder recently implemented greylisting, but are only whitelisting hosts by support request, not automatically, so all email from FM is currently delayed until our servers are listed). I believe the current conservative policy used has lots of upside with no downside that I can see. Anyway, greylisting isn't meant to be a full spam protection method, it's just another adjunct. So now we have the follow stages of protection: 1. RBLs (dsbl/xbl) 2. greylisting 3. spamassassin 4. backscatter detection And probably the main last piece to come will be the per-user bayes db... Rob

23rd July 2006, 04:27 AM	#6
paleolith Essential Contributor Join Date: Mar 2002 Location: Florida Posts: 487	David, First, Rob didn't say how FM is identifying "known dialups". My guess would be that the identification comes from specific lists, and that such legitimate mail servers are not in those lists. (If they are in those lists, then they are already being punished in much more severe ways.) Second, in the algorithm Rob described, the email from a greylisted server will be temporarily rejected on the first try. When the server retries, on its next dial-out, it will succeed. All mail servers include this retry logic, which is a basic and necessary part of mail transfer. Once it succeeds twice, it will be able to send without restriction. If a single mail server is sending to FM customers regularly, it will essentially never be slowed. If it only sends occasionally, it will be slowed by the time needed to retry. The key here is that spambot zombies don't retry, and thus will pass to blacklisted rather then to whitelisted. Waiting to see if the mail server retries is an excellent discriminator between valid mail servers and spam zombies. Of course, those dial-up mail servers are also probably among the ones most hurt by spam. FM is clearly working diligently to avoid collateral damage in the war on spam, and any dent in the spam scourge will help those dial-up mail servers. Edward Last edited by paleolith : 27th July 2006 at 07:29 AM.

21st July 2006, 06:28 PM	#2
JasonWard Cornerstone of the Community Join Date: Mar 2004 Location: London, UK Posts: 834	Thanks Rob, If you implemented this in the last 2 to 3 days I would say its made a noticible impact for me,as the number of SPAM seems to have halved in that time. Jason

23rd July 2006, 11:13 AM	#8
David Intergalactic Postmaster Join Date: Dec 2001 Location: location, location Posts: 9,265	Thanks DrS and paleolith for the explanation.

26th July 2006, 10:11 PM	#9
msulloway Senior Member Join Date: Mar 2005 Location: Florida Posts: 154	This seems to have stopped all the dictionary attacks to my member account. Thanks.

27th July 2006, 05:57 PM	#11
Ruhe Junior Member Join Date: Jul 2005 Posts: 17	What is "backscatter detection"?

27th July 2006, 07:26 PM	#13
Ruhe Junior Member Join Date: Jul 2005 Posts: 17	Thanks Jason. And this can be en-/disabled in FM.

27th July 2006, 08:10 PM	#14
JasonWard Cornerstone of the Community Join Date: Mar 2004 Location: London, UK Posts: 834	Hmmm, I guess, I use Sieve scripting and currently ignore the backscatter stuff (so in effect it's turned off for me). As I never see the rules screen, and I don't ever recall having seen it I am far from a good guide as to how you configure it when not using Sieve. Jason

27th July 2006, 09:16 PM	#15
BobSchneider Junior Member Join Date: Jul 2006 Posts: 4	I am not happy with your greylisting because there is a chance that it could result in mail that I need to receive being rejected....and I can't turn it off. The chance may be very very small I admit, but there is no way of telling what impact a rejected e-mail might have on me or my business. I believe that if an e-mail is addressed to me it should be my choice whether or not it passses through filters that could cause non-delivery. I also believe that if mail gets filtered into non-delivery, I should have an option to be informed about its rejection so that I can take remedial action if necessary (eg contact the sender to inform them that they have a virus). It is not acceptable to me for mail addressed to me to be silently rejected by a process which I cannot avoid. I can see why you have taken these steps, but in my book you have changed from a reliable e-mail service to an unreliable one, purely because you have taken steps that could lose e-mail addressed to me. Bob