Greylisting updates

robmueller · 2nd August 2006, 07:55 PM

I've made some updates to the policy system. As mentioned previously, there are two parts to the policy system that run indepedently: "Greylisting" and "Address Enumeration Detection" (AED)

Because of the way Verizon were using "address verifcation" of incoming email to their systems, it was causing their servers to be blocked by ours by the AED system. I now have a workaround in place for them, and something that should also work for most other places that have a sane "address verification" system in place.

There's been some concern about greylisting delaying email, so I've tightened it up some more and fixed some bugs.

1. Bugfix: It's definitely only "dialup" hosts that are now being greylisted, there was a bug that was causing some non-dialup hosts to be greylisted. (By dialup, I mean hosts that match specific criteria that make them look like a dialup host to us...)

2. Bugfix: If an IP passes greylisting twice, that host is whitelisted and not subject to greylisting for the next 24 hours. This information is now broadcast to other MX servers, rather than per-MX server.

3. Enhancement: If a host opens an SMTP session with a HELO that is not an IP address, is not the same reverse DNS as it's connecting IP, but the forward DNS of the name does resolve to the connecting IP, then that host is not subject to greylisting. (As suggested by hadaso in another thread) This actually catches quite a few machines.

An example: The machine at IP 206.223.169.73 connects to us. The reverse DNS for 206.223.169.73 is 206-223-169-73.beanfield.net, which looks like a common dialup/dynamic IP name, and would be a candidate for greylisting. However, the machine advertises itself to us with a "HELO mx3.hub.org" line. Doing a forward lookup of mx3.hub.org gives the IP 206.223.169.73, which is the same as the connecting IP, so we exclude it from greylisting.

4. Enhancement: Once a message passes greylisting and is accepted, a new header is added "X-Spam-greylist". This header tells you how many seconds the email was delayed and whether that host has been whitelisted for 24 hours.

(Technical: Well, actually the delay figure is the how long the last delay for the ip/sender/recipient combination was, so in the case of multiple emails from the same person, to the same person, from the same machine in a short time period, the figure will be a bit messy and hard to calculate)

I think all of these should help greylisting be more accurate with which hosts it greylists, while not loosing any of the functionality.

Rob

Update: fixed wording

henrico · 2nd August 2006, 08:01 PM

Thanks for the updates, Rob.

I hope it's also possible that the spam filter will be trainable in the near future...

sflorack · 2nd August 2006, 10:55 PM

Was this service beta tested prior to being rolled out to the masses? We're not talking about an interface change here -- this is something that conceivably can (and as it sounds, has) prevented delivery of emails.

robmueller · 3rd August 2006, 09:38 AM

Of course it was tested. It's just that cases were exposed in full production that testing doesn't always uncover. As noted above, there were only 2 bugfixes, and no email should have been lost because of them, because no email was ever rejected, just delayed.

Rob

sflorack · 3rd August 2006, 10:33 PM

Quote:

Because of the way Verizon were using "address verifcation" of incoming email to their systems, it was causing their servers to be blocked by ours by the AED system.

This doesn't suggest that mails went undelivered?

BinaryTB · 4th August 2006, 01:57 AM

Quote:

An example: The machine at IP 206.223.169.73 connects to us. The reverse DNS for 206.223.169.73 is 206-223-169-73.beanfield.net, which looks like a common dialup/dynamic IP name, and would be a candidate for whitelisting. However, the machine advertises itself to us with a "HELO mx3.hub.org" line. Doing a forward lookup of mx3.hub.org gives the IP 206.223.169.73, which is the same as the connecting IP, so we exclude it from whitelisting.

Is that correct?

robmueller · 4th August 2006, 11:22 AM

Quote:

Originally posted by Spooon69
Is that correct?

Ooops, I meant greylist, fixed now.

Rob

robmueller · 4th August 2006, 01:00 PM

Quote:

Originally posted by sflorack
This doesn't suggest that mails went undelivered?

All the blocks were 453 temporary deferrals, so once we fixed it, all the emails went through.

Rob

janetlorraine · 7th August 2006, 06:29 AM

I don't know if the new greylisting is impacting my email or not, but I'm suddenly not getting a lot of my email - email that I want. A lot of people who write to me use dialup connections, though some have DSL, etc. Some use Verizon but not many. I belong to a mailing list, too, and haven't been getting that mail. I don't know if this is caused by the server going down and my mail just hasn't been restored or because of the greylisting. Are there any plans for users to identify trusted addresses or identify certain types of mail as being sent from mailing lists so that it doesn't get blocked?

I have an enhanced account and havesent an email about my missing mail, requesting help, but no response to my email yet.

Janet

hadaso · 7th August 2006, 07:04 AM

Quote:

Originally posted by janetlorraine
... A lot of people who write to me use dialup connections, though some have DSL, etc. ...

They greylisting only affect senders that run server software through a consumer's connection (dialup/DSL/cables). Most people use these connections, of course, but almost all of us don't run our own server software at home, but rather configure our email software to hand our email over to our service provider's server that then sends it to the recipient (or use webmail that also sends the email from a service provider's server). Greylisting of consumer's connection only affect servers operated on a PC connected therough a consumer's connection that tries to contact FastMail's server directly and hand over a message.

don_dannielo · 7th August 2006, 01:16 PM

Am I the only one who with the new greylist, now receives spam in Inbox?

It's true that now the incoming spam is MUUUUUCH less, but some is leaking in the inbox. Before it hasn't happened.

casemods · 7th August 2006, 07:21 PM

Can you explain to me what greylisting is in noob terms?

I just want to be able to train the spam filter.

-Casemods

don_dannielo · 7th August 2006, 08:10 PM

Quote:

We've implemented a technique called greylisting to help significantly reduce the incoming spam to all accounts. Greylisting is very effective against the large number of zombie computers which deliver the vast majority of spam, which is why we've implemented it. Some naive implementations of greylisting that people may be familiar with from other providers are prone to delaying a lot of email. We've been very careful with our implementation so that the vast majority of email won't be delayed in any way. See this blog post for more details on how we've done this.

We implemented this feature about 2 weeks ago, and fine tuned it some more last week. For all users, the result will just mean that they see less spam in their accounts.

http://en.wikipedia.org/wiki/Greylisting
http://en.wikipedia.org/wiki/Zombie_computer

digp · 8th August 2006, 05:47 AM

Quote:

Originally posted by sflorack
Was this service beta tested prior to being rolled out to the masses? We're not talking about an interface change here -- this is something that conceivably can (and as it sounds, has) prevented delivery of emails.

It's not the end of the world, is it?

janetlorraine · 8th August 2006, 06:07 AM

While it may not cause the world to end, non-receipt or untimely receipt of emails can have a significant impact on an individual's world. It can impact their earnings, getting a new job, getting info about family members that's really critical... all kinds of things that have real meaning - and sometimes emotional meaning - to the people involved. Fastmail's reliability, great support and fast email delivery have always supported my decision to pay for an enhanced account and that hasn't changed. If emails weren't important to me, I'd go with a free service that everyone is used to using...

Janet

2nd August 2006, 07:55 PM	#1
robmueller Intergalactic Postmaster Join Date: Oct 2001 Location: Melbourne, Australia Posts: 6,098 Representative of: Fastmail.FM	Greylisting updates I've made some updates to the policy system. As mentioned previously, there are two parts to the policy system that run indepedently: "Greylisting" and "Address Enumeration Detection" (AED) Because of the way Verizon were using "address verifcation" of incoming email to their systems, it was causing their servers to be blocked by ours by the AED system. I now have a workaround in place for them, and something that should also work for most other places that have a sane "address verification" system in place. There's been some concern about greylisting delaying email, so I've tightened it up some more and fixed some bugs. 1. Bugfix: It's definitely only "dialup" hosts that are now being greylisted, there was a bug that was causing some non-dialup hosts to be greylisted. (By dialup, I mean hosts that match specific criteria that make them look like a dialup host to us...) 2. Bugfix: If an IP passes greylisting twice, that host is whitelisted and not subject to greylisting for the next 24 hours. This information is now broadcast to other MX servers, rather than per-MX server. 3. Enhancement: If a host opens an SMTP session with a HELO that is not an IP address, is not the same reverse DNS as it's connecting IP, but the forward DNS of the name does resolve to the connecting IP, then that host is not subject to greylisting. (As suggested by hadaso in another thread) This actually catches quite a few machines. An example: The machine at IP 206.223.169.73 connects to us. The reverse DNS for 206.223.169.73 is 206-223-169-73.beanfield.net, which looks like a common dialup/dynamic IP name, and would be a candidate for greylisting. However, the machine advertises itself to us with a "HELO mx3.hub.org" line. Doing a forward lookup of mx3.hub.org gives the IP 206.223.169.73, which is the same as the connecting IP, so we exclude it from greylisting. 4. Enhancement: Once a message passes greylisting and is accepted, a new header is added "X-Spam-greylist". This header tells you how many seconds the email was delayed and whether that host has been whitelisted for 24 hours. (Technical: Well, actually the delay figure is the how long the last delay for the ip/sender/recipient combination was, so in the case of multiple emails from the same person, to the same person, from the same machine in a short time period, the figure will be a bit messy and hard to calculate) I think all of these should help greylisting be more accurate with which hosts it greylists, while not loosing any of the functionality. Rob Update: fixed wording Last edited by robmueller : 4th August 2006 at 11:19 AM.

2nd August 2006, 08:01 PM	#2
henrico Essential Contributor Join Date: Jun 2003 Location: Netherlands Posts: 421	Thanks for the updates, Rob. I hope it's also possible that the spam filter will be trainable in the near future...

2nd August 2006, 10:55 PM	#3
sflorack The "e" in e-mail Join Date: Feb 2002 Posts: 2,638	Was this service beta tested prior to being rolled out to the masses? We're not talking about an interface change here -- this is something that conceivably can (and as it sounds, has) prevented delivery of emails.

3rd August 2006, 09:38 AM	#4
robmueller Intergalactic Postmaster Join Date: Oct 2001 Location: Melbourne, Australia Posts: 6,098 Representative of: Fastmail.FM	Of course it was tested. It's just that cases were exposed in full production that testing doesn't always uncover. As noted above, there were only 2 bugfixes, and no email should have been lost because of them, because no email was ever rejected, just delayed. Rob

7th August 2006, 06:29 AM	#9
janetlorraine Member Join Date: Jun 2002 Location: Baltimore, MD Posts: 52	I don't know if the new greylisting is impacting my email or not, but I'm suddenly not getting a lot of my email - email that I want. A lot of people who write to me use dialup connections, though some have DSL, etc. Some use Verizon but not many. I belong to a mailing list, too, and haven't been getting that mail. I don't know if this is caused by the server going down and my mail just hasn't been restored or because of the greylisting. Are there any plans for users to identify trusted addresses or identify certain types of mail as being sent from mailing lists so that it doesn't get blocked? I have an enhanced account and havesent an email about my missing mail, requesting help, but no response to my email yet. Janet

7th August 2006, 01:16 PM	#11
don_dannielo Member Join Date: Jun 2004 Posts: 34	Am I the only one who with the new greylist, now receives spam in Inbox? It's true that now the incoming spam is MUUUUUCH less, but some is leaking in the inbox. Before it hasn't happened.

7th August 2006, 07:21 PM	#12
casemods Junior Member Join Date: Aug 2006 Location: San Jose, CA, In my room, by myself, on the computer. Posts: 3	Can you explain to me what greylisting is in noob terms? I just want to be able to train the spam filter. -Casemods

8th August 2006, 06:07 AM	#15
janetlorraine Member Join Date: Jun 2002 Location: Baltimore, MD Posts: 52	While it may not cause the world to end, non-receipt or untimely receipt of emails can have a significant impact on an individual's world. It can impact their earnings, getting a new job, getting info about family members that's really critical... all kinds of things that have real meaning - and sometimes emotional meaning - to the people involved. Fastmail's reliability, great support and fast email delivery have always supported my decision to pay for an enhanced account and that hasn't changed. If emails weren't important to me, I'd go with a free service that everyone is used to using... Janet