EmailDiscussions.com  

Go Back   EmailDiscussions.com > Email Service Provider-specific Forums > FastMail Forum
Register FAQ Members List Calendar Search Today's Posts Mark Forums Read
Stay in touch wirelessly

FastMail Forum All posts relating to FastMail.FM should go here: suggestions, comments, requests for help, complaints, technical issues etc.

Reply
 
Thread Tools
Old 16 Feb 2007, 01:20 PM   #1
robmueller
Intergalactic Postmaster
 
Join Date: Oct 2001
Location: Melbourne, Australia
Posts: 6,102

Representative of:
Fastmail.FM
Recent spam checking summary

A while back we installed FuzzyOcr, a SpamAssassin plugin that used OCR programs to try and detect the surge in image spam that was going on. That worked fine for a while, but spammers changed to using obfuscated images which weren't easily readable by OCR systems. However it didn't matter that much, because:

a) Most of the new spam machines started being listed on RBL lists, so they would get spam scores regardless of the image analysis
b) Some SA rules were added by the regular sa-update that gave a score to the general form of message with attached gif

It seems these two combined in most cases would get the spams over the 5 point threshold that's the default "Normal" level protection.

Now the problem is that the RBL stuff doesn't work nearly as well for people forwarding from another service to FM. Basically SpamAssassin will in quite a few cases only look at the network "edge" where the email came from to our system because you can't trust headers beyond that. In the case of forwarding services, that means the forwarding service itself is checked against RBLs, not very helpful. I made a change a while back to SA that tries to help with that by defining some "trusted" forwarding servers. If we find those in the headers, we scan back through them to the IP of the machine that entered that system. The current list of trusted systems is:

trusted_host nic.name
trusted_host infidels.org
trusted_host zoneedit.com
trusted_host pobox.com srs
trusted_host google.com srs
trusted_host livemail.co.uk
trusted_host hotmail.com
trusted_host yahoo.com
trusted_host outblaze.com
trusted_host mailsnare.com
trusted_host runbox.com
trusted_host gmx.net
trusted_host mxes.net srs
trusted_host iki.fi

Note that being a "trusted" system doesn't mean we don't spam check it, it just means that we parse back through the Received headers to find what server delivered the email to that service, rather than using that services IP. This improves RBL checks enormously because there's no reason for any of the above services to be on an RBL.

To avoid forgery issues, we look at the IP address the Received header shows the email came from, and do a reverse and forward DNS lookup to see that they match, and then see that it's a host within one of those domain names above.

This has helped with those forwarders, but of course not all forwarding systems people have setup. So I had a quick look if we could improve the image spam scanning. An hour of fiddling, and I found a set of transforms that does amazingly well on almost all the current obfuscated image spams out there. Check out this dir:

http://forum.robm.fastmail.fm/spam/

You can see some original, and some "fixed" versions. Feeding the fixed versions into OCR software usually gives some meaningful result.

I rolled this out to our spam scanning machines for all incoming email yesterday (note I haven't rolled it out to the machines that handle Pop Links yet), and I notice that overnight I got no image spams at all in my Junk Mail folder. Hopefully a trend that continues.

Anyway, this is all just FYI for people...

Rob
robmueller is offline   Reply With Quote

Old 16 Feb 2007, 03:36 PM   #2
beq
Cornerstone of the Community
 
Join Date: Jan 2005
Location: USA
Posts: 895
Great stuff, I didn't know that you differentiate mail from trusted forwarders and check the previous hop server (I'm glad that Tuffmail's on the list).

Also, thanks for the informative write-up on your image spam scanning efforts -- always appreciated.
beq is offline   Reply With Quote
Old 16 Feb 2007, 05:30 PM   #3
sjk
Master of the @
 
Join Date: May 2002
Location: Eugene
Posts: 1,975
Quote:
Originally posted by beq
Also, thanks for the informative write-up on your image spam scanning efforts -- always appreciated.
Yep. Thanks, Rob.
sjk is offline   Reply With Quote
Old 16 Feb 2007, 08:08 PM   #4
savirr
Senior Member
 
Join Date: May 2003
Location: Cambridgeshire, UK
Posts: 155
Rob,

Is there a process to request an addition to the list of trusted forwarding systems?

Also, for trusted systems can you look further back the chain of Received headers to provide the information in the X-Spam-source header?

Simon
savirr is offline   Reply With Quote
Old 18 Feb 2007, 08:37 PM   #5
hadaso
The "e" in e-mail
 
Join Date: Oct 2002
Location: Holon, Israel.
Posts: 4,799
When a user is using a forwarding service then there is a specific SMTP envelope rcpt address the user set at the forwarder to forward to. I have several forwarders and each forwards to a specific subdomain@alias address. It would be helpful if there was a way for a user to declare that a certain incoming address is expecting mail from a certain forwarding host, and then trust the next received headers from these hosts when they send to the particular address (though I'm not sure how easy it is to implement. Most forwarding services I use insert several "Received" headers for several internal transfers.

SpamCop has their "mailhosts" configuration system for making their system learn about the forwarders each user uses, but I guess parsing "Received" headers is their specialty.
hadaso is offline   Reply With Quote
Old 19 Feb 2007, 12:09 AM   #6
meshugena313
Junior Member
 
Join Date: Nov 2005
Posts: 15
academic mail forward service

I've been seeing a massive increase in spam reaching my inbox and it all has a spam score of 0! So now I know why - it's all being forwarded through 2 university mail systems. Rob - can you somehow add "trusted" .edu mail forwarders?
meshugena313 is offline   Reply With Quote
Old 20 Feb 2007, 09:03 AM   #7
robmueller
Intergalactic Postmaster
 
Join Date: Oct 2001
Location: Melbourne, Australia
Posts: 6,102

Representative of:
Fastmail.FM
Currently adding a trusted forwarder is manual only, however I was thinking of adding an option on the Options -> Spam/Virus protection screen so people could add custom hostnames. Still, I'd like to make sure that the most common forwarders are included so things just "work" for most people.

If you email me at robm AT fastmail DOT fm with forwarder details, I'll think about adding them.

Re the X-Spam-source header. My plan had been to integrate this with the improved SA Received header tracking, I just hadn't got to it yet.

FYI, I only received 2 image spam messages over the entire weekend in my Junk Mail folder, which means all the rest must have been recognised and given a score > 10. Normally I get a dozen in my Junk Mail folder.

Rob
robmueller is offline   Reply With Quote
Old 21 Feb 2007, 07:09 PM   #8
hadaso
The "e" in e-mail
 
Join Date: Oct 2002
Location: Holon, Israel.
Posts: 4,799
Quote:
Originally posted by robmueller
If you email me at robm AT fastmail DOT fm with forwarder details, I'll think about adding them.
I think a ".edu forwarder" usually means a .forward file in a user's homedir on unix, so allowing this as "trusted forwarder" ammounts to either allowing all the .edu tld and also things like .ac.cc or .edu.cc (where cc stannds for country code) or listing lots of individual universities and many hosts within each university. My own forwarders are ams.org, openu.ac.il and my ISP (and I'm not sure if it can be "trusted").
hadaso is offline   Reply With Quote
Old 21 Feb 2007, 08:05 PM   #9
robmueller
Intergalactic Postmaster
 
Join Date: Oct 2001
Location: Melbourne, Australia
Posts: 6,102

Representative of:
Fastmail.FM
I won't add isps to trusted hosts, since they are actually indirectly the source of most spam with their users on dsl networks with compromised machines.

Actually, that's all the trusted_hosts thing is about, allowing us to correctly scan back through Received headers that are NOT an ISP. In theory, about the only machines on RBLs should be:

1. Compromised machines on DSL/dialup lines (99%)
2. Compromised servers (1%)

It's silly using RBLs that block email sources with any legitimate mail (eg hotmail/yahoo/etc) even if they do send some spam, because you're just randomly blocking some machine and users. I think RBLs are best when they just block known insecure machines that should never be sending email.

I think having the per-user option is the best way to go in the long term, but I'm happy to add forwarders people use if they email me.

Rob
robmueller is offline   Reply With Quote
Old 22 Feb 2007, 08:10 AM   #10
hadaso
The "e" in e-mail
 
Join Date: Oct 2002
Location: Holon, Israel.
Posts: 4,799
Quote:
Originally posted by robmueller
I won't add isps to trusted hosts, since they are actually indirectly the source of most spam with their users on dsl networks with compromised machines. ...
I thought that the "trusted hosts" was just about being able to trust a host to put in correct "Received" headers. So even if an ISP's outgoing SMTP server relays spam sent by broadband subscribers it usually can be trusted to prepend correct "Received" headers to the email it relays.

Anyway, how do you know where to stop accepting "Received" headers? Most forwarders I've seen add several such headers.

Here are headers from mail forwarded by the AMS:
Code:
Received: from ams.org (mail01.ams.org [130.44.1.106])
	by mx2.messagingengine.com (Postfix) with ESMTP id 907F91DE08D
	for <member@myself.123mail.or9>; Tue, 20 Feb 2007 09:53:09 -0500 (EST)
Received: from smtp.ams.org (smtp.ams.org [130.44.1.23])
	by ams.org (Switch-3.2.5/Switch-3.2.5) with ESMTP id l1KErQJm007015
	(version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO)
	for <myself@member.ams.or9>; Tue, 20 Feb 2007 09:53:27 -0500 (EST)
Received: from narkis.wisdom.weizmann.ac.il (narkis.wisdom.weizmann.ac.il [132.76.80.32])
	by smtp.ams.org (8.12.11.20060308/8.12.11) with ESMTP id l1KEr4Fg006136
	for <myself@member.ams.or9>; Tue, 20 Feb 2007 09:53:05 -0500
The bottom two "Received headers were added by the AMS.

Here's one from my ISP:
Code:
Received: from omr1.bezeqint.net (omr1.bezeqint.net [192.115.104.8])
	by mx1.messagingengine.com (Postfix) with ESMTP id C7F39A919C
	for <bezeqint@myself.123mail.or9>; Mon, 12 Feb 2007 23:43:39 -0500 (EST)
Received: from mas21.bezeqint.net (mas21.bezeqint.net [192.115.104.151])
	by omr1.bezeqint.net (Bezeq International SMTP out Mail Server) with ESMTP id 968201622D6
	for <bezeqint@myself.123mail.or9>; Tue, 13 Feb 2007 06:43:40 +0200 (IST)
Received: from localhost (localhost [127.0.0.1])
	by mas21.bezeqint.net (MOS 3.7.3a-GA)
	id CQE91258;
	Tue, 13 Feb 2007 06:43:35 +0200 (IST)
Received: from mr5.bezeqint.net (mr5.bezeqint.net [192.115.104.75])
	by mas21.bezeqint.net (MOS 3.7.3a-GA)
	with ESMTP id CQE91213;
	Tue, 13 Feb 2007 06:43:31 +0200 (IST)
Received: from SHIVUK-NET-5 (Hosting-IGLD-192-248.inter.net.il [213.8.192.248] (may be forged))
	by mr5.bezeqint.net (MOS 3.7.5a-GA)
	with SMTP id EIB12155;
	Tue, 13 Feb 2007 06:43:30 +0200 (IST)
Here the forwarder added 4 "Received" headers so if the forwarder can be trusted the bottom one shows the originating IP (personally I trust these headers and crop all but the bottom one from spam reported using SpamCop).

And here's one that adds 5 "Received" headers:
Code:
Received: from davar1.openu.ac.il (mailhost-main.openu.ac.il [192.114.2.5])
	by mx2.messagingengine.com (Postfix) with ESMTP id 3A0C71DBDAA
	for <openu@outlook.myself.tld>; Fri, 16 Feb 2007 10:03:21 -0500 (EST)
Received: from TAMAR.openu.local (rimon.openu.ac.il [147.233.6.65])
	by davar1.openu.ac.il (Postfix) with ESMTP id 197C7358827
	for <openu@outlook.myself.tld>; Fri, 16 Feb 2007 17:03:11 +0200 (IST)
Received: from metate.openu.ac.il ([147.233.197.30]) by TAMAR.openu.local
         with Microsoft SMTPSVC(6.0.3790.1830);
	 Fri, 16 Feb 2007 17:03:45 +0200
Received: from madrid.openu.ac.il (mailhost-main.openu.ac.il) by metate.openu.ac.il
 (Clearswift SMTPRS 5.2.5) with SMTP id <T7dd7ec2d7a93e9c51e9f8@metate.openu.ac.il>
 for <me@openu.ac.i1>; Fri, 16 Feb 2007 17:04:51 +0200
Received: from davar1.openu.ac.il (davar1.openu.ac.il [192.114.2.5])
	by madrid.openu.ac.il (8.12.11.20060308/8.12.11) 
        with ESMTP id l1GF29jU001194
	for <me@openu.ac.i1>; Fri, 16 Feb 2007 17:02:21 +0200
Received: from server2.kcsnet.net (kcsnet2.spd.co.il [212.199.125.71])
	by davar1.openu.ac.il (Postfix) with ESMTP id 0A1EF35882D
	for <me@openu.ac.i1>; Fri, 16 Feb 2007 17:02:14 +0200 (IST)
hadaso is offline   Reply With Quote
Old 22 Feb 2007, 09:05 AM   #11
robmueller
Intergalactic Postmaster
 
Join Date: Oct 2001
Location: Melbourne, Australia
Posts: 6,102

Representative of:
Fastmail.FM
Basically spamassassin already includes a Received: header parser that tries to break a Received header down into a common format (for the interested, it's a several hundred line function that tries to match against many, many different formats since there's no standard Received format).

http://search.cpan.org/src/FELICITY/...ta/Received.pm

Now by default, spamassassin will search back through the Received headers to find the IP the message was received from into the local system. All we do is extend the length it keeps searching back to include extra "trusted" hosts.

So lets say we added "ams.org" as a trusted host. Then this line:

Code:
Received: from ams.org (mail01.ams.org [130.44.1.106])
	by mx2.messagingengine.com (Postfix) with ESMTP id 907F91DE08D
	for <member@myself.123mail.or9>; Tue, 20 Feb 2007 09:53:09 -0500 (EST)
Would be broken down by SA into:

Code:
  my $relay = {
    ip => '130.44.1.106',
    rdns => 'mail01.ams.org',
    by => 'mx2.messagingengine.com',
    helo => 'ams.org',
    id => '907F91DE08D',
  };
We look at the rdns value (mail01.ams.org), and see if it's in our trusted host list. It's not, so we strip the /^[^.]*\./ from the front to get "ams.org" and try again. This is in our trusted host list.

Now we need to check that the header isn't forged, and the ips are actually real. So we do a DNS lookup on mail01.ams.org to get the IPs.

Code:
$ dig +short mail01.ams.org
130.44.1.106
And we see this does match the IP in the header, so we trust this Received header, and move on to the next one. Repeating this process would get us to:

Code:
Received: from narkis.wisdom.weizmann.ac.il (narkis.wisdom.weizmann.ac.il [132.76.80.32])
	by smtp.ams.org (8.12.11.20060308/8.12.11) with ESMTP id l1KEr4Fg006136
	for <myself@member.ams.or9>; Tue, 20 Feb 2007 09:53:05 -0500
As the header of interest to check the handoff IP against.

With the second example, assuming we trust bezeqint.net, we can see a similar process. The only odd one is:

Code:
Received: from localhost (localhost [127.0.0.1])
	by mas21.bezeqint.net (MOS 3.7.3a-GA)
	id CQE91258;
	Tue, 13 Feb 2007 06:43:35 +0200 (IST)
But since 127.0.0.1 is obviously an internal handoff, I'm pretty sure SA trusts this header and moves on.


Rob
robmueller is offline   Reply With Quote
Old 22 Feb 2007, 06:24 PM   #12
hadaso
The "e" in e-mail
 
Join Date: Oct 2002
Location: Holon, Israel.
Posts: 4,799
So actually SA does a pretty good job going down all received headers representing internal mail transfers inside an ISP given only the ISP's domain. So if you can get it to "trust" a user specified domain the user would only have to specify the ISP's domain and not any internal email hosts for it to work. I prefer it to be limited only to specified email address since the forwarding works is that a user specifies a particular email address to the forwarder to forward to, so there is no reason to "trust" the forwarder except for the specific email address email is forwarded to.
hadaso is offline   Reply With Quote
Old 23 Feb 2007, 02:25 PM   #13
robmueller
Intergalactic Postmaster
 
Join Date: Oct 2001
Location: Melbourne, Australia
Posts: 6,102

Representative of:
Fastmail.FM
When we say "trust" we don't really mean "anything coming from this host is ok", what we mean is that "this host just passes on email, assume it's headers are ok and find the real source"

With that definition, it doesn't really matter whether it's a specific email address or not for the hosts you "trust"

Rob
robmueller is offline   Reply With Quote
Old 23 Feb 2007, 06:18 PM   #14
hadaso
The "e" in e-mail
 
Join Date: Oct 2002
Location: Holon, Israel.
Posts: 4,799
Quote:
Originally posted by robmueller
With that definition, it doesn't really matter whether it's a specific email address or not for the hosts you "trust"
I understand that the definition ot "trusted" here is just "trusted to report the correct IP address in the 'Received' header".

But then if for instance I am a client of inter.net.il (which I would bet several FastMail users are. It's one of the biggest ISPs in Israel) then I might have then forward my email to FastMail. Then if I can tell FastMail "trust inter.net.il" when parsing "Received" headers then if I get spam sent from a broadband subscriber of inter.net.il sent directly to FastMail or to any forwarder I "trust" then I would also trust the "Received" line that the spammer put in if the spammer corectly uses the rdns of the IP address that sends the spam, and then the spammer can indicate a forged source in a forged header I "trust". For instance in the second example that I posted above, the one with bottom "Received" header saying:
Received: from SHIVUK-NET-5 (Hosting-IGLD-192-248.inter.net.il [213.8.192.248] ...
the spammer can identify as Hosting-IGLD-192-248.inter.net.il which would pass the "trust test" and then the spammer can forge another "Received" line that lets Hosting-IGLD-192-248.inter.net.il tell us that the email originated from somewhere else. This is a side effect of the fact that anything with suffix inter.net.il would be "trusted" if inter.net.il is trusted. I don't know if limiting "trust" per address set to receive forwarded mail can solve it, but it can limit this problem to spam sent within the forwarding ISP.

Last edited by hadaso : 4 Mar 2007 at 08:23 AM. Reason: typos, typos ...
hadaso is offline   Reply With Quote
Old 26 Feb 2007, 01:09 PM   #15
robmueller
Intergalactic Postmaster
 
Join Date: Oct 2001
Location: Melbourne, Australia
Posts: 6,102

Representative of:
Fastmail.FM
This is why I said above:

"I won't add isps to trusted hosts, since they are actually indirectly the source of most spam with their users on dsl networks with compromised machines."

If you use an ISP for forwarding then, you're unfortunately out of luck at the moment with this. However you are right, you can narrow down the problem by making trust a tuple of "host/rcpt-to-address" that's trusted rather than just "host". When I add "per user trusted hosts", i'll keep it in mind...

Rob
robmueller is offline   Reply With Quote
Reply


Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump


All times are GMT +9. The time now is 02:46 PM.

 

Copyright EmailDiscussions.com 1998-2022. All Rights Reserved. Privacy Policy