POP and Spam

[Bagnet]

Hi.
Those of you using POP and an e-mail client (which only downloads from the Inbox) how do you deal with the mails that go to the spam folder? Do you check the web interface regularly, or do you use a workaround?

thanks.

[xyzzy]

Periodically check the webmail. I also have other accounts and aliases I access via mail fetch that I don't want in my client so I check them in the webmail too. I wouldn't normally do that but since using FM it seems to have one of the best/fastest webmail UI's I've seen. Up until getting FM I always hated using webmails.

I suppose if you wanted to only use your client you could turn spam checking off in the webmail and send everything to your client and depend on its filtering and spam handling. At that point you are only using FM only for its server access. I don't do that but I thought I'd mention it anyway.

[Bagnet]

Thanks for the answer

Originally Posted by xyzzy

I suppose if you wanted to only use your client you could turn spam checking off in the webmail and send everything to your client and depend on its filtering and spam handling.

What if you keep spam checking on? Is there a way of having everything in your inbox but with the spam mails marked as such?

[SideshowBob]

Originally Posted by xyzzy

I suppose if you wanted to only use your client you could turn spam checking off in the webmail and send everything to your client and depend on its filtering and spam handling. At that point you are only using FM only for its server access. I don't do that but I thought I'd mention it anyway.

It's possible to let FM run the spam scanning, but skip the filtering, and let the client use the headers to determine what is spam. You need to use sieve for this.

[n5bb]

Originally Posted by Bagnet

... What if you keep spam checking on? Is there a way of having everything in your inbox but with the spam mails marked as such?

I recommend using IMAP (which allows you to see folders) or webmail. But if you insist on using POP, you can disable the Fastmail spam filing feature and add a spam tag to the subject. This requires adding some custom sieve code as shown below:

• First log into your Fastmail account and go to the Filters & Rules page.
• In the Filters section at the top, go to the Advanced settings area of Spam Protection. Change the Protection level to Custom and be sure that neither the Move or Permanently delete check boxes are checked. Then click Back at the upper left.
• At the bottom of the Rules area click Edit custom Sieve code.
• In the second unshaded Sieve code block (between the ### 3 and ### 4 code blocks) insert the following sieve code. You can change the spam score level from 5 to a different level if you wish:

if header :value "ge" :comparator "i;ascii-numeric" "X-Spam-score" "5" {
    if header :matches "Subject" "*" {
        set "subject" "${1}";
    }
    deleteheader "Subject";
    addheader :last "Subject" "[SPAM] ${subject}";
    stop;
   }

If you disable the spam filter as I described earlier, this Sieve code will insert [SPAM] at the start of the subject of messages which generate a spam score is 5 or greater. The message will remain in your Inbox where it can be retrieved via POP if no other rule moves it.

Bill

[xyzzy]

A little optimization for that code (don't need the subject variable):

if header :value "ge" :comparator "i;ascii-numeric" "X-Spam-score" "5" {
  if header :matches "Subject" "*" {
    deleteheader "Subject";
    addheader "Subject" "[SPAM] ${1}";
    stop;
  }
}

Update:
I just realized something about the Spam Protection settings I hadn't thought about before. There's a checkbox for "Share spam with other companies fighting it". I've always had that checked but I didn't think until now how or when it knows something is spam for sharing. I assume it means anything deleted from the Spam folder or that has a folder's "Spam Learning" checkbox checked. If so, doing the techniques being described here makes that "Share spam..." setting useless.

Extending that line of thought bypassing all the FM spam filters to place everything in FM's INBOX means there would be no (further) spam filter training. From FM's point of view there is no spam or at least no more after that code is put in. So how reliable is checking X-Spam-score (unless it has a reasonable history of already being trained)? If the intent is to let the client do all the spam filtering then why bother even prefixing the Subject line at all? Let the client's spam filters be trained instead.

[n5bb]

Thanks for simplifying my code!

Yes, it appears that placing all spam messages into your Inbox as I have described will disable spam learning. But if someone never uses the web interface they probably haven't read that section of the Help and set up folder training so that 200 Spam and 200 Ham are detected so that your personal Bayes filter is activated.

The X-Spam-score header is set from the sum of X-Spam-Hits component scores. For example, I just received a spam message with X-Spam-score of 4.3, and because it was new to my account the Bayes score was BAYES_00 -1.9. Counteracting that was DCC_CHECK 1.1, ME_SENDERREP_DENY 4, and MPART_ALT_DIFF 0.79.

It's also possible for a POP client to access the Spam folder directly. See:
https://www.fastmail.com/help/clients/popfolders.html
But my guess is that most people still using POP are treating their email transactionally, similar to how they use text messaging. They just want a simple way to see their new messages.

I use Fastmail in a completely different manner. I have 190,000 messages accumulated over the past 16 years I have had a Fastmail account. The oldest email I have saved (after the Fastmail signup messages) was a cartoon about the US President George W Bush while he was in his first term in April, 2004. I will make a separate post about the dramatic improvements to the Fastmail search feature, which is one of the main reasons I only use web access to my Fastmail account.

Bill

[SideshowBob]

It's possible to force everything into Inbox, let the existing sieve file a copy of spam into the spam folder and create a rule to file ham into "Incoming". You can then login occasionally to check for errors. "Incoming" could be set up as a training folder or ham could be moved to a training folder after checking.

I think, long ago, FM allowed pop retrieval from any folder by combining the folder name into the username. I guess this went a long time ago. Does anyone know?

Edit. I see this has already been mentioned. It would be a workable alternative to fetch spam from a second pop account.

[xyzzy]

Originally Posted by n5bb

It's also possible for a POP client to access the Spam folder directly. See:
https://www.fastmail.com/help/clients/popfolders.html
But my guess is that most people still using POP are treating their email transactionally, similar to how they use text messaging. They just want a simple way to see their new messages.

Heh, cute. Plus addressing for a login id in a POP client. Didn't know about that one. Thanks for pointing that out.

Of course you would need to configure the POP client with separate account settings to access each specific FM folder individually.

[SideshowBob]

Originally Posted by n5bb

For example, I just received a spam message with X-Spam-score of 4.3, and because it was new to my account the Bayes score was BAYES_00 -1.9. Counteracting that was DCC_CHECK 1.1, ME_SENDERREP_DENY 4, and MPART_ALT_DIFF 0.79.

Just as important is training the ME_SENDERREP result which got the above spam right when Bayes didn't.

Personally I find that Bayes almost always has unrecognized spam at BAYES_50. I've only had a handful at BAYES_00 in the last 10 years.

[SideshowBob]

Originally Posted by xyzzy

Of course you would need to configure the POP client with separate account settings to access each specific FM folder individually.

Or use sieve to put everything you want to retrieve by POP into a single dedicated folder. That's what I do with getmail (with IMAP retrieval).

[n5bb]

Here is a better way to do this (replacing my earlier idea):

• First log into your Fastmail account and go to the Filters & Rules page.
• In the Filters section at the top, go to the Advanced settings area of Spam Protection. Be sure that your spam protection is active and write down your spam filtering threshold (default is 5). Then click Back at the upper right.
• At the bottom of the Rules area click Edit custom Sieve code.
• In the first unshaded Sieve code block (before the ### 1 code block) insert the following sieve code. You can change the spam threshold from 5 to a different level if you wish. This spam threshold will override the normal spam setting, since this new code stops execution before the normal spam filter is reached.
• This code should copy each spam message to the Spam folder but leave a copy in your Inbox with a modified subject containing [SPAM] so it can be pulled by POP.
• I can't easily test this code until I get more spam into my test account. This may take a week or so.

if header :value "ge" :comparator "i;ascii-numeric" "X-Spam-score" "5" {
  fileinto :copy "\\Junk";
   if header :matches "Subject" "*" {
    deleteheader "Subject";
    addheader "Subject" "[SPAM] ${1}";
  }
  stop;
  }

You should also set your Spam folder to auto-delete in some interval which is long enough so you can discover good messages accidentally classified as spam. These false positives must be manually marked as non-spam by the user in the web interface. You can delete spam messages which are in the Spam folder and that should mark them as spam in your personal spam database.

The spam filtering system is really set up for use by a browser or the iOS/Android app. It is not possible to get all the benefits of the spam filtering system with learning using a POP email client. If you insist on using POP, be sure to log into the web interface periodically and mark false positive and false negative spam messages appropriately.

Bill

[SideshowBob]

The above wont work well if a POP3 client deletes mail in the Inbox, because no ham will ever be learned. Even if the client is set to delete after n days, or leave it indefinitely, it's still error prone.

I suggest doing it the way I do it, filing a separate copy in a folder dedicated for retrieval.