EmailDiscussions.com  

Go Back   EmailDiscussions.com > Email Service Provider-specific Forums > FastMail Forum
Register FAQ Members List Calendar Search Today's Posts Mark Forums Read
Stay in touch wirelessly

FastMail Forum All posts relating to FastMail.FM should go here: suggestions, comments, requests for help, complaints, technical issues etc.

Reply
 
Thread Tools
Old 9 Sep 2019, 04:47 AM   #16
JamesHenderson
Essential Contributor
 
Join Date: Jan 2003
Location: Oxfordshire, UK
Posts: 467
Quote:
Originally Posted by xyzzy View Post
When I refer to a sieve section number I am referring to the numbered comments in the portions of the sieve script you cannot edit. For example section 3 starts with the comment,

### 3. Sieve generated for spam protection

Or to use your examples section 5 is Vacation and section 6 calendar. It's thus easier and quicker to refer to the explicitly numbered comments rather than counting edit sections.

Below is the portion of my initialization code (just after the require of course). Note, I tend to comment a lot, just my coding style.

Code:
#
# Initialize known_sender to reflect what X-Spam-Known-Sender indicates (i.e., sender is in the Contacts
# list) or for contacts that use subdomains that cannot be exhaustively specified in the Contacts list.
# Note there is also a special case where X-Spam-Known-Sender will be set to "no" even if the contact is
# on the Contacts list. That can happen if there is something bogus in the headers.  When that happens
# "in-addressbook" is added to the end of X-Spam-Known-Sender so that even with "no" it is still known
# it's in the Contacts list.  The known_sender switch being true always guarantees known senders make it
# to through even those with suspicious headers.
#
if header :matches "X-Spam-Known-Sender" ["yes*", "no*in-addressbook*"] {
  set "known_sender" "true";
} elsif address :matches "From" ["*@*abc.tld",         # whitelisted patterns that contacts doesn't support
                                 "*@*xyz.tld",         # if contacts supported full globbing instead of
                                 "*@*.gov",            # just *@domain.tld these tests woudn't be needed.
                                 "*uvwh*"] {
  set "known_sender" "true"; 
} elsif address :matches "To" ["*@LISTSERV.*"] {       # allow subscribed listserv mail lists
  set "known_sender" "true";                           # "To" is the list, "From" is users posting to list
} else {
  set "known_sender" "false";
}
So I then use "known_sender" in places I want to override that might send the message into spam. For example to bypass my black list testing and more to the point to bypass FM own sieve spam check code. So just before section 1 I have:

Code:
#
# Sender is not blacklisted or foreign at this point...
#
# Always bypass the spam score checks for what is considered senders in addition to X-Spam-Known-Sender.
# Unfortunately this also bypasses save-on-SMTP identities and UI generated discard rules but that can't
# be helped since that code below cannot be edited.  Hopefully these will never be important enough for
# known contacts (or whitelisted ones).  If they are then they will have to be explicitly coded.
#
if string :is "${known_sender}" "false" {
And just before section 4 the matching brace to the above test:

Code:
} # not known sender
Hi,

So I have tried this, but everything is now ending up in my inbox. I have traced the error to this line:
HTML Code:
if header :matches "X-Spam-Known-Sender" ["yes*", "no*in-addressbook*"]
and specifically this part:
HTML Code:
"no*in-addressbook*"
For whatever reason, if "X-Spam-Known-Sender" is "no", "known_sender" is being set to "true"

Is there something wrong or out of date with "no*in-addressbook*"?

cheers,
James.
JamesHenderson is offline   Reply With Quote
Old 9 Sep 2019, 06:11 AM   #17
xyzzy
Senior Member
 
Join Date: May 2018
Posts: 175
If you look in section 3 you will see FM's sieve's own code test' X-Spam-Known-Sender for stuff being in the the contacts list as:

Code:
if not header :matches "X-Spam-Known-Sender" "yes*" {
So the first part of my test checks for the same pattern.

My second test for "no*in-addressbook*" covers cases like the following:

X-Spam-known-sender: no ("From == To and no DKIM or SPF for from domain, likely forged"); in-addressbook
X-Spam-known-sender: no ("Email failed DMARC policy for domain"); in-addressbook

because I want names in my contacts list to always get through to my inbox no matter what. Let's not get into a philosophical debate about this. It's just my preference.

The additional match ("no*in-addressbook*") is for "no" followed by anything followed by "in-addressbook" as illustrated in those examples. Probably don't need the trailing star but it shouldn't make a difference.

I wrote that code based only on my what I have seen. I only have my own email to use as source examples. If there's other situations where something in the contacts list can cause X-Spam-Known-Sender to say "no" and also not say "in-addressbook" for a contact that is in the contacts list I don't know about them.

So in all your matches that went into the inbox do you have any where X-Spam-Known-Sender was "no" and didn't contain "in-addressbook"? Are you sure there isn't some pattern matching mistake in the address :matches "From" section?
xyzzy is offline   Reply With Quote
Old 9 Sep 2019, 06:43 AM   #18
BritTim
The "e" in e-mail
 
Join Date: May 2003
Location: mostly in Thailand
Posts: 2,797
I cannot see any particular error with your code, but I always try to simplify tests as far as possible. Tests with several wildcards make me nervous because they can cause tests to timeout. I would be inclined to write those tests as
Code:
if header :contains "X-Spam-Known-Sender" ["yes", "in-addressbook"]
Since the possible values of the X-Spam-Known-Sender header are pretty constrained, I would not worry about mismatches here.
BritTim is offline   Reply With Quote
Old 9 Sep 2019, 07:03 AM   #19
JamesHenderson
Essential Contributor
 
Join Date: Jan 2003
Location: Oxfordshire, UK
Posts: 467
Quote:
Originally Posted by xyzzy View Post
If you look in section 3 you will see FM's sieve's own code test' X-Spam-Known-Sender for stuff being in the the contacts list as:

Code:
if not header :matches "X-Spam-Known-Sender" "yes*" {
So the first part of my test checks for the same pattern.

My second test for "no*in-addressbook*" covers cases like the following:

X-Spam-known-sender: no ("From == To and no DKIM or SPF for from domain, likely forged"); in-addressbook
X-Spam-known-sender: no ("Email failed DMARC policy for domain"); in-addressbook

because I want names in my contacts list to always get through to my inbox no matter what. Let's not get into a philosophical debate about this. It's just my preference.

The additional match ("no*in-addressbook*") is for "no" followed by anything followed by "in-addressbook" as illustrated in those examples. Probably don't need the trailing star but it shouldn't make a difference.

I wrote that code based only on my what I have seen. I only have my own email to use as source examples. If there's other situations where something in the contacts list can cause X-Spam-Known-Sender to say "no" and also not say "in-addressbook" for a contact that is in the contacts list I don't know about them.

So in all your matches that went into the inbox do you have any where X-Spam-Known-Sender was "no" and didn't contain "in-addressbook"? Are you sure there isn't some pattern matching mistake in the address :matches "From" section?
gotcha - looking closer at the spam that went into my inbox, they all were using my email address as the "from" address, and X-spam-known-header was:
HTML Code:
X-Spam-known-sender: no ("From == To and no DKIM or SPF for from domain, likely forged");
 in-addressbook
...so your script worked, but as I never send emails to myself, I would still want this kind of email to get spam checked in section 3 - easy enough to add a test for that situation.

thanks!
JamesHenderson is offline   Reply With Quote
Old 9 Sep 2019, 12:56 PM   #20
xyzzy
Senior Member
 
Join Date: May 2018
Posts: 175
The setting of the known_sender to true and the way it's tested is mainly to get it past the FM generated spam test (but I also use it in other places as well). So once beyond there you're own your own to filter any way you want.

I also have an organize rule for emails sent to myself. I filter them not as spam but place them in a folder named "Unexpected". No one knows my actual FM email address. Only my forwarding service's address is known. So any email sent with an explicit To of my FM address I put in a folder of their own called "unexpected". The only legitimate time the To should be my actual FM address is when I am doing it for testing purposes or FM is sending me something directly. So part of the test is to let those emails reach my inbox (with one exception mentioned later). Below is the sieve code (albeit generated with an actual organize rule - sieve code "reads" so much better and is more concise than the organize rule UI page which defined them).

Code:
if 
  allof(
  address :is "To" "[email protected]",
  not address :matches "From" "*@fastmail.com"
  )
{
  fileinto "[email protected]^com";
}
This happens to be my first organize rule in the list hence the if test and not elsif. If no other organize rule picks these off it will end up in the inbox by default.

But there is as case I do want to pick off and I can't do it with an organize rule (or if I can it's not worth the effort since it's easier in sieve). That's the case fastmail.com directly sends me email. Obviously they do know my actual FM email address too!

I have it grouped in a special block of code following section 8 where do some other similar operations (alias address filtering but that's off topic). Without going into the actual context of what I do there since that complicates the description I include a test that basically looks like:

Code:
if allof(address :is "From" "[email protected]",
                header :matches "Subject" "*.* * Mail fetch retrieval error") {
    fileinto "${1}^${2}";
  }
If this test matches I then filter those messages into the folder named as the email address since I name top level folders by their email address (that's my convention - any subfolders would be special cases of that email address which is why I have it grouped with my alias address handling - actual code is more complicated to handle subfolder name extraction and the conversion of dots to ^'s).

The reason for this test is I have (had) a mail fetch account where about 30% of the time resulted in "Mail fetch retrieval error" being reported from fastmail.com. In the "old days" when mail fetch was using POP it was a timeout error and I could test for this and just ignore it since eventually a future fetch would succeed. Now with IMAP it's just a general "Mail fetch retrieval error" so I sort them into the email address 's folder. If I didn't they would end up in my inbox or if I didn't exclude fastmail.com from my "unexpected" test they would go into "unexpected". I want these error reports to go into the mail fetch identity's folder not elsewhere.

I have 4 other mail fetch identities set up. This problem only happens with one of the (the yahoo servers). While the above test is a general filtering rule to handle these errors, I've since changed one offending email account to forward from there and disabled my mail fetch. Screw 'em!

Last edited by xyzzy : 9 Sep 2019 at 01:09 PM.
xyzzy is offline   Reply With Quote
Old 9 Sep 2019, 04:16 PM   #21
Terry
The "e" in e-mail
 
Join Date: Jul 2002
Location: VK4
Posts: 2,791
Would this work

elsif
allof(
not header :contains "X-Spam-Known-Sender" "in-addressbook",
address :is "To" "your email address"
)
{
fileinto "INBOX";
}
Terry is offline   Reply With Quote
Old 9 Sep 2019, 04:58 PM   #22
xyzzy
Senior Member
 
Join Date: May 2018
Posts: 175
It's valid code but what are you trying to accomplish and where is this code being placed? As a organize rule it would act only on non-spam.

On face value is says "send any message from anyone not in your contacts" to the inbox. The To, in most cases, will be always be you.

Is this in reference to something I said or some of the code I included in one of the previous posts? I'm a bit confused about the context here.

Last edited by xyzzy : 9 Sep 2019 at 05:30 PM.
xyzzy is offline   Reply With Quote
Old 9 Sep 2019, 05:54 PM   #23
Terry
The "e" in e-mail
 
Join Date: Jul 2002
Location: VK4
Posts: 2,791
I just thought that it may work for the OP.

I would have thought the sent email would be put in his inbox instead of the spam folder.
Terry is offline   Reply With Quote
Old 9 Sep 2019, 06:43 PM   #24
xyzzy
Senior Member
 
Join Date: May 2018
Posts: 175
Oh, all the way back there. Not sure why that email is being classed as spam if the sender is in the contacts list. Maybe the same kind of X-Spam-Known-Sender examples I show in post 17 so the spam bypass test at the start of section 3 fails and the stuff really has a high enough spam scores.

The code I showed is a more general case for overriding the spam check when the sender is in the contacts list. Senders will always get past the spam check no matter what. Your code isn't going to work since as a organize rule it's too late since the OP's message would still be picked off by the spam test. And if you place your code before the spam check almost all email that's not in the contacts will end up in the inbox.

So to summarize the object of the "sport" was to only bypass the spam check if the sender is in the contacts list leaving everything else the same. That's all. Once past the spam check organize rules can do their thing to filter what they want and what's left will end up in the inbox by default.

Last edited by xyzzy : 9 Sep 2019 at 07:08 PM.
xyzzy is offline   Reply With Quote
Old 9 Sep 2019, 06:51 PM   #25
JamesHenderson
Essential Contributor
 
Join Date: Jan 2003
Location: Oxfordshire, UK
Posts: 467
Quote:
Originally Posted by xyzzy View Post
Not sure why that email is being classed as spam if the sender is in the contacts list. I supposed then X-Spam-Known-Sender is no for some reason.
they spoofed my own email address so the email was:
from: me
to: me

The spam rules caught it correctly, but your script was (of course) bypassing the spam rules because the sender (me) was in my contacts list.

This will be a problem for all spam that uses your own contacts as their from: address (I guess because you also are also in their contacts list when their email was compromised). I get those from time to time.
JamesHenderson is offline   Reply With Quote
Reply


Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump


All times are GMT +9. The time now is 02:39 PM.

 

Copyright EmailDiscussions.com 1998-2013. All Rights Reserved. Privacy Policy