Why marked as spam, although in address book?

[David]

Originally Posted by jhollington

If you own the domain names, is there some reason you can't simply point the MX records directly to FastMail?You'd have to have at least a "Standard" account (under the new plans — I think "Enhanced" was the minimum under the old set of plans) to host your own domain, but it would be a MUCH cleaner way of doing this then relying on forwarding.

Thank you for the reply. I have always used a FULL account, with forwarding. It seems to be working well; when push comes to shove however I will likely have to upgrade.

[jhollington]

Originally Posted by David

Thank you for the reply. I have always used a FULL account, with forwarding. It seems to be working well; when push comes to shove however I will likely have to upgrade.

Well, if you're doing typical forwarding chances are the messages to those addresses are probably getting a higher spam score than they otherwise would, but of course you may not see the effects of this depending on how your rules and spam settings are configured.

[n5bb]

Originally Posted by David

...I have always used a FULL account, with forwarding. It seems to be working well; when push comes to shove however I will likely have to upgrade.

I did not intend to imply that forwarding never works. But a simple old-style forwarding system is likely to cause problems for several reasons:

• SPF: This system insures that the system that connected to the destination to deliver the message was authorized by the domain owner to send messages for that domain. Third party forwarding will always break SPF unless sender rewriting is supported by the forwarding and receiving servers. https://en.m.wikipedia.org/wiki/Sender_Rewriting_Scheme
• DKIM: This uses a cryptographic signature created by the sender. If the portion of the message which was signed is even slightly altered, DKIM will fail. It's not unusual for forwarding systems to cause such failures due to alterations.
• Alignment: The From domain must align with (match) the domain used by SPF and DKIM. Spammers often send messages with good DKIM and SPF, but the requirement for proper alignment won't allow them to spoof another domain name in From.
• DMARC: This is the way a domain can publish a policy specifying how to treat messages which fail both aligned SPF and aligned DKIM.

It's getting harder and harder to guarantee that forwarded messages won't run afoul of those spam prevention techniques. This is especially the case if the message is forwarded multiple times or comes from a mailing list auto forwarder where the originator uses a domain which has implemented a strict DMARC policy. So it's better long term to host your domain with an email provider (such as FastMail) which supports these standards so that incoming and outgoing messages use the best possible SPF, DKIM, and therefore DMARC spam prevention techniques.

Bill

[rabarberski]

A (very) late update.
I finally had some time to look into the details of all the great feedback you have all provided.

Originally Posted by lane

This is going to be fixed, but they are having some trouble doing so. See the writeup on the issue by the Microsoft chap who works with these things: Why does my email from Facebook, that I forward from my outlook.com account, get rejected?
Particularly, follow the comments to see the current status of the fix. I am eagerly awaiting it myself.

We are 1 month later but, according to the comments in the linked Microsoft blog post, no fix available. Unfortunatly.

Originally Posted by lane

Conclusion: you need to either wait out the fix, or do something else:

1. You could set the spam score required to move something to Spam much higher. Downside might be extra spam in your Inbox.
2. You could have Fastmail POP the mail down from Hotmail instead of using forwarding. Downside: you no longer get nearly instant delivery.

I am still in the "wait out a fix" mode. POP-ing is no solution (too much delay being added), raising the spam score will result in too much false positives.

Originally Posted by n5bb

The ugly truth is that forwarding in this manner just isn't compatible with current techniques for spam prevention. So the best solution is to log into the sources of those emails and change the email address they use to your FastMail address. I also suggest that you use a subdomain address for two reasons:
...
So if you owned the email address rab@example.com, you could use linkedin@rab.example.com as a subdomain address. The received message would by default be delivered to rab+linkedin@example.com, so if you had a "LinkedIn" folder it would receive those messages.

I've considered this before. But I like the simplicity of my current workflow where I have one (hotmail) address that I use for all web services that do not need to contact me personally. Switching to the scheme you suggest would require me to "remember" which email address i use for which service. For some it will be easy (like linkedin), for others it might get fuzzy (when the service changes name etc.)

Originally Posted by n5bb

So it's better long term to host your domain with an email provider (such as FastMail) which supports these standards ...

I agree. However, I still like the fact that, when I tell people my email address, they can easily remember or write down the email domain I am using because it is such a common one.

Originally Posted by BritTim

In theory, custom sieve code could be used to resolve the problem completely. It could check for the specific case of "known sender" being ignored for DKIM reasons, and apply a higher threshold for spam in that situation. Bit messy, of course.

Custom sieve seems like a good solution, as I am forwarding from my hotmail account to a specific fastmail alias (something like myspamhotmail@rab.fastmail.com). So I can identify from which hotmail account it was forwared from easily.
I've never used sieve scripts. And I didn't want to switch to it because it used to be that you either used the all sieve solution, or the web gui solution. I do have the feeling this has now changed, and that it is possible to both keep using the GUI for simpel rules (like filing into folders) and extra sieve rules for tweaking.

Question 1: Can anybody confirm that you can now use the GUI + sieve, before I make any breaking changes? (I didn't find anything about this in the the linked Fastmail help )

Question 2:Can anybody provide any clue on how such a sieve entry would look? And would it go under "### 3. Sieve generated for spam protection" before the actual spam score check?

[n5bb]

Originally Posted by rabarberski

...
Question 1: Can anybody confirm that you can now use the GUI + sieve, before I make any breaking changes? (I didn't find anything about this in the the linked Fastmail help )

Question 2:Can anybody provide any clue on how such a sieve entry would look? And would it go under "### 3. Sieve generated for spam protection" before the actual spam score check?

1. Yes, you can log into the current web interface and use the "normal" rules as well as adding custom sieve script (by using Edit custom sieve code at the bottom of the rules screen). You insert your custom sieve between sieve sections automatically created from the GUI rules (and rules automatically created by the spam filtering and other Fastmail features). So you can't actually modify what is automatically created, but you can cause the automatic sieve to be ignored. I will include some examples below.
2. No, you can't insert code to bypass the automatic spam filter after the filter (section ### 3).
   • You will note that after filing apparently undesirable messages into the Junk (Spam) folder, the automatic spam sieve code includes a stop statement. When a stop statement is reached all sieve processing stops, so after a message is caught by the spam filter it's all over. And, as I described above, all of the sieve you add is between the automatic rules and you can't can't change the automatically generated code.
   • But you can add sieve code before ##1 so that the discard code (in section ### 2) and spam filtering code (in section ### 3) is bypassed for certain messages. For example, you could insert the following into the top user sieve block (before ### 1):
     

     if not header :contains "X-Delivered-To" ["myspamhotmail@rab.fastmail.com"] {

     Then you need to add a closing brace at the start of the user sieve area after ### 3:
     

     }

     That should cause all of the discard and spam rules to be bypassed if a message arrives for that special delivery address. I just tested this (with a discard rule) and it works as planned.

If you want to cause more than one delivery address to bypass the discard and spam rules, add them inside square brackets and separated with commas as shown here:

if not header :contains "X-Delivered-To" ["myspamhotmail@rab.fastmail.com","myotherspamhotmail@rab.fastmail.com"] {

Bill

[rabarberski]

Bill, thank you for you very clear and detailed explanation (as always).
Very much appreciated!

I would have expected something like below before ###1, (i.e., remove the "not" and add a "stop")

if header :contains "X-Delivered-To" ["myspamhotmail@rab.fastmail.com"] {
    stop;
}

But your solution cleverly allows to still use the forwarding/vacation/organizing rules for that address.

[glass]

Another option is to ignore DMARC failure. I did this by effectively bumping up the spam score threshold for mail with DMARC failures so it counteracts the additional spam points added by the DMARC rules. An example of such a sieve rule is here: http://emaildiscussions.com/showpost...2&postcount=12

As a fun test, look through your spam folder and see if it contains any mail that 1) failed DMARC, and 2) would have made it into your inbox if DMARC rules weren't being enforced.

Since Fastmail started enforcing DMARC, I have received exactly 0 such emails, but every message I have received that has failed DMARC and would have otherwise had a spam score lower than 5 has been a legitimate email. In other words, for me at least, spam classifying based on DMARC failure has a false-positive rate of 100% while preventing 0% of additional spam.

The only thing DMARC enforcement detects is incorrect DMARC policies -- and there are many, many, domains with such policies. Too many for DMARC to be useful.

[rabarberski]

@glass: great suggestion. Did it just now (copy/paste), and will see if it helps in the next days.

[rabarberski]

Some quick feedback: seems to work fine so far.


Another question, as I was reading the Fastmail blog post (https://blog.fastmail.com/2016/12/01...ear-in-review/) where they mention Postbox.

I understand that basically email forward becomes more and more "broken" with all the email security features that are grudaually added (SPF, DKIM, DMARC...)
As pobox is at its heart a forwarding service, does it deal with it differently that Fastmail?

(although I don't know in how much the pobox backend architecture is still different from Fastmail's backend)

[rabarberski]

Final update: this is really working well.

Would be nice if Fastmail would allow to turn on/off DMARC with a simple checkbox.
I would assume quite some of their users use forwarding in one way or another, and would like to have that sort of control (rather than doing the sieve hacks)

[Mugwhamp]

Originally Posted by lane

Conclusion: you need to either wait out the fix, or do something else:

1. You could have Fastmail POP the mail down from Hotmail instead of using forwarding. Downside: you no longer get nearly instant delivery.

Are you implying that POP'd email is properly scanned in a way that forwarded mail is not? This interests me. Why would that be the case? Sorry for my ignorance of backend matters.

[David]

Forwarded mail is always faster than Popped mail. I use POP only when forwarding is not available.

[Mugwhamp]

Originally Posted by David

Forwarded mail is always faster than Popped mail. I use POP only when forwarding is not available.

Understood, but my question doesn't relate to speed, it concerns spam scanning. Is POP3 email scanned by FM's engine, while email forwarded to FM is not? That seems to be the implication in the quotation above.

[David]

Originally Posted by Mugwhamp

Understood, but my question doesn't relate to speed, it concerns spam scanning. Is POP3 email scanned by FM's engine, while email forwarded to FM is not? That seems to be the implication in the quotation above.

I apologise for my knee jerk reply, Mugwhamp. I suspect that that best answer to your question will come from Fastmail support.

[lane]

Originally Posted by Mugwhamp

Understood, but my question doesn't relate to speed, it concerns spam scanning. Is POP3 email scanned by FM's engine, while email forwarded to FM is not? That seems to be the implication in the quotation above.

Yes, I think the best answer as to how spam scanning is done for POP'ed email, should be answered by Fastmail support. My tests seem to indicate, however, that:

1. Normal SpamAssassin scanning is performed and spam/nonspam is processed based on the scoring as usual.
2. The special checks for DMARC which add 8 or 15 points to the spam score when DMARC fails, are not done. This is because the checks for valid SPF and DKIM are not performed for POP'ed email, and DMARC depends on those.
3. The check for SPF cannot be made in any case since there is no "sending server" to Fastmail, and it would not make any sense for Fastmail to use the server it POP's from (which would normally fail anyway).
4. I think a DKIM check could theoretically be made at the time of POP'ing the email, but Fastmail does not do it. It would anyway lead to complications if you POP'ed old emails from an account (perhaps as a bulk transfer), because encryption keys might change over time.

Forwarded email is always scanned, including DMARC.