Stop from going to Spam

[JamesHenderson]

Originally Posted by xyzzy

And here's what I see in the received email (sending to myself):

X-Spam-Known-Sender: yes!
Debug: X-Spam-Known-Sender=yes ("Self sent message"); in-addressbook,  self-send

...and if I adapt that script to use my name@domain.tld email address:

### 0.  Whitelist test 
if address :matches "from" "name@domain.tld" {
	deleteheader "X-Spam-Known-Sender";
	addheader "X-Spam-Known-Sender" "yes from sieve Whitelist";	
	if header :matches "X-Spam-Known-Sender" "*" {
		addheader "Debug" "X-Spam-Known-Sender=${1}";
		}
	else {
		addheader "Debug" "Huh?";
		}
	stop;
	}

I then get:

Debug: X-Spam-Known-Sender=no
X-Spam-Known-Sender: yes from sieve Whitelist

Which lines up with your result and shows why it is still going to my junk folder,

Bizarre.

[xyzzy]

Originally Posted by JamesHenderson

...albeit I don't understand how it is possible to test a header that has been deleted.

I wondered about that too. One thought is the original message is cached and tests only look at the cached copy and not the actual message to be filed. Unless there is something "hiding" in a RFC somewhere about this I think this is a bug.

I appreciate you sending the bug report. I did ask my original question of Fastmail support but haven't received an answer yet, If you want to associate your bug report with my query, the ticket ID is: PTN636591

I'll update my ticket with a reference to yours. FWIW my ticket is PTN636757 in case you want to update yours so that both tickets are pointing to each other.

Update:
I looked at the deleteheader/addheader documentation in RFC5293. section 7 (Interaction with Other Sieve Extensions). The following paragraph looks pertinent to this problem (bold emphasis mine).

Tests and actions such as "exists", "header", or "vacation" [VACATION] that examine header fields MUST examine the current state of a header as modified by any actions that have taken place so far.

So that alone would indicate we should expect it to test the updated header unless there's an "escape clause" somewhere (?) that says that only applies to new headers and not already existing ones.

[JamesHenderson]

Originally Posted by xyzzy

FWIW my ticket is PTN636757 in case you want to update yours so that both tickets are pointing to each other.

Done!


...blah blah blah, extra words to make the message long enough to post.

[xyzzy]

Heh, you should see how long some of my posts get in those tickets. It wouldn't be so bad if they didn't make the text portion so narrow. I hate that ticket software. It's not really amiable to posting sieve problems. Leading blanks are removed when you try to show indented code. Asterisks are deleted (so trying to show header :matches "X-Spam-Known-Sender" "*" will remove the *.

By the way I added a small update to my last post.

[JamesHenderson]

Originally Posted by xyzzy

Heh, you should see how long some of my posts get in those tickets. It wouldn't be so bad if they didn't make the text portion so narrow. I hate that ticket software. It's not really amiable to posting sieve problems. Leading blanks are removed when you try to show indented code. Asterisks are deleted (so trying to show header :matches "X-Spam-Known-Sender" "*" will remove the *.

By the way I added a small update to my last post.

cheers.

I really only looked at the documentation for the editheader extension.

[SideshowBob]

Originally Posted by xyzzy

I wonder if there's something in an RFC about this.

As far as tests are concerned the updated headers have to take effect immediately. rfc5293 section 7 - it's a MUST requirement.

[JamesHenderson]

Ricardo from support has got back to me to say they are now looking into it...

[JamesHenderson]

OK, Ricardo from support recognises the problem.

It used to work but got broken when a bug in a different part of sieve was fixed.

They hope to have it fixed pretty soon, but cannot promise a date.

[xyzzy]

That sounds like the same reply he sent to me this morning. But he closed my ticket. So I reopened it with a my reply suggesting they should make this a "highe(er)" priority and not close the ticket until it actually is fixed.

[JamesHenderson]

He closed my ticked this morning gas well. I re-opened it asking that it stay open until fixed. He wrote back that that’s not how the ticketing system works - it’s closed because it is now on their task list.

[xyzzy]

One thing I forgot to mention about all this when changing X-Spam-Known-Sender to "yes" since it had nothing to do with the bug. That is, if you're a purist you might want to add in-addressbook as well to that header. Why? Because at the point in the script where it checks calendar preferences (section 6 - Sieve generated for calendar preferences) it tests "in-addressbook". I guess that's because the original X-Spam-Known-Sender could say "no" but it will always have "in-addressbook" for a known contact.

So below is my chunk of code I've been using for setting X-Spam-Known-Sender and making sure in-addressbook is also set:

if header :matches "X-Spam-Known-Sender" "no*" {    # fix up X-Spam-Known-Sender so FM-generated code
  deleteheader "X-Spam-Known-Sender";               # that checks X-Spam-Known-Sender thinks it is known
  if header :contains "X-Spam-Known-Sender" "in-addressbook" { # set yes & add in-addressbook if missing 
    addheader "X-Spam-Known-Sender" "yes!${1}";     # make this stand out a little (!) if it's displayed
  } else {                                          # also add in-addressbook if not already present
    addheader "X-Spam-Known-Sender" "yes!${1}; in-addressbook!"; # make it stand out a little too (!)
  }
}

In other words I change X-Spam-Known-Sender only where I have to but leave other info that's already there alone.

----

Update
Just got the reply from Ricardo mentioning the same thing about closing tickets. He also said the following:

It was actually pretty easy to determine pretty close to exactly how many people could be affected by this, and I did. At any rate I figured I'd pass it along.

It's definitely an obnoxious regression, and there will be a fix soon, along with updates to the automated tests so we can't break it without noticing again.

I don't know if he included that in your reply or not. At any rate I figured I'd pass it along.

[SideshowBob]

If you set Protection level to custom and uncheck the move and discard boxes you can put your own spam handling code in the block below. This is more flexible than trying to work around the generated code.

For example I've always preferred using a second spam folder for high scoring spam rather than having a discard level.

[xyzzy]

Originally Posted by SideshowBob

If you set Protection level to custom and uncheck the move and discard boxes you can put your own spam handling code in the block below. This is more flexible than trying to work around the generated code.

Ok, I guess. So you are disabling all the tests in section 3 (Sieve generated for spam protection) so that it doesn't matter what X-Spam-Known-Sender is set to? No backscatter to test either?

By the way I do have spam protection set to custom (with move and add score set). To each his own I guess. I have no problems with doing what I am doing (well other than FM causing previously fixed bugs to reappear).

---

A little off topic but a heads up about that "block below", i.e., before "old" rules sieve section 4 (Sieve generated for forwarding rules). If you are doing that using the "old" rules mode you cannot switch to "new" rules mode because that block needs to be empty to do the switch. So you have to delete it, switch to "new" and then manually add it back in. It's funny they don't like stuff in that block to allow the switch. It's in the same position with either rules mode. The only one they should be complaining about is the block before "old" rules section section 6 (Sieve generated for calendar preferences). That block no longer exists in the "new" rules sieve code. Oh well. It's not worth a ticket since I assume the "old" rules sieve code will eventually be replaced with the "new" rules sieve code. What happens your code in that block then I do not know.

[JamesHenderson]

Originally Posted by SideshowBob

If you set Protection level to custom and uncheck the move and discard boxes you can put your own spam handling code in the block below. This is more flexible than trying to work around the generated code.

For example I've always preferred using a second spam folder for high scoring spam rather than having a discard level.

That's good info - thanks!

Right now, I'm happy to use the generated code (providing they fix the bug). All I am trying to do right now is create a whitelist without the addresses being in my contacts. This is for addresses that I don't send to and so do not want to see as an option when composing eg: mailing lists and "do-not-reply" addresses etc.

[JamesHenderson]

Originally Posted by xyzzy

Update
Just got the reply from Ricardo mentioning the same thing about closing tickets. He also said the following:

I don't know if he included that in your reply or not. At any rate I figured I'd pass it along.

He hadn't, so thanks for doing that!