Sieve notify script -- two questions

[Nafri]

I have made a very basic and simple sieve script to send a summary notification (without the body text) to my_target_email_address whenever an email is received from any xyz.com email address.

if address :matches "From" "*xyz.com" {
	notify :method "mailto" :options ["my_target_email_address", "From", "Orig"] :message "$from$ / [Notify from $from$] $subject$ / ";
}

So an email sent by abc@xyz.com with the subject line "Abracadabra" results in this notification:

From: abc@xyz.com

Subject: (*1) [Notify from abc@xyz.com] Abracadabra


I have two questions.

1. Why does the subject line start with "(*1)"? Is there any way to suppress it?

2. Since I am looking for all emails from a certain domain, which of the following criteria is efficient (or preferred)?

if address :matches "From" "*xyz.com"

if address :domain :is "From" "xyz.com"

[BritTim]

On your second question, best of all is probably

if address :domain :is "From" "xyz.com"

but the practical difference is slight. Note that the From header (tested here) and the envelope sender, although most often the same in most situations, are not guaranteed to be the same. Bear this in mind if it could be important to know that the real sender is from that domain.

[somdcomputerguy]

Originally Posted by Nafri

1. Why does the subject line start with "(*1)"? Is there any way to suppress it?

That probably means there is one unread email in your box. I don't know how to suppress it. Maybe by creating a rule (or adding to the specific one you've already created) to mark the email as read (red )?

-bruce

[Nafri]

Originally Posted by BritTim

On your second question, best of all is probably

if address :domain :is "From" "xyz.com"

but the practical difference is slight.

I guess that both scripts would take a fraction of a millisecond each to execute with marginal performance differentials. In terms of code clarity, I would also prefer address :domain :is "From" "xyz.com".

Note that the From header (tested here) and the envelope sender, although most often the same in most situations, are not guaranteed to be the same. Bear this in mind if it could be important to know that the real sender is from that domain.

Will keep this in mind. Thank you.

[Nafri]

Originally Posted by somdcomputerguy

That probably means there is one unread email in your box. I don't know how to suppress it.

Bruce, this can't be the case as I have just tested the sieve with several unread emails present in both inboxes (source and target). I am sure that the "(*1)" subject prefix is generated by the sieve processor for some specific -- and, to me, unknown -- reason.

[BritTim]

Originally Posted by Nafri

Bruce, this can't be the case as I have just tested the sieve with several unread emails present in both inboxes (source and target). I am sure that the "(*1)" subject prefix is generated by the sieve processor for some specific -- and, to me, unknown -- reason.

I am reasonably sure you have found a parsing bug in Fastmail's Notify sieve extension.

Referring to the documentation at https://www.fastmail.com/help/techni...ve-notify.html, the Subject line is supposed to be <from> <subject>

• The subject will be of the form "[Notify from <from>] <subject>", where <from> is optionally compressed (see the "Compress" option, above) and the entire subject is cropped to SubjectLength characters.
• If <from> is a full email address containing a "Name" part, only the "Name" part will be included.

It looks as though the specific options you chose cause the substitution for <from> either to be omitted (my guess) or to process incorrectly.

You could report this, but I would expect you to have a fairly long wait for a fix. Sieve support is not a priority for Fastmail these days. More promising, I think, is to fiddle with the options (such as by explicitly specifying a default, for instance) to see if that causes the parsing error to go away.

[n5bb]

The subject starts with (*1) because Fastmail uses this as an email loop indicator for such messages. As I say in this post:

(*1) -- This is the loop detector. If you redirect the notification back so that a message forwarding loop is created, the value of this initial text on the subject line will increment to (*2), (*3), (*4), (*5), then the next notification loop will fail (with a warning message sent to your account).

Fastmail has added several tweaks to the notification feature in the past for some users, and it's a mess. The Fastmail staff told me a couple of years ago that they intend to clean up the Sieve modified notify code they use, but this keeps getting delayed, since it doesn't seem very important compared to other needs.

Bill

[somdcomputerguy]

Originally Posted by n5bb

The subject starts with (*1) because Fastmail uses this as an email loop indicator for such messages. As I say in this post:

Thanks for clarifying that, Bill.

-bruce

[Nafri]

Originally Posted by n5bb

The subject starts with (*1) because Fastmail uses this as an email loop indicator for such messages. As I say in this post:

Thank you, Bill. The post that you referred to was extremely useful for me in learning more about sieve scripting.

During the course of experimentation a couple of days ago, I used to get notification emails without this (*1) thing. I just tried to replicate the earlier results and here are the scripts to suppress this dreaded prefix at the expense of not getting the original email address in the From field. This was achieved by removing , "From", "Orig" from : options.

As earlier, it is assumed that an email is sent from abc@xyz .com to my_fastmail_address with the subject line "Abracadabra".

if address :matches "From" "*xyz.com" {
	notify :method "mailto" :options ["my_target_email_address"] :message "$from$ / [Notify from $from$] $subject$ / ";
}

This resulted in the following notification email:
From: my_fastmail_address
Subject: (from abc@xyz .com) [Notify from abc@xyz .com] Abracadabra

Note the sieve processor has added (from abc@xyz .com).

The second script gets rid of $from$ from :message.

if address :matches "From" "*xyz.com" {
	notify :method "mailto" :options ["my_target_email_address"] :message " / [Notify from $from$] $subject$ / ";
}

And this one resulted in the following notification email:
From: my_fastmail_address
Subject: [Notify from abc@xyz .com] Abracadabra

[Nafri]

Please correct me if I am wrong but I think that the description given on fastmail's help page entitled Sieve Notify extension is at odds with the implementation of sieve notify extension. Some examples:

- For :message <text>, the help page says that the <text> can be one of these two:

"<from> / <subject> / <body>" — This will be split apart and used to create the message subject and body.
The subject will be of the form "[Notify from <from>] <subject>"
[...]

and

"<string>" — This will generate a message with the subject of the form "[Notify] <string>"

I wasn't able to get any string in the subject line as mentioned in the second option. The string instead appears in the message body.

The documentation for the first option reads like as if the sieve processor will itself insert "Notify from". Actually it will gladly insert whatever text one provides through a string in the <subject> part of <text>.

- On the notification loop checking, the documentation says the following:

This works by looking for a [Notify …] section in the message subject and replacing it with [Notify*2 ...], and so on, until it reaches *5, at which point an error is generated and the notification is cancelled.
However, since this depends on the subject of the notification message, you still run the risk of loops if you use a custom <text> string. You should exercise caution if you use a custom <text> string to avoid this possibility.

As mentioned earlier in this thread, the sieve processor generates (*1) only if "From", "Orig" (or "From", "Full") are present in : options. Secondly, I believe that the "default" <text> (as opposed to the custom <text> ) would be "$from$ / $subject$ / $text$". This doesn't insert the Notify word.

[n5bb]

I am again asking Fastmail staff about their plans to rework the Sieve notify feature.

Bill

[brong]

Originally Posted by n5bb

I am again asking Fastmail staff about their plans to rework the Sieve notify feature.

Bill

Thanks Bill.

We have a major project working on the Cyrus server for the next 2 months, because Robert S is coming over from Austria to spend October and November co-working with us in the FastMail offices and concentrating on JMAP work.

We're also planning to add the ability to re-process messages through sieve again - in particular to handle the use-case where a message was delivered to Spam but is actually legitimate, so the "report Not-Spam" will cause it to be delivered to the correct folder rather than dumped in Inbox.

Support for RFC 5435 (aka "enotify", rather than the pre-standard "notify" that we currently have) is not an immediate priority, largely because we'd have to rewrite all existing scripts! Annoyingly the earlier draft which our server is based on uses the same keyword, so you can't mix the two styles together!

I do want to go through cyrus-imapd and find all the draft standards that we're implementing where there's an actual standard, but I can't bump that up above all the other really important work. I'm in the middle of a massive redesign of some internals to allow greater than 32bit cache sizes and label support on messages for JMAP. That's going to have to take priority! Sorry.

Regarding the (*1) we need a way to catch mail loops through external systems which strip headers, which is why we're modifying the subject. Mail forwarding and automated creation is fraught, and mail loops suck They tend to be the things that wake us up with a middle of the night page and everything stuck for many other users - it's hard to justify changing it.

[n5bb]

We appreciate the update, Bron.

Mail loops can be horrible! A few months someone at my company did a reply-all to an automated IT "we will respond later when we have it figured out" message, and they added a large group of the CC list. I'm not sure if it was "out of office" auto-responses or something else which triggered the problem, but every few minutes the IT auto-responder sent out another blast to many dozens of recipients, and each message had an additional "user response" tacked to the bottom. I guess the IT auto-responder was adding it's own response to the user comments. This happened on a weekend or overnight, and the number of auto-replies was also increasing. So one auto-reply turned into two longer auto-replies then three even longer auto-replies. By the time a human IT person saw the mess, our email inboxes were filled with pages of increasingly long junk messages.

By the way, Rob M had told me several years ago (2010 or earlier, I think) that tweaks had been made to the Sieve notify feature for some customers. The side effect of these (and the vague old Sieve notify spectication) has led to requiring users to experiment to discover the secret tricks. My suggestion is to throw everything away and start over from scratch with something which meets modern RFC's.

Bill

[BritTim]

Originally Posted by brong

Thanks Bill.

We have a major project working on the Cyrus server for the next 2 months, because Robert S is coming over from Austria to spend October and November co-working with us in the FastMail offices and concentrating on JMAP work.

We're also planning to add the ability to re-process messages through sieve again - in particular to handle the use-case where a message was delivered to Spam but is actually legitimate, so the "report Not-Spam" will cause it to be delivered to the correct folder rather than dumped in Inbox.

Support for RFC 5435 (aka "enotify", rather than the pre-standard "notify" that we currently have) is not an immediate priority, largely because we'd have to rewrite all existing scripts! Annoyingly the earlier draft which our server is based on uses the same keyword, so you can't mix the two styles together!

I do want to go through cyrus-imapd and find all the draft standards that we're implementing where there's an actual standard, but I can't bump that up above all the other really important work. I'm in the middle of a massive redesign of some internals to allow greater than 32bit cache sizes and label support on messages for JMAP. That's going to have to take priority! Sorry.

Regarding the (*1) we need a way to catch mail loops through external systems which strip headers, which is why we're modifying the subject. Mail forwarding and automated creation is fraught, and mail loops suck They tend to be the things that wake us up with a middle of the night page and everything stuck for many other users - it's hard to justify changing it.

Hi Brong ... great to see you are still around!

I appreciate why the notify header cleanup is down the priority list. To be honest, my priority (if sieve was going to get attention) would be editheader. However, purely academic, rather than changing the Subject header, for loop detection, would it not make more sense just to use a separate header like x-loop-detect or something?

[n5bb]

Originally Posted by BritTim

... However, purely academic, rather than changing the Subject header, for loop detection, would it not make more sense just to use a separate header like x-loop-detect or something?

I agree that headers would be a better method in a perfect world, but as Bron pointed out some systems may strip off headers. The worst situation is when two auto-responders get in a loop with each other (or, as I described, an auto-reponder loops with itself) in a manner where the subject (and useful headers) of each auto-response never changes. About all that FastMail can do in that situation is to put a rate limit on how many messages to allow every hour.

Bill