Personality "Save Sent" & "Bcc" via SMTP

robmueller · 2 Jan 2007, 01:35 PM

If you login to the beta server (http://www.fastmail.fm/beta/), and go to Options -> Personalities, you'll see two new checkboxes:

BCC on SMTP - Send to Bcc addresses when sending via email software
Sent Items on SMTP - Save to folder when sending via email software

These attempt to do the same thing as the Personality would do when sending from the web interface, but when you send via SMTP.

The reasons people might want this:
1. Bcc all emails to another host for backups
2. Avoid the double upload when sending (eg SMTP + IMAP upload)
3. Actually allow save to Sent Items sanely (eg Outlook)

A couple of things to note:
1. The actual saving to Sent Items is done by sending another copy of the message to the user and then catching and filing it via a special rule added to the start of the users script, so if you use custom advanced rules, you need to add the special rule to your script as well
2. We match the personality to use by looking at the "From" header in the email, and looking for a personality with the same email address
3. If we find multiple personalities with the same email address, we try and match the "name" part of the email address with the "Full name" of the personality. If matches are found, we use them, if not, we use the list from step 2 still
4. If still multiple matches, prefer the "Default" personality if in the list, otherwise use a random one since we can't guess a "best" one

Cheers

Rob

Edward Velo · 2 Jan 2007, 05:37 PM

Thanks, Rob! I really wanted this feature ( http://www.emaildiscussions.com/show...threadid=44019 ). And together with the recently implemented "web only" domain hosting, this is great news!

PS. Maybe the text could be changed to "Also send to BCC addresses when sending via email software" because as it is now (without the "also"), it could be interpreted as (when checked) not sending to BCC addresses when sending via the webinterface? Or maybe two checkboxes: one for using BCC via the webinterface, and one for using BCC via SMTP?

hadaso · 2 Jan 2007, 05:47 PM

Quote:

Originally posted by robmueller
1. The actual saving to Sent Items is done by sending another copy of the message to the user and then catching and filing it via a special rule added to the start of the users script, so if you use custom advanced rules, you need to add the special rule to your script as well

Is there any particular header/headers added to the copy being delivered to the user to make this rule work? Perhaps you can post sample Sieve here?

robmueller · 2 Jan 2007, 06:07 PM

Quote:

Originally posted by hadaso
Is there any particular header/headers added to the copy being delivered to the user to make this rule work? Perhaps you can post sample Sieve here?

It's implemented by just sending another copy of the message to a magic address:

username+personalitysentitem-${personalityid}@domain.tld

The sieve rule added is just at the start of the script and is of the form:

Code:

if header :contains ["x-resolved-to"] "[email protected]" {
  setflag "\\Seen";
  fileinto "INBOX.Sent Items";
  stop;
  }

The extra bcc'ing is done internally in a similar way, but instead of adding the above extra magic recipient, it just adds the extra bcc addresses to the recipients list.

Rob

2 Jan 2007, 10:30 PM

Thanks, it's useful. I'm using the Web interface almost exclusively, but I enabled this anyway

(I'll probably get a bunch of sent messages in my inbox next time I use a client, though, until I remember to add that magic rule to my script.)

davidbstanley · 2 Jan 2007, 10:38 PM

I have been doing this for sometime now, by BCC'ing in the client and with some sieve code to move [email protected] to the sent folder. Today it has stopped working and the BCC's appear in the inbox. I guess it is related. Any ideas? I know the new FM system will be better, i just wonder why my solution has stopped working.

frankd1 · 3 Jan 2007, 05:38 AM

Is there any way to sent up an "account-wide" save sent option? I personally want every outgoing message saved to the sent folder, and it would be nice to not have the added step of going into the personalities of new aliases to check the save box.

robmueller · 3 Jan 2007, 06:15 AM

Quote:

Originally posted by davidbstanley
I have been doing this for sometime now, by BCC'ing in the client and with some sieve code to move [email protected] to the sent folder. Today it has stopped working and the BCC's appear in the inbox. I guess it is related. Any ideas? I know the new FM system will be better, i just wonder why my solution has stopped working.

Oh, I might know what's causing this. Previously when you sent a message via SMTP to a local FM recipient, it was going via the local delivery path, while email for outgoing hosts was sent via the outgoing path. To make this new feature work, I had to make them both go via the outgoing path, but thinking about this, the X-Delivered-to header will actually be the same as X-Resolved-to for email sent via authenticated SMTP to a FastMail account. Can you check that in the header of the email? I'll look into fixing this.

Rob

**Sherry** · 3 Jan 2007, 08:47 AM

Finally got it to partly work after figuring out what email address to put in the personality. Anyway to get it to use a wild card address like *@mydomain so any email sent using OE and my personal domain would work? As it is now I think I have to make a Personality for every "Dummy" address setup in OE for sending email "From". I might get by with the 8 Personalities I get but the wild card from would be great or frankd1's question for an ""account-wide" setup?

Oh, when I got it working I wound up with 3 copies of the bcc address. 1 in the local Sent Items (which I expected) and 1 in the FM IMAP Sent Items (which I hoped for) but I also got a third copy in the IMAP Inbox...

Sherry

Edit: Forgot to mention that the "mydomain" is a domain that is forwarded to FM from GoDaddy, not setup in FM with the MX record stuff.

robmueller · 3 Jan 2007, 09:12 AM

Have you had a look at the headers of the email that ended up in your inbox? The Received: headers should be able to show you the path it took, that might help work out what happened.

Rob

**Sherry** · 3 Jan 2007, 10:10 AM

I'm not sure what to look for in the received part of the header.

This is from the Inbox. Starting from the bottom I'll show parts of it (but maybe not enough?)

This is the bottom one: (basicaly showing it's from my pc)
Received: from "removed" (removed[removed])
by mail.messagingengine.com (Postfix)

Received: from heartbeat1.messagingengine.com ([10.202.2.160])
by out1.internal (MEProxy)

Received: from out1.internal (unknown [10.202.2.149])
by out1.messagingengine.com (Postfix)

This is the final one after all the X-stuff:
Received: from out1.internal (out1.internal [10.202.2.134])
by store13m.internal

Sherry

n5bb · 3 Jan 2007, 12:10 PM

Rob, I think some more work needs to be done on this feature. It seems to be confusing for simple webmail use. Here is what I did, and what resulted on the beta server:

I reverted to my default Sieve script. The system sent me an email with my customized script! Thanks for this neat new feature -- it must have been added in the last month or so.
In Personalities, I set my default personality to use the Sent Items folder for sent items. I also checked "Also save to folder when sending via email software".
I composed a short message to an external address, using my default personality. I left "Save Copy" set to the default Personality value.
After sending this message, I checked the Sent Items folder.
Two messages were in Sent Items. One appeared to be directly sent (in the original manner), and the other was filed via the Sieve rule set up by the Personality selection.
If I change the "Save Copy" field on the Compose screen to None, then I only get one message in Sent Items (filed via the Sieve script).

If the webmail system did not generate the additional message to be saved, the original save-to-folder features would work fine. I often use this save-to-folder-on-compose feature, so please don't remove it! I guess you could change the default save-to-folder setting to "None" -- then the message would still get saved to the folder specified in the Personality (by the Sieve script), but the second copy would not be generated.

On new features such as this one (and the backscatter filter) which cause existing Sieve custom scripts to break or fail to do what the menus say, you need to clearly warn existing custom Sieve users about these effects and how to rebuild a corrected Sieve script. In this case, it's too bad there isn't a feature which would insert certain option menu script changes at the top (after the "require" line) of an existing customized Sieve script. Ideally the user would be given a choice in a menu of whether to automatically insert certain sections of automatically generated code, such as backscatter filtering, personality filing (this thread), basic whitelist/spam filtering/filing, etc. Then the advanced user could keep a customized filter, but get the benefits of these newer features tied in with various menus.

Bill

robmueller · 3 Jan 2007, 01:04 PM

Ah yes, of course, when you send via the web interface it double triggers (standard save to Sent Items via IMAP + the second save via sieve). I'll look into that.

Custom sieve scripts are complex to deal with, and I don't want to go inserting bits and pieces of code in them, it's bound to cause problems.

As it is, if you do something that makes a change to your sieve script and have custom rules, it appends the "generated" sieve script to the end of your custom sieve script with a header line:

# ===BEGIN-GENERATED=== Text between markers will be automatically replaced

What should probably happen, is that you get a warning message at the top of the screen about how your rules needed to be regenerated, but you have a custom sieve script, so only the commented out version of the script was updated. That would at least warn people when they made a change (eg Spam virus/protection screen, Personalities screen, Pop Links screen) that required rules changes that couldn't actually be done.

Hmmm, you know what, the Spam/virus screen and Pop Links screen already do this. I've factored out the same code and made the Personalities screen use it as well now. So if you have a custom sieve script, and click Done on the personalities screen you get:

Generated script has been commented and appended to your custom sieve script, which you have previously modified by hand.

Rob

robmueller · 3 Jan 2007, 01:55 PM

Ok, the problem of sending via the web interface causing "double" items to appear in the Sent Items folder or Bcc'd should be fixed now as well.

Rob

davidbstanley · 3 Jan 2007, 03:20 PM

Quote:

Originally posted by robmueller
Oh, I might know what's causing this. Previously when you sent a message via SMTP to a local FM recipient, it was going via the local delivery path, while email for outgoing hosts was sent via the outgoing path. To make this new feature work, I had to make them both go via the outgoing path, but thinking about this, the X-Delivered-to header will actually be the same as X-Resolved-to for email sent via authenticated SMTP to a FastMail account. Can you check that in the header of the email? I'll look into fixing this.

Rob

Yes, the problem is caused by the X-Delivered-to header being set to the same as the X-Resolved-to header. Will this be resolved? I am sure I am not the only one doing this.

2 Jan 2007, 01:35 PM	#1
robmueller Intergalactic Postmaster Join Date: Oct 2001 Location: Melbourne, Australia Posts: 6,102 Representative of: Fastmail.FM	Personality "Save Sent" & "Bcc" via SMTP If you login to the beta server (http://www.fastmail.fm/beta/), and go to Options -> Personalities, you'll see two new checkboxes: BCC on SMTP - Send to Bcc addresses when sending via email software Sent Items on SMTP - Save to folder when sending via email software These attempt to do the same thing as the Personality would do when sending from the web interface, but when you send via SMTP. The reasons people might want this: 1. Bcc all emails to another host for backups 2. Avoid the double upload when sending (eg SMTP + IMAP upload) 3. Actually allow save to Sent Items sanely (eg Outlook) A couple of things to note: 1. The actual saving to Sent Items is done by sending another copy of the message to the user and then catching and filing it via a special rule added to the start of the users script, so if you use custom advanced rules, you need to add the special rule to your script as well 2. We match the personality to use by looking at the "From" header in the email, and looking for a personality with the same email address 3. If we find multiple personalities with the same email address, we try and match the "name" part of the email address with the "Full name" of the personality. If matches are found, we use them, if not, we use the list from step 2 still 4. If still multiple matches, prefer the "Default" personality if in the list, otherwise use a random one since we can't guess a "best" one Cheers Rob

2 Jan 2007, 05:37 PM	#2
Edward Velo Essential Contributor Join Date: Jun 2004 Location: Brussels, Belgium Posts: 424	Thanks, Rob! I really wanted this feature ( http://www.emaildiscussions.com/show...threadid=44019 ). And together with the recently implemented "web only" domain hosting, this is great news! PS. Maybe the text could be changed to "Also send to BCC addresses when sending via email software" because as it is now (without the "also"), it could be interpreted as (when checked) not sending to BCC addresses when sending via the webinterface? Or maybe two checkboxes: one for using BCC via the webinterface, and one for using BCC via SMTP? Last edited by Edward Velo : 2 Jan 2007 at 05:45 PM.

3 Jan 2007, 08:47 AM	#9
Sherry Moderator Join Date: Dec 2002 Location: USA Posts: 8,686	Finally got it to partly work after figuring out what email address to put in the personality. Anyway to get it to use a wild card address like @mydomain so any email sent using OE and my personal domain would work? As it is now I think I have to make a Personality for every "Dummy" address setup in OE for sending email "From". I might get by with the 8 Personalities I get but the wild card from would be great or frankd1's question for an ""account-wide" setup? Oh, when I got it working I wound up with 3 copies of the bcc address. 1 in the local Sent Items (which I expected) and 1 in the FM IMAP Sent Items (which I hoped for) but I also got a third copy in the IMAP Inbox... Sherry Edit: Forgot to mention that the "mydomain" is a domain that is forwarded to FM from GoDaddy, not setup in FM with the MX record stuff. Last edited by Sherry : 3 Jan 2007 at 08:53 AM.*

3 Jan 2007, 12:10 PM	#12
n5bb Intergalactic Postmaster Join Date: May 2004 Location: Irving, Texas Posts: 8,586	Extra copy placed in Sent Items when using webmail Rob, I think some more work needs to be done on this feature. It seems to be confusing for simple webmail use. Here is what I did, and what resulted on the beta server: I reverted to my default Sieve script. *The system sent me an email with my customized script! Thanks for this neat new feature -- it must have been added in the last month or so.* In Personalities, I set my default personality to use the Sent Items folder for sent items. I also checked "Also save to folder when sending via email software". I composed a short message to an external address, using my default personality. I left "Save Copy" set to the default Personality value. After sending this message, I checked the Sent Items folder. Two messages were in Sent Items. One appeared to be directly sent (in the original manner), and the other was filed via the Sieve rule set up by the Personality selection. If I change the "Save Copy" field on the Compose screen to None, then I only get one message in Sent Items (filed via the Sieve script). If the webmail system did not generate the additional message to be saved, the original save-to-folder features would work fine. I often use this save-to-folder-on-compose feature, so please don't remove it! I guess you could change the default save-to-folder setting to "None" -- then the message would still get saved to the folder specified in the Personality (by the Sieve script), but the second copy would not be generated. On new features such as this one (and the backscatter filter) which cause existing Sieve custom scripts to break or fail to do what the menus say, you need to clearly warn existing custom Sieve users about these effects and how to rebuild a corrected Sieve script. In this case, it's too bad there isn't a feature which would insert certain option menu script changes at the top (after the "require" line) of an existing customized Sieve script. Ideally the user would be given a choice in a menu of whether to automatically insert certain sections of automatically generated code, such as backscatter filtering, personality filing (this thread), basic whitelist/spam filtering/filing, etc. Then the advanced user could keep a customized filter, but get the benefits of these newer features tied in with various menus. Bill

2 Jan 2007, 10:30 PM	#5
Daniel S Guest Posts: n/a	Thanks, it's useful. I'm using the Web interface almost exclusively, but I enabled this anyway (I'll probably get a bunch of sent messages in my inbox next time I use a client, though, until I remember to add that magic rule to my script.)

2 Jan 2007, 10:38 PM	#6
davidbstanley Cornerstone of the Community Join Date: Aug 2002 Location: Kent, UK Posts: 691	I have been doing this for sometime now, by BCC'ing in the client and with some sieve code to move [email protected] to the sent folder. Today it has stopped working and the BCC's appear in the inbox. I guess it is related. Any ideas? I know the new FM system will be better, i just wonder why my solution has stopped working.

3 Jan 2007, 05:38 AM	#7
frankd1 Junior Member Join Date: Mar 2006 Posts: 23	Is there any way to sent up an "account-wide" save sent option? I personally want every outgoing message saved to the sent folder, and it would be nice to not have the added step of going into the personalities of new aliases to check the save box.

3 Jan 2007, 09:12 AM	#10
robmueller Intergalactic Postmaster Join Date: Oct 2001 Location: Melbourne, Australia Posts: 6,102 Representative of: Fastmail.FM	Have you had a look at the headers of the email that ended up in your inbox? The Received: headers should be able to show you the path it took, that might help work out what happened. Rob

3 Jan 2007, 10:10 AM	#11
Sherry Moderator Join Date: Dec 2002 Location: USA Posts: 8,686	I'm not sure what to look for in the received part of the header. This is from the Inbox. Starting from the bottom I'll show parts of it (but maybe not enough?) This is the bottom one: (basicaly showing it's from my pc) Received: from "removed" (removed[removed]) by mail.messagingengine.com (Postfix) Received: from heartbeat1.messagingengine.com ([10.202.2.160]) by out1.internal (MEProxy) Received: from out1.internal (unknown [10.202.2.149]) by out1.messagingengine.com (Postfix) This is the final one after all the X-stuff: Received: from out1.internal (out1.internal [10.202.2.134]) by store13m.internal Sherry

3 Jan 2007, 01:04 PM	#13
robmueller Intergalactic Postmaster Join Date: Oct 2001 Location: Melbourne, Australia Posts: 6,102 Representative of: Fastmail.FM	Ah yes, of course, when you send via the web interface it double triggers (standard save to Sent Items via IMAP + the second save via sieve). I'll look into that. Custom sieve scripts are complex to deal with, and I don't want to go inserting bits and pieces of code in them, it's bound to cause problems. As it is, if you do something that makes a change to your sieve script and have custom rules, it appends the "generated" sieve script to the end of your custom sieve script with a header line: # ===BEGIN-GENERATED=== Text between markers will be automatically replaced What should probably happen, is that you get a warning message at the top of the screen about how your rules needed to be regenerated, but you have a custom sieve script, so only the commented out version of the script was updated. That would at least warn people when they made a change (eg Spam virus/protection screen, Personalities screen, Pop Links screen) that required rules changes that couldn't actually be done. Hmmm, you know what, the Spam/virus screen and Pop Links screen already do this. I've factored out the same code and made the Personalities screen use it as well now. So if you have a custom sieve script, and click Done on the personalities screen you get: Generated script has been commented and appended to your custom sieve script, which you have previously modified by hand. Rob

3 Jan 2007, 01:55 PM	#14
robmueller Intergalactic Postmaster Join Date: Oct 2001 Location: Melbourne, Australia Posts: 6,102 Representative of: Fastmail.FM	Ok, the problem of sending via the web interface causing "double" items to appear in the Sent Items folder or Bcc'd should be fixed now as well. Rob