Sieve filtering failure

tenekun · 17 May 2003, 07:36 AM

I just want to know how emails with these subjects:

Quote:

Subject: Prescrition weight loss .................. rudy

Quote:

Subject: Lose Weight Without Dieting..

Quote:

Subject: Government Grants Can Save You! acnz

and

Quote:

Subject: Printer Cartridges - Save Up To 80%!

got past the following filter (clipped for readability):

Code:

if not  header :contains "subject" "secret phrase" {
if header :contains ["subject", "from"]  
["...", "Grants", "...", "save up to", "...", "weight", "..."] {
 reject text: 
  This is the rejection message. 
.
   ;
}
}

--Brandon Siegel

SanjaySheth · 17 May 2003, 11:31 AM

Can you run one of those emails against your sieve script in the Sieve Tester?

That should reveal if a mistake was made.

tenekun · 17 May 2003, 12:28 PM

How does that work exactly? Do I just paste in the headers? I have tried that and I get no output (blank line). If not, how do I include the body?

--Brandon Siegel

SanjaySheth · 17 May 2003, 12:39 PM

View the email in raw mode and copy the entire contents into the bottom box and the entire sieve script into the top box and then hit run.

tenekun · 17 May 2003, 12:42 PM

OK when I paste in the body below the headers, the result is this:

Quote:

Envelope body of 'from'? Envelope body of 'to'? sieve_execute_script() returns -1237848061

This is on the e-mail with the following headers (I edited out the naughty word myself, it didn't come that way):

Code:

 Return-Path: <ojyqg2k38t8@aol.com>
Received: from server3.fastmail.fm (server3.internal [10.202.2.134])
        by server2.fastmail.fm (Cyrus v2.1.9) with LMTP; Fri, 16 May 2003 21:21:02 -0400
X-Sieve: CMU Sieve 2.2
Received: from smtp.us.messagingengine.com (server3.internal [10.202.2.134])
        by server3.fastmail.fm (Cyrus v2.1.9) with LMTP; Fri, 16 May 2003 21:21:01 -0400
Received: from smtp.us.messagingengine.com (localhost [127.0.0.1])
        by server3.messagingengine.com (Postfix) with ESMTP id 8968C78387
        for <bsiegel@fastmail.fm>; Fri, 16 May 2003 21:21:01 -0400 (EDT)
Received: from 127.0.0.1 ([127.0.0.1] helo=smtp.us.messagingengine.com) by messagingengine.com
  with SMTP; Fri, 16 May 2003 21:21:01 -0400
X-Mail-from: ojyqg2k38t8@aol.com
X-Delivered-to: <bsiegel@fastmail.fm>
Received: from beetle.vosn.net (beetle.vosn.net [209.197.236.31])
        by smtp.us.messagingengine.com (Postfix) with ESMTP id DA4F078352
        for <bsiegel@fastmail.fm>; Fri, 16 May 2003 21:21:00 -0400 (EDT)
Received: from herring-d1.cpe.mvllo.al.charter.com ([24.196.3.96])
        by beetle.vosn.net with smtp (Exim 3.36 #1)
        id 19GqNi-0000fs-00
        for brandoo@nukenet.com; Fri, 16 May 2003 18:21:05 -0700
Received: from oav.11euxo.com [138.76.9.42] by HERRING-D1.cpe.mvllo.al.charter.com id a0OxuFHwfw3x for <brandoo@nukenet.com>; Thu, 15 May 2003 11:19:00 +0100
Message-ID: <h02$d3m6$--9-1$0j@6lwgq09afi0.ad>
From: "Mitzi Bliss" <ojyqg2k38t8@aol.com>
To: brandoo@nukenet.com
Subject: Free Membership - Unbelievable XxX Barely Legal Teen F***ing! lloorgqowrdzzrdhs d
Date: Thu, 15 May 03 11:19:00 GMT
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook, Build 10.0.2627
MIME-Version: 1.0
Content-Type: multipart/alternative;
        boundary=".B47.7EBFEA"
X-AntiAbuse: This header was added to track abuse, please include it with any abuse report
X-AntiAbuse: Primary Hostname - beetle.vosn.net
X-AntiAbuse: Original Domain - nukenet.com
X-AntiAbuse: Originator/Caller UID/GID - [0 0] / [0 0]
X-AntiAbuse: Sender Address Domain - aol.com

According to my script, it ought to have rejected it on the basis of both the phrases "xxx" and "teen".

Ideas?

SanjaySheth · 17 May 2003, 12:44 PM

A very very curious result .. that almost seems like the problem is within the sieve parser.

Can you email me your entire sieve script and the entire email at sanjay AT fastmail.fm .. I'll try it via the console and see if there's some problem with the tester or somewhere else.

SanjaySheth · 17 May 2003, 12:47 PM

ok i just ran a quick test with the email you have above... did you include the contents of the email body itself when you ran the test? Otherwise the tester fails. If you don't want to include the actual contents, try just leaving a blank line after the headers and then typing something on the next line like test (that will fool the tester enough to make the difference).

tenekun · 17 May 2003, 12:49 PM

When there is no body, it returns no result (a blank line). When I included the body, I got the response I pasted above.

--Brandon Siegel

SanjaySheth · 17 May 2003, 02:30 PM

It's kind of interesting .. I took your sieve script and worked my way through it, commenting out various sections and testing to see what triggered the error.

I've got your sieve script boiled down to this:

Code:

require ["reject"];

if not false 
{  reject "#1"; }

if true
{ reject "#2";  }
elsif not true 
{  reject "#3"; }

Run the above code through the tester and it will err out against any email sample (I was testing against just the simplest case)

Code:

From: <abc@xyz.com>

body starts here

If you remove the not from either the first or 3rd rules above, the error goes away. It looks like the actual tests themselves are not the reason for the error (which is why i replaced them with true/false). Only the logical order seems to matter.
Very weird.

As for your script,

If you can figure out a way to remove the not from your last rule (the exists test) or the date test rule, the error goes away in the tester. Since the tester uses the same parsing logic as the sieve filter itself, the filter might be producing errors against certain emails (which end up matching various tests in above logical order) and causing an internal sieve parse error. Sieve's default action when it gets an internal error is to just keep the email in the Inbox which is probably what is happening here.

I'm going to send the above test sample to Ken Murchison (one of main honchos for cyrus and sieve) and see what he thinks is the issue.

DrStrabismus · 18 May 2003, 01:15 AM

I'm not sure there is anything wrong here. The interpreter correctly returns an error code on the above because two reject actions is an illegal combination.

Removing the first "not" should fix the problem, as it does.

You say that removing the second "not" also fixes the problem, but I cannot reproduce this on your tester page.

DrStrabismus · 18 May 2003, 01:27 AM

Assuming the original problem is to do multiple rejects.

Either rewrite the logic so this doesn't happen, or put a stop after each reject.

Discard can be overridden by other actions, but reject shouldn't be combined with any other action.

tenekun · 18 May 2003, 02:11 AM

Ahh so it is because it keeps going after hitting one of the reject clauses. Would simply putting "stop;" after each case fix it?

Edit: I did add stops after each reject and it seems to pass in the tester now. Very good. Also I discovered the date condition was too limiting and I made it only search for "03" in the Date header. Now that seems to be OK.

--Brandon Siegel

bitequator · 18 May 2003, 07:03 AM

Quote:

Originally posted by DrStrabismus
Discard can be overridden by other actions, but reject shouldn't be combined with any other action.

Sorry, I haven't read through the prob being discussed in this thread. But just wanted to mention off-topic that from my brief testing earlier I think that the Cyrus/FM implementation treats reject + other actions (in any order) the same as the way the RFC requires for discard + other actions -- namely the reject (or discard) is cancelled. I didn't test multiple rejects though.

Quote:

Originally posted by SanjaySheth
I'm going to send the above test sample to Ken Murchison (one of main honchos for cyrus and sieve) and see what he thinks is the issue.

I knew it! So Murchison's Subaddress Extension WAS done because he's an FM user?

Or maybe it's general Cyrus teehee

DrStrabismus · 18 May 2003, 08:24 PM

Quote:

Originally posted by bitequator
But just wanted to mention off-topic that from my brief testing earlier I think that the Cyrus/FM implementation treats reject + other actions (in any order) the same as the way the RFC requires for discard + other actions -- namely the reject (or discard) is cancelled.

No, the behaviour of reject and discard are very different, see here.

SanjaySheth · 18 May 2003, 08:56 PM

Doc, I think you are right about the problem being that tenekun's code was trying to do 2 or more rejects on the same email. However, sieve is also at fault here since the parser is kicking out in the middle of the parsing cycle and the function is basically dying in the middle (at the very least, it should identify the problem as being multiple rejects and give a syntax error message to that effect)

Ken has emailed me back that he will take a look at the issue on Tuesday when he gets time.

17 May 2003, 12:49 PM	#8
tenekun Member Join Date: Dec 2001 Location: Off the deep end Posts: 87	When there is no body, it returns no result (a blank line). When I included the body, I got the response I pasted above. --Brandon Siegel Last edited by tenekun : 17 May 2003 at 01:02 PM.

17 May 2003, 02:30 PM	#9
SanjaySheth Master of the @ Join Date: Mar 2002 Location: Chicago Posts: 1,079	It's kind of interesting .. I took your sieve script and worked my way through it, commenting out various sections and testing to see what triggered the error. I've got your sieve script boiled down to this: Code: require ["reject"]; if not false { reject "#1"; } if true { reject "#2"; } elsif not true { reject "#3"; } Run the above code through the tester and it will err out against any email sample (I was testing against just the simplest case) Code: From: <abc@xyz.com> body starts here If you remove the not from either the first or 3rd rules above, the error goes away. It looks like the actual tests themselves are not the reason for the error (which is why i replaced them with true/false). Only the logical order seems to matter. Very weird. As for your script, If you can figure out a way to remove the not from your last rule (the exists test) or the date test rule, the error goes away in the tester. Since the tester uses the same parsing logic as the sieve filter itself, the filter might be producing errors against certain emails (which end up matching various tests in above logical order) and causing an internal sieve parse error. Sieve's default action when it gets an internal error is to just keep the email in the Inbox which is probably what is happening here. I'm going to send the above test sample to Ken Murchison (one of main honchos for cyrus and sieve) and see what he thinks is the issue. Last edited by SanjaySheth : 17 May 2003 at 02:33 PM.

18 May 2003, 02:11 AM	#12
tenekun Member Join Date: Dec 2001 Location: Off the deep end Posts: 87	Ahh so it is because it keeps going after hitting one of the reject clauses. Would simply putting "stop;" after each case fix it? Edit: I did add stops after each reject and it seems to pass in the tester now. Very good. Also I discovered the date condition was too limiting and I made it only search for "03" in the Date header. Now that seems to be OK. --Brandon Siegel Last edited by tenekun : 18 May 2003 at 02:25 AM.

17 May 2003, 11:31 AM	#2
SanjaySheth Master of the @ Join Date: Mar 2002 Location: Chicago Posts: 1,079	Can you run one of those emails against your sieve script in the Sieve Tester? That should reveal if a mistake was made.

17 May 2003, 12:28 PM	#3
tenekun Member Join Date: Dec 2001 Location: Off the deep end Posts: 87	How does that work exactly? Do I just paste in the headers? I have tried that and I get no output (blank line). If not, how do I include the body? --Brandon Siegel

17 May 2003, 12:39 PM	#4
SanjaySheth Master of the @ Join Date: Mar 2002 Location: Chicago Posts: 1,079	View the email in raw mode and copy the entire contents into the bottom box and the entire sieve script into the top box and then hit run.

17 May 2003, 12:44 PM	#6
SanjaySheth Master of the @ Join Date: Mar 2002 Location: Chicago Posts: 1,079	A very very curious result .. that almost seems like the problem is within the sieve parser. Can you email me your entire sieve script and the entire email at sanjay AT fastmail.fm .. I'll try it via the console and see if there's some problem with the tester or somewhere else.

17 May 2003, 12:47 PM	#7
SanjaySheth Master of the @ Join Date: Mar 2002 Location: Chicago Posts: 1,079	ok i just ran a quick test with the email you have above... did you include the contents of the email body itself when you ran the test? Otherwise the tester fails. If you don't want to include the actual contents, try just leaving a blank line after the headers and then typing something on the next line like test (that will fool the tester enough to make the difference).

18 May 2003, 01:15 AM	#10
DrStrabismus The "e" in e-mail Join Date: May 2002 Posts: 2,804	I'm not sure there is anything wrong here. The interpreter correctly returns an error code on the above because two reject actions is an illegal combination. Removing the first "not" should fix the problem, as it does. You say that removing the second "not" also fixes the problem, but I cannot reproduce this on your tester page.

18 May 2003, 01:27 AM	#11
DrStrabismus The "e" in e-mail Join Date: May 2002 Posts: 2,804	Assuming the original problem is to do multiple rejects. Either rewrite the logic so this doesn't happen, or put a stop after each reject. Discard can be overridden by other actions, but reject shouldn't be combined with any other action.

18 May 2003, 08:56 PM	#15
SanjaySheth Master of the @ Join Date: Mar 2002 Location: Chicago Posts: 1,079	Doc, I think you are right about the problem being that tenekun's code was trying to do 2 or more rejects on the same email. However, sieve is also at fault here since the parser is kicking out in the middle of the parsing cycle and the function is basically dying in the middle (at the very least, it should identify the problem as being multiple rejects and give a syntax error message to that effect) Ken has emailed me back that he will take a look at the issue on Tuesday when he gets time.