Warning: incorrect result from search after:

[chrisjj]

Search after:<date> erroneously includes messages on <date>!

Seemingly search has mistaken after:<date> to mean after:<date and time 00:00:00.000...0>.

FM Support says the team "suggested that this behaviour is expected"

[BritTim]

In my view, this behaviour is acceptable, but should be documented.

[chrisjj]

Originally Posted by BritTim

In my view, this behaviour is acceptable, but should be documented.

Interesting. Why do you think it is acceptable?

[JeremyNicoll]

Originally Posted by BritTim

In my view, this behaviour is acceptable, but should be documented.

I'd disagree. My view is that it is perfectly acceptable for an API (a programming interface) to work that way - and it's understandable why it does. But /programmers/ are expected to read API documentation and then make use of the interfaces appropriately. Here, I'd expect that (if, as it seems, timestamps are tested not just day numbers) that "after date" should be treated as "greater than daynumber.999999" (to whatever precision FM's system records fractional days) rather than "greater than daynumber.000000". That is, I'd view this as an error in translating a parameter specified in a user interface into what's required for the programming interface. (Lots of things commonly need translated back and forth between external user-facing representations and internal forms; this is not unusual.)

Users on the other hand are not expected to (need to) understand anything about the underlying architecture. There is, in my view, no good reason why a search that specifies "after a date" should not mean exactly what it does in everyday life. If I say I'll do something "after Wednesday" I would not expect anybody to think I might start on Wednesday.

Moreover, FM's GUI search interface lacks immediately visible documentation. There's no indicator that "after" might not be intuitive, and no link to a help page. So no normal user is ever going to find their way to the documentation, such as it is. Currently the search help page says nothing at all about "after date" meaning "after the first nanosecond of that day".

There's a principle, "least astonishment", that's used in some programming languages and in interface design - see: https://en.wikipedia.org/wiki/Princi...t_astonishment - in which interfaces are meant to be designed so that they'll do what most users would expect. My view would be that this feature violates that principle.

[chrisjj]

Originally Posted by JeremyNicoll

Moreover, FM's GUI search interface lacks immediately visible documentation. There's no indicator that "after" might not be intuitive, and no link to a help page.

Moreover, what help there is confirms: "Use before: or after: to search for messages before or after certain dates."

I can see no grounds to doubt the implementation is simply defective. FM's denial disconcerts me.

Originally Posted by JeremyNicoll

Currently the search help page says nothing at all about "after date" meaning "after the first nanosecond of that day".

That as expected - given such a self-contradiction would surely have alerted the team to the fault at the outset.

Originally Posted by JeremyNicoll

There's a principle, "least astonishment", that's used in some programming languages and in interface design - see: https://en.wikipedia.org/wiki/Princi...t_astonishment - in which interfaces are meant to be designed so that they'll do what most users would expect. My view would be that this feature violates that principle.

Agreed, but this issue does not rest on user expectation. It is at simplest a failure of program to behave as documented.

PS FM would find this tricky to fix without perturbing existing stored searches.

[BritTim]

Originally Posted by chrisjj

Interesting. Why do you think it is acceptable?

When you only specify a date range, number of decisions needed to be made. Whether after:2023-07-13 means all messages received after 00:00 on 2023-07-13 or 23:59 on 2023-07-13 is only one of them. Should you use a neutral time zone like UTC? Should you use the time zone that the user was employing at the time the message was received? Is the right option the time zone of the user currently running the search (in which case two people making the same search of the same shared folder would get different results)? And so on ... The most important thing is that the results are well defined and documented.

Literally person centuries of talented IT professional time was invested to deal with all the complications around date and time handling in POSIX.

[chrisjj]

Originally Posted by BritTim

When you only specify a date range, number of decisions needed to be made. Whether after:2023-07-13 means all messages received after 00:00 on 2023-07-13 or 23:59 on 2023-07-13 is only one of them.

I disagree. It is none of them. "After a day" means after that day.

Originally Posted by BritTim

Literally person centuries of talented IT professional time was invested to deal with all the complications around date and time handling in POSIX.

Sure, but I can't see how any of that relates to this issue. After = after.

[BritTim]

Originally Posted by chrisjj

Sure, but I can't see how any of that relates to this issue. After = after.

You ask me when I sent you an email. You search for it using the date I give you. Assuming you do not know what time zone I am in (or was in at the time I sent the message) what date do you specify in your search? Are you upset when you fail to find the message using the date I provided to you?

If you are the person sending, receiving and searching for the email, everything is simple (as long as JMAP and Fastmail are aware of that). In the general case, the most important thing is to know what the search actually does (which the current documentation does not tell you).

NOTE: Actually, the behaviour of JMAP after is (inadequately) documented in https://jmap.io/spec-mail.html#emailquery

after: UTCDate The receivedAt date-time of the Email must be the same or after this date-time to match the condition.

Obviously, time defaults to 00:00.

[BritTim]

Further to my last message, note that Fastmail needs to document that you can specify something like

after:"2023-07-13 06:00"

to find messages received after 6:00 UTC on July 13th 2023 (as well, of course, as indicating the default).

[BritTim]

Originally Posted by JeremyNicoll

I'd disagree. My view is that it is perfectly acceptable for an API (a programming interface) to work that way - and it's understandable why it does. But /programmers/ are expected to read API documentation and then make use of the interfaces appropriately. Here, I'd expect that (if, as it seems, timestamps are tested not just day numbers) that "after date" should be treated as "greater than daynumber.999999" (to whatever precision FM's system records fractional days) rather than "greater than daynumber.000000". That is, I'd view this as an error in translating a parameter specified in a user interface into what's required for the programming interface. (Lots of things commonly need translated back and forth between external user-facing representations and internal forms; this is not unusual.)

Users on the other hand are not expected to (need to) understand anything about the underlying architecture. There is, in my view, no good reason why a search that specifies "after a date" should not mean exactly what it does in everyday life. If I say I'll do something "after Wednesday" I would not expect anybody to think I might start on Wednesday.

Moreover, FM's GUI search interface lacks immediately visible documentation. There's no indicator that "after" might not be intuitive, and no link to a help page. So no normal user is ever going to find their way to the documentation, such as it is. Currently the search help page says nothing at all about "after date" meaning "after the first nanosecond of that day".

There's a principle, "least astonishment", that's used in some programming languages and in interface design - see: https://en.wikipedia.org/wiki/Princi...t_astonishment - in which interfaces are meant to be designed so that they'll do what most users would expect. My view would be that this feature violates that principle.

The problem is that times are tested (as I explained above) but Fastmail does not properly document this.

[chrisjj]

Originally Posted by BritTim

You ask me when I sent you an email. You search for it using the date I give you.

That's not "after:".

Originally Posted by BritTim

Assuming you do not know what time zone I am in (or was in at the time I sent the message) what date do you specify in your search? Are you upset when you fail to find the message using the date I provided to you?

Yes that's an issue, but a distinct issue.

Originally Posted by BritTim

If you are the person sending, receiving and searching for the email, everything is simple (as long as JMAP and Fastmail are aware of that).

.. and provided there's no intervening UK "clocks change".

Originally Posted by BritTim

In the general case, the most important thing is to know what the search actually does (which the current documentation does not tell you).

The current documentation does tell you. It tells you wrong.

NOTE: Actually, the behaviour of JMAP after is (inadequately) documented in https://jmap.io/spec-mail.html#emailquery

Are we to know that FM searches are "JMAP"?

On "inadequate", I agree. e.g.

"after: UTCDate The receivedAt date-time of the Email must be the same or after this date-time to match the condition."

UTCDate is undefined. Woe betide anyone who guesses it is a date... given it is here apparently being compared to a date-time. Did someone/thing overlook a type conversion error?

Originally Posted by BritTim

Obviously, time defaults to 00:00.

Oops. It shouldn't default at all. Day means day - all of it.

[chrisjj]

Originally Posted by BritTim

The problem is that times are tested (as I explained above) but Fastmail does not properly document this.

I disagree. The problem is that times are tested, period. Day == day - all of it.

[JeremyNicoll]

Originally Posted by BritTim

The problem is that times are tested (as I explained above) but Fastmail does not properly document this.

Yes, but whether they document it or not (somewhere the user is not going to find unless they search for it) is not really the point. That level of detail (and timezones etc) is API detail. Also, we don't know whether the search process looks at the dates within headers (ie in the exact character by character format they are in, possibly malformed) or whether it's looking at validated binary dates extracted from emails and stored in FM's indexes.

The user interface is dumbed-down/vague to the point where whatever interpretation a user puts on something is going to be "normal everyday usage" so "after date" should mean what it would do in ordinary life.

A big part of the problem is that the user interface is set up so that a user can type something generic into a box (and hope that the system will interpret it the way they expect). The only immediately visible hint is that one could type eg "25 Nov" or "last Wednesday". There may be lots of other formats that work, but there's no clue in thatt dialog box what they are.

Rather than just have a single unformatted field where a user can type something, I think the box that opens up when you choose "After" should be two boxes, with both date and time. The date box could offer a dropdown so one could pick a date from a calendar widget, or something like "last Xxxday" .. and - if 00:00:00 is the implied time of day that should be prefilled in the time field. Yes of course this would be more complicated (for FM to code) but it would always be completely clear what value was going to be used.

[chrisjj]

Originally Posted by JeremyNicoll

Also, we don't know whether the search process looks at the dates within headers (ie in the exact character by character format they are in, possibly malformed) or whether it's looking at validated binary dates extracted from emails and stored in FM's indexes.

And even then, the UI doesn't actually state what the displayed message date is. The unlabelled date appearances on the message list and message top right often differ by one minute from the appearance labelled "Date:" in "Show details". And "Show details" can show two different versions: https://i.imgur.com/DRJh9KF.png . I'm hoping FM doesn't like Gmail mix between sent and received dates, but who knows?

[quote=JeremyNicoll;630135]The user interface is dumbed-down/vague to the point where whatever interpretation a user puts on something is going to be "normal everyday usage" so "after date" should mean what it would do in ordinary life.[quote=JeremyNicoll;630135]

I think that's unfair on "after:". There's nothing dumb about it's interface. Even smart writers/readers knows "After 2023" means after 2023. What's dumb is only the implementation - where "after:2023" includes all emails in 2023.

Originally Posted by JeremyNicoll

Rather than just have a single unformatted field where a user can type something, I think the box that opens up when you choose "After" should be two boxes, with both date and time.

It is labelled Date. Not Date-time.

Anyway, note were discussing the after: operator, not the After control where I believe the issues are different.

[JeremyNicoll]

Originally Posted by chrisjj

And "Show details" can show two different versions: https://i.imgur.com/DRJh9KF.png .

What do the date headers in that email show?

You can't blame FM if the email itself contains more than one Date: header. Also, there's now a huge range of current and obsolete but still meant to be supported valid formats for parts of a date/time specification. See eg RFC5322.

Originally Posted by chrisjj

I'm hoping FM doesn't like Gmail mix between sent and received dates, but who knows?

Not me...

Originally Posted by chrisjj

It is labelled Date. Not Date-time.

Anyway, note were discussing the after: operator, not the After control where I believe the issues are different.

Hmm, bear in mind that the header is called "Date:" even though it contains a time as well.

I've never used the "after:" operator at all, always used the Advanced Search GUI to specify what I want done.