[BUG] org-agenda thinks timestamps after 23:00 correspond to the next day [9.5.2 (release_9.5.2-25-gaf6f12 @ /home/ignacio/repos/emacs/lisp/org/)]

[BUG] org-agenda thinks timestamps after 23:00 correspond to the next day [9.5.2 (release_9.5.2-25-gaf6f12 @ /home/ignacio/repos/emacs/lisp/org/)]

[Ignacio Casso]

Hello,

After last Saturday's hour change in Spain, org-agenda thinks that
timestamps after 23:00 correspond to the next day in Emacs 29. I'm not
actually sure if that is the reason, since I usually use Emacs 27, but I
guess it must be that if I have found out three days after the hour
change.

I have tried to track down the problem, and it doesn't seem to be the
fault of any org-mode code change. The problem is that
(org-time-string-to-time timestamp), defined as (encode-time
(org-parse-time-string timestamp)), returns different things in Emacs 27
and Emacs 29.

Let's consider the timestamp "<2022-03-29 mar 23:00>" as an example:

1) (org-parse-time-string "<2022-03-29 mar 23:00>") returns (0 0 23 29 3
2022 nil nil nil).

2) (encode-time '(0 0 23 29 3 2022 nil nil nil)) returns '(25155 29520)
in Emacs 27, but (25155 33120) in Emacs 29

3.1) (time-to-days '(25155 29520)) returns 738243

3.2) (time-to-days '(25155 33120)) returns 738244

4) (org-today) returns 738243

Therefore, org-agenda thinks that "<2022-03-29 mar 23:00>" is today in
Emacs 27, but tomorrow in Emacs 29.

`encode-time' is defined in C, and is probably system dependent, so this
is probably not an org-mode bug. But maybe org-mode code could try to be
smart about this? I don't know if it's even possible.

And if this should not be fixed in org-mode, do you know were it should?
It could be an Emacs bug? Or maybe the problem is in my system?

Regards,

--Ignacio


Emacs  : GNU Emacs 29.0.50 (build 19, x86_64-pc-linux-gnu, GTK+ Version 3.24.20, cairo version 1.16.0)
 of 2022-03-29
Package: Org mode version 9.5.2 (release_9.5.2-25-gaf6f12 @ /home/ignacio/repos/emacs/lisp/org/)

Re: [BUG] org-agenda thinks timestamps after 23:00 correspond to the next day [9.5.2 (release_9.5.2-25-gaf6f12 @ /home/ignacio/repos/emacs/lisp/org/)]

[Ignacio Casso]

Actually, this only happens with SCHEDULED timestamps, so it might be
considered org-mode's fault since the handling of normal and SCHEDULED
timestamps is not always consistent.

The reason it is different is that `org-agenda-get-timestamps' looks for
timestamps in the org buffer using a regular expression that already
matches only the expected date. `org-agenda-get-scheduled' can't do
that, so it uses a regular expression that matches any timestamp,
converts the timestamp to an absolute date using
`org-agenda--timestamp-to-absolute', and compares with the current
agenda date.

`org-agenda-get-timestamps' also has to do something similar for
repeated tasks. However, the function used to get the absolute date is
different for those timestamps. The function for scheduled timestamps
boils down to

  (time-to-days (encode-time (org-parse-time-string timestamp)))

but for repeating timestamps, it boils down to

  (calendar-absolute-from-gregorian (org-date-to-gregorian timestamp))

For the timestamp "<2022-03-30 mié 23:00>", the second form returns
738244 in my machine, which corresponds to (org-today) at 30/03/2022,
but the second returns 738245. Thus, repeated timestamps still work, but
scheduled timestamps don't.

Is there a reason why two different ways to obtain those dates are used?

For completion, I have also checked deadline timestamps, and they suffer
from the same problem as scheduled timestamps. A deadline for today at
23:00 will appear in the agenda as it would a deadline for tomorrow at
22:59, that is, with a warning that it is due in one day, and not as a
deadline for today at 22:59 would appear.

Regards,

Ignacio


Ignacio Casso <ignaciocasso@hotmail.com> writes:

> Hello,
>
> After last Saturday's hour change in Spain, org-agenda thinks that
> timestamps after 23:00 correspond to the next day in Emacs 29. I'm not
> actually sure if that is the reason, since I usually use Emacs 27, but I
> guess it must be that if I have found out three days after the hour
> change.
>
> I have tried to track down the problem, and it doesn't seem to be the
> fault of any org-mode code change. The problem is that
> (org-time-string-to-time timestamp), defined as (encode-time
> (org-parse-time-string timestamp)), returns different things in Emacs 27
> and Emacs 29.
>
> Let's consider the timestamp "<2022-03-29 mar 23:00>" as an example:
>
> 1) (org-parse-time-string "<2022-03-29 mar 23:00>") returns (0 0 23 29 3
> 2022 nil nil nil).
>
> 2) (encode-time '(0 0 23 29 3 2022 nil nil nil)) returns '(25155 29520)
> in Emacs 27, but (25155 33120) in Emacs 29
>
> 3.1) (time-to-days '(25155 29520)) returns 738243
>
> 3.2) (time-to-days '(25155 33120)) returns 738244
>
> 4) (org-today) returns 738243
>
> Therefore, org-agenda thinks that "<2022-03-29 mar 23:00>" is today in
> Emacs 27, but tomorrow in Emacs 29.
>
> `encode-time' is defined in C, and is probably system dependent, so this
> is probably not an org-mode bug. But maybe org-mode code could try to be
> smart about this? I don't know if it's even possible.
>
> And if this should not be fixed in org-mode, do you know were it should?
> It could be an Emacs bug? Or maybe the problem is in my system?
>
> Regards,
>
> --Ignacio
>
>
> Emacs  : GNU Emacs 29.0.50 (build 19, x86_64-pc-linux-gnu, GTK+ Version 3.24.20, cairo version 1.16.0)
>  of 2022-03-29
> Package: Org mode version 9.5.2 (release_9.5.2-25-gaf6f12 @ /home/ignacio/repos/emacs/lisp/org/)

Re: [BUG] org-agenda thinks timestamps after 23:00 correspond to the next day

[Max Nikulin]

On 30/03/2022 14:13, Ignacio Casso wrote:
> Actually, this only happens with SCHEDULED timestamps, so it might be
> considered org-mode's fault since the handling of normal and SCHEDULED
> timestamps is not always consistent.

A first step to debug date issues is to check current timezone

     (getenv "TZ")

     $ timedatectl

I have not looked what changed in emacs in respect to timezones.

Re: [BUG] org-agenda thinks timestamps after 23:00 correspond to the next day

[Ignacio Casso]

> A first step to debug date issues is to check current timezone

Thanks Max.

>     (getenv "TZ")

This returns nil

>     $ timedatectl

And this returns

               Local time: mié 2022-03-30 12:17:36 CEST
           Universal time: mié 2022-03-30 10:17:36 UTC
                 RTC time: mié 2022-03-30 10:17:37
                Time zone: Europe/Madrid (CEST, +0200)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

> I have not looked what changed in emacs in respect to timezones.

The function `encode-time', which is implemented in C, has changed,
since it returns different things in Emacs 27 and Emacs 29, when it
receives as argument the return value of (org-parse-time-string
<timestamp>).

Re: [BUG] org-agenda thinks timestamps after 23:00 correspond to the next day [9.5.2 (release_9.5.2-25-gaf6f12 @ /home/ignacio/repos/emacs/lisp/org/)]

[Max Nikulin]

On 29/03/2022 20:09, Ignacio Casso wrote:
> 
> Let's consider the timestamp "<2022-03-29 mar 23:00>" as an example:
> 
> 1) (org-parse-time-string "<2022-03-29 mar 23:00>") returns (0 0 23 29 3
> 2022 nil nil nil).
> 
> 2) (encode-time '(0 0 23 29 3 2022 nil nil nil)) returns '(25155 29520)
> in Emacs 27, but (25155 33120) in Emacs 29

Ubuntu-20.04 LTS has emacs-26.3, so...

Could you, please, try

(encode-time '(0 0 23 29 3 2022 nil -1 nil))
                                     ^^^

Accordingly to
"TIME is a list (SECOND MINUTE HOUR DAY MONTH YEAR IGNORED DST ZONE)."
it should be daylight saving time flag and -1 means "guess" unlike nil 
that tells "no DST".

Emacs-26 does not support DST:
     (encode-time SECOND MINUTE HOUR DAY MONTH YEAR &optional ZONE)
Current master HEAD:
     (encode-time TIME &rest OBSOLESCENT-ARGUMENTS)

If my guess is right, `org-parse-time-string' should add -1 instead of 
nil in that position.

Re: [BUG] org-agenda thinks timestamps after 23:00 correspond to the next day [9.5.2 (release_9.5.2-25-gaf6f12 @ /home/ignacio/repos/emacs/lisp/org/)]

[Ignacio Casso]

>
> Ubuntu-20.04 LTS has emacs-26.3, so...
>
> Could you, please, try
>
> (encode-time '(0 0 23 29 3 2022 nil -1 nil))
>                                     ^^^

Thanks. Using -1 as DST argument indeed fixes it.

However, while testing that, I've realized that I made a mistake in the
bug report. `encode-time' and `org-parse-time-string' return the same
both in old and current Emacs and org-mode versions (by old Emacs I mean
Emacs 27, not 26). The problem is that in ELPA's org-mode,
`org-time-string-to-time' is defined as

  (apply #'encode-time (org-parse-time-string s)))

and in org-mode's built-in version of Emacs 29, it is defined as

  (encode-time (org-parse-time-string s)))

and those two return different things. In the second case, the actual
three last elements of the list returned by `org-parse-time-string' are
used, and they are all nil. In the first case, according to the
docstring of `encode-time', the last two elements of the list are
ignored and -1 and nil are used as default.

> If my guess is right, `org-parse-time-string' should add -1 instead of
> nil in that position.

So your guess is right, and doing this would restore the previous
behavior. However, the curious things is that in the org-mode repository
at git://git.savannah.gnu.org/emacs/org-mode.git,
`org-time-string-to-time' still uses `apply', and always has. The
change, and thus the bug, was only introduced directly in Emacs 29 a few
months ago.

So I guess this is an Emacs 29 bug? I didn't know the two repositories
could diverge like that. Should I report it to Emacs 29 maintainers?
Or can org-mode maintainers fix it in the Emacs repository?

Re: [BUG] org-agenda thinks timestamps after 23:00 correspond to the next day [9.5.2 (release_9.5.2-25-gaf6f12 @ /home/ignacio/repos/emacs/lisp/org/)]

[Max Nikulin]

Confirmed

Emacs copy of Org changed the way of calling `encode-time' as a result 
interpretation of last nils returned by `org-parse-string' altered from 
ignored to "no DST".

Kyle, be aware of the conflict with `org-time-string-to-time' when you 
will port emacs commit dd0727e1ec1f535b9b06be88173b4d3ccd55abcb
Paul Eggert <eggert@cs.ucla.edu> Thu Dec 16 09:40:21 2021 -0800
encode-time simplifications

I hope, it is safe to pad `org-parse-time-string' return value with nil 
-1 nil instead of nil nil nil, but I have not tried to run tests (if 
they exist).

New calling convention for `encode-time' exists since emacs-27.1, so it 
is incompatible with yet supported emacs-26. It is unfortunate that 
sources in Org and in Emacs repository diverged, but I am unsure if it 
reasonable to introduce a new compatibility wrapper. Emacs developer may 
be unhappy if the change will be reverted to way deprecated way to call 
`encode-time'. For almost a year I am reading messages here that 
Emacs-28 is about to be released, so Emacs-26 is almost unsupported in Org.

On 31/03/2022 21:11, Ignacio Casso wrote:
>>
>> Could you, please, try
>>
>> (encode-time '(0 0 23 29 3 2022 nil -1 nil))
> 
> Thanks. Using -1 as DST argument indeed fixes it.
> 
> However, while testing that, I've realized that I made a mistake in the
> bug report. `encode-time' and `org-parse-time-string' return the same
> both in old and current Emacs and org-mode versions (by old Emacs I mean
> Emacs 27, not 26). The problem is that in ELPA's org-mode,
> `org-time-string-to-time' is defined as
> 
>    (apply #'encode-time (org-parse-time-string s)))
> 
> and in org-mode's built-in version of Emacs 29, it is defined as
> 
>    (encode-time (org-parse-time-string s)))

Thank you, Ignacio. I read the code of `encode-time' once more and see 
that the DST argument is taken into account only when date-time fields 
are passed as a list. It explains change of behavior despite of absence 
of apparent recent changes in the code of the function.

> and those two return different things. In the second case, the actual
> three last elements of the list returned by `org-parse-time-string' are
> used, and they are all nil. In the first case, according to the
> docstring of `encode-time', the last two elements of the list are
> ignored and -1 and nil are used as default.
> 
>> If my guess is right, `org-parse-time-string' should add -1 instead of
>> nil in that position.
> 
> So your guess is right, and doing this would restore the previous
> behavior. However, the curious things is that in the org-mode repository
> at git://git.savannah.gnu.org/emacs/org-mode.git,
> `org-time-string-to-time' still uses `apply', and always has. The
> change, and thus the bug, was only introduced directly in Emacs 29 a few
> months ago.
> 
> So I guess this is an Emacs 29 bug? I didn't know the two repositories
> could diverge like that. Should I report it to Emacs 29 maintainers?
> Or can org-mode maintainers fix it in the Emacs repository?

The problem is a consequence of grep-refactoring of Emacs code, but 
likely it should be fixed at the Org side.

Re: [BUG] org-agenda thinks timestamps after 23:00 correspond to the next day [9.5.2 (release_9.5.2-25-gaf6f12 @ /home/ignacio/repos/emacs/lisp/org/)]

[Tim Cross]

Max Nikulin <manikulin@gmail.com> writes:

> New calling convention for `encode-time' exists since emacs-27.1, so it is
> incompatible with yet supported emacs-26. It is unfortunate that sources in Org
> and in Emacs repository diverged, but I am unsure if it reasonable to introduce
> a new compatibility wrapper. Emacs developer may be unhappy if the change will
> be reverted to way deprecated way to call `encode-time'. For almost a year I am
> reading messages here that Emacs-28 is about to be released, so Emacs-26 is
> almost unsupported in Org.
>

I don' think that is correct. The policy is to support the previous 2
major versions, which means when Emacs 28 is released, org will need to
support both 27.x and 26.x. It will likely be necessary to have a
compatibility layer for 26.x given the changes.

Re: [BUG] org-agenda thinks timestamps after 23:00 correspond to the next day [9.5.2 (release_9.5.2-25-gaf6f12 @ /home/ignacio/repos/emacs/lisp/org/)]

[Kyle Meyer]

Max Nikulin writes:

> Confirmed
>
> Emacs copy of Org changed the way of calling `encode-time' as a result 
> interpretation of last nils returned by `org-parse-string' altered from 
> ignored to "no DST".
>
> Kyle, be aware of the conflict with `org-time-string-to-time' when you 
> will port emacs commit dd0727e1ec1f535b9b06be88173b4d3ccd55abcb
> Paul Eggert <eggert@cs.ucla.edu> Thu Dec 16 09:40:21 2021 -0800
> encode-time simplifications

Thank you, Max, for looping me into the discussion (and thanks, Ignacio,
for the initial report and debugging).

> New calling convention for `encode-time' exists since emacs-27.1, so it 
> is incompatible with yet supported emacs-26. [...]

>> So I guess this is an Emacs 29 bug? I didn't know the two repositories
>> could diverge like that. Should I report it to Emacs 29 maintainers?
>> Or can org-mode maintainers fix it in the Emacs repository?
>
> The problem is a consequence of grep-refactoring of Emacs code, but 
> likely it should be fixed at the Org side.

The long tail of 9e1b9fe62 (Port more time-related changes, 2019-08-18)

My suggestion:

  1. Send a report to bug-gnu-emacs@gnu.org describing the issue.  Ask
     that Paul revert those changes.

     I can do this at some point this week.

  2. Audit and update the call sites on our side, along with some
     compatibility layer.

The first isn't necessary, but it avoids the problem living in the Emacs
master branch until the updated Org code base (main branch) is synced
with it (which hasn't started yet).

Re: [BUG] org-agenda thinks timestamps after 23:00 correspond to the next day [9.5.2 (release_9.5.2-25-gaf6f12 @ /home/ignacio/repos/emacs/lisp/org/)]

[Max Nikulin]

On 05/04/2022 11:20, Kyle Meyer wrote:
> Max Nikulin writes:
> 
>> Emacs copy of Org changed the way of calling `encode-time' as a result
>> interpretation of last nils returned by `org-parse-string' altered from
>> ignored to "no DST".
>>
> My suggestion:
> 
>    1. Send a report to bug-gnu-emacs@gnu.org describing the issue.  Ask
>       that Paul revert those changes.
> 
>       I can do this at some point this week.

Ignacio, have you tried recent emacs master branch? Paul reverted most 
of his changed, see

8ef37913d Paul Eggert 2022-04-06 07:48:05
Port Org encode-time usage back to Emacs 25

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=54731

>    2. Audit and update the call sites on our side, along with some
>       compatibility layer.
> 
> The first isn't necessary, but it avoids the problem living in the Emacs
> master branch until the updated Org code base (main branch) is synced
> with it (which hasn't started yet).

Re: [BUG] org-agenda thinks timestamps after 23:00 correspond to the next day [9.5.2 (release_9.5.2-25-gaf6f12 @ /home/ignacio/repos/emacs/lisp/org/)]

[Ignacio Casso]

Max Nikulin <manikulin@gmail.com> writes:

> On 05/04/2022 11:20, Kyle Meyer wrote:
>> Max Nikulin writes:
>> 
>>> Emacs copy of Org changed the way of calling `encode-time' as a result
>>> interpretation of last nils returned by `org-parse-string' altered from
>>> ignored to "no DST".
>>>
>> My suggestion:
>>    1. Send a report to bug-gnu-emacs@gnu.org describing the issue.
>> Ask
>>       that Paul revert those changes.
>>       I can do this at some point this week.
>
> Ignacio, have you tried recent emacs master branch? Paul reverted most
> of his changed, see
>
> 8ef37913d Paul Eggert 2022-04-06 07:48:05
> Port Org encode-time usage back to Emacs 25
>
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=54731
>

Yes, I have just tried, and everything works fine now. Thanks!

Re: [BUG] org-agenda thinks timestamps after 23:00 correspond to the next day [9.5.2 (release_9.5.2-25-gaf6f12 @ /home/ignacio/repos/emacs/lisp/org/)]

[Max Nikulin]

On 23/04/2022 14:20, Ignacio Casso wrote:
> Max Nikulin writes:
> 
>>
>> Ignacio, have you tried recent emacs master branch? Paul reverted most
>> of his changed, see
...
> Yes, I have just tried, and everything works fine now. Thanks!

Greet.

I would prefer to keep this issue tracked on https://updates.orgmode.org 
till a couple of places will be fixed in the Org repository to avoid nil 
and use -1 instead where `encode-time' expects DST when called in the 
currently recommended way. I do not insist though, so fill free to close 
the issue.

I mean some of Paul's patch
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=54764;msg=10;att=1;filename=0001-Improve-Org-usage-of-timestamps.patch
I highlighted these changes in
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=54764#21
also available as 
https://list.orgmode.org/572f6010-a544-2f28-c466-e07192f7d244@gmail.com
but I have not managed to add a "patch" entry to updates.orgmode.org

Other modifications may be committed as well, but I would prefer another 
approach with compatibility wrapper:
Max Nikulin. [DRAFT][PATCH] org-encode-time compatibility and 
convenience helper. Mon, 11 Apr 2022 22:22:48 +0700.
https://lists.orgmode.org/7f4ea652-7d22-fb61-f873-5e92f078c9e6@gmail.com

Re: [BUG] org-agenda thinks timestamps after 23:00 correspond to the next day [9.5.2 (release_9.5.2-25-gaf6f12 @ /home/ignacio/repos/emacs/lisp/org/)]

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>> Yes, I have just tried, and everything works fine now. Thanks!
>
> Greet.
>
> I would prefer to keep this issue tracked on https://updates.orgmode.org 
> till a couple of places will be fixed in the Org repository to avoid nil 
> and use -1 instead where `encode-time' expects DST when called in the 
> currently recommended way. I do not insist though, so fill free to close 
> the issue.
>
> I mean some of Paul's patch
> ...
> https://list.orgmode.org/572f6010-a544-2f28-c466-e07192f7d244@gmail.com
>
> Other modifications may be committed as well, but I would prefer another 
> approach with compatibility wrapper:
> ...
> Max Nikulin. [DRAFT][PATCH] org-encode-time compatibility and 
> convenience helper. Mon, 11 Apr 2022 22:22:48 +0700.
> https://lists.orgmode.org/7f4ea652-7d22-fb61-f873-5e92f078c9e6@gmail.com

Max,
Do I understand correctly that both the patches you referred to have
been installed? If so, is it safe to close this bug report?

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92