Re: Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Allen Li <vianchielfaura@gmail.com>]

On Wed, Nov 1, 2017 at 5:09 PM, Tim Cross <theophilusx@gmail.com> wrote:
>
> Let me try to clarify and see if that helps.
>
> Org is only forcing time into UTC format for functions which deal with
> time comparisons. i.e. those which use org-2ft. The reason for doing
> this is to try and avoid issues relating to daylight savings timezone
> changes when performing comparison operations.
>
> It is not clear to me exactly how this is actually creating a problem
> for your scenario as there does not seem to be anywhere in org where
> values created by org-2ft are converted back to human readable time
> strings. If you can provide more details on how this is a practical
> issue in org-mode that might help.

I think I made the problem quite clear multiple times.

Assuming today is 2017-10-31, SCHEDULED<"<now>" will match results
that have SCHEDULED=<2017-11-01> depending on your local time zone.

>
> I do think there was an error in your example analysis.
>
> In your examples of
>
> (current-time-string (float-time))
> "Mon Oct 30 21:21:31 2017" ; right
>
> (current-time-string (org-time-today))
> "Mon Oct 30 00:00:00 2017" ; right
>
> (current-time-string (org-2ft "<2017-10-31>"))
> "Mon Oct 30 17:00:00 2017" ; wrong
>
> the last one is only wrong because you have failed to tell
> current-time-string that the value you are converting is a UTC time. If
> you change the code to
>
> (current-time-string (org-2ft "<2017-10-31>") "UTC")
> "Mon Oct 30 00:00:00 2017" ; correct

No, that is wrong. Mon Oct 30 00:00:00 2017 UTC != Mon Oct 30 00:00:00
2017 UTC-7

(format-time-string "%FT%T%z" (org-2ft "<2017-10-31>"))
"2017-10-30T17:00:00+0700" ; wrong
(format-time-string "%FT%T%z" (org-2ft "<2017-10-31>") "UTC")
"2017-10-31T00:00:00+0000" ; wrong

In fact, you haven't solved a thing, since (current-time-string
(org-2ft "<2017-10-31>") "UTC") == (current-time-string (org-2ft
"<2017-10-31>"))  The time is still the same, and still wrong, just
formatted differently if you silently drop the TZ info from the time
string.

>
> Now, considering your suggestion to change org-2ft to not use UTC.
>
> If we change org-2ft to not use UTC time, we end up with time
> related calculations which cross daylight savings boarders becoming
> incorrect. So, if you have a clock table where you want to calculate the
> number of hours between two timestamps and one is within daylgiht
> savings time and the other is outside it, your calculation will be out
> by an hour.

Or rather, we end up with time related calculations which cross
daylight savings becoming correct.

But I am beginning to see the picture; someone wanted time related
calculations crossing DST to be incorrect to appear "right" to time
zone naive applications.

>
> As org-2ft is not used in conversion of 'epoch' time to times strings by
> org, I'm not sure reverting the change gains us anything and will likely
> adversely impact things like clock table calculations. What may be
> justified is to add a note to the manual which clearly states that if
> you are using org-2ft in your own code and plan to convert the result
> back to a timestamp string, you need to ensure you explicitly tell the
> function performing the conversion the input value is in UTC and not
> local time.

Yes, org-2ft documentation should be updated at the very least,
assuming that this behavior is desired.

>
> I agree that enhancing org timestamps to include TZ information would be
> a non-trivial chunk of work, I do feel it is the only solid way to
> address all of these issues. I suspect there are still edge cases in
> time related computations which fail even with the UTC approach and
> there are certainly extensions which could benefit from explicit TZ
> information (for example, taskgjuggler, which does time calculations for
> planning and where such calculations can easily cross multiple daylight
> savings TZ changes or include operations covering different timezones
> etc).
>
> Tim
>
> Allen Li <vianchielfaura@gmail.com> writes:
>
>> Apologies, my reply was partially omitted.  My full reply follows
>>
>> On Wed, Nov 1, 2017 at 6:09 AM, Tim Cross <theophilusx@gmail.com> wrote:
>>>
>>> Allen Li <vianchielfaura@gmail.com> writes:
>>>
>>>> On Wed, Nov 1, 2017 at 12:18 AM, Tim Cross <theophilusx@gmail.com> wrote:
>>>>>
>>>>> My preferences would be
>>>>>
>>>>> 1. If a timestamp does not include the TZ, then assume the local TZ
>>>>> 2. If a timestamp does include the TZ, honour that TZ
>>>>
>>>> Org mode does not support TZ in time strings and adding support isn't
>>>> the topic of the current bug.
>>>>
>>>
>>> I disagree. The root cause of the bug is due to a lack of clarity
>>> regarding timezones. I also suspect this is also the basic cause of
>>> issues relating to the use of timestamps/time strings and daylight
>>> savings, especially when it comes to performing calculations etc.
>>
>> Unless I missed something, Org mode does not support explicit
>> timezones in Org timestamps, and adding such support would take a
>> large effort [1].
>>
>> [1]: https://lists.gnu.org/archive/html/emacs-orgmode/2011-04/msg00299.html
>>
>> Unfortunately, I do not have time for the task.  Thus, Org mode
>> will have to make due with only Org timestamps without an
>> explicit timezone unless a savior appears.
>>
>>>
>>>> I want to emphasize the distinction between timestamps and time
>>>> strings.  Timestamps are a
>>>> float value, Unix timestamps.  Time strings are Org mode specific,
>>>> like <2017-10-30 12:34:56>
>>>
>>> Org mode refers to what you are calling time strings as timestamps. In
>>> reality, there is no difference - one is just a numeric representation
>>> and the other is a string representation. It is good you have clarified
>>> your definition to reduce confusion. However, I think the problems are
>>> arising because of a lack of explicit TZ handling.
>>
>> There's a pretty significant difference.  Unix timestamps are
>> standardized and used widely by most computers and Emacs itself.
>> Unix timestamps are time zone agnostics.
>>
>> Org timestamps (time strings, whatever) are used exclusively by
>> Org mode.  As Org timestamps do not support explicit timezones,
>> they are not time zone agnostic.  The interpretation of an Org
>> timestamp depends on the time zone of the local machine, whereas
>> the interpretation of a Unix timestamp does not depend on the
>> time zone of the local machine.
>>
>>>
>>>>
>>>> Timestamps do not have timezone information since they describe an
>>>> exact (well, minus leap seconds)
>>>> point in time, the number of seconds after the epoch.
>>>>
>>>
>>> A point in time is measured in number of seconds since epoch. However,
>>> if you want to use local (wall) time to display that point, you have to
>>> include TZ information when converting that number to a more
>>> meaningful/usable (for humans) format.
>>
>> I'm not sure what you mean.  Time zone information is provided by
>> the local machine OS.  If I pass a Unix timestamp to Emacs/the OS
>> and tell it to format it in human readable format in the local
>> time zone (the default behavior), then it will be done, without
>> my having to attach time zone information anywhere.
>>
>>> The point 'now' for me is UTC+1100
>>> and for you (based on your previous post) UTC-0700, so our
>>> representations in string format of this value will be different. Even
>>> on a single machine, it is also relevant. For example, if I have two org
>>> timestamps (your times strings) and I want to calculate the number of
>>> hours between the two, I need to include TZ information as one timestamp
>>> might be during daylight savings time and the other outside daylight
>>> savings time.
>>
>> Ideally, except Org mode does not support including explicit TZ
>> information in Org timestamps.  Thus, Org mode can only current
>> interpret Org timestamps according to some blessed time zone.
>> Four months ago, that blessed time zone changed from the local
>> machine's time zone to UTC.
>>
>>> Any calculation which fails to consider TZ information in
>>> this case will be out by an hour.
>>>
>>> If we just take the simple solution and ignore TZ information,
>>
>> Org timestamps do not have TZ information.  It's not a matter of
>> not ignoring TZ information, support for TZ information would
>> have to be added, which as I understand is non-trivial.
>>
>>> we will
>>> break calculations which cross daylight savings 'boarders'. I also
>>> suspect we will get inconsistent behaviour when integrating with other
>>> systems (such as external calendars, ical integration or sharing
>>> timestamp data with others etc).
>>
>> As long as we use Unix timestamps when integrating with other
>> systems, there is no ambiguity.
>>
>>>
>>>>> 3. If the timestamp does not include a time component, default to 0:00:0
>>>>
>>>> This is what Org mode does
>>>>
>>>>> for the local TZ
>>>>
>>>> This is what Org mode used to do, now it interprets it as 00:00:00 UTC.
>>>>
>>>>> 4. org-2ft should not enforce the UTC TZ. I agree this is incorrect.
>>>>>
>>>>> Rationale is that the user should have ability to fully control how
>>>>> their timestamps are represented. However, adding the TZ is probably
>>>>> just a pain for the majority of use cases, so defaulting to the local
>>>>> (wall) TZ is OK provided you can override this consistently by adding
>>>>> explicit TZ values.
>>>>>
>>>>> However, there is some devil in the details we need to work out. For
>>>>> example, should we support both TZ names (like AEDT or Australia/Sydney)
>>>>> and POSIX style +11/+11:00/+1100, should we add an option to tell org to
>>>>> always add TZ info in timestamps which include time components and if
>>>>> so, how complex will this need to be e.g. handle setting a future/past
>>>>> timestamp which is in a different (daylight savings) offset and what
>>>>> about the additional complexity in dealing with timestamp calculations
>>>>> where dates might be in different offsets due to daylight savings -
>>>>> while all quite possible, it does add significant complexity and this
>>>>> may have adverse impact on performance. Not to say we shouldn't do it,
>>>>> just that it will take significant work.
>>>>>
>>>>> I suspect just the first part won't have major impact - at least no more
>>>>> than enforcing UTC in org-2ft.
>>>>
>>>> Org mode does not support TZ in time strings currently.  While I would
>>>> like such a feature very much,
>>>> adding TZ support isn't the topic of the current bug, just fixing how
>>>> time strings are interpreted.
>>>>
>>>
>>> We could make the decision that all org timestamps are relative to the
>>> local TZ (ignoring the daylight savings issue) and that would probably
>>> be OK for most users
>>
>> I agree, and think that this makes more sense than using UTC.
>>
>>> but I feel the whole issue is due to a lack of
>>> adequate/consistent TZ interpretation.
>>
>> Also agreed, but I cannot volunteer for the task of adding that
>> unfortunately.
>>
>>> The problem with org-2ft is that
>>> it forces the conversion into UTC when the input time string may
>>> actually be in local time, resulting in a number which is out by
>>> whatever the local offset is.
>>
>> Exactly
>>
>>> When you then convert back, your string
>>> representation will be out by at least that amount (possibly more
>>> depending on whether TZ info is applied in the conversion back to a
>>> string).
>>
>> TZ info is not applied in that way going back, since Unix
>> timestamps are time zone agnostic.  It may affect how the string
>> representation is formatted, but the time will not be even more
>> incorrect.
>>
>> e.g., 2017-10-30T00:00:00+0000 vs 2017-10-30T07:00:00-0700.
>> Different strings due to the time zone used, but they both
>> describe the exact point of time as their input Unix timestamp.
>>
>>>
>>> If it wasn't for the benefits of org files being essentially plain text
>>> files, the easiest solution would possibly be to represent timestamps as
>>> a number and use an overlay or similar to convert it to a human readable
>>> value based on whatever the user sets as the timezone for
>>> display/export. Unfortunately, this breaks one of the big benefits of
>>> org-mode over other solutions i.e. the data is just plain text and can
>>> be accessed/used by anything able to read plain text.
>>>
>>> Tim
>>>
>>>
>>>>>
>>>>> Tim
>>>>>
>>>>> P.S. when you start to think about it, it is easy to see how Java
>>>>> screwed up this stuff so badly!
>>>
>>>
>>> --
>>> Tim Cross
>
>
> --
> Tim Cross