Re: Bug: org-2ft and/or float-time is wrong [9.1.2 (9.1.2-22-ga2a034-elpaplus @ ~/.emacs.d/elpa/org-plus-contrib-20171023/)]

[Tim Cross <theophilusx@gmail.com>]

Let me try to clarify and see if that helps.

Org is only forcing time into UTC format for functions which deal with
time comparisons. i.e. those which use org-2ft. The reason for doing
this is to try and avoid issues relating to daylight savings timezone
changes when performing comparison operations.

It is not clear to me exactly how this is actually creating a problem
for your scenario as there does not seem to be anywhere in org where
values created by org-2ft are converted back to human readable time
strings. If you can provide more details on how this is a practical
issue in org-mode that might help.

I do think there was an error in your example analysis.

In your examples of

(current-time-string (float-time))
"Mon Oct 30 21:21:31 2017" ; right

(current-time-string (org-time-today))
"Mon Oct 30 00:00:00 2017" ; right

(current-time-string (org-2ft "<2017-10-31>"))
"Mon Oct 30 17:00:00 2017" ; wrong

the last one is only wrong because you have failed to tell
current-time-string that the value you are converting is a UTC time. If
you change the code to

(current-time-string (org-2ft "<2017-10-31>") "UTC")
"Mon Oct 30 00:00:00 2017" ; correct

Now, considering your suggestion to change org-2ft to not use UTC.

If we change org-2ft to not use UTC time, we end up with time
related calculations which cross daylight savings boarders becoming
incorrect. So, if you have a clock table where you want to calculate the
number of hours between two timestamps and one is within daylgiht
savings time and the other is outside it, your calculation will be out
by an hour.

As org-2ft is not used in conversion of 'epoch' time to times strings by
org, I'm not sure reverting the change gains us anything and will likely
adversely impact things like clock table calculations. What may be
justified is to add a note to the manual which clearly states that if
you are using org-2ft in your own code and plan to convert the result
back to a timestamp string, you need to ensure you explicitly tell the
function performing the conversion the input value is in UTC and not
local time.

I agree that enhancing org timestamps to include TZ information would be
a non-trivial chunk of work, I do feel it is the only solid way to
address all of these issues. I suspect there are still edge cases in
time related computations which fail even with the UTC approach and
there are certainly extensions which could benefit from explicit TZ
information (for example, taskgjuggler, which does time calculations for
planning and where such calculations can easily cross multiple daylight
savings TZ changes or include operations covering different timezones
etc).  

Tim

Allen Li <vianchielfaura@gmail.com> writes:

> Apologies, my reply was partially omitted.  My full reply follows
>
> On Wed, Nov 1, 2017 at 6:09 AM, Tim Cross <theophilusx@gmail.com> wrote:
>>
>> Allen Li <vianchielfaura@gmail.com> writes:
>>
>>> On Wed, Nov 1, 2017 at 12:18 AM, Tim Cross <theophilusx@gmail.com> wrote:
>>>>
>>>> My preferences would be
>>>>
>>>> 1. If a timestamp does not include the TZ, then assume the local TZ
>>>> 2. If a timestamp does include the TZ, honour that TZ
>>>
>>> Org mode does not support TZ in time strings and adding support isn't
>>> the topic of the current bug.
>>>
>>
>> I disagree. The root cause of the bug is due to a lack of clarity
>> regarding timezones. I also suspect this is also the basic cause of
>> issues relating to the use of timestamps/time strings and daylight
>> savings, especially when it comes to performing calculations etc.
>
> Unless I missed something, Org mode does not support explicit
> timezones in Org timestamps, and adding such support would take a
> large effort [1].
>
> [1]: https://lists.gnu.org/archive/html/emacs-orgmode/2011-04/msg00299.html
>
> Unfortunately, I do not have time for the task.  Thus, Org mode
> will have to make due with only Org timestamps without an
> explicit timezone unless a savior appears.
>
>>
>>> I want to emphasize the distinction between timestamps and time
>>> strings.  Timestamps are a
>>> float value, Unix timestamps.  Time strings are Org mode specific,
>>> like <2017-10-30 12:34:56>
>>
>> Org mode refers to what you are calling time strings as timestamps. In
>> reality, there is no difference - one is just a numeric representation
>> and the other is a string representation. It is good you have clarified
>> your definition to reduce confusion. However, I think the problems are
>> arising because of a lack of explicit TZ handling.
>
> There's a pretty significant difference.  Unix timestamps are
> standardized and used widely by most computers and Emacs itself.
> Unix timestamps are time zone agnostics.
>
> Org timestamps (time strings, whatever) are used exclusively by
> Org mode.  As Org timestamps do not support explicit timezones,
> they are not time zone agnostic.  The interpretation of an Org
> timestamp depends on the time zone of the local machine, whereas
> the interpretation of a Unix timestamp does not depend on the
> time zone of the local machine.
>
>>
>>>
>>> Timestamps do not have timezone information since they describe an
>>> exact (well, minus leap seconds)
>>> point in time, the number of seconds after the epoch.
>>>
>>
>> A point in time is measured in number of seconds since epoch. However,
>> if you want to use local (wall) time to display that point, you have to
>> include TZ information when converting that number to a more
>> meaningful/usable (for humans) format.
>
> I'm not sure what you mean.  Time zone information is provided by
> the local machine OS.  If I pass a Unix timestamp to Emacs/the OS
> and tell it to format it in human readable format in the local
> time zone (the default behavior), then it will be done, without
> my having to attach time zone information anywhere.
>
>> The point 'now' for me is UTC+1100
>> and for you (based on your previous post) UTC-0700, so our
>> representations in string format of this value will be different. Even
>> on a single machine, it is also relevant. For example, if I have two org
>> timestamps (your times strings) and I want to calculate the number of
>> hours between the two, I need to include TZ information as one timestamp
>> might be during daylight savings time and the other outside daylight
>> savings time.
>
> Ideally, except Org mode does not support including explicit TZ
> information in Org timestamps.  Thus, Org mode can only current
> interpret Org timestamps according to some blessed time zone.
> Four months ago, that blessed time zone changed from the local
> machine's time zone to UTC.
>
>> Any calculation which fails to consider TZ information in
>> this case will be out by an hour.
>>
>> If we just take the simple solution and ignore TZ information,
>
> Org timestamps do not have TZ information.  It's not a matter of
> not ignoring TZ information, support for TZ information would
> have to be added, which as I understand is non-trivial.
>
>> we will
>> break calculations which cross daylight savings 'boarders'. I also
>> suspect we will get inconsistent behaviour when integrating with other
>> systems (such as external calendars, ical integration or sharing
>> timestamp data with others etc).
>
> As long as we use Unix timestamps when integrating with other
> systems, there is no ambiguity.
>
>>
>>>> 3. If the timestamp does not include a time component, default to 0:00:0
>>>
>>> This is what Org mode does
>>>
>>>> for the local TZ
>>>
>>> This is what Org mode used to do, now it interprets it as 00:00:00 UTC.
>>>
>>>> 4. org-2ft should not enforce the UTC TZ. I agree this is incorrect.
>>>>
>>>> Rationale is that the user should have ability to fully control how
>>>> their timestamps are represented. However, adding the TZ is probably
>>>> just a pain for the majority of use cases, so defaulting to the local
>>>> (wall) TZ is OK provided you can override this consistently by adding
>>>> explicit TZ values.
>>>>
>>>> However, there is some devil in the details we need to work out. For
>>>> example, should we support both TZ names (like AEDT or Australia/Sydney)
>>>> and POSIX style +11/+11:00/+1100, should we add an option to tell org to
>>>> always add TZ info in timestamps which include time components and if
>>>> so, how complex will this need to be e.g. handle setting a future/past
>>>> timestamp which is in a different (daylight savings) offset and what
>>>> about the additional complexity in dealing with timestamp calculations
>>>> where dates might be in different offsets due to daylight savings -
>>>> while all quite possible, it does add significant complexity and this
>>>> may have adverse impact on performance. Not to say we shouldn't do it,
>>>> just that it will take significant work.
>>>>
>>>> I suspect just the first part won't have major impact - at least no more
>>>> than enforcing UTC in org-2ft.
>>>
>>> Org mode does not support TZ in time strings currently.  While I would
>>> like such a feature very much,
>>> adding TZ support isn't the topic of the current bug, just fixing how
>>> time strings are interpreted.
>>>
>>
>> We could make the decision that all org timestamps are relative to the
>> local TZ (ignoring the daylight savings issue) and that would probably
>> be OK for most users
>
> I agree, and think that this makes more sense than using UTC.
>
>> but I feel the whole issue is due to a lack of
>> adequate/consistent TZ interpretation.
>
> Also agreed, but I cannot volunteer for the task of adding that
> unfortunately.
>
>> The problem with org-2ft is that
>> it forces the conversion into UTC when the input time string may
>> actually be in local time, resulting in a number which is out by
>> whatever the local offset is.
>
> Exactly
>
>> When you then convert back, your string
>> representation will be out by at least that amount (possibly more
>> depending on whether TZ info is applied in the conversion back to a
>> string).
>
> TZ info is not applied in that way going back, since Unix
> timestamps are time zone agnostic.  It may affect how the string
> representation is formatted, but the time will not be even more
> incorrect.
>
> e.g., 2017-10-30T00:00:00+0000 vs 2017-10-30T07:00:00-0700.
> Different strings due to the time zone used, but they both
> describe the exact point of time as their input Unix timestamp.
>
>>
>> If it wasn't for the benefits of org files being essentially plain text
>> files, the easiest solution would possibly be to represent timestamps as
>> a number and use an overlay or similar to convert it to a human readable
>> value based on whatever the user sets as the timezone for
>> display/export. Unfortunately, this breaks one of the big benefits of
>> org-mode over other solutions i.e. the data is just plain text and can
>> be accessed/used by anything able to read plain text.
>>
>> Tim
>>
>>
>>>>
>>>> Tim
>>>>
>>>> P.S. when you start to think about it, it is easy to see how Java
>>>> screwed up this stuff so badly!
>>
>>
>> --
>> Tim Cross


--
Tim Cross