Idea for handling timezones

Idea for handling timezones

[Shironeko]

Hi everyone,

I, like many others on this list, have to move between timezones quite
frequently. As I gathered from the archive, it seems the main complexity in
supporting timezones is the difficulty revolving the change of timestamp format.
So I have an idea, suppose we add a new keyword, "TIMEZONE" that can be set at
the start of the file like so

	#+TIMEZONE: America/Toronto

This specifies the timezone of all timestamps in the file. Together with it,
there can be a function called e.g. org-shift-time, that shifts all the
timestamp in the file to another specified timezone and updates the keyword. Of
course, care needs to be taken when dealing with dates without time, e.g. it
should be treated as at time 00:00 when it is alone or as the start of a time
range, and be treated as at time 24:00 when it is the end of a time range.

Then there could be hooks that offer to run the function automatically when it
detects the user's system or emacs is set to a different timezone as in the file
(e.g. when they open the file, or opens the agenda). This will make sure the
timestamps always aligns with their current one (if they wish).

This change would be backwards compatible, and it should do the job well enough.

Regards,
shironeko

Idea for handling timezones

[shironeko]

Hi everyone,

I, like many others on this list, have to move between timezones quite
frequently. As I gathered from the archive, it seems the main complexity in
supporting timezones is the difficulty revolving the change of timestamp format.
So I have an idea, suppose we add a new keyword, "TIMEZONE" that can be set at
the start of the file like so

	#+TIMEZONE: America/Toronto

This specifies the timezone of all timestamps in the file. Together with it,
there can be a function called e.g. org-shift-time, that shifts all the
timestamp in the file to another specified timezone and updates the keyword. Of
course, care needs to be taken when dealing with dates without time, e.g. it
should be treated as at time 00:00 when it is alone or as the start of a time
range, and be treated as at time 24:00 when it is the end of a time range.

Then there could be hooks that offer to run the function automatically when it
detects the user's system or emacs is set to a different timezone as in the file
(e.g. when they open the file, or opens the agenda). This will make sure the
timestamps always aligns with their current one (if they wish).

This change would be backwards compatible, and it should do the job well enough.

Regards,
shironeko

Re: Idea for handling timezones

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1421 bytes --]

On Thu, Apr 01, 2021 at 07:40:47AM +0000, shironeko wrote:
> Hi everyone,
> 
> I, like many others on this list, have to move between timezones quite
> frequently. As I gathered from the archive, it seems the main complexity in
> supporting timezones is the difficulty revolving the change of timestamp format.
> So I have an idea, suppose we add a new keyword, "TIMEZONE" that can be set at
> the start of the file like so
> 
> 	#+TIMEZONE: America/Toronto
> 
> This specifies the timezone of all timestamps in the file [...]

Hm. Just a mumbling from the peanut gallery: isn't the timezone a property
of the timestamp itself?

Specifying the timezone for the whole file is progress, but imagine the
following scenario: I have a big file which is more or less a diary of
things which happened, with lots of timestamps thrown in (also LOGBOOK
entries).

If I move through timezones, only some of the timestamps are "elsewhere".

Switching "the whole file" to reflect the "current" timezone feels somehow
wrong to me (which timezone a specific timestamp "happened" in has also
some documentary value, after all).

To keep things unambiguous, more than just the timezone must be kept.
The current time offset is necessary to actually reconstruct the time
(DST and such things).

Of course, convincing Org to extend the timestamp format and regexps
might be a tough call :-)

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Idea for handling timezones

[Tim Cross]

shironeko <shironeko@tesaguri.club> writes:

> Hi everyone,
>
> I, like many others on this list, have to move between timezones quite
> frequently. As I gathered from the archive, it seems the main complexity in
> supporting timezones is the difficulty revolving the change of timestamp format.
> So I have an idea, suppose we add a new keyword, "TIMEZONE" that can be set at
> the start of the file like so
>
> 	#+TIMEZONE: America/Toronto
>
> This specifies the timezone of all timestamps in the file. Together with it,
> there can be a function called e.g. org-shift-time, that shifts all the
> timestamp in the file to another specified timezone and updates the keyword. Of
> course, care needs to be taken when dealing with dates without time, e.g. it
> should be treated as at time 00:00 when it is alone or as the start of a time
> range, and be treated as at time 24:00 when it is the end of a time range.
>
> Then there could be hooks that offer to run the function automatically when it
> detects the user's system or emacs is set to a different timezone as in the file
> (e.g. when they open the file, or opens the agenda). This will make sure the
> timestamps always aligns with their current one (if they wish).
>

I'm sorry, but I don't like this idea. In general, I think it is the
wrong approach and not sophisticated enough to work with the
complexities associated with timestamps.

This is actually a very hard problem and not one which can be adequately
addressed with something this simple. Problems include

1. Timzone alone is not sufficient. Offsets from UTC change due to
daylight savings times etc.

2. You can easily have timestamps from different timezones in the same
org file

3. Storing timestamps in local time is problematic because of the
inherent ambiguity this can have (again, due to daylight savings times
and what occurs at the 'cut over' time).

4. Sometimes, you may want the timestamp to reflect the date/time as it
was when recorded and don't want it to 'change' because your now viewing
it in a different timezone etc.

Personally, I think timestamp 'storage' and timestamp 'display' need to
be treated separately. I also think all relevant information (timezone,
offset) need to be stored with the timestamp. I also think the
fundamental base timestamp should be stored as UTC, allowing all time
calculations to be consistent (free of daylight savings time changes).
The user can then manage how the value is displayed by setting timezone
and offsets as appropriate (with perhaps the default being the local
system settings or whatever offset/tz was stored with the timestamp
itself). 

It is very difficult to predict or understand all the use cases for
timestamps. Therefore, any scheme must be extremely flexible. Experience
has taught me that one critical component is that at the lowest level,
many problems are avoided if the value is in UTC. Problem is, UTC is not
terribly human friendly. Luckily, this can largely be automated for many
common use cases. Unfortunately, it does also mean that if you are
someone who frequently moves between many timezones, your situation will
be more complicated. 

ne of the most frustrating parts of working with timestamps is daylight
saving times. This causes complications at so many levels. In
particular, I hate the fact change over dates often change and more
often than not, those changes are based around politics and at the whim
of politicians, which makes programatic handling more complex than it
needs to be.

-- 
Tim Cross

Re: Idea for handling timezones

[Samuel Wales]

org has capability of overlaying [or similar] tses with user-defined
something or other.  for locale type purposes.

perhaps, in principle, all tses for users who need tz could be in utc
or similar, and such overlays could be used to localize on the fly?

On 4/2/21, Tim Cross <theophilusx@gmail.com> wrote:
>
> shironeko <shironeko@tesaguri.club> writes:
>
>> Hi everyone,
>>
>> I, like many others on this list, have to move between timezones quite
>> frequently. As I gathered from the archive, it seems the main complexity
>> in
>> supporting timezones is the difficulty revolving the change of timestamp
>> format.
>> So I have an idea, suppose we add a new keyword, "TIMEZONE" that can be
>> set at
>> the start of the file like so
>>
>> 	#+TIMEZONE: America/Toronto
>>
>> This specifies the timezone of all timestamps in the file. Together with
>> it,
>> there can be a function called e.g. org-shift-time, that shifts all the
>> timestamp in the file to another specified timezone and updates the
>> keyword. Of
>> course, care needs to be taken when dealing with dates without time, e.g.
>> it
>> should be treated as at time 00:00 when it is alone or as the start of a
>> time
>> range, and be treated as at time 24:00 when it is the end of a time
>> range.
>>
>> Then there could be hooks that offer to run the function automatically
>> when it
>> detects the user's system or emacs is set to a different timezone as in
>> the file
>> (e.g. when they open the file, or opens the agenda). This will make sure
>> the
>> timestamps always aligns with their current one (if they wish).
>>
>
> I'm sorry, but I don't like this idea. In general, I think it is the
> wrong approach and not sophisticated enough to work with the
> complexities associated with timestamps.
>
> This is actually a very hard problem and not one which can be adequately
> addressed with something this simple. Problems include
>
> 1. Timzone alone is not sufficient. Offsets from UTC change due to
> daylight savings times etc.
>
> 2. You can easily have timestamps from different timezones in the same
> org file
>
> 3. Storing timestamps in local time is problematic because of the
> inherent ambiguity this can have (again, due to daylight savings times
> and what occurs at the 'cut over' time).
>
> 4. Sometimes, you may want the timestamp to reflect the date/time as it
> was when recorded and don't want it to 'change' because your now viewing
> it in a different timezone etc.
>
> Personally, I think timestamp 'storage' and timestamp 'display' need to
> be treated separately. I also think all relevant information (timezone,
> offset) need to be stored with the timestamp. I also think the
> fundamental base timestamp should be stored as UTC, allowing all time
> calculations to be consistent (free of daylight savings time changes).
> The user can then manage how the value is displayed by setting timezone
> and offsets as appropriate (with perhaps the default being the local
> system settings or whatever offset/tz was stored with the timestamp
> itself).
>
> It is very difficult to predict or understand all the use cases for
> timestamps. Therefore, any scheme must be extremely flexible. Experience
> has taught me that one critical component is that at the lowest level,
> many problems are avoided if the value is in UTC. Problem is, UTC is not
> terribly human friendly. Luckily, this can largely be automated for many
> common use cases. Unfortunately, it does also mean that if you are
> someone who frequently moves between many timezones, your situation will
> be more complicated.
>
> ne of the most frustrating parts of working with timestamps is daylight
> saving times. This causes complications at so many levels. In
> particular, I hate the fact change over dates often change and more
> often than not, those changes are based around politics and at the whim
> of politicians, which makes programatic handling more complex than it
> needs to be.
>
> --
> Tim Cross
>
>


-- 
The Kafka Pandemic

Please learn what misopathy is.
https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html

Re: Idea for handling timezones

[Shironeko]

[-- Attachment #1: Type: text/plain, Size: 1405 bytes --]

On Fri, 2021-04-02 at 13:34 +0200, tomas@tuxteam.de wrote:
> On Thu, Apr 01, 2021 at 07:40:47AM +0000, shironeko wrote:
> 
> Hm. Just a mumbling from the peanut gallery: isn't the timezone a property
> of the timestamp itself?
> 
> Specifying the timezone for the whole file is progress, but imagine the
> following scenario: I have a big file which is more or less a diary of
> things which happened, with lots of timestamps thrown in (also LOGBOOK
> entries).
> 
> If I move through timezones, only some of the timestamps are "elsewhere".
> 

There are separate hacks for that, see
https://emacs.stackexchange.com/questions/13463/specify-timezone-in-org-date-format

> Switching "the whole file" to reflect the "current" timezone feels somehow
> wrong to me (which timezone a specific timestamp "happened" in has also
> some documentary value, after all).
> 
> To keep things unambiguous, more than just the timezone must be kept.
> The current time offset is necessary to actually reconstruct the time
> (DST and such things).

This is why timezones need to be specified in the tz database format, it does
all the right things and makes sure the converted timestamp corresponds to the
same instant. 

> Of course, convincing Org to extend the timestamp format and regexps
> might be a tough call :-)

This is exactly the problem I'm trying to avoid.

Regards,
shiro

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Idea for handling timezones

[Shironeko]

On Sat, 2021-04-03 at 10:37 +1100, Tim Cross wrote:
> 
> 1. Timzone alone is not sufficient. Offsets from UTC change due to
> daylight savings times etc.
> 
> 2. You can easily have timestamps from different timezones in the same
> org file
> 
> 3. Storing timestamps in local time is problematic because of the
> inherent ambiguity this can have (again, due to daylight savings times
> and what occurs at the 'cut over' time).
> 
> 4. Sometimes, you may want the timestamp to reflect the date/time as it
> was when recorded and don't want it to 'change' because your now viewing
> it in a different timezone etc.

1 and 3 is addressed by the use of tz database, it makes sure the timezone
conversion is lossless. 2 and 4 is really not the target for this proposal, this
feature is completely optional and this is really meant to solve the "I want to
see when I need to get my tasks done" which is a particular headache when there
is timezone involved.

> Personally, I think timestamp 'storage' and timestamp 'display' need to
> be treated separately. I also think all relevant information (timezone,
> offset) need to be stored with the timestamp. I also think the
> fundamental base timestamp should be stored as UTC, allowing all time
> calculations to be consistent (free of daylight savings time changes).
> The user can then manage how the value is displayed by setting timezone
> and offsets as appropriate (with perhaps the default being the local
> system settings or whatever offset/tz was stored with the timestamp
> itself). 
> 
> It is very difficult to predict or understand all the use cases for
> timestamps. Therefore, any scheme must be extremely flexible. Experience
> has taught me that one critical component is that at the lowest level,
> many problems are avoided if the value is in UTC. Problem is, UTC is not
> terribly human friendly. Luckily, this can largely be automated for many
> common use cases. Unfortunately, it does also mean that if you are
> someone who frequently moves between many timezones, your situation will
> be more complicated. 
> 
> ne of the most frustrating parts of working with timestamps is daylight
> saving times. This causes complications at so many levels. In
> particular, I hate the fact change over dates often change and more
> often than not, those changes are based around politics and at the whim
> of politicians, which makes programatic handling more complex than it
> needs to be.
> 

yes, this is why I want to avoid changing the timestamp itself, since that will
never lead to working solutions soon.

Regards,
shiro

Re: Idea for handling timezones

[Samuel Wales]

what i proposed is this.  which uses text properties.  it might not
suit your needs, but might be a workaround.  at least it is a
brainstorm.  suitable for wrapping fish.

1] convert all your tses to utc [exercise for the reader]
2] make org's idea of time be utc [there /might/ be code]
3] make org display local time and tz [see below]

3 is customizable: (info "(org) Custom time format")


On 4/2/21, Shironeko <shironeko@tesaguri.club> wrote:
> On Sat, 2021-04-03 at 10:37 +1100, Tim Cross wrote:
>>
>> 1. Timzone alone is not sufficient. Offsets from UTC change due to
>> daylight savings times etc.
>>
>> 2. You can easily have timestamps from different timezones in the same
>> org file
>>
>> 3. Storing timestamps in local time is problematic because of the
>> inherent ambiguity this can have (again, due to daylight savings times
>> and what occurs at the 'cut over' time).
>>
>> 4. Sometimes, you may want the timestamp to reflect the date/time as it
>> was when recorded and don't want it to 'change' because your now viewing
>> it in a different timezone etc.
>
> 1 and 3 is addressed by the use of tz database, it makes sure the timezone
> conversion is lossless. 2 and 4 is really not the target for this proposal,
> this
> feature is completely optional and this is really meant to solve the "I want
> to
> see when I need to get my tasks done" which is a particular headache when
> there
> is timezone involved.
>
>> Personally, I think timestamp 'storage' and timestamp 'display' need to
>> be treated separately. I also think all relevant information (timezone,
>> offset) need to be stored with the timestamp. I also think the
>> fundamental base timestamp should be stored as UTC, allowing all time
>> calculations to be consistent (free of daylight savings time changes).
>> The user can then manage how the value is displayed by setting timezone
>> and offsets as appropriate (with perhaps the default being the local
>> system settings or whatever offset/tz was stored with the timestamp
>> itself).
>>
>> It is very difficult to predict or understand all the use cases for
>> timestamps. Therefore, any scheme must be extremely flexible. Experience
>> has taught me that one critical component is that at the lowest level,
>> many problems are avoided if the value is in UTC. Problem is, UTC is not
>> terribly human friendly. Luckily, this can largely be automated for many
>> common use cases. Unfortunately, it does also mean that if you are
>> someone who frequently moves between many timezones, your situation will
>> be more complicated.
>>
>> ne of the most frustrating parts of working with timestamps is daylight
>> saving times. This causes complications at so many levels. In
>> particular, I hate the fact change over dates often change and more
>> often than not, those changes are based around politics and at the whim
>> of politicians, which makes programatic handling more complex than it
>> needs to be.
>>
>
> yes, this is why I want to avoid changing the timestamp itself, since that
> will
> never lead to working solutions soon.
>
> Regards,
> shiro
>
>
>


-- 
The Kafka Pandemic

Please learn what misopathy is.
https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html

Re: Idea for handling timezones

[tomas]

[-- Attachment #1: Type: text/plain, Size: 2468 bytes --]

On Sat, Apr 03, 2021 at 08:36:06AM +0800, Shironeko wrote:
> On Fri, 2021-04-02 at 13:34 +0200, tomas@tuxteam.de wrote:
> > On Thu, Apr 01, 2021 at 07:40:47AM +0000, shironeko wrote:
> > 
> > Hm. Just a mumbling from the peanut gallery: isn't the timezone a property
> > of the timestamp itself?
> > 
> > Specifying the timezone for the whole file is progress, but imagine the
> > following scenario: I have a big file which is more or less a diary of
> > things which happened, with lots of timestamps thrown in (also LOGBOOK
> > entries).
> > 
> > If I move through timezones, only some of the timestamps are "elsewhere".
> > 
> 
> There are separate hacks for that, see
> https://emacs.stackexchange.com/questions/13463/specify-timezone-in-org-date-format

Thanks for the pointer [1].

Yes, Org is a bit scared to touch time format, and in a way, I do
understand that.

I'm unhappy myself with Org timestamps, but given how conflict-laden
the topic is, I opted for keeping my local hacks and dealing with
some fallout on version changes. Not optimal, but I haven't better
ideas myself currently.

> > Switching "the whole file" to reflect the "current" timezone [...]

> This is why timezones need to be specified in the tz database format, it does
> all the right things and makes sure the converted timestamp corresponds to the
> same instant. 

I don't understand what you mean by "the tz database format" in this
context. What I meant is the '%z' information (e.g. -0400) instead
of (or in addition to) the '%Z' one (e.g. EDT), in the sense of
`format-time-string'.

But perhaps we mean the same.

> > Of course, convincing Org to extend the timestamp format and regexps
> > might be a tough call :-)
> 
> This is exactly the problem I'm trying to avoid.

Yes, I get the "design tension" there. I don't have better ideas.
A greater flexibility in timestamp representation would have seduced
me; a file-global setting... not so much.

Still, I wish you good luck :-)

Cheers

[1] BTW I hate SX. Theycl've come up with yet another of those cookie
   tortures to take revenge on the users for some mild privacy protection
   laws. Of course I don't want any cookies. You @#&$s don't even have
   to ask: you @#*&%$ can infer that from the fact that my browser doesn't
   accept cookies in the first place!  You so-called web programmers
   manage to elicit feelings in me I'm less-than-proud of.

 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Idea for handling timezones

[shironeko]

see https://en.m.wikipedia.org/wiki/Tz_database

Re: Idea for handling timezones

[tomas]

[-- Attachment #1: Type: text/plain, Size: 648 bytes --]

On Sat, Apr 03, 2021 at 08:03:18AM +0000, shironeko wrote:
> see https://en.m.wikipedia.org/wiki/Tz_database

That's where I looked at (OK, without the "m" in the URL ;-)

But I still don't understand what you are up to, so it might
be we are talking past each other.

What's described there is a database format for DST rules across
all time zones, each of the latter encoded as an area/location
pair.

What we are talking about is a specifier to a time stamp to
take the ambiguity wrt the time zone this stamp is "written"
in; my point is that this area/location format mentioned there
(if /this/ is what you mean) isn't sufficient.

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Idea for handling timezones

[Shironeko]

[-- Attachment #1: Type: text/plain, Size: 1727 bytes --]

tz database and tools using it like the date(1) command can convert between
local times taking into account all the difference like daylight savings time.
An example below, Toronto observes daylight savings whereas Shanghai does not:

$ TZ=Asia/Shanghai date --date='TZ="America/Toronto" 2021-11-07 Sun 00:30'
Sun  7 Nov 12:30:00 CST 2021
$ TZ=Asia/Shanghai date --date='TZ="America/Toronto" 2021-11-07 Sun 01:30'
Sun  7 Nov 13:30:00 CST 2021
$ TZ=Asia/Shanghai date --date='TZ="America/Toronto" 2021-11-07 Sun 02:30'
Sun  7 Nov 15:30:00 CST 2021

$ TZ=America/Toronto date --date='TZ="Asia/Shanghai" 2021-11-07 Sun 12:30'
Sun  7 Nov 00:30:00 EDT 2021
$ TZ=America/Toronto date --date='TZ="Asia/Shanghai" 2021-11-07 Sun 13:30'
Sun  7 Nov 01:30:00 EDT 2021
$ TZ=America/Toronto date --date='TZ="Asia/Shanghai" 2021-11-07 Sun 14:30'
Sun  7 Nov 01:30:00 EST 2021
$ TZ=America/Toronto date --date='TZ="Asia/Shanghai" 2021-11-07 Sun 15:30'
Sun  7 Nov 02:30:00 EST 2021

The ambiguity caused by dialing back is resolved with
$ TZ=Asia/Shanghai date --date='TZ="America/Toronto" 2021-11-07 Sun 01:30 EDT'
Sun  7 Nov 13:30:00 CST 2021
$ TZ=Asia/Shanghai date --date='TZ="America/Toronto" 2021-11-07 Sun 01:30 EST'
Sun  7 Nov 14:30:00 CST 2021

With this, say the user have

#+TIMEZONE: America/Toronto

at the start of their org file, and they moved to Shanghai, all the timestamp in
the org file is converted using something equivalent to

$ TZ=Asia/Shanghai date --date='TZ="America/Toronto" '"$TIMESTAMP"

and the file header changed to

#+TIMEZONE: Asia/Shanghai

when they get back the timestamp is returned with
$ TZ=America/Toronto date --date='TZ="Asia/Shanghai" '"$TIMESTAMP"

shiro

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Idea for handling timezones

[Greg Minshall]

hi, Shiro,

> With this, say the user have
> 
> #+TIMEZONE: America/Toronto
> 
> at the start of their org file, and they moved to Shanghai, all the timestamp in
> the org file is converted using something equivalent to
> 
> $ TZ=Asia/Shanghai date --date='TZ="America/Toronto" '"$TIMESTAMP"
> 
> and the file header changed to
> 
> #+TIMEZONE: Asia/Shanghai
> 
> when they get back the timestamp is returned with
> $ TZ=America/Toronto date --date='TZ="Asia/Shanghai" '"$TIMESTAMP"

i can imagine buffer-wide time zone settings, especially if one has
multiple .org files, could become hard to manage.  i don't know the
issues with evolving towards a syntax where time zone information is
embedded in time stamps themselves, but i wonder if that might be more
manageable.

i *can* imagine that if needed, one might do one's own hack, such as
your suggestion, while that evolution progresses.  in particular, issues
about entering, and displaying, time stamps, which need to be solved in
either case, could then be tackled.  i would think.

cheers, Greg

Re: Idea for handling timezones

[tomas]

[-- Attachment #1: Type: text/plain, Size: 1131 bytes --]

On Sat, Apr 03, 2021 at 05:26:43PM +0800, Shironeko wrote:
> tz database and tools using it like the date(1) command can convert between
> local times taking into account all the difference like daylight savings time.
> An example below, Toronto observes daylight savings whereas Shanghai does not:

[...]

> The ambiguity caused by dialing back is resolved with
> $ TZ=Asia/Shanghai date --date='TZ="America/Toronto" 2021-11-07 Sun 01:30 EDT'
> Sun  7 Nov 13:30:00 CST 2021
> $ TZ=Asia/Shanghai date --date='TZ="America/Toronto" 2021-11-07 Sun 01:30 EST'
> Sun  7 Nov 14:30:00 CST 2021

I understand that. Yo've got to specify (EDT vs EST) whether summer time was in
effect when taking the time stamp. But then, why not specify the offset right
away? Less dependency on "ambient" information. No dependency at all on whether
the database has changed [1] under you. And so on.

Of course, the ideal would be to keep time in UTC and display accordingly to
the time zone (ideally also, provided with each time stamp). But I fear this
would be pushing things too far...

Cheers

[1] This Shouldn't Happen (famous last words ;-)

 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Idea for handling timezones

[Shironeko]

[-- Attachment #1: Type: text/plain, Size: 270 bytes --]

The only ambiguous timestamp is the hour when exiting DST (dialing back), and
that time is usually set to when there is not much going on (like midnight). all
other time do not need to specify whether its DST or not, which fits in with the
current timestamp format.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Idea for handling timezones

[Shironeko]

[-- Attachment #1: Type: text/plain, Size: 773 bytes --]

The reason there is such a database is precisely that it is meant to change,
leap seconds are irregular and other laws change all the time (perhaps DST gets
removed (please)).

Setting all timestamps to UTC is the ideal solution for sure (perhaps as an
option) and other things that deal with org files need to convert it to local
time when use. But that's a big adventure that I'm not prepared for :( I don't
even know how many places need to be changed.

I would like to at least try implementing my idea and see if it works, but I'm
still very new to org. So it would be great if there are some pointers. Like how
would I go about finding all the timestamps and make changes. Perhaps somewhere
in org that does something similar.

Regards,
shironeko



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: Idea for handling timezones

[Russell Adams]

On Sat, Apr 03, 2021 at 02:23:34PM +0300, Greg Minshall wrote:
> > #+TIMEZONE: America/Toronto
> >
> > at the start of their org file, and they moved to Shanghai, all the timestamp in
> > the org file is converted using something equivalent to

I think what you're saying here is that timestamps without timezone
information should have a default timezone. I'd suggest there is a
global default setting (ie: init file), and allow a buffer local
override. Essentially all timestamps that don't specify a timezone
should fall back to the local TZ, unless there's a buffer override.

I'd be in favor of revisiting the idea of putting timezone information
into the timestamp. I know it's a deep change, but this is a kind of
incremental growth we should expect to a core feature. I frequently
fight this issue myself with meetings across timezones.

I would not suggest using UTC. I believe one of the reasons timestamps
didn't include TZ information was to keep them short and human
legible. Solutions with overlays to change a timestamp reduce the
usefulness of the plain text reading of Org (ie: less, grep,
etc). Storing timestamps with UTC is really a shortcut for the
computer, not the user.

I was just reading about Emacs' parse-time-string function, and Emacs
already has TZ conversion built in to many of the time functions. It
seems to me that we could fall back to the Emacs parser if needed.

https://www.gnu.org/software/emacs/manual/html_node/elisp/Time-Parsing.html

Today's timestamps are in the form of "YYYY-MM-DD Day HH:MM". I've
often wondered that the day name is in there, other than for human
legibility. Given timestamps are always wrapped in <> or [], for
active and inactive timestamps accordingly, parsing for a new element
at the end by time zone name doesn't sound so bad.

Staying with user friendly, I think time zone names would be more
useful than delta syntax from UTC. [2021-04-03 Sat 16:56 CEST] doesn't
sound too bad, given the timezones aren't expected to be in every
timestamp.

I really think that the key issue making adoption difficult will be
all the tooling reading these, not the timestamps themselves.

------------------------------------------------------------------
Russell Adams                            RLAdams@AdamsInfoServ.com

PGP Key ID:     0x1160DCB3           http://www.adamsinfoserv.com/

Fingerprint:    1723 D8CA 4280 1EC9 557F  66E8 1154 E018 1160 DCB3

Re: Idea for handling timezones

[Greg Minshall]

Russell,

> I would not suggest using UTC. I believe one of the reasons timestamps
> didn't include TZ information was to keep them short and human
> legible. Solutions with overlays to change a timestamp reduce the
> usefulness of the plain text reading of Org (ie: less, grep,
> etc). Storing timestamps with UTC is really a shortcut for the
> computer, not the user.

i wonder if this is perhaps another place where "org mode as simple
text" conflicts with "org mode to solve difficult problems".  that comes
up from time to time (using dollar signs as delimiters for math is one
recent example, another is "-" being "filled" into column one and taken
as an item in a new list).

it's a hard line to straddle.

Greg

Re: Idea for handling timezones

[Samuel Wales]

the suggestion to change all your tses to utc was not a general
suggestion for all org users or a change to org.  for op only.

but the point about grep and such then breaking is a good one.  it is
possibly not a useful workaround for the op.

On 4/3/21, Greg Minshall <minshall@umich.edu> wrote:
> Russell,
>
>> I would not suggest using UTC. I believe one of the reasons timestamps
>> didn't include TZ information was to keep them short and human
>> legible. Solutions with overlays to change a timestamp reduce the
>> usefulness of the plain text reading of Org (ie: less, grep,
>> etc). Storing timestamps with UTC is really a shortcut for the
>> computer, not the user.
>
> i wonder if this is perhaps another place where "org mode as simple
> text" conflicts with "org mode to solve difficult problems".  that comes
> up from time to time (using dollar signs as delimiters for math is one
> recent example, another is "-" being "filled" into column one and taken
> as an item in a new list).
>
> it's a hard line to straddle.
>
> Greg
>
>


-- 
The Kafka Pandemic

Please learn what misopathy is.
https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html

Re: Idea for handling timezones

[Tim Cross]

Greg Minshall <minshall@umich.edu> writes:

> Russell,
>
>> I would not suggest using UTC. I believe one of the reasons timestamps
>> didn't include TZ information was to keep them short and human
>> legible. Solutions with overlays to change a timestamp reduce the
>> usefulness of the plain text reading of Org (ie: less, grep,
>> etc). Storing timestamps with UTC is really a shortcut for the
>> computer, not the user.
>
> i wonder if this is perhaps another place where "org mode as simple
> text" conflicts with "org mode to solve difficult problems".  that comes
> up from time to time (using dollar signs as delimiters for math is one
> recent example, another is "-" being "filled" into column one and taken
> as an item in a new list).
>
> it's a hard line to straddle.
>

I think you are correct. There are fairly frequent issues associated
with timestamps and much of it is due to the tension between trying to
have something which is human friendly and having something which lends
itself to being used for calculations or in an environment where the
timezone regularly changes (i.e. from someone who travels a lot).

Adding timezone infomation to timestamps by default is not difficult.
Org uses Emacs' time handling functions under the hood and supports
setting custom timestamp formats. If I was someone who regularly moved
between timezones, I would definitely modify the default format to
ensure timezone information is recorded with the timestamp. This would
at least let the user know what timezone was being referenced when the
timestamp was created and what needs to be done to (even mentally)
convert it to whatever the current timezone is.

For any calculations involving timestamps, the only sane way to do
things is to convert all timestamps to UTC, perform the calculation and
if necessary, convert back to localtime for final result. It was
mentioned elsewhere in this thread that issues associated with DST
changes are OK because they occur early in the morning and not much
happens around then. However, this ignores use cases that depend on
calculation of time intervals. Provided all calculations are performed
using UTC, everything should work as long as timestamps include TZ info.

I don't like the idea of having a header variable which records the
timezone. I think this will be a source of errors and confusion and just
won't work well for many setups (for example, people like me who have
many org files with timestamps in them).

I feel a better result would be to

- Update default timestamp format to include timezones

- Add a new function which would either (or all of)
   - Convert an existing timestamp to a specified tz
   - Convert all timestamps in a file to a specific tz
   - Convert all timestamps in agenda files to a specific tz
   - Convert all timestamps in all org files to a specific tz
  User can then run the function as required (for example, after
  changing timezone location). 

- Ensure all exporter back ends support the ability to set an export
 timestamp function, allowing changing/stripping of TZ data during export
 as you frequently want a more concise format in exported documents. I
 believe this is already supported to some extent. 

The other things we could probably add is a timestamp display
tooltip/overlay which could be defined to popup when the mouse/cursor is
on the timestamp and which would display the timestamp in some timezone
which the user could specify as an option and which defaults to whatever
the local timezone is at the time. Then, when you see a timestamp which
is in lets say US EDT and your in Tokyo, moving your cursor or placing
your mouse on that timestamp would display a tooltip showing the local
time equivalent. 

-- 
Tim Cross

Re: Idea for handling timezones

[Tom Gillespie]

Time zones are tricky, but there are some requirements that make it
possible to rule out many apparent solutions because they violate some
critical invariant. For example. Timezone cannot be scoped to the
file. This is because many users (myself included) have a single org
file that is used in and across multiple timezones. Thus the original
proposal in this thread to have a #+timezone: keyword is insufficient
for many use cases, and can lead to significant confusion and bad
data.

Similarly it is probably a bad idea to always use UTC because even
though it will technically always be correct (while on Earth ...), it
means that the user will need to have kept track of their spatial
coordinates in addition, and in addition there will be an eternal
dependency on the timezone database in order to correctly reconstruct
what the timezone would have been. All in all it is better to assume
that the user has correctly configured their clock for their needs,
and record the timestamp with timezone offset (NOTE textual timezones
cannot be used in timestamps for a variety of reasons not the least of
which is that they are ambiguous, see CST for example). Sadly, the
ISO8601 timestamp specification for a negative timezone offset uses a
hyphen minus, the same as the date separator, so that baggage is going
to be with us for the rest of time (at least until someone comes up
with something better than ISO8601), but at least it requires no
additional information to capture all the information needed to
correctly reconstruct the time that it stamped.

Finding a timestamp format that has a regular grammar, is invariant to
changes in the location and time of the user (What if the user is on
Mars? What happens after year 9999? What happens if someone needs to
reference a date in the far future? What if someone wants to mention a
historical date e.g. 1000BCE?), doesn't require external information,
and is also somewhat human readable, is a major challenge. If we could
serialize something like the unix epoch into a file and render it
differently in the buffer that might work, however that defeats one of
the major points of using org as a plain text format.

My proposal at the moment based on the constraints imposed by the
current timestamp format that includes the repeat or delay syntax
would be the following (date and time and day are separated by a
space)

date: ([+-][0-9]\+|[0-9]{4})(-[0-9]{2}){2}
time: ([0-9]{2}:[0-9]{2}(:[0-9]{2}(,[0-9]{1,9})?)?[+-][0-9]{2}:[0-9]{2})?
day: ([a-zA-Z0-9]\+)?

followed by a space and then the repeat or delay for example
[+10000-01-01 10:11:00,992315771-04:00 Sat] or
[-480-08-20 05:00+01:00]
or more temporally local
[2021-03-03 17:43-07:00 Sat]

This is the closest I have been able to get while working through
formalizing all of org syntax and trying to come up with more robust
solutions for timestamps.
For comparison see org-ts-regex and friends in org.el. I have also not
come up with a better solution for the repeat or delay syntax, though
ISO8601 interval
specification might be an option. Similarly there are extensions for
dealing with uncertain dates and times, but I don't have good
proposals for those right now,
and the use cases are also somewhat out of scope.

Best,
Tom

Re: Idea for handling timezones

[Maxim Nikulin]

The notes below are quite general and unrelated to particular message.

On 04/04/2021 07:51, Tom Gillespie wrote:
> 
> followed by a space and then the repeat or delay for example
> [+10000-01-01 10:11:00,992315771-04:00 Sat] or
> [-480-08-20 05:00+01:00]
> or more temporally local
> [2021-03-03 17:43-07:00 Sat]

Just an idea, I do not know if it is feasible and convenient. Have 
anyone considered a kind of src blocks to describe timestamps? Current 
inline syntax is suitable for simple cases, sometimes more details are 
required, some information could be redundant to allow consistency check:

- Date-time in the original form as it appeared in external source (with 
a hint related to particular format e.g. rfc822 date.
- Either it is local time or it is bound to some event as Solar eclipse.
- List if time zones to have notion of local time of all participants of 
an online meeting.
- Location for planned event since there is a chance that existing time 
zone will be split into smaller ones.

A couple of bookmarks to get impression of complexity:
- https://stackoverflow.com/tags/timezone/info
   timezone Tag Info
- 
https://infiniteundo.com/post/25326999628/falsehoods-programmers-believe-about-time
   Falsehoods programmers believe about time
   and follow-ups.

Emacs API is hardly suitable for working with multiple time zones. 
Something could be borrowed from other projects, e.g.
https://howardhinnant.github.io/date_algorithms.html
Low-Level Date Algorithms (C++)
A special attribute has been added in python to deal with local time 
ambiguity around time transitions
https://docs.python.org/3/library/datetime.html#datetime.datetime.fold
https://www.python.org/dev/peps/pep-0495/
PEP 495 -- Local Time Disambiguation

I think, it is a challenge to provide a convenient way to generate 
agenda for a trip across several time zones.

P.S. Raw UNIX timestamp as 1234567890 is convenient for some 
computations but hardly human friendly. UTC date-time 
2009-02-13T23:31:30Z is better. With particular offset 
2009-02-13T22:31:30-01:00 it is equally precise and even more 
convenient. Still neither of such forms are applicable if it is 
scheduled event with fixed local time and offset of particular timezone 
might be changed.

Re: Idea for handling timezones

[Shironeko]

Hi all,

I originally thought having the timezone in the header of the file would make
things simpler, since it meant all the timestamp would be in the same timezone,
rather than potentially different ones. But it seems that that might not be
inline with how others use their org files.

org actually already has a keyword for a per entry timezone, 
https://orgmode.org/manual/iCalendar-Export.html
The problem with it is that no other facility of org plugs into that, it only
counts when exporting into other calendar systems, which I think is strange. If
it is indeed better to be able to set timezone per entry, then I think
supporting this properly in org would be best.

Only having CST etc timezone abbreviations is indeed problematic, because it is
ambiguous and kinda difficult to enter, also people don't necessarily
communicate with the correct designation (maybe they mean EDT but used EST, but
the time is unambiguously EDT).

With repeat tasks I don't think it would be a problem for tasks that repeats
daily or longer (since by definition they add to the day/month/year directly,
not counting how many days there are in a month for example). For hourly
repeating tasks it is indeed ambiguous, but I don't have an idea on what would
be better. Maybe people that have those can chime in.

Best,
Shironeko

Re: Idea for handling timezones

[tomas]

[-- Attachment #1: Type: text/plain, Size: 2204 bytes --]

On Fri, Apr 23, 2021 at 09:45:14AM +0800, Shironeko wrote:
> Hi all,
> 
> I originally thought having the timezone in the header of the file would make
> things simpler, since it meant all the timestamp would be in the same timezone,
> rather than potentially different ones. But it seems that that might not be
> inline with how others use their org files.
> 
> org actually already has a keyword for a per entry timezone, 
> https://orgmode.org/manual/iCalendar-Export.html
> The problem with it is that no other facility of org plugs into that, it only
> counts when exporting into other calendar systems, which I think is strange. If
> it is indeed better to be able to set timezone per entry, then I think
> supporting this properly in org would be best.

Interesting idea. If I understand the doc correctly, the iCalendar
exporter picks up timezone information from a property of the
"enclosing" heading, if present.

> Only having CST etc timezone abbreviations is indeed problematic, because it is
> ambiguous and kinda difficult to enter, also people don't necessarily
> communicate with the correct designation (maybe they mean EDT but used EST, but
> the time is unambiguously EDT).

They seem more fragile than time offsets, yes. Much implicit environment.
Some folks seem to still have strong preferences for that (not me, mind
you).

> With repeat tasks I don't think it would be a problem for tasks that repeats
> daily or longer (since by definition they add to the day/month/year directly,
> not counting how many days there are in a month for example). For hourly
> repeating tasks it is indeed ambiguous, but I don't have an idea on what would
> be better. Maybe people that have those can chime in.

Actually, repeating tasks are clear use cases for those "mushy" time
zones: "we meet every second and fourth Thursday every month at 19h30" 
almost surely means "at whatever time the local clocks say it is half
past seven in the afternoon", so daylight saving corrections will
apply.

Decisions, decisions :-)

Don't get me wrong: I'm not trying to discourage you to pick that up,
I'm happy someone has the spirit to do so!

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]