Re: [FEATURE REQUEST] Timezone support in org-mode datestamps and org-agenda

[Ihor Radchenko <yantar92@posteo.net>]

Sterling Hooten <hooten@gmail.com> writes:

> This is an initial glossary compiled from various standards and sources;
> it's incomplete, probably incorrect, and open to critique, but is useful
> in articulating a possible road map forward.

Do I understand correctly that the terms are simply taken from ISO (https://www.iso.org/obp/ui/#iso:std:iso:8601:-1:ed-1:v1:en)? 

> What does format does Org have now?
>
> • The format currently in use for timestamps is floating, field-based,
>   and has a resolution/precision of minutes.
>
> What kinds of representations would a calendar system capable of
> handling timezones require?
>
> • Instant (fixed)
>   • This is referring to an unambiguous moment in time
>   • e.g., 2007-02-03T05:00:00.000Z
> • Offset (fixed)
>   • This captures the idea of "when did it happen for the person who
>     made the observation"
>   • e.g., 2007-02-03T04:00:00.000+01:00
> • Instant with explicit offset and zone (fixed)
>   • e.g., 2007-01-01T02:00:00.000+01:00[America/Chicago]
> • Zoned local date time (floating)
>   • Tricky, requires decisions about how to interpret timestamps after
>     political changes.
>   • e.g., 2007-01-01T01:00:00.000[America/Chicago]

Note that representing the time zone does not require second/sub-second
precision. Let's not complicate the matters.

Also, [time zone] is not compatible with Org's delimiting syntax. We
need to come up with something else.

> What would a roadmap be?
>
> • Design and implement an absolute and offset timestamp system
>   • Decide on a time scale

UTC. Because we are bound by capabilities of Emacs time API.

>   • Decide on a format and syntax

Sure.

>   • Implement instant timestamps
>   • Implement offeset timestamps

This is all available via Emacs time API as implemented in glibc.
Time db is supported. Time zone names are supported. Fixed offsets are
also supported via TZ POSIX standard
(https://www.gnu.org/software/libc/manual/html_node/TZ-Variable.html).

> • Design and implement the time zone aware calendar system This is a
>   separate project.

Org support for time zones is exactly the calendar system you are
talking about. This will be most of the work TBD.

> What format and syntax should Org use?
>
> A heretical suggestion: We should abandon the day of week abbreviation
> and use a new format.

As you saw from a number of replies, people do find week abbreviation
useful.

> The current format generates a three leter abbreviation of the day of
> the week [2023-01-25 Wed 12:12]. I suggest supporting this as a
> legacy/simple format but switch to a format/encoding like
> [2023-01-25T15:13:42Z] for the new system. Specifically I'm advocating
> for an extended ISO 8601 format, compatible with expanded dates and
> Level 2 of the EDTF, with some (bracket?) notation surrounding it such
> that Org can parse the syntax as a timestamp. I advocate further for the
> use of durations and repeating intervals to follow the same standard
> format. Thus instead of a range being formatted as:
>
> [2023-01-25 Wed 13:57]–[2023-01-26 Thu 13:57]
>
> it would be:
>
> [2023-01-25T16:57:42Z/2023-01-26T16:57:42Z].

As I (and others) replied, we will not do it. We aim for human-readable
representation. Hence, while we can draw ideas from EDTF and ISO, we
will not follow them precisely.

> If the square bracket delimiter syntax is insufficient or too difficult
> to parse unambiguously, we could just encapsulate the ISO format in a
> sub-syntax (e.g., [&&(ISO format)] similar to the [%%(diary sexp)]
> technique). This is ugly, but perhaps a stepping stone during
> development to separate syntax parsing concerns from calculating etc.

This would be a breaking change that will force all the libraries adapt.
If we need supporting raw ISO syntax at any point, it can be simply made
a subset of [%%(diary sexp)]. At the end, diary sexps are nothing but
Elisp functions. We don't need to invent yet another syntax for ISO.

> What are the advantages of switching to a standard format for the new
> system?
> ...
> • We have a way of distinguishing new timestamps from legacy/simple ones
>   By making a change in syntax we reduce (or eliminate?) the possibility
>   of ambiguity between "which version" of a timestamp is being parsed. A
>   legacy timestamp can be treated as such, and new timestamps are easily
>   identified by the 'T' present instead of spaces, or in the delimiters
>   wrapping the representation.

We should not introduce this problem to start with. The versions should
be mutually compatible if at all possible. Note, for example, that
<2023-01-30 Mon 14:00 @Europe/Berlin> does not even break the existing
Org versions, except that time zone part is ignored.

I am strongly against introducing a brand-new timestamp syntax and
abandoning (or maintaining infinitely) the current one.

Proliferation of multiple parallel syntax variants is something we
really want to avoid maintenance-wise.

> • We free ourselves from the constraints of the legacy timestamp format
>   Trying to engineer a new syntax which also parses as an extension of
>   the legacy one is more complex and embeds things like "day of the
>   week" and the use of spaces as separators into this new system. Easier
>   to have two side by side.

This should only be the last resort. We are not quite there.

> • We can defer to existing parsing and calculating systems There are
>   already programs written which support the ISO standard and EDTF.

No. We can only defer the existing parsing to Elisp time API. Which does
not support EDTF. And supporting approximate dates from EDTF would be
hard to implement anyway - do we really need those?

>   • We can directly (or nearly directly) import the regular expressions
>     and parsing mechanisms already written.

It's not like these regular expressions are particularly complex. And we
are not going to make them complex either way.

>   • These enable decent testing suites as we build the system, as we can
>     check against existing packages to see if our parsing and
>     calculations agree.

Or we can write a simple converter from Org syntax to ISO and do the
same. Though I do not know any existing testing suite we could utilize
here.

>   • Users who wish to use external libraries (irrespective of language
>     or license) can extract the new timestamp and parse or calculate
>     externally.

I would not be so dramatic here. We do provide syntax description in
https://orgmode.org/worg/org-syntax.html#Timestamps It is not a big deal
to follow it.

> • Org is part of a standard
>   • We are able to defer to experts and 35 years of knowledge rather
>     than debate among ourselves

How so?

>   • Interfacing with other programs is simplified as the area inside the
>     delimiter notation can be passed as a string without parsing.

No. Org is not context-free. Parsing will be required anyway.

>   • New users and collaborators can be onboarded faster without needing
>     to learn a new system

Is it really so much decisive factor? I doubt so.

>   • Org documentation can refer to the standard instead of bearing the
>     burden of exposition.

That's a plus. But not decisive one.

> • The move to include time zones in the format is simplified
>   • The ISO standard has recently adopted a format for time zones from
>     RFC3339 and JAVAZDT, we can adhere to 8601 and keep things
>     consistent.

We have to follow POSIX TZ anyway. Maybe with minor additions.

> What other perspectives should the new format support?
>
> In addition to the representations necessary for a timezone aware
> calendar system, I suggest the new format be compatible with two other
> representations: finer/ arbitrary resolution for scientific work, and
> Level 2 of the Extended Date/Time Format for bibliographic and metadata
> systems.

Please frame it as a separate proposal. We have enough complexities with
time zones to add this on top.

> The Extended Data/Time Format (EDTF) was designed by the Library of
> Congress to address limitations of the ISO standard for metadata and
> archival purposes. A draft specification was created in 2012 and EDTF
> functionality has now been integrated into ISO 8601-2019. Of great
> interest is the ability to express the concepts of uncertainty and
> approximation. Archival work includes scenarios where the precise date
> may be unknown, so a format was created with qualifiers capable of
> handling these situations. In the EDTF format '1984?' expresses possibly
> the year 1984, but not definitely, while '2004-06~' expresses year-month
> approximate. This format has been implemented by multiple library
> systems and in 2021 Wikibase added an extension to support EDTF.

That's all nice but what a headache will it be to implement. What will
2004-06~ mean for agenda, for example?

In any case, it is also out of scope of this specific feature request.

> The initial technical or code burden to support these perspectives is
> minimal. Both can be parsed and calculated with by existing libraries,
> and the functionality to actually calculate with them can be delayed.
> The important thing is selecting a format which won't exclude them.

No. They cannot. Not by the existing Elisp libraries, available in Emacs
core.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>