org-id fixups and minor changes

org-id fixups and minor changes

[Gustav Wikström]

Hi!

I've pushed a couple of fixes and changes to master related to org-id.

First; a fix and a (major) speedup and method-change for how the
global caching works for ID's. The change in method is that providing
file's as arguments to org-id-update-id-locations no longer breaks the
existing id locations.

Second; I've added a customization so one can choose to cache the
id-locations as relative to the org-id-locations-file instead of as
absolute links. This helps a lot when running a system mirrored on
multiple machines where the absolute paths to ones documents might
differ, but where it's all the same relative to the
org-id-locations-file.

Third; I've added another ID generator method. The extremists might
say the new method is not unique enough but org-mode is a personal
system first and foremost, so I think there is merit to this new ID
method. It creates ID's based on current timestamp and doesn't try to
hide the timestamp from the user. One might call the ID's "natural"
since they are contain information in themselves. Certainly a good
thing for some. The new method will not be active unless explicitly
chosen by the user.

    As a side note, if using ID's together with attachments, try this
    new ID-generator out by setting org-id-method to ts (short for
    timestamp) and change the org-attach-id-to-path-function to
    something like "YYYY/MM/DDTHHMMSS" for more human readable
    attachment folders.

Org-id is next to undocumented in the manual so I didn't find a good
place to add this. A few lines are added to org-news though.

This is the first push by me without first doing an RFC. So,
naturally, if anything is out of order mail back and/or make changes
directly in the repo if needed. Tests passed anyways.

Kind regards
Gustav

Re: org-id fixups and minor changes

[Carsten Dominik]

[-- Attachment #1: Type: text/plain, Size: 2765 bytes --]

Hi Gustav,

I can see that it feels more natural to use timestamps.  I certainly see
that relative file names are good for across-computer compatibility.

You system assumes, if I see that correctly (have not studied it yet), that
not more that one ID will be created per second.  Or do you have something
in place that will catch this, for example if someone uses the mapping API
to assign IDs to a whole bunch of entries in a short time?

You are right that this is not documented in the manual, even though it is
used for lings and for attachments.  Maybe it would be good to explain it
in an appendix to the manual?  Would you like to draft such a section,
since you have been looking into this problem?

Do you think the default setting for Org should be modified?

Carsten

On Fri, Aug 2, 2019 at 5:14 PM Gustav Wikström <gustav@whil.se> wrote:

> Hi!
>
> I've pushed a couple of fixes and changes to master related to org-id.
>
> First; a fix and a (major) speedup and method-change for how the
> global caching works for ID's. The change in method is that providing
> file's as arguments to org-id-update-id-locations no longer breaks the
> existing id locations.
>
> Second; I've added a customization so one can choose to cache the
> id-locations as relative to the org-id-locations-file instead of as
> absolute links. This helps a lot when running a system mirrored on
> multiple machines where the absolute paths to ones documents might
> differ, but where it's all the same relative to the
> org-id-locations-file.
>
> Third; I've added another ID generator method. The extremists might
> say the new method is not unique enough but org-mode is a personal
> system first and foremost, so I think there is merit to this new ID
> method. It creates ID's based on current timestamp and doesn't try to
> hide the timestamp from the user. One might call the ID's "natural"
> since they are contain information in themselves. Certainly a good
> thing for some. The new method will not be active unless explicitly
> chosen by the user.
>
>     As a side note, if using ID's together with attachments, try this
>     new ID-generator out by setting org-id-method to ts (short for
>     timestamp) and change the org-attach-id-to-path-function to
>     something like "YYYY/MM/DDTHHMMSS" for more human readable
>     attachment folders.
>
> Org-id is next to undocumented in the manual so I didn't find a good
> place to add this. A few lines are added to org-news though.
>
> This is the first push by me without first doing an RFC. So,
> naturally, if anything is out of order mail back and/or make changes
> directly in the repo if needed. Tests passed anyways.
>
> Kind regards
> Gustav
>
>

[-- Attachment #2: Type: text/html, Size: 3304 bytes --]

Re: org-id fixups and minor changes

[Adam Porter]

Carsten Dominik <dominik@uva.nl> writes:

> You system assumes, if I see that correctly (have not studied it yet),
> that not more that one ID will be created per second.  Or do you have
> something in place that will catch this, for example if someone uses
> the mapping API to assign IDs to a whole bunch of entries in a short
> time?

That's a good point.  It might not be officially ISO8601 anymore, but
sub-second precision could be added (from format-time-string):

    %N is the nanosecond, %6N the microsecond, %3N the millisecond, etc.

I guess it's not possible for multiple entries to be created in a single
nanosecond.  :)

Re: org-id fixups and minor changes

[Gustav Wikström]

[-- Attachment #1: Type: text/plain, Size: 4121 bytes --]

Hi Carsten,

Yeah – you’re right, I didn’t think that much about automated ID creation so I stopped at seconds. I agree that it would be more general with more precision but that will also add some more noise into each ID. Maybe that’s not significant. But I also wonder how common it will be to try to batch-add ID’s…? I have nothing against adding more precision though, if that’s requested. What do you think?

Regarding documentation I’ll try to give it some thought. Maybe I’ll find some time to describe this area better 😊.

I wouldn’t mind changing the default from random to timestamp. But I’m not so sure about all the others? One thing that complicates things is the way attachment functionality parse the ID. If we use timestamps as the default ID it makes sense to change the default way org-attach parses the ID into folders as well. Something like “YYYY/MM/DD_and_the_rest”. But that will be a breaking changing. The existing folder-structure for attachments wouldn’t match the new any longer. Cleverness in code might solve the breakage though… If there is interest in changing the default I can try to solve the issue with the breaking change as well.

Regards
Gustav

From: Carsten Dominik <dominik@uva.nl>
Sent: den 10 augusti 2019 00:34
To: Gustav Wikström <gustav@whil.se>
Cc: emacs-orgmode@gnu.org
Subject: Re: [O] org-id fixups and minor changes

Hi Gustav,

I can see that it feels more natural to use timestamps.  I certainly see that relative file names are good for across-computer compatibility.

You system assumes, if I see that correctly (have not studied it yet), that not more that one ID will be created per second.  Or do you have something in place that will catch this, for example if someone uses the mapping API to assign IDs to a whole bunch of entries in a short time?

You are right that this is not documented in the manual, even though it is used for lings and for attachments.  Maybe it would be good to explain it in an appendix to the manual?  Would you like to draft such a section, since you have been looking into this problem?

Do you think the default setting for Org should be modified?

Carsten

On Fri, Aug 2, 2019 at 5:14 PM Gustav Wikström <gustav@whil.se<mailto:gustav@whil.se>> wrote:
Hi!

I've pushed a couple of fixes and changes to master related to org-id.

First; a fix and a (major) speedup and method-change for how the
global caching works for ID's. The change in method is that providing
file's as arguments to org-id-update-id-locations no longer breaks the
existing id locations.

Second; I've added a customization so one can choose to cache the
id-locations as relative to the org-id-locations-file instead of as
absolute links. This helps a lot when running a system mirrored on
multiple machines where the absolute paths to ones documents might
differ, but where it's all the same relative to the
org-id-locations-file.

Third; I've added another ID generator method. The extremists might
say the new method is not unique enough but org-mode is a personal
system first and foremost, so I think there is merit to this new ID
method. It creates ID's based on current timestamp and doesn't try to
hide the timestamp from the user. One might call the ID's "natural"
since they are contain information in themselves. Certainly a good
thing for some. The new method will not be active unless explicitly
chosen by the user.

    As a side note, if using ID's together with attachments, try this
    new ID-generator out by setting org-id-method to ts (short for
    timestamp) and change the org-attach-id-to-path-function to
    something like "YYYY/MM/DDTHHMMSS" for more human readable
    attachment folders.

Org-id is next to undocumented in the manual so I didn't find a good
place to add this. A few lines are added to org-news though.

This is the first push by me without first doing an RFC. So,
naturally, if anything is out of order mail back and/or make changes
directly in the repo if needed. Tests passed anyways.

Kind regards
Gustav

[-- Attachment #2: Type: text/html, Size: 8809 bytes --]

Re: org-id fixups and minor changes

[Adam Porter]

Gustav Wikström <gustav@whil.se> writes:

> Yeah – you’re right, I didn’t think that much about automated ID
> creation so I stopped at seconds. I agree that it would be more
> general with more precision but that will also add some more noise
> into each ID. Maybe that’s not significant. But I also wonder how
> common it will be to try to batch-add ID’s…? I have nothing against
> adding more precision though, if that’s requested. What do you think?

For any one user, it probably won't cause a problem.  But remember that
Org is used by many people.  If you add a method that can cause ID
conflicts, it's inevitable that it will happen to someone eventually.
It would be best to avoid conflicts in the first place.

> I wouldn’t mind changing the default from random to timestamp. But I’m
> not so sure about all the others?

Please don't change the default, because the default works, and has for
years.

>  One thing that complicates things is the way attachment functionality
> parse the ID. If we use timestamps as the default ID it makes sense to
> change the default way org-attach parses the ID into folders as
> well. Something like “YYYY/MM/DD_and_the_rest”. But that will be a
> breaking changing. The existing folder-structure for attachments
> wouldn’t match the new any longer. Cleverness in code might solve the
> breakage though… If there is interest in changing the default I can
> try to solve the issue with the breaking change as well.

Especially, please do not make a change like this.  Attempted cleverness
such as that should be avoided, because, considering how many users use
Org, each with their own customizations and unique workflows, it's
inevitable that such a change will lead to lost (or misplaced) data for
some users.

As a completely optional feature, it could be useful, however it would
need to cope with both storage formats, because existing users would
likely have data stored in the old locations when they enable the
feature.

Consider as well that more third-party software which uses Org formats
is becoming popular, including non-Emacs software.  Changes like these
are much more likely to cause breakage and incompatibilities.  For
example, software like Orgzly and org-web are not yet even fully
compatible with all of Org, but they make Org formats useful on
platforms which are hard to use Emacs on.  I think we should try not to
make the work of those authors more difficult by making Org hard to keep
up with.  :)

Re: org-id fixups and minor changes

[Stig Brautaset]

Hi Gustav,

Gustav Wikström <gustav@whil.se> writes:
> [...] I also wonder how common it will be to try to batch-add ID’s…?

Not especially uncommon, I think.  Both the org-rss and org-drill
packages batch-add IDs on first use.

Regards,
Stig

Re: org-id fixups and minor changes

[Carsten Dominik]

[-- Attachment #1: Type: text/plain, Size: 447 bytes --]

On Sun, Sep 1, 2019, 08:49 Stig Brautaset <stig@brautaset.org> wrote:

> Hi Gustav,
>
> Gustav Wikström <gustav@whil.se> writes:
> > [...] I also wonder how common it will be to try to batch-add ID’s…?
>
> Not especially uncommon, I think.  Both the org-rss and org-drill
> packages batch-add IDs on first use.
>

And even if that would be uncomment - at least the defaults need to be
safe.

Carsten


> Regards,
> Stig
>

[-- Attachment #2: Type: text/html, Size: 1102 bytes --]

Re: org-id fixups and minor changes

[Gustav Wikström]

[-- Attachment #1: Type: text/plain, Size: 906 bytes --]

I’ve committed a minor fix to add parts of a second to the ISO 8601 version if the ID.

https://code.orgmode.org/bzg/org-mode/commit/dea0c70c7b9036f386d36dfc8864ac0e431f9d25

/G

From: Carsten Dominik <dominik@uva.nl>
Sent: den 1 september 2019 10:36
To: Stig Brautaset <stig@brautaset.org>
Cc: Gustav Wikström <gustav@whil.se>; org-mode list <emacs-orgmode@gnu.org>
Subject: Re: [O] org-id fixups and minor changes


On Sun, Sep 1, 2019, 08:49 Stig Brautaset <stig@brautaset.org<mailto:stig@brautaset.org>> wrote:
Hi Gustav,

Gustav Wikström <gustav@whil.se<mailto:gustav@whil.se>> writes:
> [...] I also wonder how common it will be to try to batch-add ID’s…?

Not especially uncommon, I think.  Both the org-rss and org-drill
packages batch-add IDs on first use.

And even if that would be uncomment - at least the defaults need to be safe.

Carsten


Regards,
Stig

[-- Attachment #2: Type: text/html, Size: 4888 bytes --]

Re: org-id fixups and minor changes

[Gustav Wikström]

[-- Attachment #1: Type: text/plain, Size: 2773 bytes --]

Hi again,

FYI, I’ve applied a patch that will make it easier to change the default without breaking any existing functionality. I.e. we can avoid breakage for existing attachment-folders if we at some point in the future would choose to change default for ID’s to timestamp.

Commit 42b8db0d3

/G

From: Gustav Wikström
Sent: den 31 augusti 2019 21:17
To: Carsten Dominik <dominik@uva.nl>
Cc: emacs-orgmode@gnu.org
Subject: RE: [O] org-id fixups and minor changes

Hi Carsten,

Yeah – you’re right, I didn’t think that much about automated ID creation so I stopped at seconds. I agree that it would be more general with more precision but that will also add some more noise into each ID. Maybe that’s not significant. But I also wonder how common it will be to try to batch-add ID’s…? I have nothing against adding more precision though, if that’s requested. What do you think?

Regarding documentation I’ll try to give it some thought. Maybe I’ll find some time to describe this area better 😊.

I wouldn’t mind changing the default from random to timestamp. But I’m not so sure about all the others? One thing that complicates things is the way attachment functionality parse the ID. If we use timestamps as the default ID it makes sense to change the default way org-attach parses the ID into folders as well. Something like “YYYY/MM/DD_and_the_rest”. But that will be a breaking changing. The existing folder-structure for attachments wouldn’t match the new any longer. Cleverness in code might solve the breakage though… If there is interest in changing the default I can try to solve the issue with the breaking change as well.

Regards
Gustav

From: Carsten Dominik <dominik@uva.nl<mailto:dominik@uva.nl>>
Sent: den 10 augusti 2019 00:34
To: Gustav Wikström <gustav@whil.se<mailto:gustav@whil.se>>
Cc: emacs-orgmode@gnu.org<mailto:emacs-orgmode@gnu.org>
Subject: Re: [O] org-id fixups and minor changes

Hi Gustav,

I can see that it feels more natural to use timestamps.  I certainly see that relative file names are good for across-computer compatibility.

You system assumes, if I see that correctly (have not studied it yet), that not more that one ID will be created per second.  Or do you have something in place that will catch this, for example if someone uses the mapping API to assign IDs to a whole bunch of entries in a short time?

You are right that this is not documented in the manual, even though it is used for lings and for attachments.  Maybe it would be good to explain it in an appendix to the manual?  Would you like to draft such a section, since you have been looking into this problem?

Do you think the default setting for Org should be modified?

Carsten


[-- Attachment #2: Type: text/html, Size: 9033 bytes --]