emacs-orgmode@gnu.org archives
 help / color / mirror / code / Atom feed
* org-assert-version considered harmful
@ 2022-09-12 18:27 Stefan Monnier
  2022-09-13  1:52 ` Ihor Radchenko
  0 siblings, 1 reply; 14+ messages in thread
From: Stefan Monnier @ 2022-09-12 18:27 UTC (permalink / raw)
  To: emacs-orgmode

I can see the reason for this macro, and I don't really have a better
solution to offer, but as someone who has a Git clone of Org that is
regularly updated&compiled using (basically) the normal compilation
system used in Emacs (i.e. not systematically recompiling all the files,
but only those whose `.el` is more recent than their `.elc`), it means
that whenever Org's version changes, I have to manually erase all the
`.elc` files otherwise this macro will (incorrectly)
complain everywhere  :-(


        Stefan



^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: org-assert-version considered harmful
  2022-09-12 18:27 org-assert-version considered harmful Stefan Monnier
@ 2022-09-13  1:52 ` Ihor Radchenko
  2022-09-13  2:16   ` Timothy
  2022-09-13  2:53   ` Stefan Monnier
  0 siblings, 2 replies; 14+ messages in thread
From: Ihor Radchenko @ 2022-09-13  1:52 UTC (permalink / raw)
  To: Stefan Monnier; +Cc: emacs-orgmode

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I can see the reason for this macro, and I don't really have a better
> solution to offer, but as someone who has a Git clone of Org that is
> regularly updated&compiled using (basically) the normal compilation
> system used in Emacs (i.e. not systematically recompiling all the files,
> but only those whose `.el` is more recent than their `.elc`), it means
> that whenever Org's version changes, I have to manually erase all the
> `.elc` files otherwise this macro will (incorrectly)
> complain everywhere  :-(

Does it help if you run make autoloads after git fetch?

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: org-assert-version considered harmful
  2022-09-13  1:52 ` Ihor Radchenko
@ 2022-09-13  2:16   ` Timothy
  2022-09-13  2:53   ` Stefan Monnier
  1 sibling, 0 replies; 14+ messages in thread
From: Timothy @ 2022-09-13  2:16 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: Stefan Monnier, emacs-orgmode

[-- Attachment #1: Type: text/plain, Size: 768 bytes --]

Hi Ihor and Stefan,

Considering the separate cases of:
• Installing an Org version different to the bundled Org
• Having a development Org version with some slightly-out-of-date cache/autoload
  files.

I’d think the second case has a dramatically reduced chance of issues. Could it
be reasonable to change the `org-assert-version' check to act like so?
• Check if `org-version' matches, if not error
• If `org-version' matches, but `org-git-version' does not, show a warning message.

Alternatively, we could create a variable `org-assert-version-allow-git-mismatch'
which can be set before loading Org (by people who know what they’re doing, e.g.
Stefan) to enable this proposed behaviour.

What do you think?

All the best,
Timothy

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: org-assert-version considered harmful
  2022-09-13  1:52 ` Ihor Radchenko
  2022-09-13  2:16   ` Timothy
@ 2022-09-13  2:53   ` Stefan Monnier
  2022-09-13  3:18     ` Ihor Radchenko
  1 sibling, 1 reply; 14+ messages in thread
From: Stefan Monnier @ 2022-09-13  2:53 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode

Ihor Radchenko [2022-09-13 09:52:37] wrote:
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> I can see the reason for this macro, and I don't really have a better
>> solution to offer, but as someone who has a Git clone of Org that is
>> regularly updated&compiled using (basically) the normal compilation
>> system used in Emacs (i.e. not systematically recompiling all the files,
>> but only those whose `.el` is more recent than their `.elc`), it means
>> that whenever Org's version changes, I have to manually erase all the
>> `.elc` files otherwise this macro will (incorrectly)
>> complain everywhere  :-(
> Does it help if you run make autoloads after git fetch?

Not that I can tell, no.  But note that I'm not using Org's makefile to
compile the files, I'm using the elpa.git scripts instead, which don't
take into account dependencies between files.

So the situation is simply something like:

- git pull  =>  switches to 9.5.5, but several .el files are left unchanged.
- make autoloads  =>  this refreshes the autoloads, but the .elc files
  of those .el files which didn't change still won't be recompiled.

I'm not really reporting a bug; I'm not sure how to solve the problem
without throwing away the benefits of `org-assert-version`.  But I just
wanted to mention that it has unintended side effects :-)


        Stefan


PS: BTW, I notice that when Org is installed as a normal `package.el`
    package, in Emacs≥28 it will be activated before the `.emacs` file
    is loaded, which should prevent occurrence of the problems that
    `org-assert-version` aims to catch (unless you use, say, an
    org-babel file for the `early-init.el` :-)

PPS: Maybe instead of calling `org-assert-version` everywhere, the
     `org-autoloads.el` (i.e. the file that sets up the `load-path` and
     the autoloads) could look for traces of Org files in the
     `load-history` and signal an error if such files are found coming
     from a different directory.



^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: org-assert-version considered harmful
  2022-09-13  2:53   ` Stefan Monnier
@ 2022-09-13  3:18     ` Ihor Radchenko
  2022-09-13 13:26       ` Stefan Monnier
  0 siblings, 1 reply; 14+ messages in thread
From: Ihor Radchenko @ 2022-09-13  3:18 UTC (permalink / raw)
  To: Stefan Monnier; +Cc: emacs-orgmode

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Does it help if you run make autoloads after git fetch?
>
> Not that I can tell, no.  But note that I'm not using Org's makefile to
> compile the files, I'm using the elpa.git scripts instead, which don't
> take into account dependencies between files.
>
> So the situation is simply something like:
>
> - git pull  =>  switches to 9.5.5, but several .el files are left unchanged.
> - make autoloads  =>  this refreshes the autoloads, but the .elc files
>   of those .el files which didn't change still won't be recompiled.

Isn't it a bug in the elpa scripts then?
If a macro definition is changed and the .elc file using that macro is
not changed, it still needs to be re-compiled. Otherwise, all kinds of
unexpected side effects may appear.

> I'm not really reporting a bug; I'm not sure how to solve the problem
> without throwing away the benefits of `org-assert-version`.  But I just
> wanted to mention that it has unintended side effects :-)

I understand. As Timothy proposed, we can make less strict checks for Org
version consistency. But the question is whether we really need to make
a more lax assertion or maybe it is better to keep the assertion as is
and use it to catch potential issues e.g. in Elpa.

> PS: BTW, I notice that when Org is installed as a normal `package.el`
>     package, in Emacs≥28 it will be activated before the `.emacs` file
>     is loaded, which should prevent occurrence of the problems that
>     `org-assert-version` aims to catch (unless you use, say, an
>     org-babel file for the `early-init.el` :-)

Indeed. Version mismatch issue has been fixed in package.el years back.
But it is starting to pop up again as alternative package managers are
getting traction. There is a constant influx of bug reports caused by
mixed installation. Especially by users of Doom Emacs and straight.el.

Unfortunately, the problem cannot be easily solved, say, on straight.el
side --- straight.el basic design is causing the issue to appear.

> PPS: Maybe instead of calling `org-assert-version` everywhere, the
>      `org-autoloads.el` (i.e. the file that sets up the `load-path` and
>      the autoloads) could look for traces of Org files in the
>      `load-history` and signal an error if such files are found coming
>      from a different directory.

No, unfortunately.

org-autoloads, when loaded from built-in Emacs version will not help
to catch newer Org libraries being loaded after built-in Org version is
loaded.

Moreover, I consider loading personal forks of built-in Org libraries a
valid use-case. Demanding all the org libraries to be loaded from the
same directory will limit this possibility.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: org-assert-version considered harmful
  2022-09-13  3:18     ` Ihor Radchenko
@ 2022-09-13 13:26       ` Stefan Monnier
  2022-09-13 14:42         ` Ihor Radchenko
  0 siblings, 1 reply; 14+ messages in thread
From: Stefan Monnier @ 2022-09-13 13:26 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode

>> - git pull  =>  switches to 9.5.5, but several .el files are left unchanged.
>> - make autoloads  =>  this refreshes the autoloads, but the .elc files
>>   of those .el files which didn't change still won't be recompiled.
>
> Isn't it a bug in the elpa scripts then?
> If a macro definition is changed and the .elc file using that macro is
> not changed, it still needs to be re-compiled. Otherwise, all kinds of
> unexpected side effects may appear.

Yup.  But there's no option to automatically find those dependencies in
ELisp, and (IIRC from last time I looked at it, in many packages obeying
such dependencies would end up introducing circular dependencies in
the Makefile), so we'd have to depend on the package's author to provide
a working set of file dependencies.

Note that the same problem applies to Emacs's own ELisp files.
In Emacs we have `make bootstrap` to manually get out of such
a bad compilation.

>> PPS: Maybe instead of calling `org-assert-version` everywhere, the
>>      `org-autoloads.el` (i.e. the file that sets up the `load-path` and
>>      the autoloads) could look for traces of Org files in the
>>      `load-history` and signal an error if such files are found coming
>>      from a different directory.
>
> No, unfortunately.
>
> org-autoloads, when loaded from built-in Emacs version will not help
> to catch newer Org libraries being loaded after built-in Org version is
> loaded.

Hmm... after new-org-autoloads.el is loaded, the old-Org files will be
relegated to "late in the `load-path`" (i.e. after the directory that
holds the new-Org file) and should hence not be loaded any more (unless
someone goes through the trouble to explicitly load an old-Org files
with an absolute file name).

> Moreover, I consider loading personal forks of built-in Org libraries a
> valid use-case. Demanding all the org libraries to be loaded from the
> same directory will limit this possibility.

As long as they're loaded after new-org-autoloads.el, it would still be
fine since the test is only performed once when loading the
new-org-autoloads.el.


        Stefan



^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: org-assert-version considered harmful
  2022-09-13 13:26       ` Stefan Monnier
@ 2022-09-13 14:42         ` Ihor Radchenko
  2022-09-13 16:13           ` Stefan Monnier
  0 siblings, 1 reply; 14+ messages in thread
From: Ihor Radchenko @ 2022-09-13 14:42 UTC (permalink / raw)
  To: Stefan Monnier; +Cc: emacs-orgmode

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> - git pull  =>  switches to 9.5.5, but several .el files are left unchanged.
>>> - make autoloads  =>  this refreshes the autoloads, but the .elc files
>>>   of those .el files which didn't change still won't be recompiled.
>>
>> Isn't it a bug in the elpa scripts then?
>> If a macro definition is changed and the .elc file using that macro is
>> not changed, it still needs to be re-compiled. Otherwise, all kinds of
>> unexpected side effects may appear.
>
> Yup.  But there's no option to automatically find those dependencies in
> ELisp, and (IIRC from last time I looked at it, in many packages obeying
> such dependencies would end up introducing circular dependencies in
> the Makefile), so we'd have to depend on the package's author to provide
> a working set of file dependencies.

It would be nice to have such an option. At least, for the most critical
macros. Something similar to declare statements.

>>> PPS: Maybe instead of calling `org-assert-version` everywhere, the
>>>      `org-autoloads.el` (i.e. the file that sets up the `load-path` and
>>>      the autoloads) could look for traces of Org files in the
>>>      `load-history` and signal an error if such files are found coming
>>>      from a different directory.
>>
>> No, unfortunately.
>>
>> org-autoloads, when loaded from built-in Emacs version will not help
>> to catch newer Org libraries being loaded after built-in Org version is
>> loaded.
>
> Hmm... after new-org-autoloads.el is loaded, the old-Org files will be
> relegated to "late in the `load-path`" (i.e. after the directory that
> holds the new-Org file) and should hence not be loaded any more (unless
> someone goes through the trouble to explicitly load an old-Org files
> with an absolute file name).

I admit that I do not have sufficient knowledge about the autoload magic
Emacs uses when loading packages.

For reference, one simple way to trigger "mixed" state of Org is doing
something like:

1. emacs -Q
2. (require 'org)
3. Add the newer Org version to load-path
4. (require 'ob-python)

When and which version of org-autoloads.el will be loaded in such scenario?

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: org-assert-version considered harmful
  2022-09-13 14:42         ` Ihor Radchenko
@ 2022-09-13 16:13           ` Stefan Monnier
  2022-09-14  2:46             ` Ihor Radchenko
  0 siblings, 1 reply; 14+ messages in thread
From: Stefan Monnier @ 2022-09-13 16:13 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode

>> Yup.  But there's no option to automatically find those dependencies in
>> ELisp, and (IIRC from last time I looked at it, in many packages obeying
>> such dependencies would end up introducing circular dependencies in
>> the Makefile), so we'd have to depend on the package's author to provide
>> a working set of file dependencies.
>
> It would be nice to have such an option.

Agreed.  The "last time" mentioned above, I looked at changing the
byte-compiler to keep track of the macros that were expanded so we can
auto-generate the dependencies.

> At least, for the most critical macros.  Something similar to
> declare statements.

But that also requires manual intervention :-(

>> Hmm... after new-org-autoloads.el is loaded, the old-Org files will be
>> relegated to "late in the `load-path`" (i.e. after the directory that
>> holds the new-Org file) and should hence not be loaded any more (unless
>> someone goes through the trouble to explicitly load an old-Org files
>> with an absolute file name).
>
> I admit that I do not have sufficient knowledge about the autoload magic
> Emacs uses when loading packages.
>
> For reference, one simple way to trigger "mixed" state of Org is doing
> something like:
>
> 1. emacs -Q
> 2. (require 'org)
> 3. Add the newer Org version to load-path
> 4. (require 'ob-python)
>
> When and which version of org-autoloads.el will be loaded in
> such scenario?

None :-(

In my book step 3 above is a mistake (even if moved to step 2).

The `org-autoloads.el` is the file that adds the dir to `load-path`
(and in a normal ELPA install, that's the file that `package.el` loads
for you at startup).
So step 3 above is replaced by (load ".../org-autoloads"), and that's
where the problem would be detected.

But admittedly, that won't help users who made the mistake of
manually adding to `load-path` :-(


        Stefan



^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: org-assert-version considered harmful
  2022-09-13 16:13           ` Stefan Monnier
@ 2022-09-14  2:46             ` Ihor Radchenko
  2022-09-14 14:08               ` Stefan Monnier
  2022-09-25  2:39               ` Bastien
  0 siblings, 2 replies; 14+ messages in thread
From: Ihor Radchenko @ 2022-09-14  2:46 UTC (permalink / raw)
  To: Stefan Monnier; +Cc: emacs-orgmode

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Yup.  But there's no option to automatically find those dependencies in
>>> ELisp, and (IIRC from last time I looked at it, in many packages obeying
>>> such dependencies would end up introducing circular dependencies in
>>> the Makefile), so we'd have to depend on the package's author to provide
>>> a working set of file dependencies.
>>
>> It would be nice to have such an option.
>
> Agreed.  The "last time" mentioned above, I looked at changing the
> byte-compiler to keep track of the macros that were expanded so we can
> auto-generate the dependencies.

Then, I am inclined towards easing the version check to (org-version)
instead of (org-git-version). It will be no worse than the existing
situation with re-compiling updated .el files.

>> For reference, one simple way to trigger "mixed" state of Org is doing
>> something like:
>>
>> 1. emacs -Q
>> 2. (require 'org)
>> 3. Add the newer Org version to load-path
>> 4. (require 'ob-python)
>>
>> When and which version of org-autoloads.el will be loaded in
>> such scenario?
>
> None :-(
>
> In my book step 3 above is a mistake (even if moved to step 2).

I am confused.
AFAIK, changing the load-path is a common way for users to install
packages manually.

We do recommend setting the load-path in
https://orgmode.org/org.html#Installation and
https://orgmode.org/manual/Feedback.html

Moreover, it is also the recommendation from Emacs manual section
"28.8 Libraries of Lisp Code for Emacs":

>>>    The default value of ‘load-path’ is a list of directories where the
>>> Lisp code for Emacs itself is stored.  If you have libraries of your own
>>> in another directory, you can add that directory to the load path.
>>> Unlike most other variables described in this manual, ‘load-path’ cannot
>>> be changed via the Customize interface (*note Easy Customization::), but
>>> you can add a directory to it by putting a line like this in your init
>>> file (*note Init File::):
>>> 
>>>      (add-to-list 'load-path "/path/to/my/lisp/library")

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: org-assert-version considered harmful
  2022-09-14  2:46             ` Ihor Radchenko
@ 2022-09-14 14:08               ` Stefan Monnier
  2022-09-14 19:13                 ` Tim Cross
  2022-09-25  2:39               ` Bastien
  1 sibling, 1 reply; 14+ messages in thread
From: Stefan Monnier @ 2022-09-14 14:08 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode

>> In my book step 3 above is a mistake (even if moved to step 2).
> I am confused.
> AFAIK, changing the load-path is a common way for users to install
> packages manually.

No, you're not confused, I just think that installing packages manually
(including messing with `load-path` and writing `(autoload ...)` in your
init file) is very last century :-)


        Stefan



^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: org-assert-version considered harmful
  2022-09-14 14:08               ` Stefan Monnier
@ 2022-09-14 19:13                 ` Tim Cross
  0 siblings, 0 replies; 14+ messages in thread
From: Tim Cross @ 2022-09-14 19:13 UTC (permalink / raw)
  To: emacs-orgmode


Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> In my book step 3 above is a mistake (even if moved to step 2).
>> I am confused.
>> AFAIK, changing the load-path is a common way for users to install
>> packages manually.
>
> No, you're not confused, I just think that installing packages manually
> (including messing with `load-path` and writing `(autoload ...)` in your
> init file) is very last century :-)
>

but Emacs *is* last century! :-)

It is the fact we *can* manipulate load-path, autoloads and manually
control what is installed which makes it so powerful. See how far you
get when a core VS Code extension you rely on changes in a manner you
don't like and you want to revert to the previous version.

I know your comment was tongue in cheek, but I also do see some danger
in a future where we only interact with the well defined 'surface' layer
of software like Emacs and only a few hard core devs actually get into
crafting their init.el file.

It could be the reason we seem to be seeing an increase in the type of
issues which kicked this off is because fewer people are familiar with
manipulating load-path and autoloads, Less familiarity means less
familiarity with the common pitfalls and issues you may run into. 


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: org-assert-version considered harmful
  2022-09-14  2:46             ` Ihor Radchenko
  2022-09-14 14:08               ` Stefan Monnier
@ 2022-09-25  2:39               ` Bastien
  2022-09-25  3:15                 ` Ihor Radchenko
  1 sibling, 1 reply; 14+ messages in thread
From: Bastien @ 2022-09-25  2:39 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: Stefan Monnier, emacs-orgmode

Ihor Radchenko <yantar92@gmail.com> writes:

> Then, I am inclined towards easing the version check to (org-version)
> instead of (org-git-version).

FWIW strong +1 here.

-- 
 Bastien


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: org-assert-version considered harmful
  2022-09-25  2:39               ` Bastien
@ 2022-09-25  3:15                 ` Ihor Radchenko
  2022-09-25  4:27                   ` Timothy
  0 siblings, 1 reply; 14+ messages in thread
From: Ihor Radchenko @ 2022-09-25  3:15 UTC (permalink / raw)
  To: Bastien; +Cc: Stefan Monnier, emacs-orgmode

Bastien <bzg@gnu.org> writes:

> Ihor Radchenko <yantar92@gmail.com> writes:
>
>> Then, I am inclined towards easing the version check to (org-version)
>> instead of (org-git-version).
>
> FWIW strong +1 here.

There is one more concern we may need to solve prior to changing
org-git-version to org-version.

Currently, main and bugfix branches both have (org-version) ; => "9.5.5"
As a result, the assertion will not catch the important case when users
mix Org version installed via package.el and Org version installed from
git.

Should we use the next planned release version number on main branch as
a convention?

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: org-assert-version considered harmful
  2022-09-25  3:15                 ` Ihor Radchenko
@ 2022-09-25  4:27                   ` Timothy
  0 siblings, 0 replies; 14+ messages in thread
From: Timothy @ 2022-09-25  4:27 UTC (permalink / raw)
  To: emacs-orgmode; +Cc: Bastien, Stefan Monnier, emacs-orgmode

[-- Attachment #1: Type: text/plain, Size: 590 bytes --]

Hi Ihor,

> Should we use the next planned release version number on main branch as
> a convention?

For what it’s worth, in my own build of Org I set `org-release' to
`MAJOR.(MINOR+1).0', e.g. when the last tag is `9.5.5' I set `org-release' to `9.6.0'.
The `.0' suffix serves to differentiate this from the later `9.6' release, looking
at historical first-minor-version releases (`9.5', `9.4', `9.3', etc.) the patch
version seems to be omitted in each initial minor release.

I think it could make sense to apply this approach to the main branch.

All the best,
Timothy

^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2022-09-25  4:35 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-09-12 18:27 org-assert-version considered harmful Stefan Monnier
2022-09-13  1:52 ` Ihor Radchenko
2022-09-13  2:16   ` Timothy
2022-09-13  2:53   ` Stefan Monnier
2022-09-13  3:18     ` Ihor Radchenko
2022-09-13 13:26       ` Stefan Monnier
2022-09-13 14:42         ` Ihor Radchenko
2022-09-13 16:13           ` Stefan Monnier
2022-09-14  2:46             ` Ihor Radchenko
2022-09-14 14:08               ` Stefan Monnier
2022-09-14 19:13                 ` Tim Cross
2022-09-25  2:39               ` Bastien
2022-09-25  3:15                 ` Ihor Radchenko
2022-09-25  4:27                   ` Timothy

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).