Multiple versions of Org in load-path problem

Multiple versions of Org in load-path problem

[David Masterson]

I'm using Debian v11.5 on my Chromebook.  I have used 'apt install' to
load the pre-built version of Emacs 27.1.  I've just found out that it
installs a pre-built version of Org-9.3 in /usr/share/emacs/27.1/lisp.
That directory is early in the load-path, so it appears to be loaded out
of startup.el(?).  Everything seems to be working fine in my setup of
Emacs and Org.

I'd like to upgrade to org-9.6 from ELPA via the package-install in
Emacs which installs it into ~/.emacs.d/elpa.  I have a lot of
use-package calls in my ,emacs to load everything, but it runs into
problems with the mixed Org environment -- 9.3 is loaded at startup and
9.6 is "sort of" loaded by use-package.  The errors I get are from some
calls to use-package to load packages and are 'org-assert-version'
errors (a few of the packages [AFAIK] have no relationship to Org [like
hypernole]).

Is there a proper way to do this?  Do I really have to delete the
built-in 9.3 version (by hand or package-delete?) or will that mess up
something else?  I saw mention of 'unload-feature' which is new to me --
would that be safe to use?

-- 
David Masterson

Re: Multiple versions of Org in load-path problem

[tomas]

[-- Attachment #1: Type: text/plain, Size: 763 bytes --]

On Mon, Dec 05, 2022 at 06:50:33PM -0800, David Masterson wrote:
> I'm using Debian v11.5 on my Chromebook.  I have used 'apt install' to
> load the pre-built version of Emacs 27.1.  I've just found out that it
> installs a pre-built version of Org-9.3 in /usr/share/emacs/27.1/lisp.
> That directory is early in the load-path, so it appears to be loaded out
> of startup.el(?).  Everything seems to be working fine in my setup of
> Emacs and Org.

Hm. I run a self-compiled Emacs, so I can't check at the moment;
take this with a grain of salt.

That said for a distro putting the distro-provided places earlier
in the load-path than the local- or user-provided ones seems wrong;
admins/users have no chance to override them, no?

Cheers
-- 
t

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Multiple versions of Org in load-path problem

[David Masterson]

<tomas@tuxteam.de> writes:

> On Mon, Dec 05, 2022 at 06:50:33PM -0800, David Masterson wrote:
>> I'm using Debian v11.5 on my Chromebook.  I have used 'apt install' to
>> load the pre-built version of Emacs 27.1.  I've just found out that it
>> installs a pre-built version of Org-9.3 in /usr/share/emacs/27.1/lisp.
>> That directory is early in the load-path, so it appears to be loaded out
>> of startup.el(?).  Everything seems to be working fine in my setup of
>> Emacs and Org.
>
> Hm. I run a self-compiled Emacs, so I can't check at the moment;
> take this with a grain of salt.
>
> That said for a distro putting the distro-provided places earlier
> in the load-path than the local- or user-provided ones seems wrong;
> admins/users have no chance to override them, no?

Hmm.  You could be correct.  Which means it's the order of loading in my
.emacs file.  I'll have to test this to see when Org is first loaded.

-- 
David Masterson

Re: Multiple versions of Org in load-path problem

[Ihor Radchenko]

David Masterson <dsmasterson@gmail.com> writes:

> Is there a proper way to do this?  Do I really have to delete the
> built-in 9.3 version (by hand or package-delete?) or will that mess up
> something else?  I saw mention of 'unload-feature' which is new to me --
> would that be safe to use?

In theory, Emacs should handle such updates even when built-in Org is
loaded. See https://debbugs.gnu.org/cgi/bugreport.cgi?bug=59882

unload-feature is the last resort. We are still discussing the available
options in https://orgmode.org/list/87k033h218.fsf@localhost.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Multiple versions of Org in load-path problem

[David Masterson]

Ihor Radchenko <yantar92@posteo.net> writes:

> David Masterson <dsmasterson@gmail.com> writes:
>
>> Is there a proper way to do this?  Do I really have to delete the
>> built-in 9.3 version (by hand or package-delete?) or will that mess up
>> something else?  I saw mention of 'unload-feature' which is new to me --
>> would that be safe to use?
>
> In theory, Emacs should handle such updates even when built-in Org is
> loaded. See https://debbugs.gnu.org/cgi/bugreport.cgi?bug=59882

I'm going to test if (in my case) the buit-in 9.3 is already loaded when
my .emacs runs,  I also have to see when 9.6 loads in my .emacs.

> unload-feature is the last resort. We are still discussing the available
> options in https://orgmode.org/list/87k033h218.fsf@localhost.

I have to play with the solution I suggested.  I'm beginning to wonder
if it will cause other problems with (seeming) Org dependent packages.
In my case, the error I get is from various use-package calls --
something like "Error: org-assert-version is not known".  This is
confusing because I've arranged my .emacs as (almost) an alphabetical
loading of the packages and some of the use-package errors are coming
from packages loaded before Org.  IOW, org-assert-version from 9.6
shouldn't be loaded yet and 9.3 doesn't have org-assert-version. So
what's referring to org-assert-version?

-- 
David Masterson

Re: Multiple versions of Org in load-path problem

[David Masterson]

David Masterson <dsmasterson@gmail.com> writes:

> Ihor Radchenko <yantar92@posteo.net> writes:
>
>> David Masterson <dsmasterson@gmail.com> writes:
>>
>>> Is there a proper way to do this?  Do I really have to delete the
>>> built-in 9.3 version (by hand or package-delete?) or will that mess up
>>> something else?  I saw mention of 'unload-feature' which is new to me --
>>> would that be safe to use?
>>
>> In theory, Emacs should handle such updates even when built-in Org is
>> loaded. See https://debbugs.gnu.org/cgi/bugreport.cgi?bug=59882
>
> I'm going to test if (in my case) the buit-in 9.3 is already loaded when
> my .emacs runs,  I also have to see when 9.6 loads in my .emacs.
>
>> unload-feature is the last resort. We are still discussing the available
>> options in https://orgmode.org/list/87k033h218.fsf@localhost.
>
> I have to play with the solution I suggested.  I'm beginning to wonder
> if it will cause other problems with (seeming) Org dependent packages.
> In my case, the error I get is from various use-package calls --
> something like "Error: org-assert-version is not known".  This is
> confusing because I've arranged my .emacs as (almost) an alphabetical
> loading of the packages and some of the use-package errors are coming
> from packages loaded before Org.  IOW, org-assert-version from 9.6
> shouldn't be loaded yet and 9.3 doesn't have org-assert-version. So
> what's referring to org-assert-version?

I went so far as to add the following to early-init.el:

(if (featurep 'org) (load-feature 'org))
(add-to-list 'load-path (expand-file-name "~/.emacs.d/elpa/org-9.6")
(load-library "org")

The load-library still fails with the org-assert-version error.

Could it be that, when you package-install Org-9.6, it runs through a
check where (I think) it compiles the package?  I get a lot of warnings
and 34(?) failed files.  Could it be that some of the files are not
built and, so, when I load-library, I'm picking up some of the wrong
files from the built-in Org-9.3?  Do we need to remove all versions of
Org that is not the current version from the load-path at the very
beginning of org.el to ensure no fall-through?  Kind of a hack, but...

-- 
David Masterson

Re: Multiple versions of Org in load-path problem

[David Masterson]

David Masterson <dsmasterson@gmail.com> writes:

> David Masterson <dsmasterson@gmail.com> writes:
>
>> Ihor Radchenko <yantar92@posteo.net> writes:
>>
>>> David Masterson <dsmasterson@gmail.com> writes:
>>>
>>>> Is there a proper way to do this?  Do I really have to delete the
>>>> built-in 9.3 version (by hand or package-delete?) or will that mess up
>>>> something else?  I saw mention of 'unload-feature' which is new to me --
>>>> would that be safe to use?
>>>
>>> In theory, Emacs should handle such updates even when built-in Org is
>>> loaded. See https://debbugs.gnu.org/cgi/bugreport.cgi?bug=59882
>>
>> I'm going to test if (in my case) the buit-in 9.3 is already loaded when
>> my .emacs runs,  I also have to see when 9.6 loads in my .emacs.
>>
>>> unload-feature is the last resort. We are still discussing the available
>>> options in https://orgmode.org/list/87k033h218.fsf@localhost.
>>
>> I have to play with the solution I suggested.  I'm beginning to wonder
>> if it will cause other problems with (seeming) Org dependent packages.
>> In my case, the error I get is from various use-package calls --
>> something like "Error: org-assert-version is not known".  This is
>> confusing because I've arranged my .emacs as (almost) an alphabetical
>> loading of the packages and some of the use-package errors are coming
>> from packages loaded before Org.  IOW, org-assert-version from 9.6
>> shouldn't be loaded yet and 9.3 doesn't have org-assert-version. So
>> what's referring to org-assert-version?
>
> I went so far as to add the following to early-init.el:
>
> (if (featurep 'org) (load-feature 'org))

typo -- unload-feature

> (add-to-list 'load-path (expand-file-name "~/.emacs.d/elpa/org-9.6")
> (load-library "org")
>
> The load-library still fails with the org-assert-version error.
>
> Could it be that, when you package-install Org-9.6, it runs through a
> check where (I think) it compiles the package?  I get a lot of warnings
> and 34(?) failed files.  Could it be that some of the files are not
> built and, so, when I load-library, I'm picking up some of the wrong
> files from the built-in Org-9.3?  Do we need to remove all versions of
> Org that is not the current version from the load-path at the very
> beginning of org.el to ensure no fall-through?  Kind of a hack, but...

-- 
David Masterson

Re: Multiple versions of Org in load-path problem

[Ihor Radchenko]

David Masterson <dsmasterson@gmail.com> writes:

> I went so far as to add the following to early-init.el:
>
> (if (featurep 'org) (load-feature 'org))
> (add-to-list 'load-path (expand-file-name "~/.emacs.d/elpa/org-9.6")
> (load-library "org")
>
> The load-library still fails with the org-assert-version error.

Indeed. It is to be expected. M-x package-install should not err here
though. In theory.

I suggest you to read `org-assert-version'. It aims to catch the
situation above and similar.

Note, however, that installation problem is different. Or rather it
should be different.

> Could it be that, when you package-install Org-9.6, it runs through a
> check where (I think) it compiles the package?  I get a lot of warnings
> and 34(?) failed files.  Could it be that some of the files are not
> built and, so, when I load-library, I'm picking up some of the wrong
> files from the built-in Org-9.3?  Do we need to remove all versions of
> Org that is not the current version from the load-path at the very
> beginning of org.el to ensure no fall-through?  Kind of a hack, but...

Maybe. It should not. That's why I raised
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=59882
I'd like to see if Emacs devs have anything to say here.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Multiple versions of Org in load-path problem

[Ihor Radchenko]

David Masterson <dsmasterson@gmail.com> writes:

> I have to play with the solution I suggested.  I'm beginning to wonder
> if it will cause other problems with (seeming) Org dependent packages.
> In my case, the error I get is from various use-package calls --
> something like "Error: org-assert-version is not known".  This is
> confusing because I've arranged my .emacs as (almost) an alphabetical
> loading of the packages and some of the use-package errors are coming
> from packages loaded before Org.  IOW, org-assert-version from 9.6
> shouldn't be loaded yet and 9.3 doesn't have org-assert-version. So
> what's referring to org-assert-version?

If you install Org via package-install (or :ensure t), Org 9.6 should
get loaded once package system is initialized. Emacs should correctly
put ELPA's version of Org in from of the load-path. However, it looks
like Org 9.6 compilation is off and `org-assert-version' macro is
compiled as a function call instead of a macro.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Multiple versions of Org in load-path problem

[David Masterson]

Ihor Radchenko <yantar92@posteo.net> writes:

> David Masterson <dsmasterson@gmail.com> writes:
>
>> I went so far as to add the following to early-init.el:
>>
>> (if (featurep 'org) (unload-feature 'org))
>> (add-to-list 'load-path (expand-file-name "~/.emacs.d/elpa/org-9.6")
>> (load-library "org")
>>
>> The load-library still fails with the org-assert-version error.
>
> Indeed. It is to be expected. M-x package-install should not err here
> though. In theory.

My testing says that, in early-init.el, Org is not yet a feature, so the
built-in Org-9.3 is not yet loaded and the first 'if' fails. Adding the
path for Org-9.6 should put it at the front of the load-path and
(hopefully) override Org-9.3 if it is in the load-path (I think I
checked that).  The error in the load-library is, therefore, a problem
with Org-9.6 installed by package-install. 

> I suggest you to read `org-assert-version'. It aims to catch the
> situation above and similar.

This is chicken and egg problem.  I can't use 'org-assert-version'
unless I can get Org to load.

> Note, however, that installation problem is different. Or rather it
> should be different.
>
>> Could it be that, when you package-install Org-9.6, it runs through a
>> check where (I think) it compiles the package?  I get a lot of warnings
>> and 34(?) failed files.  Could it be that some of the files are not
>> built and, so, when I load-library, I'm picking up some of the wrong
>> files from the built-in Org-9.3?  Do we need to remove all versions of
>> Org that is not the current version from the load-path at the very
>> beginning of org.el to ensure no fall-through?  Kind of a hack, but...
>
> Maybe. It should not. That's why I raised
> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=59882
> I'd like to see if Emacs devs have anything to say here.

Good. I'm not used to bug reporting in Emacs, so I raised a question on
use-package in emacs-devel@gnu.org, but they pointed me at your bug.
I'll see if I can add anything there.

-- 
David Masterson

Re: Multiple versions of Org in load-path problem

[Michel Schinz]

Just for the record, I also ran into problems when installing Org 9.6
using Emacs' package system on top of an older version that came with
Emacs. If I tried to install it as usual (M-x list-packages, then
install the package from there), I had errors during compilation related
to `org-assert-version`, and then if I restarted Emacs, I would get a
fatal error in an unrelated package.

I managed to solve that problem by:
1. uninstalling Org 9.6 and exiting Emacs,
2. starting Emacs with -q,
3. installing Org 9.6 from there (using M-x list-packages as usual),
4. restarting Emacs.

I'm not sure this is related to your problem, or whether that helps (but
I hope it does)...

Michel.

On Thu, 8 Dec 2022, at 19:24, David Masterson wrote:
> Ihor Radchenko <yantar92@posteo.net> writes:
>
>> David Masterson <dsmasterson@gmail.com> writes:
>>
>>> I went so far as to add the following to early-init.el:
>>>
>>> (if (featurep 'org) (unload-feature 'org))
>>> (add-to-list 'load-path (expand-file-name "~/.emacs.d/elpa/org-9.6")
>>> (load-library "org")
>>>
>>> The load-library still fails with the org-assert-version error.
>>
>> Indeed. It is to be expected. M-x package-install should not err here
>> though. In theory.
>
> My testing says that, in early-init.el, Org is not yet a feature, so the
> built-in Org-9.3 is not yet loaded and the first 'if' fails. Adding the
> path for Org-9.6 should put it at the front of the load-path and
> (hopefully) override Org-9.3 if it is in the load-path (I think I
> checked that).  The error in the load-library is, therefore, a problem
> with Org-9.6 installed by package-install. 
>
>> I suggest you to read `org-assert-version'. It aims to catch the
>> situation above and similar.
>
> This is chicken and egg problem.  I can't use 'org-assert-version'
> unless I can get Org to load.
>
>> Note, however, that installation problem is different. Or rather it
>> should be different.
>>
>>> Could it be that, when you package-install Org-9.6, it runs through a
>>> check where (I think) it compiles the package?  I get a lot of warnings
>>> and 34(?) failed files.  Could it be that some of the files are not
>>> built and, so, when I load-library, I'm picking up some of the wrong
>>> files from the built-in Org-9.3?  Do we need to remove all versions of
>>> Org that is not the current version from the load-path at the very
>>> beginning of org.el to ensure no fall-through?  Kind of a hack, but...
>>
>> Maybe. It should not. That's why I raised
>> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=59882
>> I'd like to see if Emacs devs have anything to say here.
>
> Good. I'm not used to bug reporting in Emacs, so I raised a question on
> use-package in emacs-devel@gnu.org, but they pointed me at your bug.
> I'll see if I can add anything there.
>
> -- 
> David Masterson

RE: Multiple versions of Org in load-path problem

[Cook, Malcolm]

>Ihor Radchenko <mailto:yantar92@posteo.net> writes:
>
>> David Masterson <mailto:dsmasterson@gmail.com> writes:
>>
>>> I went so far as to add the following to early-init.el:
>>>
>>> (if (featurep 'org) (unload-feature 'org))
>>> (add-to-list 'load-path (expand-file-name "~/.emacs.d/elpa/org-9.6")
>>> (load-library "org")
>>>
>>> The load-library still fails with the org-assert-version error.
>>
>> Indeed. It is to be expected. M-x package-install should not err here
>> though. In theory.
>
>My testing says that, in early-init.el, Org is not yet a feature, so the
>built-in Org-9.3 is not yet loaded and the first 'if' fails. Adding the
>path for Org-9.6 should put it at the front of the load-path and
>(hopefully) override Org-9.3 if it is in the load-path (I think I
>checked that). The error in the load-library is, therefore, a problem
>with Org-9.6 installed by package-install. 
>
>> I suggest you to read `org-assert-version'. It aims to catch the
>> situation above and similar.
>
>This is chicken and egg problem. I can't use 'org-assert-version'
>unless I can get Org to load.
>
>> Note, however, that installation problem is different. Or rather it
>> should be different.
>>
>>> Could it be that, when you package-install Org-9.6, it runs through a
>>> check where (I think) it compiles the package? I get a lot of warnings
>>> and 34(?) failed files. Could it be that some of the files are not
>>> built and, so, when I load-library, I'm picking up some of the wrong
>>> files from the built-in Org-9.3? Do we need to remove all versions of
>>> Org that is not the current version from the load-path at the very
>>> beginning of org.el to ensure no fall-through? Kind of a hack, but...

Hi,

I've been reading along as I've been similarly hounded in years past with org version changes.

It really sounds like you compiled org-9.6 with Org-9.3 already loaded.

I'd recommend reinstalling.

Here is one way to do it that is pretty safe:

	> emacs -Q -batch -eval "(progn (require 'package) (package-initialize)  (package-refresh-contents) (package-install 'org))"

If after doing this, and running emacs, you find M-x org-version does not return 9.6, or you find the package assert problem, I'd try the more paranoid:
	> emacs -Q -batch -eval "(progn (require 'cl-seq) (delete (car (cl-member "lisp/org" load-path :test  #'string-match)) load-path) (require 'package) (package-initialize)  (package-refresh-contents) (package-install 'org))"

And possibly consider adding to your init.el at the very top:

	(require 'cl-seq) (delete (car (cl-member "lisp/org" load-path :test  #'string-match)) load-path)

YMMV,

~ Malcolm

>>
>> Maybe. It should not. That's why I raised
>> https://debbugs.gnu.org/cgi/bugreport.cgi?bug=59882
>> I'd like to see if Emacs devs have anything to say here.
>
>Good. I'm not used to bug reporting in Emacs, so I raised a question on
>use-package in mailto:emacs-devel@gnu.org, but they pointed me at your bug.
>I'll see if I can add anything there.
>
>-- 
>David Masterson
>

Re: Multiple versions of Org in load-path problem

[David Masterson]

Adding this to bug #59882

"Michel Schinz" <michel@schi.nz> writes:

> Just for the record, I also ran into problems when installing Org 9.6
> using Emacs' package system on top of an older version that came with
> Emacs. If I tried to install it as usual (M-x list-packages, then
> install the package from there), I had errors during compilation related
> to `org-assert-version`, and then if I restarted Emacs, I would get a
> fatal error in an unrelated package.
>
> I managed to solve that problem by:
> 1. uninstalling Org 9.6 and exiting Emacs,
> 2. starting Emacs with -q,
> 3. installing Org 9.6 from there (using M-x list-packages as usual),
> 4. restarting Emacs.

Interesting!  I tried this (essentially) and it worked for my case.  In
my case, I had a built-in Org-9.3 and I was trying to use list-packages
to install Org-9.6. I checked that using -q still added Org-9.3 to the
load-path, but, since Org wasn't loaded, the install via list-packages
worked.

The question is what's the proper way of doing this without '-q'?

> I'm not sure this is related to your problem, or whether that helps (but
> I hope it does)...

I think it does.

Side note:

In my testing, I found a strange case where, in *scratch*, I get:

(message "%s" org-version)
;; Error undefined
;; Do 'C-h v org-version'
(message "%s" org-version)
9.3

So, 'describe-variable' on org-version causes Org to be loaded?!?  Why
do I have a feeling this is related to this bug?

-- 
David Masterson

Re: Multiple versions of Org in load-path problem

[Tim Cross]

David Masterson <dsmasterson@gmail.com> writes:

> Adding this to bug #59882
>
> "Michel Schinz" <michel@schi.nz> writes:
>
>> Just for the record, I also ran into problems when installing Org 9.6
>> using Emacs' package system on top of an older version that came with
>> Emacs. If I tried to install it as usual (M-x list-packages, then
>> install the package from there), I had errors during compilation related
>> to `org-assert-version`, and then if I restarted Emacs, I would get a
>> fatal error in an unrelated package.
>>
>> I managed to solve that problem by:
>> 1. uninstalling Org 9.6 and exiting Emacs,
>> 2. starting Emacs with -q,
>> 3. installing Org 9.6 from there (using M-x list-packages as usual),
>> 4. restarting Emacs.
>
> Interesting!  I tried this (essentially) and it worked for my case.  In
> my case, I had a built-in Org-9.3 and I was trying to use list-packages
> to install Org-9.6. I checked that using -q still added Org-9.3 to the
> load-path, but, since Org wasn't loaded, the install via list-packages
> worked.
>
> The question is what's the proper way of doing this without '-q'?
>
>> I'm not sure this is related to your problem, or whether that helps (but
>> I hope it does)...
>
> I think it does.
>
> Side note:
>
> In my testing, I found a strange case where, in *scratch*, I get:
>
> (message "%s" org-version)
> ;; Error undefined
> ;; Do 'C-h v org-version'
> (message "%s" org-version)
> 9.3
>
> So, 'describe-variable' on org-version causes Org to be loaded?!?  Why
> do I have a feeling this is related to this bug?

I don't think there is any safe way to install an updated version of
org-mode other than

1. Use the -q approach outlined above

2. Craft your init.el file such that org functionality is only loaded
when explicitly requested and always update as the first action after
starting emacs.

The first approach is actually the easiest. The second is hard to get
right and very fragile because packages like use-package and more
specifically, other packages with leverage off org functionality, make it
impossible to reliably know exactly when org is loaded.

An approach used by many 'canned' distributions is to postpone package
updates. You have a function you run to check for updates which
generates a list of packages to update and writes that list to a
file. Each time emacs is started, it looks for this update list and if
it finds it, it installs packages updates at the very beginning of the
init process (before any of your other init.el code or custom
blocks). The process also looks for org in the list of packages to
update and if it is found, updates it first. 

I don't think there is a safe way to load org mode after the init
process i.e. after booting emacs by M-x package-update.

I've had good success using straight.el. I had to be careful regarding
how I structured my init.el file (ensuring any straight stuff happens
first and the first use package stanza is for org. The main reason
straight works well for me is that my work flow is to do a M-x
straight-pull-all when I want to update my packages. This does a git
pull for all the sources, but does not do any build/install. This occurs
when I next start Emacs and because I have all the straight stuff at the
start and because org mode is the first straight-use-package, the update
and install happens before any other org functionality is loaded,
avoiding mixed version issues.

Re: Multiple versions of Org in load-path problem

[David Masterson]

Tim Cross <theophilusx@gmail.com> writes:

> David Masterson <dsmasterson@gmail.com> writes:
>
>> "Michel Schinz" <michel@schi.nz> writes:
>>
>>> Just for the record, I also ran into problems when installing Org 9.6
>>> using Emacs' package system on top of an older version that came with
>>> Emacs. If I tried to install it as usual (M-x list-packages, then
>>> install the package from there), I had errors during compilation related
>>> to `org-assert-version`, and then if I restarted Emacs, I would get a
>>> fatal error in an unrelated package.
>>>
>>> I managed to solve that problem by:
>>> 1. uninstalling Org 9.6 and exiting Emacs,
>>> 2. starting Emacs with -q,
>>> 3. installing Org 9.6 from there (using M-x list-packages as usual),
>>> 4. restarting Emacs.
>>
>> Interesting!  I tried this (essentially) and it worked for my case.  In
>> my case, I had a built-in Org-9.3 and I was trying to use list-packages
>> to install Org-9.6. I checked that using -q still added Org-9.3 to
>> the
>> load-path, but, since Org wasn't loaded, the install via list-packages
>> worked.
>>
>> The question is what's the proper way of doing this without '-q'?

[...]

> I don't think there is any safe way to install an updated version of
> org-mode other than
>
> 1. Use the -q approach outlined above

Thinking about it, this only works if Org is in elpa as melpa (etc.) are
not added to package-archives.  You'd have to do some handwritten elisp
out of *scratch* to setup package-archives if Org-9.6 was still coming
out of melpa. That's why this can only be labeled as a hack and not a
solution.

> 2. Craft your init.el file such that org functionality is only loaded
> when explicitly requested and always update as the first action after
> starting emacs.

In this case, something happened in package-install when trying to
install Org-9.6 with a built-in Org-9.3.  During the compilation check
(.el -> .elc) many files failed because the new 'org-assert-version'
macro was not defined.  Sort of like, after package-install started
working on Org-9.6, org-macs.el (where org-assert-version should be) got
loaded *before* the new load-path had been set causing it to load the
old one from 9.3.  Thereafter, everything went awry. 

> The first approach is actually the easiest. The second is hard to get
> right and very fragile because packages like use-package and more
> specifically, other packages with leverage off org functionality, make it
> impossible to reliably know exactly when org is loaded.

Using ':after" in use-package is supposed to help that, but I'm not sure
it is reliable.  Packages are often incomplete about what other packages
it depends on.

> An approach used by many 'canned' distributions is to postpone package
> updates. You have a function you run to check for updates which
> generates a list of packages to update and writes that list to a
> file. Each time emacs is started, it looks for this update list and if
> it finds it, it installs packages updates at the very beginning of the
> init process (before any of your other init.el code or custom
> blocks). The process also looks for org in the list of packages to
> update and if it is found, updates it first. 

Probably doesn't work in this case as you would need to be able to use
package.el suggesting that the load-path has been updated for all
built-ins already.  The thing I note is that the load-path has already
been updated for built-ins at the beginning of
'~/.emacs.d/early-init.el', but the libraries haven't been loaded yet
(unless needed). That's okay if the newer version of a package then
cleanly replaces all the files in the old version.

> I don't think there is a safe way to load org mode after the init
> process i.e. after booting emacs by M-x package-update.

Where is package-update called in the boot process of emacs?  I don't
see package-update in Emacs v2.7.

> I've had good success using straight.el. I had to be careful regarding
> how I structured my init.el file (ensuring any straight stuff happens
> first and the first use package stanza is for org. The main reason
> straight works well for me is that my work flow is to do a M-x
> straight-pull-all when I want to update my packages. This does a git
> pull for all the sources, but does not do any build/install. This occurs
> when I next start Emacs and because I have all the straight stuff at the
> start and because org mode is the first straight-use-package, the update
> and install happens before any other org functionality is loaded,
> avoiding mixed version issues.

Where do you get straight.el?  I don't see it in [m]elpa.

-- 
David Masterson

Re: Multiple versions of Org in load-path problem

[Tim Cross]

David Masterson <dsmasterson@gmail.com> writes:

> Tim Cross <theophilusx@gmail.com> writes:
>
>> David Masterson <dsmasterson@gmail.com> writes:
>>
>>> "Michel Schinz" <michel@schi.nz> writes:
>>>
>>>> Just for the record, I also ran into problems when installing Org 9.6
>>>> using Emacs' package system on top of an older version that came with
>>>> Emacs. If I tried to install it as usual (M-x list-packages, then
>>>> install the package from there), I had errors during compilation related
>>>> to `org-assert-version`, and then if I restarted Emacs, I would get a
>>>> fatal error in an unrelated package.
>>>>
>>>> I managed to solve that problem by:
>>>> 1. uninstalling Org 9.6 and exiting Emacs,
>>>> 2. starting Emacs with -q,
>>>> 3. installing Org 9.6 from there (using M-x list-packages as usual),
>>>> 4. restarting Emacs.
>>>
>>> Interesting!  I tried this (essentially) and it worked for my case.  In
>>> my case, I had a built-in Org-9.3 and I was trying to use list-packages
>>> to install Org-9.6. I checked that using -q still added Org-9.3 to
>>> the
>>> load-path, but, since Org wasn't loaded, the install via list-packages
>>> worked.
>>>
>>> The question is what's the proper way of doing this without '-q'?
>
> [...]
>
>> I don't think there is any safe way to install an updated version of
>> org-mode other than
>>
>> 1. Use the -q approach outlined above
>
> Thinking about it, this only works if Org is in elpa as melpa (etc.) are
> not added to package-archives.  You'd have to do some handwritten elisp
> out of *scratch* to setup package-archives if Org-9.6 was still coming
> out of melpa. That's why this can only be labeled as a hack and not a
> solution.
>

Well, yes, if your going to use this technique to load a package which
is not in the default package archives you would need to add that
archive first. People who use this technique often just have a
'update.el' file which they load/evaluate when starting Emacs with -q. 


>> 2. Craft your init.el file such that org functionality is only loaded
>> when explicitly requested and always update as the first action after
>> starting emacs.
>
> In this case, something happened in package-install when trying to
> install Org-9.6 with a built-in Org-9.3.  During the compilation check
> (.el -> .elc) many files failed because the new 'org-assert-version'
> macro was not defined.  Sort of like, after package-install started
> working on Org-9.6, org-macs.el (where org-assert-version should be) got
> loaded *before* the new load-path had been set causing it to load the
> old one from 9.3.  Thereafter, everything went awry. 
>

You must have some custom code in your init.el or are using something
like use-package as package.el doesn't try to install or upgrade
packages during init by default.

Where people can come undone is when they are using use-package and set
the :ensure t option. In this case, use-package will not know about the
bundled version and will attempt to install org from ELPA. If use
package runs after org has already been loaded (possibly because some
other package has been loaded which depends on/requires org mode and has
loaded the bundled version) then things will break because you will end
up with a mixed version install. This is why I always ensure org is the very
first use-package in my init.el and it comes before any other code which
loads or initialises anything.  

>> The first approach is actually the easiest. The second is hard to get
>> right and very fragile because packages like use-package and more
>> specifically, other packages with leverage off org functionality, make it
>> impossible to reliably know exactly when org is loaded.
>
> Using ':after" in use-package is supposed to help that, but I'm not sure
> it is reliable.  Packages are often incomplete about what other packages
> it depends on.
>

You cannot rely on :after. The problem is, other packages may require
org functionality and will load org when they are loaded. This can be
very subtle as there are a lot of packages which only make very small
use of org mode, but even that requires that org mode is loaded. 

>> An approach used by many 'canned' distributions is to postpone package
>> updates. You have a function you run to check for updates which
>> generates a list of packages to update and writes that list to a
>> file. Each time emacs is started, it looks for this update list and if
>> it finds it, it installs packages updates at the very beginning of the
>> init process (before any of your other init.el code or custom
>> blocks). The process also looks for org in the list of packages to
>> update and if it is found, updates it first. 
>
> Probably doesn't work in this case as you would need to be able to use
> package.el suggesting that the load-path has been updated for all
> built-ins already.  The thing I note is that the load-path has already
> been updated for built-ins at the beginning of
> '~/.emacs.d/early-init.el', but the libraries haven't been loaded yet
> (unless needed). That's okay if the newer version of a package then
> cleanly replaces all the files in the old version.
>

It does work. Distributions like Sapcemacs use this approach and do not
have the mixed versions issue. I've not looked closely at their
implementation, but they well might do all this work as part of the
early-init.el file.


>> I don't think there is a safe way to load org mode after the init
>> process i.e. after booting emacs by M-x package-update.
>
> Where is package-update called in the boot process of emacs?  I don't
> see package-update in Emacs v2.7.
>

Sorry, I wasn't clear.

What I meant is that there is no clean and reliable way of running the
package update process after emacs has initialised. The package.el API
has changed, but previously, you would run M-x package-update-packages
to update installed packages. Now it is M-x package-update-all. With
many packages, you can run this command and update your installed
packages. However, more complex packages, like org-mode, cannot handle
this due to potential mixed version problems - a problem which has
existed for many years. 

By default, there is no automated package update setting (unless that
has been added in recent versions). There are some add-on packages which
can do this and the elisp to make this happen is trivial (but much
harder to do it and make it robust). 


>> I've had good success using straight.el. I had to be careful regarding
>> how I structured my init.el file (ensuring any straight stuff happens
>> first and the first use package stanza is for org. The main reason
>> straight works well for me is that my work flow is to do a M-x
>> straight-pull-all when I want to update my packages. This does a git
>> pull for all the sources, but does not do any build/install. This occurs
>> when I next start Emacs and because I have all the straight stuff at the
>> start and because org mode is the first straight-use-package, the update
>> and install happens before any other org functionality is loaded,
>> avoiding mixed version issues.
>
> Where do you get straight.el?  I don't see it in [m]elpa.

No, to use straight.el, you add some code to your init.el which will
retrieve the current straight.el file from the github
repository. Straight.el does not use the ELPA/NONGNU ELPA/MELPA archives
- at least not directly. It does use information from those archives to
build recipes which it sues to obtain the code. Essentially, straight.el
clones the package source repositories and builds it locally. It give a
lot more control and is really great if you want to modify the code
(basically, it creates a local fork). Of course, with all that extra
power, there is also additional responsibility placed on the user. You
may need to tell it which versions you want to use, which ones to 'pin'
to, which branches or fix merge conflicts when you have made changes and
want to update from the upstream soruces etc. I've been using it for
about 12 months and find it really good. YMMV.

Re: bug#59882: Multiple versions of Org in load-path problem

[Eli Zaretskii]

> Cc: emacs-orgmode@gnu.org, 59882@debbugs.gnu.org
> From: David Masterson <dsmasterson@gmail.com>
> Date: Thu, 08 Dec 2022 13:56:03 -0800
> 
> In my testing, I found a strange case where, in *scratch*, I get:
> 
> (message "%s" org-version)
> ;; Error undefined
> ;; Do 'C-h v org-version'
> (message "%s" org-version)
> 9.3
> 
> So, 'describe-variable' on org-version causes Org to be loaded?!?

Why is it so surprising?  Is this the first time you see that a Help
command loads something to do its job?

Re: bug#59882: Multiple versions of Org in load-path problem

[David Masterson]

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: emacs-orgmode@gnu.org, 59882@debbugs.gnu.org
>> From: David Masterson <dsmasterson@gmail.com>
>> Date: Thu, 08 Dec 2022 13:56:03 -0800
>> 
>> In my testing, I found a strange case where, in *scratch*, I get:
>> 
>> (message "%s" org-version)
>> ;; Error undefined
>> ;; Do 'C-h v org-version'
>> (message "%s" org-version)
>> 9.3
>> 
>> So, 'describe-variable' on org-version causes Org to be loaded?!?
>
> Why is it so surprising?  Is this the first time you see that a Help
> command loads something to do its job?

Yes, it is surprising.  How did Emacs know to load Org to find
org-version?  I didn't request it to by this action.  Or did it do a
package-activate-all (or similar) to ensure that everything was loaded?
That would be overkill.  Sometimes the best answer is 'nil'.

-- 
David Masterson

Re: bug#59882: Multiple versions of Org in load-path problem

[Eli Zaretskii]

> From: David Masterson <dsmasterson@gmail.com>
> Cc: michel@schi.nz,  emacs-orgmode@gnu.org,  59882@debbugs.gnu.org
> Date: Thu, 08 Dec 2022 23:43:05 -0800
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> So, 'describe-variable' on org-version causes Org to be loaded?!?
> >
> > Why is it so surprising?  Is this the first time you see that a Help
> > command loads something to do its job?
> 
> Yes, it is surprising.  How did Emacs know to load Org to find
> org-version?  I didn't request it to by this action.  Or did it do a
> package-activate-all (or similar) to ensure that everything was loaded?
> That would be overkill.  Sometimes the best answer is 'nil'.

I don't think Emacs does package-activate-all.  But Help commands
frequently load packages to provide the documentation of the symbols
you ask about.  If this is surprising, then I suggest to get used to
it.

Re: Multiple versions of Org in load-path problem

[Sharon Kimble]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Tim Cross <theophilusx@gmail.com> writes:

> David Masterson <dsmasterson@gmail.com> writes:
>
>> Tim Cross <theophilusx@gmail.com> writes:
>>
>>> David Masterson <dsmasterson@gmail.com> writes:
>>>
>>>> "Michel Schinz" <michel@schi.nz> writes:
>>>>
>>>>> Just for the record, I also ran into problems when installing Org 9.6
>>>>> using Emacs' package system on top of an older version that came with
>>>>> Emacs. If I tried to install it as usual (M-x list-packages, then
>>>>> install the package from there), I had errors during compilation related
>>>>> to `org-assert-version`, and then if I restarted Emacs, I would get a
>>>>> fatal error in an unrelated package.
>>>>>
>>>>> I managed to solve that problem by:
>>>>> 1. uninstalling Org 9.6 and exiting Emacs,
>>>>> 2. starting Emacs with -q,
>>>>> 3. installing Org 9.6 from there (using M-x list-packages as usual),
>>>>> 4. restarting Emacs.
>>>>
>>>> Interesting!  I tried this (essentially) and it worked for my case.  In
>>>> my case, I had a built-in Org-9.3 and I was trying to use list-packages
>>>> to install Org-9.6. I checked that using -q still added Org-9.3 to
>>>> the
>>>> load-path, but, since Org wasn't loaded, the install via list-packages
>>>> worked.
>>>>
>>>> The question is what's the proper way of doing this without '-q'?

This is what is in my 'init.el' file -

=====
;;; init.el --- sharon's config -*- eval: (read-only-mode 1) -*-
;; Make sure that Git version of Org mode is being loaded instead of the built-in version.
;;; (add-to-list 'load-path (expand-file-name "/home/boudiccas/.emacs.d/elpa/org-9.5.5"))
(add-to-list 'load-path (expand-file-name "/home/boudiccas/.emacs.d/elpa/org-9.6"))
(add-to-list 'load-path (expand-file-name "/home/boudiccas/git/org-contrib/lisp"))

(require 'package)
(setq package-enable-at-startup nil)
(package-initialize)

(require 'ob-tangle)
(org-babel-load-file "/home/boudiccas/.emacs.d/config22-2.org")    
=====

And the first thing that is loaded is org-9.6 *before* anything else! And you could equally have org-mode loading from a git repository on your hard drive, just by changing the filepath.

Super-easy without having to faff about with 'emacs -q' etc.

And as you can see I'm using org-9.6 from elpa, and I didn't move over to it until this morning, so far so good.  

Thanks
  Sharon.  
  
>>
>> [...]
>>
>>> I don't think there is any safe way to install an updated version of
>>> org-mode other than
>>>
>>> 1. Use the -q approach outlined above
>>
>> Thinking about it, this only works if Org is in elpa as melpa (etc.) are
>> not added to package-archives.  You'd have to do some handwritten elisp
>> out of *scratch* to setup package-archives if Org-9.6 was still coming
>> out of melpa. That's why this can only be labeled as a hack and not a
>> solution.
>>
>
> Well, yes, if your going to use this technique to load a package which
> is not in the default package archives you would need to add that
> archive first. People who use this technique often just have a
> 'update.el' file which they load/evaluate when starting Emacs with -q. 
>
>
>>> 2. Craft your init.el file such that org functionality is only loaded
>>> when explicitly requested and always update as the first action after
>>> starting emacs.
>>
>> In this case, something happened in package-install when trying to
>> install Org-9.6 with a built-in Org-9.3.  During the compilation check
>> (.el -> .elc) many files failed because the new 'org-assert-version'
>> macro was not defined.  Sort of like, after package-install started
>> working on Org-9.6, org-macs.el (where org-assert-version should be) got
>> loaded *before* the new load-path had been set causing it to load the
>> old one from 9.3.  Thereafter, everything went awry. 
>>
>
> You must have some custom code in your init.el or are using something
> like use-package as package.el doesn't try to install or upgrade
> packages during init by default.
>
> Where people can come undone is when they are using use-package and set
> the :ensure t option. In this case, use-package will not know about the
> bundled version and will attempt to install org from ELPA. If use
> package runs after org has already been loaded (possibly because some
> other package has been loaded which depends on/requires org mode and has
> loaded the bundled version) then things will break because you will end
> up with a mixed version install. This is why I always ensure org is the very
> first use-package in my init.el and it comes before any other code which
> loads or initialises anything.  
>
>>> The first approach is actually the easiest. The second is hard to get
>>> right and very fragile because packages like use-package and more
>>> specifically, other packages with leverage off org functionality, make it
>>> impossible to reliably know exactly when org is loaded.
>>
>> Using ':after" in use-package is supposed to help that, but I'm not sure
>> it is reliable.  Packages are often incomplete about what other packages
>> it depends on.
>>
>
> You cannot rely on :after. The problem is, other packages may require
> org functionality and will load org when they are loaded. This can be
> very subtle as there are a lot of packages which only make very small
> use of org mode, but even that requires that org mode is loaded. 
>
>>> An approach used by many 'canned' distributions is to postpone package
>>> updates. You have a function you run to check for updates which
>>> generates a list of packages to update and writes that list to a
>>> file. Each time emacs is started, it looks for this update list and if
>>> it finds it, it installs packages updates at the very beginning of the
>>> init process (before any of your other init.el code or custom
>>> blocks). The process also looks for org in the list of packages to
>>> update and if it is found, updates it first. 
>>
>> Probably doesn't work in this case as you would need to be able to use
>> package.el suggesting that the load-path has been updated for all
>> built-ins already.  The thing I note is that the load-path has already
>> been updated for built-ins at the beginning of
>> '~/.emacs.d/early-init.el', but the libraries haven't been loaded yet
>> (unless needed). That's okay if the newer version of a package then
>> cleanly replaces all the files in the old version.
>>
>
> It does work. Distributions like Sapcemacs use this approach and do not
> have the mixed versions issue. I've not looked closely at their
> implementation, but they well might do all this work as part of the
> early-init.el file.
>
>
>>> I don't think there is a safe way to load org mode after the init
>>> process i.e. after booting emacs by M-x package-update.
>>
>> Where is package-update called in the boot process of emacs?  I don't
>> see package-update in Emacs v2.7.
>>
>
> Sorry, I wasn't clear.
>
> What I meant is that there is no clean and reliable way of running the
> package update process after emacs has initialised. The package.el API
> has changed, but previously, you would run M-x package-update-packages
> to update installed packages. Now it is M-x package-update-all. With
> many packages, you can run this command and update your installed
> packages. However, more complex packages, like org-mode, cannot handle
> this due to potential mixed version problems - a problem which has
> existed for many years. 
>
> By default, there is no automated package update setting (unless that
> has been added in recent versions). There are some add-on packages which
> can do this and the elisp to make this happen is trivial (but much
> harder to do it and make it robust). 
>
>
>>> I've had good success using straight.el. I had to be careful regarding
>>> how I structured my init.el file (ensuring any straight stuff happens
>>> first and the first use package stanza is for org. The main reason
>>> straight works well for me is that my work flow is to do a M-x
>>> straight-pull-all when I want to update my packages. This does a git
>>> pull for all the sources, but does not do any build/install. This occurs
>>> when I next start Emacs and because I have all the straight stuff at the
>>> start and because org mode is the first straight-use-package, the update
>>> and install happens before any other org functionality is loaded,
>>> avoiding mixed version issues.
>>
>> Where do you get straight.el?  I don't see it in [m]elpa.
>
> No, to use straight.el, you add some code to your init.el which will
> retrieve the current straight.el file from the github
> repository. Straight.el does not use the ELPA/NONGNU ELPA/MELPA archives
> - at least not directly. It does use information from those archives to
> build recipes which it sues to obtain the code. Essentially, straight.el
> clones the package source repositories and builds it locally. It give a
> lot more control and is really great if you want to modify the code
> (basically, it creates a local fork). Of course, with all that extra
> power, there is also additional responsibility placed on the user. You
> may need to tell it which versions you want to use, which ones to 'pin'
> to, which branches or fix merge conflicts when you have made changes and
> want to update from the upstream soruces etc. I've been using it for
> about 12 months and find it really good. YMMV.

- -- 
A taste of linux = http://www.sharons.org.uk
TGmeds = http://www.tgmeds.org.uk
DrugFacts = https://www.drugfacts.org.uk
Debian 11.5, fluxbox 1.3.7, emacs 29.0.50, org 9.6 
-----BEGIN PGP SIGNATURE-----

iQJRBAEBCgA7FiEELSc/6QwVBIYugJDbNoGAGQr4g1sFAmOTWQodHGJvdWRpY2Nh
c0Bza2ltYmxlMDkucGx1cy5jb20ACgkQNoGAGQr4g1sBchAAkByRu8RZ6EV9WkAi
tbYOMCW9+rBQ9Oib2i6pj4XuGC1opjjjgMr84y2+UfgFm1bbcFo6bv1hi33UFvM2
9D8ORzSPW/wcPjGILcwHS6w1hbwkflB8qwRDwL7TxUFuZqCZ3mr9q7aLZR4ZVuf9
3WFzUHd30O61H6rLKrJWRPJgB2zYcUS4pbsiurOTK35sYoiogAshoV4akTNxQefV
TmLFcoHmJvK0M+L/5BhPJblvrG0jkudUyMCYPeAHGxDfDpXiW/aFdYJuibqhk4mm
nblCIe1EFBa5Lxu3m0lTkY7X9I/XtS1cf92gkj7qA9Y/ZYtVfWO54FyDA5sJFOZ2
aHcQrGS2vqoU1HNYv9u0CVj+/QJQbUSGrI60lrwIikOe+Xt+iHVlyx6dmmgKHLzt
YHuZVKwMZxOEJtUjVwHc23+MzDmHc1WbJ8o5yOomFnYivFg/P0GQpY0o9O3CDJCV
yV/ux7qPSPlyQZcJqKVHjrg05y/t76i5CPsiYAeUUQPa8jdU3F5SHxtpmuf9KPgQ
+5EmUYkYodcNyss93YAuPEamKI491nLuWkc+F4t+fi24QS/t3FJaemkfW+UIIBuk
gpg8/fgMkOdrs8E4IEwb7wZkCEHl52J0WypsGv9Of+vcrb5YXdPk7x/dJoHUmADm
SremPednv0xBoZCKg12orlIQzIM=
=cfTm
-----END PGP SIGNATURE-----

RE: bug#59882: Multiple versions of Org in load-path problem

[Cook, Malcolm]

> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Cc: emacs-orgmode@gnu.org, 59882@debbugs.gnu.org
> >> From: David Masterson <dsmasterson@gmail.com>
> >> Date: Thu, 08 Dec 2022 13:56:03 -0800
> >>
> >> In my testing, I found a strange case where, in *scratch*, I get:
> >>
> >> (message "%s" org-version)
> >> ;; Error undefined
> >> ;; Do 'C-h v org-version'
> >> (message "%s" org-version)
> >> 9.3
> >>
> >> So, 'describe-variable' on org-version causes Org to be loaded?!?
> >
> > Why is it so surprising?  Is this the first time you see that a Help
> > command loads something to do its job?
> 
> Yes, it is surprising.  How did Emacs know to load Org to find org-version? 

This is provided for via emacs' autoload by prefix mechanism.

See:

https://www.gnu.org/software/emacs/manual/html_node/elisp/Autoload-by-Prefix.html

What you observed will NOT happen if you first run:

	(clrhash  definition-prefixes)

> didn't request it to by this action.  Or did it do a package-activate-all (or
> similar) to ensure that everything was loaded?
> That would be overkill.  Sometimes the best answer is 'nil'.
> 
> --
> David Masterson

RE: Multiple versions of Org in load-path problem

[Cook, Malcolm]

> Adding this to bug #59882
> 
> "Michel Schinz" <michel@schi.nz> writes:
> 
> > Just for the record, I also ran into problems when installing Org 9.6
> > using Emacs' package system on top of an older version that came with
> > Emacs. If I tried to install it as usual (M-x list-packages, then
> > install the package from there), I had errors during compilation
> > related to `org-assert-version`, and then if I restarted Emacs, I
> > would get a fatal error in an unrelated package.
> >
> > I managed to solve that problem by:
> > 1. uninstalling Org 9.6 and exiting Emacs, 2. starting Emacs with -q,
> > 3. installing Org 9.6 from there (using M-x list-packages as usual),
> > 4. restarting Emacs.
> 
> Interesting!  I tried this (essentially) and it worked for my case.  In my case, I
> had a built-in Org-9.3 and I was trying to use list-packages to install Org-9.6. I
> checked that using -q still added Org-9.3 to the load-path, but, since Org
> wasn't loaded, the install via list-packages worked.
> 
> The question is what's the proper way of doing this without '-q'?
> 
> > I'm not sure this is related to your problem, or whether that helps
> > (but I hope it does)...
> 
> I think it does.
> 
> Side note:
> 
> In my testing, I found a strange case where, in *scratch*, I get:
> 
> (message "%s" org-version)
> ;; Error undefined
> ;; Do 'C-h v org-version'
> (message "%s" org-version)
> 9.3
> 
> So, 'describe-variable' on org-version causes Org to be loaded?!?  [Cook, Malcolm] 

Yes.  See https://www.gnu.org/software/emacs/manual/html_node/elisp/Autoload-by-Prefix.html

But it will NOT be autoloaded if you first:

(clrhash  definition-prefixes)

HTH & YMMV,

Malcolm

> Why do I
> have a feeling this is related to this bug?

Yeah, I'm not sure what you consider the bug to be now.  Remind me?

> 
> --
> David Masterson

Re: bug#59882: Multiple versions of Org in load-path problem

[David Masterson]

"Cook, Malcolm" <MEC@stowers.org> writes:

>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> Cc: emacs-orgmode@gnu.org, 59882@debbugs.gnu.org
>> >> From: David Masterson <dsmasterson@gmail.com>
>> >> Date: Thu, 08 Dec 2022 13:56:03 -0800
>> >>
>> >> In my testing, I found a strange case where, in *scratch*, I get:
>> >>
>> >> (message "%s" org-version)
>> >> ;; Error undefined
>> >> ;; Do 'C-h v org-version'
>> >> (message "%s" org-version)
>> >> 9.3
>> >>
>> >> So, 'describe-variable' on org-version causes Org to be loaded?!?
>> >
>> > Why is it so surprising?  Is this the first time you see that a Help
>> > command loads something to do its job?
>> 
>> Yes, it is surprising.  How did Emacs know to load Org to find org-version? 
>
> This is provided for via emacs' autoload by prefix mechanism.
>
> See:
>
> https://www.gnu.org/software/emacs/manual/html_node/elisp/Autoload-by-Prefix.html
>
> What you observed will NOT happen if you first run:
>
> 	(clrhash  definition-prefixes)

Ah! Thank you.  In many ways, I still think of Emacs from what I learned
about it in the early 80s (yes, pre-Elisp).  I didn't expect this and,
so, didn't investigate the possibility that it's planned (the Info
documentation has gotten huge).  Apologies to Eli.

-- 
David Masterson

Re: bug#59882: Multiple versions of Org in load-path problem

[Eli Zaretskii]

> From: David Masterson <dsmasterson@gmail.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  "michel@schi.nz" <michel@schi.nz>,
>   "emacs-orgmode@gnu.org" <emacs-orgmode@gnu.org>,  "59882@debbugs.gnu.org"
>  <59882@debbugs.gnu.org>
> Date: Fri, 09 Dec 2022 11:51:29 -0800
> 
> > https://www.gnu.org/software/emacs/manual/html_node/elisp/Autoload-by-Prefix.html
> >
> > What you observed will NOT happen if you first run:
> >
> > 	(clrhash  definition-prefixes)
> 
> Ah! Thank you.  In many ways, I still think of Emacs from what I learned
> about it in the early 80s (yes, pre-Elisp).  I didn't expect this and,
> so, didn't investigate the possibility that it's planned (the Info
> documentation has gotten huge).  Apologies to Eli.

No apologies needed, we all learn something new about Emacs every day.

Re: bug#59882: Multiple versions of Org in load-path problem

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> In my testing, I found a strange case where, in *scratch*, I get:
>> 
>> (message "%s" org-version)
>> ;; Error undefined
>> ;; Do 'C-h v org-version'
>> (message "%s" org-version)
>> 9.3
>> 
>> So, 'describe-variable' on org-version causes Org to be loaded?!?
>
> Why is it so surprising?  Is this the first time you see that a Help
> command loads something to do its job?

Please note that David's recipe is _not_ what I described in the bug
report.

I am referring to the problem people run into when using M-x
package-install.

AFAIU, package-install must not depend on the older version of a package
being loaded/not loaded.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Multiple versions of Org in load-path problem

[Tom Gillespie]

> I don't think there is any safe way to install an updated version of
> org-mode other than

I don't know whether it counts as safe, but since I don't see it
mentioned anywhere else in the thread you can do the following.

(require 'package)
(unless package--initialized
  (package-initialize))
(assq-delete-all 'org package--builtins)
(assq-delete-all 'org package--builtin-versions)
(package-install 'org)

Re: bug#59882: Multiple versions of Org in load-path problem

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> So, 'describe-variable' on org-version causes Org to be loaded?!?
>
> Why is it so surprising?  Is this the first time you see that a Help
> command loads something to do its job?

On a related note, is it normal that package like org-mouse.el gets
loaded upon calling C-h f org-mouse- <TAB> completion prompt?
org-mouse.el does not contain a single autoload cookie.

I am asking because org-mouse.el advises some functions and its loading
may cause unexpected consequences. See
https://orgmode.org/list/87r0x6sju1.fsf@fastmail.fm

At the same time, we have users complaining about some defcustoms not
being available without loading.
https://orgmode.org/list/25496.19258.682537.503614@gargle.gargle.HOWL

I feel confused about the situation.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bug#59882: Multiple versions of Org in load-path problem

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: David Masterson <dsmasterson@gmail.com>, emacs-orgmode@gnu.org,
>  michel@schi.nz, 59882@debbugs.gnu.org
> Date: Wed, 14 Dec 2022 09:51:01 +0000
> 
> On a related note, is it normal that package like org-mouse.el gets
> loaded upon calling C-h f org-mouse- <TAB> completion prompt?
> org-mouse.el does not contain a single autoload cookie.

I think it's expected, because the doc strings need to be loaded.
Stefan, am I right?

> I am asking because org-mouse.el advises some functions and its loading
> may cause unexpected consequences. See
> https://orgmode.org/list/87r0x6sju1.fsf@fastmail.fm

I think org-mouse.el should be fixed not to cause such effects just by
loading it.  It should do that only when it is actually used.

> At the same time, we have users complaining about some defcustoms not
> being available without loading.
> https://orgmode.org/list/25496.19258.682537.503614@gargle.gargle.HOWL

That's too vague: it talks about "whether a variable exists", which is
ambiguous.  That a variable is not boundp until its definition is
loaded is perfectly expected, and I don't think defcustom changes that
in any way (unless you autoload the defcustom, which is considered not
the best style around here).

So I guess some clarification is in order: what exactly was the
problem with that variable being "nonexistent"?

Re: bug#59882: Multiple versions of Org in load-path problem

[Ihor Radchenko]

Eli Zaretskii <eliz@gnu.org> writes:

>> I am asking because org-mouse.el advises some functions and its loading
>> may cause unexpected consequences. See
>> https://orgmode.org/list/87r0x6sju1.fsf@fastmail.fm
>
> I think org-mouse.el should be fixed not to cause such effects just by
> loading it.  It should do that only when it is actually used.

Yes, we plan to do exactly this https://orgmode.org/list/87zgbqqukg.fsf@localhost
However, it will be a breaking backward-incompatible change. And we have
similar issues with several important Org components (ol-* and ob-*). In
particular, ob-*.el packages affect Org simply by defining function
symbols with specific name patters (by ob-core.el design).

Achieving no side effects is difficult in practice.

Also, note that side effects are not uncommon in Elisp packages.
Including, built-ins, like rect.el.
dired+ comes to mind among popular third-party packages.

>> At the same time, we have users complaining about some defcustoms not
>> being available without loading.
>> https://orgmode.org/list/25496.19258.682537.503614@gargle.gargle.HOWL
>
> That's too vague: it talks about "whether a variable exists", which is
> ambiguous.  That a variable is not boundp until its definition is
> loaded is perfectly expected, and I don't think defcustom changes that
> in any way (unless you autoload the defcustom, which is considered not
> the best style around here).
>
> So I guess some clarification is in order: what exactly was the
> problem with that variable being "nonexistent"?

Agree. See the quote below:
https://list.orgmode.org/orgmode/25497.41832.663858.836948@gargle.gargle.HOWL/

    Like I said in another message that I sent just before receiving yours
    my conclusion came from the fact that hitting 'C-h v' with the cursor
    on 'org-goto-interface' provided nothing.  It was the first time this
    ever happened to me.  I did try to explicitly enter the variable's
    name by entering 'org-goto<TAB>', which (like 'org-go<TAB>') is not
    enough: it is completed to 'org--goto'.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bug#59882: Multiple versions of Org in load-path problem

[Stefan Monnier]

>> On a related note, is it normal that package like org-mouse.el gets
>> loaded upon calling C-h f org-mouse- <TAB> completion prompt?

Yes.  See `help-enable-completion-autoload`.

>> I am asking because org-mouse.el advises some functions and its loading
>> may cause unexpected consequences. See
>> https://orgmode.org/list/87r0x6sju1.fsf@fastmail.fm
> I think org-mouse.el should be fixed not to cause such effects just by
> loading it.  It should do that only when it is actually used.

Exactly.  Merely loading a file should not affect Emacs's behavior in
any significant way.


        Stefan

Re: bug#59882: Multiple versions of Org in load-path problem

[Stefan Monnier]

> similar issues with several important Org components (ol-* and ob-*). In
> particular, ob-*.el packages affect Org simply by defining function
> symbols with specific name patters (by ob-core.el design).
> Achieving no side effects is difficult in practice.

That's true.  It can be a balancing act.

> Also, note that side effects are not uncommon in Elisp packages.
> Including, built-ins, like rect.el.
> dired+ comes to mind among popular third-party packages.

Most of them are plain bugs, tho (typically fixed by introducing an
autoloaded minor-mode, so the package is activated by enabling the
minor mode rather than by `require`ing the package).

>     Like I said in another message that I sent just before receiving yours
>     my conclusion came from the fact that hitting 'C-h v' with the cursor
>     on 'org-goto-interface' provided nothing.  It was the first time this
>     ever happened to me.  I did try to explicitly enter the variable's
>     name by entering 'org-goto<TAB>', which (like 'org-go<TAB>') is not
>     enough: it is completed to 'org--goto'.

AFAICT this is a missing feature: we obey `help-enable-autoload` in
`describe-function` but we fail to do the same autoloading dance in
`describe-variable`.


        Stefan

Re: bug#59882: Multiple versions of Org in load-path problem

[Max Nikulin]

On 14/12/2022 21:35, Stefan Monnier wrote:
> 
>>      Like I said in another message that I sent just before receiving yours
>>      my conclusion came from the fact that hitting 'C-h v' with the cursor
>>      on 'org-goto-interface' provided nothing.  It was the first time this
>>      ever happened to me.  I did try to explicitly enter the variable's
>>      name by entering 'org-goto<TAB>', which (like 'org-go<TAB>') is not
>>      enough: it is completed to 'org--goto'.
> 
> AFAICT this is a missing feature: we obey `help-enable-autoload` in
> `describe-function` but we fail to do the same autoloading dance in
> `describe-variable`.

I think, the difference is interactive vs. non-interactive calls rather 
than `describe-function' vs. `describe-variable'. What misses this 
feature is `customize-variable'.

Completion failure for "org-to" happened because result of 
`register-definition-prefixes' calls is not considered as options. I am 
aware that it may give false positives, but I still believe they should 
be added.

Though I believe that org mixed version issue happens due to 
transitional dependency of some third party package on org or something 
like (require 'org-protocol) that can not be loaded on demand.

RE: bug#59882: Multiple versions of Org in load-path problem

[Cook, Malcolm]

>On a related note, is it normal that package like org-mouse.el gets
>loaded upon calling C-h f org-mouse- <TAB> completion prompt?
>org-mouse.el does not contain a single autoload cookie.

There need not be which is apparently by design:

The autoload happens as a consequence of emacs' documented feature to [Autoload by Prefix](https://www.gnu.org/software/emacs/manual/html_node/elisp/Autoload-by-Prefix.html) 

Given that:

(equal (gethash "org-mouse-"  definition-prefixes) '("org-mouse"))

Re: bug#59882: Multiple versions of Org in load-path problem

[Stefan Monnier]

Max Nikulin [2022-12-14 23:02:53] wrote:
> On 14/12/2022 21:35, Stefan Monnier wrote:
>>>      Like I said in another message that I sent just before receiving yours
>>>      my conclusion came from the fact that hitting 'C-h v' with the cursor
>>>      on 'org-goto-interface' provided nothing.  It was the first time this
>>>      ever happened to me.  I did try to explicitly enter the variable's
>>>      name by entering 'org-goto<TAB>', which (like 'org-go<TAB>') is not
>>>      enough: it is completed to 'org--goto'.
>> AFAICT this is a missing feature: we obey `help-enable-autoload` in
>> `describe-function` but we fail to do the same autoloading dance in
>> `describe-variable`.
>
> I think, the difference is interactive vs. non-interactive calls rather than
> `describe-function' vs. `describe-variable'. What misses this feature is
> `customize-variable'.
>
> Completion failure for "org-to" happened because result of
> `register-definition-prefixes' calls is not considered as options. I am
> aware that it may give false positives, but I still believe they should
> be added.

I'm sorry, but I don't understand what you're saying.

> Though I believe that org mixed version issue happens due to transitional
> dependency of some third party package on org or something like (require
> 'org-protocol) that can not be loaded on demand.

Hmm... indeed the "Subject:" talks about mixed versions, yet I was
discussing something not directly related.  Looks like I got off-topic
somewhere along the way without noticing, sorry.


        Stefan

Re: bug#59882: Multiple versions of Org in load-path problem

[Ihor Radchenko]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Though I believe that org mixed version issue happens due to transitional
>> dependency of some third party package on org or something like (require
>> 'org-protocol) that can not be loaded on demand.
>
> Hmm... indeed the "Subject:" talks about mixed versions, yet I was
> discussing something not directly related.  Looks like I got off-topic
> somewhere along the way without noticing, sorry.

And the off-topic is what has been CCed to Org mailing list :)

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bug#59882: Multiple versions of Org in load-path problem

[Max Nikulin]

On 15/12/2022 11:17, Stefan Monnier wrote:
> Max Nikulin [2022-12-14 23:02:53] wrote:
>> Completion failure for "org-to" happened because result of
>> `register-definition-prefixes' calls is not considered as options. I am
>> aware that it may give false positives, but I still believe they should
>> be added.
> 
> I'm sorry, but I don't understand what you're saying.

My bad. "org-go" is completed to "org--goto" while "org-goto-" is not 
offered. The latter might be a variant because "org-goto" is registered 
as an autoload prefix. Ihor kindly created https://debbugs.gnu.org/60085 
"`help-enable-autoload' is not fully obeyed", so it is better to discuss 
it there.

>> Though I believe that org mixed version issue happens due to transitional
>> dependency of some third party package on org or something like (require
>> 'org-protocol) that can not be loaded on demand.
> 
> Hmm... indeed the "Subject:" talks about mixed versions, yet I was
> discussing something not directly related.  Looks like I got off-topic
> somewhere along the way without noticing, sorry.

I have impression that there is no recipe how to reproduce this bug yet, 
just reports that people have faced it. So aspects of autoload may be 
considered as a part of brainstorm.

Other ideas when built-in Org version may be loaded too early:
- restoring previous session.
- user configured Org agenda as startup buffer.

Can it happen that already loaded version affects compilation of a new 
version by package-install.

Re: bug#59882: Multiple versions of Org in load-path problem

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> I have impression that there is no recipe how to reproduce this bug yet, 
> just reports that people have faced it. So aspects of autoload may be 
> considered as a part of brainstorm.
>
> Other ideas when built-in Org version may be loaded too early:
> - restoring previous session.
> - user configured Org agenda as startup buffer.
>
> Can it happen that already loaded version affects compilation of a new 
> version by package-install.

Yes. It is what appears to be happening:

https://irreal.org/blog/?p=10996
https://irreal.org/blog/?p=10999

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bug#59882: Multiple versions of Org in load-path problem

[Max Nikulin]

On 16/12/2022 21:47, Ihor Radchenko wrote:
>> Can it happen that already loaded version affects compilation of a new
>> version by package-install.
> 
> Yes. It is what appears to be happening:
> 
> https://irreal.org/blog/?p=10996
> https://irreal.org/blog/?p=10999

I would not call it steps to reproduce.

 From my point of view, there is a number of issues with updating of 
packages from emacs session.

1. If compilation of a package failed then it is not withdrawn from the 
load path.

2. There is no reliable way to determine if package compilation failed. 
(byte-recompile-directory DIR 0 t) returns human readable string, not 
signalling an error or returning a value convenient to check in the 
code. It is the cause of the previous issue.

3. Emacs package system is not ready for complex multifile packages 
containing macro definitions. It does not push user to update packages 
just after emacs restart in a clean state. As a result, files may be 
compiled with old or missed macro definitions.

4. Because of the item 2 a package has no chance to implement reliable 
internal sanity check that prevents updating to a broken state.

I have tried the following in Emacs-27. For Emacs >= 29 add -L to Org <= 
9.5.

- "emacs -l org" to simulate a regular user session with some opened files.
- Update to Org-9.6 from ELPA.
- Close emacs and start it again "emacs -l org"

Result:
byte-code: Invalid function: org-assert-version

Notice attempt to load Org-9.6 despite warnings and even error during 
update:

> Compiling file /home/emcs/.emacs.d/elpa/org-9.6/ob-C.el at Sun Dec 18 04:30:49 2022
> Entering directory ‘/home/emcs/.emacs.d/elpa/org-9.6/’
> 
> In end of data:
> ob-C.el:509:1:Warning: the function ‘org-assert-version’ is not known to be
>     defined. 
...
> Compiling file /home/emcs/.emacs.d/elpa/org-9.6/ob-matlab.el at Sun Dec 18 04:30:50 2022
> ob-matlab.el:43:1:Error: Symbol’s function definition is void: org-assert-version

So emacs can not handle update of a package with changed macro 
definitions when they reside in another package file.

Re: bug#59882: Multiple versions of Org in load-path problem

[Stefan Monnier]

> - "emacs -l org" to simulate a regular user session with some opened files.
> - Update to Org-9.6 from ELPA.
> - Close emacs and start it again "emacs -l org"
>
> Result:
> byte-code: Invalid function: org-assert-version

Hmm... there's code in `package.el` intended to avoid that problem, but
clearly it's failing here.  Could Someone™ investigate why that code
doesn't do its job.

The code in question is in `package--reload-previously-loaded` and
my understanding is that it should cause `org-macs` to be reloaded from
the Org-9.6 version of the code before the new code gets compiled, and
that new version does define the `org-assert-version` version, so those
macro calls should then be compiled correctly.

[ The approach used in `package--reload-previously-loaded` has its
  weaknesses, but AFAIK it *should* be able to avoid the above error.  ]


        Stefan

Re: bug#59882: Multiple versions of Org in load-path problem

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> I have tried the following in Emacs-27. For Emacs >= 29 add -L to Org <= 
> 9.5.
>
> - "emacs -l org" to simulate a regular user session with some opened files.
> - Update to Org-9.6 from ELPA.
> - Close emacs and start it again "emacs -l org"
>
> Result:
> byte-code: Invalid function: org-assert-version

I tried to follow these steps, but unfortunately I am unable to
reproduce. Everything works fine using Emacs 27 on my side. Strange.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bug#59882: Multiple versions of Org in load-path problem

[Max Nikulin]

On 18/12/2022 22:40, Stefan Monnier wrote:
> Could Someone™ investigate why that code
> doesn't do its job.
> 
> The code in question is in `package--reload-previously-loaded` and
> my understanding is that it should cause `org-macs` to be reloaded

I do not see obvious flaws in `package--reload-previously-loaded' code. 
My guess is that we get reports from users who have installed released 
versions of emacs.

Notice that my complain concerning failure `byte-recompile-directory', 
hidden from its caller, is still valid.

On 25/12/2022 16:48, Ihor Radchenko wrote:
> Max Nikulin writes:
> 
>> I have tried the following in Emacs-27. For Emacs >= 29 add -L to Org <=
>> 9.5.

Stefan convinced me that I was too optimistic concerning Emacs-29.

>> - "emacs -l org" to simulate a regular user session with some opened files.
>> - Update to Org-9.6 from ELPA.
>> - Close emacs and start it again "emacs -l org"
>>
>> Result:
>> byte-code: Invalid function: org-assert-version
> 
> I tried to follow these steps, but unfortunately I am unable to
> reproduce. Everything works fine using Emacs 27 on my side. Strange.

I tried it once more in a minimal LXC container with Ubuntu-22.04 LTS 
jammy. This time I even removed the elpa-org-9.5.2 system package, so 
built-in version of org is really used. So, emacs-27

- emacs -l org
- M-x list-packages RET
- / n org RET
- move cursor to org, install it
- i x y
- compilation buffers pops up with warning and errors.

Due to "-l org" some libraries like org, org-macs, ob are loaded. Files 
that `require' already loaded libraries generate warnings

Compiling file /home/emcs/.emacs.d/elpa/org-9.6/ob-R.el at Sun Dec 18 
04:30:49 2022

In end of data:
ob-R.el:562:1:Warning: the function ‘org-assert-version’ is not known to be
     defined.

because `org-assert-version' is defined in org-macs.el, but the old 
version without it is already loaded. Call to `org-assert-version' is 
just added to the .elc file.

E.g. org-keys.el produce the same warning. Another case is

Compiling file /home/emcs/.emacs.d/elpa/org-9.6/ob-matlab.el at Sun Dec 
18 04:30:50 2022
ob-matlab.el:43:1:Error: Symbol’s function definition is void: 
org-assert-version

This file contains "(require 'ob-octave)". Attempt of load new version 
of ob-octave fails because during execution of it contents causes an 
error due to the `org-assert-version' call.

In particular, compilation of org.el fails because the new version has 
new "(require 'org-fold)" and the latter also tries to call undefined 
`org-assert-version'.

So at this point we have partially compiled broken install. Some files 
are not compiled, others have call to `org-assert-version' instead of 
its expansion.

Let's try to start emacs again and load org. Since org.el is not 
compiled, "(org-assert-version)" inside it is unable to catch any 
problem due to mixed-version compilation. Now `org-release' returns new 
version. The failure happens during "(require 'org-keys)" that was 
compiled with the warning.

Debugger entered--Lisp error: (invalid-function org-assert-version)
   org-assert-version()
   byte-code("\300\301!\210\302 \210\300\303!\210\300\304!\207" [require 
org-macs org-assert-version cl-lib oc] 2)
   require(org-keys)

So we have an obscure load time error.  I would say that compile *error* 
messages are obscure as well because it is unclear that errors happen in 
`require'. E.g. when compiling org-matlab.el the reported line number is 
for "(require 'ob-maxima)" while compiling of ob.maxima.el finished with 
a warning, not an error and it is really confusing. "ob-octave:34:" in 
addition to "ob-matlab:43:" during compiling of ob-matlab.el would be 
more informative.

Re: bug#59882: Multiple versions of Org in load-path problem

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

>> I tried to follow these steps, but unfortunately I am unable to
>> reproduce. Everything works fine using Emacs 27 on my side. Strange.
>
> I tried it once more in a minimal LXC container with Ubuntu-22.04 LTS 
> jammy. This time I even removed the elpa-org-9.5.2 system package, so 
> built-in version of org is really used. So, emacs-27
>
> - emacs -l org
> - M-x list-packages RET
> - / n org RET
> - move cursor to org, install it

On my side, it will give nothing useful: the package list is not updated
and I get:

Package org is built-in.

     Status: Built-In.
    Version: 9.4.4
    Summary: Export Framework for Org Mode

Detailed steps I tried (with package refresh) using with-emacs.sh [1]:

1. ~/Git/with-emacs.sh/with-emacs.sh -e emacs-27 -O -- -l org
2. M-x package-list-packages RET
3. C-s outline-based RET
4. RET
5. C-x o C-s instal RET
6. RET y
7. The warnings buffer appears, but only contains
   Warning (defvaralias): Overwriting value of ‘org-tab-first-hook’ by aliasing to ‘org-cycle-tab-first-hook’

[1] https://github.com/alphapapa/with-emacs.sh

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bug#59882: Multiple versions of Org in load-path problem

[Ihor Radchenko]

Ihor Radchenko <yantar92@posteo.net> writes:

> 1. ~/Git/with-emacs.sh/with-emacs.sh -e emacs-27 -O -- -l org

I was finally able to reproduce using emacs-26.

Maybe your Emacs 27 is older than mine? I have GNU Emacs 27.2

I also tried with Emacs 28. Not able to reproduce.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bug#59882: Multiple versions of Org in load-path problem

[Max Nikulin]

On 26/12/2022 18:01, Ihor Radchenko wrote:
> Ihor Radchenko writes:
>> 1. ~/Git/with-emacs.sh/with-emacs.sh -e emacs-27 -O -- -l org
> I was finally able to reproduce using emacs-26.
> 
> Maybe your Emacs 27 is older than mine? I have GNU Emacs 27.2

Debian stable and Ubuntu LTS have Emacs-27.1 only, so I am testing 
1:27.1+1-3ubuntu5

Following the with-emacs.sh script I have tried

mkdir /tmp/emcs
emacs --quick --eval '(setq user-emacs-directory "/tmp/emcs/")' --eval 
'(setq user-init-file "/tmp/emcs/init.el")' -l package --eval '(setq 
package-user-dir "/tmp/emcs/elpa/")' --eval '(package-initialize)' 
--eval '(package-refresh-contents)' -l org

I had no problem to select Org-9.6 from the buffer generated by 
`list-packages'. Compilation errors and the error during next startup 
are the same as I reported before.

I found just single commit to lisp/emacs-lisp/package.el between 
emacs-27.1 and emacs-27.2:

1fc9de4b81c 2020-10-30 19:20:24 -0700 Glenn Morris: Improve 
reproducibility of generated -pkg.el files

So I am still surprised that you can not reproduce the issue.

Re: bug#59882: Multiple versions of Org in load-path problem

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> I found just single commit to lisp/emacs-lisp/package.el between 
> emacs-27.1 and emacs-27.2:
>
> 1fc9de4b81c 2020-10-30 19:20:24 -0700 Glenn Morris: Improve 
> reproducibility of generated -pkg.el files
>
> So I am still surprised that you can not reproduce the issue.

Can you then try to test using Emacs 28?
The main question if whether this has been fixed in newer Emacs releases
or it is also something to do with OS environment.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bug#59882: Multiple versions of Org in load-path problem

[Max Nikulin]

On 27/12/2022 16:47, Ihor Radchenko wrote:
> Can you then try to test using Emacs 28?
> The main question if whether this has been fixed in newer Emacs releases
> or it is also something to do with OS environment.

I see quite the same issue with Emacs-28.2 in Debian testing. Compile 
buffer displays a bit more warnings and usual `org-assert-version' 
warnings and errors are present as well. It might have another level of 
complexity due to .eln files. I am unsure what happens with calls to 
undefined macros.

Re: bug#59882: Multiple versions of Org in load-path problem

[Ihor Radchenko]

Max Nikulin <manikulin@gmail.com> writes:

> On 27/12/2022 16:47, Ihor Radchenko wrote:
>> Can you then try to test using Emacs 28?
>> The main question if whether this has been fixed in newer Emacs releases
>> or it is also something to do with OS environment.
>
> I see quite the same issue with Emacs-28.2 in Debian testing. Compile 
> buffer displays a bit more warnings and usual `org-assert-version' 
> warnings and errors are present as well. It might have another level of 
> complexity due to .eln files. I am unsure what happens with calls to 
> undefined macros.

I asked people around to test using Debian, and we do have a
confirmation that Debian + Emacs 27 and Debian + Emacs 28 do trigger the
error.

I also installed Debian 11.6.0 on virtual machine, and I was also able to
trigger the error, following the provided steps, using the Emacs 27
installed via apt-get.

The problem seems to be real and appears to be some combination of
Debian/Ubuntu + Emacs.

Considering the popularity of Debian-based distros, may someone take a
closer look on what may be going on? Since the latest Emacs release also
suffers from the problem, I am afraid that the issue will be present in
the coming Emacs 29 as well (I recall no related fixes in recent Emacs).

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bug#59882: Multiple versions of Org in load-path problem

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>, emacs-orgmode@gnu.org, Eli
>  Zaretskii <eliz@gnu.org>, 59882@debbugs.gnu.org
> Date: Fri, 03 Feb 2023 11:02:21 +0000
> 
> Considering the popularity of Debian-based distros, may someone take a
> closer look on what may be going on? Since the latest Emacs release also
> suffers from the problem, I am afraid that the issue will be present in
> the coming Emacs 29 as well (I recall no related fixes in recent Emacs).

I don't know enough about package.el's peculiarities, sorry.  And my
plate is too full ATM.  So I'm afraid someone else will have to
investigate this.  (I think Debian downstream maintainers are the
natural candidates, so I've taken the liberty of CC'ing them.)

Re: bug#59882: Multiple versions of Org in load-path problem

[Tim Cross]

Ihor Radchenko <yantar92@posteo.net> writes:

> Max Nikulin <manikulin@gmail.com> writes:
>
>> On 27/12/2022 16:47, Ihor Radchenko wrote:
>>> Can you then try to test using Emacs 28?
>>> The main question if whether this has been fixed in newer Emacs releases
>>> or it is also something to do with OS environment.
>>
>> I see quite the same issue with Emacs-28.2 in Debian testing. Compile 
>> buffer displays a bit more warnings and usual `org-assert-version' 
>> warnings and errors are present as well. It might have another level of 
>> complexity due to .eln files. I am unsure what happens with calls to 
>> undefined macros.
>
> I asked people around to test using Debian, and we do have a
> confirmation that Debian + Emacs 27 and Debian + Emacs 28 do trigger the
> error.
>
> I also installed Debian 11.6.0 on virtual machine, and I was also able to
> trigger the error, following the provided steps, using the Emacs 27
> installed via apt-get.
>
> The problem seems to be real and appears to be some combination of
> Debian/Ubuntu + Emacs.
>
> Considering the popularity of Debian-based distros, may someone take a
> closer look on what may be going on? Since the latest Emacs release also
> suffers from the problem, I am afraid that the issue will be present in
> the coming Emacs 29 as well (I recall no related fixes in recent Emacs).

I don't run Debian or Ubuntu anymore. However, I do recall that debian
does use a modified Emacs startup which is not part of the standard
Emacs distribution. They do this to enable the ability to have multiple
versions of Emacs installed at the same time.

If you are unable to reproduce the issue with an Emacs built from
sources and only from the versions of Emacs installed bia apt, then I
would be looking at the sources for the apt package and the modified
*.el files it uses as it could be something in there which is triggering
the issue (seem unlikely to be coincidence that a customization to allow
running of multiple versions is also triggering a version conflict in
org?)

. 

Re: bug#59882: Multiple versions of Org in load-path problem

[Gregor Zattler]

Hi Tim, org-mode and emacs developers,
* Tim Cross <theophilusx@gmail.com> [2023-02-04; 07:01 +11]:
> Ihor Radchenko <yantar92@posteo.net> writes:
>> Max Nikulin <manikulin@gmail.com> writes:
>>> On 27/12/2022 16:47, Ihor Radchenko wrote:
>>>> Can you then try to test using Emacs 28?
>>>> The main question if whether this has been fixed in newer Emacs releases
>>>> or it is also something to do with OS environment.
>>> I see quite the same issue with Emacs-28.2 in Debian testing. Compile
>>> buffer displays a bit more warnings and usual `org-assert-version'
>>> warnings and errors are present as well. It might have another level of
>>> complexity due to .eln files. I am unsure what happens with calls to
>>> undefined macros.
>>
>> I asked people around to test using Debian, and we do have a
>> confirmation that Debian + Emacs 27 and Debian + Emacs 28 do trigger the
>> error.
>>
>> I also installed Debian 11.6.0 on virtual machine, and I was also able to
>> trigger the error, following the provided steps, using the Emacs 27
>> installed via apt-get.
>>
>> The problem seems to be real and appears to be some combination of
>> Debian/Ubuntu + Emacs.
>>
>> Considering the popularity of Debian-based distros, may someone take a
>> closer look on what may be going on? Since the latest Emacs release also
>> suffers from the problem, I am afraid that the issue will be present in
>> the coming Emacs 29 as well (I recall no related fixes in recent Emacs).
>
> I don't run Debian or Ubuntu anymore. However, I do recall that debian
> does use a modified Emacs startup which is not part of the standard
> Emacs distribution. They do this to enable the ability to have multiple
> versions of Emacs installed at the same time.

that would be /usr/share/emacs/site-lisp/debian-startup.el


Ciao; Gregor
--
 -... --- .-. . -.. ..--.. ...-.-

Re: bug#59882: Multiple versions of Org in load-path problem

[Max Nikulin]

On 15/02/2023 05:53, Gregor Zattler wrote:
> * Tim Cross [2023-02-04; 07:01 +11]:
>>
>> I don't run Debian or Ubuntu anymore. However, I do recall that debian
>> does use a modified Emacs startup which is not part of the standard
>> Emacs distribution. They do this to enable the ability to have multiple
>> versions of Emacs installed at the same time.
> 
> that would be /usr/share/emacs/site-lisp/debian-startup.el

This file should be ignored when Emacs is started with --quick (-Q) 
option, see
https://debbugs.gnu.org/cgi/bugreport.cgi?bug=59882#86

I have not checked patches that may be applied to Emacs core files.

Re: bug#59882: Multiple versions of Org in load-path problem

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > >> I don't run Debian or Ubuntu anymore. However, I do recall that debian
  > >> does use a modified Emacs startup which is not part of the standard
  > >> Emacs distribution. They do this to enable the ability to have multiple
  > >> versions of Emacs installed at the same time.

Would we like to add a feature like that to Emacs?
I don't know how much work it would be, but I think
it would be a desirable change if it is easy.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: bug#59882: Multiple versions of Org in load-path problem

[Eli Zaretskii]

> Cc: emacs-orgmode@gnu.org, 59882@debbugs.gnu.org
> From: Richard Stallman <rms@gnu.org>
> Date: Tue, 21 Feb 2023 00:22:04 -0500
> 
>   > >> I don't run Debian or Ubuntu anymore. However, I do recall that debian
>   > >> does use a modified Emacs startup which is not part of the standard
>   > >> Emacs distribution. They do this to enable the ability to have multiple
>   > >> versions of Emacs installed at the same time.
> 
> Would we like to add a feature like that to Emacs?
> I don't know how much work it would be, but I think
> it would be a desirable change if it is easy.

I'm not sure I understand: feature like what?  If you are talking
about having several Emacs versions installed on the same system, then
I think we already support that: I use that all the time.  That's why
we install various files in version-specific directories.

If you mean something else, please elaborate.  (The bug discussion is
very long, and the above quote is from a tangent, so I'm not sure it's
worth my while reading all the discussion from the beginning trying to
understand what is being alluded to here.)

Re: bug#59882: Multiple versions of Org in load-path problem

[Bhavin Gandhi]

On Fri, 3 Feb 2023 at 16:35, Ihor Radchenko <yantar92@posteo.net> wrote:
>
> Max Nikulin <manikulin@gmail.com> writes:
>
> > On 27/12/2022 16:47, Ihor Radchenko wrote:
> >> Can you then try to test using Emacs 28?
> >> The main question if whether this has been fixed in newer Emacs releases
> >> or it is also something to do with OS environment.
> >
> > I see quite the same issue with Emacs-28.2 in Debian testing. Compile
> > buffer displays a bit more warnings and usual `org-assert-version'
> > warnings and errors are present as well. It might have another level of
> > complexity due to .eln files. I am unsure what happens with calls to
> > undefined macros.
>
> I asked people around to test using Debian, and we do have a
> confirmation that Debian + Emacs 27 and Debian + Emacs 28 do trigger the
> error.
>
> I also installed Debian 11.6.0 on virtual machine, and I was also able to
> trigger the error, following the provided steps, using the Emacs 27
> installed via apt-get.
>
> The problem seems to be real and appears to be some combination of
> Debian/Ubuntu + Emacs.

I'm able to reproduce this on Fedora 37, Emacs 28.2.

In Fedora we have:
- /usr/share/emacs/site-lisp/site-start.el (has code to load .el and .elc
from /usr/share/emacs/site-lisp/site-start.d).
- /usr/share/emacs/site-lisp/default.el (sets smime-CA-directory).

There are other patches but most of them are backports and seem to be
unrelated to package.el:
https://src.fedoraproject.org/rpms/emacs/tree/rawhide

Steps using with-emacs.sh (https://github.com/alphapapa/with-emacs.sh):

1. mkdir /tmp/emacs-59882
2. with-emacs.sh -d /tmp/emacs-59882 -- -l org
3. M-x package-list-packages
4. Install Org 9.6.1 from ELPA
5. When the package is compiled you get this error at the end (it is
there multiple times)
   ox.el:82:1: Error: Symbol’s function definition is void: org-assert-version
6. Close Emacs and start it again with
   with-emacs.sh -d /tmp/emacs-59882 -- -l org
7. I get byte-code: Invalid function: org-assert-version

-- 
Bhavin Gandhi (bhavin192) | https://geeksocket.in

Re: bug#59882: Multiple versions of Org in load-path problem

[Ihor Radchenko]

Bhavin Gandhi <bhavin7392@gmail.com> writes:

> I'm able to reproduce this on Fedora 37, Emacs 28.2.
>
> In Fedora we have:
> - /usr/share/emacs/site-lisp/site-start.el (has code to load .el and .elc
> from /usr/share/emacs/site-lisp/site-start.d).
> - /usr/share/emacs/site-lisp/default.el (sets smime-CA-directory).

May I know what is the contents of
/usr/share/emacs/site-lisp/site-start.d ?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bug#59882: Multiple versions of Org in load-path problem

[Bhavin Gandhi]

On Wed, 22 Feb 2023 at 16:17, Ihor Radchenko <yantar92@posteo.net> wrote:
>
> Bhavin Gandhi <bhavin7392@gmail.com> writes:
>
> > I'm able to reproduce this on Fedora 37, Emacs 28.2.
> >
> > In Fedora we have:
> > - /usr/share/emacs/site-lisp/site-start.el (has code to load .el and .elc
> > from /usr/share/emacs/site-lisp/site-start.d).
> > - /usr/share/emacs/site-lisp/default.el (sets smime-CA-directory).
>
> May I know what is the contents of
> /usr/share/emacs/site-lisp/site-start.d ?

These are init files specific to other packages installed via OS package
manager. But those are not loaded, it seems in the above steps. I tried to
run (featurep 'clang-format) and it returned nil, while this returns t
in my normal Emacs instance.

$ ls -1 /usr/share/emacs/site-lisp/site-start.d/
autoconf-init.el
  https://src.fedoraproject.org/rpms/autoconf/blob/rawhide/f/autoconf-init.el
clang-format.el
clang-include-fixer.el
clang-rename.el
cmake-init.el
  https://src.fedoraproject.org/rpms/cmake/blob/rawhide/f/cmake-init.el
desktop-entry-mode-init.el
  https://src.fedoraproject.org/rpms/desktop-file-utils/blob/rawhide/f/desktop-entry-mode-init.el
gtypist-init.el
  https://src.fedoraproject.org/rpms/gtypist/blob/rawhide/f/gtypist.spec#_43-46
mercurial-site-start.el
  https://src.fedoraproject.org/rpms/mercurial/blob/rawhide/f/mercurial-site-start.el

None of these load org-mode.

--
Bhavin Gandhi (bhavin192) | https://geeksocket.in

Re: bug#59882: Multiple versions of Org in load-path problem

[Ihor Radchenko]

Bhavin Gandhi <bhavin7392@gmail.com> writes:

> These are init files specific to other packages installed via OS package
> manager. But those are not loaded, it seems in the above steps. I tried to
> run (featurep 'clang-format) and it returned nil, while this returns t
> in my normal Emacs instance.
> ...
> None of these load org-mode.

They just define auto-loading, but I see not how they would affect Org
loading.

Very confusing...

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: bug#59882: Multiple versions of Org in load-path problem

[Max Nikulin]

On 03/02/2023 18:02, Ihor Radchenko wrote:
> 
> The problem seems to be real and appears to be some combination of
> Debian/Ubuntu + Emacs.

It is not a Debian packaging issue. I have reproduced the "invalid 
function `org-assert-version'" bug using emacs built from emacs-28.2 git 
tag. It is important to execute "make install" and to launch emacs from 
from that directory.

I have no idea why behavior is different when emacs is launched from the 
build tree.

Environment: Debian-12 bookworm (frozen testing).

There is no problem when emacs is built by just "make" from debian 
sources with all patches applied. I admit that this experiment was not 
really pure since I did not use configure options from the 
debian/control file.

Re: bug#59882: Multiple versions of Org in load-path problem

[Max Nikulin]

On 18/12/2022 22:40, Stefan Monnier via Bug reports for GNU Emacs, the 
Swiss army knife of text editors wrote:
>> - "emacs -l org" to simulate a regular user session with some opened files.
>> - Update to Org-9.6 from ELPA.
>> - Close emacs and start it again "emacs -l org"
>>
>> Result:
>> byte-code: Invalid function: org-assert-version
> 
> Hmm... there's code in `package.el` intended to avoid that problem, but
> clearly it's failing here.  Could Someone™ investigate why that code
> doesn't do its job.
> 
> The code in question is in `package--reload-previously-loaded`

Since I managed to reproduce the issue with emacs-28.2 built from source 
tree, I tried current master (30.0.50) as well. I used the following 
trick to shadow org-assert-version:

sed -i -e 's/org-assert-version/org-assert-version-old/g' lisp/org/*.el

I am unable to reproduce the "Invalid function: org-assert-version" 
issue. Org compilation finishes without warnings.

I have noticed that neither `package--reload-previously-loaded` nor code 
loading already compiled package unload the loaddefs file. E.g. I added 
org-stale-function marked as autoload. It is possible to get help for 
this function, but attempts to call this function causes "Autoloading 
file" error. Ideally emacs should completely forget about such functions 
when newer elpa package is installed.

Re: bug#59882: Multiple versions of Org in load-path problem

[Stefan Monnier]

> I have noticed that neither `package--reload-previously-loaded` nor code
> loading already compiled package unload the loaddefs file. E.g. I added

The state of our unloading support is not good enough to use it blindly,
indeed: it's too easy/common to have packages where `unload` leaves you
with an Emacs that has serious problems (including being unusable).

So, instead we limit ourselves to force-reloading files which tends to
be much more harmless (tho in theory of course it's just as bad).
It also tends to be sufficient.


        Stefan

Re: bug#59882: Multiple versions of Org in load-path problem

[Tom Gillespie]

> The state of our unloading support is not good enough to use it blindly

I have managed to use unload-feature to reload a new version of org
after the builtin version has been previously loaded, but the dance you
have to do to avoid nasty errors (and segfaults if using native comp!)
is about 100 lines of arcane footwork, and if any one step is off you
will encounter one of the many hidden landmines.

unload-feature cannot be used blindly, and there are a very large
number of side effects along the path that are not controlled which
makes any process that tries to use unload-feature brittle and hard
to debug.

https://github.com/tgbugs/orgstrap/blob/f35bccde556b0f82515e79ee69f4379469276356/ow.el#L1010-L1064
https://github.com/tgbugs/orgstrap/blob/f35bccde556b0f82515e79ee69f4379469276356/ow.el#L1073-L1177