missing autoloads / (void-variable org-version)

missing autoloads / (void-variable org-version)

[Achim Gratz]

Since my Makefile branch was merged I've been getting flak for breaking
certain setups.  Now, the change didn't actually break them, but I did
make (perhaps foolishly) a deliberate decision to make that particular
breakage fatal rather than silent.  It would be very easy to continue to
paper over the breakage and pretend things are working when they're not.
What irritates me greatly is that quite a few of the people that insist
on this "solution" are the ones who would have to deal with the
Heisenbugs thus introduced.  So let me explain once more what is broken:


Consider a core emacs library "test", consisting of the main "test.el"
and auxiliary functions in "test-aux.el".  The main entry points are in
"test.el" and thus you'd "(require 'test)" in your .emacs to activate
the package.  The auxiliary functions are not needed in "test.el", so to
make them available to the user or whatever function that needs to call
them, these entry points are autoloaded (let's say the only function
there is "test-aux").  When this package gets installed into emacs, the
autoload forms will be extracted into "loaddefs.el" in the main lisp
directory.  You can then call "(test-aux)" without having to explicitly
load "test-aux.el", since the autoload form has registered that
association and will load "test-aux.el" automatically when encountering
the first use of the function test-aux.

Now the development version of this same package gets installed locally.
The load-path is altered so that emacs finds the development version
first.  Thus "(require 'test)" will load the development version.  If
you now try to "(test-aux)", what happens?  Autoload still has all the
associations from "loaddefs.el" and thus tries to load "test-aux.el",
which will end up finding the file in the development version. So all is
well, right?

Development progresses and as "test.el" grows larger some functions are
moved to "test-extra.el".  Also, "test-aux.el" is split into
"test-aux1.el", while the function test-aux (yes, the same name as the
old, but with different arguments) is moved to "test-aux2.el".  The
functions in "test-extra.el" become unavailable since autoload doesn't
know about them — unless you explicitely load "test-aux.el" of course.
Also, the calls to "(test-aux)" flip-flop between calling the old and
new test-aux: if that call is made before a load of "test-aux2.el",
emacs will look for it in "test-aux.el" and only find that file in its
own install directory.  If you happened to have loaded "test-aux2.el"
(maybe due to a "(require 'test-aux2)" in "test-extra.el"), then the new
definition of "(test-aux)" will be used.  So you will have irritated
users and developers alike and bugs that don't reproduce (or only with
certain versions of emacs that happen to have specific versions of
library test integrated).

The real solution of course is to re-generate the autoloads file for the
development version and explicitely load it, say "(require
'test-install)".

That still leaves an open flank if you delete both a function and it's
associated source file from the development version: if someone then
uses "(test-deleted-function)", the autoload definition in Emacs will
happily find the stale source it has installed since it is not shadowed
anymore by the same file in the development version.  I've no idea yet
if that is fixable without touching the actual installation and how
difficult it would be, but that's a minor worry at this point.


If you've made it here, please weigh in on whether you want to have
things correctly fixed or simply papered over.  Either one is fine with
me...


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf rackAttack V1.04R1:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: missing autoloads / (void-variable org-version)

[Sebastien Vauban]

Hi Achim,

Achim Gratz wrote:
> Since my Makefile branch was merged I've been getting flak for breaking
> certain setups.  Now, the change didn't actually break them, but I did
> make (perhaps foolishly) a deliberate decision to make that particular
> breakage fatal rather than silent.  It would be very easy to continue to
> paper over the breakage and pretend things are working when they're not.
> What irritates me greatly is that quite a few of the people that insist
> on this "solution" are the ones who would have to deal with the
> Heisenbugs thus introduced.  So let me explain once more what is broken:

Did not know that term... Thanks.

http://en.wikipedia.org/wiki/Heisenbug

> Consider a core emacs library "test", consisting of the main "test.el"
> and auxiliary functions in "test-aux.el".  The main entry points are in
> "test.el" and thus you'd "(require 'test)" in your .emacs to activate
> the package.  The auxiliary functions are not needed in "test.el", so to
> make them available to the user or whatever function that needs to call
> them, these entry points are autoloaded (let's say the only function
> there is "test-aux").  When this package gets installed into emacs, the
> autoload forms will be extracted into "loaddefs.el" in the main lisp
> directory.  You can then call "(test-aux)" without having to explicitly
> load "test-aux.el", since the autoload form has registered that
> association and will load "test-aux.el" automatically when encountering
> the first use of the function test-aux.
>
> Now the development version of this same package gets installed locally.
> The load-path is altered so that emacs finds the development version
> first.  Thus "(require 'test)" will load the development version.  If
> you now try to "(test-aux)", what happens?  Autoload still has all the
> associations from "loaddefs.el" and thus tries to load "test-aux.el",
> which will end up finding the file in the development version. So all is
> well, right?

Here, I would have thought that invoking the function `text-aux' would open
the "production" Lisp file (that is, the one in Emacs code base) as you use
Emacs loaddefs.el associations, no?

> Development progresses and as "test.el" grows larger some functions are
> moved to "test-extra.el".  Also, "test-aux.el" is split into
> "test-aux1.el", while the function test-aux (yes, the same name as the
> old, but with different arguments) is moved to "test-aux2.el".  The
> functions in "test-extra.el" become unavailable since autoload doesn't
> know about them — unless you explicitely load "test-aux.el" of course.
> Also, the calls to "(test-aux)" flip-flop between calling the old and
> new test-aux: if that call is made before a load of "test-aux2.el",
> emacs will look for it in "test-aux.el" and only find that file in its
> own install directory.  If you happened to have loaded "test-aux2.el"
> (maybe due to a "(require 'test-aux2)" in "test-extra.el"), then the new
> definition of "(test-aux)" will be used.  So you will have irritated
> users and developers alike and bugs that don't reproduce (or only with
> certain versions of emacs that happen to have specific versions of
> library test integrated).
>
> The real solution of course is to re-generate the autoloads file for the
> development version and explicitely load it, say "(require
> 'test-install)".
>
> That still leaves an open flank if you delete both a function and it's
> associated source file from the development version: if someone then
> uses "(test-deleted-function)", the autoload definition in Emacs will
> happily find the stale source it has installed since it is not shadowed
> anymore by the same file in the development version.  I've no idea yet
> if that is fixable without touching the actual installation and how
> difficult it would be, but that's a minor worry at this point.
>
>
> If you've made it here, please weigh in on whether you want to have
> things correctly fixed or simply papered over.  Either one is fine with
> me...

I've experienced enough couple of hours lost in the past, searching for
problems due to mixed versions loaded, to know what the correct answer is.

Often, such problems are even not detectable in such a case:

1. some function loads an Org function (without you being aware), hence the
   Emacs Org version.

2. you update your load-path to point to Org latest dev version (.. but not at
   the very top of your Emacs init file).

3. you load other Org components.

4. you have problems.

5. you don't notice them by inspecting `load-path' variable: your Org dev
   directory is in front of Emacs Org directory, letting you think everything
   is correct!

Best regards,
  Seb

-- 
Sebastien Vauban

Re: missing autoloads / (void-variable org-version)

[Detlef Steuer]

> Since my Makefile branch was merged I've been getting flak for breaking
> certain setups.  Now, the change didn't actually break them, but I did
> make (perhaps foolishly) a deliberate decision to make that particular
> breakage fatal rather than silent.  It would be very easy to continue to
> paper over the breakage and pretend things are working when they're not.
> What irritates me greatly is that quite a few of the people that insist
> on this "solution" are the ones who would have to deal with the
> Heisenbugs thus introduced.  So let me explain once more what is broken:
> 

Hmm. I was biten by the makefile change but hopefully my report didn't
count as "flak". 

I have absolutely no problem with any reasonable change in the makefile.
But your description of the situation before and after the makefile
change is not entirely correct.  
Before: May be I had a broken setup and didn't know. Nevertheless it
worked. 
After: May be conceptually your new makefile improved the build
process. But now my setup was broken. ;-)

And here comes the moment when you just wanted to do some routine work,
namely publish some notes for the upcoming lecture, normally a one
minute job, well set-up and tested. And all you get is some error. Time
is ticking away, next org-mode scheduled APPT in 5 min. We all know
this.  

My guess: It was too easy to miss that action was required on a user's
side. May be a hint that make targets changed and that under some
circumstances make update could fail and a hint with some exclamation
marks that batch exporting now requires --load org-install.el and not
only --load org.el would have reduced mail traffic a lot. If there was
such a note at least I missed it. 

Definitely keep up your work and improve the build process! I'm happy
to change my scripts if I know I need to and better still if I
understand why I should!  (Thx for your explanation!)

Detlef

Re: missing autoloads / (void-variable org-version)

[Martyn Jago]

Hi Achim

Achim Gratz <Stromeko@nexgo.de> writes:

> Since my Makefile branch was merged I've been getting flak for breaking
> certain setups.  Now, the change didn't actually break them, but I did
> make (perhaps foolishly) a deliberate decision to make that particular
> breakage fatal rather than silent.  It would be very easy to continue to
> paper over the breakage and pretend things are working when they're not.
> What irritates me greatly is that quite a few of the people that insist
> on this "solution" are the ones who would have to deal with the
> Heisenbugs thus introduced.  So let me explain once more what is broken:
>

[...]

Thanks for this post - it has certainly helped me see the `rock' and the
`hard place'. 

As an aside, I haven't used MS windows for some two years (using OS X
and Linux), but this will change shortly as my next project will require
a working Windows system with MSys. I could certainly document the setup
of Org-mode including functioning tests etc. on a line-by-line basis if
it will help.

Best, Martyn

>
> Regards,
> Achim.

Re: missing autoloads / (void-variable org-version)

[Bastien]

Hi all,

thanks Achim for this lengthy post, it's definitely worth reading for
anyone who got confused by autoloading stuff once in his life (I did,
who didn't?)

Now, I should have warned the mailing list about possible breakage in
people's configuration, I apologize for this.

There is a fix that makes the new Makefile compatible with everyone's
configuration *without* requiring to add the version number directly in
the org.el file.  I'm discussing it with Achim and will commit it as
soon as possible.

"Breaking" a broken configuration is definitely not a bug, especially
when this comes with a conceptually clean approach -- which is precisely
the point of Achim's work.  

Still, a missing org-version should not prevent the whole Org spaceship
to take off...  so, let's bare with us a little bit.

Thanks!

-- 
 Bastien

Re: missing autoloads / (void-variable org-version)

[Achim Gratz]

Sebastien Vauban writes:
> Here, I would have thought that invoking the function `text-aux' would open
> the "production" Lisp file (that is, the one in Emacs code base) as you use
> Emacs loaddefs.el associations, no?

It could be set up that way, but isn't.  You wouldn't be able to move an
installed emacs tree if that were the case, and it could quite likely
break in other mysterious ways.

> I've experienced enough couple of hours lost in the past, searching for
> problems due to mixed versions loaded, to know what the correct answer is.

...so, I count that as a "yes, fix it please"?


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Terratec KOMPLEXER:
http://Synth.Stromeko.net/Downloads.html#KomplexerWaves

Re: missing autoloads / (void-variable org-version)

[Achim Gratz]

Martyn Jago writes:
> As an aside, I haven't used MS windows for some two years (using OS X
> and Linux), but this will change shortly as my next project will require
> a working Windows system with MSys. I could certainly document the setup
> of Org-mode including functioning tests etc. on a line-by-line basis if
> it will help.

This simply works with minimal setup which I already posted here.  If
you want to go with MSys, consider using the MSys-Git netinstall
version, since it comes with all the development tools you need.  The
standalone Git for Windows doesn't have make and some other stuff you
need for the documentation.  If you're serious about working on Windows
you should probably just install Cygwin.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Samples for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra

Re: missing autoloads / (void-variable org-version)

[Achim Gratz]

Bastien writes:
> There is a fix that makes the new Makefile compatible with everyone's
> configuration *without* requiring to add the version number directly in
> the org.el file.  I'm discussing it with Achim and will commit it as
> soon as possible.

Towards that end, I can test Emacs23 and Emacs24 on Linux.  Anyone still
using XEmacs and willing to check that everything keeps working there?
It's pretty difficult to even check the documentation for it when you
don't have it installed... and there are certainly differences in some
areas we'd need to poke around in that even I know.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: missing autoloads / (void-variable org-version)

[Sebastien Vauban]

Hi Achim,

Achim Gratz wrote:
> Sebastien Vauban writes:
>> Here, I would have thought that invoking the function `text-aux' would open
>> the "production" Lisp file (that is, the one in Emacs code base) as you use
>> Emacs loaddefs.el associations, no?
>
> It could be set up that way, but isn't.  You wouldn't be able to move an
> installed emacs tree if that were the case, and it could quite likely
> break in other mysterious ways.

I think you did not understand what I meant, or the opposite. Whatever, this
presentation is not the most important thing. What you're trying to achieve
*is*. First things first...

To come back to what I understood from your text:

1. you have `test.el' in, let's say,
   `c:/Program Files/Emacs-24.0/lisp/test-mode'

2. the autoload definition for the function `test-aux' is in
   `c:/Program Files/Emacs-24.0/lisp'

3. you have latest Test package (dev version) in, let's say,
   `c:/home/sva/src/test-mode', with 2 files in there:
   + `test.el'
   + `test-aux.el'

4. you add `c:/home/sva/src/test-mode' *in front of* your `load-path' list

5. you, then, require `test.el' in your Emacs startup file; it loads the dev
   version (that is, `c:/home/sva/src/test-mode/test.el')

6. you, then, invoke the command `test-aux'. As the autoload says that, for
   getting the definition of that function, you need to load the file
   `test-aux.el', Emacs searches for that file, and finds the first occurrence
   in version `c:/home/sva/src/test-mode/test-aux.el'

Here, due to a too quick reflexion, I thought that Emacs would load the base
version in `c:/Program Files/Emacs-24.0/lisp/test-mode/test-aux.el'. But
that's _not_ true, as:

- only file names are given in `loaddefs.el', not full paths
- the dev version directory has been placed first in `load-path'.

So, I was wrong about my comment... Sorry for the noise.

However, a problem that has bitten me twice already is that, in the above
case, I would have this in my Emacs startup file:

#+begin_src emacs-lisp
(require 'xyz-package) ;; this one requires `test.el' but you don't see/know it
...
...
...
(add-to-list 'load-path "c:/home/sva/src/test-mode") ;; before use of Test package
(require 'test) ;; no-op in this case
(test-aux)
#+end_src

You end up in mixed versions of the different functions. Typically the case
where Org version looks like:

    Org-mode version 7.7 (release_7.8.09-400-g4a0e5b)

But, when having a look at the value of `load-path', everything seems correct:
you dev version is in front of your Emacs core version.

You think that everything related to the Test package would have been loaded
correctly. Not the case, "obviously"...

Are we on the same wavelength?

>> I've experienced enough couple of hours lost in the past, searching for
>> problems due to mixed versions loaded, to know what the correct answer is.
>
> ...so, I count that as a "yes, fix it please"?

Yep!  I consider important that you fix things which you consider as flaws or
broken pieces.

Best regards,
  Seb

-- 
Sebastien Vauban

Re: missing autoloads / (void-variable org-version)

[Bastien]

Achim Gratz <Stromeko@nexgo.de> writes:

> Bastien writes:
>> There is a fix that makes the new Makefile compatible with everyone's
>> configuration *without* requiring to add the version number directly in
>> the org.el file.  I'm discussing it with Achim and will commit it as
>> soon as possible.
>
> Towards that end, I can test Emacs23 and Emacs24 on Linux.  Anyone still
> using XEmacs and willing to check that everything keeps working there?

Michael Sperber is our XEmacs fairy around, hopefully he we be able to 
test the patch.

-- 
 Bastien

Re: missing autoloads / (void-variable org-version)

[Bastien]

Hi Sébastien,

"Sebastien Vauban"
<wxhgmqzgwmuf-geNee64TY+gS+FvcfC7Uqw@public.gmane.org> writes:

> You think that everything related to the Test package would have been loaded
> correctly. Not the case, "obviously"...

More precisely: it has been _loaded_ correctly (i.e. it is in the load
path), but the functions have been _autoloaded_ from a previous package.
Looking for the package from the autoloaded function will point at the
first matching package in load-path, which might not be the one from
which functions have been autoloaded... 

Yep, it took me a while to understand it, so this email is a tribute to
Achim's patience :)

> Are we on the same wavelength?

I think we are.

-- 
 Bastien

Re: missing autoloads / (void-variable org-version)

[Bastien]

Hi Sébastien,

"Sebastien Vauban"
<wxhgmqzgwmuf-geNee64TY+gS+FvcfC7Uqw@public.gmane.org> writes:

> You think that everything related to the Test package would have been loaded
> correctly. Not the case, "obviously"...

More precisely: it has been _loaded_ correctly (i.e. it is in the load
path), but the functions have been _autoloaded_ from a previous package.
Looking for the package from the autoloaded function will point at the
first matching package in load-path, which might not be the one from
which functions have been autoloaded... 

Yep, it took me a while to understand it, so this email is a tribute to
Achim's patience :)

> Are we on the same wavelength?

I think we are.

-- 
 Bastien

Re: missing autoloads / (void-variable org-version)

[Michael Sperber]

Bastien <bzg@gnu.org> writes:

> Achim Gratz <Stromeko@nexgo.de> writes:
>
>> Bastien writes:
>>> There is a fix that makes the new Makefile compatible with everyone's
>>> configuration *without* requiring to add the version number directly in
>>> the org.el file.  I'm discussing it with Achim and will commit it as
>>> soon as possible.
>>
>> Towards that end, I can test Emacs23 and Emacs24 on Linux.  Anyone still
>> using XEmacs and willing to check that everything keeps working there?
>
> Michael Sperber is our XEmacs fairy around, hopefully he we be able to 
> test the patch.

Yes.  I've also had to apply a few fixes to make the current XEmacs
work.  I intend to send a new batch of patches in two weeks or so, once
everything has settled down on my end.  (Until then, I'm quite swamped.
If I don't answer e-mail in that timeframe, please don't assume I'm not
interested.)

-- 
Cheers =8-} Mike
Friede, Völkerverständigung und überhaupt blabla

Re: missing autoloads / (void-variable org-version)

[Achim Gratz]

Michael Sperber writes:
> Yes.  I've also had to apply a few fixes to make the current XEmacs
> work.  I intend to send a new batch of patches in two weeks or so, once
> everything has settled down on my end.  (Until then, I'm quite swamped.
> If I don't answer e-mail in that timeframe, please don't assume I'm not
> interested.)

I have found a few things to adapt, it does at least compile (even if
XEmacs complains a lot).  Testing is unfortunately not possible, unless
somebody has ert ported to XEmacs (prominently, XEmacs doesn't have
button.el).


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf rackAttack V1.04R1:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada