Policy proposal: Do not move existing functions/macros except in major version increments

Policy proposal: Do not move existing functions/macros except in major version increments

[Adam Porter]

The relatively recent moving of org-get-outline-path to org-refile.el
has caused breakage in Org itself in several places, e.g.

https://lists.gnu.org/archive/html/emacs-orgmode/2020-04/msg00260.html
https://lists.gnu.org/archive/html/emacs-orgmode/2020-04/msg00259.html
https://lists.gnu.org/archive/html/emacs-orgmode/2020-04/msg00261.html

Thankfully, Kyle has proposed a patch to revert that change.  I hope it
is merged.

If it is not, when a new Org version is released with those changes
(actually, sooner, because some users run the master branch), it will
cause further breakage in out-of-tree packages and code in user
configurations.

I think changes like this should not be made without very careful
consideration of the wider implications.  These kinds of changes create
a not-insubstantial burden on maintainers of Org-related packages to
keep up with churn and maintain compatibility with multiple Org versions
(which are used in the wild for years--I know of users still using Org
8, as well as Org 9 versions that are included with older Emacs versions
(e.g. Emacs 26.3 is still stuck in Debian unstable, not migrating to
testing, stable, or backports)).

For my own packsges, I would expect to get multiple bug reports for
several of my packages, which means that for each one, I then have to
deal with the report, make a fix, test it, log it, push it, close the
bug report...and for all that, nothing is gained.  It adds up, and it's
frustrating and demoralizing.

Of course, I am not opposed, in principle, to refactoring and
reorganization of this sort.  Org is a huge project, and it certainly
could benefit from these kinds of changes, in general.

So, I propose that changes like these should not be made except in major
version increments, e.g. this change should have been delayed until the
release of Org 10.0.  It would be helpful for users and package authors
if they knew that changes like these would not be made until the next
major version increment.

If this is agreeable to the Org maintainers, I'd ask that it be
documented in the project and announced in the NEWS file.

Thanks,
Adam

Re: Policy proposal: Do not move existing functions/macros except in major version increments

[Nicolas Goaziou]

Hello,

Adam Porter <adam@alphapapa.net> writes:

> The relatively recent moving of org-get-outline-path to org-refile.el
> has caused breakage in Org itself in several places, e.g.

[...]

> Thankfully, Kyle has proposed a patch to revert that change.  I hope it
> is merged.
>
> If it is not, when a new Org version is released with those changes

What makes you think a new Org would be released in this situation,
without fixing it first?

> I think changes like this should not be made without very careful
> consideration of the wider implications.  These kinds of changes create
> a not-insubstantial burden on maintainers of Org-related packages to
> keep up with churn and maintain compatibility with multiple Org versions
> (which are used in the wild for years--I know of users still using Org
> 8, as well as Org 9 versions that are included with older Emacs versions
> (e.g. Emacs 26.3 is still stuck in Debian unstable, not migrating to
> testing, stable, or backports)).

[...]

> So, I propose that changes like these should not be made except in major
> version increments, e.g. this change should have been delayed until the
> release of Org 10.0.  It would be helpful for users and package authors
> if they knew that changes like these would not be made until the next
> major version increment.

FWIW, I think this is too strong a requirement.

This breakage is unfortunate, and I can hear the consequences it has on
the Org ecosystem, but, hey, it happens. Note that moving part of Org
elsewhere usually introduces less friction. This is a relatively
exceptional situation.

Master is an unstable branch, relatively open to experimentation.
Moreover, that experimentation happens before deciding if the next
release should be 10.0 or 9.4, so it wouldn't help users or package
authors.

It doesn't mean we cannot do better here. For example, I think we could
improve the way Org loads its libraries. Ideally, external libraries
should only (require 'org), and optionally, (require 'ox-*) or (require
'ol-*). Thus, changes like the one discussed here would be
implementation details. For example, "ox-hugo.el" requires directly
"ob-core.el", and now "org-refile.el", which is, IMO, a path to
troubles. It should only require "org.el".

The current situation is tricky though: "org.el" requires some libraries
(e.g., "org-key.el", "org-table.el", ...), and some libraries require
"org.el" (e.g., "org-capture.el", "org-element.el", ...). I expect
"org.el" to be the entry point for the Org package, so loading should
happen in one-way only.

This would not solve everything, but it would certainly make things
smoother for external libraries.

WDYT?

Regards,

-- 
Nicolas Goaziou

Re: Policy proposal: Do not move existing functions/macros except in major version increments

[Adam Porter]

Hi Nicolas,

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Hello,
>
> Adam Porter <adam@alphapapa.net> writes:
>
>> The relatively recent moving of org-get-outline-path to org-refile.el
>> has caused breakage in Org itself in several places, e.g.
>
> [...]
>
>> Thankfully, Kyle has proposed a patch to revert that change.  I hope
>> it is merged.
>>
>> If it is not, when a new Org version is released with those changes
>
> What makes you think a new Org would be released in this situation,
> without fixing it first?

I don't know what will happen.  I don't know whether the Org maintainers
would consider the problem serious enough to avert (as you said later,
"it happens").  That's why I pointed out what the consequences would be
if the patch isn't merged, to encourage its merging.

>> I think changes like this should not be made without very careful
>> consideration of the wider implications.  These kinds of changes
>> create a not-insubstantial burden on maintainers of Org-related
>> packages to keep up with churn and maintain compatibility with
>> multiple Org versions (which are used in the wild for years--I know
>> of users still using Org 8, as well as Org 9 versions that are
>> included with older Emacs versions (e.g. Emacs 26.3 is still stuck in
>> Debian unstable, not migrating to testing, stable, or backports)).
>
> [...]
>
>> So, I propose that changes like these should not be made except in
>> major version increments, e.g. this change should have been delayed
>> until the release of Org 10.0.  It would be helpful for users and
>> package authors if they knew that changes like these would not be
>> made until the next major version increment.
>
> FWIW, I think this is too strong a requirement.
>
> This breakage is unfortunate, and I can hear the consequences it has
> on the Org ecosystem, but, hey, it happens. Note that moving part of
> Org elsewhere usually introduces less friction. This is a relatively
> exceptional situation.

Of course, I am biased here, but I feel like it's not quite as
exceptional as it ought to be.  The more Org-related packages I make and
maintain, the more version-specific workarounds I have to add due to
changed function names, signatures, etc.  These are sometimes necessary,
of course, but I think they should be made more carefully and
deliberately.

Of course, Org doesn't make any promises to third-party packages.  But
that is the issue, isn't it?  I'm saying that I think it should start
taking this issue a little more seriously.  :)

> Master is an unstable branch, relatively open to experimentation.
> Moreover, that experimentation happens before deciding if the next
> release should be 10.0 or 9.4, so it wouldn't help users or package
> authors.

That is a matter of policy, which is what I'm proposing.  When such a
change is desired (symbol name, function signature, etc), it should be
targeted at the next major version increment.  If the project as a whole
is not ready to make that increment, that change should be delayed until
then--it can be developed in a branch and merged when preparing the
major release.  These kinds of changes could even be documented in
advance, e.g. in a ROADMAP or PLANS file, or whatever you want to call
it, which would be more explicit and referenceable than merely a mailing
list post.

> It doesn't mean we cannot do better here. For example, I think we
> could improve the way Org loads its libraries. Ideally, external
> libraries should only (require 'org), and optionally, (require 'ox-*)
> or (require 'ol-*). Thus, changes like the one discussed here would be
> implementation details. For example, "ox-hugo.el" requires directly
> "ob-core.el", and now "org-refile.el", which is, IMO, a path to
> troubles. It should only require "org.el".
>
> The current situation is tricky though: "org.el" requires some
> libraries (e.g., "org-key.el", "org-table.el", ...), and some
> libraries require "org.el" (e.g., "org-capture.el", "org-element.el",
> ...). I expect "org.el" to be the entry point for the Org package, so
> loading should happen in one-way only.
>
> This would not solve everything, but it would certainly make things
> smoother for external libraries.
>
> WDYT?

That is a good idea, and one that needs addressing.  I'd be glad to give
some feedback on it, but in a separate thread, please, because it seems
like a different matter from the issue I'm raising and the proposal I'm
making.

Thanks,
Adam

Re: Policy proposal: Do not move existing functions/macros except in major version increments

[Nicolas Goaziou]

Hello,

Adam Porter <adam@alphapapa.net> writes:

> Of course, I am biased here, but I feel like it's not quite as
> exceptional as it ought to be.  The more Org-related packages I make and
> maintain, the more version-specific workarounds I have to add due to
> changed function names, signatures, etc.  These are sometimes necessary,
> of course, but I think they should be made more carefully and
> deliberately.
>
> Of course, Org doesn't make any promises to third-party packages.  But
> that is the issue, isn't it?  I'm saying that I think it should start
> taking this issue a little more seriously.  :)

[...]

> That is a matter of policy, which is what I'm proposing.  When such a
> change is desired (symbol name, function signature, etc), it should be
> targeted at the next major version increment.  If the project as a whole
> is not ready to make that increment, that change should be delayed until
> then--it can be developed in a branch and merged when preparing the
> major release.  These kinds of changes could even be documented in
> advance, e.g. in a ROADMAP or PLANS file, or whatever you want to call
> it, which would be more explicit and referenceable than merely a mailing
> list post.

I think it isn't a realistic policy, because it is not even close to how
Org development happens.

Unlike what you write, even with a smiley, I am taking compatibility
issues and breakages as seriously as I can. I do my best to not
introduce them in the first place, and, if I do, because I think that's
for the better, I do my best to make the transition easier. Yet,
breakage happens, and I am sincerely sorry about that. Now, fire me if
you want.

However, please remember that Org development happens in a best effort,
and in a somewhat opportunistic, fashion. Most internal changes, and
therefore breakages, stem off a bug report or an enhancement request,
not from a big ROADMAP file in the sky. Call it amateurism, but I think
it is in its most noble meaning.

Maybe in the not-so-distant future, professional-minded folks will lead
the project, defining road maps, strict rules for features integration,
proper issue tracking, and whatnot. And maybe that will be a good thing.
Any help is welcome. Until then, we have two branches:

1. master, where breakage can happen, which implies at least minor
   version numbers bumps.

2. maint, where breakage is to be avoided at all cost, which implies
   only bugfix version number bumps.

Close to a release, we also happen to have "feature-freeze" periods. In
that situation, development happens in a temporary third branch: next.

You may consider it to be an insufficient policy, but AFAICT, this is
all we have at the moment, and, possibly, all we can afford, too.

Of course, this is only my opinion. I wouldn't dare talking for other
developers. AFAIC, I already do all I can, with the time I have;
I cannot follow your policy.

> That is a good idea, and one that needs addressing.  I'd be glad to give
> some feedback on it, but in a separate thread, please, because it seems
> like a different matter from the issue I'm raising and the proposal I'm
> making.

I thought most third-party breakages came from misuse of internal
functions in Org. But if you say it is orthogonal, and that I am
off-topic, so be it.

Regards,

-- 
Nicolas Goaziou

Re: Policy proposal: Do not move existing functions/macros except in major version increments

[Bastien]

Hi Adam,

Adam Porter <adam@alphapapa.net> writes:

> The relatively recent moving of org-get-outline-path to org-refile.el
> has caused breakage in Org itself in several places, e.g.
>
> https://lists.gnu.org/archive/html/emacs-orgmode/2020-04/msg00260.html
> https://lists.gnu.org/archive/html/emacs-orgmode/2020-04/msg00259.html
> https://lists.gnu.org/archive/html/emacs-orgmode/2020-04/msg00261.html

ahem, my bad.  I made this bold (and wrong) move, and I broke code out
of org-mode.

I understand your proposal, and it's always good to be reminded that
many people depend on Org's code out there.  It is not easy to spend
time working on Org *and* tracking all these interesting extensions.

I agree with Nicolas that we should not put more constraints on the
shoulders of Org current developers, especially because their time is 
limited - and obviously not enough to cope with every request.

That said, we can make it easier for third-party developers to know
what changes will be released in the future.

See the "Upcoming changes" in https://updates.orgmode.org

You can subscribe to this RSS feed:
https://updates.orgmode.org/feed/changes

Or check the data directly:
https://updates.orgmode.org/data/changes

To announce the change you see, I just used this email header:

  X-Woof-Change: 9092c289 9.4

That is the commit number where the change happens and the version
in which the change will be released.

The list of upcoming changes is emptied when a new major released is
done, because the changes are then advertized in this file, as usual:
https://orgmode.org/Changes.html

I think that's a tiny distributed effort for developers and a easy
way to track changes for third-party developers.

Best,

-- 
 Bastien

Re: Policy proposal: Do not move existing functions/macros except in major version increments

[Adam Porter]

Hi Bastien,

On 6/1/20, Bastien <bzg@gnu.org> wrote:
> Hi Adam,
>
> Adam Porter <adam@alphapapa.net> writes:
>
>> The relatively recent moving of org-get-outline-path to org-refile.el
>> has caused breakage in Org itself in several places, e.g.
>>
>> https://lists.gnu.org/archive/html/emacs-orgmode/2020-04/msg00260.html
>> https://lists.gnu.org/archive/html/emacs-orgmode/2020-04/msg00259.html
>> https://lists.gnu.org/archive/html/emacs-orgmode/2020-04/msg00261.html
>
> ahem, my bad.  I made this bold (and wrong) move, and I broke code out
> of org-mode.
>
> I understand your proposal, and it's always good to be reminded that
> many people depend on Org's code out there.  It is not easy to spend
> time working on Org *and* tracking all these interesting extensions.

Of course, no one could be expected to keep track of all those things.
Such is the nature of writing software that runs in a Lisp image,
where anyone can use anything, and does.

> I agree with Nicolas that we should not put more constraints on the
> shoulders of Org current developers, especially because their time is
> limited - and obviously not enough to cope with every request.

I mostly agree with you.  My request is simply that, when a change has
the potential to break third-party packages, and it's a change, such
as this one, that mostly moves code around for organizational
purposes, that it be delayed until the next major version.  My goal
for such a policy would be to reduce the frequency of such changes
that third-party package authors have to write compatibility code for.
The (if (version< org-version ...)) workarounds become confusing for
authors and users, and somewhat of a maintenance burden.

> That said, we can make it easier for third-party developers to know
> what changes will be released in the future.
>
> See the "Upcoming changes" in https://updates.orgmode.org
>
> You can subscribe to this RSS feed:
> https://updates.orgmode.org/feed/changes
>
> Or check the data directly:
> https://updates.orgmode.org/data/changes
>
> To announce the change you see, I just used this email header:
>
>   X-Woof-Change: 9092c289 9.4
>
> That is the commit number where the change happens and the version
> in which the change will be released.
>
> The list of upcoming changes is emptied when a new major released is
> done, because the changes are then advertized in this file, as usual:
> https://orgmode.org/Changes.html
>
> I think that's a tiny distributed effort for developers and a easy
> way to track changes for third-party developers.

That's a very clever way to do it!  Thanks for your work on this.  If
it's feasible, you might also consider allowing committers to put such
a header in the git commit log and parsing it out of that, which could
make it even easier.

Thanks,
Adam

Re: Policy proposal: Do not move existing functions/macros except in major version increments

[Bastien]

Hi Adam,

Adam Porter <adam@alphapapa.net> writes:

> I mostly agree with you.  My request is simply that, when a change
> has the potential to break third-party packages, and it's a change,
> such as this one, that mostly moves code around for organizational
> purposes, that it be delayed until the next major version.

IIRC this was the case for the change that triggered your first email,
it was in master, no?

> My goal for such a policy would be to reduce the frequency of such
> changes that third-party package authors have to write compatibility
> code for.  The (if (version< org-version ...)) workarounds become
> confusing for authors and users, and somewhat of a maintenance
> burden.

I understand the burden.  We don't really follow the conventions of
semantic versioning.  We use "x.y.z" versions with z being for bugfix
releases and both y and x for... the rest.

Because we are slow at releasing so-called minor releases, they end up
containing some changes that users and developers should be aware of.
Hence the need to let them know about such changes.  I don't think
using a policy will help here, because definitions of "breaking" may
vary: is a change in a function's signature always a breaking change?
For every function?  Or just for core functions?  If so, which ones?
(See the many discussions in the npm universe about what to consider
a breaking change and worth a major release.)

In a word: even though stricter rules could somehow help, I do think
the main issue is about *communication*, not conventions.

> That's a very clever way to do it!  Thanks for your work on this.  

Thanks, I hope it will be useful.

> If
> it's feasible, you might also consider allowing committers to put such
> a header in the git commit log and parsing it out of that, which could
> make it even easier.

The idea is that updates that are important enough to be listed on
updates.orgmode.org are the ones that *should* be somehow announced on
the mailing list.  If we let updates.orgmode.org monitor commits, then
people who are only reading the list will miss updates (which we
don't want, I think.)

That said, it would be nice to have a function that checks the data
from https://updates.orgmode.org/data/updates and warn the user when
there is a new upcoming change.

-- 
 Bastien