Re: Sync up the org in emacs master to org maint branch?

[Edward John Steere <edward.steere@gmail.com>]

> It's not like packages communicate with Emacs over a well
> defined RESTful interface. In other words: CEDET and Gnus are not
> loosely coupled, quite the opposite: they are extremely dependend on
> many obscure Emacs internals (not sure about Org).

Shouldn't we be trying to avoid this situation?  It's sure to come back
and bite us in the future.  If we continue to develop a package
(wherever it ends up being developed) which is so tightly coupled that
any kind of re factoring in core becomes a nightmare, because we have to
consider the umpteen ways in which it'll break the package, then surely
that's a bad methodology to continue?  (I'm not suggesting that we
rewrite it to make it more loosely coupled, just that it seems like a
bad idea to continue allowing this going forward.)

> As a consequence, we
> and also the Gnus guys decided to not do separate releases anymore.  We
> used to provide CEDET for different Emacs versions, and it was a *huge*
> amount of work. I had a buildbot running with 7 or 8 Emacs versions to
> run the test suite, and things broke pretty regularly.
> Now you might say: fine, only release a package for current master. But
> let's say we put CEDET into ELPA. Emacs 26 gets released, and work on
> Emacs 27 starts. Now there are changes happening in Emacs 27 that
> require changes in CEDET, which make it incompatible with Emacs 26. So
> you have to create two packages: one for 26, one for 27? Not only is
> this confusing, but it most definitely increases my workload.

I feel like this problem isn't intractable.  We can mark some state of
CEDET as being stable under the various versions of Emacs (because it
was at the time) and then only support the current release for the
latest package.  This would most likely require changes to package to
ensure that you get an appropriate version of the package when you
download it.

Consider the problem which our users currently face.  The built in CEDET
is miles behind and missing very important bug fixes and features.  The
upstream CEDET can be a real pain to setup, but it has the latest
features.  This is not a good place to be.  If we merge CEDET in and
only release it with the realeses of Emacs then we are heading for a
state where you only get the new features and bug fixes every time Emacs
is realesed, i.e. our users might have to wait up to two years to have
something fixed.  This is also a bad place to be.

>> We can arrange things so that a Git clone of Emacs includes pulling the
>> submodules (or trees, or ELPA.git, or what not) that are considered part of
>> "main Emacs development", including some of those batteries. I see this all as
>> a process issue, and that "living in one Git repository" has just been an
>> implementation strategy to satisfy that process.
>
> Obviously, I'm very skeptical towards such an approach. Our tooling
> around Emacs development is already very intricate and only works
> because a few people work quietly behind the scenes to keep this all
> running. Introducing even more complexity through
> submodules/subtrees/whatever will do the opposite of what you want to
> achieve: it creates more work for those few people who manage the Emacs
> infrastructure. But I'd love to hear what for instance Glenn and Paul
> think about this.

I'm interested in exploring more with regards to how the subtree
approach would work.  In particular how it would impact the Makefiles
and build process.  I don't think that introducing features like this
necessarily increases the burden of maintaining our tooling.  If we get
it right it could reduce it.  For example we could establish some sort
of convention for building submodules/subtrees which allows us to
simplify the related Makefiles.

>> Why do the split at all? Core becomes smaller,
>
> First off, the Emacs repo isn't that big w.r.t. the number of
> files. Secondly, while there surely is an inverse correlation between
> repo size and maintainability, I would argue that as long as the Big
> Three are well maintained, they are not the problem. When did CEDET,
> Gnus or Org ever significantly delay an Emacs release?  When was an
> Emacs core developer ever forced to fix a critical thing in those
> packages he did not break?  These are the questions we should be
> asking. From watching this list over the past years, I don't get the
> feeling that the inclusion of these packages has been a significant
> burden, but I may be wrong.

I think that it's a worthwhile goal to make core smaller.  It may not be
a gigantic enterprise system with tens of thousands of source files,
like some of us are accustomed to working on, but I think that it
becomes easier to reason about the features and behaviour of core when
it's smaller.

>> updating packages within it is no longer a major issue, and (I hope)
>> it will be clearer when something is a core issue vs. a package issue.
>
> I find this whole core vs package argument strange. If you ship Emacs
> with Org, Gnus and CEDET, they are part of Emacs, so it's in the
> interest of all Emacs developers that they work well, whether they use
> them or not. The users won't care if they originate from a separate repo
> and are considered a "package".

I think that the distinction becomes clearer when you consider what
other parts of Emacs ought to be able to depend on.  If mode-x started
building dependencies on CEDET because the author discovered some useful
functions in CEDET.  Then mode-x would build a dependency which is
difficult to maintain because changes in CEDET might have unintended
effects on mode-x.  If the function really is useful, then I think
that we should instead consider extracting it from CEDET and placing it
into some core library.  As far as I can tell this has already happened
with numerous functions which originated from CEDET.

> So if Paul is determined to fix all
> occurences of "compatilibity" in the doc-strings, why would he only do
> that for core?  People won't care if it's a CEDET doc-string, they'd
> just say "Teh Emacs people cant spell!1!!". It's no big deal for him
> anyway if everything is in one repo. Likewise when Stefan does some
> refactoring, like renaming 'defmethod' to 'cl-defmethod'. Doing this for
> all files is a matter of seconds, then commit, done. If Gnus, CEDET and
> Org are separate, you have to create separate commits for them, with
> their own ChangeLogs. I would argue that it is almost always *less* work
> for all people involved if things get fixed right away from the right
> person, and putting our built-in packages in different repos will make
> this more difficult.

I agree with this point.  Definitely a downside to the separate packages
approach and one which we don't have a solution for so far.

> The question is *how* did they do this. You think they just copied it
> over and hoped for the best? Or maybe they did have one repo were
> everything was checked in, and where they carefully tested it, maybe
> even applied their own patches to Django which they couldn't or wouldn't
> get upstream?  I don't know! Since it is non-free, we cannot check,
> unfortunately.
>
> -David

I don't think that anyone is suggesting that we "copy them over and hope
for the best."  This would have to form part of the QA process.  As for
testing -- I would be surprised if CEDET in core has really received the
amount of testing it needs to declare that it's a stable release.

As to the matter of user testing in particular.  I'm fairly certain that
the majority of our users have opted to go through the pain of setting
up upstream for the reason that it includes the latest features.  I feel
that if we were to distribute it as a package then that too would
receive more user testing than core CEDET.  People tend to lose interest
in projects which stagnate.