Sync up the org in emacs master to org maint branch?

Sync up the org in emacs master to org maint branch?

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 1030 bytes --]

Hi all,

I am aware that in emacs 26, there are plans to change the way in how
certain packages can be moved out of the emacs master and still can be
installed seamlessly using the tarballs of those.

Currently the org-mode version in emacs master is 8.2.10 and that it too
old (> 2 years, ref: http://orgmode.org/cgit.cgi/org-mode.git/refs/). The
current stable version of org-mode is 9.0.4 (released yesterday).

At the time of releasing emacs 25.1, the org-mode in emacs master could
have been synced up with the then 1.5 years newer and stable version of org
(probably 8.3.5 or 8.3.6). But that got missed due to some reason.

As a precaution that that does not repeat when emacs 26.x is released,
should the org version in emacs master be synced with the now latest stable
org version 9.0.4?

If we are able the release the new packaging method in emacs 26.x, then we
can remove org from emacs master completely, but if not, then at least as
backup we have a newer org version to go out with that release.

-- 

Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 1335 bytes --]

Re: Sync up the org in emacs master to org maint branch?

[Rasmus]

Hi,

Kaushal Modi <kaushal.modi@gmail.com> writes:

> I am aware that in emacs 26, there are plans to change the way in how
> certain packages can be moved out of the emacs master and still can be
> installed seamlessly using the tarballs of those.

Indeed.

> Currently the org-mode version in emacs master is 8.2.10 and that it too
> old (> 2 years, ref: http://orgmode.org/cgit.cgi/org-mode.git/refs/). The
> current stable version of org-mode is 9.0.4 (released yesterday).
>
> At the time of releasing emacs 25.1, the org-mode in emacs master could
> have been synced up with the then 1.5 years newer and stable version of org
> (probably 8.3.5 or 8.3.6). But that got missed due to some reason.

*AFAIR* it was too late and would thus not have received enough test from
the general Emacs community.

> As a precaution that that does not repeat when emacs 26.x is released,
> should the org version in emacs master be synced with the now latest stable
> org version 9.0.4?

Yes.

> If we are able the release the new packaging method in emacs 26.x, then we
> can remove org from emacs master completely, but if not, then at least as
> backup we have a newer org version to go out with that release.

What is the current status?  I am a bit confused about the policy at this
point.  I'm happy to try to update master to 9.0.4, but I was somehow
under the impression that we were waiting for a solution to include ELPA
packages in the Emacs tarball.

Thanks,
Rasmus

-- 
However beautiful the theory, one should occasionally look at the evidence

Re: Sync up the org in emacs master to org maint branch?

[Eli Zaretskii]

> From: Rasmus <rasmus@gmx.us>
> Date: Wed, 25 Jan 2017 17:54:48 +0100
> Cc: emacs-devel@gnu.org
> 
> What is the current status?  I am a bit confused about the policy at this
> point.  I'm happy to try to update master to 9.0.4, but I was somehow
> under the impression that we were waiting for a solution to include ELPA
> packages in the Emacs tarball.

That could take a while, AFAIU, so I wouldn't recommend delaying the
update on that behalf.

Thanks.

Re: Sync up the org in emacs master to org maint branch?

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 1495 bytes --]

On Thu, Jan 26, 2017 at 1:19 AM Kyle Meyer <kyle@kyleam.com> wrote:

> Rasmus <rasmus@gmx.us> writes:
> We'd want at least one more release from maint, I think, so that'd be
> 9.0.5.
>

Would it be OK to sync the current stable 9.0.4, and keep on updating with
each stable release as time comes? We never know, we might end up with even
higher stable releases by the time emacs 26.1 is released.

I have been using emacs master and org master for ages now without any
issues. So emacs master + org 9.0.4 should not cause any serious problems.
The major issues I forsee are the few backward incompatible changes people
might have to make when org changes from 8.2.x to 9.x (though all those
changes are documented in ORG-NEWS).


On Thu, Jan 26, 2017 at 8:46 AM Rasmus <rasmus@gmx.us> wrote:

> So would now be a good time to sync the Emacs master?  I guess the
> appropriate way would be to make a new branch that can eventually be
> merged.
>

Going by the same argument as above, do you think that merging org maint
into emacs master directly is that risky? org master + emacs master has
been super-stable for me.


On Thu, Jan 26, 2017 at 9:22 AM Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> > So would now be a good time to sync the Emacs master?
>
> On `master`, "now" is pretty much *always* a good time to sync.
> More specifically, it's better to always keep `master` in sync with the
> upstream (applies not just to Org).
>
> "sync early, sync often",
>

+1!


-- 

Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 2816 bytes --]

Re: Sync up the org in emacs master to org maint branch?

[Kyle Meyer]

Kaushal Modi <kaushal.modi@gmail.com> writes:

> On Thu, Jan 26, 2017 at 1:19 AM Kyle Meyer <kyle@kyleam.com> wrote:
>
>> We'd want at least one more release from maint, I think, so that'd be
>> 9.0.5.
>
> Would it be OK to sync the current stable 9.0.4,

I don't think that's a good idea.  Since 9.0.4, I've backported one
remaining commit from Emacs, I've adjusted :version in defcustoms to the
appropriate version for a sync targeting Emacs's master, and I've
cleaned up the spacing in a few places so that all the files pass
Emacs's pre-commit check.

> and keep on updating with each stable release as time comes?

Yes, that should be done.

> We never know, we might end up with even higher stable releases by the
> time emacs 26.1 is released.

I suspect we will, given that 9.0.1 -> 9.0.4 have all been released
since this November.

-- 
Kyle

Re: Sync up the org in emacs master to org maint branch?

[Kaushal Modi]

[-- Attachment #1: Type: text/plain, Size: 1155 bytes --]

On Thu, Jan 26, 2017 at 10:39 AM Kyle Meyer <kyle@kyleam.com> wrote:

> Kaushal Modi <kaushal.modi@gmail.com> writes:
>
> > On Thu, Jan 26, 2017 at 1:19 AM Kyle Meyer <kyle@kyleam.com> wrote:
> >
> >> We'd want at least one more release from maint, I think, so that'd be
> >> 9.0.5.
> >
> > Would it be OK to sync the current stable 9.0.4,
>
> I don't think that's a good idea.  Since 9.0.4, I've backported one
> remaining commit from Emacs, I've adjusted :version in defcustoms to the
> appropriate version for a sync targeting Emacs's master, and I've
> cleaned up the spacing in a few places so that all the files pass
> Emacs's pre-commit check.
>

Makes sense. Thanks for the explanation.


> > We never know, we might end up with even higher stable releases by the
> > time emacs 26.1 is released.
>
> I suspect we will, given that 9.0.1 -> 9.0.4 have all been released
> since this November.
>

I don't believe that the target date has yet been set for releasing 26.1.
We are currently on the release candidate testing stage of 25.2. So I will
not be surprised if 26.1 get released even 3 months or 6 months from now.
Eli? John?
-- 

Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 2278 bytes --]

Re: Sync up the org in emacs master to org maint branch?

[Eli Zaretskii]

> From: Kaushal Modi <kaushal.modi@gmail.com>
> Date: Thu, 26 Jan 2017 16:01:24 +0000
> Cc: emacs-org list <emacs-orgmode@gnu.org>,
> 	Stefan Monnier <monnier@iro.umontreal.ca>,
> 	Rasmus <rasmus@gmx.us>, Emacs developers <emacs-devel@gnu.org>
> 
> I don't believe that the target date has yet been set for releasing 26.1. We are currently on the release
> candidate testing stage of 25.2. So I will not be surprised if 26.1 get released even 3 months or 6 months from
> now. Eli? John?

AFAIR, we have never released a major version so quickly, and I don't
see why this one would be different.  A year at least, I'd say,
probably more.

Re: Sync up the org in emacs master to org maint branch?

[John Wiegley]

>>>>> "EZ" == Eli Zaretskii <eliz@gnu.org> writes:

EZ> AFAIR, we have never released a major version so quickly, and I don't see
EZ> why this one would be different. A year at least, I'd say, probably more.

This was my feeling as well.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Sync up the org in emacs master to org maint branch?

[Kyle Meyer]

Rasmus <rasmus@gmx.us> writes:

> Kaushal Modi <kaushal.modi@gmail.com> writes:

[...]

>> As a precaution that that does not repeat when emacs 26.x is released,
>> should the org version in emacs master be synced with the now latest stable
>> org version 9.0.4?
>
> Yes.

We'd want at least one more release from maint, I think, so that'd be
9.0.5.

>> If we are able the release the new packaging method in emacs 26.x, then we
>> can remove org from emacs master completely, but if not, then at least as
>> backup we have a newer org version to go out with that release.
>
> What is the current status?  I am a bit confused about the policy at this
> point.  I'm happy to try to update master to 9.0.4, but I was somehow
> under the impression that we were waiting for a solution to include ELPA
> packages in the Emacs tarball.

Rasmus, I've pushed a branch (emacs-sync) to the Org repo that applies
several patches on top of maint.  These make a few changes to org.texi
and orgcard.tex that I think are appropriate for the sync.  Feel free to
ignore the branch if it's not helpful.

Thanks.

-- 
Kyle

Re: Sync up the org in emacs master to org maint branch?

[Michael Albinus]

Rasmus <rasmus@gmx.us> writes:

> Hi,

Hi,

> *AFAIR* it was too late and would thus not have received enough test from
> the general Emacs community.

org-mode comes with some hundred ert test cases. It would be great if
they'll land also in the Emacs master branch; this will give much more
testing.

> Thanks,
> Rasmus

Best regards, Michael.

Re: Sync up the org in emacs master to org maint branch?

[John Wiegley]

>>>>> "KM" == Kaushal Modi <kaushal.modi@gmail.com> writes:

KM> If we are able the release the new packaging method in emacs 26.x, then we
KM> can remove org from emacs master completely, but if not, then at least as
KM> backup we have a newer org version to go out with that release.

For Emacs 26, I intend the new ELPA process to be in place, whereby "default"
packages can be developed separately, and declare a way to get slip-streamed
into the release tarball so users are unaware of the separate nature of their
development.

The CEDET developers have agreed to support this, and it sounds like you are
willing to as well. If Lars is game, I'd like for Gnus to be third major
package we do this for initially. That will reduce considerably the number of
external files we track in Emacs.git.

The precise technical details have yet to be worked out, but it shouldn't be
too difficult. Phillip Lord has already began advance work on alternatives,
and I've received offers of help from others to work on this new process.

I think now is a good time to begin. The first step is to solidify what is
meant by "tarball EPLA", and the means of slip-streaming a package's contents.
This will require at least two bits:

  - Some form of declaration to indicate how external files should appear in
    the tarball. In order for the first version of this scheme to be as low
    impact as possible, this should probably be done with a sexp in a data
    file, to be checked in alongside the EPLA.git import of the project.

  - changes to "make dist" to integrate these files, and setup autoloading so
    their inclusion is transparent to end users.

Please comment with your recommendations for the first, and supporting changes
for the second, if anyone has ideas. Phillip, how is your work on these coming
along?

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Sync up the org in emacs master to org maint branch?

[Stefan Monnier]

> Please comment with your recommendations for the first, and supporting
> changes for the second, if anyone has ideas.  Phillip, how is your
> work on these coming along?

BTW, the easiest packages with which to start experimenting are those
which are already in elpa.git.


        Stefan

Re: Sync up the org in emacs master to org maint branch?

[David Engster]

John Wiegley writes:
>>>>>> "KM" == Kaushal Modi <kaushal.modi@gmail.com> writes:
>
> KM> If we are able the release the new packaging method in emacs 26.x, then we
> KM> can remove org from emacs master completely, but if not, then at least as
> KM> backup we have a newer org version to go out with that release.
>
> For Emacs 26, I intend the new ELPA process to be in place, whereby "default"
> packages can be developed separately, and declare a way to get slip-streamed
> into the release tarball so users are unaware of the separate nature of their
> development.
>
> The CEDET developers have agreed to support this, and it sounds like you are
> willing to as well.

This is a misunderstanding. I said I wanted to move support for certain
languages and project types into ELPA, not CEDET core. I'm still of the
opinion that moving it completely to ELPA is a mistake.

> If Lars is game, I'd like for Gnus to be third major
> package we do this for initially. That will reduce considerably the number of
> external files we track in Emacs.git.

CEDET and Gnus are not external anymore. Both have abandonded their
upstream repos and moved to Emacs, because the faster development of
Emacs has made that necessary.

-David

Re: Sync up the org in emacs master to org maint branch?

[John Wiegley]

>>>>> "DE" == David Engster <deng@randomsample.de> writes:

DE> This is a misunderstanding. I said I wanted to move support for certain
DE> languages and project types into ELPA, not CEDET core. I'm still of the
DE> opinion that moving it completely to ELPA is a mistake.

It would only be a mistake if other parts of core need to use it. If only
users make use of it, then having it ELPA will be invisible to them.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Sync up the org in emacs master to org maint branch?

[Edward John Steere]

>
> DE> This is a misunderstanding. I said I wanted to move support for certain
> DE> languages and project types into ELPA, not CEDET core. I'm still of the
> DE> opinion that moving it completely to ELPA is a mistake.
>
> It would only be a mistake if other parts of core need to use it. If only
> users make use of it, then having it ELPA will be invisible to them.

Apologies in advance for the significant verbage.

I just want to provide context.

* Context regarding CEDET:
(apologies: I know this thread is more concerned with org mode and, if
you'll bare with me, I think that this is relevant.)

I started the CEDET merge process a few months ago.  There was talk on
the CEDET mailing list regarding the difficulty of getting going with
CEDET as a user and/or a developer.  One of the ideas put forward was
that there ought to be a merge into Emacs so that one wouldn't have to
clone the repo, build the code and install it with some ELisp
config. hacking.

I got in touch with Eric directly and bounced some ideas passed him.
Subsequently I volunteered to help with the merge.  I got in touch with
JohnW and (not being very familiar with Emacs development) asked some
basic questions about how to go about accomplishing the merge.  What
resulted instead was the idea that we should try to look at streamlining
how CEDET get's included with the Emacs tarball rather than having it
live in two repositories.  I agreed with the idea and got to work on
getting a version of CEDET together which would work with 26.  I merged
in the Emacs -> CEDET direction and ended up with a version of CEDET
which is a WIP and works with Emacs 26.

Some time passed between then and the start of the discussions here.  I
think that the approach has evolved past what I was originally planning
on working towards and is now something along the lines of: do a final
merge of core CEDET components and make the rest into a series of ELPA
packages.


* What I think that we shouldn't lose sight of (if I may suggest it): is
that packaging CEDET, Org Mode and other packages like them in a process
which integrates them only when producing the tarball would serve to
simplify things for everyone.  Emacs core wouldn't be able to depend on
packages which are more relevant to the users (and package developers)
of Emacs, but these packages would still there like they always have
been when one downloads binary Emacs.

I'm new around here and I know that I lack the context and experience in
this project to make such swooping suggestions or judge the validity of
these points, but I thought that they would be worth raising.

Kind regards,

Edward Steere

Re: Sync up the org in emacs master to org maint branch?

[John Wiegley]

>>>>> "EJS" == Edward John Steere <edward.steere@gmail.com> writes:

EJS> What I think that we shouldn't lose sight of (if I may suggest it): is
EJS> that packaging CEDET, Org Mode and other packages like them in a process
EJS> which integrates them only when producing the tarball would serve to
EJS> simplify things for everyone. Emacs core wouldn't be able to depend on
EJS> packages which are more relevant to the users (and package developers) of
EJS> Emacs, but these packages would still there like they always have been
EJS> when one downloads binary Emacs.

This is the very point I was hoping to make, thanks Edward. If it can simplify
things for everyone, great; if it really will only complicate things, not so
great.

Many software projects have package ecosystems surrounding them that deal with
similar issues, and I can't, off-hand, think of cases where the answer was
"let's move some of those packages into core development to ease ___". This is
why I find it a bit surprising that some feel so strongly about keeping these
large, external packages in core.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Sync up the org in emacs master to org maint branch?

[David Engster]

John Wiegley writes:
> Many software projects have package ecosystems surrounding them that deal with
> similar issues, and I can't, off-hand, think of cases where the answer was
> "let's move some of those packages into core development to ease
> ___".

Just from the software I worked with over the years, I remember
KDevelop, Typo3, Drupal and MediaWiki moving plugins to core. It's
really not uncommon. I still use Drupal, and moving important plugins to
core has made updates *much* less brittle. I just wish Jenkins would do
the same already...

-David

Re: Sync up the org in emacs master to org maint branch?

[David Engster]

John Wiegley writes:
>>>>>> "DE" == David Engster <deng@randomsample.de> writes:
>
> DE> This is a misunderstanding. I said I wanted to move support for certain
> DE> languages and project types into ELPA, not CEDET core. I'm still of the
> DE> opinion that moving it completely to ELPA is a mistake.
>
> It would only be a mistake if other parts of core need to use it. If only
> users make use of it, then having it ELPA will be invisible to them.

It is a mistake because you are creating more moving targets and bring
them together very late in the release process. This reduces the amount
of testing that is done for those packages, so bugs will be noticed
later and the quality of the relases suffer. It moves even more work
into the RC-phase, which is already crowded and where people who can fix
those bugs might not be readily available. It removes those packages
from Emacs CI, so that breakages due to changes in core are not
immediately noticed, and often times they have to be fixed not by those
who created the breakage, but by those who notice them.

-David

Re: Sync up the org in emacs master to org maint branch?

[John Wiegley]

>>>>> "DE" == David Engster <deng@randomsample.de> writes:

DE> It is a mistake because you are creating more moving targets and bring
DE> them together very late in the release process. This reduces the amount of
DE> testing that is done for those packages, so bugs will be noticed later and
DE> the quality of the relases suffer. It moves even more work into the
DE> RC-phase, which is already crowded and where people who can fix those bugs
DE> might not be readily available. It removes those packages from Emacs CI,
DE> so that breakages due to changes in core are not immediately noticed, and
DE> often times they have to be fixed not by those who created the breakage,
DE> but by those who notice them.

These are issues to be fixed in the way ELPA integrates with our development
process. I recognize today's ELPA may have these drawbacks, but I believe they
can be fixed.

We're moving toward a future where Emacs.git will represent "core Emacs", and
only contain what core needs (plus a few historical bits, I'm sure). There
should be no argument for keeping a project in core just to gain auxiliary
benefits.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Sync up the org in emacs master to org maint branch?

[Lars Ingebrigtsen]

John Wiegley <jwiegley@gmail.com> writes:

> We're moving toward a future where Emacs.git will represent "core
> Emacs", and only contain what core needs (plus a few historical bits,
> I'm sure). There should be no argument for keeping a project in core
> just to gain auxiliary benefits.

I'm massively unenthusiastic about this future.  Things in ELPA has to
be backwards-and-forwards compatible with a wide Emacs version range,
which makes maintaining things much more work.  When you develop things
in "Emacs core", you have one specific target and can make large
internal changes without these considerations.

Emacs doesn't seem to have a massive surfeit of developers, so I wonder
where this plan comes from.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Sync up the org in emacs master to org maint branch?

[John Wiegley]

>>>>> "LI" == Lars Ingebrigtsen <larsi@gnus.org> writes:

LI> I'm massively unenthusiastic about this future. Things in ELPA has to be
LI> backwards-and-forwards compatible with a wide Emacs version range, which
LI> makes maintaining things much more work. When you develop things in "Emacs
LI> core", you have one specific target and can make large internal changes
LI> without these considerations.

So far, all of these arguments against a tighter development integration with
ELPA have been predicated on the way that ELPA is used today. ELPA is under
our control; we can adjust our process to suit the needs of Emacs development.

LI> Emacs doesn't seem to have a massive surfeit of developers, so I wonder
LI> where this plan comes from.

It comes from the desire to decouple the development of large, mostly external
projects, from core Emacs. They don't belong in Emacs.git.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Sync up the org in emacs master to org maint branch?

[Lars Ingebrigtsen]

John Wiegley <jwiegley@gmail.com> writes:

> So far, all of these arguments against a tighter development integration with
> ELPA have been predicated on the way that ELPA is used today. ELPA is under
> our control; we can adjust our process to suit the needs of Emacs development.

Yes, but external packages lose much of their value if they aren't
developed in a compatible manner.

> LI> Emacs doesn't seem to have a massive surfeit of developers, so I wonder
> LI> where this plan comes from.
>
> It comes from the desire to decouple the development of large, mostly external
> projects, from core Emacs. They don't belong in Emacs.git.

But you're talking about coupling ELPA tighter with core Emacs, too.
"They don't belong" isn't really much of an argument here.

The question is: What is the most effective way for Emacs developers to
spend their time?  I can't really see that anybody has made the case
that shifting stuff from Emacs core to ELPA will mean less work for...
well, anybody.

(Except perhaps CEDET.  There seems to be a lot of merging problems
there.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Sync up the org in emacs master to org maint branch?

[David Engster]

Lars Ingebrigtsen writes:
> The question is: What is the most effective way for Emacs developers to
> spend their time?  I can't really see that anybody has made the case
> that shifting stuff from Emacs core to ELPA will mean less work for...
> well, anybody.
>
> (Except perhaps CEDET.  There seems to be a lot of merging problems
> there.)

This is precisely why we're currently moving all development to Emacs
and abandon the external repo. At least we were planning to...

-David

Re: Sync up the org in emacs master to org maint branch?

[Lars Ingebrigtsen]

David Engster <deng@randomsample.de> writes:

>> (Except perhaps CEDET.  There seems to be a lot of merging problems
>> there.)
>
> This is precisely why we're currently moving all development to Emacs
> and abandon the external repo. At least we were planning to...

Oh, great.  One less thing that would be helped by moving to ELPA.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Sync up the org in emacs master to org maint branch?

[Stephen Leake]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> John Wiegley <jwiegley@gmail.com> writes:
>
>> We're moving toward a future where Emacs.git will represent "core
>> Emacs", and only contain what core needs (plus a few historical bits,
>> I'm sure). There should be no argument for keeping a project in core
>> just to gain auxiliary benefits.
>
> I'm massively unenthusiastic about this future.  Things in ELPA has to
> be backwards-and-forwards compatible with a wide Emacs version range,

no, they don't.

That is one possible policy, but each package decides for itself whether
to follow it. You can have

;; package-requires: ((emacs "25.2"))

if you want.

If you want to maintain two versions, one for older emacs, one for
newer, you'll need two different package names, similar to gtk2, gtk3.


It does appear we need a "next release" branch in ELPA git similar to
"master" in emacs git, so the released ELPA package is still available,
but those working on emacs master can access the leading edge ELPA
packages.

> Emacs doesn't seem to have a massive surfeit of developers, so I wonder
> where this plan comes from.

Several developers prefer the decoupled development cycle of ELPA
packages.

It is primarily up to the package developers whether a package is in
core or not, at least until it is clear that there is no advantage to
being in core.

-- 
-- Stephe

Re: Sync up the org in emacs master to org maint branch?

[Lars Ingebrigtsen]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

>> I'm massively unenthusiastic about this future.  Things in ELPA has to
>> be backwards-and-forwards compatible with a wide Emacs version range,
>
> no, they don't.
>
> That is one possible policy, but each package decides for itself whether
> to follow it. You can have
>
> ;; package-requires: ((emacs "25.2"))
>
> if you want.

That's technically correct.  (The best kind of correct, as they say.)
But for a package archive to be useful, it should support a range of
Emacs releases, otherwise users will be sad.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Sync up the org in emacs master to org maint branch?

[Aaron Ecay]

Hi Lars,

2017ko urtarrilak 31an, Lars Ingebrigtsen-ek idatzi zuen:
> 
> John Wiegley <jwiegley@gmail.com> writes:
> 
>> We're moving toward a future where Emacs.git will represent "core
>> Emacs", and only contain what core needs (plus a few historical bits,
>> I'm sure). There should be no argument for keeping a project in core
>> just to gain auxiliary benefits.
> 
> I'm massively unenthusiastic about this future.  Things in ELPA has to
> be backwards-and-forwards compatible with a wide Emacs version range,

This seems like a technical limitation of ELPAʼs current implementation,
rather than a conceptual impossibility.  If ELPA made available (on the
server for downloading, and in the client for installing) old versions
of packages, then users could always be offered the latest compatible
version, but not later incompatible ones.

Developers would have to be a little more diligent about declaring their
packagesʼ dependencies on emacs major versions (or on other packages, if
they depend on parts of core that have migrated to ELPA), but this would
be a small hurdle.

Aaron

PS Speaking of dependency management, Iʼd be more worried that this kind
of approach will accelerate the advent of dependency hell with ELPA
packages...but I think all package repos have to confront that problem
eventually.  So Iʼd file that thought under “inevitable growing pains”
rather than “arguments against”.

-- 
Aaron Ecay

Re: Sync up the org in emacs master to org maint branch?

[Lars Ingebrigtsen]

Aaron Ecay <aaronecay@gmail.com> writes:

> If ELPA made available (on the server for downloading, and in the
> client for installing) old versions of packages, then users could
> always be offered the latest compatible version, but not later
> incompatible ones.

I don't think having Emacs developers fixing bugs in a number of
different versions of a package sounds like a good way to spend time,
either.

We're not infinite monkeys, I'm sad to say.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Sync up the org in emacs master to org maint branch?

[Phillip Lord]

On Wed, February 1, 2017 6:51 pm, Lars Ingebrigtsen wrote:
> Aaron Ecay <aaronecay@gmail.com> writes:
>
>
>> If ELPA made available (on the server for downloading, and in the
>> client for installing) old versions of packages, then users could always
>> be offered the latest compatible version, but not later incompatible
>> ones.
>
> I don't think having Emacs developers fixing bugs in a number of
> different versions of a package sounds like a good way to spend time,
> either.

Well, they do that anyway. Org-mode, in Emacs core is quite a way behind
org-mode latest. That's the start point of this discussion. And, for
packages which are maintained only in core (not synced to an external repo
like org), well, we currently have Emacs-25 and Emacs-26 in active
development.

Currently development of Emacs core with slow releases contributes to
this, because the old versions of org remain around and in active use for
a long period of time. In the case of org, this was exacerbated when it
changed the features it provides, meaning that upgrading to ELPA worked
imperfectly.

There is even a reasonable possibility that with a smaller core, and
faster releases, fewer changes would happen in core, so supporting
multiple versions might well become easier because the differences would
be smaller.

These are complex questions, and it's hard to get evidence one way or
another. But many other systems do support numerous small packages,
building up into a greater whole: consider npm, CPAN, CRAN, CTAN or,
indeed, debian. And, yes, sometimes we do end up in version hell, but
mostly we don't.

Phil

Re: Sync up the org in emacs master to org maint branch?

[Stefan Monnier]

> Well, they do that anyway. Org-mode, in Emacs core is quite a way behind
> org-mode latest. That's the start point of this discussion.  And, for
> packages which are maintained only in core (not synced to an external repo
> like org), well, we currently have Emacs-25 and Emacs-26 in active
> development.

Org-mode is (for me) one of the best candidates to move to elpa.git
exactly for that reason: the main issue is to avoid having
separate branches.  Moving it to elpa.git doesn't impose any extra
hassle of backward compatibility since it's already distributed
separately and hence already has to cater to backward compatibility.

But that doesn't apply to all packages.


        Stefan

Re: Sync up the org in emacs master to org maint branch?

[David Engster]

John Wiegley writes:
>>>>>> "DE" == David Engster <deng@randomsample.de> writes:
>
> DE> It is a mistake because you are creating more moving targets and bring
> DE> them together very late in the release process. This reduces the amount of
> DE> testing that is done for those packages, so bugs will be noticed later and
> DE> the quality of the relases suffer. It moves even more work into the
> DE> RC-phase, which is already crowded and where people who can fix those bugs
> DE> might not be readily available. It removes those packages from Emacs CI,
> DE> so that breakages due to changes in core are not immediately noticed, and
> DE> often times they have to be fixed not by those who created the breakage,
> DE> but by those who notice them.
>
> These are issues to be fixed in the way ELPA integrates with our development
> process. I recognize today's ELPA may have these drawbacks, but I believe they
> can be fixed.

This really has not much to do with how ELPA works. My points above are
about the underlying concept you are proposing: moving packages out of
Emacs core and hence removing them from current Emacs development, but
still bundling them with the release. It's a have-and-eat-cake concept.

> We're moving toward a future where Emacs.git will represent "core Emacs", and
> only contain what core needs (plus a few historical bits, I'm sure). There
> should be no argument for keeping a project in core just to gain auxiliary
> benefits.

Of the points I raise above, which fall under "just to gain auxiliary
benefits"? I'm honestly confused. I'm specifically talking about the
quality of the Emacs relases.

Also, I currently have no idea how to continue with CEDET, as the future
where development should happen is unclear, and I get the feeling we're
just waisting our time with the ongoing merge.

-David

Re: Sync up the org in emacs master to org maint branch?

[John Wiegley]

>>>>> "DE" == David Engster <deng@randomsample.de> writes:

DE> Also, I currently have no idea how to continue with CEDET, as the future
DE> where development should happen is unclear, and I get the feeling we're
DE> just waisting our time with the ongoing merge.

Until the dust has settled, please proceed, assuming nothing has changed. Move
your primary development into Emacs.git.

The changes I'm proposing don't have to happen tomorrow, and I can still be
convinced they're unnecessary. My gut tells me, however, that we're supporting
an unnecessarily monolithic development model for no better reason than "we're
used to it".

In fact, what we're doing feels like if Python included Django in its main
repository, just to solve Django's problems of compatibility, testing, and
making its bugs known to the main Python developers.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Sync up the org in emacs master to org maint branch?

[Lars Ingebrigtsen]

John Wiegley <jwiegley@gmail.com> writes:

> In fact, what we're doing feels like if Python included Django in its main
> repository, just to solve Django's problems of compatibility, testing, and
> making its bugs known to the main Python developers.

I guess that would be a fair comparison.

If Django had traditionally always been distributed along with Python,
and maintained in the Python repo, and the suggestion now would be to
move Django to a part of the Python repo that very few developers look
at, but Django would continue to be distributed with Python, and all
Django bug reports would continue to go to the Python bug repository,
and Python developers would continue to be responsible for QA and bug
fixing of Django.

See?  It's totally the same thing.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Sync up the org in emacs master to org maint branch?

[John Wiegley]

>>>>> "LI" == Lars Ingebrigtsen <larsi@gnus.org> writes:

LI> If Django had traditionally always been distributed along with Python, and
LI> maintained in the Python repo, and the suggestion now would be to move
LI> Django to a part of the Python repo that very few developers look at, but
LI> Django would continue to be distributed with Python, and all Django bug
LI> reports would continue to go to the Python bug repository, and Python
LI> developers would continue to be responsible for QA and bug fixing of
LI> Django.

OK, to continue the analogy, what is the right answer? Technically it doesn't
seem as though Django belongs there, even if culturally it sounds hard to
separate. Should it stay indefinitely, or should the development model change?

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Sync up the org in emacs master to org maint branch?

[Lars Ingebrigtsen]

John Wiegley <jwiegley@gmail.com> writes:

> OK, to continue the analogy, what is the right answer? Technically it
> doesn't seem as though Django belongs there, even if culturally it
> sounds hard to separate. Should it stay indefinitely, or should the
> development model change?

If somebody genuinely offered to take over, say, rmail, and maintain it
separately, and handle bug reports, and, like, be the maintainer, that
would be a help.  Great, go ahead, and put the resulting thing in ELPA.

But nobody has made that offer?  Or have they, and I just missed it?

I fail to see how just shuffling rmail from Emacs to ELPA helps us in
any way with the maintainership.  Instead it creates an extra burden on
us, since we (in addition to all the normal maintainership) will also
have to consider Emacs version compatibility.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Sync up the org in emacs master to org maint branch?

[Dmitry Gutov]

On 02.02.2017 14:10, Lars Ingebrigtsen wrote:

> If Django had traditionally always been distributed along with Python,
> and maintained in the Python repo,

I'm pretty sure the first versions of Emacs came without Gnus. Later, it 
got bundled. Some time after that, Org and CEDET joined too.

All of that was before we got ELPA and with it, a very easy way for 
users to install third-party packages.

> and the suggestion now would be to
> move Django to a part of the Python repo that very few developers look
> at,

I've never looked at the Gnus source code either, even inside the Emacs 
repository.

> but Django would continue to be distributed with Python, and all
> Django bug reports would continue to go to the Python bug repository,
> and Python developers would continue to be responsible for QA and bug
> fixing of Django.

You could introduce a separate bug tracker, if that helps.

Re: Sync up the org in emacs master to org maint branch?

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Thu, 02 Feb 2017 13:10:07 +0100
> Cc: Bastien Guerry <bzg@gnu.org>, Emacs developers <emacs-devel@gnu.org>,
> 	Phillip Lord <phillip.lord@russet.org.uk>,
> 	emacs-org list <emacs-orgmode@gnu.org>,
> 	Kaushal Modi <kaushal.modi@gmail.com>
> 
> If Django had traditionally always been distributed along with Python,
> and maintained in the Python repo, and the suggestion now would be to
> move Django to a part of the Python repo that very few developers look
> at, but Django would continue to be distributed with Python, and all
> Django bug reports would continue to go to the Python bug repository,
> and Python developers would continue to be responsible for QA and bug
> fixing of Django.

I believe the intent is to make it so that checking out and building
Emacs also checks out and builds all the packages that are intended to
be part of a release tarball.  If we indeed do that this way, there
will be no difference, QA-wise, between core packages and ELPA
packages that are logically part of an Emacs release.

Re: Sync up the org in emacs master to org maint branch?

[David Engster]

Eli Zaretskii writes:
>> From: Lars Ingebrigtsen <larsi@gnus.org>
>> Date: Thu, 02 Feb 2017 13:10:07 +0100
>> Cc: Bastien Guerry <bzg@gnu.org>, Emacs developers <emacs-devel@gnu.org>,
>
>> 	Phillip Lord <phillip.lord@russet.org.uk>,
>> 	emacs-org list <emacs-orgmode@gnu.org>,
>> 	Kaushal Modi <kaushal.modi@gmail.com>
>> 
>> If Django had traditionally always been distributed along with Python,
>> and maintained in the Python repo, and the suggestion now would be to
>> move Django to a part of the Python repo that very few developers look
>> at, but Django would continue to be distributed with Python, and all
>> Django bug reports would continue to go to the Python bug repository,
>> and Python developers would continue to be responsible for QA and bug
>> fixing of Django.
>
> I believe the intent is to make it so that checking out and building
> Emacs also checks out and builds all the packages that are intended to
> be part of a release tarball.  If we indeed do that this way, there
> will be no difference, QA-wise, between core packages and ELPA
> packages that are logically part of an Emacs release.

That's not how I understood it. It was always said that Emacs must not
depend on those ELPA packages that are shipped with the release, which
implies that they are not supposed to be present at a "normal"
checkout. Otherwise, what difference would it make? (Besides requiring
more cumbersome tooling to make this all work.)

-David

Re: Sync up the org in emacs master to org maint branch?

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Cc: Lars Ingebrigtsen <larsi@gnus.org>,  bzg@gnu.org,  emacs-devel@gnu.org,  phillip.lord@russet.org.uk,  emacs-orgmode@gnu.org,  kaushal.modi@gmail.com
> Date: Thu, 02 Feb 2017 18:47:49 +0100
> 
> > I believe the intent is to make it so that checking out and building
> > Emacs also checks out and builds all the packages that are intended to
> > be part of a release tarball.  If we indeed do that this way, there
> > will be no difference, QA-wise, between core packages and ELPA
> > packages that are logically part of an Emacs release.
> 
> That's not how I understood it.

I hope you have misunderstood, and not me.

> It was always said that Emacs must not depend on those ELPA packages
> that are shipped with the release

I'm talking about building Emacs from Git, not from a release
tarball.  For the latter, you are right, and we are in agreement.  But
that's not relevant for the issue at hand, which appears to be the
attention which such ELPA packages will get from Emacs developers.
For that, building a fresh checkout should also build the latest
versions from ELPA, from a branch that the package maintainers will
designate as the equivalent of the Emacs master branch.

> which implies that they are not supposed to be present at a "normal"
> checkout.

I don't see how it implies that.  Release tarballs are prepared
specially for a release, so their build procedures don't have to be
identical to building from Git.

> Otherwise, what difference would it make?

Ask the package maintainers, they see significant advantages in being
able to release interim versions independent of Emacs releases.  But
we are not talking about that aspect, we are talking about the
parallel development of core and the packages.

Re: Sync up the org in emacs master to org maint branch?

[David Engster]

Eli Zaretskii writes:
>> From: David Engster <deng@randomsample.de>
>> Cc: Lars Ingebrigtsen <larsi@gnus.org>, bzg@gnu.org,
>> emacs-devel@gnu.org, phillip.lord@russet.org.uk,
>
>> which implies that they are not supposed to be present at a "normal"
>> checkout.
>
> I don't see how it implies that.  Release tarballs are prepared
> specially for a release, so their build procedures don't have to be
> identical to building from Git.
>
>> Otherwise, what difference would it make?
>
> Ask the package maintainers, they see significant advantages in being
> able to release interim versions independent of Emacs releases.

But we are discussing moving CEDET, Gnus and Org out of Emacs core, and
at least the former two do not plan to provide updates between Emacs
releases. We already had this discussion in 2015, where John was pretty
determined to remove CEDET from core:

https://lists.gnu.org/archive/html/emacs-devel/2015-11/msg00877.html

-David

Re: Sync up the org in emacs master to org maint branch?

[Eli Zaretskii]

> From: David Engster <deng@randomsample.de>
> Cc: emacs-devel@gnu.org,  bzg@gnu.org,  emacs-orgmode@gnu.org,  kaushal.modi@gmail.com,  larsi@gnus.org,  phillip.lord@russet.org.uk
> Date: Thu, 02 Feb 2017 21:57:02 +0100
> 
> > Ask the package maintainers, they see significant advantages in being
> > able to release interim versions independent of Emacs releases.
> 
> But we are discussing moving CEDET, Gnus and Org out of Emacs core, and
> at least the former two do not plan to provide updates between Emacs
> releases.

That's fine with me, I have nothing against leaving some packages in
emacs.git if their maintainers so wish.  I was replying to a single
aspect of this discussion: the expectation that packages which will be
moved to ELPA will necessarily and inevitably suffer in terms of QA
and the developers' attention they receive.  I'm saying that AFAIU
this is not supposed to happen, for the reasons I presented.

Re: Sync up the org in emacs master to org maint branch?

[Stefan Monnier]

> In fact, what we're doing feels like if Python included Django in its main
> repository, just to solve Django's problems of compatibility, testing, and
> making its bugs known to the main Python developers.

No: the reason CEDET is inside Emacs is double:
1- We wanted to make CEDET's features available to more users.
2- We wanted to integrate it more tightly with Emacs (not in terms of
   bug-tracking and releasing schedule, but in terms of making it
   possible for generic Emacs code to use some of CEDET, and to
   encourage more major modes and other features to use CEDET).

Moving all of CEDET to elpa.git (and then including it in the tarball)
would still get us point nb 1 but not point nb 2.

The main problem so far with CEDET is the "two branches" situation, made
worse by the "time-limited" copyright assignment of Eric.

I'm not sure what's going on w.r.t the Eric's copyright assignment, but
if that problem is still with us, then it will plague CEDET-in-elpa just
as much as it did CEDET-in-emacs.

I think the main plan should be to consolidate the development into
a single branch (of course, that can also be several branches, as long
as they are constantly kept in sync).  The easiest would be to have that
"upstream" be in emacs.git for now (since moving Emacs's CEDET outside
of Emacs would likely take time anyway).  It might also be beneficial to
move some of CEDET to elpa.git (those features which aren't "core" and
are hence unlikely to be used by other parts of Emacs), so their
development can be decoupled.  But I think CEDET-core (such as
lisp/cedet/semantic) should stay in emacs.git (and more packages should
make use of it).


        Stefan

Using CEDET modules from Emacs core

[John Wiegley]

>>>>> "SM" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

SM> 2- We wanted to integrate it more tightly with Emacs (not in terms of
SM>    bug-tracking and releasing schedule, but in terms of making it
SM>    possible for generic Emacs code to use some of CEDET, and to
SM>    encourage more major modes and other features to use CEDET).

Hi Stefan,

Can you clarify what the plans are here?  Which CEDET features would we want
to use from core?

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Using CEDET modules from Emacs core

[Stefan Monnier]

SM> 2- We wanted to integrate it more tightly with Emacs (not in terms of
SM> bug-tracking and releasing schedule, but in terms of making it
SM> possible for generic Emacs code to use some of CEDET, and to
SM> encourage more major modes and other features to use CEDET).
> Can you clarify what the plans are here?

The plans were not very clear, no.  Just a general feeling that there's
a lot of opportunity for integration.  It has not materialized the way
we had hoped, admittedly.  I guess I'd consider the xref work as
something in that direction, although it happened more by replacing
CEDET's system than by integrating it.

> Which CEDET features would we want to use from core?

For one, I'd like to see more major modes come with support for Semantic
right in the major mode's own definition (rather than have it part of
CEDET).  E.g. for Elisp mode, CC-mode, ...

The idea is to get to the point where Semantic support is just another
thing that a major mode should aim to support alongside syntax-tables,
indentation, font-lock, outline-minor-mode, ...


        Stefan

Re: Using CEDET modules from Emacs core

[Edward John Steere]

>> Which CEDET features would we want to use from core?
>
> For one, I'd like to see more major modes come with support for Semantic
> right in the major mode's own definition (rather than have it part of
> CEDET).  E.g. for Elisp mode, CC-mode, ...
>
> The idea is to get to the point where Semantic support is just another
> thing that a major mode should aim to support alongside syntax-tables,
> indentation, font-lock, outline-minor-mode, ...

This sounds like a great idea!  Semantic appears to be to be stable
enough that we might want to consider extracting it from CEDET in core
like EIEIO was.

Perhaps it's worth considering this line of thought: that are parts of
CEDET which are worthy of becoming part of Emacs proper.  As Stefan
said, semantic is a perfect example of something which built in modes
could benefit from.

There are other parts of CEDET which I don't think meet the criteria of
being stable and general enough that they should be considered for this.
Such as EDE and the external databases for semantic db.  All of which
are useful, but unstable (and sometimes very slow) and I feel like they
shouldn't be expected as part of the core editing experience --
i.e. that one should have to buy into their use.

Re: Using CEDET modules from Emacs core

[John Wiegley]

>>>>> "SM" == Stefan Monnier <monnier@iro.umontreal.ca> writes:

SM> For one, I'd like to see more major modes come with support for Semantic
SM> right in the major mode's own definition (rather than have it part of
SM> CEDET). E.g. for Elisp mode, CC-mode, ...

SM> The idea is to get to the point where Semantic support is just another
SM> thing that a major mode should aim to support alongside syntax-tables,
SM> indentation, font-lock, outline-minor-mode, ...

Is the semantic support really at the point of warranting that? Does it have
many users currently? Is it something major-modes would want to include
default support for?

The last time I tried it, for C++ code, it was far too slow. Are you saying
it's effective for other languages, like Python or Javascript or Go?

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Using CEDET modules from Emacs core

[Stefan Monnier]

SM> For one, I'd like to see more major modes come with support for Semantic
SM> right in the major mode's own definition (rather than have it part of
SM> CEDET). E.g. for Elisp mode, CC-mode, ...
SM> The idea is to get to the point where Semantic support is just another
SM> thing that a major mode should aim to support alongside syntax-tables,
SM> indentation, font-lock, outline-minor-mode, ...
> Is the semantic support really at the point of warranting that?

What more would you want?

> Does it have many users currently?

Chicken, meet Egg!

> Is it something major-modes would want to include default support for?

It provides a functionality that's usually requested by users, so I'm
surprised you'd ask.

We don't have enough infrastructure support in general for "advanced"
editing functionality such as type/scope-aware completion.  So we have
various add-on thingies (like company-mode, cedet, and auto-complete)
which provide such support for specific modes, but really these should
move to core, so that major modes can themselves provide support for
such completion.

I've done some of that with completion-at-point-functions and moving
some company-<foo>.el to <foo>-mode.el, but there's a lot more to do.


        Stefan

Re: Using CEDET modules from Emacs core

[Dmitry Gutov]

On 12.02.2017 05:33, Stefan Monnier wrote:

> We don't have enough infrastructure support in general for "advanced"
> editing functionality such as type/scope-aware completion.  So we have
> various add-on thingies (like company-mode, cedet, and auto-complete)
> which provide such support for specific modes, but really these should
> move to core, so that major modes can themselves provide support for
> such completion.

That doesn't always equate to "add semantic-mode support", and in many 
cases won't be optimal.

I don't have anything against supporting Semantic more widely, but we 
should understand that it isn't something all users want. And the 
"Semantic is too slow for C++" complaint (e.g. compared to Clang-based 
background process solutions) is unlikely to go away.

Re: Using CEDET modules from Emacs core

[Eric Ludlam]

On 02/12/2017 11:00 AM, Dmitry Gutov wrote:
> On 12.02.2017 05:33, Stefan Monnier wrote:
> I don't have anything against supporting Semantic more widely, but we
> should understand that it isn't something all users want. And the
> "Semantic is too slow for C++" complaint (e.g. compared to Clang-based
> background process solutions) is unlikely to go away.

There are a lot of ways to use semantic which changes its performance 
profile.  It depends on what features you want.  Most configuration help 
assumes you want all the most time consuming features, but there are 
also simple helpful features that are supported with minimal parsing 
support.   To boot, C++ is also the oldest parser in the suite and it 
wasn't updated to the newer/faster parser generator.

While I haven't had time to work on CEDET  lately, I'd be happy to 
discuss specific performance issues and share ideas on how to improve 
them, presumably after the most recent merge is completed.

Eric

Re: Using CEDET modules from Emacs core

[Stefan Monnier]

>> "Semantic is too slow for C++" complaint (e.g. compared to Clang-based
>> background process solutions) is unlikely to go away.
> While I haven't had time to work on CEDET lately, I'd be happy to discuss
> specific performance issues and share ideas on how to improve them,
> presumably after the most recent merge is completed.

Isn't it the case that CEDET could also make use of a Clang backend?


        Stefan

Re: Sync up the org in emacs master to org maint branch?

[David Engster]

John Wiegley writes:
>>>>>> "DE" == David Engster <deng@randomsample.de> writes:
>
> DE> Also, I currently have no idea how to continue with CEDET, as the future
> DE> where development should happen is unclear, and I get the feeling we're
> DE> just waisting our time with the ongoing merge.
>
> Until the dust has settled, please proceed, assuming nothing has
> changed. Move your primary development into Emacs.git.
>
> The changes I'm proposing don't have to happen tomorrow, and I can still be
> convinced they're unnecessary.

So if you don't get convinced, we'll just move again, right? No big
deal.

> My gut tells me, however, that we're supporting an unnecessarily
> monolithic development model for no better reason than "we're used to
> it".
>
> In fact, what we're doing feels like if Python included Django in its main
> repository, just to solve Django's problems of compatibility, testing, and
> making its bugs known to the main Python developers.

You are insinuating that my motivation is to delegate CEDET development
to the core Emacs developers. This is simply not true, and I don't see
how my original mail could be interpreted like that.

So let me try again: What I find completely misguided is to move
packages out of core *but still putting them into the release*. In other
words, in my opinion there are really just two options that make sense:
you either keep a package in core, or you kick it out and don't ship it
with the release.

Say the Python developers would decide: Hey, many people like Django, so
let's just put their latest git master into our release and ship
it. Would you think that is a good approach?

-David

Re: Sync up the org in emacs master to org maint branch?

[Stefan Monnier]

> So let me try again: What I find completely misguided is to move
> packages out of core *but still putting them into the release*. In other
> words, in my opinion there are really just two options that make sense:
> you either keep a package in core, or you kick it out and don't ship it
> with the release.

AFAIK I am the one who started with the idea of including ELPA packages
in the tarball, so I feel like I have to answer here: The idea was not
to push packages out to elpa.git only to then include them in
the tarball.

Instead, the idea was just to allow some packages into the tarball
without having to move them from elpa.git to emacs.git.  Their inclusion
in the tarball is simply a way to make that package more
widely available.

E.g., I'd like to include company-mode in Emacs's tarball since this
kind of completion is very popular (and also because I think Emacs as
a whole suffers from the dilution of effort between company-mode and
auto-complete, and I have the delusion that integrating one of the two
into the tarball would help solve this problem).  We could do it by
moving company-mode to emacs.git, but IIRC the maintainer of
company-mode would prefer to keep it in elpa.git, so the other option is
to bundle that ELPA package into the tarball.

I never thought of this as a way to move stuff out of emacs.git.

Yet, there is one case where I think it would be beneficial to move
a package out of emacs.git only to re-integrate it back into the
tarball: Org mode.  Org mode is really managed as a separate package
(which sadly isn't even present in elpa.git) that happens to be bundled
in the tarball: almost no code in the rest of Emacs cares about Org, and
the Org code is not tightly dependent on the specific Emacs version with
which it is shipped.
So IMO it would make more sense to keep Org outside of emacs.git (and
save us the trouble of syncing), tho we'd still want to distribute it in
Emacs's tarball, since it's an extremely popular package.


        Stefan

Re: Sync up the org in emacs master to org maint branch?

[John Wiegley]

>>>>> "DE" == David Engster <deng@randomsample.de> writes:

DE> So if you don't get convinced, we'll just move again, right? No big deal.

I suppose I'm asking that of you, yes.

DE> You are insinuating that my motivation is to delegate CEDET development to
DE> the core Emacs developers. This is simply not true, and I don't see how my
DE> original mail could be interpreted like that.

I didn't mean to insinuate anything; it seems we may have gotten off on the
wrong foot, my intention is to make your life easier, not harder. If all this
would do is make more work for people, it's not worth it.

DE> So let me try again: What I find completely misguided is to move packages
DE> out of core *but still putting them into the release*. In other words, in
DE> my opinion there are really just two options that make sense: you either
DE> keep a package in core, or you kick it out and don't ship it with the
DE> release.

Why is this so? Right now I see the Emacs release as more than just releasing
Emacs core; it's more of a "batteries included" release, combining the editor
with lots of other default packages. It makes sense to me to move these
batteries outside the core repository, than to put them all together in the
same Git repository.

We can arrange things so that a Git clone of Emacs includes pulling the
submodules (or trees, or ELPA.git, or what not) that are considered part of
"main Emacs development", including some of those batteries. I see this all as
a process issue, and that "living in one Git repository" has just been an
implementation strategy to satisfy that process.

Why do the split at all? Core becomes smaller, its future history less
cluttered, updating packages within it is no longer a major issue, and (I
hope) it will be clearer when something is a core issue vs. a package issue.
Also, people wanting to contribute new code to Emacs will not feel we're
consigning them to disuse by saying it will go in ELPA. I've seen a few
arguments already for things going into core, just to ensure more people would
use it.

DE> Say the Python developers would decide: Hey, many people like Django, so
DE> let's just put their latest git master into our release and ship it. Would
DE> you think that is a good approach?

Some companies have actually done this. I remember when ActivePython bundled
quite a few things, making it an attractive alternate to installing core
Python (back when package management was still very poor in Python world).

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Sync up the org in emacs master to org maint branch?

[Stefan Monnier]

> a process issue, and that "living in one Git repository" has just been an
> implementation strategy to satisfy that process.

While there can be good reasons to use separate repositories and/or
branches, using a single repository&branch can also make a lot of sense.
IIUC Google's in-house revision control system is basically
single-repository/single-branch and holds "the world plus the kitchen
sink" (except for some exceptions, mostly those few packages which are
also exposed to the outside world and hence use another revision control
system).
If it ain't broken, don't fix it.


        Stefan

Re: Sync up the org in emacs master to org maint branch?

[David Engster]

John Wiegley <jwiegley@gmail.com> writes:
>>>>>> "DE" == David Engster <deng@randomsample.de> writes:
>
> DE> So if you don't get convinced, we'll just move again, right? No big deal.
>
> I suppose I'm asking that of you, yes.

Sorry, but I rather wait.

> DE> You are insinuating that my motivation is to delegate CEDET development to
> DE> the core Emacs developers. This is simply not true, and I don't see how my
> DE> original mail could be interpreted like that.
>
> I didn't mean to insinuate anything; it seems we may have gotten off on the
> wrong foot, my intention is to make your life easier, not harder. If all this
> would do is make more work for people, it's not worth it.

This will most definitely make things harder for me. 

> DE> So let me try again: What I find completely misguided is to move packages
> DE> out of core *but still putting them into the release*. In other words, in
> DE> my opinion there are really just two options that make sense: you either
[+]
> DE> keep a package in core, or you kick it out and don't ship it with the
> DE> release.
>
> Why is this so? Right now I see the Emacs release as more than just releasing
> Emacs core; it's more of a "batteries included" release, combining the editor
> with lots of other default packages. It makes sense to me to move these
> batteries outside the core repository, than to put them all together in the
> same Git repository.

For package developers, keeping up with Emacs has become much harder in
recent years, as its development has accelerated (which is a good
thing). It's not like packages communicate with Emacs over a well
defined RESTful interface. In other words: CEDET and Gnus are not
loosely coupled, quite the opposite: they are extremely dependend on
many obscure Emacs internals (not sure about Org). As a consequence, we
and also the Gnus guys decided to not do separate releases anymore.  We
used to provide CEDET for different Emacs versions, and it was a *huge*
amount of work. I had a buildbot running with 7 or 8 Emacs versions to
run the test suite, and things broke pretty regularly.

Now you might say: fine, only release a package for current master. But
let's say we put CEDET into ELPA. Emacs 26 gets released, and work on
Emacs 27 starts. Now there are changes happening in Emacs 27 that
require changes in CEDET, which make it incompatible with Emacs 26. So
you have to create two packages: one for 26, one for 27? Not only is
this confusing, but it most definitely increases my workload.

> We can arrange things so that a Git clone of Emacs includes pulling the
> submodules (or trees, or ELPA.git, or what not) that are considered part of
> "main Emacs development", including some of those batteries. I see this all as
> a process issue, and that "living in one Git repository" has just been an
> implementation strategy to satisfy that process.

Obviously, I'm very skeptical towards such an approach. Our tooling
around Emacs development is already very intricate and only works
because a few people work quietly behind the scenes to keep this all
running. Introducing even more complexity through
submodules/subtrees/whatever will do the opposite of what you want to
achieve: it creates more work for those few people who manage the Emacs
infrastructure. But I'd love to hear what for instance Glenn and Paul
think about this.

> Why do the split at all? Core becomes smaller,

First off, the Emacs repo isn't that big w.r.t. the number of
files. Secondly, while there surely is an inverse correlation between
repo size and maintainability, I would argue that as long as the Big
Three are well maintained, they are not the problem. When did CEDET,
Gnus or Org ever significantly delay an Emacs release?  When was an
Emacs core developer ever forced to fix a critical thing in those
packages he did not break?  These are the questions we should be
asking. From watching this list over the past years, I don't get the
feeling that the inclusion of these packages has been a significant
burden, but I may be wrong.

> its future history less cluttered,

That's actually a bit funny, since we gave up an uncluttered history
when we switched to git's spaghetti DAG.

> updating packages within it is no longer a major issue, and (I hope)
> it will be clearer when something is a core issue vs. a package issue.

I find this whole core vs package argument strange. If you ship Emacs
with Org, Gnus and CEDET, they are part of Emacs, so it's in the
interest of all Emacs developers that they work well, whether they use
them or not. The users won't care if they originate from a separate repo
and are considered a "package". So if Paul is determined to fix all
occurences of "compatilibity" in the doc-strings, why would he only do
that for core?  People won't care if it's a CEDET doc-string, they'd
just say "Teh Emacs people cant spell!1!!". It's no big deal for him
anyway if everything is in one repo. Likewise when Stefan does some
refactoring, like renaming 'defmethod' to 'cl-defmethod'. Doing this for
all files is a matter of seconds, then commit, done. If Gnus, CEDET and
Org are separate, you have to create separate commits for them, with
their own ChangeLogs. I would argue that it is almost always *less* work
for all people involved if things get fixed right away from the right
person, and putting our built-in packages in different repos will make
this more difficult.

> DE> Say the Python developers would decide: Hey, many people like Django, so
> DE> let's just put their latest git master into our release and ship it. Would
> DE> you think that is a good approach?
>
> Some companies have actually done this. I remember when ActivePython bundled
> quite a few things, making it an attractive alternate to installing core
> Python (back when package management was still very poor in Python world).

The question is *how* did they do this. You think they just copied it
over and hoped for the best? Or maybe they did have one repo were
everything was checked in, and where they carefully tested it, maybe
even applied their own patches to Django which they couldn't or wouldn't
get upstream?  I don't know! Since it is non-free, we cannot check,
unfortunately.

-David

Re: Sync up the org in emacs master to org maint branch?

[Edward John Steere]

> It's not like packages communicate with Emacs over a well
> defined RESTful interface. In other words: CEDET and Gnus are not
> loosely coupled, quite the opposite: they are extremely dependend on
> many obscure Emacs internals (not sure about Org).

Shouldn't we be trying to avoid this situation?  It's sure to come back
and bite us in the future.  If we continue to develop a package
(wherever it ends up being developed) which is so tightly coupled that
any kind of re factoring in core becomes a nightmare, because we have to
consider the umpteen ways in which it'll break the package, then surely
that's a bad methodology to continue?  (I'm not suggesting that we
rewrite it to make it more loosely coupled, just that it seems like a
bad idea to continue allowing this going forward.)

> As a consequence, we
> and also the Gnus guys decided to not do separate releases anymore.  We
> used to provide CEDET for different Emacs versions, and it was a *huge*
> amount of work. I had a buildbot running with 7 or 8 Emacs versions to
> run the test suite, and things broke pretty regularly.
> Now you might say: fine, only release a package for current master. But
> let's say we put CEDET into ELPA. Emacs 26 gets released, and work on
> Emacs 27 starts. Now there are changes happening in Emacs 27 that
> require changes in CEDET, which make it incompatible with Emacs 26. So
> you have to create two packages: one for 26, one for 27? Not only is
> this confusing, but it most definitely increases my workload.

I feel like this problem isn't intractable.  We can mark some state of
CEDET as being stable under the various versions of Emacs (because it
was at the time) and then only support the current release for the
latest package.  This would most likely require changes to package to
ensure that you get an appropriate version of the package when you
download it.

Consider the problem which our users currently face.  The built in CEDET
is miles behind and missing very important bug fixes and features.  The
upstream CEDET can be a real pain to setup, but it has the latest
features.  This is not a good place to be.  If we merge CEDET in and
only release it with the realeses of Emacs then we are heading for a
state where you only get the new features and bug fixes every time Emacs
is realesed, i.e. our users might have to wait up to two years to have
something fixed.  This is also a bad place to be.

>> We can arrange things so that a Git clone of Emacs includes pulling the
>> submodules (or trees, or ELPA.git, or what not) that are considered part of
>> "main Emacs development", including some of those batteries. I see this all as
>> a process issue, and that "living in one Git repository" has just been an
>> implementation strategy to satisfy that process.
>
> Obviously, I'm very skeptical towards such an approach. Our tooling
> around Emacs development is already very intricate and only works
> because a few people work quietly behind the scenes to keep this all
> running. Introducing even more complexity through
> submodules/subtrees/whatever will do the opposite of what you want to
> achieve: it creates more work for those few people who manage the Emacs
> infrastructure. But I'd love to hear what for instance Glenn and Paul
> think about this.

I'm interested in exploring more with regards to how the subtree
approach would work.  In particular how it would impact the Makefiles
and build process.  I don't think that introducing features like this
necessarily increases the burden of maintaining our tooling.  If we get
it right it could reduce it.  For example we could establish some sort
of convention for building submodules/subtrees which allows us to
simplify the related Makefiles.

>> Why do the split at all? Core becomes smaller,
>
> First off, the Emacs repo isn't that big w.r.t. the number of
> files. Secondly, while there surely is an inverse correlation between
> repo size and maintainability, I would argue that as long as the Big
> Three are well maintained, they are not the problem. When did CEDET,
> Gnus or Org ever significantly delay an Emacs release?  When was an
> Emacs core developer ever forced to fix a critical thing in those
> packages he did not break?  These are the questions we should be
> asking. From watching this list over the past years, I don't get the
> feeling that the inclusion of these packages has been a significant
> burden, but I may be wrong.

I think that it's a worthwhile goal to make core smaller.  It may not be
a gigantic enterprise system with tens of thousands of source files,
like some of us are accustomed to working on, but I think that it
becomes easier to reason about the features and behaviour of core when
it's smaller.

>> updating packages within it is no longer a major issue, and (I hope)
>> it will be clearer when something is a core issue vs. a package issue.
>
> I find this whole core vs package argument strange. If you ship Emacs
> with Org, Gnus and CEDET, they are part of Emacs, so it's in the
> interest of all Emacs developers that they work well, whether they use
> them or not. The users won't care if they originate from a separate repo
> and are considered a "package".

I think that the distinction becomes clearer when you consider what
other parts of Emacs ought to be able to depend on.  If mode-x started
building dependencies on CEDET because the author discovered some useful
functions in CEDET.  Then mode-x would build a dependency which is
difficult to maintain because changes in CEDET might have unintended
effects on mode-x.  If the function really is useful, then I think
that we should instead consider extracting it from CEDET and placing it
into some core library.  As far as I can tell this has already happened
with numerous functions which originated from CEDET.

> So if Paul is determined to fix all
> occurences of "compatilibity" in the doc-strings, why would he only do
> that for core?  People won't care if it's a CEDET doc-string, they'd
> just say "Teh Emacs people cant spell!1!!". It's no big deal for him
> anyway if everything is in one repo. Likewise when Stefan does some
> refactoring, like renaming 'defmethod' to 'cl-defmethod'. Doing this for
> all files is a matter of seconds, then commit, done. If Gnus, CEDET and
> Org are separate, you have to create separate commits for them, with
> their own ChangeLogs. I would argue that it is almost always *less* work
> for all people involved if things get fixed right away from the right
> person, and putting our built-in packages in different repos will make
> this more difficult.

I agree with this point.  Definitely a downside to the separate packages
approach and one which we don't have a solution for so far.

> The question is *how* did they do this. You think they just copied it
> over and hoped for the best? Or maybe they did have one repo were
> everything was checked in, and where they carefully tested it, maybe
> even applied their own patches to Django which they couldn't or wouldn't
> get upstream?  I don't know! Since it is non-free, we cannot check,
> unfortunately.
>
> -David

I don't think that anyone is suggesting that we "copy them over and hope
for the best."  This would have to form part of the QA process.  As for
testing -- I would be surprised if CEDET in core has really received the
amount of testing it needs to declare that it's a stable release.

As to the matter of user testing in particular.  I'm fairly certain that
the majority of our users have opted to go through the pain of setting
up upstream for the reason that it includes the latest features.  I feel
that if we were to distribute it as a package then that too would
receive more user testing than core CEDET.  People tend to lose interest
in projects which stagnate.

Re: Sync up the org in emacs master to org maint branch?

[Eli Zaretskii]

> From: Edward John Steere <edward.steere@gmail.com>
> Date: Sun, 05 Feb 2017 11:03:31 +0200
> Cc: Bastien Guerry <bzg@gnu.org>, Emacs developers <emacs-devel@gnu.org>,
> 	Phillip Lord <phillip.lord@russet.org.uk>,
> 	emacs-org list <emacs-orgmode@gnu.org>,
> 	Kaushal Modi <kaushal.modi@gmail.com>
> 
> > It's not like packages communicate with Emacs over a well
> > defined RESTful interface. In other words: CEDET and Gnus are not
> > loosely coupled, quite the opposite: they are extremely dependend on
> > many obscure Emacs internals (not sure about Org).
> 
> Shouldn't we be trying to avoid this situation?  It's sure to come back
> and bite us in the future.  If we continue to develop a package
> (wherever it ends up being developed) which is so tightly coupled that
> any kind of re factoring in core becomes a nightmare, because we have to
> consider the umpteen ways in which it'll break the package, then surely
> that's a bad methodology to continue?  (I'm not suggesting that we
> rewrite it to make it more loosely coupled, just that it seems like a
> bad idea to continue allowing this going forward.)

How would you go about not allowing this to go forward?  I don't
think I see any practical way to do that; do you?

IMO, this is up to the package developers: if they want to depend less
on the Emacs internals, and thus be more loosely coupled with a
particular Emacs version, they will need to invest the extra effort
for that, e.g., by providing some compatibility shims.

IOW, this isn't an issue that can be solved once and for all by the
way we develop the core or the packages, or by the way we integrate
packages with core, the solution is on a different level.

> > As a consequence, we
> > and also the Gnus guys decided to not do separate releases anymore.  We
> > used to provide CEDET for different Emacs versions, and it was a *huge*
> > amount of work. I had a buildbot running with 7 or 8 Emacs versions to
> > run the test suite, and things broke pretty regularly.
> > Now you might say: fine, only release a package for current master. But
> > let's say we put CEDET into ELPA. Emacs 26 gets released, and work on
> > Emacs 27 starts. Now there are changes happening in Emacs 27 that
> > require changes in CEDET, which make it incompatible with Emacs 26. So
> > you have to create two packages: one for 26, one for 27? Not only is
> > this confusing, but it most definitely increases my workload.
> 
> I feel like this problem isn't intractable.  We can mark some state of
> CEDET as being stable under the various versions of Emacs (because it
> was at the time) and then only support the current release for the
> latest package.  This would most likely require changes to package to
> ensure that you get an appropriate version of the package when you
> download it.

IF (and its a big "if") package developers want to be able to target
more than the single Emacs version on a single branch of the Emacs
repo, then they will need to provide at least 3 branches:

  . "development" branch that tracks the Emacs master branch and
    introduces exciting new features
  . "bugfix" branch that tracks the Emacs release branch without
    adding any new features
  . "stable" branch that is compatible with the Emacs release branch,
    but also introduces some new features, the ones that don't need
    core developments available only on the Emacs master branch

I envision that some packages will want the above (or maybe even a
more elaborate scheme), because they can afford that, and because
their users expect that.  These are mostly those packages whose
developers welcome the move to put more of Emacs on ELPA.  Other
packages, and I guess CEDET is one of them, will not want to do this,
because it adds too much work to an already complicated and
time-consuming job.

IOW, once again the solution is not part of the way we develop the
core or integrate non-core packages, it's elsewhere.

Bottom line, I think people are bothered by aspects of the "move to
ELPA" process that are not supposed to be affected by that move in any
way.  They are aspects that need to be resolved on entirely different
levels, and the resolution is up to the package maintainers.  That
includes a possible decision of the developers that some package will
not move to ELPA; I don't think that if a package developers say they
want to stay in emacs.git, they will be told to get out regardless.

Re: Sync up the org in emacs master to org maint branch?

[David Engster]

Edward John Steere writes:
>> It's not like packages communicate with Emacs over a well
>> defined RESTful interface. In other words: CEDET and Gnus are not
>> loosely coupled, quite the opposite: they are extremely dependend on
>> many obscure Emacs internals (not sure about Org).
>
> Shouldn't we be trying to avoid this situation?  It's sure to come back
> and bite us in the future.  If we continue to develop a package
> (wherever it ends up being developed) which is so tightly coupled that
> any kind of re factoring in core becomes a nightmare, because we have to
> consider the umpteen ways in which it'll break the package, then surely
> that's a bad methodology to continue?

I don't know what you have in mind. All I can say is: CEDET couldn't do
what it does if we'd restrict ourselves to some subset of Emacs.

>> As a consequence, we and also the Gnus guys decided to not do
>> separate releases anymore.  We used to provide CEDET for different
>> Emacs versions, and it was a *huge* amount of work. I had a buildbot
>> running with 7 or 8 Emacs versions to run the test suite, and things
>> broke pretty regularly.  Now you might say: fine, only release a
>> package for current master. But let's say we put CEDET into
>> ELPA. Emacs 26 gets released, and work on Emacs 27 starts. Now there
>> are changes happening in Emacs 27 that require changes in CEDET,
>> which make it incompatible with Emacs 26. So you have to create two
>> packages: one for 26, one for 27? Not only is this confusing, but it
>> most definitely increases my workload.
>
> I feel like this problem isn't intractable.

I didn't say it's intractable. I just said it means more work for me.

> We can mark some state of CEDET as being stable under the various
> versions of Emacs (because it was at the time) and then only support
> the current release for the latest package.  This would most likely
> require changes to package to ensure that you get an appropriate
> version of the package when you download it.

As I said: we did that. It was a huge amount of work. Please understand
where I'm coming from: if you look through the CEDET history, you will
see that in the past few years I almost exclusively did infrastructure
work and no real coding. All I can say is: *I* won't do this anymore,
and I don't want to be part in something which will increase grunt
work. We did not make this decision lightly. But with the amount of
developers we have, I'm convinced it's the right thing to do.

> Consider the problem which our users currently face.  The built in CEDET
> is miles behind and missing very important bug fixes and features.  The
> upstream CEDET can be a real pain to setup, but it has the latest
> features.  This is not a good place to be.  If we merge CEDET in and
> only release it with the realeses of Emacs then we are heading for a
> state where you only get the new features and bug fixes every time Emacs
> is realesed, i.e. our users might have to wait up to two years to have
> something fixed.  This is also a bad place to be.

Bug fixes can go with point release, which we should have every
year. But yes, if you want the latest, greatest and buggiest, you'll
have to use Emacs master. But that goes for a lot of Emacs features, so
I don't see why it's particularly bad for CEDET.

>>> We can arrange things so that a Git clone of Emacs includes pulling the
>>> submodules (or trees, or ELPA.git, or what not) that are considered part of
>>> "main Emacs development", including some of those batteries. I see this all as
>>> a process issue, and that "living in one Git repository" has just been an
>>> implementation strategy to satisfy that process.
>>
>> Obviously, I'm very skeptical towards such an approach. Our tooling
>> around Emacs development is already very intricate and only works
>> because a few people work quietly behind the scenes to keep this all
>> running. Introducing even more complexity through
>> submodules/subtrees/whatever will do the opposite of what you want to
>> achieve: it creates more work for those few people who manage the Emacs
>> infrastructure. But I'd love to hear what for instance Glenn and Paul
>> think about this.
>
> I'm interested in exploring more with regards to how the subtree
> approach would work.  In particular how it would impact the Makefiles
> and build process.  I don't think that introducing features like this
> necessarily increases the burden of maintaining our tooling.  If we get
> it right it could reduce it.

Well, we cannot really discuss this since there's no real plan how this
all should work. I can only speak from experience.

> I think that it's a worthwhile goal to make core smaller.  It may not be
> a gigantic enterprise system with tens of thousands of source files,
> like some of us are accustomed to working on, but I think that it
> becomes easier to reason about the features and behaviour of core when
> it's smaller.

How does CEDET, Gnus and Org affect the rest of Emacs? They strongly
depend on Emacs "core" (whatever exactly that is), not the other way
round.

>>> updating packages within it is no longer a major issue, and (I hope)
>>> it will be clearer when something is a core issue vs. a package issue.
>>
>> I find this whole core vs package argument strange. If you ship Emacs
>> with Org, Gnus and CEDET, they are part of Emacs, so it's in the
>> interest of all Emacs developers that they work well, whether they use
>> them or not. The users won't care if they originate from a separate repo
>> and are considered a "package".
>
> I think that the distinction becomes clearer when you consider what
> other parts of Emacs ought to be able to depend on.  If mode-x started
> building dependencies on CEDET because the author discovered some useful
> functions in CEDET.  Then mode-x would build a dependency which is
> difficult to maintain because changes in CEDET might have unintended
> effects on mode-x.  If the function really is useful, then I think
> that we should instead consider extracting it from CEDET and placing it
> into some core library.  As far as I can tell this has already happened
> with numerous functions which originated from CEDET.

Aside from EIEIO, I wouldn't know any. And with EIEIO it had exactly the
opposite effect: CEDET has to adapt, not the other way round.

> I don't think that anyone is suggesting that we "copy them over and hope
> for the best."  This would have to form part of the QA process.  As for
> testing -- I would be surprised if CEDET in core has really received the
> amount of testing it needs to declare that it's a stable release.

Well, that's precisely why I want to move development to the Emacs
repository.

-David

Re: Sync up the org in emacs master to org maint branch?

[Edward John Steere]

>>> they are extremely dependend on
>>> many obscure Emacs internals (not sure about Org).
>>
>> Shouldn't we be trying to avoid this situation?  It's sure to come back
>> and bite us in the future.  If we continue to develop a package
>> (wherever it ends up being developed) which is so tightly coupled that
>> any kind of re factoring in core becomes a nightmare, because we have to
>> consider the umpteen ways in which it'll break the package, then surely
>> that's a bad methodology to continue?
>
> I don't know what you have in mind. All I can say is: CEDET couldn't do
> what it does if we'd restrict ourselves to some subset of Emacs.

In particular I was worried by the phrasing "extremely dependent".  I
agree that in order to make a system like CEDET without re-inventing the
wheel and without running into performance problems it's necessary to
depend on more primitive parts of Emacs.  Perhaps we can improve the
relationship(?)  Perhaps this is a discussion to be had when all of this
is done though.

>> I feel like this problem isn't intractable.
>
> I didn't say it's intractable. I just said it means more work for me.
>
>> We can mark some state of CEDET as being stable under the various
>> versions of Emacs (because it was at the time) and then only support
>> the current release for the latest package.  This would most likely
>> require changes to package to ensure that you get an appropriate
>> version of the package when you download it.
>
> As I said: we did that. It was a huge amount of work. Please understand
> where I'm coming from: if you look through the CEDET history, you will
> see that in the past few years I almost exclusively did infrastructure
> work and no real coding. All I can say is: *I* won't do this anymore,
> and I don't want to be part in something which will increase grunt
> work. We did not make this decision lightly. But with the amount of
> developers we have, I'm convinced it's the right thing to do.

Fair enough.  I don't have the experience here.  It just seems like we
could meet both goals without increasing the work load in this regard.
To be clear I agree that, whatever decision this discussion arrives at,
we definitely don't want to we waste the time of any developer with
grunt work.

Continuing this line of thought.  I'm not sure that we're thinking of
the same process here.  What I'm suggesting is as follows:

 - Suppose that Emacs 22.0 is the current release and Emacs 22.1 is then
   released; CEDET is at <some-hash-from-master-on-git>
 - we update a registry somewhere indicating that Emacs 22.0 works with
   <some-hash-from-master-on-git> and 22.1 works with
   <some-hash-from-master-on-git>
 - When we make updates to CEDET we mark 22.1 as working with
   <some-new-hash-from-master-on-git> but we don't change that reference
   for 22.0 (or any older versions)
 - When someone complains that there's a bug in CEDET for 22.0 we
   indicate that it's no longer supported and that they should update
   Emacs to receive updates

This process would almost be the same as what you get just by bundling
CEDET with Emacs except that:

 - You can get the latest CEDET *if* you have the latest Emacs
 - The version of CEDET for any particular version of Emacs is as far as
   CEDET got before the next release of Emacs came out

If this is what you were thinking of then please could you elaborate on
what ended up being the problem which added more work.

Also, would this be a problem for our users?  i.e. would we be inundated
with emails requesting continued support on older versions or would
users generally accept this kind of change.  I mostly work on back end
systems so I don't have a good feeling for how this would go down with
users (I would find this reasonable as a user).

> Bug fixes can go with point release, which we should have every
> year. But yes, if you want the latest, greatest and buggiest, you'll
> have to use Emacs master. But that goes for a lot of Emacs features, so
> I don't see why it's particularly bad for CEDET.

I feel like there are aspects of CEDET which are still under
development.  Perhaps I'm just unlucky in my particular usage patterns
of upstream and the way things are going this will be fine (with the
un-merged parts going into packages.)

>> I'm interested in exploring more with regards to how the subtree
>> approach would work.  In particular how it would impact the Makefiles
>> and build process.  I don't think that introducing features like this
>> necessarily increases the burden of maintaining our tooling.  If we get
>> it right it could reduce it.
>
> Well, we cannot really discuss this since there's no real plan how this
> all should work. I can only speak from experience.

We can still put ideas forward though.  Haven't come up with anything
myself yet though.

>> I think that it's a worthwhile goal to make core smaller.  It may not be
>> a gigantic enterprise system with tens of thousands of source files,
>> like some of us are accustomed to working on, but I think that it
>> becomes easier to reason about the features and behaviour of core when
>> it's smaller.
>
> How does CEDET, Gnus and Org affect the rest of Emacs? They strongly
> depend on Emacs "core" (whatever exactly that is), not the other way
> round.

I believe that one of the intentions of the move is to enforce this so
we can't build bad dependencies -- am I wrong?

>> I think that the distinction becomes clearer when you consider what
>> other parts of Emacs ought to be able to depend on.  If mode-x started
>> building dependencies on CEDET because the author discovered some useful
>> functions in CEDET.  Then mode-x would build a dependency which is
>> difficult to maintain because changes in CEDET might have unintended
>> effects on mode-x.  If the function really is useful, then I think
>> that we should instead consider extracting it from CEDET and placing it
>> into some core library.  As far as I can tell this has already happened
>> with numerous functions which originated from CEDET.
>
> Aside from EIEIO, I wouldn't know any. And with EIEIO it had exactly the
> opposite effect: CEDET has to adapt, not the other way round.

I think that we missed each other here.  I was saying that moving EIEIO
out of CEDET and giving it a home where other packages could depend on
it was a good thing and that CEDET has to adapt to changes in EIEIO is
the way it should be.  I can think of one other function off the top of
my head: `locate-dominating-file'.  I believe that it came from EDE.

>> I don't think that anyone is suggesting that we "copy them over and hope
>> for the best."  This would have to form part of the QA process.  As for
>> testing -- I would be surprised if CEDET in core has really received the
>> amount of testing it needs to declare that it's a stable release.
>
> Well, that's precisely why I want to move development to the Emacs
> repository.

Perhaps I'm wrong, but to my mind the package approach would afford us
with more testing before we get to the point of another release of
Emacs.

Kind regards,

Edward Steere

Re: Sync up the org in emacs master to org maint branch?

[David Engster]

Edward John Steere writes:
>  - Suppose that Emacs 22.0 is the current release and Emacs 22.1 is then
>    released; CEDET is at <some-hash-from-master-on-git>
>  - we update a registry somewhere indicating that Emacs 22.0 works with
>    <some-hash-from-master-on-git> and 22.1 works with
>    <some-hash-from-master-on-git>
>  - When we make updates to CEDET we mark 22.1 as working with
>    <some-new-hash-from-master-on-git> but we don't change that reference
>    for 22.0 (or any older versions)
>  - When someone complains that there's a bug in CEDET for 22.0 we
>    indicate that it's no longer supported and that they should update
>    Emacs to receive updates
>
> This process would almost be the same as what you get just by bundling
> CEDET with Emacs except that:
>
>  - You can get the latest CEDET *if* you have the latest Emacs

No. We have two branches: emacs-25 and master. The CEDET from master
will usually not work on any 25.x version.

>  - The version of CEDET for any particular version of Emacs is as far as
>    CEDET got before the next release of Emacs came out
>
> If this is what you were thinking of then please could you elaborate on
> what ended up being the problem which added more work.

First off, CEDET is currently *not* a package, although that notion gets
thrown around a lot. It is scattered across the Emacs code base:
lisp/cedet, admin/grammars, etc/srecode, documentation, and test
suite. All this needs to be packaged in a way so that it can be
integrated into Emacs during a normal checkout. It needs to build and
test in such a normal checkout, but also separately when installed from
ELPA, including grammar compilation. And you need this twice: one for
emacs-25, one for master, with the possiblity to merge between the two.

Then there's this "registry". No one has said how that should
work. "Submodules/Subtrees" are *not* a sufficient answer, they are just
tools. People will need to say how the *workflow* should be, including
such things like merges from stable, ChangeLog generation, AUTHORS,
NEWS, creating release tarballs, and so on. Once someone has written
this down *in detail*, we can discuss again if this indeed will make
things simpler and reduce our workload.

> I feel like there are aspects of CEDET which are still under
> development.

I hope so.

>> Well, we cannot really discuss this since there's no real plan how this
>> all should work. I can only speak from experience.
>
> We can still put ideas forward though.  Haven't come up with anything
> myself yet though.

Yes, you can, but it has a cost. Once again, the CEDET merge is stalled,
and we spend our time writing mails. I find this situation incredibly
frustrating.

>> How does CEDET, Gnus and Org affect the rest of Emacs? They strongly
>> depend on Emacs "core" (whatever exactly that is), not the other way
>> round.
>
> I believe that one of the intentions of the move is to enforce this so
> we can't build bad dependencies -- am I wrong?

I think other modes should use Semantic.

> Perhaps I'm wrong, but to my mind the package approach would afford us
> with more testing before we get to the point of another release of
> Emacs.

If you develop on Emacs 'master', you can use all the new features (like
threading and FFI), but you loose testers on 25.x. A package won't
change that.

-David

Re: Sync up the org in emacs master to org maint branch?

[Edward John Steere]

>>  - Suppose that Emacs 22.0 is the current release and Emacs 22.1 is then
>>    released; CEDET is at <some-hash-from-master-on-git>
>>  - we update a registry somewhere indicating that Emacs 22.0 works with
>>    <some-hash-from-master-on-git> and 22.1 works with
>>    <some-hash-from-master-on-git>
>>  - When we make updates to CEDET we mark 22.1 as working with
>>    <some-new-hash-from-master-on-git> but we don't change that reference
>>    for 22.0 (or any older versions)
>>  - When someone complains that there's a bug in CEDET for 22.0 we
>>    indicate that it's no longer supported and that they should update
>>    Emacs to receive updates
>>
>> This process would almost be the same as what you get just by bundling
>> CEDET with Emacs except that:
>>
>>  - You can get the latest CEDET *if* you have the latest Emacs
>
> No. We have two branches: emacs-25 and master. The CEDET from master
> will usually not work on any 25.x version.

In the process which I proposed we would be developing against the most
recently released Emacs.  I now see that there's a trade off.  If we
were to go with my scheme then users would have access to the latest
version, but we would:
 1. miss out on the new features being developed on the Emacs master
    branch (which CEDET stands to gain a lot from);
 2. have to endure a difficult and error prone release process every time
    Emacs is released because that's not what all the testing is done
    against;

If we go ahead and merge it in then we would be able to benefit from all
the new Emacs features and the release would be less painful.  Of course
our users would still have to wait for the latest features, but at least
it would be worth the wait because we would be able to consider using
features like threading.

>>  - The version of CEDET for any particular version of Emacs is as far as
>>    CEDET got before the next release of Emacs came out
>>
>> If this is what you were thinking of then please could you elaborate on
>> what ended up being the problem which added more work.
>
> First off, CEDET is currently *not* a package, although that notion gets
> thrown around a lot. It is scattered across the Emacs code base:
> lisp/cedet, admin/grammars, etc/srecode, documentation, and test
> suite. All this needs to be packaged in a way so that it can be
> integrated into Emacs during a normal checkout. It needs to build and
> test in such a normal checkout, but also separately when installed from
> ELPA, including grammar compilation. And you need this twice: one for
> emacs-25, one for master, with the possiblity to merge between the two.

Yes, this would be a pain.  I'm in favour of Phillip's approach in which
the packages are self contained somewhere in the Emacs source tree.
This would eliminate the need for a complicated copying process and
eliminate the problem of where things go and what files should be
updated.  No other folder would be touched because a package contains
it's own source, documentation etc.  Unfortunately the idea isn't
gaining a lot of traction.  I'm not satisfied that the alternative would
make things easier for anyone because, as you say, it would need to
describe a complex copying process which intertwines CEDET with various
parts of Emacs.

> Then there's this "registry". No one has said how that should
> work. "Submodules/Subtrees" are *not* a sufficient answer, they are just
> tools. People will need to say how the *workflow* should be, including
> such things like merges from stable, ChangeLog generation, AUTHORS,
> NEWS, creating release tarballs, and so on. Once someone has written
> this down *in detail*, we can discuss again if this indeed will make
> things simpler and reduce our workload.

I haven't heard the registry mentioned before.  I mentioned it as a
detail required by my process for supporting multiple versions, but I'm
beginning to think that it's the wrong line of thought to pursue.

>>> Well, we cannot really discuss this since there's no real plan how this
>>> all should work. I can only speak from experience.
>>
>> We can still put ideas forward though.  Haven't come up with anything
>> myself yet though.
>
> Yes, you can, but it has a cost. Once again, the CEDET merge is stalled,
> and we spend our time writing mails. I find this situation incredibly
> frustrating.

In lieu of any input from others the best we have is Phillip's idea of
using Elpa.  That solves how the files are copied in and compiled.

>>> How does CEDET, Gnus and Org affect the rest of Emacs? They strongly
>>> depend on Emacs "core" (whatever exactly that is), not the other way
>>> round.
>>
>> I believe that one of the intentions of the move is to enforce this so
>> we can't build bad dependencies -- am I wrong?
>
> I think other modes should use Semantic.

Agreed.  I mentioned this in my response to Stefan's email.  Shouldn't
we then consider moving it out of CEDET in Emacs?

>> Perhaps I'm wrong, but to my mind the package approach would afford us
>> with more testing before we get to the point of another release of
>> Emacs.
>
> If you develop on Emacs 'master', you can use all the new features (like
> threading and FFI), but you loose testers on 25.x. A package won't
> change that.

Yes.  As I said earlier in this email I'm now starting to appreciate
this problem.  Perhaps it's worth considering which aspects of CEDET
would really benefit from being integrated with Emacs and which could
live without the latest features (and be turned into packages).

There are four parts to CEDET which I'm aware of (at least in the CEDET
which is included in Emacs):
 1. Semantic
 2. Semanticdb
 3. EDE
 4. Srecode

I think that the key parts are Semantic and Semanticdb.  What if we
merged Semantic and Semanticdb into Emacs proper (i.e. not under a
folder named cedet in the source tree) and worked on turning EDE and
Srecode into packages which are included for distribution?

The only obstacle I can think of is that there are some features in
Semantic (ia and db) which rely on project variables (such as the
location of system includes).  We could consider decoupling Semantic to
solve this problem.  e.g. by providing an interface to which other modes
can publish known information about the context of a source file in a
project.

Re: Sync up the org in emacs master to org maint branch?

[John Wiegley]

>>>>> "DE" == David Engster <deng@randomsample.de> writes:

DE> I find this whole core vs package argument strange. If you ship Emacs with
DE> Org, Gnus and CEDET, they are part of Emacs, so it's in the interest of
DE> all Emacs developers that they work well, whether they use them or not.
DE> The users won't care if they originate from a separate repo and are
DE> considered a "package". So if Paul is determined to fix all occurences of
DE> "compatilibity" in the doc-strings, why would he only do that for core?

If pulling Emacs.git also pulls ELPA.git (as it should, since it's within our
purview as well), then global search&replace should still affect all the
files.

I hear your other points, so I'm curious now as to what more people think
about this who work on Emacs core: Do you want more modularity with regard
ELPA, or does the monolithic model work better for you?

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Sync up the org in emacs master to org maint branch?

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Date: Sat, 11 Feb 2017 18:46:09 -0800
> Cc: Bastien Guerry <bzg@gnu.org>, Emacs developers <emacs-devel@gnu.org>,
> 	Phillip Lord <phillip.lord@russet.org.uk>,
> 	emacs-org list <emacs-orgmode@gnu.org>,
> 	Kaushal Modi <kaushal.modi@gmail.com>
> 
> I hear your other points, so I'm curious now as to what more people think
> about this who work on Emacs core: Do you want more modularity with regard
> ELPA, or does the monolithic model work better for you?

AFAIU, the main motivation for the "drive to ELPA" comes from
developers of individual packages, not from those working on the core
(even though some package developers also work on core).

Re: Sync up the org in emacs master to org maint branch?

[Stefan Monnier]

> AFAIU, the main motivation for the "drive to ELPA" comes from
> developers of individual packages, not from those working on the core

Not sure what you mean exactly by "drive to ELPA".  If you mean "drive
to put packages in GNU ELPA rather than in core", then this drive is
linked to various aspects:
- we can put packages in elpa.git without worrying too much
  about whether or not they are "good/important enough", contrary to
  packages in core.
- packages in elpa.git don't come with the same kind of implicit
  guarantee of compatibility, availability, and maintenance as those in
  core.
- sync'ing with an external upstream is easier in elpa.git since
  there's no need to pay attention to Emacs release schedule.
- sync'ing with an external upstream can be made technically easier
  because we can use a separate branch for the package.


        Stefan

Re: Sync up the org in emacs master to org maint branch?

[Stefan Monnier]

> It is a mistake because you are creating more moving targets and bring
> them together very late in the release process. This reduces the amount
> of testing that is done for those packages, so bugs will be noticed
> later and the quality of the relases suffer. It moves even more work
> into the RC-phase, which is already crowded and where people who can fix
> those bugs might not be readily available. It removes those packages
> from Emacs CI, so that breakages due to changes in core are not
> immediately noticed, and often times they have to be fixed not by those
> who created the breakage, but by those who notice them.

Indeed, moving something to elpa.git has benefits but also downsides.

Another downside is that ELPA packages need to be compatible with
various Emacs releases, whereas bundled code can happily ignore those
compatibility issues.  Of course, the reverse is true as well: if
CEDET-core is an ELPA package, then CEDET-client packages don't need to
be compatible with older CEDET-core.

I think the most important issue is to avoid duplicating branches.
Moving (parts of) CEDET to elpa.git is/was mostly motivated by the
desire to avoid the difficulties of sync'ing between the Emacs code and
the upstream code, since the elpa.git branch could hopefully evolve
in-sync with the upstream (or could even *be* the uptream), whereas that
was more difficult for the code in emacs.git.

If the CEDET code in emacs.git becomes "the upstream", then moving it to
elpa.git is much less interesting.


        Stefan

Re: Sync up the org in emacs master to org maint branch?

[Phillip Lord]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> "KM" == Kaushal Modi <kaushal.modi@gmail.com> writes:
>
> KM> If we are able the release the new packaging method in emacs 26.x, then we
> KM> can remove org from emacs master completely, but if not, then at least as
> KM> backup we have a newer org version to go out with that release.
>
> For Emacs 26, I intend the new ELPA process to be in place, whereby "default"
> packages can be developed separately, and declare a way to get slip-streamed
> into the release tarball so users are unaware of the separate nature of their
> development.
>
> The CEDET developers have agreed to support this, and it sounds like you are
> willing to as well. If Lars is game, I'd like for Gnus to be third major
> package we do this for initially. That will reduce considerably the number of
> external files we track in Emacs.git.
>
> The precise technical details have yet to be worked out, but it shouldn't be
> too difficult. Phillip Lord has already began advance work on alternatives,
> and I've received offers of help from others to work on this new process.
>
> I think now is a good time to begin. The first step is to solidify what is
> meant by "tarball EPLA", and the means of slip-streaming a package's contents.
> This will require at least two bits:
>
>   - Some form of declaration to indicate how external files should appear in
>     the tarball. In order for the first version of this scheme to be as low
>     impact as possible, this should probably be done with a sexp in a data
>     file, to be checked in alongside the EPLA.git import of the project.
>
>   - changes to "make dist" to integrate these files, and setup autoloading so
>     their inclusion is transparent to end users.
>
> Please comment with your recommendations for the first, and supporting changes
> for the second, if anyone has ideas. Phillip, how is your work on these coming
> along?

At the moment it isn't. My original plan, if you remember, to have emacs
core load ELPA packages with package.el. This requires some minor
re-working of package.el (so that the -Q doesn't block them), some
Makefile fiddling and introducing some standards to test file location
in ELPA, so that it's possible to run ELPA tests from within core.

The alternative proposal is, essentially, to copy files into the Emacs
core build structure and move from there.

Reading the discussion reinforces my feeling that the latter is the
wrong approach, because it reinforces a distinction between packages on
ELPA and packages in core above and beyond the location that they are
stored and versioned. I can't see the advantage of doing this.

I will probably try to work a little further on my package.el solution,
although there seems little enthusiasm for this as the way forward.

Phil

Re: Sync up the org in emacs master to org maint branch?

[John Wiegley]

>>>>> "PL" == Phillip Lord <phillip.lord@russet.org.uk> writes:

PL> The alternative proposal is, essentially, to copy files into the Emacs
PL> core build structure and move from there.

PL> Reading the discussion reinforces my feeling that the latter is the wrong
PL> approach, because it reinforces a distinction between packages on ELPA and
PL> packages in core above and beyond the location that they are stored and
PL> versioned. I can't see the advantage of doing this.

PL> I will probably try to work a little further on my package.el solution,
PL> although there seems little enthusiasm for this as the way forward.

Correct, I don't want to move forward with the package.el option yet. Just
copying files into the core build structure should be very lightweight, and it
lets us focus on the ELPA separation first. I believe the package.el coupling
is something we can examine at a later date.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2