Git merge tool for Org files

Git merge tool for Org files

[Carsten Dominik]

Hi,

I was wondering if anyone here has the skills and interest to write
a git merge driver for Org mode files, in the way

   http://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=lib/git-merge-changelog.c

does this for GNU style ChangeLog files?

One of the obstacle for using Org-mode for collaborative programs is that
many operations add or remove text from the beginning or end of a file,
or which add or remove a subnode from an outline tree.
These are operations that confuse the git merger, in particular
if two people have added something to a file, or removed/added
sequential sibling nodes.

I believe that this could be solved with a dedicated merge driver
that understands the integrity of an outline (sub)tree, and that
knows that the sequence of two new subtrees added by different people
does not matter.

The idea for this is actually (I believe) from Brian Gough who asked me
about it at FOSDEM.  I have now also run into this problem and would
really find it great if such a merge driver could be written.
Unfortunately, I do not have the skill for this.

- Carsten

Re: Git merge tool for Org files

[Andrea Crotti]

On 08/18/2011 12:45 PM, Carsten Dominik wrote:
> Hi,
>
> I was wondering if anyone here has the skills and interest to write
> a git merge driver for Org mode files, in the way
>
>     http://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=lib/git-merge-changelog.c
>
> does this for GNU style ChangeLog files?
>
> One of the obstacle for using Org-mode for collaborative programs is that
> many operations add or remove text from the beginning or end of a file,
> or which add or remove a subnode from an outline tree.
> These are operations that confuse the git merger, in particular
> if two people have added something to a file, or removed/added
> sequential sibling nodes.
>
> I believe that this could be solved with a dedicated merge driver
> that understands the integrity of an outline (sub)tree, and that
> knows that the sequence of two new subtrees added by different people
> does not matter.
>
> The idea for this is actually (I believe) from Brian Gough who asked me
> about it at FOSDEM.  I have now also run into this problem and would
> really find it great if such a merge driver could be written.
> Unfortunately, I do not have the skill for this.
>
> - Carsten
Does the merge driver need to be written in C?
It looks quite a complex task considering that org-mode files
are not so easy to parse...

Anyway a partly unrelated thing, with git attributes I'm able
to see the diff of crypted org files very easily (maybe someone
is interested), adding to .git/config:

[diff "gpg"]
       textconv = gpg2 --decrypt

and to .gitattributes:
*.gpg diff=gpg

Re: Git merge tool for Org files

[Carsten Dominik]

On 20.8.2011, at 16:17, Andrea Crotti wrote:

> On 08/18/2011 12:45 PM, Carsten Dominik wrote:
>> Hi,
>> 
>> I was wondering if anyone here has the skills and interest to write
>> a git merge driver for Org mode files, in the way
>> 
>>    http://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=lib/git-merge-changelog.c
>> 
>> does this for GNU style ChangeLog files?
>> 
>> One of the obstacle for using Org-mode for collaborative programs is that
>> many operations add or remove text from the beginning or end of a file,
>> or which add or remove a subnode from an outline tree.
>> These are operations that confuse the git merger, in particular
>> if two people have added something to a file, or removed/added
>> sequential sibling nodes.
>> 
>> I believe that this could be solved with a dedicated merge driver
>> that understands the integrity of an outline (sub)tree, and that
>> knows that the sequence of two new subtrees added by different people
>> does not matter.
>> 
>> The idea for this is actually (I believe) from Brian Gough who asked me
>> about it at FOSDEM.  I have now also run into this problem and would
>> really find it great if such a merge driver could be written.
>> Unfortunately, I do not have the skill for this.
>> 
>> - Carsten
> Does the merge driver need to be written in C?

I have no idea, but I would think that it could also be written in Perl,
for example.

> It looks quite a complex task considering that org-mode files
> are not so easy to parse...
> 
> Anyway a partly unrelated thing, with git attributes I'm able
> to see the diff of crypted org files very easily (maybe someone
> is interested), adding to .git/config:
> 
> [diff "gpg"]
>      textconv = gpg2 --decrypt
> 
> and to .gitattributes:
> *.gpg diff=gpg

interesting!

- Carsten

Re: Git merge tool for Org files

[Andrea Crotti]

On 08/20/2011 04:42 PM, Carsten Dominik wrote:
> I have no idea, but I would think that it could also be written in Perl,
> for example.
>

Yes from what I understand from the source it can be any executable
that outputs things as git expects them.
There are no compile time dependencies on git as far as I see.

So maybe why not writing it (even a prototype) directly in Elisp?
That would solve many things right?

Re: Git merge tool for Org files

[Nick Dokos]

Andrea Crotti <andrea.crotti.0@gmail.com> wrote:

> On 08/18/2011 12:45 PM, Carsten Dominik wrote:
> > Hi,
> >
> > I was wondering if anyone here has the skills and interest to write
> > a git merge driver for Org mode files, in the way
> >
> >     http://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=lib/git-merge-changelog.c
> >
> > does this for GNU style ChangeLog files?
> >
> > One of the obstacle for using Org-mode for collaborative programs is that
> > many operations add or remove text from the beginning or end of a file,
> > or which add or remove a subnode from an outline tree.
> > These are operations that confuse the git merger, in particular
> > if two people have added something to a file, or removed/added
> > sequential sibling nodes.
> >
> > I believe that this could be solved with a dedicated merge driver
> > that understands the integrity of an outline (sub)tree, and that
> > knows that the sequence of two new subtrees added by different people
> > does not matter.
> >
> > The idea for this is actually (I believe) from Brian Gough who asked me
> > about it at FOSDEM.  I have now also run into this problem and would
> > really find it great if such a merge driver could be written.
> > Unfortunately, I do not have the skill for this.
> >
> > - Carsten
> Does the merge driver need to be written in C?
> It looks quite a complex task considering that org-mode files
> are not so easy to parse...
> 

and

> On 08/20/2011 04:42 PM, Carsten Dominik wrote:
> > I have no idea, but I would think that it could also be written in Perl,
> > for example.
> >
> 
> Yes from what I understand from the source it can be any executable
> that outputs things as git expects them.
> There are no compile time dependencies on git as far as I see.
> 
> So maybe why not writing it (even a prototype) directly in Elisp?
> That would solve many things right?
> 
> 

Perhaps a better way is to not insist on automatic conflict resolution
(changelogs are relatively simple, whereas org files would be more
complicated: one would have to impose lots of rules to make sure that
the merge would be successful). A relatively painless manual,
interactive merge might fit the bill better.

In his book "Pro Git"[fn:1], Scott Chacon describes how to hook an
external merge tool into git (he uses the -free- Perforce Visual Merge
Tool for illustration, but it should be possible to use anything).  And
emacs has the emerge tool already built in.  In fact, Chacon mentions
emerge as one of the presets in git, so it may be even simpler than what
he describes.

This is all theory and we all know that [fn:2] "In theory, there is no
difference between theory and practice - in practice, there is."  But it
seems worth investigating this a bit, before striking off into an
unknown unknown. [fn:3]

Nick

Footnotes:

[fn:1] Available on the web at http://progit.org/book, as well as on
paper (see the website for details). Section 7.1 is the relevant part.

[fn:2] ...in the words of that immortal philosopher, Yogi Berra, ...

[fn:3] OK, ok - Donald Rumsfeld said the "unknown unknown" thing first.

Re: Git merge tool for Org files

[suvayu ali]

Hello everyone,

On Sat, Aug 20, 2011 at 6:04 PM, Nick Dokos <nicholas.dokos@hp.com> wrote:
>  And emacs has the emerge tool already built in.  In fact, Chacon
> mentions emerge as one of the presets in git, so it may be even
> simpler than what he describes.

When there is a conflict, this should work:

$ git mergetool --tool=emerge

This pops out an emacs window for every conflicting file. But I believe
it is possible to specify a custom mergetool rather easily using `emacs
-nw` or `emacsclient -t`.

Excerpt from `man git-config`:

  merge.tool
      Controls which merge resolution program is used by
      git-mergetool(1). Valid built-in values are: "kdiff3", "tkdiff",
      "meld", "xxdiff", "emerge", "vimdiff", "gvimdiff", "diffuse",
      "ecmerge", "tortoisemerge", "p4merge", "araxis" and "opendiff".
      Any other value is treated is custom merge tool and there must be
      a corresponding mergetool.<tool>.cmd option.


However what I have found is it is rather difficult to merge blocks of
text reasonably trivially. Often diff (or git diff) is thrown off when:

1. There is a TODO/tag change very close to some other change in text
2. A headline is promoted or demoted without any change in the content
3. There is minor change in a body of text but line wrapping makes the
   diff very difficult to follow as it lists many more changes.

I think the problem is all tools like patch/diff are made for source
code. So they understand source code structures like code blocks {..} in
most languages or methods/functions like someFunc() {..} in C/C++ or
(defun somefn () ...) in lisp. Where as plain text / Org text is
structured more as paragraphs and trees, line wrapping is usually
inconsequential. So a dedicated git driver to merge org files might be
worth it. It could automatically take care of differences solely due to
wrapping and other formatting changes like promoting or demoting a tree,
whereas only present to the users areas where text content has been
changed (a driver that understands the extra word is what I care about
and not the apparent 5 line change because of the wrapping).

Maybe the way to do this would be to first simplify the text into a form
where formatting is irrelevant. The driver could then take the
formatting merge related decisions and present the content merging
decisions to the user. Of course this is all talk and probably a *big*
project in itself. :-p

Hopefully I was able to express myself clearly.

-- 
Suvayu

Open source is the future. It sets us free.

Re: Git merge tool for Org files

[Achim Gratz]

Carsten Dominik <carsten.dominik@gmail.com> writes:
> One of the obstacle for using Org-mode for collaborative programs is
> that many operations add or remove text from the beginning or end of a
> file, or which add or remove a subnode from an outline tree.

On of the things that I think would be necessary is a way to put an Org
document into a "normalized" representation and then evaluate the
differences between two of these.  The generic exporter started by
Bastien could probably deliver such a representation.

> These are operations that confuse the git merger, in particular
> if two people have added something to a file, or removed/added
> sequential sibling nodes.

Git merge only works reasonably well for program code (and only if
there's no changelog stuff inside the file).  The merge driver
functionality is there to solve that problem for when you _can_ be
smarter.  It is absolutely not necessary to write that driver in any
particular language, all that git does is to call the driver with three
versions of the same file as the argument.

> I believe that this could be solved with a dedicated merge driver
> that understands the integrity of an outline (sub)tree, and that
> knows that the sequence of two new subtrees added by different people
> does not matter.

You ask a potentially very big question here, this should be constrained
to keep it manageable.  I think a reasonable strategy is to merge two
Org documents "from the outside in", that is starting with the topmost
tree elements.  If you are merging at the tree level, anything inside
the tree should not matter at all.  I'm not sure the sequence doesn't
matter, but that can hardly be decided by the merge driver unless it
figures out how to sort elements of a tree.  The second step would then
be to check the leaf nodes of the tree for concurrent changes, which
probably still need to be resolved manually.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables