[GSoC] org-merge-driver weekly update

[GSoC] org-merge-driver weekly update

[Andrew Young]

Hi everyone,
small weekly update;

I'm almost finished creating the parser.  Because of the nature
of org mode grammar, the files are being parsed with regexs and
(f)lex.

Because Orgmode files can be written in any character encoding
supported in Emacs, I am looking into using libiconv to support
as many file encodings as possible.

Sincerely,
Andrew Young

[GSoC] org-merge-driver weekly update

[Andrew Young]

[-- Attachment #1: Type: text/plain, Size: 370 bytes --]

Hello everyone,

I would just like to let everyone know that a prototype for
org-merge-driver is available.  It is not complete, so of course please do
not use it for your repository! ;)

You can see some example use of it at the the worg project page:
http://orgmode.org/worg/org-contrib/gsoc2012/student-projects/git-merge-tool/prototype.html

Sincerely,
Andrew Young

[-- Attachment #2: Type: text/html, Size: 574 bytes --]

Re: [GSoC] org-merge-driver weekly update

[Carsten Dominik]

Hi Andrew,

On 30.5.2012, at 16:36, Andrew Young wrote:

> Hello everyone,
> 
> I would just like to let everyone know that a prototype for org-merge-driver is available.  It is not complete, so of course please do not use it for your repository! ;)
> 
> You can see some example use of it at the the worg project page: http://orgmode.org/worg/org-contrib/gsoc2012/student-projects/git-merge-tool/prototype.html


A nice start!

In your first example, it seems incorrect to me that "* A new heading in file 2"
ends up being the first top-level headline in the merged file.  It is added as
second in file two, so I clearly would expect it to appear as second in the merged file.

Greetings

- Carsten

P.S.  Bastien, should comments and discussions like this one be on the list, or off list?

- Carsten

Re: [GSoC] org-merge-driver weekly update

[Bastien]

Hi Carsten,

Carsten Dominik <carsten.dominik@gmail.com> writes:

> P.S.  Bastien, should comments and discussions like this one be on the
> list, or off list?

I think it's fine to comment and discuss publicly available work on this
list.  Involving the community is part of the GSoC "rules".

Thanks,

-- 
 Bastien

Re: [GSoC] org-merge-driver weekly update

[Andrew Young]

Hi Carsten,

On Wed, May 30, 2012 at 1:15 PM, Carsten Dominik
<carsten.dominik@gmail.com> wrote:
>
> Hi Andrew,
>
> On 30.5.2012, at 16:36, Andrew Young wrote:
>
> > Hello everyone,
> >
> > I would just like to let everyone know that a prototype for org-merge-driver is available.  It is not complete, so of course please do not use it for your repository! ;)
> >
> > You can see some example use of it at the the worg project page: http://orgmode.org/worg/org-contrib/gsoc2012/student-projects/git-merge-tool/prototype.html
>
>
> A nice start!
>
> In your first example, it seems incorrect to me that "* A new heading in file 2"
> ends up being the first top-level headline in the merged file.  It is added as
> second in file two, so I clearly would expect it to appear as second in the merged file.
>
> Greetings
>
> - Carsten
>
> P.S.  Bastien, should comments and discussions like this one be on the list, or off list?
>
> - Carsten

I agree that the prototype is not correct in this respect.  I
originally planned to separate orgmode elements into two types:
ordered and unordered.  Unordered elements would be things like
plain lists and headings.  Ordered entities would be things like
numbered plain lists.

This decision led to the prototype not tracking the order which
headings appear.  Only hierachical position was tracked and used
in the prototype.  As a result, new headings are always shown at
the start of a level, and reordered headings may or may show up
in the new order.  I have since changed my mind about how to
handle this situation.

The new strategy is to track the same positional data for both
ordered and unordered elements.  The difference between the
element types will be in how reorders from both files will be
merged together.

I can modify the prototype to print headings in the correct
order, but I'm mostly just interested in moving on.

Sincerely,
Andrew

Re: [GSoC] org-merge-driver weekly update

[Carsten Dominik]

Hi Andrew,

On 31.5.2012, at 09:31, Andrew Young wrote:

> Hi Carsten,
> 
> On Wed, May 30, 2012 at 1:15 PM, Carsten Dominik
> <carsten.dominik@gmail.com> wrote:
>> 
>> Hi Andrew,
>> 
>> On 30.5.2012, at 16:36, Andrew Young wrote:
>> 
>>> Hello everyone,
>>> 
>>> I would just like to let everyone know that a prototype for org-merge-driver is available.  It is not complete, so of course please do not use it for your repository! ;)
>>> 
>>> You can see some example use of it at the the worg project page: http://orgmode.org/worg/org-contrib/gsoc2012/student-projects/git-merge-tool/prototype.html
>> 
>> 
>> A nice start!
>> 
>> In your first example, it seems incorrect to me that "* A new heading in file 2"
>> ends up being the first top-level headline in the merged file.  It is added as
>> second in file two, so I clearly would expect it to appear as second in the merged file.
>> 
>> Greetings
>> 
>> - Carsten
>> 
>> P.S.  Bastien, should comments and discussions like this one be on the list, or off list?
>> 
>> - Carsten
> 
> I agree that the prototype is not correct in this respect.  I
> originally planned to separate orgmode elements into two types:
> ordered and unordered.  Unordered elements would be things like
> plain lists and headings.  Ordered entities would be things like
> numbered plain lists.
> 
> This decision led to the prototype not tracking the order which
> headings appear.  Only hierachical position was tracked and used
> in the prototype.  As a result, new headings are always shown at
> the start of a level, and reordered headings may or may show up
> in the new order.  I have since changed my mind about how to
> handle this situation.
> 
> The new strategy is to track the same positional data for both
> ordered and unordered elements.  The difference between the
> element types will be in how reorders from both files will be
> merged together.

OK, this sounds right.

> 
> I can modify the prototype to print headings in the correct
> order, but I'm mostly just interested in moving on.

I see no compelling reason to update the prototype - just wanted to give feedback.

- Carsten

Re: [GSoC] org-merge-driver weekly update

[Robert Horn]

Another area that would be nice to address is taking advantage of the
information in date-trees so assist with merging.  This is similar to
the logic around keeping headlines in order.  With date trees there is a
date and sometimes time tag to help.

In addition to the occurrence order, there is also an ordering constraint on date trees that can be used to determine the proper delta.  You can use the date and time information in the headlines to determine the proper sequencing.

For example, the delta/merger for two files of the form:
     File 1:
     * Year
     ** Year-Month
     *** Year-Month-Day
     **** Y-M-D-Time1 stuff1 ...
     **** Y-M-D-Time2 stuff2 ...
     **** Y-M-D-Time4 stuff4 ...
     **** Y-M-D-Time5 stuff5 ...
     **** Y-M-D-Time9 stuff9 ...
     File 2:
     * Year
     ** Year-Month
     *** Year-Month-Day
     **** Y-M-D-Time1 stuff1 ...
     **** Y-M-D-Time2 stuff2 ...
     **** Y-M-D-Time3 stuff3 ...
     **** Y-M-D-Time6 stuff6 ...
     **** Y-M-D-Time7 stuff7 ...

     Should be:
     * Year
     ** Year-Month
     *** Year-Month-Day
     **** Y-M-D-Time1 stuff1 ...
     **** Y-M-D-Time2 stuff2 ...
     **** Y-M-D-Time3 stuff3 ...
     **** Y-M-D-Time4 stuff4 ...
     **** Y-M-D-Time5 stuff5 ...
     **** Y-M-D-Time6 stuff6 ...
     **** Y-M-D-Time7 stuff7 ...
     **** Y-M-D-Time9 stuff9 ...

This time aware merge logic will apply similarly to all levels of the date tree.

Date trees are recognizable by the combination of headlines in this
format.  A date tree can occur anywhere in an org file, but it will
begin with a level one headline of the form "* YYYY", etc.

R Horn
rjhorn@alum.mit.edu

Re: [GSoC] org-merge-driver weekly update

[Andrew Young]

Hi Robert Horn,

On Mon, Jun 4, 2012 at 11:30 AM, Robert Horn <rjhorn@alum.mit.edu> wrote:
> Another area that would be nice to address is taking advantage of the
> information in date-trees so assist with merging.  This is similar to
> the logic around keeping headlines in order.  With date trees there is a
> date and sometimes time tag to help.
>
> In addition to the occurrence order, there is also an ordering constraint on date trees that can be used to determine the proper delta.  You can use the date and time information in the headlines to determine the proper sequencing.
>
> For example, the delta/merger for two files of the form:
>     File 1:
>     * Year
>     ** Year-Month
>     *** Year-Month-Day
>     **** Y-M-D-Time1 stuff1 ...
>     **** Y-M-D-Time2 stuff2 ...
>     **** Y-M-D-Time4 stuff4 ...
>     **** Y-M-D-Time5 stuff5 ...
>     **** Y-M-D-Time9 stuff9 ...
>     File 2:
>     * Year
>     ** Year-Month
>     *** Year-Month-Day
>     **** Y-M-D-Time1 stuff1 ...
>     **** Y-M-D-Time2 stuff2 ...
>     **** Y-M-D-Time3 stuff3 ...
>     **** Y-M-D-Time6 stuff6 ...
>     **** Y-M-D-Time7 stuff7 ...
>
>     Should be:
>     * Year
>     ** Year-Month
>     *** Year-Month-Day
>     **** Y-M-D-Time1 stuff1 ...
>     **** Y-M-D-Time2 stuff2 ...
>     **** Y-M-D-Time3 stuff3 ...
>     **** Y-M-D-Time4 stuff4 ...
>     **** Y-M-D-Time5 stuff5 ...
>     **** Y-M-D-Time6 stuff6 ...
>     **** Y-M-D-Time7 stuff7 ...
>     **** Y-M-D-Time9 stuff9 ...
>
> This time aware merge logic will apply similarly to all levels of the date tree.
>
> Date trees are recognizable by the combination of headlines in this
> format.  A date tree can occur anywhere in an org file, but it will
> begin with a level one headline of the form "* YYYY", etc.
>
> R Horn
> rjhorn@alum.mit.edu

Thank you for the suggestion!  The program should support date trees.

I wonder if date trees specifically should be aggressively resorted
during the merge (reordering more headings than necessary, without
regards to the in-file ordering).  It is currently my opinion that the
program should try to retain the original ordering as much as
possible, only sorting the minimum number of headings necessary when
merging has made the ordering ambiguous.

Sincerely,
Andrew Young

Re: [GSoC] org-merge-driver weekly update

[Carsten Dominik]

On 6.6.2012, at 15:50, Andrew Young wrote:

> Hi Robert Horn,
> 
> On Mon, Jun 4, 2012 at 11:30 AM, Robert Horn <rjhorn@alum.mit.edu> wrote:
>> Another area that would be nice to address is taking advantage of the
>> information in date-trees so assist with merging.  This is similar to
>> the logic around keeping headlines in order.  With date trees there is a
>> date and sometimes time tag to help.
>> 
>> In addition to the occurrence order, there is also an ordering constraint on date trees that can be used to determine the proper delta.  You can use the date and time information in the headlines to determine the proper sequencing.
>> 
>> For example, the delta/merger for two files of the form:
>>    File 1:
>>    * Year
>>    ** Year-Month
>>    *** Year-Month-Day
>>    **** Y-M-D-Time1 stuff1 ...
>>    **** Y-M-D-Time2 stuff2 ...
>>    **** Y-M-D-Time4 stuff4 ...
>>    **** Y-M-D-Time5 stuff5 ...
>>    **** Y-M-D-Time9 stuff9 ...
>>    File 2:
>>    * Year
>>    ** Year-Month
>>    *** Year-Month-Day
>>    **** Y-M-D-Time1 stuff1 ...
>>    **** Y-M-D-Time2 stuff2 ...
>>    **** Y-M-D-Time3 stuff3 ...
>>    **** Y-M-D-Time6 stuff6 ...
>>    **** Y-M-D-Time7 stuff7 ...
>> 
>>    Should be:
>>    * Year
>>    ** Year-Month
>>    *** Year-Month-Day
>>    **** Y-M-D-Time1 stuff1 ...
>>    **** Y-M-D-Time2 stuff2 ...
>>    **** Y-M-D-Time3 stuff3 ...
>>    **** Y-M-D-Time4 stuff4 ...
>>    **** Y-M-D-Time5 stuff5 ...
>>    **** Y-M-D-Time6 stuff6 ...
>>    **** Y-M-D-Time7 stuff7 ...
>>    **** Y-M-D-Time9 stuff9 ...
>> 
>> This time aware merge logic will apply similarly to all levels of the date tree.
>> 
>> Date trees are recognizable by the combination of headlines in this
>> format.  A date tree can occur anywhere in an org file, but it will
>> begin with a level one headline of the form "* YYYY", etc.
>> 
>> R Horn
>> rjhorn@alum.mit.edu
> 
> Thank you for the suggestion!  The program should support date trees.
> 
> I wonder if date trees specifically should be aggressively resorted
> during the merge (reordering more headings than necessary, without
> regards to the in-file ordering).  It is currently my opinion that the
> program should try to retain the original ordering as much as
> possible, only sorting the minimum number of headings necessary when
> merging has made the ordering ambiguous.

As a general remark, if you implement things like aggressive resorting,
I think it would be good to have such features optional, in some
way configurable.  Turning off all bells would then do a simple stable
outline tree inserting like you have in your prototype.  Increasing
complexity can and should be implemented, but I would like them optional
for users (opt-out is OK).

Greetings

- Carsten

> 
> Sincerely,
> Andrew Young

Re: [GSoC] org-merge-driver weekly update

[Robert Horn]

Carsten Dominik <carsten.dominik@gmail.com> writes:

> On 6.6.2012, at 15:50, Andrew Young wrote:
> As a general remark, if you implement things like aggressive resorting,
> I think it would be good to have such features optional, in some
> way configurable.  Turning off all bells would then do a simple stable
> outline tree inserting like you have in your prototype.  Increasing
> complexity can and should be implemented, but I would like them optional
> for users (opt-out is OK).
>

I agree.  I'm thinking about a problem that I routinely have.  I work on
multiple computers capturing notes into journals that are date-trees.  I
resynchronize these journals using git, and git is often confused about
what to do when it sees the same base tree with different additions from
the different computers.

In my usage there are never deletes, and I would expect that the order
of headlines in the original versions would be preserved as the order of
those headlines in the merged version.  I don't get out of order
headlines because the capture function manages the date tree for me.

If I had an improperly ordered input I would prefer to have the merger
fail and ask for manual help.  If it's out of order that means either
something went wrong with the capture function, or I manually did
something that I might want preserved.  For instance, I might archive
the 2010 notes into some other file, and replace them with a link to
that file.  I would want to preserve this.  (Actually the issue of how
to manage such archival is one that I haven't given much thought, since
disk space is cheap and it's easy to collapse the old stuff.)

Robert Horn