Rudel - Real-Time collaborative editing of Org-Mode files

Rudel - Real-Time collaborative editing of Org-Mode files

[Ciaran Mulloy]

Hi,
I was wondering whether anyone had experience, good bad or indifferent 
in the use of minor mode Rudel (http://rudel.sourceforge.net/) for 
collaborative editing of org-mode file by a number of different people 
over the internet.

I am hoping that it would provide similar collaborative working features 
that's available in Google Docs.

Regards,
Ciaran Mulloy

Re: Rudel - Real-Time collaborative editing of Org-Mode files

[Bastien]

Hi Ciaran,

Ciaran Mulloy <crmulloy@gmail.com> writes:

> I was wondering whether anyone had experience, good bad or indifferent in
> the use of minor mode Rudel (http://rudel.sourceforge.net/) for
> collaborative editing of org-mode file by a number of different people over
> the internet.
>
> I am hoping that it would provide similar collaborative working features
> that's available in Google Docs.

I don't have experience with Rudel but I'm also interested in hearing
from others -- maybe at some point we can organize some co-writing
session where we try to update Worg pages.

On the same topic, I'm interested to know if anyone is using prose.io
to collaborate on markdown docs:

  http://prose.io/about.html

The nice thing is that the code is free, and it should be fairly easy
for javascript programmers to adapt it so that it can handle basic Org
structure... and let *anyone* write Org doc collaboratively and
seemlessly.

  https://github.com/prose

Com'on, let's start something great :)

-- 
 Bastien

Re: Rudel - Real-Time collaborative editing of Org-Mode files

[François Pinard]

Bastien <bzg@altern.org> writes:

> I don't have experience with Rudel but I'm also interested in hearing
> from others -- maybe at some point we can organize some co-writing
> session where we try to update Worg pages.

I tried Rudel, a good while ago, and did not find it satisfying enough
to be usable, besides an emulation of the old "talk" functionality.

After your message, I quickly revisited its Web site.  While it got more
interesting on the side of supporting various standards and methods, and
without trying it this time, I got the feeling its UI did not really
improve, or at least, is still way too far from Org.

Or maybe it's just that I lack enthusiasm and vision?

>   http://prose.io/about.html
>   https://github.com/prose

Saving these pointers, might be worth further checking?

> [...] let *anyone* write Org doc collaboratively and seemlessly.
> Com'on, let's start something great :)

It would be great indeed!  Where I work, Org has no chance of spreading
to the remainder of the development team, exactly because it is not
real-time collaborative.  So we use other (flashy but weaker) means
instead.

François

Re: Rudel - Real-Time collaborative editing of Org-Mode files

[François Pinard]

François Pinard <pinard@iro.umontreal.ca> writes:

> Bastien <bzg@altern.org> writes:

>> [...] let *anyone* write Org doc collaboratively and seemlessly.
>> Com'on, let's start something great :)

> It would be great indeed!

On the road this morning, I gave some thought on this problem, which
does not seem trivial at all, to me at least.  Unless taken under the
appropriate angle — which I do not know yet! — I think I see that many
design choices could trap developers in a maze of sad consequences.

Nevertheless, it might be an interesting problem.  Maybe it would be
worth seeking a few avenues on this mailing list, trying to find a
workable solution?  Would there be an interest for such a discussion?

François

Re: Rudel - Real-Time collaborative editing of Org-Mode files

[Eric Schulte]

François Pinard <pinard@iro.umontreal.ca> writes:

> François Pinard <pinard@iro.umontreal.ca> writes:
>
>> Bastien <bzg@altern.org> writes:
>
>>> [...] let *anyone* write Org doc collaboratively and seemlessly.
>>> Com'on, let's start something great :)
>
>> It would be great indeed!
>

Unfortunately Rudel doesn't appear to be maintained (at least as far as
I could tell the last mailing list post was in 2011 and the last git
commit was in 2010).  Compiling Rudel from ELPA and from source both
result in the following error [1], which is caused by this line from the
automatically generated rudel-loaddefs.el file [2].

Rather than an Org-mode specific solution, I think adopting Rudel or
developing something similar which provides emacs-wide support for
standard collaborative editing protocols (assuming any currently exist)
would be the best way forward.

Cheers,

Footnotes: 
[1]  Debugger entered--Lisp error: (wrong-type-argument hash-table-p unbound)
  gethash(protocol unbound)
  (or (gethash category (slot-value this (quote factories))) (puthash category (rudel-backend-factory category) (slot-value this (quote factories))))
  (progn (or (gethash category (slot-value this (quote factories))) (puthash category (rudel-backend-factory category) (slot-value this (quote factories)))))
  (symbol-macrolet ((factories (slot-value this (quote factories)))) (or (gethash category factories) (puthash category (rudel-backend-factory category) factories)))
  (with-slots (factories) this (or (gethash category factories) (puthash category (rudel-backend-factory category) factories)))
  rudel-backend-factory(rudel-backend-factory protocol)
  apply(rudel-backend-factory (rudel-backend-factory protocol))
  eieio-generic-call(rudel-get-factory (rudel-backend-factory protocol))
  rudel-get-factory(rudel-backend-factory protocol)
  rudel-backend-get-factory(protocol)
  (rudel-add-backend (rudel-backend-get-factory (quote protocol)) (quote infinote) (quote rudel-infinote-backend))
  eval-buffer(#<buffer  *load*> nil "/home/eschulte/.emacs.d/src/rudel/rudel-loaddefs.el" nil t)  ; Reading at buffer position 955
  load-with-code-conversion("/home/eschulte/.emacs.d/src/rudel/rudel-loaddefs.el" "/home/eschulte/.emacs.d/src/rudel/rudel-loaddefs.el" nil nil)
  load("/home/eschulte/.emacs.d/src/rudel/rudel-loaddefs.el" nil nil t)
  load-file("rudel-loaddefs.el")
  eval((load-file "rudel-loaddefs.el") nil)
  eval-expression((load-file "rudel-loaddefs.el") nil)
  call-interactively(eval-expression nil nil)

[2]  (rudel-add-backend (rudel-backend-get-factory 'protocol) 'infinote 'rudel-infinote-backend)


-- 
Eric Schulte
http://cs.unm.edu/~eschulte

Re: Rudel - Real-Time collaborative editing of Org-Mode files

[François Pinard]

Eric Schulte <schulte.eric@gmail.com> writes:

> Unfortunately Rudel doesn't appear to be maintained [...]

Moreover, looking around, I saw a few comparisons and comments in which
other tools using the Obby protocol, which Rudel primarily supports, had
negative light.  I also found out that the synchronization problem and
issues are far, far more complex than I initially thought.  There are
really many avenues, while none seem perfect so far.

> Rather than an Org-mode specific solution, I think adopting Rudel or
> developing something similar which provides emacs-wide support for
> standard collaborative editing protocols (assuming any currently exist)
> would be the best way forward.

Agreed that any nice and general solution which overwhelms [not sure of
that word in English] Org, and even Emacs, should be considered more
tempting.  An full Emacs Lisp solution would be a wrong solution, as the
protocol itself should be implemented without the need of Emacs.  On the
other hand, any solution encumbered with explicit editing minutiae (make
this bold, change font, etc.) is less attractive, because it would mean
spurious and unwanted burden for Org files.

In my wildest dreams :-), I would see real-time collaboration between
people with most participant working on the text of a document, I mean
what follow headers or the header text itself, while a few others
reorganize the structure by moving headers around.  Moving headers
should be nothing more than moving headers, it should not imply deletion
followed by transmission of re-inserted text, as this would seriously
disrupt those altering the text: the re-organization should happen
magically under participant's feet while they are editing, nice and
easy.

Now, the notion of structure may be rendered by a synchronization
mechanism in a way which overwhelms Org by the concept of efficiently
handled nested documents, an single Org file would itself be a
collection of such nested documents.  Just an idea of course, there
might be other avenues as well which are acceptable as long as they
allow structure reorganization without text being transmitted again.

I have the vague, and admittedly strange intuition that Git internals
have the potential for representing nested documents through the
repository structure, for discovering both structural and textual
overhaul, and even for transmitting differences over the wires.  But I
doubt, all efficient that Git may already be, that it would be speedy
enough to be part of a solution.  That might even be elegant, so I hope
I'm wrong on the speed issue :-).

Maybe such a protocol already exists in an efficient or acceptable way,
yet I would tend to doubt it, and it seems like a serious undertaking to
develop one.  It also much depends if we want to accept a solution based
on a central broker for modifications, which is less difficult than a
solution based on modifications flooding over many pairwise connections,
I suspect the latter could yield conflicts and oscillation loops.

Another problem is the initial contact between two widely differing Org
files declared to represent a single one.  I do not like the solution of
having a broker holding the "official" copy of it.  Once a collaboration
session terminates, I would like that no copy be especially official in
a technical sense.  Humans would decide between them which is which!

François

Re: Rudel - Real-Time collaborative editing of Org-Mode files

[Torben Hoffmann]

[-- Attachment #1: Type: text/plain, Size: 3624 bytes --]

On Thu, Dec 27, 2012 at 3:18 AM, François Pinard <pinard@iro.umontreal.ca>wrote:

> Eric Schulte <schulte.eric@gmail.com> writes:
>
> > Unfortunately Rudel doesn't appear to be maintained [...]
>
> Moreover, looking around, I saw a few comparisons and comments in which
> other tools using the Obby protocol, which Rudel primarily supports, had
> negative light.  I also found out that the synchronization problem and
> issues are far, far more complex than I initially thought.  There are
> really many avenues, while none seem perfect so far.
>
> > Rather than an Org-mode specific solution, I think adopting Rudel or
> > developing something similar which provides emacs-wide support for
> > standard collaborative editing protocols (assuming any currently exist)
> > would be the best way forward.
>
> Agreed that any nice and general solution which overwhelms [not sure of
> that word in English] Org, and even Emacs, should be considered more
> tempting.  An full Emacs Lisp solution would be a wrong solution, as the
> protocol itself should be implemented without the need of Emacs.  On the
> other hand, any solution encumbered with explicit editing minutiae (make
> this bold, change font, etc.) is less attractive, because it would mean
> spurious and unwanted burden for Org files.
>
> In my wildest dreams :-), I would see real-time collaboration between
> people with most participant working on the text of a document, I mean
> what follow headers or the header text itself, while a few others
> reorganize the structure by moving headers around.  Moving headers
> should be nothing more than moving headers, it should not imply deletion
> followed by transmission of re-inserted text, as this would seriously
> disrupt those altering the text: the re-organization should happen
> magically under participant's feet while they are editing, nice and
> easy.
>
> Now, the notion of structure may be rendered by a synchronization
> mechanism in a way which overwhelms Org by the concept of efficiently
> handled nested documents, an single Org file would itself be a
> collection of such nested documents.  Just an idea of course, there
> might be other avenues as well which are acceptable as long as they
> allow structure reorganization without text being transmitted again.
>
> I have the vague, and admittedly strange intuition that Git internals
> have the potential for representing nested documents through the
> repository structure, for discovering both structural and textual
> overhaul, and even for transmitting differences over the wires.  But I
> doubt, all efficient that Git may already be, that it would be speedy
> enough to be part of a solution.  That might even be elegant, so I hope
> I'm wrong on the speed issue :-).
>

Not sure if it would be speedy enough, but gitit is based on git and since
it is a wiki running on top of git with the ability to edit the documents
either through a traditional wiki frontend in a browser or as a raw file in
an editor of your choice on your own machine.

Inventing a protocol to deal with synchronisation is not trivial, so a good
starting point might be gitit or raw git with the intention of learning
about what the real issues are before creating a system from scratch to
solve what might not be the right problem to solve.

I have not used org enough to be able to judge these issues, but I would
like to have a good multi-user solution since org seems to be one of the
better ways to collaborate.

<snip>

Cheers,
__
 /orben

-- 
http://www.linkedin.com/in/torbenhoffmann

[-- Attachment #2: Type: text/html, Size: 4417 bytes --]

Re: Rudel - Real-Time collaborative editing of Org-Mode files

[François Pinard]

Torben Hoffmann <torben.lehoff@gmail.com> writes:

> Not sure if it would be speedy enough, but gitit [...]

Thanks for the pointer! :-)

> Inventing a protocol to deal with synchronisation is not trivial, so a
> good starting point might be gitit or raw git with the intention of
> learning about what the real issues are before creating a system from
> scratch to solve what might not be the right problem to solve.

I tried to see a bit how to get hold on a synchronization protocol which
would be easy to use, and have been overwhelmed so far by the complexity
of the specifications, and also the implementations I quickly looked at.
I really want something simple enough for me to grasp (you know, I've
short limits!), and would be rather willing and ready to compromise on
the generality or speed, in the sake of simplicity.

So, inventing a protocol is still an avenue which I naively seek, and
for which I dared giving some thought recently, trying to stay on the
side of simplicity.

> I have not used org enough to be able to judge these issues, but I
> would like to have a good multi-user solution since org seems to be
> one of the better ways to collaborate.

Sigh!  Yes! :-)

François

colorg: Protocol [was: Re: Rudel - Real-Time collaborative editing of Org-Mode files]

[François Pinard]

François Pinard <pinard@iro.umontreal.ca> writes:

> So, inventing a protocol is still an avenue which I naively seek, and
> for which I dared giving some thought recently, trying to stay on the
> side of simplicity.

Here is a stab at a simple protocol, which I documented in:

   https://github.com/pinard/ColOrg/wiki/Protocol

I have the start of colorg.el, and now intend to write the skeleton of a
server.  Before committing them, I'll stretch both enough so they speak
to one another.  Then we will have a sandbox to play with. :-)

François

Re: colorg: Protocol [was: Re: Rudel - Real-Time collaborative editing of Org-Mode files]

[Samuel Loury]

[-- Attachment #1: Type: text/plain, Size: 2180 bytes --]

Hi,

This sounds really promising. But instead of creating your own protocol,
have you thought about extending an already existing one? I see that you
have read negative comments about tools using the obby protocol, but
have you read about the protocol itself?

For instance, even if the gobby seems quite a dead (not commit since
2011-08-07T16:07:00Z+0200), the obby protocol sounds like it has been
well thought and should worth give a look. See for instance the
annotated obby session[1].

By recreating a new protocol, you might be facing the same issues in
synchronization that gooby faced at some time and spending useless
effort trying to fix it.

As far as I can see, the only thing that appears to be missing in the
obby protocol is the possibility to move entries without deleting and
reinserting. This makes sense since it is specific to outlined
documents. Why not adding this feature to the obby protocol?

If the obby protocol or any other RTCE protocol does not fit your needs
causing the creation of a new protocol, I think it would be a good idea
to write why on your wiki page.

By the way, I tried this week end gobby server 0.4 and rudel client
(last git version) and it did not manage to connect to the gobby server
while a gobby client 0.4 succeeded. So sad...

I can't wait to see RTCE of org document!

Sincerely,

[1] http://gobby.0x539.de/trac/wiki/AnnotatedObbySession
François Pinard <pinard@iro.umontreal.ca> writes:

> François Pinard <pinard@iro.umontreal.ca> writes:
>
>> So, inventing a protocol is still an avenue which I naively seek, and
>> for which I dared giving some thought recently, trying to stay on the
>> side of simplicity.
>
> Here is a stab at a simple protocol, which I documented in:
>
>    https://github.com/pinard/ColOrg/wiki/Protocol
>
> I have the start of colorg.el, and now intend to write the skeleton of a
> server.  Before committing them, I'll stretch both enough so they speak
> to one another.  Then we will have a sandbox to play with. :-)
>
> François
>

-- 
Konubinix
GPG Key    : 7439106A
Fingerprint: 5993 BE7A DA65 E2D9 06CE  5C36 75D2 3CED 7439 106A

[-- Attachment #2: Type: application/pgp-signature, Size: 489 bytes --]

Re: colorg: Protocol [was: Re: Rudel - Real-Time collaborative editing of Org-Mode files]

[François Pinard]

Samuel Loury <konubinix@gmail.com> writes:

> But instead of creating your own protocol, have you thought about
> extending an already existing one?

Yes, of course.  My goal is getting some solution, not creating my own
thing.  I only tried to look at the internals of Rudel and
Etherpad-lite, and also to read some literature on the topic, starting
with Wikipedia.  In all cases, I felt stupid and overwhelmed. :-)  This
is not simple, as far as I can see.

> I see that you have read negative comments about tools using the obby
> protocol, but have you read about the protocol itself?

Besides Rudel, no.

> By recreating a new protocol, you might be facing the same issues in
> synchronization that gooby faced at some time and spending useless
> effort trying to fix it.

While not being fully sure, I think I have some understanding of the
problem, and the solution I have in head might have no issue.  Its
optimization is going to be a bit hairy however, and there lies the
danger for introducing errors.  My fix would then be to not optimize, so
with at least an inefficient solution, the effort is not useless. :-).

> As far as I can see, the only thing that appears to be missing in the
> obby protocol is the possibility to move entries without deleting and
> reinserting.  This makes sense since it is specific to outlined
> documents.  Why not adding this feature to the obby protocol?

Because of the bad press, which gave me the unverified impression that
by adopting Obby, I would have to spouse its problems, and get to solve
them.  I guess people much more brilliant than me already tried, and
failed or abandoned, so I just have no chance of succeeding :-).

It sounds important to me, for Org mode, to support some "Move Block"
operation, which combines delete and reinsert the same contents as a
single operation instead of two, as I suspect this is frequent when
someone is reorganize an Org outline, and I would ideally like that
people editing within a block which is being moved by someone else does
barely notice s/he is being shuffled elsewhere.  This is an Org
specialty, that is unlikely part of other protocols, and this
consideration pushed me into attempting something.  Not that I currently
have a "Move Block" operation in the protocol, but it should be easier
to add to something that I well understand.

> By the way, I tried this week end gobby server 0.4 and rudel client
> (last git version) and it did not manage to connect to the gobby server
> while a gobby client 0.4 succeeded. So sad...

I also got quick failures in my tries, of many kinds.

> [1] http://gobby.0x539.de/trac/wiki/AnnotatedObbySession

Thanks for this one, which I did not see.  I'll take a closer look!

> If the obby protocol or any other RTCE protocol does not fit your needs
> causing the creation of a new protocol, I think it would be a good idea
> to write why on your wiki page.

I'm saving these messages and recycling their content on the Wiki.  I
should get more documentation on the Wiki, but did not have much time
since I started, Friday evening, as I rather wanted to push on the code
to have by Sunday at least some skeleton that moves a bit.  And even
then, I took the time to move some previous comments to the Wiki, and
explain at least the current state of protocol.  Documenting early helps
at avoiding design errors.

> I can't wait to see RTCE of org document!

Ista Zahn published a working solution on the Wiki, that one could use
if in a hurry.  It is said to work well, see:

   https://github.com/pinard/ColOrg/wiki/emacsclient

François

Re: colorg: Protocol [was: Re: Rudel - Real-Time collaborative editing of Org-Mode files]

[Torben Hoffmann]

[-- Attachment #1: Type: text/plain, Size: 4550 bytes --]

Hi François,

I recently read an interesting article on Convergent and Commutative
Replicated Data Types (
http://hal.inria.fr/docs/00/55/55/88/PDF/techreport.pdf) which happened to
have a section called "Co-operative text editing" that seems spot on for
the problem you are trying to solved. They mention two existing solutions -
Logoot and Treedoc - that might be worth investigating...

I'm not saying that any of these will solve the problem, but by nature (and
my heavy schooling in Math) I am too lazy not to consider stealing a
solution instead of inventing my own! ;-)

Cheers,
__
 /orben


On Mon, Jan 14, 2013 at 3:38 PM, François Pinard <pinard@iro.umontreal.ca>wrote:

> Samuel Loury <konubinix@gmail.com> writes:
>
> > But instead of creating your own protocol, have you thought about
> > extending an already existing one?
>
> Yes, of course.  My goal is getting some solution, not creating my own
> thing.  I only tried to look at the internals of Rudel and
> Etherpad-lite, and also to read some literature on the topic, starting
> with Wikipedia.  In all cases, I felt stupid and overwhelmed. :-)  This
> is not simple, as far as I can see.
>
> > I see that you have read negative comments about tools using the obby
> > protocol, but have you read about the protocol itself?
>
> Besides Rudel, no.
>
> > By recreating a new protocol, you might be facing the same issues in
> > synchronization that gooby faced at some time and spending useless
> > effort trying to fix it.
>
> While not being fully sure, I think I have some understanding of the
> problem, and the solution I have in head might have no issue.  Its
> optimization is going to be a bit hairy however, and there lies the
> danger for introducing errors.  My fix would then be to not optimize, so
> with at least an inefficient solution, the effort is not useless. :-).
>
> > As far as I can see, the only thing that appears to be missing in the
> > obby protocol is the possibility to move entries without deleting and
> > reinserting.  This makes sense since it is specific to outlined
> > documents.  Why not adding this feature to the obby protocol?
>
> Because of the bad press, which gave me the unverified impression that
> by adopting Obby, I would have to spouse its problems, and get to solve
> them.  I guess people much more brilliant than me already tried, and
> failed or abandoned, so I just have no chance of succeeding :-).
>
> It sounds important to me, for Org mode, to support some "Move Block"
> operation, which combines delete and reinsert the same contents as a
> single operation instead of two, as I suspect this is frequent when
> someone is reorganize an Org outline, and I would ideally like that
> people editing within a block which is being moved by someone else does
> barely notice s/he is being shuffled elsewhere.  This is an Org
> specialty, that is unlikely part of other protocols, and this
> consideration pushed me into attempting something.  Not that I currently
> have a "Move Block" operation in the protocol, but it should be easier
> to add to something that I well understand.
>
> > By the way, I tried this week end gobby server 0.4 and rudel client
> > (last git version) and it did not manage to connect to the gobby server
> > while a gobby client 0.4 succeeded. So sad...
>
> I also got quick failures in my tries, of many kinds.
>
> > [1] http://gobby.0x539.de/trac/wiki/AnnotatedObbySession
>
> Thanks for this one, which I did not see.  I'll take a closer look!
>
> > If the obby protocol or any other RTCE protocol does not fit your needs
> > causing the creation of a new protocol, I think it would be a good idea
> > to write why on your wiki page.
>
> I'm saving these messages and recycling their content on the Wiki.  I
> should get more documentation on the Wiki, but did not have much time
> since I started, Friday evening, as I rather wanted to push on the code
> to have by Sunday at least some skeleton that moves a bit.  And even
> then, I took the time to move some previous comments to the Wiki, and
> explain at least the current state of protocol.  Documenting early helps
> at avoiding design errors.
>
> > I can't wait to see RTCE of org document!
>
> Ista Zahn published a working solution on the Wiki, that one could use
> if in a hurry.  It is said to work well, see:
>
>    https://github.com/pinard/ColOrg/wiki/emacsclient
>
> François
>



-- 
http://www.linkedin.com/in/torbenhoffmann

[-- Attachment #2: Type: text/html, Size: 5746 bytes --]