org-git-link does not support locational information within file

org-git-link does not support locational information within file

[Gregor Kappler]

Hi org-moders,

I started using orgmode a while ago - and it revoluzionized my
workflow.  I could not keep up with my work without it!  Thanks for
the most versatile tool I ever used and still learn!

I recently found out about org-git-link 
(http://orgmode.org/worg/org-contrib/org-git-link.html), 
and realized it would be near perfect for my research usecase:

** Use Case: Project workflow with moving files, linked to in orgmode
1. Often projects do not arrive with a defined structure: In the
   initial phase I might get several data files and other documents -
   with little to no long-term specification.  Often it is unclear
   whether the files will evolve into some bigger coauthoring project
   involving data analyzes.
2. Consequently, in the initial phase of a project, my file management
   often is messy.  Only within my org files the structure is kept
   clean, respectively is evolving into an appropriate structure.  (I
   heavily use orgmode linking of files and mails.)  Also, (nearly)
   all files are kept in git.
3. I might be tidying file system folders several times during a
   project.
   1. This breaks links.
   2. I thought storing links to revisions of files would solve broken
      links, as links to moved files can be recovered with git.

I guess this might be a scenario some fellow org-moders are facing?

** Shortcomings of git-link in current org HEAD
Yet, org-git-link currently is too greedy for my daily use:
 1. they kill org-links for org headings, if the org files are
    versioned in a git repository (and all of mine are!) and
 2. they kill in-file-search information for versioned non-org files.

** My ugly hack
My intermediate solution is to remove/add the git-link hook as
appropriate with
(add-hook 'org-store-link-functions 'org-git-store-link)
(remove-hook 'org-store-link-functions 'org-git-store-link)

** Proposal of a better solution
I think a better solution for me would be:
 1. use org-git-store-link.
 2. use git versioned files transparently, i.e. org-git-store-link
    should support search (org-ids and text files) in linked git
    revisions of files.
 3. define an interactive function that can update the revision
    information of a link at mark to the current branch head of the
    file (so I can update all links to new FS folder structure.)


I am still lame at elisp - so my implementation skills are
limited. With the great work in org-git-link all backend stuff seems
there, only needing more glue. Any hints how to achieve this would be
very welcome!


Thanks, Gregor


-- 
--

Dr. Gregor Kappler

Fakultät für Psychologie 
Institut für Entwicklungspsychologie und 
Psychologische Diagnostik
http://www.univie.ac.at/Psychologie

Universität Wien
Liebiggasse 5
A-1010 Wien
mail: gregor.kappler@univie.ac.at
tel: +43 1 4277 47866

Re: org-git-link does not support locational information within file

[Bastien]

Hi Gregor,

Gregor Kappler <gregor.kappler@univie.ac.at> writes:

>  2. use git versioned files transparently, i.e. org-git-store-link
>     should support search (org-ids and text files) in linked git
>     revisions of files.

I've look into this.  We could code things to add a search string:

  [[git:~/my.org::master@{2011-02-07}::Org code]]
                                     ^^

... but I'm reluctant to change the general syntax of links, even 
if that's just for git links.

>  3. define an interactive function that can update the revision
>     information of a link at mark to the current branch head of the
>     file (so I can update all links to new FS folder structure.)

You mean update

  [[git:~/my.org::master@{2011-02-07}::Org code]]

to 

  [[git:~/my.org::master@{2011-02-11}::Org code]]

?

Can you provide an explicit example?

> I am still lame at elisp - so my implementation skills are
> limited. With the great work in org-git-link all backend stuff seems
> there, only needing more glue. Any hints how to achieve this would be
> very welcome!

org-git-link.el is quite readable, and I'd welcome ideas on how to
extend it to fulfill your wishes without extending Org's link syntax 
too much...

Thanks!

-- 
 Bastien

Re: org-git-link does not support locational information within file

[Samuel Wales]

Hi Bastien,

I think your reluctance to change the syntax is understandable.  Then
again, I'm a proponent of simple syntax.  That is one reason I like
Lisp.

> org-git-link.el is quite readable, and I'd welcome ideas on how to
> extend it to fulfill your wishes without extending Org's link syntax
> too much...

It can be done without extending Org's link syntax at all.

I think questions of syntax are important.  Over time, syntaxes get
complicated, and we need more features, but fear to add them.
Sometimes we end up stuck in the middle, with complicated regexps, not
always factored, not quite sure how it will export or whether it can
be nested or combined or what other syntaxes it will work with or how
search will find it, but also lacking a feature somebody wants.
Adding a feature can sometimes raise questions of how to quote or
escape literal strings that look exactly like the special syntax for
the feature.  I wrote about this in a post called parsing risk with
greater care than I can apply here.

For new features, I think it would be good to consider extensible
syntax, which is a specific, documented proposal for a universal
syntax in which you can add things without breaking other things.  A
very small amount of code is necessary to add a new subfeature to a
feature, and it is even possible to open it up to users.  The parsing
and semantics are worked out once, and apply to all uses of extensible
syntax for all future features and subfeatures.  You can have
confidence that the feature or subfeature you are adding will not have
syntax problems.

(By the way, extensible syntax is a specific proposal for org that
enables many different possible future features, not the general idea
of extending syntax.  Important not to be confused about that.  If you
want to add to link syntax, you are not doing extensible syntax.  But
you can use extensible syntax to implement /any type of link you want
with any subfeatures you want including git features/.  For example, I
supplied an example that allows link coloring according to whether the
link was visited recently.  And I have been wanting another where you
can have bidirectional links using Org-IDs so that you can move both
ends of the link anywhere you want -- and automatic labels.  All of
this is feasible with a single syntax, so we don't have to pull our
hair out over unintended consequences.)

In the case of git links, we can add as many new git features as we
want without breaking anything.  The syntax can follow git's syntax
without having to figure out how to translate it or delimit it or work
out special cases.

I have more notes on this but cannot supply them now.

Some previous posts:

  http://www.mail-archive.com/emacs-orgmode@gnu.org/msg28464.html
  http://thread.gmane.org/gmane.emacs.orgmode/11896
  http://thread.gmane.org/gmane.emacs.orgmode/10204/focus=10240

Perhaps this is something to consider.

Samuel

-- 
The Kafka Pandemic:
http://thekafkapandemic.blogspot.com/2010/12/welcome-to-kafka-pandemic-two-forces_9182.html
I support the Whittemore-Peterson Institute (WPI)
===
I want to see the original (pre-hold) Lo et al. 2010 NIH/FDA/Harvard MLV paper.

Re: org-git-link does not support locational information within file

[Samuel Wales]

For a concise and relevant post:
http://www.mail-archive.com/emacs-orgmode@gnu.org/msg28464.html

-- 
The Kafka Pandemic:
http://thekafkapandemic.blogspot.com/2010/12/welcome-to-kafka-pandemic-two-forces_9182.html
I support the Whittemore-Peterson Institute (WPI)
===
I want to see the original (pre-hold) Lo et al. 2010 NIH/FDA/Harvard MLV paper.

Re: org-git-link does not support locational information within file

[Bastien]

Hi Samuel,

Samuel Wales <samologist@gmail.com> writes:

> I think your reluctance to change the syntax is understandable.  Then
> again, I'm a proponent of simple syntax.  That is one reason I like
> Lisp.

So do I.  My other concern is backward compatibility, and burden any
change about this may put on third-part tools (like the python parsers
and so on.)

>> org-git-link.el is quite readable, and I'd welcome ideas on how to
>> extend it to fulfill your wishes without extending Org's link syntax
>> too much...
>
> It can be done without extending Org's link syntax at all.

How?  I managed to hack something with git links looking like

  git:/file/::master{...}::textsearchstring

When I say it doesn't stick to the simple stored link syntax, I mean
it requires additional "::".  Which is not a big deal _per se_, and
maybe I'll end up implementing this.

> I think questions of syntax are important.  Over time, syntaxes get
> complicated, and we need more features, but fear to add them.
> Sometimes we end up stuck in the middle, with complicated regexps, not
> always factored, not quite sure how it will export or whether it can
> be nested or combined or what other syntaxes it will work with or how
> search will find it, but also lacking a feature somebody wants.
> Adding a feature can sometimes raise questions of how to quote or
> escape literal strings that look exactly like the special syntax for
> the feature.  I wrote about this in a post called parsing risk with
> greater care than I can apply here.
>
> For new features, I think it would be good to consider extensible
> syntax, which is a specific, documented proposal for a universal
> syntax in which you can add things without breaking other things.  A
> very small amount of code is necessary to add a new subfeature to a
> feature, and it is even possible to open it up to users.  The parsing
> and semantics are worked out once, and apply to all uses of extensible
> syntax for all future features and subfeatures.  You can have
> confidence that the feature or subfeature you are adding will not have
> syntax problems.
>
> (By the way, extensible syntax is a specific proposal for org that
> enables many different possible future features, not the general idea
> of extending syntax.  Important not to be confused about that.  If you
> want to add to link syntax, you are not doing extensible syntax.  But
> you can use extensible syntax to implement /any type of link you want
> with any subfeatures you want including git features/.  For example, I
> supplied an example that allows link coloring according to whether the
> link was visited recently.  And I have been wanting another where you
> can have bidirectional links using Org-IDs so that you can move both
> ends of the link anywhere you want -- and automatic labels.  All of
> this is feasible with a single syntax, so we don't have to pull our
> hair out over unintended consequences.)

Nice thoughts, I share the spirit of it.

> Some previous posts:
>
>   http://www.mail-archive.com/emacs-orgmode@gnu.org/msg28464.html

I've reread this.  The proposed syntax is clean, but implementing this
is certainly a daunting task, and discussing the theorical benefits of
such a change is not high on my todo list right now... 

> Perhaps this is something to consider.

It is -- there are enough brains here to let the discussion grow in a
useful way, and this list is not a place where we close doors :)

Thanks,

-- 
 Bastien

Re: org-git-link does not support locational information within file

[Samuel Wales]

I think it will always be much easier to add a little syntax here and
there to existing syntaxes.

To me, the question is whether it is worth it in the big picture,
long-term perspective to keep adding small syntax extensions to
existing syntax.

Yes, we have great people on this list who might be able to do
something with this.

Thanks.

Samuel

-- 
The Kafka Pandemic:
http://thekafkapandemic.blogspot.com/2010/12/welcome-to-kafka-pandemic-two-forces_9182.html
I support the Whittemore-Peterson Institute (WPI)
===
I want to see the original (pre-hold) Lo et al. 2010 NIH/FDA/Harvard MLV paper.

Re: org-git-link does not support locational information within file

[Achim Gratz]

Bastien <bastien.guerry@wikimedia.fr> writes:
>>  2. use git versioned files transparently, i.e. org-git-store-link
>>     should support search (org-ids and text files) in linked git
>>     revisions of files.
>
> I've look into this.  We could code things to add a search string:
>
>   [[git:~/my.org::master@{2011-02-07}::Org code]]

Just like org-protocol and org-annotate-helper this uses an opaque URI
and positional arguments, something I'm not too fond of.  It's easily
broken and hard to extend in a backwards-compatible manner.  If all uses
of URI-like syntax were stratified to a single general form within
orgmode, it would become a lot easier to move data back and forth using
these URI, IMHO.

Since the function is searching for something after the resource has
been retrieved by the application that requested it, it should probably
have URI fragment syntax, see RFC 2396 (this RFC actually replaces
RFC1738 that org-protocol still references).  Likewise the specification
of what version to retrieve from Git should probably be in query syntax
since it's done by the server of the request.  Last but not least, the
git scheme is already in use by git for specifying remotes, so I'd
suggest not clobbering that namespace from within org.  The use of "@"
is reserved in certain places and use of braces is labeled "unwise".
Hence something like:

 [[org://authority/git:/~/my.org?b=master;d=2011-02-07#Org%20code]]                                 

looks to be more friendly considering that maybe these links diffuse
outside of org and are tried to be interpreted by URI parsers.  It opens
up a way to ask org servers for their data (when the authority segment
is present, if not it uses the local data as implied by the org document
containing the URI) and enables a uniform way to introduce sub-handlers
(git:/ in this case).  Org-protocol does some of that already and should
be able to be extended to include the proposal above.  The arguments are
also named, so they can come in any order.  Just a thought.

[Note: emacsclient does only strip down consecutive slashes for what it
considers the path to files.  It's perfectly OK to give it a lisp
expression that contains a string with as many slashes as you see fit,
like this one.  This might be a better way to get URI arguments to emacs
server, then: emacsclient -e '(org-parse-URI "org:...")']


Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf Q+, Q and microQ:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: org-git-link does not support locational information within file

[Gregor Kappler]

Hi Bastien,

Thanks for your answer. I was surprised and glad to hear so much interest in 
my particular problem.

On Fri, 11 Feb 2011 18:17:58 +0100, Bastien <bastien.guerry@wikimedia.fr> wrote:
> >  2. use git versioned files transparently, i.e. org-git-store-link
> >     should support search (org-ids and text files) in linked git
> >     revisions of files.
> 
> I've look into this.  We could code things to add a search string:
> 
>   [[git:~/my.org::master@{2011-02-07}::Org code]]
>                                      ^^
> 
> ... but I'm reluctant to change the general syntax of links, even 
> if that's just for git links.

It might be better to stick to org-mode's convention of storing the "location within the file" after "::", a convention your current syntax is not adhering to.
I think you could stay consistent with the current general syntax of orgmode by switching from

[[git:~/my.org::master@{2011-02-11}::Org code]]

to

[[git:~/my.org@master{2011-02-11}::Org code]]
or 
[[git:~/my.org?master{2011-02-11}::Org code]]

This would fully break existing links.
Achim Gratz contributed some good points about changing the git-link syntax and made a more founded proposal for a changed syntax though. Achim seems to focus strongly on machine readability. For me, I like to be able to read links easily.


[[git:<repository or file>?<revision information>::<location in file>]]
<revision information> can be
- a date+time value (for me, time is important here)
- a SHA
- "HEAD"

> 
> >  3. define an interactive function that can update the revision
> >     information of a link at mark to the current branch head of the
> >     file (so I can update all links to new FS folder structure.)
> 
> You mean update
> 
>   [[git:~/my.org::master@{2011-02-07}::Org code]]
> 
> to 
> 
>   [[git:~/my.org::master@{2011-02-11}::Org code]]
> 
> ?
> 
> Can you provide an explicit example?

Let the function be of name org-git-link-update-to-head.
I will adhere to the syntax in your post. 
I guess all scenarios can be made explicit by two conditions:

1. The file is still of the same name. Calling org-git-link-update-to-head on 

   [[git:~/my.org::master@{2011-02-11}::Org code]]

   would change the link to

   [[git:~/my.org::master@{2011-02-15}::Org code]]

   (today being the 15th)

2. more interesting is the case that the file was moved by e.g by

   git mv my.org your.org

   Then optimally git would be queried for the actual location of a newer version in HEAD (if unique). 
   (I am actually not sure how to do this with git, but should be viable)

   The link should be rewritten by org-git-link-update-to-head to
   [[git:~/your.org::master@{2011-02-15}::Org code]]


   This file might have been edited meanwhile by e.g. 

   echo "* annoying tail heading" >> your.org && git add your.org && git commit -m ""

   Still the new link should become
   [[git:~/your.org::master@{2011-02-14}::Org code]]

If HEAD would be added to the possible revision definition, org-git-link-update-to-head cannot identify the file after moving. On the other hand, having this function one would not want to point to head.

Another possibility would be not to rewrite links but to resolve the
current revision as outlined on the fly in org-open-at-point.

> > I am still lame at elisp - so my implementation skills are
> > limited. With the great work in org-git-link all backend stuff seems
> > there, only needing more glue. Any hints how to achieve this would be
> > very welcome!
> 
> org-git-link.el is quite readable, and I'd welcome ideas on how to
> extend it to fulfill your wishes without extending Org's link syntax 
> too much...

I try to learn elisp bit by bit. Found a good tutorial finally.
I got busy times around me - need to constrain fiddle-time.


Cheers and thanks,

Gregor

-- 
--

Dr. Gregor Kappler

Fakultät für Psychologie 
Institut für Entwicklungspsychologie und 
Psychologische Diagnostik
http://www.univie.ac.at/Psychologie

Universität Wien
Liebiggasse 5
A-1010 Wien
mail: gregor.kappler@univie.ac.at
tel: +43 1 4277 47866

Re: org-git-link does not support locational information within file

[Leo Alekseyev]

> ** Shortcomings of git-link in current org HEAD
> Yet, org-git-link currently is too greedy for my daily use:
>  1. they kill org-links for org headings, if the org files are
>    versioned in a git repository (and all of mine are!) and
>  2. they kill in-file-search information for versioned non-org files.

The discussion so far focused on extending the link syntax to allow
multiple pieces of "location" information (e.g. location within the
repository + location within the file), which is a good idea.
However, I think the bigger problem with org-git-link in its current
incarnation is that it forces me to use git:// links for all files
under version control, which is NOT what I want to do 90% of the time.
 I have a quick hack to deal with this -- namely, commenting out

;; (add-hook 'org-store-link-functions 'org-git-store-link t)

and using a separate keybinding for storing git links using the
following function:

(defun org-git-store-link-interactively (arg)
  "Store git link to current file."
  (interactive "P")
  (let ((org-store-link-functions (cons 'org-git-store-link
org-store-link-functions)))
    (call-interactively 'org-store-link arg)
))


In addition, I'm not crazy about using the "branch@{date}" format for
storing links by default, so I hacked something that uses SHA1
instead...  I could post a patch if anyone is curious.

--Leo

Re: org-git-link does not support locational information within file

[Gregor Kappler]

Hi Leo,

> However, I think the bigger problem with org-git-link in its current
> incarnation is that it forces me to use git:// links for all files
> under version control, which is NOT what I want to do 90% of the time.
>  I have a quick hack to deal with this -- namely, commenting out
> 
> ;; (add-hook 'org-store-link-functions 'org-git-store-link t)
> 
> and using a separate keybinding for storing git links using the
> following function:
> 
> (defun org-git-store-link-interactively (arg)
>   "Store git link to current file."
>   (interactive "P")
>   (let ((org-store-link-functions (cons 'org-git-store-link
> org-store-link-functions)))
>     (call-interactively 'org-store-link arg)
> ))

Thanks a lot for this!

> 
> 
> In addition, I'm not crazy about using the "branch@{date}" format for
> storing links by default, so I hacked something that uses SHA1
> instead...  I could post a patch if anyone is curious.
I would be interested, and I think SHA1 should be supported as it is so
common with git.

Cheers,
Gregor

-- 
--

Dr. Gregor Kappler

Fakultät für Psychologie 
Institut für Entwicklungspsychologie und 
Psychologische Diagnostik
http://www.univie.ac.at/Psychologie

Universität Wien
Liebiggasse 5
A-1010 Wien
mail: gregor.kappler@univie.ac.at
tel: +43 1 4277 47866