transcluding some org-elements in multiple places

transcluding some org-elements in multiple places

[John Kitchin]

Hi all,

I have an idea for how I could transclude "copies" or links to
org-elements in multiple places and keep them up to date. A prototypical
example of this is I have a set of org-contacts in one place, and I want
to create a new list of people for a committee in a new place made of
"copies" of the contact headlines. But I do not really want to duplicate
the headlines, and if I modify one, I want it reflected in the other
places. I do not want just links to those contacts, because then I can
not do things with org-map-entries, and other org-machinery which needs
the actual headlines/properties present. Another example might be I want
a table in two places, but the contents of them should stay
synchronized, ditto for a code block.

This idea was inspired by https://github.com/gregdetre/emacs-freex.

The idea starts with creating (wait for it...) a new link ;) In a
document where I want to transclude a headline, I would enter something like:

[[transclude:some-file.org::*headline title]]

Then, I would rely on the font-lock system to replace that link with the
headline and its contents (via the :activate-func link property), and to
put an overlay on it with a bunch of useful properties, including
modification hooks that would update the source if I change the the
element in this document, and some visual indication that it is
transcluded (e.g. light gray background/tooltip).

I would create a kill-buffer hook function that would replace that
transcluded content with the original link. A focus-in hook function
would make sure the transcluded content is updated when you enter the
frame. So when the file is not open, there is just a transclude link
indicating what should be put there, and when it is open, the overlay
modification hooks and focus hook should ensure everything stays
synchronized (as long as external processes are not modifying the
contents).

It seems like this could work well for headlines, and named tables, src
blocks, and probably any other element that can be addressed by a
name/ID.

Any thoughts? Do you see any potential issues?




-- 
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: transcluding some org-elements in multiple places

[Matt Price]

[-- Attachment #1: Type: text/plain, Size: 76 bytes --]

Replying only to say that I think both the name and the idea are brilliant.

[-- Attachment #2: Type: text/html, Size: 179 bytes --]

Re: transcluding some org-elements in multiple places

[Matt Price]

[-- Attachment #1: Type: text/plain, Size: 364 bytes --]

On Thu, Dec 15, 2016 at 7:26 PM, Matt Price <moptop99@gmail.com> wrote:

> Replying only to say that I think both the name and the idea are brilliant.
>
> In fact am in the middle of a very hectic time trying to write stuff up
and if I had transclusion available right now it would be BRILLIANT.
Instead resorting to Slack, etc., a much worse writing environment.

[-- Attachment #2: Type: text/html, Size: 765 bytes --]

Re: transcluding some org-elements in multiple places

[Eric S Fraga]

[-- Attachment #1: Type: text/plain, Size: 1364 bytes --]

On Friday,  9 Dec 2016 at 15:29, John Kitchin wrote:
> Hi all,
>
> I have an idea for how I could transclude "copies" or links to

[...]

> Any thoughts? Do you see any potential issues?

Potentially very interesting and useful idea.  Some thoughts/issues:

- not sure about the name but that's just æsthetics...

- can I link to a transcluded headline?  i.e. [[#localheadline]]

- what about more than level of indirection: link to a trancluded
  headline which transcludes to another etc.  What is shown?  Is there a
  performance hit?

- definitely not sure about the name... ;-)

- what happens if the destination moves or gets deleted?  the link to
  it, which looked like something "complete" is now meaningless.  This
  could be quite confusing.  This is probably the most serious issue.

- the table aspect is almost a "view" on a database which is really
  appealing but would benefit from a fully defined syntax a la sql?
  Would table formulas work as expected from the view of the contents?

- maybe view: as the link type?

- could we have "read-only" views?  I may not want the original touched.

- how does an export work?  On the view or the actual contents?

That's it for the moment based on stream of consciousness thinking!

-- 
: Eric S Fraga (0xFFFCF67D), Emacs 26.0.50.1, Org release_9.0.1-49-ga43eac

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 162 bytes --]

Re: transcluding some org-elements in multiple places

[David Talmage]

[-- Attachment #1: Type: text/plain, Size: 2031 bytes --]

On Fri, Dec 9, 2016 at 11:28 AM, Eric S Fraga <e.fraga@ucl.ac.uk> wrote:

> On Friday,  9 Dec 2016 at 15:29, John Kitchin wrote:
> > Hi all,
> >
> > I have an idea for how I could transclude "copies" or links to
>
> [...]
>
> > Any thoughts? Do you see any potential issues?
>
> Potentially very interesting and useful idea.  Some thoughts/issues:
>
>
I'm interested in this, too.  In my lab notebook org file, I have a section
that should be put into its own file and either linked or transcluded in
several places.  Transclusion would be better because I could see the text
without following a link.  It suits my lazy nature. ;-)


> ...
>
> - what about more than level of indirection: link to a trancluded
>   headline which transcludes to another etc.  What is shown?  Is there a
>   performance hit?
>
>
It seems to me that transclusion should be transitive.  A transcluded org
file should be able to transclude other org files.  It should be
transclusion all the way down.

It also seems to me that org-mode transclusion must detect cycles so that
it does not infinitely transclude documents.

- what happens if the destination moves or gets deleted?  the link to
>   it, which looked like something "complete" is now meaningless.  This
>   could be quite confusing.  This is probably the most serious issue.
>

How does org-mode handle broken links now?  That's probably the right way
to handle broken transclusion.



> - the table aspect is almost a "view" on a database which is really
>   appealing but would benefit from a fully defined syntax a la sql?
>   Would table formulas work as expected from the view of the contents?
>
>
Why wouldn't table formulas work?  Transclusion is/could be/should be just
another way to put org-mode formatted text into an org-mode buffer.

...
>
> - could we have "read-only" views?  I may not want the original touched.
>

Perhaps that would be property of the transcluded document.



> - how does an export work?  On the view or the actual contents?
>
>
Export works on the view.

[-- Attachment #2: Type: text/html, Size: 3398 bytes --]

Re: transcluding some org-elements in multiple places

[John Kitchin]

David Talmage writes:

> On Fri, Dec 9, 2016 at 11:28 AM, Eric S Fraga <e.fraga@ucl.ac.uk> wrote:
>
>> On Friday,  9 Dec 2016 at 15:29, John Kitchin wrote:
>> > Hi all,
>> >
>> > I have an idea for how I could transclude "copies" or links to
>>
>> [...]
>>
>> > Any thoughts? Do you see any potential issues?
>>
>> Potentially very interesting and useful idea.  Some thoughts/issues:
>>
>>
> I'm interested in this, too.  In my lab notebook org file, I have a section
> that should be put into its own file and either linked or transcluded in
> several places.  Transclusion would be better because I could see the text
> without following a link.  It suits my lazy nature. ;-)

It turns out to be pretty easy to see the text; that can be done with a
simple overlay. What is not easy, is to be able to edit the text, or use
the text with org-machinery, e.g. org-map-entries. The text in the
overlay is not visible to the org-machinery, so you can not get
properties, edit it, etc... Editing could be made possible with a
special buffer, but it doesn't seem likely that you can make the overlay
display work with many org-functions.

I did have some success making bi-directional editing possible with
modification-hooks on an overlay on the copied text, but it is tricky to
do and I ran into some other difficulties I have not been able to get past.

1. having a transcluded headline at a different level than the source.
E.g. I want to group headlines under a heading in one document, but they
are all first level headings in another document.

2. having an org-id in more than one location confuses org-id and causes
some issues. This suggests you might want some kind of transcluded id,
that is different than the source id.

>
>
>> ...
>>
>> - what about more than level of indirection: link to a trancluded
>>   headline which transcludes to another etc.  What is shown?  Is there a
>>   performance hit?
>>
>>
> It seems to me that transclusion should be transitive.  A transcluded org
> file should be able to transclude other org files.  It should be
> transclusion all the way down.
>
> It also seems to me that org-mode transclusion must detect cycles so that
> it does not infinitely transclude documents.

that would be a good idea!

>
> - what happens if the destination moves or gets deleted?  the link to
>>   it, which looked like something "complete" is now meaningless.  This
>>   could be quite confusing.  This is probably the most serious issue.
>>
>
> How does org-mode handle broken links now?  That's probably the right way
> to handle broken transclusion.
>
>
>
>> - the table aspect is almost a "view" on a database which is really
>>   appealing but would benefit from a fully defined syntax a la sql?
>>   Would table formulas work as expected from the view of the contents?
>>
>>
> Why wouldn't table formulas work?  Transclusion is/could be/should be just
> another way to put org-mode formatted text into an org-mode buffer.
>
> ...
>>
>> - could we have "read-only" views?  I may not want the original touched.
>>
>
> Perhaps that would be property of the transcluded document.
>
>
>
>> - how does an export work?  On the view or the actual contents?
>>
>>
> Export works on the view.


-- 
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: transcluding some org-elements in multiple places

[Matt Price]

[-- Attachment #1: Type: text/plain, Size: 1726 bytes --]

On Wed, Dec 14, 2016 at 11:55 AM, John Kitchin <jkitchin@andrew.cmu.edu>
wrote:

>
> David Talmage writes:
>
> > On Fri, Dec 9, 2016 at 11:28 AM, Eric S Fraga <e.fraga@ucl.ac.uk> wrote:
> >
> >> On Friday,  9 Dec 2016 at 15:29, John Kitchin wrote:
> >> > Hi all,
> >> >
> >> > I have an idea for how I could transclude "copies" or links to
> >>
> >> [...]
> >>
> >> > Any thoughts? Do you see any potential issues?
> >>
> >> Potentially very interesting and useful idea.  Some thoughts/issues:
> >>
> >>
> > I'm interested in this, too.  In my lab notebook org file, I have a
> section
> > that should be put into its own file and either linked or transcluded in
> > several places.  Transclusion would be better because I could see the
> text
> > without following a link.  It suits my lazy nature. ;-)
>
> It turns out to be pretty easy to see the text; that can be done with a
> simple overlay. What is not easy, is to be able to edit the text, or use
> the text with org-machinery, e.g. org-map-entries. The text in the
> overlay is not visible to the org-machinery, so you can not get
> properties, edit it, etc... Editing could be made possible with a
> special buffer, but it doesn't seem likely that you can make the overlay
> display work with many org-functions.
>
Have you had an opportunity to work o nthis at all?  For my purposes, even
having the transcluded text visible in a buffer + included in an export
would be a major improvement over, say, a link.  attempting to edit would,
I guess, pop up a special narrowed buffer that edits the original text.
Those 3 features together would cover maybe 90% of what I personally need,
so if you have the code for this, I would really love to see it. Thank you!

Matt

[-- Attachment #2: Type: text/html, Size: 2341 bytes --]

Re: transcluding some org-elements in multiple places

[John Kitchin]

I played with it a bit. Here is a link that displays file links as
overlays. Clicking on them jump to the source, and C-click will toggle
the overlay. It works ok, but I didn't test it a lot.

Say you have a table named like this in a file.

#+name: test
| 555 | 66      |
|  te | tef     |
| 555 | ftre 5f |
| 555 | ftre 5f |

This link will display the table.

transclude:file:transcluding-org-elements.org::test                        

This link will display the headline named archive.

transclude:file:transcluding-org-elements.org::*archive


Here is the code. 

#+BEGIN_SRC emacs-lisp :results silent
(defvar *transclude* t "Put overlays on or not")
(setq *transclude* t)

(org-link-set-parameters
 "transclude"
 :face '(:background "gray80")
 :follow (lambda (path)
	   (org-open-link-from-string path))
 :keymap (let ((map (copy-keymap org-mouse-map)))
           (define-key map [C-mouse-1] (lambda ()
					 (interactive)
					 (setq *transclude* (not *transclude*))
					 (unless *transclude*
					   (ov-clear 'transclude))
					 (font-lock-fontify-buffer)))
           map)
 :help-echo "Transcluded element. Click to open source. C-mouse-1 to toggle overlay."
 :activate-func (lambda (start end path bracketp)
		  (if *transclude*
		      (let ((ov (make-overlay start end))
			    el disp)
			(ov-put ov 'transclude t)
			(save-window-excursion
			  (org-open-link-from-string path)
			  (setq el (org-element-context))
			  (setq disp (buffer-substring
				      (org-element-property :begin el)
				      (- (org-element-property :end el)
					 (or (org-element-property :post-blank el) 0))))
			  (ov-put ov 'display disp)))
		    (ov-clear 'transclude 'any start end))))
#+END_SRC



Matt Price writes:

> On Wed, Dec 14, 2016 at 11:55 AM, John Kitchin <jkitchin@andrew.cmu.edu>
> wrote:
>
>>
>> David Talmage writes:
>>
>> > On Fri, Dec 9, 2016 at 11:28 AM, Eric S Fraga <e.fraga@ucl.ac.uk> wrote:
>> >
>> >> On Friday,  9 Dec 2016 at 15:29, John Kitchin wrote:
>> >> > Hi all,
>> >> >
>> >> > I have an idea for how I could transclude "copies" or links to
>> >>
>> >> [...]
>> >>
>> >> > Any thoughts? Do you see any potential issues?
>> >>
>> >> Potentially very interesting and useful idea.  Some thoughts/issues:
>> >>
>> >>
>> > I'm interested in this, too.  In my lab notebook org file, I have a
>> section
>> > that should be put into its own file and either linked or transcluded in
>> > several places.  Transclusion would be better because I could see the
>> text
>> > without following a link.  It suits my lazy nature. ;-)
>>
>> It turns out to be pretty easy to see the text; that can be done with a
>> simple overlay. What is not easy, is to be able to edit the text, or use
>> the text with org-machinery, e.g. org-map-entries. The text in the
>> overlay is not visible to the org-machinery, so you can not get
>> properties, edit it, etc... Editing could be made possible with a
>> special buffer, but it doesn't seem likely that you can make the overlay
>> display work with many org-functions.
>>
> Have you had an opportunity to work o nthis at all?  For my purposes, even
> having the transcluded text visible in a buffer + included in an export
> would be a major improvement over, say, a link.  attempting to edit would,
> I guess, pop up a special narrowed buffer that edits the original text.
> Those 3 features together would cover maybe 90% of what I personally need,
> so if you have the code for this, I would really love to see it. Thank you!
>
> Matt


-- 
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: transcluding some org-elements in multiple places

[Matt Price]

[-- Attachment #1: Type: text/plain, Size: 567 bytes --]

On Thu, Jan 5, 2017 at 7:30 PM, John Kitchin <jkitchin@andrew.cmu.edu>
wrote:

> I didn't test it with other files, and it does not seem to work with
> them. For some reason (org-open-link-from-string path) does not seem to
> actually open the file at path. I am not sure why, since the follow code
> works just fine.
>
> Here is a hackier version that seems to work for me:
>

Might be hackier but it works for me! Thanks John!

Would be cool to somehow include in export. But that's a little too hard
for me to think about.  This is already pretty good. Thank you!

[-- Attachment #2: Type: text/html, Size: 966 bytes --]

Re: transcluding some org-elements in multiple places

[John Kitchin]

[-- Attachment #1: Type: text/plain, Size: 1094 bytes --]

It would be easy to have it export the plain text of the transcluded
heading. to export the actual heading, I would setup a preprocessing hook
that replaces the link with the content it points to.

John

-----------------------------------
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu


On Thu, Jan 5, 2017 at 9:17 PM, Matt Price <moptop99@gmail.com> wrote:

>
>
> On Thu, Jan 5, 2017 at 7:30 PM, John Kitchin <jkitchin@andrew.cmu.edu>
> wrote:
>
>> I didn't test it with other files, and it does not seem to work with
>> them. For some reason (org-open-link-from-string path) does not seem to
>> actually open the file at path. I am not sure why, since the follow code
>> works just fine.
>>
>> Here is a hackier version that seems to work for me:
>>
>
> Might be hackier but it works for me! Thanks John!
>
> Would be cool to somehow include in export. But that's a little too hard
> for me to think about.  This is already pretty good. Thank you!
>
>

[-- Attachment #2: Type: text/html, Size: 2039 bytes --]

Re: transcluding some org-elements in multiple places

[Matt Price]

[-- Attachment #1: Type: text/plain, Size: 594 bytes --]

makes sense, will try to figure that out in the next couple of days.

On Thu, Jan 5, 2017 at 9:19 PM, John Kitchin <jkitchin@andrew.cmu.edu>
wrote:

> It would be easy to have it export the plain text of the transcluded
> heading. to export the actual heading, I would setup a preprocessing hook
> that replaces the link with the content it points to.
>
> John
>
> On Thu, Jan 5, 2017 at 9:17 PM, Matt Price <moptop99@gmail.com> wrote:
>
>>
>> Would be cool to somehow include in export. But that's a little too hard
>> for me to think about.  This is already pretty good. Thank you!
>>
>>
>
>

[-- Attachment #2: Type: text/html, Size: 1487 bytes --]