adding a new org-element?

adding a new org-element?

[John Kitchin]

Suppose one wanted to add a new org-element/syntax to org-mode. Where
would one start?

I am interested in something like the following syntax:

$(arbitrary stuff inside the sexp)$

with a mechanism to call an export function to transcode it. 

Any pointers to getting started?


--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: adding a new org-element?

[Eric S Fraga]

On Monday, 21 Mar 2016 at 21:51, John Kitchin wrote:
> Suppose one wanted to add a new org-element/syntax to org-mode. Where
> would one start?

I cannot help but I am curious:

> I am interested in something like the following syntax:
> $(arbitrary stuff inside the sexp)$

Given the power of lisp, can't you almost already do this using

[[elisp:(almost arbitrary stuff inside the sexp)]]

?  My curiousity is piqued, wondering what interesting capabilities you
are thinking of adding to org!

-- 
: Eric S Fraga (0xFFFCF67D), Emacs 25.0.91.1, Org release_8.3.4-626-gb62d55

Re: adding a new org-element?

[John Kitchin]

Eric S Fraga writes:

> On Monday, 21 Mar 2016 at 21:51, John Kitchin wrote:
>> Suppose one wanted to add a new org-element/syntax to org-mode. Where
>> would one start?
>
> I cannot help but I am curious:
>
>> I am interested in something like the following syntax:
>> $(arbitrary stuff inside the sexp)$
>
> Given the power of lisp, can't you almost already do this using
>
> [[elisp:(almost arbitrary stuff inside the sexp)]]

I could, and more or less have done it here:
http://kitchingroup.cheme.cmu.edu/blog/2015/02/05/Extending-the-org-mode-link-syntax-with-attributes/

the elisp link is a good idea, but I am looking into an idea for a
chemical markup language where you might have a $(molecule + data)$ and
reaction descriptions $(molecule -> new molecules)$ that become
machine-readable as well.

> ?  My curiousity is piqued, wondering what interesting capabilities you
> are thinking of adding to org!

I want to explore a deeper integration of text and data, something like
RDFa but in s-expressions, and with export. That isn't critical, I can
always do pre-processing to transform them, but I think it would be nice
in the long run if it hooked into the org-machinery directly.


--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: adding a new org-element?

[Eric S Fraga]

On Tuesday, 22 Mar 2016 at 07:34, John Kitchin wrote:

[...]

> the elisp link is a good idea, but I am looking into an idea for a
> chemical markup language where you might have a $(molecule + data)$ and
> reaction descriptions $(molecule -> new molecules)$ that become
> machine-readable as well.

Sounds nice and I would probably find a use for this!  With elisp, you
could of course define functions (reaction ...), (molecule ...),
etc.  Ummmm something to think about for me now that term is finishing
:-)

The nice thing about elisp is having the full power of lisp. even if it
is emacs lisp ;-)

Anyway, keep us posted!

-- 
: Eric S Fraga (0xFFFCF67D), Emacs 25.0.91.1, Org release_8.3.4-626-gb62d55

Re: adding a new org-element?

[John Kitchin]

Eric S Fraga writes:

> On Tuesday, 22 Mar 2016 at 07:34, John Kitchin wrote:
>
> [...]
>
>> the elisp link is a good idea, but I am looking into an idea for a
>> chemical markup language where you might have a $(molecule + data)$ and
>> reaction descriptions $(molecule -> new molecules)$ that become
>> machine-readable as well.
>
> Sounds nice and I would probably find a use for this!  With elisp, you
> could of course define functions (reaction ...), (molecule ...),
> etc.  Ummmm something to think about for me now that term is finishing
> :-)

I also did something like that here:
http://kitchingroup.cheme.cmu.edu/blog/2016/01/17/Side-by-side-figures-in-org-mode-for-different-export-outputs/

which worked ok as long as everything fits into a block and you don't
want things inline. That implementation had some limitations, like no
communication between the blocks, and the need to load different
functions for different exports. The latter could be fixed the way
org-links are exported by backend specific outputs.

I could define a lisp helper lib with those definitions, and a link that
would provide some functionality (e.g. view the molecule, or compute
some property) and export.

The main reason for wanting a new element is just to be able to map over
them. With links, I have to map over all the links, and check if the
type is what I want. It's not terrible, but... It is also somewhat
tedious still to refontify some link types, e.g. ir org-ref I make the
cite, ref and label links specific colors, and show the full cite links
if there are descriptions. Also, I am looking for alternatives to my
\(mis\|ab\| \)use of links for this kind of stuff ;)

I was inspired by this paper:
http://pubs.rsc.org/en/Content/ArticleLanding/2001/NJ/b008780g#!divAbstract
that uses an XML based approach for "Development of chemical markup
language (CML) as a system for handling complex chemical content" and
I wondered what this would look like if I wrote the paper in org-mode
and used a lisp markup for the data.


>
> The nice thing about elisp is having the full power of lisp. even if it
> is emacs lisp ;-)
>
> Anyway, keep us posted!


--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: adding a new org-element?

[Eric S Fraga]

Interesting points raised in your last email.  And also reminiscent of
the citation discussion... for better or for worse ;-)

org currently has effective support for literate programming with babel;
however, it has only rudimentary support for data: tables and properties
(and maybe tags).  More and more we are finding the desire to work with
data more generally outwith the constraints imposed by the current
support.

Links provide another interface to data but also rather rudimentary.

Maybe it is time to generalise some of these concepts while keeping
parsing straightforward.

I would be strongly in favour of some type of structure that supported
the equivalent of JSON in terms of data representation but with programming
functionality for export, interaction and display as provided by links
to some degree.

However, the easiest solution may be to extend the link syntax and
implementation, or maybe just the implementation, to address some of the
current limitations, especially in terms of display but also in terms of
linking to other objects in the org file (or even to other org files)?

At present, links have follow and export functionality.  The follow
functionality is a start towards actions on the data and is complete, in
the Turing sense, given that the full power of elisp is there.  Likewise
for export.  Two things are missing: linking and display.

Linking (confusing terminology: maybe cross-referencing) between "link
objects" could be imposed on the description which can then be processed
by the follow parameter.  Nevertheless, it probably would be desirable
to have a naming capability for individual link instances, one of the
aspects discussed in the citation thread IIRC.  What is missing entirely
in links is display functionality; this could be added as a third
argument to the link definition.

-- 
: Eric S Fraga (0xFFFCF67D), Emacs 25.0.91.1, Org release_8.3.4-626-gb62d55

Re: adding a new org-element?

[John Kitchin]

Eric S Fraga writes:

> Interesting points raised in your last email.  And also reminiscent of
> the citation discussion... for better or for worse ;-)

True, I recall saying or thinking something to the effect of how
"someone" might consider how to use that expanded syntax ;)

> org currently has effective support for literate programming with babel;
> however, it has only rudimentary support for data: tables and properties
> (and maybe tags).  More and more we are finding the desire to work with
> data more generally outwith the constraints imposed by the current
> support.

I would add blocks (code and special) to this.

>
> Links provide another interface to data but also rather rudimentary.

I think I agree with this. The cite links, for example in org-ref point
to "Data" in a bibtex file, and it does so by an id/bibtex-key. If you
build in parsing of the path, and potentially the description it is
possible to use existing capability to expand this, but they are always
"links" and to operate on a particular type you have to filter them out.

> Maybe it is time to generalise some of these concepts while keeping
> parsing straightforward.

This is the gist of what I am thinking about for sure. Parsing and
authoring should always be as straightforward as possible. If parsing is
hard, nobody will write the code, and if authoring is difficult nobody
will use it.


> I would be strongly in favour of some type of structure that supported
> the equivalent of JSON in terms of data representation but with programming
> functionality for export, interaction and display as provided by links
> to some degree.

My current thinking is the s-expression is sufficiently general to
represent a lot of data. Nothing would keep you from doing something
like: $(json "\"employees\":[
    {\"firstName\":\"John\", \"lastName\":\"Doe\"},
    {\"firstName\":\"Anna\", \"lastName\":\"Smith\"},
    {\"firstName\":\"Peter\",\"lastName\":\"Jones\"}
]")$

or in s-exp:

$(employees
  '((:firstname "John" :lastname "Doe")
    (:firstname "Anna" :lastname "Smith")
    (:firstname "Peter" :lastname "Jones")))$

These both can be plain text, or there could be a representation
function that displays them as a list of names in an overlay, for
example, or as "List of employees" as a clickable link with actions, and
a tooltip of some kind.

> However, the easiest solution may be to extend the link syntax and
> implementation, or maybe just the implementation, to address some of the
> current limitations, especially in terms of display but also in terms of
> linking to other objects in the org file (or even to other org files)?
>
> At present, links have follow and export functionality.  The follow
> functionality is a start towards actions on the data and is complete, in
> the Turing sense, given that the full power of elisp is there.  Likewise
> for export.  Two things are missing: linking and display.

I am mostly satisfied with the follow capability; the ability to call a
function with a menu of options makes it pretty much possible to do
anything from a link!

Display is less missing, in the sense that what isn't handled at export
can be handled by font-lock. This is already done to some extent when
org-mode makes the [[]] disappear in a link, and collapses the links
with descriptions. org-ref also does it a bit by changing the color of
links, and uncollapsing links with descriptions. It isn't that easy to
do this, at the moment, and it takes some font-lock skill that is hard
to come by ;) That is a feature of font-lock though.

It would be great if it were possible to call a display function during
font-lock that did stuff on an org element, e.g. set faces, properties,
etc...

What is tricky about that for links, is they are all one
element(object?), but there are different types of links, so the
"org-font-lock-link" function might need to call out to another
"org-font-lock-link-<type>" function, or refer to some kind of alist of
(type . font-lock-func) to easily modify individual link types.

> Linking (confusing terminology: maybe cross-referencing) between "link
> objects" could be imposed on the description which can then be processed
> by the follow parameter.  Nevertheless, it probably would be desirable
> to have a naming capability for individual link instances, one of the
> aspects discussed in the citation thread IIRC.  What is missing entirely
> in links is display functionality; this could be added as a third
> argument to the link definition.

I don't think we need a third argument for this. I feel like this is
like a css sheet in HTML, you can choose the display representation and
change it, somewhat in between what you see in the buffer, and what you
see in export (which has a filter system to change the representation).
The fallback is plain text, which might determine if you put the data in
the text, or if you put it elsewhere and link to it. You would just load
different el files to determine the representation. These might even be
specified in the file, like they are for xml.

Being able to link to the $(employees)$ link above and even use it for
something is probably useful. It might not always be what you want to
do, compared to say, just querying a database in a code block, or
reading the data in from a file. But, suppose you wanted to mine papers
that contained $(molecule inchi-key LFQSCWFLJHTTHZ-UHFFFAOYSA-N formula
C2H5OH)$ and $(float id melting-point inchi-key
LFQSCWFLJHTTHZ-UHFFFAOYSA-N value -173.2 units "degF")$, then you might
link to the melting-point value somehow in a new paper without having to
copy it.

Anyway, the thoughts are still a little loose in my head. I want to try
it, and write a paper with it, and see if it was useful enough to write
another paper that way ;)


--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: adding a new org-element?

[Eric S Fraga]

On Tuesday, 22 Mar 2016 at 15:18, John Kitchin wrote:

[...]

> Anyway, the thoughts are still a little loose in my head. I want to try
> it, and write a paper with it, and see if it was useful enough to write
> another paper that way ;)

I look forward to hearing about your experiences.  It's always in trying
to use something that you learn what it can do.
-- 
: Eric S Fraga (0xFFFCF67D), Emacs 25.0.92.1, Org release_8.3.4-668-g809a83

Re: adding a new org-element?

[Samuel W. Flint]

:: Eric S Fraga writes:

ESF> On Monday, 21 Mar 2016 at 21:51, John Kitchin wrote:
>> Suppose one wanted to add a new org-element/syntax to org-mode. Where
>> would one start?

ESF> I cannot help but I am curious:

>> I am interested in something like the following syntax: $(arbitrary
>> stuff inside the sexp)$

ESF> Given the power of lisp, can't you almost already do this using

ESF> [[elisp:(almost arbitrary stuff inside the sexp)]]

ESF> ?  My curiousity is piqued, wondering what interesting capabilities
ESF> you are thinking of adding to org!

I agree, this does sound useful, but I think some other syntax would be
more useful (LaTeX Equations are kinda important to me).

Sam

ESF> -- : Eric S Fraga (0xFFFCF67D), Emacs 25.0.91.1, Org
ESF> release_8.3.4-626-gb62d55

-- 
Samuel W. Flint
4096R/266596F4
      (9477 D23E 389E 40C5 2F10  DE19 68E5 318E 2665 96F4)
(λs.s s) λs.s s

Re: adding a new org-element?

[Eric S Fraga]

On Tuesday, 22 Mar 2016 at 13:59, Samuel W. Flint wrote:

[...]

> I agree, this does sound useful, but I think some other syntax would be
> more useful (LaTeX Equations are kinda important to me).

I'm not sure what you mean.  LaTeX equations are already available and
the mhchem package is particularly good for typesetting reactions etc.

-- 
: Eric S Fraga (0xFFFCF67D), Emacs 25.0.92.1, Org release_8.3.4-668-g809a83

Re: [SPAM] Re: adding a new org-element?

[Samuel W. Flint]

:: Eric S Fraga writes:

ESF> On Tuesday, 22 Mar 2016 at 13:59, Samuel W. Flint wrote: [...]

>> I agree, this does sound useful, but I think some other syntax would
>> be more useful (LaTeX Equations are kinda important to me).

ESF> I'm not sure what you mean.  LaTeX equations are already available
ESF> and the mhchem package is particularly good for typesetting
ESF> reactions etc.

The '$()$' syntax is too close to LaTeX equation syntax for my liking.
Yes, I'm aware of \(\) and \[\], but the $$ and $$$$ syntax is much
faster.

ESF> -- : Eric S Fraga (0xFFFCF67D), Emacs 25.0.92.1, Org
ESF> release_8.3.4-668-g809a83 

-- 
Samuel W. Flint
4096R/266596F4
      (9477 D23E 389E 40C5 2F10  DE19 68E5 318E 2665 96F4)
(λs.s s) λs.s s

Re: adding a new org-element?

[John Kitchin]

Samuel W. Flint writes:

> :: Eric S Fraga writes:
>
> ESF> On Monday, 21 Mar 2016 at 21:51, John Kitchin wrote:
>>> Suppose one wanted to add a new org-element/syntax to org-mode. Where
>>> would one start?
>
> ESF> I cannot help but I am curious:
>
>>> I am interested in something like the following syntax: $(arbitrary
>>> stuff inside the sexp)$
>
> ESF> Given the power of lisp, can't you almost already do this using
>
> ESF> [[elisp:(almost arbitrary stuff inside the sexp)]]
>
> ESF> ?  My curiousity is piqued, wondering what interesting capabilities
> ESF> you are thinking of adding to org!
>
> I agree, this does sound useful, but I think some other syntax would be
> more useful (LaTeX Equations are kinda important to me).

I assume you mean the $()$ syntax would look like a latex fragment? This
is fine too: %(stuff)% or %[stuff]% ... I am not too particular, as long
as it easy to type, and easy to parse.




>
> Sam
>
> ESF> -- : Eric S Fraga (0xFFFCF67D), Emacs 25.0.91.1, Org
> ESF> release_8.3.4-626-gb62d55


--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: adding a new org-element?

[Rasmus]

John Kitchin <jkitchin@andrew.cmu.edu> writes:

> Suppose one wanted to add a new org-element/syntax to org-mode. Where
> would one start?
>
> I am interested in something like the following syntax:
>
> $(arbitrary stuff inside the sexp)$
>
> with a mechanism to call an export function to transcode it. 
>
> Any pointers to getting started?

I guess you'd to amend org-element.el, which I guess is not really built
to add new element types...  The citation branch is an example of
modifications to add a single new element to the parser.

Hopefully Nicolas can give a deeper explanation.

Rasmus

-- 
Dung makes an excellent fertilizer