question on org-element-interpret-data and when it works

question on org-element-interpret-data and when it works

[John Kitchin]

[-- Attachment #1: Type: text/plain, Size: 3032 bytes --]

I am trying to find some ways to programatically modify org-elements that
use fewer regexps and motion commands. It seems like org-dp (
https://github.com/tj64/org-dp) was intended to do that but it is not clear
enough how you might use it, and it also doesn't seem to support
plain-lists yet.

What I imagined happening is that I would get the element to modify as a
data structure, modify the data structure, and replace the old element with
an interpreted version of the modified data structure.

I want to do something like a radio list where only one box can be checked
at a time. Suppose I have this:

#+attr_org: :radio
- [ ] one
- [ ] two
- [ ] three

It gets represented from org-element-context as:

(plain-list
 (:type unordered :begin 579 :end 630 :contents-begin 598 :contents-end 630
:structure
((598 0 "- " nil "[ ]" nil 608)
(608 0 "- " nil "[ ]" nil 618)
(618 0 "- " nil "[ ]" nil 630))
:post-blank 0 :post-affiliated 598 :attr_org
(":radio")
:parent nil))

What I thought I could is something like (here I simulate having point in
the first item):

#+BEGIN_SRC emacs-lisp :results code
(let* ((p 600) ;where current point is
       n
       (data '(plain-list
       (:type unordered :begin 579 :end 630 :contents-begin 598
:contents-end 630 :structure
      ((598 0 "- " nil "[ ]" nil 608)
       (608 0 "- " nil "[ ]" nil 618)
       (618 0 "- " nil "[ ]" nil 630))
      :post-blank 0 :post-affiliated 598 :attr_org
      (":radio")
      :parent nil)))
       (structure (plist-get (cadr data) :structure)))
  (loop for i from 0 for item in structure
do
(if (and (>= p (first item))
(< p (seventh item)))
    ;; in the item, toggle it
    (setf (fifth item) (if (string= "[X]" (fifth item))
   "[ ]"
"[X]"))
  ;; not on the item
  (setf (fifth item) "[ ]")))

  data)
#+END_SRC

Which outputs:

#+RESULTS:
#+BEGIN_SRC emacs-lisp
(plain-list
 (:type unordered :begin 579 :end 630 :contents-begin 598 :contents-end 630
:structure
((598 0 "- " nil "[X]" nil 608)
(608 0 "- " nil "[ ]" nil 618)
(618 0 "- " nil "[ ]" nil 630))
:post-blank 0 :post-affiliated 598 :attr_org
(":radio")
:parent nil))
#+END_SRC

As a step towards getting to a data structure I could programmatically
modify, and then reinterpret, I made this little function:

(defun ointerp ()
  (interactive)
  (let ((el (org-element-context)))
    (kill-new (org-element-interpret-data el))))

It works on some things, e.g. headlines, src blocks. I put the point on one
of those things, run this command, and then I can paste it somewhere to see
that it did indeed work.

But, it does not work on plain-lists, or paragraphs. I either get an empty
string, or Wrong type argument: char-or-string-p, nil

Is it possible to do what I am describing? Am I just missing how to get the
element data in the right form?

Thanks,



John

-----------------------------------
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

[-- Attachment #2: Type: text/html, Size: 7038 bytes --]

Re: question on org-element-interpret-data and when it works

[Nicolas Goaziou]

Hello,

John Kitchin <jkitchin@andrew.cmu.edu> writes:

> I am trying to find some ways to programatically modify org-elements that
> use fewer regexps and motion commands. It seems like org-dp (
> https://github.com/tj64/org-dp) was intended to do that but it is not clear
> enough how you might use it, and it also doesn't seem to support
> plain-lists yet.

[...]

> It works on some things, e.g. headlines, src blocks. I put the point on one
> of those things, run this command, and then I can paste it somewhere to see
> that it did indeed work.
>
> But, it does not work on plain-lists, or paragraphs. I either get an empty
> string, or Wrong type argument: char-or-string-p, nil
>
> Is it possible to do what I am describing? Am I just missing how to get the
> element data in the right form?

You cannot change a non-terminal element without changing its contents.
In particular plain-lists, tables and paragraphs are fully defined by
their contents, i.e., they do not decorate contents like headlines. In
this case, data returned by `org-element-context' is incomplete in this
case. You probably need to parse the buffer between :begin and :end and
modify structure recursively.

Regards,

-- 
Nicolas Goaziou

Re: question on org-element-interpret-data and when it works

[Thorsten Jolitz]

John Kitchin <jkitchin@andrew.cmu.edu> writes:

Hello John,

> I am trying to find some ways to programatically modify org-elements
> that use fewer regexps and motion commands. It seems like org-dp
> (https://github.com/tj64/org-dp) was intended to do that

thats right, that's it's exact use case

> but it is not clear enough how you might use it, and it also doesn't
> seem to support plain-lists yet.

it's actually split into core and lib functionality, org-dp.el being the
core and org-dp-lib.el being the lib.

AFAIK it org-dp does work, and in its core it's based on just two
commands, one for CREATE and one for REWIRE (=modify) org elements
(locally, not replacing Nicolas org element framework, but rather making
it easy to use locally without the need for a whole parse tree).

The basic idea:
 - internally, all org elements look alike (plists)
 - org elements have many properties, but we are interested only in
   those used by the org element interpreter (that creates org syntax
   out of Emacs Lisp plists). These are surprisingly few.
 - luckily, plists ignore all elements that are not accessed, making
   transformation between different plists for different org elements
   much easier

You should be able to do anything you want with org elements with just
these two functions, org-dp-create and org-dp-rewire.

There is a generic prompt function two. Never write interactive specs
again, just use this one prompt functions for all org element related
prompts (or utility commands based on the prompt workhorse).

And then there is this fast and simple mapping function for org-dp:

,----[ C-h f org-dp-map RET ]
| org-dp-map is a Lisp function in ‘org-dp.el’.
| 
| (org-dp-map FUN-WITH-ARGS RGXP &optional MATCH-POS BACKWARD-SEARCH-P
| BEG END SILENT-P)
| 
| Apply quoted FUN-WITH-ARGS at every RGXP match.
| 
| [...] 
| In contrast to other mapping functions in Org-mode, this mapping
| function does not collect any information about mapped elements,
| it simply moves point quickly to all positions in a buffer(range)
| that are matched by a (forward) regexp-search and applies one of
| ‘org-dp’’s or ‘org-dp-lib’’s functions locally at that
| point (i.e. without any context information other than that about
| the parsed element-at-point).
| 
| When calling FUN ‘org-dp-create’, or ‘org-dp-rewire’ with
| argument ELEMENT given, no parsing at all takes places, but newly
| created of modified elements can be inserted at point.
| 
| This mapping function wraps its body in ‘save-excursion’ and
| ‘save-match-data’ calls, so point position and global match-data
| are preserved. It does not widen the buffer before executing its
| body, so buffer restrictions are respected. 
`----

> What I imagined happening is that I would get the element to modify as a
> data structure, modify the data structure, and replace the old element
> with an interpreted version of the modified data structure.

Thats exactly what this central org-dp workhorse function does:

,----[ C-h f org-dp-rewire RET ]
| org-dp-rewire is a Lisp function in ‘org-dp.el’.
| 
| (org-dp-rewire ELEM-TYPE &optional CONTENTS REPLACE AFFILIATED ELEMENT
| &rest ARGS)
| 
| Rewire element-at-point or ELEMENT (if given).
| 
| [...] 
| ELEM-TYPE is one of the types in ‘org-element-all-elements’. If
| it is nil, the element type of the original element is used. ARGS
| is a plist consisting of key-val pairs of all other keyword
| arguments given, defining the (rewired) element’s properties.
| 
| The former value of an element property can be reused in the
| creation of a new value by giving a ‘lambda’ expession or
| function taking two arguments (instead of just a value) to a
| key. The first argument will then be replaced by the property’s
| former value when applying the function. The second argument
| should be the parsed element itself, enabling access to its type
| and all its properties inside of the lambda expression.
`----

But, as Nicolas said in his answer, plain lists, tables, properties
etc are nested org elements, what is most obvious with tables:
a table is just a container for table rows that hold the actual data
(with some meta data in the container).

Thats why org-dp-lib.el has several related functions:

,----
| org-dp-lib.el
| 40:(defun org-dp-wrap-in-block (&optional lines user-info &rest prompt-spec)
| 204:(defun org-dp-toggle-headers (&optional action)
| 290:(defun org-dp-org-props ()
| 307:(defun org-dp-filter-node-props (filter &optional negate-p verbose-p)
| 366:(defun org-dp-create-table (row-lst &optional tblfm table-el-p insert-p)
| 416:(defun org-dp-create-plain-list (item-lst &optional insert-p)
| 452:(defun org-dp-create-property-drawer (node-prop-lst &optional insert-p)
`----

They should hopefully be pretty well documented, since I use to do that.

To get started with org-dp, you really need these 3 (or 4)
functions. They spare you typing, and more important, they tell which
properties of an org element matter for its interpretation.

CREATE:

,----[ C-h f tempo-template-org-dp-create RET ]
| tempo-template-org-dp-create is an interactive Lisp function.
| 
| (tempo-template-org-dp-create &optional ARG)
| 
| Insert org-dp-create template.
| 
| [back]
`----

or this one with explanations to get started (not so much for productive use):

,----[ C-h f tempo-template-org-dp-create-with-comments RET ]
| tempo-template-org-dp-create-with-comments is an interactive Lisp
| function.
| 
| (tempo-template-org-dp-create-with-comments &optional ARG)
| 
| Insert org-dp-create template.
| 
| [back]
`----

Example: you will be prompted for the org element (in this case I
decided to use "example-block"), you don't have to type anything, just
do this single selection when prompted.

,----
| ;; Affiliated keywords: '(:kw1 val1 :kw2 val2 ...)
| ;; :name "val"
| ;; :plot "val"
| ;; :results ("val") or ("val" "key")
| ;; :header ("val") or ("val1" "val2")
| ;; :caption (("val")) or (("val" "key")) or
| ;;          (("val2" "key2") ("val2" "key2"))
| ;; :attr_xyz ("val") or ("val1" "val2")
| (org-dp-create 'example-block nil t ;cont ins
| nil ;aff 
| :switches ""
| :preserve-indent ""
| :value ""
| )
`----

What you see: you can just write "Hello World" in the 

,----
| :value ""
`----

parameter and will get an example block with "Hello World" as content
(funny enough, some org elements have 'value' as their content, others
'content').  Check the docstring of org-dp-create for more info on its
args.  You can add affiliated keywords too, as demonstrated (as one
function arg).


REWIRE:

,----[ C-h f tempo-template-org-dp-rewire RET ]
| tempo-template-org-dp-rewire is an interactive Lisp function.
| 
| (tempo-template-org-dp-rewire &optional ARG)
| 
| Insert org-dp-rewire template.
| 
| [back]
`----

or this one, if you don't just want to set properties, but need a lambda
function to do work in calculation the new property (with access to the
old property and the whole plist of the old element you want to
rewire/modify):

,----[ C-h f tempo-template-org-dp-rewire-lambda RET ]
| tempo-template-org-dp-rewire-lambda is an interactive Lisp function.
| 
| (tempo-template-org-dp-rewire-lambda &optional ARG)
| 
| Insert org-dp-rewire template with lambdas.
| 
| [back]
`----

Examples:

,----
| (org-dp-rewire 'example-block nil t ;cont ins
| nil ;aff 
| nil ;elem 
| :switches ""
| :preserve-indent ""
| :value ""
| )
`----

,----
| (org-dp-rewire 'example-block nil t ;cont ins
| nil ;aff 
| nil ;elem 
| :switches '(lambda (old elem)  )
| :preserve-indent '(lambda (old elem)  )
| :value '(lambda (old elem)  )
`----

Again, check the docstring of org-dp-rewire for more info on its args.

I hope this helps already, feel free to ask any further questions (maybe
put me in Cc since I'm not a very frecuent visitor of the list).

If I find time and motivation I might look at your specific example/use
case, but I cannot promise that.

-- 
cheers,
Thorsten

Re: question on org-element-interpret-data and when it works

[Nicolas Goaziou]

Hello,

Thorsten Jolitz <tjolitz@gmail.com> writes:

> (funny enough, some org elements have 'value' as their content, others
> 'content').

Could you point out where there is such discrepancy in "org-element.el"?

Thank you.

Regards,

-- 
Nicolas Goaziou

Re: question on org-element-interpret-data and when it works

[Thorsten Jolitz]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

Hello,

> Thorsten Jolitz <tjolitz@gmail.com> writes:
>
>> (funny enough, some org elements have 'value' as their content, others
>> 'content').
>
> Could you point out where there is such discrepancy in "org-element.el"?

I have defined these two constants in org-dp.el to work around this
discrepancy (and to know which elements do not have interpreted content
at all):

,----
| (defconst org-dp-no-content-elems
|   (list 'babel-call 'clock 'comment 'comment-block 'diary-sexp
| 	'example-block 'fixed-width 'horizontal-rule 'keyword
| 	'latex-environment 'node-property 'planning 'src-block)
|   "List of Org elements without interpreted .")
| 
| (defconst org-dp-value-blocks
|   (list 'comment-block 'example-block 'src-block)
|   "List of Org block that have a :value instead of contents.")
`----

PS 
should probably read "... without interpreted content" in the first
defconst

-- 
cheers,
Thorsten

Re: question on org-element-interpret-data and when it works

[Nicolas Goaziou]

Thorsten Jolitz <tjolitz@gmail.com> writes:

> I have defined these two constants in org-dp.el to work around this
> discrepancy (and to know which elements do not have interpreted content
> at all):
>
> ,----
> | (defconst org-dp-no-content-elems
> |   (list 'babel-call 'clock 'comment 'comment-block 'diary-sexp
> | 	'example-block 'fixed-width 'horizontal-rule 'keyword
> | 	'latex-environment 'node-property 'planning 'src-block)
> |   "List of Org elements without interpreted .")
> | 
> | (defconst org-dp-value-blocks
> |   (list 'comment-block 'example-block 'src-block)
> |   "List of Org block that have a :value instead of contents.")
> `----

I don't understand where you think there is a discrepancy here.

Re: question on org-element-interpret-data and when it works

[Thorsten Jolitz]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Thorsten Jolitz <tjolitz@gmail.com> writes:
>
>> I have defined these two constants in org-dp.el to work around this
>> discrepancy (and to know which elements do not have interpreted content
>> at all):
>>
>> ,----
>> | (defconst org-dp-no-content-elems
>> |   (list 'babel-call 'clock 'comment 'comment-block 'diary-sexp
>> | 	'example-block 'fixed-width 'horizontal-rule 'keyword
>> | 	'latex-environment 'node-property 'planning 'src-block)
>> |   "List of Org elements without interpreted .")
>> | 
>> | (defconst org-dp-value-blocks
>> |   (list 'comment-block 'example-block 'src-block)
>> |   "List of Org block that have a :value instead of contents.")
>> `----
>
> I don't understand where you think there is a discrepancy here.

You used the word 'discrepancy', I simply needed to know for each org
element what is interpreted and what not. And some have a content,
others a :value. So if I pass 'Hello World' as content to an example
block, nothing happens, if I pass it via :value, it appears as the
blocks ... well, content.

This is no critique, and no problem, and please don't change it (since
it would be a breaking change in this context). 

With the org-dp tempo-templates, its no problem for org-dp users either,
since these are smart: they offer you a content arg (cont) to fill, if
it makes sense, they hide it, if not. And for some block types they
offer the :value parameter, since it is interpreted.

So I don't see any problem, just something an org-dp user probably
should be aware of.

-- 
cheers,
Thorsten

Re: question on org-element-interpret-data and when it works

[Nicolas Goaziou]

Thorsten Jolitz <tjolitz@gmail.com> writes:

> You used the word 'discrepancy',

True. I inferred it from

  (funny enough, some org elements have 'value' as their content, others
  'content').

which, IMO, sounds like it is a surprising fact.

> I simply needed to know for each org
> element what is interpreted and what not. And some have a content,
> others a :value.

As in every AST, some nodes are terminal (no contents), and some are not
(contents).

This distinction is made in `org-element-greater-elements', i.e.,
non-terminal elements. See also `org-element-recursive-objects' for
non-terminal objects.

> So if I pass 'Hello World' as content to an example
> block, nothing happens, if I pass it via :value, it appears as the
> blocks ... well, content.

Contents imply Org syntax. This would defeat the purpose of an example
block.

Re: question on org-element-interpret-data and when it works

[Thorsten Jolitz]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Thorsten Jolitz <tjolitz@gmail.com> writes:
>
>> You used the word 'discrepancy',
>
> True. I inferred it from
>
>   (funny enough, some org elements have 'value' as their content, others
>   'content').
>
> which, IMO, sounds like it is a surprising fact.
>
>> I simply needed to know for each org
>> element what is interpreted and what not. And some have a content,
>> others a :value.
>
> As in every AST, some nodes are terminal (no contents), and some are not
> (contents).
>
> This distinction is made in `org-element-greater-elements', i.e.,
> non-terminal elements. See also `org-element-recursive-objects' for
> non-terminal objects.
>
>> So if I pass 'Hello World' as content to an example
>> block, nothing happens, if I pass it via :value, it appears as the
>> blocks ... well, content.
>
> Contents imply Org syntax. This would defeat the purpose of an example
> block.

Ok, so its just a matter of wording.  

On the computer science side of things, content seems to be org elements
or objects contained in other org elements (like table rows in a table),
and on the laymans side of things the text inside of an example block
looks very much like the blocks content too (while its technically named
'value' in this case).

-- 
cheers,
Thorsten