Programmatically handling org files

Programmatically handling org files

[Joost Kremers]

Hi all,

I was wondering if there is some sort of (semi)official API for handling
org files programmatically. That's to say, is there a documented way for
non-org Emacs packages to manipulate (the contents of) org files?

Specifically, I'm wondering about creating and deleting entries (by
entries I mean headers with their contents), changing TODO state, moving
entries, that sort of thing.

Also, I was wondering if there's a way to hook into org-store-link. For
a particular major mode, I would like to be able to define what kind of
link is created when the user calls `org-store-link`. I looked at the
source of `org-store-link` and it looks like the answer is no, but I
thought I'd ask anyway. I could of course create the link myself and add
it to `org-stored-links` but that feels rather hackish and I suspect
will blow up in my face at some point in the future.

TIA


--
Joost Kremers
Life has its moments

Re: Programmatically handling org files

[John Kitchin]

Joost Kremers writes:

> Hi all,
>
> I was wondering if there is some sort of (semi)official API for handling
> org files programmatically. That's to say, is there a documented way for
> non-org Emacs packages to manipulate (the contents of) org files?

None that I know of. A non-elisp lib would have to be able to parse the
org-files. The grammar of org-files is documented here:
http://orgmode.org/worg/dev/org-syntax.html

There are libs that can parse org-mode, e.g.
http://orgmode.org/worg/org-tools/, some of them may also be able modify
the files. The elisp org libraries are light years ahead of them though
for what you describe.

> Specifically, I'm wondering about creating and deleting entries (by
> entries I mean headers with their contents), changing TODO state, moving
> entries, that sort of thing.
>
> Also, I was wondering if there's a way to hook into org-store-link. For
> a particular major mode, I would like to be able to define what kind of
> link is created when the user calls `org-store-link`. I looked at the
> source of `org-store-link` and it looks like the answer is no, but I
> thought I'd ask anyway. I could of course create the link myself and add
> it to `org-stored-links` but that feels rather hackish and I suspect
> will blow up in my face at some point in the future.

You want to add a function to org-store-link-functions. The function
should check if it is responsible for creating this link (for example by
looking at the major mode). If not, it must exit and return nil. If yes,
it should return a non-nil value after a calling `org-store-link-props'
with a list of properties and values.

>
> TIA


-- 
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: Programmatically handling org files

[Thorsten Jolitz]

Joost Kremers <joostkremers@fastmail.fm> writes:

Hi,

> I was wondering if there is some sort of (semi)official API for handling
> org files programmatically. That's to say, is there a documented way for
> non-org Emacs packages to manipulate (the contents of) org files?
>
> Specifically, I'm wondering about creating and deleting entries (by
> entries I mean headers with their contents), changing TODO state, moving
> entries, that sort of thing.

by "non-org Emacs packages" you mean Emacs packages written in Elisp,
but not part of Org-mode? 

The org-mode parser converts an Org document into a nested list and
provides many convenience functions to work on this parse tree. So
org-element.el (and maybe ox.el too) is the core library for converting
an Org text document into an Elisp data structure and working with that,
have a look at these two functions:

,----
| 3965:(defun org-element-parse-buffer (&optional granularity visible-only)
| 4043:(defun org-element-map
`----

If you feel you don't need the whole parse tree, but rather want to act
locally on the Org element at point, you might want to look at 
org-dp.el with just two core functions (create and rewire an Org
element) and a mapping functions (plus quite a few utilities in
org-dp.el and org-dp-lib.el):

,----
| 523:(cl-defun org-dp-create
|        (elem-type &optional contents insert-p affiliated &rest args)
| 642:(cl-defun org-dp-rewire
|        (elem-type &optional contents replace affiliated element &rest args)
| 766:(defun org-dp-map
|        (fun-with-args rgxp &optional match-pos backward-search-p beg end
|         silent-p)
`----

Note that I recently added 4 "Tempo" Templates (a bit like Yasnippets)
to org-dp that make it easy to insert a 'create' or 'rewire' call with
just those parameters the interpreter uses for the element to be created
or rewired:

,----
| M-x tempo-template-org-dp-create
| M-x tempo-template-org-dp-create-with-comments
| M-x tempo-template-org-dp-rewire
| M-x tempo-template-org-dp-rewire-lambda
`----
 
e.g. calling the third one with elem type "headline" enters this template

#+BEGIN_SRC emacs-lisp
  (org-dp-rewire 'headline
  "\n" ;cont
  t ;ins 
  nil ;aff 
  nil ;elem 
  :level 1 ;1..8
  :priority nil ;65|66|67
  :todo-keyword TODO
  :title ""
  :tags '( )
  :commentedp nil
  :pre-blank 0
  :footnote-section-p nil
  )
#+END_SRC

while for elem type "example block" its only:

#+BEGIN_SRC emacs-lisp
  (org-dp-rewire 'example-block nil t ;cont ins
  nil ;aff 
  nil ;elem 
  :switches ""
  :preserve-indent ""
  :value ""
  )
#+END_SRC

since example-blocks have no content (:value is their content) and only
3 parameters that are actually interpreted.

Using this system, creating or rewiring an Org Element from Elisp
requires only to define the values of the interpreted parameters, all
the low level stuff (actually creating and inserting the new/modified
element in text form) is left to the interpreters (from org-element.el).

You just declare what you want and don't worry anymore how it is done
(=> dp stands for declarative programming, in this context at least ;-)

-- 
cheers,
Thorsten

Re: Programmatically handling org files

[John Kitchin]

[-- Attachment #1: Type: text/plain, Size: 4165 bytes --]

indeed! Maybe I misunderstood the OP.

You can find more on the API here:

http://orgmode.org/worg/dev/org-element-api.html

There are also a bunch of functions to do other things, e.g.

(org-todo) (org-cut-subtree) (org-entry-put) and many others

that let you change properties of headings, cut them, etc...

These are documented in the org code. The best way to learn how to use them
is to reverse engineer how you would make the change you want manually,
e.g. which keys do you press, then use C-h k on those keys to see what
commands get run.

John

-----------------------------------
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu


On Mon, Sep 12, 2016 at 2:21 PM, Thorsten Jolitz <tjolitz@gmail.com> wrote:

> Joost Kremers <joostkremers@fastmail.fm> writes:
>
> Hi,
>
> > I was wondering if there is some sort of (semi)official API for handling
> > org files programmatically. That's to say, is there a documented way for
> > non-org Emacs packages to manipulate (the contents of) org files?
> >
> > Specifically, I'm wondering about creating and deleting entries (by
> > entries I mean headers with their contents), changing TODO state, moving
> > entries, that sort of thing.
>
> by "non-org Emacs packages" you mean Emacs packages written in Elisp,
> but not part of Org-mode?
>
> The org-mode parser converts an Org document into a nested list and
> provides many convenience functions to work on this parse tree. So
> org-element.el (and maybe ox.el too) is the core library for converting
> an Org text document into an Elisp data structure and working with that,
> have a look at these two functions:
>
> ,----
> | 3965:(defun org-element-parse-buffer (&optional granularity visible-only)
> | 4043:(defun org-element-map
> `----
>
> If you feel you don't need the whole parse tree, but rather want to act
> locally on the Org element at point, you might want to look at
> org-dp.el with just two core functions (create and rewire an Org
> element) and a mapping functions (plus quite a few utilities in
> org-dp.el and org-dp-lib.el):
>
> ,----
> | 523:(cl-defun org-dp-create
> |        (elem-type &optional contents insert-p affiliated &rest args)
> | 642:(cl-defun org-dp-rewire
> |        (elem-type &optional contents replace affiliated element &rest
> args)
> | 766:(defun org-dp-map
> |        (fun-with-args rgxp &optional match-pos backward-search-p beg end
> |         silent-p)
> `----
>
> Note that I recently added 4 "Tempo" Templates (a bit like Yasnippets)
> to org-dp that make it easy to insert a 'create' or 'rewire' call with
> just those parameters the interpreter uses for the element to be created
> or rewired:
>
> ,----
> | M-x tempo-template-org-dp-create
> | M-x tempo-template-org-dp-create-with-comments
> | M-x tempo-template-org-dp-rewire
> | M-x tempo-template-org-dp-rewire-lambda
> `----
>
> e.g. calling the third one with elem type "headline" enters this template
>
> #+BEGIN_SRC emacs-lisp
>   (org-dp-rewire 'headline
>   "\n" ;cont
>   t ;ins
>   nil ;aff
>   nil ;elem
>   :level 1 ;1..8
>   :priority nil ;65|66|67
>   :todo-keyword TODO
>   :title ""
>   :tags '( )
>   :commentedp nil
>   :pre-blank 0
>   :footnote-section-p nil
>   )
> #+END_SRC
>
> while for elem type "example block" its only:
>
> #+BEGIN_SRC emacs-lisp
>   (org-dp-rewire 'example-block nil t ;cont ins
>   nil ;aff
>   nil ;elem
>   :switches ""
>   :preserve-indent ""
>   :value ""
>   )
> #+END_SRC
>
> since example-blocks have no content (:value is their content) and only
> 3 parameters that are actually interpreted.
>
> Using this system, creating or rewiring an Org Element from Elisp
> requires only to define the values of the interpreted parameters, all
> the low level stuff (actually creating and inserting the new/modified
> element in text form) is left to the interpreters (from org-element.el).
>
> You just declare what you want and don't worry anymore how it is done
> (=> dp stands for declarative programming, in this context at least ;-)
>
> --
> cheers,
> Thorsten
>
>
>

[-- Attachment #2: Type: text/html, Size: 5617 bytes --]

Re: Programmatically handling org files

[Joost Kremers]

On Mon, Sep 12 2016, Thorsten Jolitz wrote:
> by "non-org Emacs packages" you mean Emacs packages written in Elisp,
> but not part of Org-mode? 

Yes, exactly. My wording wasn't entirely clear, I admit.

> The org-mode parser converts an Org document into a nested list and
> provides many convenience functions to work on this parse tree. So
> org-element.el (and maybe ox.el too) is the core library for converting
> an Org text document into an Elisp data structure and working with that,

So IIUC org-element.el is mainly for getting the contents of an org
buffer in such a way that a program can work with it, but not really for
modifying the contents of the buffer itself in such a way that it's
still a valid org document, right?

> If you feel you don't need the whole parse tree, but rather want to act
> locally on the Org element at point,

Yes. :-)

> you might want to look at 
> org-dp.el

Cool, I will do that!

> with just two core functions (create and rewire an Org
> element) and a mapping functions (plus quite a few utilities in
> org-dp.el and org-dp-lib.el):

[snip example]

> Using this system, creating or rewiring an Org Element from Elisp
> requires only to define the values of the interpreted parameters, all
> the low level stuff (actually creating and inserting the new/modified
> element in text form) is left to the interpreters (from org-element.el).
>
> You just declare what you want and don't worry anymore how it is done
> (=> dp stands for declarative programming, in this context at least ;-)

That sounds pretty cool, thanks!

-- 
Joost Kremers
Life has its moments

Re: Programmatically handling org files

[Joost Kremers]

On Mon, Sep 12 2016, John Kitchin wrote:
> indeed! Maybe I misunderstood the OP.
>
> You can find more on the API here:
>
> http://orgmode.org/worg/dev/org-element-api.html
>
> There are also a bunch of functions to do other things, e.g.
>
> (org-todo) (org-cut-subtree) (org-entry-put) and many others
>
> that let you change properties of headings, cut them, etc...
>
> These are documented in the org code. The best way to learn how to use them
> is to reverse engineer how you would make the change you want manually,
> e.g. which keys do you press, then use C-h k on those keys to see what
> commands get run.

Yes, I've been doing that and I've arrived at something that works, just
wasn't sure it was the Right Way™. :-)

-- 
Joost Kremers
Life has its moments

Re: Programmatically handling org files

[Joost Kremers]

On Mon, Sep 12 2016, John Kitchin wrote:
>> I was wondering if there is some sort of (semi)official API for handling
>> org files programmatically. That's to say, is there a documented way for
>> non-org Emacs packages to manipulate (the contents of) org files?
>
> None that I know of. A non-elisp lib would have to be able to parse the
> org-files.

As you've already realised, I was asking about Elisp code that's not
part of or intended to extend orgmode.

>> Also, I was wondering if there's a way to hook into org-store-link. For
>> a particular major mode, I would like to be able to define what kind of
>> link is created when the user calls `org-store-link`. I looked at the
>> source of `org-store-link` and it looks like the answer is no, but I
>> thought I'd ask anyway. I could of course create the link myself and add
>> it to `org-stored-links` but that feels rather hackish and I suspect
>> will blow up in my face at some point in the future.
>
> You want to add a function to org-store-link-functions. The function
> should check if it is responsible for creating this link (for example by
> looking at the major mode). If not, it must exit and return nil. If yes,
> it should return a non-nil value after a calling `org-store-link-props'
> with a list of properties and values.

Great, thanks! I looked at the source of `org-store-link', but this
wasn't obvious to me. :-/


-- 
Joost Kremers
Life has its moments

Re: Programmatically handling org files

[Nick Dokos]

Thorsten Jolitz <tjolitz@gmail.com> writes:

> ...
>
> If you feel you don't need the whole parse tree, but rather want to act
> locally on the Org element at point, you might want to look at 
> org-dp.el
> ...

Just a note for the benefit of newcomers to the list: Thorsten's
org-dp.el (and many other goodies) are on his github pages:

    https://github.com/tj64/org-dp
    https://github.com/tj64?tab=repositories
    
-- 
Nick