Three questions about Org-mode API

Three questions about Org-mode API

[Dmitri Minaev]

Hello,

I use org-mode as a front-end to view and edit the data stored in a
list. It's easy and fast to render the list as an Org-mode tree, but I
have some problems with the performance when I convert the data back
to the list.

1. If I want to retrieve all properties of an entry (a thousand or two
of entries, for that matter), what would be faster: using
org-entry-get for every property, org-entry-properties or using custom
regexps?

2. Is there a way to obtain programmatically the contents of an entry,
the text in it (assuming the entry has no children nodes)?

3. It would be nice to mark the edited entries as `dirty' to avoid the
conversion of non-changed entries. Any ideas?

Thanks.

-- 
With best regards,
Dmitri Minaev

Russian history blog: http://minaev.blogspot.com

Re: Three questions about Org-mode API

[Carsten Dominik]

On May 11, 2008, at 9:19 PM, Dmitri Minaev wrote:

> Hello,
>
> I use org-mode as a front-end to view and edit the data stored in a
> list. It's easy and fast to render the list as an Org-mode tree, but I
> have some problems with the performance when I convert the data back
> to the list.
>
> 1. If I want to retrieve all properties of an entry (a thousand or two
> of entries, for that matter), what would be faster: using
> org-entry-get for every property, org-entry-properties or using custom
> regexps?

If you are talking only about the standard properties (i.e. not the  
TODO state or the tags, but just the properties in the drawer, the  
fastest inside-org way would be

(org-entry-get nil 'standard)

at each headline, which would get rid of the considerable overhead of  
getting the special properties.  This still has some overhead, so  
custom regexps could speed this up a bit.

If speed is an issue, I would write an external program in perl.
I think I could write a perl parser that is at least a factor of 10  
faster
than anything in emacs lisp.

One could also think of an external database, but that only would work  
will for a linear list of entries, and structure editing does ruin  
such things.

> 2. Is there a way to obtain programmatically the contents of an entry,
> the text in it (assuming the entry has no children nodes)?

Check out Bastien's parser, I think it is in some branch in the git repo
(right Bastien???).  Although I don't know how fast this would be.

> 3. It would be nice to mark the edited entries as `dirty' to avoid the
> conversion of non-changed entries. Any ideas?

This is hard, because you don't want to put any contraints on how the  
entry can be edited.  One could use text properties (during a single  
session) or Org properties, both triggered with after-change- 
functions, but that is a lot of editing overhead.

- Carsten

Re: Three questions about Org-mode API

[Dmitri Minaev]

On Mon, May 12, 2008 at 4:03 PM, Carsten Dominik <dominik@science.uva.nl> wrote:
>  If you are talking only about the standard properties (i.e. not the TODO
> state or the tags, but just the properties in the drawer, the fastest
> inside-org way would be
>
>  (org-entry-get nil 'standard)

No, unfortunately, it won't do. I will need tags, TODO states and
priorities, among other things.

>  If speed is an issue, I would write an external program in perl.
>  I think I could write a perl parser that is at least a factor of 10 faster
>  than anything in emacs lisp.

Perhaps, this is true. But this is my first program in elisp and I
would like to take the chance to learn it :)

What if I ditch the org-mode tools and write a specialized parser in
elisp? My org file has a rather regular structure, with the uniform
properties located in the same order in all entries. Do you think it
would be faster?

>  One could also think of an external database, but that only would work will
> for a linear list of entries, and structure editing does ruin such things.

Hmm... How's that?

Well, what I'm writing is an ebooks catalog. I keep the "database" in
a list. To browse the catalog, I render it into an org-mode-compliant
text buffer and run org-mode. Here I can change tags, priorities, TODO
(toread) state, edit the description and, in some cases, the
information stored in standard properties: title, authors, genre, path
to the file. The database may be rendered in three modes: by title (1
level); by author/title (2 levels) and by genre/author/title (3
levels). When I've done with browsing and editing, I have to convert
the org-mode buffer back into the list. The number of books should be
large enough. As for now, I can deal with 1,000 of them with a
tolerable speed. But I hope to make the library work with up to 10,000
books. So, the list is not linear. And still... An external database?
How?

>  Check out Bastien's parser, I think it is in some branch in the git repo
>  (right Bastien???).  Although I don't know how fast this would be.

Thanks, I'll have a look at it.

> > 3. It would be nice to mark the edited entries as `dirty' to avoid the
> > conversion of non-changed entries. Any ideas?
>
>  This is hard, because you don't want to put any contraints on how the entry
> can be edited.  One could use text properties (during a single session) or
> Org properties, both triggered with after-change-functions, but that is a
> lot of editing overhead.

Could I use some hook that would add an extra property for every changed entry?

-- 
With best regards,
Dmitri Minaev

Russian history blog: http://minaev.blogspot.com

Re: Three questions about Org-mode API

[Carsten Dominik]

Hi Dmitri,

On May 12, 2008, at 8:36 PM, Dmitri Minaev wrote:

> On Mon, May 12, 2008 at 4:03 PM, Carsten Dominik <dominik@science.uva.nl 
> > wrote:
>> If you are talking only about the standard properties (i.e. not the  
>> TODO
>> state or the tags, but just the properties in the drawer, the fastest
>> inside-org way would be
>>
>> (org-entry-get nil 'standard)
>
> No, unfortunately, it won't do. I will need tags, TODO states and
> priorities, among other things.

OK.

>
>
>> If speed is an issue, I would write an external program in perl.
>> I think I could write a perl parser that is at least a factor of 10  
>> faster
>> than anything in emacs lisp.
>
> Perhaps, this is true. But this is my first program in elisp and I
> would like to take the chance to learn it :)

That is certainly a good opportunity to learn this stuff.

> What if I ditch the org-mode tools and write a specialized parser in
> elisp? My org file has a rather regular structure, with the uniform
> properties located in the same order in all entries. Do you think it
> would be faster?

Yes, I think it would be faster.  Maybe a factor 2-3?  Not much more  
(I hope, or my code is really bad :-)


I think if you search for org-complex-heading-regexp and use that to  
extract TODO state, level, priority, heading text, this will already  
be more efficient than getting the special properties through calling  
org-entry-properties, because you get it in a single match.

>> One could also think of an external database, but that only would  
>> work will
>> for a linear list of entries, and structure editing does ruin such  
>> things.
>
> Hmm... How's that?
>
> Well, what I'm writing is an ebooks catalog. I keep the "database" in
> a list. To browse the catalog, I render it into an org-mode-compliant
> text buffer and run org-mode. Here I can change tags, priorities, TODO
> (toread) state, edit the description and, in some cases, the
> information stored in standard properties: title, authors, genre, path
> to the file. The database may be rendered in three modes: by title (1
> level); by author/title (2 levels) and by genre/author/title (3
> levels). When I've done with browsing and editing, I have to convert
> the org-mode buffer back into the list. The number of books should be
> large enough. As for now, I can deal with 1,000 of them with a
> tolerable speed. But I hope to make the library work with up to 10,000
> books. So, the list is not linear. And still... An external database?
> How?


What I meant to say:  One could keep all the stuf in a database, and  
then pull out individual entries to edit them.  The problems with such  
an approach is that

- You loose structure information from an outline structure in the org  
file
- You only get to edit individual entries

If you like the convenience of having the whole ting in a single file  
to edit it, this is not a solution, I agree.


>
>
>> Check out Bastien's parser, I think it is in some branch in the git  
>> repo
>> (right Bastien???).  Although I don't know how fast this would be.
>
> Thanks, I'll have a look at it.
>
>>> 3. It would be nice to mark the edited entries as `dirty' to avoid  
>>> the
>>> conversion of non-changed entries. Any ideas?
>>
>> This is hard, because you don't want to put any contraints on how  
>> the entry
>> can be edited.  One could use text properties (during a single  
>> session) or
>> Org properties, both triggered with after-change-functions, but  
>> that is a
>> lot of editing overhead.
>
> Could I use some hook that would add an extra property for every  
> changed entry?

As I said, this would have to be after-change-functions.  Check out
the Elisp documentation for this hook.

- Carsten