[BABEL] ob-rec.el and some questions

[BABEL] ob-rec.el and some questions

[Jose E. Marchesi]

[-- Attachment #1: Type: text/plain, Size: 1851 bytes --]


Hi.

I have been playing a bit with org babel, the goal being to be able to
query a recfile[1] and insert the result as a rec table.  Thanks to the
superb design of org-babel and the existing examples, I quickly wrote
the attached little hack.

The hack allows to query a rec file 'foo.rec' as follows:

#+begin_src rec :data hackers.rec :fields Name,Email :type Hacker
Papers = 'requested' && CreatedAt << '01 January 2011'
#+end_src

Where :data points to the recfile (or list of files), :fields is a
comma-separated list of fields (allowing subscripts) and :type selects
the record set where to make the query. :cmdline can also be used to
specify any other command line option to recsel.  All parameters but
:data are optional.

The selection expression can be any expression allowed by recsel in the
-e command line option.

Of course you have to install the recutils for it to work.

It roughly works and I want to add more functionality, but due to my
lack of experience with org-babel I would like to ask some questions
before continuing working on it.

- Is there a way to execute empty code blocks?  It would be quite common
  to require all the records stored in a record set, e.g.

  #+begin_src rec :data inventory.rec
  #+end_src

  C-cC-c in that block echoes "Local setup has been refreshed" and
  nothing happens.  A workaround would be to use a selection expression
  that always evaluates to "true", such as:

  #+begin_src rec :data inventory.rec
  1
  #+end_src

  but it would be nice to avoid it.

- Is there a way to dinamically change the value of the :results
  parameter in the org-babel-execute:rec function?  I would like to add
  a :template parameter containing a template for recfmt, and in that
  case the default value "raw" would not be appropriate.

Thanks in advance :)

[1] http://www.gnu.org/software/recutils


[-- Attachment #2: ob-rec.el --]
[-- Type: application/emacs-lisp, Size: 2025 bytes --]

[-- Attachment #3: Type: text/plain, Size: 80 bytes --]


-- 
Jose E. Marchesi    jemarch@gnu.org
GNU Project         http://www.gnu.org

[-- Attachment #4: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: [BABEL] ob-rec.el and some questions

[Eric Schulte]

[-- Attachment #1: Type: text/plain, Size: 3172 bytes --]

jemarch@gnu.org (Jose E. Marchesi) writes:

> Hi.
>
> I have been playing a bit with org babel, the goal being to be able to
> query a recfile[1] and insert the result as a rec table.  Thanks to the
> superb design of org-babel and the existing examples, I quickly wrote
> the attached little hack.
>
> The hack allows to query a rec file 'foo.rec' as follows:
>
> #+begin_src rec :data hackers.rec :fields Name,Email :type Hacker
> Papers = 'requested' && CreatedAt << '01 January 2011'
> #+end_src
>

Very cool!  Now is the time when I ask, would you consider adding this
to the Org-mode core, which requires filling out the FSF copyright
assignment form?  http://orgmode.org/worg/org-contribute.html#sec-2

>
> Where :data points to the recfile (or list of files), :fields is a
> comma-separated list of fields (allowing subscripts) and :type selects
> the record set where to make the query. :cmdline can also be used to
> specify any other command line option to recsel.  All parameters but
> :data are optional.
>
> The selection expression can be any expression allowed by recsel in the
> -e command line option.
>
> Of course you have to install the recutils for it to work.
>
> It roughly works and I want to add more functionality, but due to my
> lack of experience with org-babel I would like to ask some questions
> before continuing working on it.
>
> - Is there a way to execute empty code blocks?  It would be quite common
>   to require all the records stored in a record set, e.g.
>
>   #+begin_src rec :data inventory.rec
>   #+end_src
>
>   C-cC-c in that block echoes "Local setup has been refreshed" and
>   nothing happens.  A workaround would be to use a selection expression
>   that always evaluates to "true", such as:
>

With my attached version of ob-rec.el it is possible to have a mostly
empty body as follows, however, currently `org-babel-get-src-block-info'
*does* require that there be at least 1 character as well as a newline
in the body.  This could be changed, however I'd want to do some local
testing before pushing up such a change, as I fear it may break existing
code blocks.

#+begin_src rec :data book.rec :fields Title,Author
  
#+end_src

#+results:
| Title...
| GNU Emacs Manual...

>
>   #+begin_src rec :data inventory.rec
>   1
>   #+end_src
>
>   but it would be nice to avoid it.
>
> - Is there a way to dinamically change the value of the :results
>   parameter in the org-babel-execute:rec function?  I would like to add
>   a :template parameter containing a template for recfmt, and in that
>   case the default value "raw" would not be appropriate.
>

Please see my attached revision of ob-rec.el.  I do not believe that the
default value of raw is appropriate currently as Babel should know when
the returned results are a table.  I've slightly modified your existing
code so that raw is no longer the default value, and so that tables are
passed back as tables.  Does this change subsume your format question?

>

Also, it sounds like there is currently only support for selecting
records *from* a .rec file, do you plan on adding support for inserting
records *into* a .rec file?

Thanks for sharing! -- Eric


[-- Attachment #2: ob-rec.el --]
[-- Type: application/emacs-lisp, Size: 2319 bytes --]

[-- Attachment #3: Type: text/plain, Size: 71 bytes --]


>
> Thanks in advance :)
>
> [1] http://www.gnu.org/software/recutils

[-- Attachment #4: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: [BABEL] ob-rec.el and some questions

[Jose E. Marchesi]

    > #+begin_src rec :data hackers.rec :fields Name,Email :type Hacker
    > Papers = 'requested' && CreatedAt << '01 January 2011'
    > #+end_src
    >
    
    Very cool!  Now is the time when I ask, would you consider adding this
    to the Org-mode core, which requires filling out the FSF copyright
    assignment form?  http://orgmode.org/worg/org-contribute.html#sec-2

Sure.  I assigned future changes to Emacs some years ago.

    >   C-cC-c in that block echoes "Local setup has been refreshed" and
    >   nothing happens.  A workaround would be to use a selection expression
    >   that always evaluates to "true", such as:
    >
    
    With my attached version of ob-rec.el it is possible to have a
    mostly empty body as follows, however, currently
    `org-babel-get-src-block-info' *does* require that there be at least
    1 character as well as a newline in the body.  This could be
    changed, however I'd want to do some local testing before pushing up
    such a change, as I fear it may break existing code blocks.

That looks like a reasonable workaround.

    > - Is there a way to dinamically change the value of the :results
    >   parameter in the org-babel-execute:rec function?  I would like to add
    >   a :template parameter containing a template for recfmt, and in that
    >   case the default value "raw" would not be appropriate.
    >
    
    Please see my attached revision of ob-rec.el.  I do not believe that the
    default value of raw is appropriate currently as Babel should know when
    the returned results are a table.  I've slightly modified your existing
    code so that raw is no longer the default value, and so that tables are
    passed back as tables.  Does this change subsume your format
    question?

Yes, now it is clear: I have to adapt the returned data to the
result-params settings, and not the other way around.

Thanks for the fixes :)
    
    Also, it sounds like there is currently only support for selecting
    records *from* a .rec file, do you plan on adding support for inserting
    records *into* a .rec file?

Extracting the records from org tables or raw rec data, you mean?  It
would be easy to use csv2rec and recins to implement a recins block:

#+begin_src recins :data foo.rec
| Title            | Author              |
| GNU Emacs Manual | Richard M. Stallman |
...
#+end_src

or

#+begin_src recins :data foo.rec
%rec: Book
...

Title: GNU Emacs Manual
Author: Richard M. Stallman

...
#+end_src

By evaluating all the code blocks in the file in a sequential way,
additional "rec" blocks could then be used to make (and publish)
selections of the table inserted above:

#+begin_src rec :data foo.rec
some selection expression
#+end_src

#+results
| Title            | Author              |
| ...              | ...                 |


That sounds like fun! org-mode could then be used as a GUI for recutils
:D

-- 
Jose E. Marchesi    jemarch@gnu.org
GNU Project         http://www.gnu.org

Re: [BABEL] ob-rec.el and some questions

[Eric Schulte]

jemarch@gnu.org (Jose E. Marchesi) writes:

>     > #+begin_src rec :data hackers.rec :fields Name,Email :type Hacker
>     > Papers = 'requested' && CreatedAt << '01 January 2011'
>     > #+end_src
>     >
>     
>     Very cool!  Now is the time when I ask, would you consider adding this
>     to the Org-mode core, which requires filling out the FSF copyright
>     assignment form?  http://orgmode.org/worg/org-contribute.html#sec-2
>
> Sure.  I assigned future changes to Emacs some years ago.
>

Ah, I should have noticed the @gnu in your email address.  That's great,
once this file settles down I'll add it to the repository.

>
>     >   C-cC-c in that block echoes "Local setup has been refreshed" and
>     >   nothing happens.  A workaround would be to use a selection expression
>     >   that always evaluates to "true", such as:
>     >
>     
>     With my attached version of ob-rec.el it is possible to have a
>     mostly empty body as follows, however, currently
>     `org-babel-get-src-block-info' *does* require that there be at least
>     1 character as well as a newline in the body.  This could be
>     changed, however I'd want to do some local testing before pushing up
>     such a change, as I fear it may break existing code blocks.
>
> That looks like a reasonable workaround.
>
>     > - Is there a way to dinamically change the value of the :results
>     >   parameter in the org-babel-execute:rec function?  I would like to add
>     >   a :template parameter containing a template for recfmt, and in that
>     >   case the default value "raw" would not be appropriate.
>     >
>     
>     Please see my attached revision of ob-rec.el.  I do not believe that the
>     default value of raw is appropriate currently as Babel should know when
>     the returned results are a table.  I've slightly modified your existing
>     code so that raw is no longer the default value, and so that tables are
>     passed back as tables.  Does this change subsume your format
>     question?
>
> Yes, now it is clear: I have to adapt the returned data to the
> result-params settings, and not the other way around.
>

well put

>
> Thanks for the fixes :)
>     
>     Also, it sounds like there is currently only support for selecting
>     records *from* a .rec file, do you plan on adding support for inserting
>     records *into* a .rec file?
>
> Extracting the records from org tables or raw rec data, you mean?  It
> would be easy to use csv2rec and recins to implement a recins block:
>

Yes, I was thinking of something like the following.  Rather than create
a new block type, simply add a header argument (or body content) that
(somehow) indicates that we are writing not reading.  So for example
writing an Org-mode table into a fresh .rec file would look like the
following.

#+tblname: org-data
| Foo | Bar |
|-----+-----|
|   1 | a   |
|   2 | b   |
|   3 | c   |
|   4 | d   |

#+begin_src rec :data something.rec :var data=org-data :write
  some rec code to dump data into the rec file...
#+end_src

[...]
>
> By evaluating all the code blocks in the file in a sequential way,
> additional "rec" blocks could then be used to make (and publish)
> selections of the table inserted above:
>

Yes, and once two-way movement of data is possible, then other languages
could be used to update the contents of column in rec-files from within
an Org-mode document.  Running an arbitrary python function over the
"Foo" column of a .rec file and saving the results should be as easy as
evaluating the write-foos code block below.

1. Select Foo for all records
   #+source: all-foos
   #+begin_src rec :data something.rec :fields Foo
     1
   #+end_src

2. Run all foos through some arbitrary processing in any Babel language
   #+source: process-foos
   #+begin_src python :var foos=all-foos
     do something to foos
   #+end_src

3. Save the processed values back into the foos column of the original
   rec file
   #+source: write-foos
   #+begin_src rec :data something.rec :fields Foo :write :var foos=process-foos
     insert foos
   #+end_src

Hope this makes sense.  Best -- Eric

>
> #+begin_src rec :data foo.rec
> some selection expression
> #+end_src
>
> #+results
> | Title            | Author              |
> | ...              | ...                 |
>
>
> That sounds like fun! org-mode could then be used as a GUI for recutils
> :D

Re: [BABEL] ob-rec.el and some questions

[Jose E. Marchesi]

    > Extracting the records from org tables or raw rec data, you mean?  It
    > would be easy to use csv2rec and recins to implement a recins block:
    >
    
    Yes, I was thinking of something like the following.  Rather than create
    a new block type, simply add a header argument (or body content) that
    (somehow) indicates that we are writing not reading.  So for example
    writing an Org-mode table into a fresh .rec file would look like the
    following.
    
    #+tblname: org-data
    | Foo | Bar |
    |-----+-----|
    |   1 | a   |
    |   2 | b   |
    |   3 | c   |
    |   4 | d   |

    #+begin_src rec :data something.rec :var data=org-data :write
      some rec code to dump data into the rec file...
    #+end_src

I like that approach, but I would put the raw rec code into the src
block:

#+begin_src rec :data something.rec :write
Foo: 1
Bar: a

Foo: 2
Bar: b

...  
#+end_src

Then use a :table parameter in case we want to insert the contents of a
table:

#+begin_src rec :data something.rec :write :table org-data
  
#+end_src

I will work on this and will be back as soon as I have something
working.  Many thanks for the feedback :)

-- 
Jose E. Marchesi    jemarch@gnu.org
GNU Project         http://www.gnu.org