Exploring properties

Exploring properties

[Vikas Rawal]

[-- Attachment #1: Type: text/plain, Size: 356 bytes --]

I don't know if I have missed something obvious. But I am trying to read
all the properties data (key-value pairs) into an R source code block
(essentially to gather into a data.frame before further processing).

I thought properties would be a neat way to put together a small database,
and then process it in R.

Would appreciate a clue.

Warmly,

Vikas

[-- Attachment #2: Type: text/html, Size: 493 bytes --]

Re: Exploring properties

[Bastien]

Hi Vikas,

Vikas Rawal <vikasrawal@gmail.com> writes:

> I don't know if I have missed something obvious. But I am trying to
> read all the properties data (key-value pairs) into an R source code
> block (essentially to gather into a data.frame before further
> processing).

You might want to look at `org-map-entries'.

HTH,

-- 
 Bastien

Re: Exploring properties

[Vikas Rawal]

>
> > I don't know if I have missed something obvious. But I am trying to
> > read all the properties data (key-value pairs) into an R source code
> > block (essentially to gather into a data.frame before further
> > processing).
>
> You might want to look at `org-map-entries'.

Thanks. It seems it will take more lisp than I can handle. But let me try.

I think there is a potential to combine orgmode tags and properties, with a little more lisp may be, along with version control to create small database applications.

It would be good to design something as a proof-of-concept.

Vikas

Re: Exploring properties

[John Kitchin]

[-- Attachment #1: Type: text/plain, Size: 1324 bytes --]

org-ql (https://github.com/alphapapa/org-ql) is also quite good for this
kind of thing.

Here is even a database like query to find the heading, and EMAIL property
from all the files in a list returned by the function nb-org-files.

(org-ql-query
  :select '(cons (fifth (org-heading-components))
 (org-entry-get (point) "EMAIL"))
  :from (nb-org-files)
  :where '(and (property "EMAIL")))


John

-----------------------------------
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu



On Sun, Feb 23, 2020 at 8:25 AM Vikas Rawal <vikasrawal@gmail.com> wrote:

>
> >
> > > I don't know if I have missed something obvious. But I am trying to
> > > read all the properties data (key-value pairs) into an R source code
> > > block (essentially to gather into a data.frame before further
> > > processing).
> >
> > You might want to look at `org-map-entries'.
>
> Thanks. It seems it will take more lisp than I can handle. But let me try.
>
> I think there is a potential to combine orgmode tags and properties, with
> a little more lisp may be, along with version control to create small
> database applications.
>
> It would be good to design something as a proof-of-concept.
>
> Vikas
>
>

[-- Attachment #2: Type: text/html, Size: 2160 bytes --]

Re: Exploring properties

[Vikas Rawal]

>
> You might want to look at `org-map-entries'.
>
Would you think that an extension of org-collector is the direction to look at? Doesn't look like org-collector has seen any updates in a long time.

Vikas

Re: Exploring properties

[Bastien]

Hi Vikas,

Vikas Rawal <vikasrawal@gmail.com> writes:

> Would you think that an extension of org-collector is the direction to
> look at? Doesn't look like org-collector has seen any updates in a
> long time.

You might want to look at recutils instead.

It was announced on org-mode mailing list here 10 years ago:
https://lists.gnu.org/archive/html/emacs-orgmode/2010-12/msg00219.html

Jose is still developing it actively - it made the news on HN
recently: https://news.ycombinator.com/item?id=22153665

There is *some* integration with org-mode, although I have not tested
it. I've also found this, but this seems outdated and not maintained:
https://github.com/millarc/ob-rec.el

I'm cc'ing Jose as he may be able to share his own wishlist on how Org
could better interact with recutils.

HTH,

-- 
 Bastien

Re: Exploring properties

[Vikas Rawal]

[-- Attachment #1: Type: text/plain, Size: 1228 bytes --]

Thanks Bastien for advice and for connecting with Jose. It would be useful
to write some lisp for inserting and adding data from org into rec files.
It could either be from org-mode tables or from properties. I am
essentially thinking of org-mode providing an interface for feeding data to
recutils.

Vikas

On Sun, 23 Feb 2020 at 19:10, Bastien <bzg@gnu.org> wrote:

> Hi Vikas,
>
> Vikas Rawal <vikasrawal@gmail.com> writes:
>
> > Would you think that an extension of org-collector is the direction to
> > look at? Doesn't look like org-collector has seen any updates in a
> > long time.
>
> You might want to look at recutils instead.
>
> It was announced on org-mode mailing list here 10 years ago:
> https://lists.gnu.org/archive/html/emacs-orgmode/2010-12/msg00219.html
>
> Jose is still developing it actively - it made the news on HN
> recently: https://news.ycombinator.com/item?id=22153665
>
> There is *some* integration with org-mode, although I have not tested
> it. I've also found this, but this seems outdated and not maintained:
> https://github.com/millarc/ob-rec.el
>
> I'm cc'ing Jose as he may be able to share his own wishlist on how Org
> could better interact with recutils.
>
> HTH,
>
> --
>  Bastien
>

[-- Attachment #2: Type: text/html, Size: 2012 bytes --]

Re: Exploring properties

[Bastien]

Hi Vikas,

Vikas Rawal <vikasrawal@gmail.com> writes:

> I am essentially thinking of org-mode providing an
> interface for feeding data to recutils.

Yes, that would probably be useful.

If you want to write a feature request, please write it as if the
reader does not know recutils and you precise use-case, so that we
understand all implication and motivate possible contributors.

Thanks,

-- 
 Bastien

Re: Exploring properties

[Vikas Rawal]

I don't know if this is useful. But this is what I could come up with. This might at least motivate somebody to think of other possible advantages/uses of building org-mode capabilities to interact with databases.

Vikas

---

Feature request: To build tools to facilitate using org-mode as a front-end for interacting with a database.

The idea would be to use org-mode to select, insert and update records in a database. It is natural to think of recutils, a plain text database, as the database backend. But a more generalised solution may allow other choices of databases. With a text-based database backend like recutils, we could harness a version control system like git to semi-automate collaboraion.

Why would one use org-mode for creating a database application? One can think of many advantages. But the biggest advantage (to me) would be that, in this setup, a multi-user database application can work without constant internet connectivity. You can query recutils files, insert/update records, and then let git deal with synchronisation across the team members.

There are other advantages as well. We do not yet have a text-based, easy-to-deploy database application system. Recutils provides the infrastructure for a backend. But we do not have a fully-developed front-end system to interact with recutils. Org-mode is clearly best placed to provide the frontend for working with recutils.

Needless to say, this would be particularly interesting to members of the org-mode/emacs fan-club. Over the last few years, org-mode has come to be used for many tasks that go way beyond what Carsten had in mind when he first built org-mode. From web-publishing to writing books, org-mode provides excellent tools. Being able to create and use databases from within org-mode would be a very useful addition to this toolkit.

We already have tools that can be used to read data from recutils (and other databases) and create reports in org-mode. The missing feature is to be able to use org-mode to create/update records systemmatically.

There are two possibilities here: to use org-mode tables or org-mode properties to interact with the database. The advantage of doing this using the org-mode properties is that the column-view of properties provides an easy to use interface for entering data. There is already a mechanism for defining "Allowed" values for any field which speeds up data entry and helps avoid typing errors.

There are several challenges. Some of these are:

1. Extending "Allowed" values to specify type of data that can be recorded (numeric, char) or range of values.
2. How to deal with relationships/foreign keys. Can property inheritance be used to deal with at least simple foreign key constraints?
3. Org-mode macros provide {{{property(PROPERTYNAME)}}} syntax for macro replacement during export. But nothing as simple as this is available for use in source code blocks. Something like this would allow using some code to add/update records in a database. We should perhaps build on the property API or org-ql for creating something like this.

If we can resolve some of these and creating a simple application for demonstration, it might help in showing the potential and identifying other challenges.

On Mon, Feb 24, 2020 at 09:39:51AM +0100, Bastien wrote:
> Hi Vikas,
>
> Vikas Rawal <vikasrawal@gmail.com> writes:
>
> > I am essentially thinking of org-mode providing an
> > interface for feeding data to recutils.
>
> Yes, that would probably be useful.
>
> If you want to write a feature request, please write it as if the
> reader does not know recutils and you precise use-case, so that we
> understand all implication and motivate possible contributors.
>
> Thanks,
>
> --
>  Bastien
>

Re: Exploring properties

[John Kitchin]

You might take a look at ebib, which is something like this for bibtex
files (also a plain text db). You can bend bibtex into being a database,
with custom entry types. What ebib does for bibtex might also be
possible for recutils too.

Similarly, bbdb offered an emacsy interface to a database of contacts (I
think just stored as elisp code).

elfeed has a git like database structure it uses to store things in, and
a front-end for displaying them.

For recutils, one could make a temporary org buffer full of recutil
entries in org syntax. Then, you could edit it, and write it back out, I
guess from a save-buffer-hook. The main benefit of this in my mind is
editing many records at a time, leveraging the search/navigation/outline
view of org-mode. I guess a row in a table per entry is all that could
be reliably done without a custom template that would allow you to
create headings per row. A table might be awkward for multiline entries
though. There is already a rec-mode with recutils that provides features
for editing, navigating, etc. within a rec file. I guess you can't edit
several entries at a time though.

The downside of headings might be then you also need custom code to
write it out. I guess this also goes by creating a list of sexp entries
that can be written out.

One simple application might be a recutils contact database, based on
org-contacts. These are basically headings with EMAIL properties, where
the heading is the contact name. You might also save tags in the entry,
and any other properties. You could store the body of the heading as
well. Some one will have to see how it scales with contacts, if it is
easy to search, etc.

Why contacts? org-contacts works ok, if you don't have too many, and
they aren't in too many files. I have 5000+ org-contacts spread all over
the place, so I have to use a cache (i.e. db) that is fast to query to
use them. If recutils was good for that, it might work for me.


Vikas Rawal <vikasrawal@gmail.com> writes:

> I don't know if this is useful. But this is what I could come up with. This might at least motivate somebody to think of other possible advantages/uses of building org-mode capabilities to interact with databases.
>
> Vikas
>
> ---
>
> Feature request: To build tools to facilitate using org-mode as a front-end for interacting with a database.
>
> The idea would be to use org-mode to select, insert and update records in a database. It is natural to think of recutils, a plain text database, as the database backend. But a more generalised solution may allow other choices of databases. With a text-based database backend like recutils, we could harness a version control system like git to semi-automate collaboraion.
>
> Why would one use org-mode for creating a database application? One can think of many advantages. But the biggest advantage (to me) would be that, in this setup, a multi-user database application can work without constant internet connectivity. You can query recutils files, insert/update records, and then let git deal with synchronisation across the team members.
>
> There are other advantages as well. We do not yet have a text-based,
> easy-to-deploy database application system. Recutils provides the
> infrastructure for a backend. But we do not have a fully-developed
> front-end system to interact with recutils. Org-mode is clearly best
> placed to provide the frontend for working with recutils.

>
> Needless to say, this would be particularly interesting to members of the org-mode/emacs fan-club. Over the last few years, org-mode has come to be used for many tasks that go way beyond what Carsten had in mind when he first built org-mode. From web-publishing to writing books, org-mode provides excellent tools. Being able to create and use databases from within org-mode would be a very useful addition to this toolkit.
>
> We already have tools that can be used to read data from recutils (and other databases) and create reports in org-mode. The missing feature is to be able to use org-mode to create/update records systemmatically.
>
> There are two possibilities here: to use org-mode tables or org-mode properties to interact with the database. The advantage of doing this using the org-mode properties is that the column-view of properties provides an easy to use interface for entering data. There is already a mechanism for defining "Allowed" values for any field which speeds up data entry and helps avoid typing errors.
>
> There are several challenges. Some of these are:
>
> 1. Extending "Allowed" values to specify type of data that can be recorded (numeric, char) or range of values.
> 2. How to deal with relationships/foreign keys. Can property inheritance be used to deal with at least simple foreign key constraints?
> 3. Org-mode macros provide {{{property(PROPERTYNAME)}}} syntax for macro replacement during export. But nothing as simple as this is available for use in source code blocks. Something like this would allow using some code to add/update records in a database. We should perhaps build on the property API or org-ql for creating something like this.
>
> If we can resolve some of these and creating a simple application for demonstration, it might help in showing the potential and identifying other challenges.
>
> On Mon, Feb 24, 2020 at 09:39:51AM +0100, Bastien wrote:
>> Hi Vikas,
>>
>> Vikas Rawal <vikasrawal@gmail.com> writes:
>>
>> > I am essentially thinking of org-mode providing an
>> > interface for feeding data to recutils.
>>
>> Yes, that would probably be useful.
>>
>> If you want to write a feature request, please write it as if the
>> reader does not know recutils and you precise use-case, so that we
>> understand all implication and motivate possible contributors.
>>
>> Thanks,
>>
>> --
>>  Bastien
>>


--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu