* [OT] recutils @ 2010-12-03 21:16 Jose E. Marchesi 2010-12-04 7:17 ` Michael Brand 0 siblings, 1 reply; 10+ messages in thread From: Jose E. Marchesi @ 2010-12-03 21:16 UTC (permalink / raw) To: emacs-orgmode Hi. I apologize in advance for the off-topic, but this could be of interest to people liking to "organize their life" in plain text. This is about "storing your data" in plain text :D I just made the first release of the GNU recutils, a set of tools and libraries to access human-editable, text-based databases called recfiles. The data is stored as a sequence of records, each record containing an arbitrary number of named fields. Advanced capabilities usually found in other data storage systems are supported by GNU recutils: data types, data integrity (keys, mandatory fields, etc) as well as the ability of records to refer to other records (sort of foreign keys). Despite its simplicity, recfiles can be used to store medium-sized databases. I have been using org-mode for years and it rocks. In some way it changed my life. It is perfect to store all kind of hierarchical data. At some point I though about applying a similar philosophy to "sequential" repetitive data with relationships. Data that we would store in a relational database. A nice complement to org. The recutils are the result of those thoughts. I would highly appreciate any comment or suggestion on improving the format, the utilities, or whatever. Please see the GNU recutils homepage for more information: http://www.gnu.org/software/recutils (due to the crack of savannah the webpages replication facility is not yet working, so the page there is old and does not reflect the release). The tarball can be downloaded from ftp://ftp.gnu.org/gnu/recutils/recutils-1.0.tar.gz The GNU recutils suite comprises: - A texinfo manual, describing the Rec format and the accompaying software. - A C library (librec) that provides a rich set of functions to manipulate rec data. - A set of C utilities that can be used in shell scripts and in the command line to operate on rec files: + `recinf' prints information about the contents of recfiles. + `recsel' prints records and field values. + `recins' adds new records to a recfile. + `recdel' deletes records. + `recset' manipulates fields. + `recfix' checks and fixes recfiles. + `csv2rec' converts comma-separated-values files to recfiles. + `mdb2rec' converts mdb files (used by MS-Access) to recfiles. - rec-mode.el - An emacs mode to edit and navigate recfiles. Still experimental and without a manual. Whatever... sorry for the spam! XD -- Jose E. Marchesi jemarch@gnu.org GNU Project http://www.gnu.org ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [OT] recutils 2010-12-03 21:16 [OT] recutils Jose E. Marchesi @ 2010-12-04 7:17 ` Michael Brand 2010-12-04 8:25 ` Russell Adams 2010-12-04 12:28 ` Jose E. Marchesi 0 siblings, 2 replies; 10+ messages in thread From: Michael Brand @ 2010-12-04 7:17 UTC (permalink / raw) To: Jose E. Marchesi; +Cc: emacs-orgmode On Fri, Dec 3, 2010 at 22:16, Jose E. Marchesi <jemarch@gnu.org> wrote: > I apologize in advance for the off-topic, but this could be of interest > to people liking to "organize their life" in plain text. This is about > "storing your data" in plain text :D > > I just made the first release of the GNU recutils, a set of tools and > libraries to access human-editable, text-based databases called > recfiles. The data is stored as a sequence of records, each record > containing an arbitrary number of named fields. Advanced capabilities > usually found in other data storage systems are supported by GNU > recutils: data types, data integrity (keys, mandatory fields, etc) as > well as the ability of records to refer to other records (sort of > foreign keys). Despite its simplicity, recfiles can be used to store > medium-sized databases. > > I have been using org-mode for years and it rocks. In some way it > changed my life. It is perfect to store all kind of hierarchical data. > At some point I though about applying a similar philosophy to > "sequential" repetitive data with relationships. Data that we would > store in a relational database. A nice complement to org. The recutils > are the result of those thoughts. > > I would highly appreciate any comment or suggestion on improving the > format, the utilities, or whatever. Hi Jose Wow! The ability to have a _relational_ database with foreign keys in a simple text file is so great news for me. A dream I had for many years now. Or is it lack of knowledge from my side about already existing solutions before recutils? Thank you a lot for your work, sharing it to the community, posting it also on this list here and for asking for suggestions which I would like to give at the end. I was thinking about using sqlite from the command line and together with shell scripts for stuff like my collection of music with recordings, MIDI files, scores and so on. Because at least for me editing a text file is by far simpler, more interactive and more convenient than editing with SQL I will prefer recutils over sqlite. But since recfiles, the text files of recutils, are not hierarchically organized I am still considering to use only Org for the case of my collection of music. Org would have the advantages of outlining, hyperlinks, column view, todo, tags, agenda view, export for publishing and many others. Thus my wish for the file format would be to somehow keep in mind Org to potentially - convert the files bidirectionally between the format of recfiles and Org, with or without something like "literate programming" of org-babel - use recutils as a language extension to org-babel, recutils reading data streamed to its stdin from org-babel and piping back to into org-babel result (no writing by recutils to the file) - as an imagination: run recutils on a file in the format of Org, only reading or even read/write. I will follow up with an example of what I plan to do for my collection of music, the database table relations squeezed hierarchically into Org as a first attempt. Michael ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [OT] recutils 2010-12-04 7:17 ` Michael Brand @ 2010-12-04 8:25 ` Russell Adams 2010-12-04 12:30 ` Jose E. Marchesi 2010-12-04 12:28 ` Jose E. Marchesi 1 sibling, 1 reply; 10+ messages in thread From: Russell Adams @ 2010-12-04 8:25 UTC (permalink / raw) To: emacs-orgmode > But since recfiles, the text files of recutils, are not hierarchically > organized I am still considering to use only Org for the case of my > collection of music. Org would have the advantages of outlining, > hyperlinks, column view, todo, tags, agenda view, export for > publishing and many others. > > Thus my wish for the file format would be to somehow keep in mind Org > to potentially I noticed the file format is very similar to the layout of property drawers. Could recutils read an org file using those drawers with minor changes? That'd be a neat way to externally cross reference Org files and do reporting! Thanks. ------------------------------------------------------------------ Russell Adams RLAdams@AdamsInfoServ.com PGP Key ID: 0x1160DCB3 http://www.adamsinfoserv.com/ Fingerprint: 1723 D8CA 4280 1EC9 557F 66E8 1154 E018 1160 DCB3 ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [OT] recutils 2010-12-04 8:25 ` Russell Adams @ 2010-12-04 12:30 ` Jose E. Marchesi 0 siblings, 0 replies; 10+ messages in thread From: Jose E. Marchesi @ 2010-12-04 12:30 UTC (permalink / raw) To: emacs-orgmode > Thus my wish for the file format would be to somehow keep in mind Org > to potentially I noticed the file format is very similar to the layout of property drawers. Could recutils read an org file using those drawers with minor changes? That'd be a neat way to externally cross reference Org files and do reporting! The parser implemented by librec would need a lot of changes to achieve this. But you can always pre-process the org files in a wrapper, that would remove anything that is not a drawer and the indentation. -- Jose E. Marchesi jemarch@gnu.org GNU Project http://www.gnu.org ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [OT] recutils 2010-12-04 7:17 ` Michael Brand 2010-12-04 8:25 ` Russell Adams @ 2010-12-04 12:28 ` Jose E. Marchesi 2010-12-04 18:50 ` Russell Adams ` (2 more replies) 1 sibling, 3 replies; 10+ messages in thread From: Jose E. Marchesi @ 2010-12-04 12:28 UTC (permalink / raw) To: Michael Brand; +Cc: emacs-orgmode > I would highly appreciate any comment or suggestion on improving the > format, the utilities, or whatever. Wow! The ability to have a _relational_ database with foreign keys in a simple text file is so great news for me. A dream I had for many years now. Or is it lack of knowledge from my side about already existing solutions before recutils? I don't think there is anything similar. Initially I just wanted a simple format to store fields in readable and writable files. Something really simple. Then additional features came to my mind and I implemented them. But note that the "relational" characteristics in recutils are not very sophisticated. You can't have keys composed by more than one field, for example. The rule here is: if you need something more complex then you probably should be using a real relational dbms instead :) I was thinking about using sqlite from the command line and together with shell scripts for stuff like my collection of music with recordings, MIDI files, scores and so on. Because at least for me editing a text file is by far simpler, more interactive and more convenient than editing with SQL I will prefer recutils over sqlite. Additionally, text files are tool-independent. You could even print your database :) But since recfiles, the text files of recutils, are not hierarchically organized I am still considering to use only Org for the case of my collection of music. Org would have the advantages of outlining, hyperlinks, column view, todo, tags, agenda view, export for publishing and many others. Thus my wish for the file format would be to somehow keep in mind Org to potentially - convert the files bidirectionally between the format of recfiles and Org, with or without something like "literate programming" of org-babel I don't think it is generally possible to map the relationships between record types in a recfile to hierarchies in an org file. For example, in a recfile you could have something like: %rec: Album %key: Name %type: Year date Name: Loving You Year: 1957 Author:Name: Elvis Presley ... %rec: Author %key: Name Name: Elvis Presley You could extract: * Albums ** Elvis Presley *** Loving You But then, what if several authors authored an album? %rec: Album Name: Sounds of Silence Author:Name: Simon Author:Name: Garfunkel You could group by Album or by Author then. - use recutils as a language extension to org-babel, recutils reading data streamed to its stdin from org-babel and piping back to into org-babel result (no writing by recutils to the file) That sounds interesting. Would be nice to store rec data into the org files. Thanks for the suggestions :) -- Jose E. Marchesi jemarch@gnu.org GNU Project http://www.gnu.org ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [OT] recutils 2010-12-04 12:28 ` Jose E. Marchesi @ 2010-12-04 18:50 ` Russell Adams 2010-12-05 18:59 ` Michael Brand 2010-12-13 12:33 ` Jean-Marie Gaillourdet 2 siblings, 0 replies; 10+ messages in thread From: Russell Adams @ 2010-12-04 18:50 UTC (permalink / raw) To: emacs-orgmode On Sat, Dec 04, 2010 at 01:28:39PM +0100, Jose E. Marchesi wrote: > You could extract: > > * Albums > ** Elvis Presley > *** Loving You > > But then, what if several authors authored an album? > > %rec: Album > > Name: Sounds of Silence > Author:Name: Simon > Author:Name: Garfunkel > * Albums ** Album 1 :PROPERTIES: :NAME: Sounds of Silence :BAND: The Beegees :YEAR: 1912 :NOTE: I haven't a clue about these bands :END: ** Album 2 :PROPERTIES: :NAME: Ethel the Aardvark goes quantity surveying :BAND: Monty Python :YEAR: 1978 :NOTE: A musical adaptation of a children's book :END: An operation across all child topics of "Albums" ought not be difficult. Just an idea. ------------------------------------------------------------------ Russell Adams RLAdams@AdamsInfoServ.com PGP Key ID: 0x1160DCB3 http://www.adamsinfoserv.com/ Fingerprint: 1723 D8CA 4280 1EC9 557F 66E8 1154 E018 1160 DCB3 ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [OT] recutils 2010-12-04 12:28 ` Jose E. Marchesi 2010-12-04 18:50 ` Russell Adams @ 2010-12-05 18:59 ` Michael Brand 2010-12-05 20:20 ` Jose E. Marchesi 2010-12-13 12:33 ` Jean-Marie Gaillourdet 2 siblings, 1 reply; 10+ messages in thread From: Michael Brand @ 2010-12-05 18:59 UTC (permalink / raw) To: Jose E. Marchesi; +Cc: emacs-orgmode Hi Jose On Sat, Dec 4, 2010 at 13:28, Jose E. Marchesi <jemarch@gnu.org> wrote: > But note that the "relational" characteristics in > recutils are not very sophisticated. > > The rule here is: if you need something more complex then you probably > should be using a real relational dbms instead :) Ok, maybe I expected a bit too much for the initial version 1.0. But is it already possible or planned to allow recsel deliver field values from two tables, joined together with a foreign key or what does the manual mean with "the ability of records to refer to other records (sort of foreign keys)"? Michael ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [OT] recutils 2010-12-05 18:59 ` Michael Brand @ 2010-12-05 20:20 ` Jose E. Marchesi 0 siblings, 0 replies; 10+ messages in thread From: Jose E. Marchesi @ 2010-12-05 20:20 UTC (permalink / raw) To: Michael Brand; +Cc: emacs-orgmode On Sat, Dec 4, 2010 at 13:28, Jose E. Marchesi <jemarch@gnu.org> wrote: > But note that the "relational" characteristics in > recutils are not very sophisticated. > > The rule here is: if you need something more complex then you probably > should be using a real relational dbms instead :) Ok, maybe I expected a bit too much for the initial version 1.0. But is it already possible or planned to allow recsel deliver field values from two tables, joined together with a foreign key or what does the manual mean with "the ability of records to refer to other records (sort of foreign keys)"? Right now the recutils use the information of the "foreign keys" (compound fields) just to pick the type of the field in the "referenced" record descriptor. For the next version it is planned to support joins in recsel. For example, having a database like: %rec: Hacker %key: Name Name: Mr. foo %rec: Package Name: GNU foo Hacker:Name: Mr. foo Name: GNU bar Hacker:Name: Mr. foo It will be possible to invoke recsel like: $ recsel -t Hacker -p Name,Package:Name foo.rec Name: Mr. Foobar Package:Name: GNU foo Name: Mr. Foobar Package:Name: GNU bar -- Jose E. Marchesi jemarch@gnu.org GNU Project http://www.gnu.org ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [OT] recutils 2010-12-04 12:28 ` Jose E. Marchesi 2010-12-04 18:50 ` Russell Adams 2010-12-05 18:59 ` Michael Brand @ 2010-12-13 12:33 ` Jean-Marie Gaillourdet 2010-12-20 18:09 ` Michael Brand 2 siblings, 1 reply; 10+ messages in thread From: Jean-Marie Gaillourdet @ 2010-12-13 12:33 UTC (permalink / raw) To: Jose E. Marchesi; +Cc: Michael Brand, emacs-orgmode Hi, jemarch@gnu.org (Jose E. Marchesi) writes: > > I would highly appreciate any comment or suggestion on improving the > > format, the utilities, or whatever. > > Wow! The ability to have a _relational_ database with foreign keys > in a simple text file is so great news for me. A dream I had for > many years now. Or is it lack of knowledge from my side about > already existing solutions before recutils? > > I don't think there is anything similar. You might be interested to hear about txt-sushi [1], a command line utility to apply SQL statements on a set of csv files, each of which represents one table. [1] http://keithsheppard.name/txt-sushi/ Cheers, Jean-Marie ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [OT] recutils 2010-12-13 12:33 ` Jean-Marie Gaillourdet @ 2010-12-20 18:09 ` Michael Brand 0 siblings, 0 replies; 10+ messages in thread From: Michael Brand @ 2010-12-20 18:09 UTC (permalink / raw) To: emacs-orgmode; +Cc: Jose E. Marchesi On Mon, Dec 13, 2010 at 13:33, Jean-Marie Gaillourdet <jmg@gaillourdet.net> wrote: > You might be interested to hear about txt-sushi [1], a command line > utility to apply SQL statements on a set of csv files, each of which > represents one table. > > [1] http://keithsheppard.name/txt-sushi/ Thank you for pointing this out. And there in the link section is also mentioned "You can now use CSV files as the storage backend for MySQL": http://dev.mysql.com/tech-resources/articles/csv-storage-engine.html Michael ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2010-12-20 18:10 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2010-12-03 21:16 [OT] recutils Jose E. Marchesi 2010-12-04 7:17 ` Michael Brand 2010-12-04 8:25 ` Russell Adams 2010-12-04 12:30 ` Jose E. Marchesi 2010-12-04 12:28 ` Jose E. Marchesi 2010-12-04 18:50 ` Russell Adams 2010-12-05 18:59 ` Michael Brand 2010-12-05 20:20 ` Jose E. Marchesi 2010-12-13 12:33 ` Jean-Marie Gaillourdet 2010-12-20 18:09 ` Michael Brand
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).