* Gather properties for use by babel source block?
@ 2011-08-02 23:57 Tim Burt
2011-08-03 14:00 ` darcamo
2012-05-07 15:52 ` Colin Hall
0 siblings, 2 replies; 9+ messages in thread
From: Tim Burt @ 2011-08-02 23:57 UTC (permalink / raw)
To: emacs-orgmode
I want to gather data from properties into something that can be used by
a babel source block (e.g. plot the data). Searches in the manual,
worg, and gmane have not yielded the method, but my best guess is that
I've missed it. If so, this is simply a request for a pointer to the
documentation I should read.
BACKGROUND: I'm trying to complete a workflow where I capture data over
time and periodically operate with it (e.g. put into a database and
plot). The excellent documentation for orgmode has made certain parts
easy going, but the current obstacle is extracting properties that can
subsequently be operated on in babel source blocks.
+ [X] Capture data into properties using templates
+ [X] View tables of properties in org file
* [X] use columnview dynamic block
* [X] use org-collector
+ [ ] Gather properties for use by babel source block
+ [-] Operate on the gathered properties
* [-] SQL or SQLite
- [X] follow simple tutorials for proof of principle
- [ ] populate database with gathered properties
* [-] R
- [X] follow simple tutorials
- [ ] calculate statistics and populate database
- [ ] plot from database
- [ ] plot from table in org file
Good evening,
Tim
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: Gather properties for use by babel source block? 2011-08-02 23:57 Gather properties for use by babel source block? Tim Burt @ 2011-08-03 14:00 ` darcamo 2011-08-03 14:35 ` tcburt 2012-05-07 15:52 ` Colin Hall 1 sibling, 1 reply; 9+ messages in thread From: darcamo @ 2011-08-03 14:00 UTC (permalink / raw) To: Tim Burt; +Cc: emacs-orgmode I have not used it myself, but isn't org-collector exactly what you want [1]. "org-collector collects headline properties into tables with optional pre-processing" Once you have a table, you can easily pass it to code blocks for further processing with org-babel. [1] - http://orgmode.org/worg/org-contrib/org-collector.html -- Darlan 2011/8/2 Tim Burt <tcburt@rochester.rr.com>: > > I want to gather data from properties into something that can be used by > a babel source block (e.g. plot the data). Searches in the manual, > worg, and gmane have not yielded the method, but my best guess is that > I've missed it. If so, this is simply a request for a pointer to the > documentation I should read. > > BACKGROUND: I'm trying to complete a workflow where I capture data over > time and periodically operate with it (e.g. put into a database and > plot). The excellent documentation for orgmode has made certain parts > easy going, but the current obstacle is extracting properties that can > subsequently be operated on in babel source blocks. > + [X] Capture data into properties using templates > + [X] View tables of properties in org file > * [X] use columnview dynamic block > * [X] use org-collector > + [ ] Gather properties for use by babel source block > + [-] Operate on the gathered properties > * [-] SQL or SQLite > - [X] follow simple tutorials for proof of principle > - [ ] populate database with gathered properties > * [-] R > - [X] follow simple tutorials > - [ ] calculate statistics and populate database > - [ ] plot from database > - [ ] plot from table in org file > > Good evening, > Tim > > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Gather properties for use by babel source block? 2011-08-03 14:00 ` darcamo @ 2011-08-03 14:35 ` tcburt 2011-08-03 15:53 ` suvayu ali 0 siblings, 1 reply; 9+ messages in thread From: tcburt @ 2011-08-03 14:35 UTC (permalink / raw) To: darcamo@gmail.com; +Cc: emacs-orgmode Hello Darlan --- ---- "darcamo@gmail.com" <darcamo@gmail.com> wrote: > I have not used it myself, but isn't org-collector exactly what you want [1]. > "org-collector collects headline properties into tables with optional > pre-processing" Indeed org-collector has allowed me to extract properties into a table, and even without much knowledge of lisp (for the conditions) the results were promising. > Once you have a table, you can easily pass it to code blocks for further > processing with org-babel. While I have the table from org-collector I do not know how to "easily pass it to code blocks". I have an (untested) idea that if #+tblname preceded the table from org-collector, then a babel source block could operate on it. If this works then almost all is well and good, except for the small detail of getting the #+tblname automatically from the org-collector process. Thank you for the response, Tim > > [1] - http://orgmode.org/worg/org-contrib/org-collector.html > > -- > Darlan > > 2011/8/2 Tim Burt <tcburt@rochester.rr.com>: > > > > I want to gather data from properties into something that can be used by > > a babel source block (e.g. plot the data). Searches in the manual, > > worg, and gmane have not yielded the method, but my best guess is that > > I've missed it. If so, this is simply a request for a pointer to the > > documentation I should read. > > > > BACKGROUND: I'm trying to complete a workflow where I capture data over > > time and periodically operate with it (e.g. put into a database and > > plot). The excellent documentation for orgmode has made certain parts > > easy going, but the current obstacle is extracting properties that can > > subsequently be operated on in babel source blocks. > > + [X] Capture data into properties using templates > > + [X] View tables of properties in org file > > * [X] use columnview dynamic block > > * [X] use org-collector > > + [ ] Gather properties for use by babel source block > > + [-] Operate on the gathered properties > > * [-] SQL or SQLite > > - [X] follow simple tutorials for proof of principle > > - [ ] populate database with gathered properties > > * [-] R > > - [X] follow simple tutorials > > - [ ] calculate statistics and populate database > > - [ ] plot from database > > - [ ] plot from table in org file > > > > Good evening, > > Tim > > > > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Gather properties for use by babel source block? 2011-08-03 14:35 ` tcburt @ 2011-08-03 15:53 ` suvayu ali 0 siblings, 0 replies; 9+ messages in thread From: suvayu ali @ 2011-08-03 15:53 UTC (permalink / raw) To: tcburt; +Cc: emacs-orgmode Hello Tim, On Wed, Aug 3, 2011 at 4:35 PM, <tcburt@rochester.rr.com> wrote: > While I have the table from org-collector I do not know how to "easily pass it to code blocks". I have put an example of passing tables to gnuplot source blocks below. I hope you can adapt it to the source block of your choice. It also illustrates the noweb syntax. I hope this helps. -------8<----------------8<------------- #+BABEL: :noweb yes :results silent * Gnuplot source preamble :src: #+srcname: gnuplot-preamble #+begin_src gnuplot reset set terminal pdfcairo color size 21cm,14.8cm set termoption enhanced set termoption font "DejaVuSerif,8" set grid back set style line 1 linewidth 2 linecolor rgb 'forest-green' set style line 2 linewidth 2 linecolor rgb 'red' set label '(pp \@ 7 TeV)' at graph -0.08,1.05 set ylabel 'K-factor' set xlabel 'W invariant mass (GeV)' set title 'K-factor = {/Symbol s}_{NLO}/{/Symbol s}_{LO}' #+end_src * HORACE K-factor ** Gnuplot source :src: #+srcname: plot-horace-k-factor #+begin_src gnuplot :var data=horace-kfactor-nnlo-vs-nnlo-w-scale-err <<gnuplot-preamble>> set output "horace-kfactor.pdf" set label 'Standard Model W' at graph 0.5,0.67 set xrange [0:1000] plot "$data" using 1:2 with linespoints linestyle 1 title 'Horace K-factor' set output #+end_src ** K-factor with uncertainties #+TBLNAME: horace-kfactor-nnlo-vs-nnlo-w-scale-err | M (GeV) | K-factor | |---------+----------| | 50 | 1.075 | | 150 | 1.01 | | 250 | 0.9932 | | 350 | 0.9778 | | 450 | 0.9621 | | 550 | 0.9498 | | 650 | 0.9365 | | 750 | 0.9253 | | 850 | 0.9122 | | 950 | 0.9001 | -------8<----------------8<------------- -- Suvayu Open source is the future. It sets us free. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Gather properties for use by babel source block? 2011-08-02 23:57 Gather properties for use by babel source block? Tim Burt 2011-08-03 14:00 ` darcamo @ 2012-05-07 15:52 ` Colin Hall 2012-05-07 14:16 ` Eric Schulte 2012-05-08 10:42 ` Tim Burt 1 sibling, 2 replies; 9+ messages in thread From: Colin Hall @ 2012-05-07 15:52 UTC (permalink / raw) To: emacs-orgmode Tim Burt <tcburt <at> rochester.rr.com> writes: > > I want to gather data from properties into something that can be used by > a babel source block (e.g. plot the data). Searches in the manual, > worg, and gmane have not yielded the method, but my best guess is that > I've missed it. If so, this is simply a request for a pointer to the > documentation I should read. Any luck with this, Tim? I'm trying to do something very similar. Cheers, Colin. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Gather properties for use by babel source block? 2012-05-07 15:52 ` Colin Hall @ 2012-05-07 14:16 ` Eric Schulte 2012-05-07 21:30 ` Colin Hall 2012-05-08 10:42 ` Tim Burt 1 sibling, 1 reply; 9+ messages in thread From: Eric Schulte @ 2012-05-07 14:16 UTC (permalink / raw) To: Colin Hall; +Cc: emacs-orgmode Colin Hall <colinghall@gmail.com> writes: > Tim Burt <tcburt <at> rochester.rr.com> writes: >> >> I want to gather data from properties into something that can be used by >> a babel source block (e.g. plot the data). Searches in the manual, >> worg, and gmane have not yielded the method, but my best guess is that >> I've missed it. If so, this is simply a request for a pointer to the >> documentation I should read. > > Any luck with this, Tim? I'm trying to do something very similar. > One approach would be to use the org-collector [1] from contrib/ to collect properties into a table. That table could then be fed as the argument to a source code block. Hope this helps, Footnotes: [1] http://orgmode.org/worg/org-contrib/org-collector.html -- Eric Schulte http://cs.unm.edu/~eschulte/ ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Gather properties for use by babel source block? 2012-05-07 14:16 ` Eric Schulte @ 2012-05-07 21:30 ` Colin Hall 0 siblings, 0 replies; 9+ messages in thread From: Colin Hall @ 2012-05-07 21:30 UTC (permalink / raw) To: Eric Schulte; +Cc: Colin Hall, emacs-orgmode On Mon, May 07, 2012 at 10:16:36AM -0400, Eric Schulte wrote: > Colin Hall <colinghall@gmail.com> writes: > > > Tim Burt <tcburt <at> rochester.rr.com> writes: > >> > >> I want to gather data from properties into something that can be used by > >> a babel source block (e.g. plot the data). Searches in the manual, > >> worg, and gmane have not yielded the method, but my best guess is that > >> I've missed it. If so, this is simply a request for a pointer to the > >> documentation I should read. > > > > Any luck with this, Tim? I'm trying to do something very similar. > > > > One approach would be to use the org-collector [1] from contrib/ to > collect properties into a table. That table could then be fed as the > argument to a source code block. > > Hope this helps, Yes, thank you, that was very helpful indeed. Cheers, Colin. -- Colin Hall ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Gather properties for use by babel source block? 2012-05-07 15:52 ` Colin Hall 2012-05-07 14:16 ` Eric Schulte @ 2012-05-08 10:42 ` Tim Burt 2012-05-09 19:30 ` Colin Hall 1 sibling, 1 reply; 9+ messages in thread From: Tim Burt @ 2012-05-08 10:42 UTC (permalink / raw) To: Colin Hall; +Cc: emacs-orgmode [-- Attachment #1: Type: text/plain, Size: 4335 bytes --] Colin Hall <colinghall@gmail.com> writes: > Tim Burt <tcburt <at> rochester.rr.com> writes: >> >> I want to gather data from properties into something that can be used by >> a babel source block (e.g. plot the data). Searches in the manual, >> worg, and gmane have not yielded the method, but my best guess is that >> I've missed it. If so, this is simply a request for a pointer to the >> documentation I should read. > > Any luck with this, Tim? I'm trying to do something very similar. After the hints of Darlan and Suvayu last August I cobbled a workflow described below. Errors and instability in the early weeks have been resolved into the current set of workable blemishes described below the workflow. The attachment is an org file that should be a working example if gnuplot is installed and configured for org-babel. Thank you Colin for asking the question about progress, because I should have shared it with the mailing list long ago warts and all. This is an example file for gathering and plotting health data (some of which has been fudged to protect the guilty ;-). The workflow is this: 1. DAILY: Use capture templates to query for data into a datetree under the heading '2011' which has an ID property. Each data headline is marked with tags for the data class (e.g. weight, blood sugar) that is used as a hook for collection. 2. ONE-TIME: Define the plots, in this case using gnuplot (thanks to Suvayu for the noweb example), under the heading 'Calculation and Visualization'. Another one-time setup is to define propview blocks that org-collector will populate for each data class (e.g. weight). Both the ID and tags are used to extract the information. 3. ONE-TIME: Another one-time setup is to define propview blocks that org-collector will populate for each data class (e.g. weight). Both the ID and tags are used to extract the information and this is under the 'Summaries' heading. - A #+tblname is placed after the #+BEGIN to provide a hook for the babel blocks for plotting. 4. ON-DEMAND: Refresh each collector block (in this case 4 separate ones), then org-babel-execute-buffer to generate the plots. This method has been mostly working but it has a few weaknesses, borne mostly from my ignorance. I haven't thought about the problem in several months, so maybe the act of writing the questions will spur me to the act of making it better. Here are my observations and questions: - At various times in the past year the ON-DEMAND step has not worked consistently, especially in regards to the #+tblname within the collector block. Sometimes the refresh would fail, often by wiping out the existing static view and replacing with an empty line under the #+tblname. No matter what happened in the past, it is stable enough for this post. - The problem could have been one of configuration (e.g. org-collector not loaded) or data (e.g. error in the date tree) or something else entirely. I have not done sufficiently thorough troubleshooting to really identify the different problems. - Collection is done by matching the tags in the data headlines, but the same type of information can be gleaned from the properties themselves. How can a match invocation be crafted to use the properties? - This weakness comes straight from my ignorance of lisp. - Collection is done for each separate table which does not scale well as the number of tables increases. Scaling itself is not the issue with just 4 tables, but forgetting to update each table is a slight problem. - Is there a way to automate the org-collector step so that one action updates all tables? - Of course, for the current story another solution would be to create one single data table with org-collector then modify the plotting routines to extract only the necessary columns. I'd like the ability to do monolithic or separate then decide which to use for which problems. - The capture templates yield blank properties that simply clutter the drawer and make data extraction logic slightly more complicated since existence alone is not a complete test. - Can a capture template be written to ask all the questions, but not yield a property if the answer is 'null'? [-- Attachment #2: TCBHealth-Example.org --] [-- Type: application/vnd.lotus-organizer, Size: 21674 bytes --] [-- Attachment #3: Type: text/plain, Size: 147 bytes --] > > Cheers, > Colin. > > > > > > -- Tim Burt www.rketburt.org "It is healthful to every sane man to utter the art within him;" -- GK Chesterton ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Gather properties for use by babel source block? 2012-05-08 10:42 ` Tim Burt @ 2012-05-09 19:30 ` Colin Hall 0 siblings, 0 replies; 9+ messages in thread From: Colin Hall @ 2012-05-09 19:30 UTC (permalink / raw) To: Tim Burt; +Cc: emacs-orgmode On Tue, May 08, 2012 at 06:42:00AM -0400, Tim Burt wrote: > Colin Hall <colinghall@gmail.com> writes: > > > Tim Burt <tcburt <at> rochester.rr.com> writes: > >> > >> I want to gather data from properties into something that can be used by > >> a babel source block (e.g. plot the data). Searches in the manual, > >> worg, and gmane have not yielded the method, but my best guess is that > >> I've missed it. If so, this is simply a request for a pointer to the > >> documentation I should read. > > > > Any luck with this, Tim? I'm trying to do something very similar. > > After the hints of Darlan and Suvayu last August I cobbled a workflow > described below. Errors and instability in the early weeks have been > resolved into the current set of workable blemishes described below the > workflow. The attachment is an org file that should be a working example > if gnuplot is installed and configured for org-babel. > > Thank you Colin for asking the question about progress, because I should > have shared it with the mailing list long ago warts and all. Thanks for the sample code, Tim, that's got me started. Cheers, Colin. -- Colin Hall ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2012-05-09 19:30 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2011-08-02 23:57 Gather properties for use by babel source block? Tim Burt 2011-08-03 14:00 ` darcamo 2011-08-03 14:35 ` tcburt 2011-08-03 15:53 ` suvayu ali 2012-05-07 15:52 ` Colin Hall 2012-05-07 14:16 ` Eric Schulte 2012-05-07 21:30 ` Colin Hall 2012-05-08 10:42 ` Tim Burt 2012-05-09 19:30 ` Colin Hall
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).