From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andy Choens Subject: Reproducible Research Template Date: Tue, 04 Jan 2011 14:55:27 -0500 Message-ID: <1294170927.2599.110.camel@Yates> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0109224704==" Return-path: Received: from [140.186.70.92] (port=55855 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1PaCyl-0000MX-D4 for emacs-orgmode@gnu.org; Tue, 04 Jan 2011 14:55:45 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PaCyi-0000cK-BS for emacs-orgmode@gnu.org; Tue, 04 Jan 2011 14:55:33 -0500 Received: from mail-qw0-f41.google.com ([209.85.216.41]:52323) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1PaCyi-0000c1-8w for emacs-orgmode@gnu.org; Tue, 04 Jan 2011 14:55:32 -0500 Received: by qwa26 with SMTP id 26so14542251qwa.0 for ; Tue, 04 Jan 2011 11:55:31 -0800 (PST) List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: Org-mode ml --===============0109224704== Content-Type: multipart/alternative; boundary="=-CIr4E+2ks0RAx6LdnkPb" --=-CIr4E+2ks0RAx6LdnkPb Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit I am developing a reproducible research template for R. I am trying to implement most of a research "compendium" in org. I say "most" because I am going to allow the actual data to exist outside of org, simply because most of the data I work with is relational or very large, which makes storage in plain text problematic or impossible. Has anyone ever implemented a reproducible research template in org that I can look at? I looked at the stuff on Worg and there are examples of a project, but not a template. In a nutshell, I am trying to do two things. 1) Provide a structure for reproducible research / programming ; 2) Provide a small set of helper functions. But, I don't want the helper functions to get in the way. I have considered two options: 1) Store the example code / template stuff in subheadings 2) Store the example code / template stuff in an external file. Using subheadings is tempting, but I'm afraid that org-babel-execute-buffer would cause problems for users who don't use all of the template functions. Using an external file, similar to the Lobrary of Babel is also tempting. It would allow me to make a cleaner template for structure and still allow users to access any helper functions. Is there a way to link to an external file, other than the Library of Babel? If so, how do I do this? Does anyone have any opinions about hiding/linking/importing example code in a template? I certainly appreciate any thoughts. --andy --=-CIr4E+2ks0RAx6LdnkPb Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: 7bit I am developing a reproducible research template for R. I am trying to implement most of a research "compendium" in org. I say "most" because I am going to allow the actual data to exist outside of org, simply because most of the data I work with is relational or very large, which makes storage in plain text problematic or impossible.

Has anyone ever implemented a reproducible research template in org that I can look at? I looked at the stuff on Worg and there are examples of a project, but not a template.

In a nutshell, I am trying to do two things.
    1) Provide a structure for reproducible research / programming ;
    2) Provide a small set of helper functions.

But, I don't want the helper functions to get in the way. I have considered two options:
    1) Store the example code / template stuff in subheadings
    2) Store the example code / template stuff in an external file. 

Using subheadings is tempting, but I'm afraid that org-babel-execute-buffer would cause problems for users who don't use all of the template functions.

Using an external file, similar to the Lobrary of Babel is also tempting. It would allow me to make a cleaner template for structure and still allow users to access any helper functions. Is there a way to link to an external file, other than the Library of Babel? If so, how do I do this?

Does anyone have any opinions about hiding/linking/importing example code in a template?

I certainly appreciate any thoughts.

--andy --=-CIr4E+2ks0RAx6LdnkPb-- --===============0109224704== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Emacs-orgmode mailing list Please use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode --===============0109224704==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Thomas S. Dye" Subject: Re: Reproducible Research Template Date: Wed, 5 Jan 2011 06:24:01 -1000 Message-ID: References: <1294170927.2599.110.camel@Yates> Mime-Version: 1.0 (Apple Message framework v936) Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Return-path: Received: from [140.186.70.92] (port=35669 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1PaW9g-0006fE-Bw for emacs-orgmode@gnu.org; Wed, 05 Jan 2011 11:24:09 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PaW9f-0008Pn-9B for emacs-orgmode@gnu.org; Wed, 05 Jan 2011 11:24:08 -0500 Received: from oproxy3-pub.bluehost.com ([69.89.21.8]:57644) by eggs.gnu.org with smtp (Exim 4.71) (envelope-from ) id 1PaW9f-0008PJ-32 for emacs-orgmode@gnu.org; Wed, 05 Jan 2011 11:24:07 -0500 In-Reply-To: <1294170927.2599.110.camel@Yates> List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: Andy Choens Cc: Org-mode ml On Jan 4, 2011, at 9:55 AM, Andy Choens wrote: > I am developing a reproducible research template for R. I am trying > to implement most of a research "compendium" in org. I say "most" > because I am going to allow the actual data to exist outside of org, > simply because most of the data I work with is relational or very > large, which makes storage in plain text problematic or impossible. > > Has anyone ever implemented a reproducible research template in org > that I can look at? I looked at the stuff on Worg and there are > examples of a project, but not a template. > > In a nutshell, I am trying to do two things. > 1) Provide a structure for reproducible research / programming ; > 2) Provide a small set of helper functions. > > But, I don't want the helper functions to get in the way. I have > considered two options: > 1) Store the example code / template stuff in subheadings > 2) Store the example code / template stuff in an external file. > > Using subheadings is tempting, but I'm afraid that org-babel-execute- > buffer would cause problems for users who don't use all of the > template functions. > > Using an external file, similar to the Lobrary of Babel is also > tempting. It would allow me to make a cleaner template for structure > and still allow users to access any helper functions. Is there a way > to link to an external file, other than the Library of Babel? If so, > how do I do this? > > Does anyone have any opinions about hiding/linking/importing example > code in a template? > > I certainly appreciate any thoughts. > > --andy Aloha Andy, Great idea. I'll be interested to see where you go with this. You can link to an external file other than the one holding the library of babel using the library of babel facility. I use this in my config file: #+source: load-local-lob #+begin_src emacs-lisp :tangle yes (org-babel-lob-ingest "~/org/local-lob.org") #+end_src I put functions there that are useful to me in more than one Org-mode buffer, but that are not likely to be useful to other Org-mode users (and therefore fit for the library of babel). Most of my projects store data in a MySQL database. My projects define queries that relate tables to one another, but the results are typically something that Org-mode understands---a flat table or a single value. The reproducible research functions that I write break the analysis workflow down into separate tasks so that the results of a SQL query are written to the Org-mode buffer. Subsequent steps in the workflow refer to the named results block. This way, the Org-mode project contains the actual data used in the analysis without the need to reproduce a relational structure. The functions that access the SQL database go in the local library of babel because I don't want to give users of a RR document direct access to my database. If I manipulate query results after they are written into the Org-mode buffer, I also like to write out intermediate results in many situations. My goal in writing RR code is not speed or compactness, but maximum transparency. Because Org-mode is language agnostic it is frequently the case that the files created by others contain code in languages I don't understand. If these are long and complex, then I'm at a loss to what is actually going on. If they are short, and intermediate results are written out in the buffer, then it is easier for me to follow the analysis. I look forward to learning from your work with RR templates. All the best, Tom From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Charles C. Berry" Subject: Re: Reproducible Research Template Date: Wed, 5 Jan 2011 08:31:27 -0800 Message-ID: References: <1294170927.2599.110.camel@Yates> Mime-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Return-path: Received: from [140.186.70.92] (port=33002 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1PaWHC-0008Vr-2w for emacs-orgmode@gnu.org; Wed, 05 Jan 2011 11:31:56 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PaWH9-0001od-9h for emacs-orgmode@gnu.org; Wed, 05 Jan 2011 11:31:54 -0500 Received: from tajo.ucsd.edu ([137.110.122.165]:53776) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1PaWH9-0001j8-42 for emacs-orgmode@gnu.org; Wed, 05 Jan 2011 11:31:51 -0500 In-Reply-To: <1294170927.2599.110.camel@Yates> List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: Andy Choens Cc: Org-mode ml On Tue, 4 Jan 2011, Andy Choens wrote: > I am developing a reproducible research template for R. I am trying to > implement most of a research "compendium" in org. I say "most" because I > am going to allow the actual data to exist outside of org, simply > because most of the data I work with is relational or very large, which > makes storage in plain text problematic or impossible. > > Has anyone ever implemented a reproducible research template in org that > I can look at? I looked at the stuff on Worg and there are examples of a > project, but not a template. > > In a nutshell, I am trying to do two things. > 1) Provide a structure for reproducible research / programming ; > 2) Provide a small set of helper functions. > > But, I don't want the helper functions to get in the way. I have > considered two options: > 1) Store the example code / template stuff in subheadings > 2) Store the example code / template stuff in an external file. One version of 'store the example code / template stuff' externally is to create an R package that holds the helper functions and whose example()s and/or demo()s show off the capabilities of helper functions. The R package 'inst' directory can hold miscellaneous files including suitable *.org templates, SQL snippets, etc. for different kinds of research projects. Then carrying out the research project would consist of starting R, require()-ing that package, issuing a command like choose.template( my.type.of.project, "myLocalWorkfile.org" , my.options ) (which would generate the starting template as myLocalWortkfile.org), and carrying on. HTH, Chuck p.s. You probably know that you can master the R package in a *.org file, but just in case: http://orgmode.org/worg/org-contrib/babel/examples/Rpackage.html#sec-6_1 > > Using subheadings is tempting, but I'm afraid that > org-babel-execute-buffer would cause problems for users who don't use > all of the template functions. > > Using an external file, similar to the Lobrary of Babel is also > tempting. It would allow me to make a cleaner template for structure and > still allow users to access any helper functions. Is there a way to link > to an external file, other than the Library of Babel? If so, how do I do > this? > > Does anyone have any opinions about hiding/linking/importing example > code in a template? > > I certainly appreciate any thoughts. > > --andy > Charles C. Berry Dept of Family/Preventive Medicine cberry@tajo.ucsd.edu UC San Diego http://famprevmed.ucsd.edu/faculty/cberry/ La Jolla, San Diego 92093-0901 From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andy Choens Subject: Re: Reproducible Research Template Date: Wed, 05 Jan 2011 12:24:28 -0500 Message-ID: <1294248268.4016.10.camel@Yates> References: <1294170927.2599.110.camel@Yates> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============0234491840==" Return-path: Received: from [140.186.70.92] (port=57392 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1PaX69-0000wc-6A for emacs-orgmode@gnu.org; Wed, 05 Jan 2011 12:24:34 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1PaX68-0004tg-2e for emacs-orgmode@gnu.org; Wed, 05 Jan 2011 12:24:33 -0500 Received: from mail-qy0-f169.google.com ([209.85.216.169]:62252) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1PaX67-0004tZ-Ut for emacs-orgmode@gnu.org; Wed, 05 Jan 2011 12:24:32 -0500 Received: by qyk7 with SMTP id 7so16760578qyk.0 for ; Wed, 05 Jan 2011 09:24:31 -0800 (PST) In-Reply-To: List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: "Thomas S. Dye" Cc: Org-mode ml --===============0234491840== Content-Type: multipart/alternative; boundary="=-2Onnij1mEp3NxMTYEoB3" --=-2Onnij1mEp3NxMTYEoB3 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit On Wed, 2011-01-05 at 06:24 -1000, Thomas S. Dye wrote: > Most of my projects store data in a MySQL database. My projects > define queries that relate tables to one another, but the results > are > typically something that Org-mode understands---a flat table or a > single value. The reproducible research functions that I write > break > the analysis workflow down into separate tasks so that the results > of > a SQL query are written to the Org-mode buffer. Subsequent steps in > the workflow refer to the named results block. This way, the > Org-mode > project contains the actual data used in the analysis without the > need > to reproduce a relational structure. The functions that access the > SQL database go in the local library of babel because I don't want > to > give users of a RR document direct access to my database. > That is very interesting. I like the idea of separating the database queries from the flat table results. This is similar to my situation. The actual storage of the data is relational, but the queries needed to assess the data result in either flat tables or single values. Very good idea. --=-2Onnij1mEp3NxMTYEoB3 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: 7bit
On Wed, 2011-01-05 at 06:24 -1000, Thomas S. Dye wrote:
Most of my projects store data in a MySQL database.  My projects 
define queries that relate tables to one another, but the results are 
typically something that Org-mode understands---a flat table or a 
single value.  The reproducible research functions that I write break 
the analysis workflow down into separate tasks so that the results of 
a SQL query are written to the Org-mode buffer.  Subsequent steps in 
the workflow refer to the named results block.  This way, the Org-mode 
project contains the actual data used in the analysis without the need 
to reproduce a relational structure.  The functions that access the 
SQL database go in the local library of babel because I don't want to 
give users of a RR document direct access to my database.


That is very interesting. I like the idea of separating the database queries from the flat table results. This is similar to my situation. The actual storage of the data is relational, but the queries needed to assess the data result in either flat tables or single values. Very good idea. --=-2Onnij1mEp3NxMTYEoB3-- --===============0234491840== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Emacs-orgmode mailing list Please use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode --===============0234491840==--