LaTeX export of tables

LaTeX export of tables

[Francesco Pizzolante]

Hi,

When exporting tables to LaTeX, I think it would be good to have the following
options to be able to define:

  - the table environment to be used: now we have the option to have tabular
    and longtable, but I could also use array, tabular*, tabularx, tabulary,
    etc.

    The general rule is:

    \begin{TABLEENV}{WIDTH}[VPOS]{COLS}
    ROWS
    \end{TABLEENV}

    TABLEENV can be any of tabular, array, tabular*, tabularx, etc., depending
    on the type of table I want.

    Some types of table need a WIDTH to be specified; in this case, I would
    add a WIDTH option. An extra vertical position VPOS option can also be
    added to some types of table.

    The COLS parameter already receives the value from the "align" org
    attribute.

    The following options

    #+ATTR_LATEX: tabularx width=\textwidth align=lX vpos=tb

    would generate the following LaTeX code:

    \begin{tabularx}{\textwidth}[tb]{lX}
    ROWS
    \end{tabularx}


  - the rule to be used as hline (similar option as in
    #+ORGTBL: :hline="\\midrule"):

    #+ATTR_LATEX: hline=midrule or #+ATTR_LATEX: hline="\\midrule"


  - define the top and bottom rules:

    #+ATTR_LATEX: toprule="\\toprule" bottomrule="\\bottomrule"


Does it make sense? Would it be possible to have these new options in a future
orgmode release?

Thanks a lot.
Francesco


_______________________________________________
Emacs-orgmode mailing list
Remember: use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: LaTeX export of tables

[Francesco Pizzolante]

Hi,

Can I give this thread a bump?

Francesco Pizzolante wrote:
> Hi,
>
> When exporting tables to LaTeX, I think it would be good to have the following
> options to be able to define:
>
>   - the table environment to be used: now we have the option to have tabular
>     and longtable, but I could also use array, tabular*, tabularx, tabulary,
>     etc.
>
>     The general rule is:
>
>     \begin{TABLEENV}{WIDTH}[VPOS]{COLS}
>     ROWS
>     \end{TABLEENV}
>
>     TABLEENV can be any of tabular, array, tabular*, tabularx, etc., depending
>     on the type of table I want.
>
>     Some types of table need a WIDTH to be specified; in this case, I would
>     add a WIDTH option. An extra vertical position VPOS option can also be
>     added to some types of table.
>
>     The COLS parameter already receives the value from the "align" org
>     attribute.
>
>     The following options
>
>     #+ATTR_LATEX: tabularx width=\textwidth align=lX vpos=tb
>
>     would generate the following LaTeX code:
>
>     \begin{tabularx}{\textwidth}[tb]{lX}
>     ROWS
>     \end{tabularx}
>
>
>   - the rule to be used as hline (similar option as in
>     #+ORGTBL: :hline="\\midrule"):
>
>     #+ATTR_LATEX: hline=midrule or #+ATTR_LATEX: hline="\\midrule"
>
>
>   - define the top and bottom rules:
>
>     #+ATTR_LATEX: toprule="\\toprule" bottomrule="\\bottomrule"
>
>
> Does it make sense? Would it be possible to have these new options in a future
> orgmode release?

I really am eager to use orgmode in order to generate all my LaTeX documents.
But, as of now, the LaTeX export of tables offers only limited flexibility.

My proposal is simply to add a few org attributes (type of the table, toprule,
bottomrule, hline, vpos) in order to get an improved flexibility when
generating LaTeX tables.

Can someone take care of my request?

Thanks a lot,
Francesco


_______________________________________________
Emacs-orgmode mailing list
Remember: use `Reply All' to send replies to the list.
Emacs-orgmode-mXXj517/zsQ@public.gmane.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: LaTeX export of tables

[Darlan Cavalcante Moreira]

If you are going to export only to LaTeX then you may just write the table in
latex syntax. Org-mode will recognize the latex environment and export it as it
is for latex. For HTML the table will be exported as a image, since it is a
"latex table" instead of an "org-mode table". It is not as convenient, but at
least you get what you want for now.

If you need to publish to another format then maybe there is a way to tell
org-mode to export certain part of the file only in latex, and other part only
in HTML. In this way you could have your elaborated table when publishing in
latex and a table following the org-mode syntax when publishing in HTML.


Darlan

At Mon, 19 Oct 2009 13:58:07 +0200,
Francesco Pizzolante <fpz@missioncriticalit.com> wrote:
> 
> Hi,
> 
> Can I give this thread a bump?
> 
> Francesco Pizzolante wrote:
> > Hi,
> >
> > When exporting tables to LaTeX, I think it would be good to have the following
> > options to be able to define:
> >
> >   - the table environment to be used: now we have the option to have tabular
> >     and longtable, but I could also use array, tabular*, tabularx, tabulary,
> >     etc.
> >
> >     The general rule is:
> >
> >     \begin{TABLEENV}{WIDTH}[VPOS]{COLS}
> >     ROWS
> >     \end{TABLEENV}
> >
> >     TABLEENV can be any of tabular, array, tabular*, tabularx, etc., depending
> >     on the type of table I want.
> >
> >     Some types of table need a WIDTH to be specified; in this case, I would
> >     add a WIDTH option. An extra vertical position VPOS option can also be
> >     added to some types of table.
> >
> >     The COLS parameter already receives the value from the "align" org
> >     attribute.
> >
> >     The following options
> >
> >     #+ATTR_LATEX: tabularx width=\textwidth align=lX vpos=tb
> >
> >     would generate the following LaTeX code:
> >
> >     \begin{tabularx}{\textwidth}[tb]{lX}
> >     ROWS
> >     \end{tabularx}
> >
> >
> >   - the rule to be used as hline (similar option as in
> >     #+ORGTBL: :hline="\\midrule"):
> >
> >     #+ATTR_LATEX: hline=midrule or #+ATTR_LATEX: hline="\\midrule"
> >
> >
> >   - define the top and bottom rules:
> >
> >     #+ATTR_LATEX: toprule="\\toprule" bottomrule="\\bottomrule"
> >
> >
> > Does it make sense? Would it be possible to have these new options in a future
> > orgmode release?
> 
> I really am eager to use orgmode in order to generate all my LaTeX documents.
> But, as of now, the LaTeX export of tables offers only limited flexibility.
> 
> My proposal is simply to add a few org attributes (type of the table, toprule,
> bottomrule, hline, vpos) in order to get an improved flexibility when
> generating LaTeX tables.
> 
> Can someone take care of my request?
> 
> Thanks a lot,
> Francesco
> 
> 
> _______________________________________________
> Emacs-orgmode mailing list
> Remember: use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: LaTeX export of tables

[Carsten Dominik]

On Oct 19, 2009, at 1:58 PM, Francesco Pizzolante wrote:

> Hi,
>
> Can I give this thread a bump?

Hi Francesco,

Org-mode LaTeX export is intended to give you a way to
export your file as LaTeX.  It can never hope to give you
all the formatting options LaTeX has.  Sure, what you are saying could  
be implemented, but so could many other things.  If you can come up  
with a patch, I would consider it, but I am so strained to keep up  
with the daily stuff, I don't expect to be able to put time into this.

An alternative to do this is to post-process your file using one of
the hooks and to switch out the environment used at that moment.

- Carsten

>
> Francesco Pizzolante wrote:
>> Hi,
>>
>> When exporting tables to LaTeX, I think it would be good to have  
>> the following
>> options to be able to define:
>>
>>  - the table environment to be used: now we have the option to have  
>> tabular
>>    and longtable, but I could also use array, tabular*, tabularx,  
>> tabulary,
>>    etc.
>>
>>    The general rule is:
>>
>>    \begin{TABLEENV}{WIDTH}[VPOS]{COLS}
>>    ROWS
>>    \end{TABLEENV}
>>
>>    TABLEENV can be any of tabular, array, tabular*, tabularx, etc.,  
>> depending
>>    on the type of table I want.
>>
>>    Some types of table need a WIDTH to be specified; in this case,  
>> I would
>>    add a WIDTH option. An extra vertical position VPOS option can  
>> also be
>>    added to some types of table.
>>
>>    The COLS parameter already receives the value from the "align" org
>>    attribute.
>>
>>    The following options
>>
>>    #+ATTR_LATEX: tabularx width=\textwidth align=lX vpos=tb
>>
>>    would generate the following LaTeX code:
>>
>>    \begin{tabularx}{\textwidth}[tb]{lX}
>>    ROWS
>>    \end{tabularx}
>>
>>
>>  - the rule to be used as hline (similar option as in
>>    #+ORGTBL: :hline="\\midrule"):
>>
>>    #+ATTR_LATEX: hline=midrule or #+ATTR_LATEX: hline="\\midrule"
>>
>>
>>  - define the top and bottom rules:
>>
>>    #+ATTR_LATEX: toprule="\\toprule" bottomrule="\\bottomrule"
>>
>>
>> Does it make sense? Would it be possible to have these new options  
>> in a future
>> orgmode release?
>
> I really am eager to use orgmode in order to generate all my LaTeX  
> documents.
> But, as of now, the LaTeX export of tables offers only limited  
> flexibility.
>
> My proposal is simply to add a few org attributes (type of the  
> table, toprule,
> bottomrule, hline, vpos) in order to get an improved flexibility when
> generating LaTeX tables.
>
> Can someone take care of my request?
>
> Thanks a lot,
> Francesco
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Remember: use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

- Carsten

Latex export of tables

[Vikas Rawal]

I am using org-mode version 8.0-pre (release_8.0-pre-247-gbc3ccd @
/home/vikas/lisp/org-mode/lisp/).

I have a table generated by a source block in a document that I would
like to export to latex. In the exported tex file, I would like org to
insert a line like the following between \end(tabular} and \end{table}

\begin{minipage}{\textwidth} \tiny Note: Some descriptive text here. \end{minipage}

Can somebody suggest what I could do to insert this?

I have been playing with something like the following but am not able
to get it right.

#+LATEX: {
#+NAME: table-name-out
#+CAPTION: Average gross value of output and net income from cultivation of wheat intercropped with rapeseed, by class, Mahatwar (Rupees per acre, 2005-06 prices)
#+attr_latex: :environment tabulary :width 15cm :align l|RRR 
#+RESULTS: source_code_name
| A | B | C |
|---+---+---|
| 1 | 1 | 1 |
| 1 | 2 | 3 |
| 2 | 2 | 4 |
#+LATEX: \begin{minipage}{} \tiny Note: Some descriptive text here.#\end{minipage}  }

Vikas

Re: Latex export of tables

[Suvayu Ali]

Hello Vikas,

On Fri, Apr 12, 2013 at 01:36:01PM +0530, Vikas Rawal wrote:
> I am using org-mode version 8.0-pre (release_8.0-pre-247-gbc3ccd @
> /home/vikas/lisp/org-mode/lisp/).
> 
> I have a table generated by a source block in a document that I would
> like to export to latex. In the exported tex file, I would like org to
> insert a line like the following between \end(tabular} and \end{table}
> 
> \begin{minipage}{\textwidth} \tiny Note: Some descriptive text here. \end{minipage}

I do not think this is possible.  You have to realise that Org does not
aim to support everything you can do with a backend natively.  One of
the primary reasons for that is the backend agnostic abstraction
provided by Org.

When in need of specific needs like this, I resort to writing LaTeX
natively.

Hope this helps somehow,

-- 
Suvayu

Open source is the future. It sets us free.

Re: Latex export of tables

[Vikas Rawal]

> > I am using org-mode version 8.0-pre (release_8.0-pre-247-gbc3ccd @
> > /home/vikas/lisp/org-mode/lisp/).
> > 
> > I have a table generated by a source block in a document that I would
> > like to export to latex. In the exported tex file, I would like org to
> > insert a line like the following between \end(tabular} and \end{table}
> > 
> > \begin{minipage}{\textwidth} \tiny Note: Some descriptive text here. \end{minipage}
> 
> I do not think this is possible.  You have to realise that Org does not
> aim to support everything you can do with a backend natively.  One of
> the primary reasons for that is the backend agnostic abstraction
> provided by Org.

I have seen some way of doing things like this. See section 13.3 at
http://orgmode.org/worg/org-tutorials/org-latex-export.html

I can't get it to work though. Will keep trying.

> 
> When in need of specific needs like this, I resort to writing LaTeX
> natively.

I guess one thing about org-mode is that it is addictive. Afterall, if
it is something to do with manipulating text, it ought to be possible
:)

There is also a reason for not doing it natively in latex even if the
org-mode solution is somewhat round-about. I am writing a research
paper using orgmode, with embedded R source blocks in it. I do not
mind embedding some latex source block into it but I would not like to
edit an exported latex file. After all, in the end, the objective is
to be able to have an org file which produces a full paper when exported.

Vikas

Re: Latex export of tables

[Thomas Alexander Gerds]

Hi Vikas

I am not sure I understand the problem correctly, but how about this?

Here is a table produced by a R-src block
with some descriptive text in a minipage:

-----------------------snip----------------------------------------------

#+BEGIN_SRC R  :results output latex  :exports results  :session *R* :cache yes 
tab <- matrix(1:12,nrow=4)
cat("\n\\begin{table}\n")
cat("\\begin{minipage}{\\textwidth}\n")
cat("\\tiny{ Note: some descriptive text}\n")
cat("\\end{minipage}\n")
nix <- apply(tab,1,function(x)cat(paste(x,collab="&"),"\\\\\n"))
cat("\\end{table}\n")
#+END_SRC

#+RESULTS[<2013-04-16 15:11:58> 964853177d477abc1cba212a72dde1f7cf3251c0]:
#+BEGIN_LaTeX

\begin{table}
\begin{minipage}{\textwidth}
\tiny{ Note: some descriptive text}
\end{minipage}
1 & 5 & 9 & \\
2 & 6 & 10 & \\
3 & 7 & 11 & \\
4 & 8 & 12 & \\
\end{table}
#+END_LaTeX

-----------------------snap----------------------------------------------

cheers Thomas

Vikas Rawal <vikaslists@agrarianresearch.org> writes:

>> > I am using org-mode version 8.0-pre (release_8.0-pre-247-gbc3ccd @
>> > /home/vikas/lisp/org-mode/lisp/).
>> > 
>> > I have a table generated by a source block in a document that I
>> > would like to export to latex. In the exported tex file, I would
>> > like org to insert a line like the following between \end(tabular}
>> > and \end{table}
>> > 
>> > \begin{minipage}{\textwidth} \tiny Note: Some descriptive text
>> > here. \end{minipage}
>>  I do not think this is possible.  You have to realise that Org does
>> not aim to support everything you can do with a backend natively.
>> One of the primary reasons for that is the backend agnostic
>> abstraction provided by Org.
>
> I have seen some way of doing things like this. See section 13.3 at
> http://orgmode.org/worg/org-tutorials/org-latex-export.html
>
> I can't get it to work though. Will keep trying.
>
>>  When in need of specific needs like this, I resort to writing LaTeX
>> natively.
>
> I guess one thing about org-mode is that it is addictive. Afterall, if
> it is something to do with manipulating text, it ought to be possible
> :)
>
> There is also a reason for not doing it natively in latex even if the
> org-mode solution is somewhat round-about. I am writing a research
> paper using orgmode, with embedded R source blocks in it. I do not
> mind embedding some latex source block into it but I would not like to
> edit an exported latex file. After all, in the end, the objective is
> to be able to have an org file which produces a full paper when
> exported.
>
> Vikas
--
Thomas A. Gerds -- Assoc. Prof. Department of Biostatistics
University of Copenhagen, Øster Farimagsgade 5, 1014 Copenhagen, Denmark
Office: CSS-15.2.07 (Gamle Kommunehospital)
tel: 35327914 (sec: 35327901) 

Re: Latex export of tables

[Suvayu Ali]

Hi Vikas,

On Tue, Apr 16, 2013 at 05:26:19PM +0530, Vikas Rawal wrote:
> > > I am using org-mode version 8.0-pre (release_8.0-pre-247-gbc3ccd @
> > > /home/vikas/lisp/org-mode/lisp/).
> > > 
> > > I have a table generated by a source block in a document that I would
> > > like to export to latex. In the exported tex file, I would like org to
> > > insert a line like the following between \end(tabular} and \end{table}
> > > 
> > > \begin{minipage}{\textwidth} \tiny Note: Some descriptive text here. \end{minipage}
> > 
> > I do not think this is possible.  You have to realise that Org does not
> > aim to support everything you can do with a backend natively.  One of
> > the primary reasons for that is the backend agnostic abstraction
> > provided by Org.
> 
> I have seen some way of doing things like this. See section 13.3 at
> http://orgmode.org/worg/org-tutorials/org-latex-export.html
> 
> I can't get it to work though. Will keep trying.

Many of the things on that page is old exporter specific and probably
will not work with the new exporter.

> There is also a reason for not doing it natively in latex even if the
> org-mode solution is somewhat round-about. I am writing a research
> paper using orgmode, with embedded R source blocks in it. I do not
> mind embedding some latex source block into it but I would not like to
> edit an exported latex file. After all, in the end, the objective is
> to be able to have an org file which produces a full paper when exported.

Then generate LaTeX tables from R not Org tables.  As far as I know, R
is capable of that.  I believe you can pass the ":wrap latex" option to
the babel block to wrap your LaTeX table with
"#+begin_latex..#+end_latex".

I'm suggesting this because if you continue on this path, i.e. litter
your Org file with hacks, soon you will end up with an extremely fragile
and complicated Org project.  I have been down that road while writing
my thesis.  At one point I realised the problem and made the decision to
split things into two kinds of files: static content (document
structuring, text, plots, etc), and dynamic content (babel, TikZ blocks
that generate tables, plots, figures, etc used by the static content
files).  It is still reproducible research, but modular and less hacky
(hence more stable).

Of course all of this is my personal opinion, it might be completely
inappropriate advise for your use case.

Good luck,

-- 
Suvayu

Open source is the future. It sets us free.

Re: Latex export of tables

[Thomas S. Dye]

Aloha all,

Jumping in here with apologies :)

Suvayu Ali <fatkasuvayu+linux@gmail.com> writes:

> Hi Vikas,
>
> On Tue, Apr 16, 2013 at 05:26:19PM +0530, Vikas Rawal wrote:
>> > > I am using org-mode version 8.0-pre (release_8.0-pre-247-gbc3ccd @
>> > > /home/vikas/lisp/org-mode/lisp/).
>> > > 
>> > > I have a table generated by a source block in a document that I would
>> > > like to export to latex. In the exported tex file, I would like org to
>> > > insert a line like the following between \end(tabular} and \end{table}
>> > > 
>> > > \begin{minipage}{\textwidth} \tiny Note: Some descriptive text
>> > > here. \end{minipage}
>> > 
>> > I do not think this is possible.  You have to realise that Org does not
>> > aim to support everything you can do with a backend natively.  One of
>> > the primary reasons for that is the backend agnostic abstraction
>> > provided by Org.
>> 
>> I have seen some way of doing things like this. See section 13.3 at
>> http://orgmode.org/worg/org-tutorials/org-latex-export.html
>> 
>> I can't get it to work though. Will keep trying.
>
> Many of the things on that page is old exporter specific and probably
> will not work with the new exporter.

Yes, this page is all about workarounds for the old exporter.  I believe
John Hendy is reorganizing this material and at some point will either
remove the tutorial or label it clearly as specific to the old exporter.

>
>> There is also a reason for not doing it natively in latex even if the
>> org-mode solution is somewhat round-about. I am writing a research
>> paper using orgmode, with embedded R source blocks in it. I do not
>> mind embedding some latex source block into it but I would not like to
>> edit an exported latex file. After all, in the end, the objective is
>> to be able to have an org file which produces a full paper when exported.
>
> Then generate LaTeX tables from R not Org tables.  As far as I know, R
> is capable of that.  I believe you can pass the ":wrap latex" option to
> the babel block to wrap your LaTeX table with
> "#+begin_latex..#+end_latex".

One reason to stick with Org tables is to ensure stylistic consistency
in the LaTeX output for all tables, regardless of where they originated.
This is more of a convenience than anything else, since the approach you
suggest can yield arbitrarily styled tables, too.

>
> I'm suggesting this because if you continue on this path, i.e. litter
> your Org file with hacks, soon you will end up with an extremely fragile
> and complicated Org project.  I have been down that road while writing
> my thesis.  At one point I realised the problem and made the decision to
> split things into two kinds of files: static content (document
> structuring, text, plots, etc), and dynamic content (babel, TikZ blocks
> that generate tables, plots, figures, etc used by the static content
> files).  It is still reproducible research, but modular and less hacky
> (hence more stable).

The path to a fragile and complicated Org project is well-worn, I've
been down it too many times myself. The habits I've developed over time
have helped, but I think they are less systematic than what you've
devised. I'd love to see some notes on your solution as a brief tutorial
or an expanded FAQ on Worg.  I'll be happy to contribute or help if you
find time to do something like this.

All the best,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: Latex export of tables

[Suvayu Ali]

Hey Tom,

On Tue, Apr 16, 2013 at 10:07:26AM -1000, Thomas S. Dye wrote:
> Aloha all,
> 
> Jumping in here with apologies :)

Isn't that why we discuss on mailing lists, so that people can jump in
;).

> > I'm suggesting this because if you continue on this path, i.e. litter
> > your Org file with hacks, soon you will end up with an extremely fragile
> > and complicated Org project.  I have been down that road while writing
> > my thesis.  At one point I realised the problem and made the decision to
> > split things into two kinds of files: static content (document
> > structuring, text, plots, etc), and dynamic content (babel, TikZ blocks
> > that generate tables, plots, figures, etc used by the static content
> > files).  It is still reproducible research, but modular and less hacky
> > (hence more stable).
> 
> The path to a fragile and complicated Org project is well-worn, I've
> been down it too many times myself. The habits I've developed over time
> have helped, but I think they are less systematic than what you've
> devised. I'd love to see some notes on your solution as a brief tutorial
> or an expanded FAQ on Worg.  I'll be happy to contribute or help if you
> find time to do something like this.

Actually, I am working on a workflow for large writing projects, my PhD
thesis in this case :-p.  What I have in mind is have a Makefile based
build system that uses `emacs --batch' to export to LaTeX and html.  The
images are generated separately using babel blocks or standalone TeX
files with TikZ code.  I intend to make pdf images for LaTeX and convert
them to png/svg with imagemagick/inkscape.  Of course all of this is
still a pipe dream; if I get it working, I'll definitely write a Worg
page on it.  Of course it will be great if people with interesting ideas
pitch in :).  I expect to have an early working environment in a couple
of months.

:)

-- 
Suvayu

Open source is the future. It sets us free.

Best practices for literate programming [was: Latex export of tables]

[Vikas Rawal]

> > I'm suggesting this because if you continue on this path, i.e. litter
> > your Org file with hacks, soon you will end up with an extremely fragile
> > and complicated Org project.  I have been down that road while writing
> > my thesis.  

I see the point. I think there is a need for documenting different
approaches people have used so far to avoid this. I suggest we use
this thread to start a discussion. If we get useful content, it should
perhaps land up somewhere on worg eventually.

> At one point I realised the problem and made the decision to
> > split things into two kinds of files: static content (document
> > structuring, text, plots, etc), and dynamic content (babel, TikZ blocks
> > that generate tables, plots, figures, etc used by the static content
> > files).  It is still reproducible research, but modular and less hacky
> > (hence more stable).

Suvayu,

This is indeed a very neat approach. Would you kindly elaborate?

Would it be too much work for you to get some illustrations from your
work?

In your scheme of things, how do you finally combine the static and
the dynamic content?

Any chance that you could release the source of something like a
chapter of your thesis for people to see? Or may be create something
with dummy content?

> I've been down it too many times myself. The habits I've developed
> over time have helped, but I think they are less systematic than
> what you've devised.

Tom, do tell us more about what these habits are.

>  I'd love to see some notes on your solution as
> a brief tutorial or an expanded FAQ on Worg. 

+1

> I'll be happy to
> contribute or help if you find time to do something like this.

and +1.

Vikas

Re: Latex export of tables

[Thomas S. Dye]

Hi Suvayu,

Suvayu Ali <fatkasuvayu+linux@gmail.com> writes:

> Hey Tom,
>
> Actually, I am working on a workflow for large writing projects, my PhD
> thesis in this case :-p.  What I have in mind is have a Makefile based
> build system that uses `emacs --batch' to export to LaTeX and html.  The
> images are generated separately using babel blocks or standalone TeX
> files with TikZ code.  I intend to make pdf images for LaTeX and convert
> them to png/svg with imagemagick/inkscape.  Of course all of this is
> still a pipe dream; if I get it working, I'll definitely write a Worg
> page on it.  Of course it will be great if people with interesting ideas
> pitch in :).  I expect to have an early working environment in a couple
> of months.

This sounds like the system I was using near the end of the old
exporter's life. When Nicolas added asynchronous export to the new
exporter I was able to abandon the Makefile. Lately, I've been tangling
init.el files from my source document. I really like this approach
because the asynchronous exporter starts off with a fresh emacs instance
(the infamous emacs -Q, I think) and you don't have to worry if your
.emacs has an old setting that has some weird effect on export--all of
the setup is in the babel block that generates the init.el file. I like
that the init.el setup travels with the document in the Org mode file. I
think it helps with reproducibility.

All the best,
Tom

-- 
T.S. Dye & Colleagues, Archaeologists
735 Bishop St, Suite 315, Honolulu, HI 96813
Tel: 808-529-0866, Fax: 808-529-0884
http://www.tsdye.com

Re: Best practices for literate programming [was: Latex export of tables]

[Thomas S. Dye]

Aloha Vikas,

Vikas Rawal <vikaslists@agrarianresearch.org> writes:

>> I've been down it too many times myself. The habits I've developed
>> over time have helped, but I think they are less systematic than
>> what you've devised.
>
> Tom, do tell us more about what these habits are.

The new exporter is really your friend.  Where before I might choose to
generate a LaTeX block, now I look to generate Org output and then count
on the exporter to do the right thing on the way to pdf.  

The exporter's attribute system is very easy to use.  The attributes you
need to access are always right there.

I've also come to rely on filters quite a bit. I use them for
non-breaking spaces, the plus/minus symbol, and for the multiple
citation commands used by biblatex (e.g., \parencites). There seems to
be a move afoot to collect filters so they can be widely distributed.
I'd like to see the filters go to the Library of Babel, but for
reproducible research it is probably best to keep them with the source
document so there is no doubt about the fidelity of filter code.

All the best,
Tom

-- 
T.S. Dye & Colleagues, Archaeologists
735 Bishop St, Suite 315, Honolulu, HI 96813
Tel: 808-529-0866, Fax: 808-529-0884
http://www.tsdye.com

Re: Best practices for literate programming [was: Latex export of tables]

[Suvayu Ali]

[-- Attachment #1: Type: text/plain, Size: 2425 bytes --]

Hi Vikas,

On Wed, Apr 17, 2013 at 03:40:22AM +0530, Vikas Rawal wrote:
> 
> > At one point I realised the problem and made the decision to
> > split things into two kinds of files: static content (document
> > structuring, text, plots, etc), and dynamic content (babel, TikZ blocks
> > that generate tables, plots, figures, etc used by the static content
> > files).  It is still reproducible research, but modular and less hacky
> > (hence more stable).
> 
> This is indeed a very neat approach. Would you kindly elaborate?
> 
> Would it be too much work for you to get some illustrations from your
> work?

Well ... it was couple of years back, the Org version was quite
different, e.g. babel was rapidly evolving.  It might be a fair bit of
work to get it working again.  That said, last year I gave a talk in an
internal workshop, I made the plots with the attached file.  I didn't
spend time to make sure everything is pretty, so the legend and titles
might be a little wonky.  Just evaluating the two main source blocks
should give you two plots in pdf files.

> In your scheme of things, how do you finally combine the static and
> the dynamic content?
> 
> Any chance that you could release the source of something like a
> chapter of your thesis for people to see? Or may be create something
> with dummy content?

The idea is to keep the dynamic content on separate org files which you
export less frequently during the course of your writing, e.g. any
tables that are inputs for source blocks.  Evaluating these blocks, or
exporting these dynamic files (whichever is your preference) generates
the graphic which is then used in the static file.  This is not limited
to plots, you could write org/LaTeX tables to separate files.  You can
then easily include those in your static files.

My main motivation for this was to make the export process simpler.  And
since the complicated interacting bits are all isolated and modularised,
there are fewer things that go wrong and many files are updated only
when required, hence faster too!

Anyway, this is all probably very vague without working examples.  I'll
try to come up with something, but I have been rather busy for the last
year or so and do not see any sign of respite in the near future :-/.
I'll get this fleshed out at some point, just don't know how soon.

Hope this was helpful in some way,

:)

-- 
Suvayu

Open source is the future. It sets us free.

[-- Attachment #2: plots.org --]
[-- Type: text/plain, Size: 5338 bytes --]

#+STARTUP: overview
#+PROPERTY: noweb yes
#+PROPERTY: results silent
#+BIND: org-confirm-babel-evaluate nil


* Gnuplot source preamble                                               :src:
  :PROPERTIES:
  :VISIBILITY: folded
  :END:

#+name: gnuplot-preamble
#+begin_src gnuplot
  reset
  set terminal pdfcairo color size 21cm,14.8cm
  set termoption enhanced
  set encoding utf8
  set termoption font "DejaVuSerif,8"
  # set output '|display png:-'
  set grid back
  set style line 1 linewidth 9 pointtype 1  linecolor rgb 'orange'
  set style line 2 pointsize 1 pointtype 5  linecolor rgb 'forest-green'
  set style line 3 pointsize 1 pointtype 7  linecolor rgb 'red'
  set style line 4 pointsize 1 pointtype 9  linecolor rgb 'blue'
  set style line 5 pointsize 1 pointtype 11 linecolor rgb 'dark-gray'
  set style line 6 pointsize 1 pointtype 13 linecolor rgb 'brown'
  set style line 7 linewidth 7 pointtype 19 linecolor rgb 'black'
  set style line 10 linewidth 2 linecolor rgb 'black'
  set style line 11 linewidth 5 linecolor rgb 'red'
  set key outside
  set key box linestyle 10
#+end_src


* BF Upper Limit summary plots
** Gnuplot source 							:src:
#+name: limits-preamble
#+begin_src gnuplot
  set log y
  set format y "10^{%L}"
  set ylabel 'BF Upper Limit'
  set xtics nomirror rotate by 90 offset character 0,-3
#+end_src

*** B⁺ → h⁻l⁺l⁺ / D⁻l⁺l⁺					      :Bplus:
#+begin_src gnuplot :noweb yes :var limits=Bpluslimits
  <<gnuplot-preamble>>
  <<limits-preamble>>
  set xrange [0:8]
  set yrange [1E-14:1E-5]
  set label 'BF Upper Limits:' at graph 1.02,0.55 font ',10'
  set label ' B⁺ → h⁻l⁺l⁺' at graph 1.02,0.5
  set label ' B⁺ → D⁽*⁾⁻l⁺l⁺' at graph 1.02,0.45
  set label 'LHCb limits \@ 95% C.L.' at graph 1.02,0.37 font ',7'
  set label 'Other limits \@ 90% C.L.' at graph 1.02,0.33 font ',7'
  set xtics ("K⁻e⁺e⁺" 1, "K⁻μ⁺μ⁺" 2, "π⁻e⁺e⁺" 3, "π⁻μ⁺μ⁺" 4, "D⁻e⁺e⁺" 5, "D⁻μ⁺μ⁺" 6, "D*⁻μ⁺μ⁺" 7)
  set output "Bpluslimits.pdf"
  plot "$limits" using 1:2 title 'Theory' linestyle 1, \
       "$limits" using 1:3 title 'BaBar' linestyle 2, \
       "$limits" using 1:4 title 'Belle' linestyle 3, \
       "$limits" using 1:5 title 'LHCb' linestyle 4, \
       "$limits" using 1:6 title 'LHCb year-end' linestyle 5, \
       "$limits" using 1:7 title 'LHCb upgrade' linestyle 6
       # 1E-10 with lines linestyle 10 title ''
       # 3.1E-9 with lines linestyle 11
  set output
#+end_src

*** D⁺ → h⁻l⁺l⁺ / Dₛ⁺ → h⁻l⁺l⁺					      :Dplus:
#+begin_src gnuplot :noweb yes :var limits=Dpluslimits
  <<gnuplot-preamble>>
  <<limits-preamble>>
  set xrange [0:7]
  set yrange [1E-10:1E-5]
  set label 'BF Upper Limits:' at graph 0.7,0.55 font ',10'
  set label ' D⁺ → π⁻l⁺l⁺' at graph 0.7,0.5
  set label ' Dₛ⁺ → h⁻l⁺l⁺' at graph 0.7,0.45
  set label 'All limits \@ 90% C.L.' at graph 0.7,0.37 font ',7'
  set xtics nomirror rotate by 90 offset character 0,-5
  set xtics ("D⁺→π⁻e⁺e⁺" 1, "D⁺→π⁻μ⁺μ⁺" 2, "Dₛ⁺→π⁻e⁺e⁺" 3, "Dₛ⁺→π⁻μ⁺μ⁺" 4, "Dₛ⁺→K⁻e⁺e⁺" 5, "Dₛ⁺→K⁻μ⁺μ⁺" 6)
  set output "Dpluslimits.pdf"
  plot "$limits" using 1:2 title 'Theory' linestyle 1, \
       "$limits" using 1:3 title 'BaBar' linestyle 2, \
       "$limits" using 1:4 title 'SuperB' linestyle 7
       # 1E-10 with lines linestyle 10 title ''
       # 3.1E-9 with lines linestyle 11
  set output
#+end_src

** Tables								:tbl:
*** B⁺ → h⁻l⁺l⁺ / D⁻l⁺l⁺					      :Bplus:
#+tblname: Bpluslimits
|   |  Theory |   BaBar |  Belle |       LHCb |     LHCb |    LHCb |                             |
|   |         |         |        | (95% C.L.) | year-end | upgrade |                             |
|---+---------+---------+--------+------------+----------+---------+-----------------------------|
| 1 | 3.6E-14 |  3.0E-8 |        |            |          |         | \(B^+ \to K^-e^+e^+\)       |
| 2 | 3.6E-14 |  6.7E-8 |        |     5.4E-8 |   6.5E-9 |  1.1E-9 | \(B^+ \to K^-\mu^+\mu^+\)   |
| 3 | 6.3E-13 |  2.3E-8 |        |            |          |         | \(B^+ \to \pi^-e^+e^+\)     |
| 4 | 6.3E-13 | 10.7E-8 |        |     5.8E-8 |   7.0E-9 |  1.1E-9 | \(B^+ \to \pi^-\mu^+\mu^+\) |
| 5 | 1.7E-14 |         | 2.6E-6 |            |          |         | \(B^+ \to D^-e^+e^+\)       |
| 6 | 1.7E-14 |         | 1.1E-6 |     6.9E-7 |   2.8E-7 |  4.5E-8 | \(B^+ \to D^-\mu^+\mu^+\)   |
| 7 |         |         |        |     2.4E-6 |   9.6E-7 |  1.6E-7 | \(B^+ \to D*^-\mu^+\mu^+\)  |
#+tblfm: $1=(@#-2)

*** D⁺ → h⁻l⁺l⁺ / Dₛ⁺ → h⁻l⁺l⁺					      :Dplus:
#+tblname: Dpluslimits
|   |  Theory |  BaBar | SuperB |                               |
|---+---------+--------+--------+-------------------------------|
| 1 | 4.5E-10 | 1.9E-6 | 1E-8   | \(D^+ \to \pi^-e^+e^+\)       |
| 2 | 4.5E-10 | 2.0E-6 | 1E-8   | \(D^+ \to \pi^-\mu^+\mu^+\)   |
| 3 |  6.9E-9 | 4.1E-6 |        | \(D^+_s \to \pi^-e^+e^+\)     |
| 4 |  6.9E-9 | 1.4E-5 |        | \(D^+_s \to \pi^-\mu^+\mu^+\) |
| 5 | 2.2E-10 | 5.2E-6 |        | \(D^+_s \to K^-e^+e^+\)       |
| 6 | 2.2E-10 | 1.3E-5 |        | \(D^+_s \to K^-\mu^+\mu^+\)   |
#+tblfm: $1=(@#-1)


* COMMENT local setup

# Local Variables:
# org-export-allow-BIND: t
# End:

Re: Best practices for literate programming [was: Latex export of tables]

[Rainer M. Krug]

Suvayu Ali <fatkasuvayu+linux@gmail.com> writes:

> Hi Vikas,
>
> On Wed, Apr 17, 2013 at 03:40:22AM +0530, Vikas Rawal wrote:
>> 
>> > At one point I realised the problem and made the decision to
>> > split things into two kinds of files: static content (document
>> > structuring, text, plots, etc), and dynamic content (babel, TikZ blocks
>> > that generate tables, plots, figures, etc used by the static content
>> > files).  It is still reproducible research, but modular and less hacky
>> > (hence more stable).
>> 
>> This is indeed a very neat approach. Would you kindly elaborate?
>> 
>> Would it be too much work for you to get some illustrations from your
>> work?
>
> Well ... it was couple of years back, the Org version was quite
> different, e.g. babel was rapidly evolving.  It might be a fair bit of
> work to get it working again.  That said, last year I gave a talk in an
> internal workshop, I made the plots with the attached file.  I didn't
> spend time to make sure everything is pretty, so the legend and titles
> might be a little wonky.  Just evaluating the two main source blocks
> should give you two plots in pdf files.
>
>> In your scheme of things, how do you finally combine the static and
>> the dynamic content?
>> 
>> Any chance that you could release the source of something like a
>> chapter of your thesis for people to see? Or may be create something
>> with dummy content?
>
> The idea is to keep the dynamic content on separate org files which you
> export less frequently during the course of your writing, e.g. any
> tables that are inputs for source blocks.  Evaluating these blocks, or
> exporting these dynamic files (whichever is your preference) generates
> the graphic which is then used in the static file.  This is not limited
> to plots, you could write org/LaTeX tables to separate files.  You can
> then easily include those in your static files.
>
> My main motivation for this was to make the export process simpler.  And
> since the complicated interacting bits are all isolated and modularised,
> there are fewer things that go wrong and many files are updated only
> when required, hence faster too!
>
> Anyway, this is all probably very vague without working examples.  I'll
> try to come up with something, but I have been rather busy for the last
> year or so and do not see any sign of respite in the near future :-/.
> I'll get this fleshed out at some point, just don't know how soon.
>
> Hope this was helpful in some way,
>
> :)
<#secure method=pgpmime mode=sign>
I did not follow the initial thread, but the new header caught my
attentian, as I am doing something similar with papers. Nothing against
org for writing papers, but I prefer LyX [1]. But for doing the analysis,
org together, nothing beats org. So in my org file I have the
analysis which creates graphs on export (and a basic report of the
analysis, including all the source code necessary, which I can then use
as an appendix for the paper).

These graphs are then inserted in the lyx file. I assume, you used
something similar, only that the oputput can then be used in the org
file (thesis) - correct?

Cheers,

Rainer

Footnotes: 
[1]  http://www.lyx.org - very nice LaTeX frontend.

-- 
Rainer M. Krug, PhD (Conservation Ecology, SUN), MSc (Conservation Biology, UCT), Dipl. Phys. (Germany)

Centre of Excellence for Invasion Biology
Stellenbosch University
South Africa

Tel :       +33 - (0)9 53 10 27 44
Cell:       +33 - (0)6 85 62 59 98
Fax :       +33 - (0)9 58 10 27 44

Fax (D):    +49 - (0)3 21 21 25 22 44

email:      Rainer@krugs.de

Skype:      RMkrug

Re: Best practices for literate programming [was: Latex export of tables]

[Suvayu Ali]

Hi Rainer,

On Wed, Apr 17, 2013 at 11:55:50AM +0200, Rainer M. Krug wrote:
> 
> I did not follow the initial thread, but the new header caught my
> attentian, as I am doing something similar with papers. Nothing against
> org for writing papers, but I prefer LyX [1]. But for doing the analysis,
> org together, nothing beats org. So in my org file I have the
> analysis which creates graphs on export (and a basic report of the
> analysis, including all the source code necessary, which I can then use
> as an appendix for the paper).
> 
> These graphs are then inserted in the lyx file. I assume, you used
> something similar, only that the oputput can then be used in the org
> file (thesis) - correct?

Yes something like that; usually for me analysis code is so complicated
that doing it inside Org would be madness :-p, I have dedicated software
projects for that.  I only use Org for simple spreadsheet operations in
tables and eventually plotting them.  These then get included in the
final "thesis" file.

Cheers,

-- 
Suvayu

Open source is the future. It sets us free.

Re: Latex export of tables

[Myles English]

Hi Suvayu,

Suvayu Ali writes:

> Actually, I am working on a workflow for large writing projects, my PhD
> thesis in this case :-p.  What I have in mind is have a Makefile based
> build system that uses `emacs --batch' to export to LaTeX and html.  The
> images are generated separately using babel blocks or standalone TeX
> files with TikZ code.  I intend to make pdf images for LaTeX and convert
> them to png/svg with imagemagick/inkscape.  Of course all of this is
> still a pipe dream; if I get it working, I'll definitely write a Worg
> page on it.  Of course it will be great if people with interesting ideas
> pitch in :).  I expect to have an early working environment in a couple
> of months.

I am looking forward to seeing what you come up with.  (I have mentioned
this before, however) I have been using a CMake system for LaTeX export:

http://cmake.org/Wiki/images/8/80/UseLATEX.cmake

This allows (amongst other things) generating graphics from stand alone
R scripts that get their own data from a database.  I believe the new
version has some kind of support for SVG.  I used this system for
several papers and my thesis and it has three main advantages:

1) out-of-source builds, so my working dir doesn't get cluttered with
LaTeX files

2) asynchronous export of org to LaTeX (although this is now available
with the new exporter)

3) the graphics are regenerated every time so I know it all still
works

I tangle a CMakeList.txt from every org file that I want to export, then
run cmake ~/path; make from the commandline.

(Version control as part of this workflow is possibly another topic: I
have been using git subtrees to make project-local copies of files such
as mystyle.sty and mybiblatex.bib and then update the main git repo with
the changes, and pull the changes down to another project.)

I'll just paste the whole CMakeList.txt so you can see what it looks
like, but the most interesting part is the emacs --batch command:

#+BEGIN_SRC sh :tangle CMakeLists.txt
  cmake_minimum_required(VERSION 2.8)
    
  project(relk NONE)
    
  include(/usr/share/cmake-2.8/Modules/UseLATEX.cmake)
    
  # export the .tex file from the .org file
  # using the emacs orgmode exporter
  latex_get_output_path(OUTPUT_DIR)
    
  file(COPY ${CMAKE_CURRENT_SOURCE_DIR}/relkpaper.org DESTINATION ${OUTPUT_DIR}/ )
  file(COPY /home/myles/lib/lisp/my-export.el DESTINATION ${OUTPUT_DIR}/ )
  
  add_custom_target( orgfile ALL
      DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/relkpaper.org )
  
  add_custom_target( elfile ALL
      DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/my-export.el )
  
  add_custom_command(
      OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/relkpaper.tex
      COMMAND emacs --batch --eval \"(progn
          (add-to-list 'load-path
          (expand-file-name \\"~/.emacs.d/plugins/org-mode/lisp/\\"))
        (add-to-list 'load-path
              (expand-file-name \\"~/.emacs.d/plugins/org-mode/contrib/lisp/\\" t))
          (require 'org)
        (require 'ox)
          (require 'org-exp)
        (require 'org-inlinetask)
          (org-babel-do-load-languages
              'org-babel-load-languages
          '((emacs-lisp . t)
              (sh . t)))
          (setq org-confirm-babel-evaluate nil)
          (setq org-export-with-todo-keywords nil)
          (setq org-export-babel-evaluate nil)
          (load-file \\"my-export.el\\")
          (add-to-list 'org-export-before-parsing-hook 'my-export-delete-headlines-tagged-noheading)
          (add-to-list 'org-export-filter-link-functions
          'my-autoref-filter-link-func)
          (find-file \\"${CMAKE_CURRENT_BINARY_DIR}/relkpaper.org\\")
          (org-latex-export-to-latex))\"
      DEPENDS orgfile elfile
      COMMENT "Exporting orgmode file to LaTeX using emacs")
  
  add_custom_target( mainfile ALL
      DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/relkpaper.tex )
    
  # Set R executable
  set(R_COMPILE "/usr/bin/Rscript")
  # Set the location of data files
  ##set(DATA_DIR data)
  # Set the location of the directory for image files
  set(IMAGE_DIR graphicsauto)
    
  # Get a list of R files
  file(GLOB_RECURSE R_FILES "*.R")
    
  # Copy over all R scripts
  # add_custom_command(
  #   OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/R
  #   DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/R
  #   COMMAND ${CMAKE_COMMAND} -E copy_directory
  #     ${CMAKE_CURRENT_SOURCE_DIR}/R
  #     ${CMAKE_CURRENT_BINARY_DIR}/R
  # )
  file( COPY ${CMAKE_CURRENT_SOURCE_DIR}/R
      DESTINATION ${CMAKE_CURRENT_BINARY_DIR})
  file(MAKE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/${IMAGE_DIR})
    
  foreach(file ${R_FILES})
      message("processing ${file}")
      get_filename_component(basename "${file}" NAME_WE)
      #file(COPY ${CMAKE_CURRENT_SOURCE_DIR}/R/${file} DESTINATION ${CMAKE_CURRENT_BINARY_DIR}/R/${file})
      # # Replace strings in R files so data files can be found
      # file(READ
      # ${CMAKE_CURRENT_SOURCE_DIR}/${IMAGE_DIR}/${basename}.R
      # file_contents
      # )
      # string(REPLACE "${DATA_DIR}" "${IMAGE_DIR}/${DATA_DIR}"
      # changed_file_contents ${file_contents}
      # )
      # file(WRITE
      # ${CMAKE_CURRENT_BINARY_DIR}/${IMAGE_DIR}/${basename}.R
      # ${changed_file_contents}
      # )
    
        # Command to run R
        if(R_COMPILE)
          message("Adding ........... ${CMAKE_CURRENT_BINARY_DIR}/R/${basename}.R")
    
          add_custom_command(
            OUTPUT
              ${CMAKE_CURRENT_BINARY_DIR}/${IMAGE_DIR}/${basename}.eps
            DEPENDS
              ${CMAKE_CURRENT_BINARY_DIR}/R/${basename}.R
              #      ${CMAKE_CURRENT_BINARY_DIR}/${IMAGE_DIR}/${DATA_DIR}
            COMMAND
              ${R_COMPILE}
            ARGS
              ${CMAKE_CURRENT_BINARY_DIR}/R/${basename}.R
              ${CMAKE_CURRENT_BINARY_DIR}/${IMAGE_DIR}/${basename}.eps
          )
    message("Running ${R_COMPILE} ${CMAKE_CURRENT_BINARY_DIR}/R/${basename}.R ${CMAKE_CURRENT_BINARY_DIR}/${IMAGE_DIR}/${basename}.eps")
    
       # add_dependencies( ${CMAKE_CURRENT_BINARY_DIR}/R/${basename}.R
     # ${CMAKE_CURRENT_BINARY_DIR}/${IMAGE_DIR}/${basename}.eps
    #)
    endif(R_COMPILE)
    
        # Make a list of all R files (for ADD_LATEX_DOCUMENT depend)
        set(ALL_R_FILES ${ALL_R_FILES}
          ${CMAKE_CURRENT_BINARY_DIR}/${IMAGE_DIR}/${basename}.eps
        )
  endforeach(file)
    
  # # Copy over all data files needed to generate R graphs
  # add_custom_command(
  #   OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/${IMAGE_DIR}/${DATA_DIR}
  #   DEPENDS ${CMAKE_CURRENT_SOURCE_DIR}/${IMAGE_DIR}/${DATA_DIR}
  #   COMMAND ${CMAKE_COMMAND} -E copy_directory
  #     ${CMAKE_CURRENT_SOURCE_DIR}/${IMAGE_DIR}/${DATA_DIR}
  #     ${CMAKE_CURRENT_BINARY_DIR}/${IMAGE_DIR}/${DATA_DIR}
  # )
    
  add_latex_document( a.tex
      INPUTS texlib/mystyle.sty
      IMAGE_DIRS img/scans ${IMAGE_DIR} img
      DEPENDS ${ALL_R_FILES}
              mainfile
      BIBFILES texlib/mybiblatex.bib
      DEFAULT_PDF
      USE_NOMENCL
  )
    
#+END_SRC

Myles

Re: Best practices for literate programming [was: Latex export of tables]

[Rasmus]

Thomas,

>> Tom, do tell us more about what these habits are.
>
> The new exporter is really your friend.  Where before I might choose to
> generate a LaTeX block, now I look to generate Org output and then count
> on the exporter to do the right thing on the way to pdf.  
>
> The exporter's attribute system is very easy to use.  The attributes you
> need to access are always right there.
>
> I've also come to rely on filters quite a bit. I use them for
> non-breaking spaces, the plus/minus symbol, and for the multiple
> citation commands used by biblatex (e.g., \parencites). There seems to
> be a move afoot to collect filters so they can be widely distributed.
> I'd like to see the filters go to the Library of Babel, but for
> reproducible research it is probably best to keep them with the source
> document so there is no doubt about the fidelity of filter code.

I too rely heavily on filters and customizations.  I haven't been able
to fully appreciate the asynchronous exporter yet.

For instance I set some defaults for tables, pictures, add lots of
entities etc. in my init file, and I went as far as writing a separate
init file just loading just the org stuff.  Now, this is clearly /not/
a very reproducible way of doing this.

So I'm really interested in hearing or seeing implementation where the
goal is reproducibility.  On one hand I can appreciate keeping Org
close to a vanilla state.  On the other hand, I'd have to overwrite
defaults every time (e.g. I /always/ want booktab tables).  I guess I
could keep an emacs-lisp block in the top of the file specifying
stuff, but it also seems kind of tedious (copy-pasting every time).
(Perhaps this could be resolved by loading external files hosted
somewhere accessible).

–Rasmus

-- 
. . . The proofs are technical in nature and provides no real understanding.

Re: Best practices for literate programming [was: Latex export of tables]

[Aaron Ecay]

Hi Rasmus,

2013ko apirilak 18an, Rasmus-ek idatzi zuen:
> I too rely heavily on filters and customizations.  I haven't been able
> to fully appreciate the asynchronous exporter yet.
> 
> For instance I set some defaults for tables, pictures, add lots of
> entities etc. in my init file, and I went as far as writing a separate
> init file just loading just the org stuff.  Now, this is clearly /not/
> a very reproducible way of doing this.
> 
> So I'm really interested in hearing or seeing implementation where the
> goal is reproducibility.  On one hand I can appreciate keeping Org
> close to a vanilla state.  On the other hand, I'd have to overwrite
> defaults every time (e.g. I /always/ want booktab tables).  I guess I
> could keep an emacs-lisp block in the top of the file specifying
> stuff, but it also seems kind of tedious (copy-pasting every time).
> (Perhaps this could be resolved by loading external files hosted
> somewhere accessible).

If your external org configuration file were kept under version control
(I’ll discuss git but the principle is general), then reproducibility
would be possible.  There are ways of embedding git hashes in LaTeX
documents (for one example:
http://thorehusfeldt.net/2011/05/13/including-git-revision-identifiers-in-latex/),
and of course org could help automate this.  Including the git hash of
the document itself, the config file, and org-mode’s own code (assuming
these are kept in 3 separate repos) should allow perfect reproducibility
(modulo incompatible changes in emacs, I guess).

It would be interesting for org to have an ability to reference files
not just by name, but by git revision.  So that you could do something
like (where 123456 is some git hash):
#+include: [[gitbare:/path/to/repo::123456:my-org-setup-file.org]]
and have org take care of checking out the proper revision and loading
the file in the usual way.  This syntax is already implemented, for
plain links, in contrib/lisp/org-git-link.el, so it is just a matter
of making #+include and friends understand links in addition to
filenames.

-- 
Aaron Ecay

Re: Best practices for literate programming [was: Latex export of tables]

[Rasmus]

Aaron Ecay <aaronecay@gmail.com> writes:

> If your external org configuration file were kept under version control
> (I’ll discuss git but the principle is general), then reproducibility
> would be possible.  There are ways of embedding git hashes in LaTeX
> documents (for one example:
> http://thorehusfeldt.net/2011/05/13/including-git-revision-identifiers-in-latex/),
> and of course org could help automate this.  Including the git hash of
> the document itself, the config file, and org-mode’s own code (assuming
> these are kept in 3 separate repos) should allow perfect reproducibility
> (modulo incompatible changes in emacs, I guess).

Sounds interesting.  I'll check it out. 


> It would be interesting for org to have an ability to reference files
> not just by name, but by git revision.  So that you could do something
> like (where 123456 is some git hash):
> #+include: [[gitbare:/path/to/repo::123456:my-org-setup-file.org]]
> and have org take care of checking out the proper revision and loading
> the file in the usual way.  This syntax is already implemented, for
> plain links, in contrib/lisp/org-git-link.el, so it is just a matter
> of making #+include and friends understand links in addition to
> filenames.

Now that is a great idea that allows for both incremental
improvements while still retaining compatibility for old files.

–Rasmus

-- 
And let me remind you also that moderation in the pursuit of justice
is no virtue

Re: Best practices for literate programming [was: Latex export of tables]

[Thomas S. Dye]

Aloha Rasmus,

Rasmus <rasmus@gmx.us> writes:

> The following message is a courtesy copy of an article
> that has been posted to gmane.emacs.orgmode as well.
>
>
> Thomas,
>
>>> Tom, do tell us more about what these habits are.
>>
>> The new exporter is really your friend.  Where before I might choose to
>> generate a LaTeX block, now I look to generate Org output and then count
>> on the exporter to do the right thing on the way to pdf.  
>>
>> The exporter's attribute system is very easy to use.  The attributes you
>> need to access are always right there.
>>
>> I've also come to rely on filters quite a bit. I use them for
>> non-breaking spaces, the plus/minus symbol, and for the multiple
>> citation commands used by biblatex (e.g., \parencites). There seems to
>> be a move afoot to collect filters so they can be widely distributed.
>> I'd like to see the filters go to the Library of Babel, but for
>> reproducible research it is probably best to keep them with the source
>> document so there is no doubt about the fidelity of filter code.
>
> I too rely heavily on filters and customizations.  I haven't been able
> to fully appreciate the asynchronous exporter yet.
>
> For instance I set some defaults for tables, pictures, add lots of
> entities etc. in my init file, and I went as far as writing a separate
> init file just loading just the org stuff.  Now, this is clearly /not/
> a very reproducible way of doing this.

I suppose this depends on what is meant by "reproducible."

My goal is to produce a compendium as defined by Gentleman and Lang
(see Gentleman R, Lang DT (2004). "Statistical Analyses and Reproducible
Research." Technical report, Bioconductor Project. URL
http://www.bepress.com/bioconductor/paper2).  

I keep the init.el file as a babel source block with the reproducible
document, so it can be tangled. I also have an editing setup in a babel
source block that activates many of the same features handled by the
init.el file, but also configures the new exporter to look for init.el
(which might have a different name). The filters are all part of the Org
document, too, and get pulled into the init.el file with noweb
references.

A compendium with this structure gets past the problem, often aired on
the ML, that there is "something in my setup" that causes unexpected
behavior.  The Org setup is completely contained in the compendium.

I am able to distribute the compendium, typically as a single document
(sometimes with associated data files produced by an on-line service
that can't be used programmatically), which I believe is a good step
toward reproducibility.

Of course, it leaves open the question of changes in the underlying
software. This is a real problem. Org 8.0, with its new (and sweet)
exporter has broken my first two compendia. Conceivably, changes in
Emacs might break a compendium, as could changes in all the other
software referenced by babel code blocks.  Aaron Ecay seems to be on to
a possible mechanism to take care of at least some of this.  AFAICT,
however, his solution doesn't change the utility of the compendium,
which seems to me an integral part of the reproducibility equation.

What do you think?  

>
> So I'm really interested in hearing or seeing implementation where the
> goal is reproducibility.  On one hand I can appreciate keeping Org
> close to a vanilla state.  On the other hand, I'd have to overwrite
> defaults every time (e.g. I /always/ want booktab tables).  I guess I
> could keep an emacs-lisp block in the top of the file specifying
> stuff, but it also seems kind of tedious (copy-pasting every time).
> (Perhaps this could be resolved by loading external files hosted
> somewhere accessible).

Some journals specify which LaTeX packages can or cannot be used.
PLOS-One, for instance, doesn't use booktab tables, so a reproducible
research document sent to them couldn't include your default setting.
My sense of the publishing world is that it is sufficiently variable
eventually to break almost any default you might hope to establish.   

Incidentally, I think this is an area ripe for growth within Org
mode--additions to the Library of Babel that configure a compendium to
produce LaTeX code that meets the requirements of particular publishing
venues. It would be ideal to do something like <<init-plos-one>> and
then, when the journal sends back your paper with a digital pink slip,
change that to something like <<init-nature>> and send it off again.

All the best,
Tom 

-- 
T.S. Dye & Colleagues, Archaeologists
735 Bishop St, Suite 315, Honolulu, HI 96813
Tel: 808-529-0866, Fax: 808-529-0884
http://www.tsdye.com

Re: Best practices for literate programming [was: Latex export of tables]

[Achim Gratz]

Aaron Ecay writes:
> If your external org configuration file were kept under version control
> (I’ll discuss git but the principle is general), then reproducibility
> would be possible.

There's a lot more to reproducibility then just this, but yes, the
configuration files would have to be part of it.

> There are ways of embedding git hashes in LaTeX
> documents (for one example:
> http://thorehusfeldt.net/2011/05/13/including-git-revision-identifiers-in-latex/),
> and of course org could help automate this.  Including the git hash of
> the document itself, the config file, and org-mode’s own code (assuming
> these are kept in 3 separate repos) should allow perfect reproducibility
> (modulo incompatible changes in emacs, I guess).

This is confused thinking and doesn't help anyway with the problem at
hand.  The purpose of Git is to record (and later re-create) the
complete state of your work tree, so monkeying around with hashes
embedded in document sources isn't making progress and recording several
hashes is either superfluous or a sign of incomplete control over the
work tree (you'd maybe want to use submodules).

What you can and should do however is putting a Git hash into the final
document so that this can be linked back to some state of the worktree
(like Org does for its manual and installed sources).

> It would be interesting for org to have an ability to reference files
> not just by name, but by git revision.  So that you could do something
> like (where 123456 is some git hash):
> #+include: [[gitbare:/path/to/repo::123456:my-org-setup-file.org]]
> and have org take care of checking out the proper revision and loading
> the file in the usual way.  This syntax is already implemented, for
> plain links, in contrib/lisp/org-git-link.el, so it is just a matter
> of making #+include and friends understand links in addition to
> filenames.

Git revisions are for the whole tree (or more precisely the commit that
references the tree), not single files.  If you access a file blob by
it's SHA-1 rather than name, then Git lets you do that, but you bypass
most of Git that way (much like you'd bypass the file system if you
started to access files by block numbers).


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: Best practices for literate programming [was: Latex export of tables]

[Rasmus Pank Roulund]

Dear Tom,

> I suppose this depends on what is meant by "reproducible."
>
> My goal is to produce a compendium as defined by Gentleman and Lang
> (see Gentleman R, Lang DT (2004). "Statistical Analyses and Reproducible
> Research." Technical report, Bioconductor Project. URL
> http://www.bepress.com/bioconductor/paper2).  
>
> I keep the init.el file as a babel source block with the reproducible
> document, so it can be tangled. I also have an editing setup in a babel
> source block that activates many of the same features handled by the
> init.el file, but also configures the new exporter to look for init.el
> (which might have a different name). The filters are all part of the Org
> document, too, and get pulled into the init.el file with noweb
> references.

My issue here is that this approach might lead to copy-paste
"preambles" which may or may not be desirable.  I can certainly see
the attraction in being able to just tangle the setup.  In fact for my
thesis I also had a preamble.tex blog in my file.  Your proposed setup
here is perhaps better in that it uses emacs-lisp.

Still, say I'm working on two files A and B.  If I fix a bug in
"preamble" A I would have to manually copy it over to B.  

Thus, the main question is how to distribute updates?  I guess one
could keep a separate file, but then we are back at square one in a
way. . .

One possibility might be a file structure like this

setup.org
A/project-A.org
A/setup-A.org
B/project-B.org
B/setup-B.org

where A and B both has a block like
#+BEGIN_SRC org
* Preamlbe    :noexport:
#+INCLUDE: "../setup.org"
#+INCLUDE: "setup-A.org"
#+END_SRC

To ship it off one would only have to write a command to replacing
#+INCLUDE with its content.  The exporter could likely be used for
this and one could produce an archive version when signing off a
project.

Even more robust, #+INCLUDE: would look for files in org-directory (it
might already do, I didn't check).

Am I missing something obvious (probably?) in the above stream of
random thoughts?  It's kind of a LaTeX-ish way of dealing with it, I
guess.

> I am able to distribute the compendium, typically as a single
> document (sometimes with associated data files produced by an
> on-line service that can't be used programmatically), which I
> believe is a good step toward reproducibility.

Agreed.

–Rasmus

-- 
Send from my Emacs