From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dov Grobgeld Subject: Re: org-mode and python pandas Date: Tue, 28 Apr 2015 11:36:53 +0300 Message-ID: References: <87bo6nkv0e.fsf@gmail.com> <87y59qdysz.fsf@Rainer.invalid> <87zju6jjo3.fsf@pank.eu> <8761wrpwew.fsf@gmail.com> Mime-Version: 1.0 Content-Type: multipart/alternative; boundary=047d7ba97b30bb12d10514c4c35c Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:59258) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Yn10j-0004Ys-Fp for emacs-orgmode@gnu.org; Tue, 28 Apr 2015 04:36:59 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Yn10h-0005qX-6U for emacs-orgmode@gnu.org; Tue, 28 Apr 2015 04:36:57 -0400 Received: from mail-wi0-x22c.google.com ([2a00:1450:400c:c05::22c]:38276) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Yn10g-0005qR-Qg for emacs-orgmode@gnu.org; Tue, 28 Apr 2015 04:36:55 -0400 Received: by wiun10 with SMTP id n10so19792060wiu.1 for ; Tue, 28 Apr 2015 01:36:53 -0700 (PDT) In-Reply-To: <8761wrpwew.fsf@gmail.com> List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: Eric Schulte Cc: emacs-orgmode , Rasmus --047d7ba97b30bb12d10514c4c35c Content-Type: text/plain; charset=UTF-8 I returned to this issue recently and tried to get the ob-python to output a table with a header, but didn't manage. Here is the code: #+BEGIN_SRC python :colnames true return [['','A','B','C'], [0,0.628365,0.424279,0.619791], [1,0.799666,0.527572,0.132928]] #+END_SRC #+RESULTS: | | A | B | C | | 0 | 0.628365 | 0.424279 | 0.619791 | | 1 | 0.799666 | 0.527572 | 0.132928 | What I want is: | | A | B | C | |---+----------+----------+----------| | 0 | 0.628365 | 0.424279 | 0.619791 | | 1 | 0.799666 | 0.527572 | 0.132928 | Is there any way to do that besides using the :results raw option? Thanks! Dov On Wed, Jul 3, 2013 at 5:09 PM, Eric Schulte wrote: > Dov Grobgeld writes: > > > Thanks for the answers, but there is still something missing in order > > to get it to work. Part of it seems to be connected to the python > > parsing. E.g. the following translation of Eric's sh example doesn't > > output correctly with python: > > > > > > #+BEGIN_SRC python :results output > > print """,A,B,C > > 0,0.628365,0.424279,0.619791 > > 1,0.799666,0.527572,0.132928 > > 2,0.837255,0.138906,0.408233 > > 3,0.388080,0.146212,0.575346 > > """ > > #+END_SRC > > > > #+RESULTS: > > : ,A,B,C > > : 0,0.628365,0.424279,0.619791 > > : 1,0.799666,0.527572,0.132928 > > : 2,0.837255,0.138906,0.408233 > > : 3,0.388080,0.146212,0.575346 > > > > > > #+BEGIN_SRC python :results table > > return """,A,B,C > > 0,0.628365,0.424279,0.619791 > > 1,0.799666,0.527572,0.132928 > > 2,0.837255,0.138906,0.408233 > > 3,0.388080,0.146212,0.575346 > > """ > > #+END_SRC > > > > #+RESULTS: > > | > ,A,B,C\n\n0,0.628365,0.424279,0.619791\n\n1,0.799666,0.527572,0.132928\n\n2,0.837255,0.138906,0.408233\n\n3,0.388080,0.146212,0.575346 > > | > > > > It seems that the only way to get a table from python is by outputting > > a two dimensional python structure: > > > > #+BEGIN_SRC python > > return [[0,0.628365,0.424279,0.619791], > > [1,0.799666,0.527572,0.132928]] > > #+END_SRC > > > > #+RESULTS: > > | 0 | 0.628365 | 0.424279 | 0.619791 | > > | 1 | 0.799666 | 0.527572 | 0.132928 | > > > > This seems quite limiting.... > > > > In most cases this is what one wants when returning data from python > code. The following elisp defined a "panda" code block, which is just > like python, only it assumes that the results will be these sort of > human readable strings instead of python code. > > ;; -*- emacs-lisp -*- > (defun org-babel-execute:panda (body params) > (let ((results > (org-babel-execute:python > body (org-babel-merge-params '((:results . "scalar")) > params)))) > (org-babel-result-cond (cdr (assoc :result-params params)) > results > (let ((tmp-file (org-babel-temp-file "sh-"))) > (with-temp-file tmp-file (insert results)) > (org-babel-import-elisp-from-file tmp-file))))) > > With the above evaluated the following works > > #+BEGIN_SRC panda > return """,A,B,C > 0,0.628365,0.424279,0.619791 > 1,0.799666,0.527572,0.132928 > 2,0.837255,0.138906,0.408233 > 3,0.388080,0.146212,0.575346 > """ > #+END_SRC > > #+RESULTS: > | | A | B | C | > | 0 | 0.628365 | 0.424279 | 0.619791 | > | 1 | 0.799666 | 0.527572 | 0.132928 | > | 2 | 0.837255 | 0.138906 | 0.408233 | > | 3 | 0.38808 | 0.146212 | 0.575346 | > > > > > Another related question is if there is any support for header tables? > > I.e. instead of this: > > > > | | A | B | C | > > | 0 | 0.827817 | 0.664009 | 0.089161 | > > | 1 | 0.170031 | 0.729214 | 0.110918 | > > | 2 | 0.575918 | 0.863924 | 0.757536 | > > | 3 | 0.682722 | 0.774445 | 0.992041 | > > > > I want this: > > > > | | A | B | C | > > |---+----------+----------+----------| > > | 0 | 0.827817 | 0.664009 | 0.089161 | > > | 1 | 0.170031 | 0.729214 | 0.110918 | > > | 2 | 0.575918 | 0.863924 | 0.757536 | > > | 3 | 0.682722 | 0.774445 | 0.992041 | > > > > I guess that if I start playing around with the python ob module, it > > should be possible to get this working? > > > > See the :colnames header argument in the manual. > > Best, > > > > > Regards, > > Dov > > > > On Mon, Jul 1, 2013 at 8:04 PM, Rasmus wrote: > >> Achim Gratz writes: > >> > >>>>> 2. Add to pandas the option of globally influencing the text > >>>>> formatting so that it outputs something more parsable by org-mode. > >>>> > >>>> This sounds promising, if pandas support csv output that will be > >>>> correctly parsed by Org-mode. > >>> > >>> The package already has CSV export, so one could use that. I don't > know > >>> if you could echo the result directly to the output, all examples > >>> revolve around putting the CSV into a file. For Org, TSV output would > >>> be more natural. > >> > >> Something like: > >> > >> from pandas import DataFrame > >> from numpy.random import rand > >> from sys import stdout > >> df = DataFrame(rand(10,3), columns = list('abc')) > >> df > >> df.to_csv(stdout, sep="\t", header = True, cols=(1,2)) > >> > >> I was completely unable to get ob-python working this morning, so I > >> haven't tested it. I'm using python3, build in python mode and elpy. > >> > >> In any case, the csv route might be better, as Pandas doesn't print > >> the table if it's too big (try changing 10 to 1000 above). > >> > >> -- > >> Powered by magic pixies! > >> > >> > > > > -- > Eric Schulte > http://cs.unm.edu/~eschulte > --047d7ba97b30bb12d10514c4c35c Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
I returned to this issue recently and tried to get the = ob-python to output a table with a header, but didn't manage. Here is t= he code:

#+BEGIN_SRC python :colnames true
return [['',&#= 39;A','B','C'],
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 [0,0.628365,0.424279,0.619791],
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 [1,0.799666,0.527572,0.132928]]
#+END_SRC

#+RESULTS:|=C2=A0=C2=A0 |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 A |=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 B |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 C |
| 0 | 0.628365 | 0.424279 | 0.619791 |
| 1 | 0.799666 | 0.527= 572 | 0.132928 |

What I want is:

|=C2=A0=C2=A0 |=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 A |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 B |=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 C |
|---+----= ------+----------+----------|
| 0 | 0.628365 | 0.424279 | 0.619791 |
= | 1 | 0.799666 | 0.527572 | 0.132928 |

Is there any way to = do that besides using the :results raw option?

Thanks!
<= /div>
Dov


On Wed, Jul 3, 2013 at 5:09 PM, Eric Schulte <schulte.e= ric@gmail.com> wrote:
Dov Grobgeld <dov.grobgeld@gmail.com> writes:

> Thanks for the answers, but there is still something missing in order<= br> > to get it to work. Part of it seems to be connected to the python
> parsing. E.g. the following translation of Eric's sh example doesn= 't
> output correctly with python:
>
>
> #+BEGIN_SRC python :results output
> print """,A,B,C
> 0,0.628365,0.424279,0.619791
> 1,0.799666,0.527572,0.132928
> 2,0.837255,0.138906,0.408233
> 3,0.388080,0.146212,0.575346
> """
> #+END_SRC
>
> #+RESULTS:
> : ,A,B,C
> : 0,0.628365,0.424279,0.619791
> : 1,0.799666,0.527572,0.132928
> : 2,0.837255,0.138906,0.408233
> : 3,0.388080,0.146212,0.575346
>
>
> #+BEGIN_SRC python :results table
> return """,A,B,C
> 0,0.628365,0.424279,0.619791
> 1,0.799666,0.527572,0.132928
> 2,0.837255,0.138906,0.408233
> 3,0.388080,0.146212,0.575346
> """
> #+END_SRC
>
> #+RESULTS:
> | ,A,B,C\n\n0,0.628365,0.424279,0.619791\n\n1,0.799666,0.527572,0.1329= 28\n\n2,0.837255,0.138906,0.408233\n\n3,0.388080,0.146212,0.575346
> |
>
> It seems that the only way to get a table from python is by outputting=
> a two dimensional python structure:
>
> #+BEGIN_SRC python
> return [[0,0.628365,0.424279,0.619791],
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0[1,0.799666,0.527572,0.132928]]
> #+END_SRC
>
> #+RESULTS:
> | 0 | 0.628365 | 0.424279 | 0.619791 |
> | 1 | 0.799666 | 0.527572 | 0.132928 |
>
> This seems quite limiting....
>

In most cases this is what one wants when returning data from p= ython
code.=C2=A0 The following elisp defined a "panda" code block, whi= ch is just
like python, only it assumes that the results will be these sort of
human readable strings instead of python code.

=C2=A0 =C2=A0 ;; -*- emacs-lisp -*-
=C2=A0 =C2=A0 (defun org-babel-execute:panda (body params)
=C2=A0 =C2=A0 =C2=A0 (let ((results
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(org-babel-execute:python =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 body (org-babel-merge-para= ms '((:results . "scalar")) params))))
=C2=A0 =C2=A0 =C2=A0 =C2=A0 (org-babel-result-cond (cdr (assoc :result-para= ms params))
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 results
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (let ((tmp-file (org-babel-temp-file &qu= ot;sh-")))
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (with-temp-file tmp-file (insert = results))
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (org-babel-import-elisp-from-file= tmp-file)))))

With the above evaluated the following works

=C2=A0 =C2=A0 #+BEGIN_SRC panda
=C2=A0 =C2=A0 return """,A,B,C
=C2=A0 =C2=A0 0,0.628365,0.424279,0.619791
=C2=A0 =C2=A0 1,0.799666,0.527572,0.132928
=C2=A0 =C2=A0 2,0.837255,0.138906,0.408233
=C2=A0 =C2=A0 3,0.388080,0.146212,0.575346
=C2=A0 =C2=A0 """
=C2=A0 =C2=A0 #+END_SRC

=C2=A0 =C2=A0 #+RESULTS:
=C2=A0 =C2=A0 |=C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 A |=C2=A0 = =C2=A0 =C2=A0 =C2=A0 B |=C2=A0 =C2=A0 =C2=A0 =C2=A0 C |
=C2=A0 =C2=A0 | 0 | 0.628365 | 0.424279 | 0.619791 |
=C2=A0 =C2=A0 | 1 | 0.799666 | 0.527572 | 0.132928 |
=C2=A0 =C2=A0 | 2 | 0.837255 | 0.138906 | 0.408233 = |
=C2=A0 =C2=A0 | 3 |=C2=A0 0.38808 | 0.146212 | 0.575346 |

>
> Another related question is if there is any su= pport for header tables?
> I.e. instead of this:
>
> |=C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 A |=C2=A0 =C2=A0 =C2=A0 =C2= =A0 B |=C2=A0 =C2=A0 =C2=A0 =C2=A0 C |
> | 0 | 0.827817 | 0.664009 | 0.089161 |
> | 1 | 0.170031 | 0.729214 | 0.110918 |
> | 2 | 0.575918 | 0.863924 | 0.757536 |
> | 3 | 0.682722 | 0.774445 | 0.992041 |
>
> I want this:
>
> |=C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 A |=C2=A0 =C2=A0 =C2=A0 =C2= =A0 B |=C2=A0 =C2=A0 =C2=A0 =C2=A0 C |
> |---+----------+----------+----------|
> | 0 | 0.827817 | 0.664009 | 0.089161 |
> | 1 | 0.170031 | 0.729214 | 0.110918 |
> | 2 | 0.575918 | 0.863924 | 0.757536 |
> | 3 | 0.682722 | 0.774445 | 0.992041 |
>
> I guess that if I start playing around with the python ob module, it > should be possible to get this working?
>

See the :colnames header argument in the manual.

Best,

>
> Regards,
> Dov
>
> On Mon, Jul 1, 2013 at 8:04 PM, Rasmus <rasmus@gmx.us> wrote:
>> Achim Gratz <Stromeko@nexg= o.de> writes:
>>
>>>>> 2. Add to pandas the option of globally influencing th= e text
>>>>> formatting so that it outputs something more parsable = by org-mode.
>>>>
>>>> This sounds promising, if pandas support csv output that w= ill be
>>>> correctly parsed by Org-mode.
>>>
>>> The package already has CSV export, so one could use that.=C2= =A0 I don't know
>>> if you could echo the result directly to the output, all examp= les
>>> revolve around putting the CSV into a file.=C2=A0 For Org, TSV= output would
>>> be more natural.
>>
>> Something like:
>>
>> from pandas import DataFrame
>> from numpy.random import rand
>> from sys import stdout
>> df =3D DataFrame(rand(10,3), columns =3D list('abc'))
>> df
>> df.to_csv(stdout, sep=3D"\t", header =3D True, cols=3D(1= ,2))
>>
>> I was completely unable to get ob-python working this morning, so = I
>> haven't tested it.=C2=A0 I'm using python3, build in pytho= n mode and elpy.
>>
>> In any case, the csv route might be better, as Pandas doesn't = print
>> the table if it's too big (try changing 10 to 1000 above).
>>
>> --
>> Powered by magic pixies!
>>
>>
>

--
Eric Schulte
http://cs.unm.edu= /~eschulte

--047d7ba97b30bb12d10514c4c35c--