emacs-orgmode@gnu.org archives
 help / color / mirror / code / Atom feed
* org-mode and python pandas
@ 2013-06-28  5:26 Dov Grobgeld
  2013-06-30 23:15 ` Eric Schulte
  0 siblings, 1 reply; 9+ messages in thread
From: Dov Grobgeld @ 2013-06-28  5:26 UTC (permalink / raw)
  To: emacs-orgmode

Has anyone used org-mode with the python pandas package? Pandas is in
a certain way an alternative to R, but with the (for me) familiar
syntax of python. See: http://pandas.pydata.org/

Pandas is very much built to be used interactively, and it outputs its
data in space separated tabular format. E.g. in ipython:

In [1]: import pandas as pd
In [2]: import numpy as np

In [3]: pd.DataFrame(np.random.random((4,3)), columns=['A','B','C'])
Out[3]:
          A         B         C
0  0.628365  0.424279  0.619791
1  0.799666  0.527572  0.132928
2  0.837255  0.138906  0.408233
3  0.388080  0.146212  0.575346

Unfortunately this doesn't output as nicely when used from org-mode:

#+BEGIN_SRC python
import pandas as pd
import numpy as np

return pd.DataFrame(np.random.random((4,3)), columns=list('ABC'))
#+END_SRC

#+RESULTS:
: A         B         C
: 0  0.827817  0.664009  0.089161
: 1  0.170031  0.729214  0.110918
: 2  0.575918  0.863924  0.757536
: 3  0.682722  0.774445  0.992041

while I would like to have:

|   |        A |        B |        C |
|---+----------+----------+----------|
| 0 | 0.827817 | 0.664009 | 0.089161 |
| 1 | 0.170031 | 0.729214 | 0.110918 |
| 2 | 0.575918 | 0.863924 | 0.757536 |
| 3 | 0.682722 | 0.774445 | 0.992041 |

The question is how to get this? Here are a few ideas:

1. Write a general filter in the org-mode elisp than uses heuristics
to recognize ascii aligned tables and change these to org-tables.
2. Add to pandas the option of globally influencing the text
formatting so that it outputs something more parsable by org-mode.
3. Create a special language "pandas" that recognize the ascii aligned
tables and saves the need to import pandas and np?
4. And the obvious approach of writing a python function that writes a
org-mode parsable table and always call it as part of the return.

Which is the preferable approach? Any other ideas?

Regards,
Dov

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: org-mode and python pandas
  2013-06-28  5:26 org-mode and python pandas Dov Grobgeld
@ 2013-06-30 23:15 ` Eric Schulte
  2013-07-01 16:34   ` Achim Gratz
  0 siblings, 1 reply; 9+ messages in thread
From: Eric Schulte @ 2013-06-30 23:15 UTC (permalink / raw)
  To: Dov Grobgeld; +Cc: emacs-orgmode

Dov Grobgeld <dov.grobgeld@gmail.com> writes:

> Has anyone used org-mode with the python pandas package? Pandas is in
> a certain way an alternative to R, but with the (for me) familiar
> syntax of python. See: http://pandas.pydata.org/
>
> Pandas is very much built to be used interactively, and it outputs its
> data in space separated tabular format. E.g. in ipython:
>
> In [1]: import pandas as pd
> In [2]: import numpy as np
>
> In [3]: pd.DataFrame(np.random.random((4,3)), columns=['A','B','C'])
> Out[3]:
>           A         B         C
> 0  0.628365  0.424279  0.619791
> 1  0.799666  0.527572  0.132928
> 2  0.837255  0.138906  0.408233
> 3  0.388080  0.146212  0.575346
>
> Unfortunately this doesn't output as nicely when used from org-mode:
>
> #+BEGIN_SRC python
> import pandas as pd
> import numpy as np
>
> return pd.DataFrame(np.random.random((4,3)), columns=list('ABC'))
> #+END_SRC
>
> #+RESULTS:
> : A         B         C
> : 0  0.827817  0.664009  0.089161
> : 1  0.170031  0.729214  0.110918
> : 2  0.575918  0.863924  0.757536
> : 3  0.682722  0.774445  0.992041
>
> while I would like to have:
>
> |   |        A |        B |        C |
> |---+----------+----------+----------|
> | 0 | 0.827817 | 0.664009 | 0.089161 |
> | 1 | 0.170031 | 0.729214 | 0.110918 |
> | 2 | 0.575918 | 0.863924 | 0.757536 |
> | 3 | 0.682722 | 0.774445 | 0.992041 |
>

What happens if you add ":results table" to your code block?  Would that
be sufficient?

>
> The question is how to get this? Here are a few ideas:
>
> 1. Write a general filter in the org-mode elisp than uses heuristics
> to recognize ascii aligned tables and change these to org-tables.

The default value should be to convert multi-line output to tables, the
":results table" option above will force this conversion in case it is
currently not taking place due to the default header arguments in use.

> 
> 2. Add to pandas the option of globally influencing the text
> formatting so that it outputs something more parsable by org-mode.

This sounds promising, if pandas support csv output that will be
correctly parsed by Org-mode.

> 
> 3. Create a special language "pandas" that recognize the ascii aligned
> tables and saves the need to import pandas and np?  4. And the obvious
> approach of writing a python function that writes a org-mode parsable
> table and always call it as part of the return.
>
> Which is the preferable approach? Any other ideas?
>

I think a header-argument-based approach would be ideal, I'd look at the
value of org-babel-default-header-args:python, and read the portion of
the manual related to the "results" header arguments.

I don't understand multi-line strings in python, but I get the following
behavior from simple shell script blocks.

#+begin_src sh
  cat <<EOF
            A         B         C
  0  0.628365  0.424279  0.619791
  1  0.799666  0.527572  0.132928
  2  0.837255  0.138906  0.408233
  3  0.388080  0.146212  0.575346
  EOF
#+end_src

#+RESULTS:
| A |        B |        C |          |
| 0 | 0.628365 | 0.424279 | 0.619791 |
| 1 | 0.799666 | 0.527572 | 0.132928 |
| 2 | 0.837255 | 0.138906 | 0.408233 |
| 3 |  0.38808 | 0.146212 | 0.575346 |

#+begin_src sh
  cat <<EOF
  ,A,B,C
  0,0.628365,0.424279,0.619791
  1,0.799666,0.527572,0.132928
  2,0.837255,0.138906,0.408233
  3,0.388080,0.146212,0.575346
  EOF
#+end_src

#+RESULTS:
|   |        A |        B |        C |
| 0 | 0.628365 | 0.424279 | 0.619791 |
| 1 | 0.799666 | 0.527572 | 0.132928 |
| 2 | 0.837255 | 0.138906 | 0.408233 |
| 3 |  0.38808 | 0.146212 | 0.575346 |

Hope this helps,

>
> Regards,
> Dov
>

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: org-mode and python pandas
  2013-06-30 23:15 ` Eric Schulte
@ 2013-07-01 16:34   ` Achim Gratz
  2013-07-01 17:04     ` Rasmus
  0 siblings, 1 reply; 9+ messages in thread
From: Achim Gratz @ 2013-07-01 16:34 UTC (permalink / raw)
  To: emacs-orgmode

Eric Schulte writes:
>> |   |        A |        B |        C |
>> |---+----------+----------+----------|
>> | 0 | 0.827817 | 0.664009 | 0.089161 |
>> | 1 | 0.170031 | 0.729214 | 0.110918 |
>> | 2 | 0.575918 | 0.863924 | 0.757536 |
>> | 3 | 0.682722 | 0.774445 | 0.992041 |
>>
>
> What happens if you add ":results table" to your code block?  Would that
> be sufficient?

The problem is that he's trying to consume human-readable output.  The
alignment in this output is done with spaces, not tabs (as Org would
expect).  There are more problems considering that the numbers likely
lose precision in this format (nota bene: there are other Babel
languages that have that same problem).

> The default value should be to convert multi-line output to tables, the
> ":results table" option above will force this conversion in case it is
> currently not taking place due to the default header arguments in use.

Since there are no tabs in there, it is impossible to skip empty cells
like the first one (they'd have to be written as "" instead, as would be
cells that should contain literal spaces).

>> 2. Add to pandas the option of globally influencing the text
>> formatting so that it outputs something more parsable by org-mode.
>
> This sounds promising, if pandas support csv output that will be
> correctly parsed by Org-mode.

The package already has CSV export, so one could use that.  I don't know
if you could echo the result directly to the output, all examples
revolve around putting the CSV into a file.  For Org, TSV output would
be more natural.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Waldorf MIDI Implementation & additional documentation:
http://Synth.Stromeko.net/Downloads.html#WaldorfDocs

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: org-mode and python pandas
  2013-07-01 16:34   ` Achim Gratz
@ 2013-07-01 17:04     ` Rasmus
  2013-07-03  9:15       ` Dov Grobgeld
  0 siblings, 1 reply; 9+ messages in thread
From: Rasmus @ 2013-07-01 17:04 UTC (permalink / raw)
  To: emacs-orgmode

Achim Gratz <Stromeko@nexgo.de> writes:

>>> 2. Add to pandas the option of globally influencing the text
>>> formatting so that it outputs something more parsable by org-mode.
>>
>> This sounds promising, if pandas support csv output that will be
>> correctly parsed by Org-mode.
>
> The package already has CSV export, so one could use that.  I don't know
> if you could echo the result directly to the output, all examples
> revolve around putting the CSV into a file.  For Org, TSV output would
> be more natural.

Something like:

from pandas import DataFrame
from numpy.random import rand
from sys import stdout
df = DataFrame(rand(10,3), columns = list('abc'))
df
df.to_csv(stdout, sep="\t", header = True, cols=(1,2))

I was completely unable to get ob-python working this morning, so I
haven't tested it.  I'm using python3, build in python mode and elpy.

In any case, the csv route might be better, as Pandas doesn't print
the table if it's too big (try changing 10 to 1000 above).

-- 
Powered by magic pixies!

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: org-mode and python pandas
  2013-07-01 17:04     ` Rasmus
@ 2013-07-03  9:15       ` Dov Grobgeld
  2013-07-03 10:31         ` Rasmus
  2013-07-03 14:09         ` Eric Schulte
  0 siblings, 2 replies; 9+ messages in thread
From: Dov Grobgeld @ 2013-07-03  9:15 UTC (permalink / raw)
  To: Rasmus; +Cc: emacs-orgmode

Thanks for the answers, but there is still something missing in order
to get it to work. Part of it seems to be connected to the python
parsing. E.g. the following translation of Eric's sh example doesn't
output correctly with python:


#+BEGIN_SRC python :results output
print """,A,B,C
0,0.628365,0.424279,0.619791
1,0.799666,0.527572,0.132928
2,0.837255,0.138906,0.408233
3,0.388080,0.146212,0.575346
"""
#+END_SRC

#+RESULTS:
: ,A,B,C
: 0,0.628365,0.424279,0.619791
: 1,0.799666,0.527572,0.132928
: 2,0.837255,0.138906,0.408233
: 3,0.388080,0.146212,0.575346


#+BEGIN_SRC python :results table
return """,A,B,C
0,0.628365,0.424279,0.619791
1,0.799666,0.527572,0.132928
2,0.837255,0.138906,0.408233
3,0.388080,0.146212,0.575346
"""
#+END_SRC

#+RESULTS:
| ,A,B,C\n\n0,0.628365,0.424279,0.619791\n\n1,0.799666,0.527572,0.132928\n\n2,0.837255,0.138906,0.408233\n\n3,0.388080,0.146212,0.575346
|

It seems that the only way to get a table from python is by outputting
a two dimensional python structure:

#+BEGIN_SRC python
return [[0,0.628365,0.424279,0.619791],
        [1,0.799666,0.527572,0.132928]]
#+END_SRC

#+RESULTS:
| 0 | 0.628365 | 0.424279 | 0.619791 |
| 1 | 0.799666 | 0.527572 | 0.132928 |

This seems quite limiting....

Another related question is if there is any support for header tables?
I.e. instead of this:

|   |        A |        B |        C |
| 0 | 0.827817 | 0.664009 | 0.089161 |
| 1 | 0.170031 | 0.729214 | 0.110918 |
| 2 | 0.575918 | 0.863924 | 0.757536 |
| 3 | 0.682722 | 0.774445 | 0.992041 |

I want this:

|   |        A |        B |        C |
|---+----------+----------+----------|
| 0 | 0.827817 | 0.664009 | 0.089161 |
| 1 | 0.170031 | 0.729214 | 0.110918 |
| 2 | 0.575918 | 0.863924 | 0.757536 |
| 3 | 0.682722 | 0.774445 | 0.992041 |

I guess that if I start playing around with the python ob module, it
should be possible to get this working?

Regards,
Dov

On Mon, Jul 1, 2013 at 8:04 PM, Rasmus <rasmus@gmx.us> wrote:
> Achim Gratz <Stromeko@nexgo.de> writes:
>
>>>> 2. Add to pandas the option of globally influencing the text
>>>> formatting so that it outputs something more parsable by org-mode.
>>>
>>> This sounds promising, if pandas support csv output that will be
>>> correctly parsed by Org-mode.
>>
>> The package already has CSV export, so one could use that.  I don't know
>> if you could echo the result directly to the output, all examples
>> revolve around putting the CSV into a file.  For Org, TSV output would
>> be more natural.
>
> Something like:
>
> from pandas import DataFrame
> from numpy.random import rand
> from sys import stdout
> df = DataFrame(rand(10,3), columns = list('abc'))
> df
> df.to_csv(stdout, sep="\t", header = True, cols=(1,2))
>
> I was completely unable to get ob-python working this morning, so I
> haven't tested it.  I'm using python3, build in python mode and elpy.
>
> In any case, the csv route might be better, as Pandas doesn't print
> the table if it's too big (try changing 10 to 1000 above).
>
> --
> Powered by magic pixies!
>
>

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: org-mode and python pandas
  2013-07-03  9:15       ` Dov Grobgeld
@ 2013-07-03 10:31         ` Rasmus
  2013-07-03 14:09         ` Eric Schulte
  1 sibling, 0 replies; 9+ messages in thread
From: Rasmus @ 2013-07-03 10:31 UTC (permalink / raw)
  To: emacs-orgmode

Hi Dov,

> Another related question is if there is any support for header tables?
> I.e. instead of this:
>
> |   |        A |        B |        C |
> | 0 | 0.827817 | 0.664009 | 0.089161 |
> | 1 | 0.170031 | 0.729214 | 0.110918 |
> | 2 | 0.575918 | 0.863924 | 0.757536 |
> | 3 | 0.682722 | 0.774445 | 0.992041 |
>
> I want this:
>
> |   |        A |        B |        C |
> |---+----------+----------+----------|
> | 0 | 0.827817 | 0.664009 | 0.089161 |
> | 1 | 0.170031 | 0.729214 | 0.110918 |
> | 2 | 0.575918 | 0.863924 | 0.757536 |
> | 3 | 0.682722 | 0.774445 | 0.992041 |

This is an issue in R as well, e.g. often I might want to have 
|A|
|-|
|1|
|2|
|-|
|3|

where the last is a summary statistic.  The last hline can be gotten
as 

#+BEGIN_SRC emacs-lisp :var X=mytabel :exports results
    (let
        ((L (- (length X) 1)))
      (append
       (subseq X 0 L)
       (list 'hline)
       (subseq X L)))
#+END_SRC

Perhaps a better way exists?

–Rasmus

-- 
. . . It begins of course with The Internet.  A Net of Peers.

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: org-mode and python pandas
  2013-07-03  9:15       ` Dov Grobgeld
  2013-07-03 10:31         ` Rasmus
@ 2013-07-03 14:09         ` Eric Schulte
  2015-04-28  8:36           ` Dov Grobgeld
  1 sibling, 1 reply; 9+ messages in thread
From: Eric Schulte @ 2013-07-03 14:09 UTC (permalink / raw)
  To: Dov Grobgeld; +Cc: emacs-orgmode, Rasmus

Dov Grobgeld <dov.grobgeld@gmail.com> writes:

> Thanks for the answers, but there is still something missing in order
> to get it to work. Part of it seems to be connected to the python
> parsing. E.g. the following translation of Eric's sh example doesn't
> output correctly with python:
>
>
> #+BEGIN_SRC python :results output
> print """,A,B,C
> 0,0.628365,0.424279,0.619791
> 1,0.799666,0.527572,0.132928
> 2,0.837255,0.138906,0.408233
> 3,0.388080,0.146212,0.575346
> """
> #+END_SRC
>
> #+RESULTS:
> : ,A,B,C
> : 0,0.628365,0.424279,0.619791
> : 1,0.799666,0.527572,0.132928
> : 2,0.837255,0.138906,0.408233
> : 3,0.388080,0.146212,0.575346
>
>
> #+BEGIN_SRC python :results table
> return """,A,B,C
> 0,0.628365,0.424279,0.619791
> 1,0.799666,0.527572,0.132928
> 2,0.837255,0.138906,0.408233
> 3,0.388080,0.146212,0.575346
> """
> #+END_SRC
>
> #+RESULTS:
> | ,A,B,C\n\n0,0.628365,0.424279,0.619791\n\n1,0.799666,0.527572,0.132928\n\n2,0.837255,0.138906,0.408233\n\n3,0.388080,0.146212,0.575346
> |
>
> It seems that the only way to get a table from python is by outputting
> a two dimensional python structure:
>
> #+BEGIN_SRC python
> return [[0,0.628365,0.424279,0.619791],
>         [1,0.799666,0.527572,0.132928]]
> #+END_SRC
>
> #+RESULTS:
> | 0 | 0.628365 | 0.424279 | 0.619791 |
> | 1 | 0.799666 | 0.527572 | 0.132928 |
>
> This seems quite limiting....
>

In most cases this is what one wants when returning data from python
code.  The following elisp defined a "panda" code block, which is just
like python, only it assumes that the results will be these sort of
human readable strings instead of python code.

    ;; -*- emacs-lisp -*-
    (defun org-babel-execute:panda (body params)
      (let ((results
             (org-babel-execute:python
              body (org-babel-merge-params '((:results . "scalar")) params))))
        (org-babel-result-cond (cdr (assoc :result-params params))
          results
          (let ((tmp-file (org-babel-temp-file "sh-")))
            (with-temp-file tmp-file (insert results))
            (org-babel-import-elisp-from-file tmp-file)))))

With the above evaluated the following works

    #+BEGIN_SRC panda
    return """,A,B,C
    0,0.628365,0.424279,0.619791
    1,0.799666,0.527572,0.132928
    2,0.837255,0.138906,0.408233
    3,0.388080,0.146212,0.575346
    """
    #+END_SRC

    #+RESULTS:
    |   |        A |        B |        C |
    | 0 | 0.628365 | 0.424279 | 0.619791 |
    | 1 | 0.799666 | 0.527572 | 0.132928 |
    | 2 | 0.837255 | 0.138906 | 0.408233 |
    | 3 |  0.38808 | 0.146212 | 0.575346 |

>
> Another related question is if there is any support for header tables?
> I.e. instead of this:
>
> |   |        A |        B |        C |
> | 0 | 0.827817 | 0.664009 | 0.089161 |
> | 1 | 0.170031 | 0.729214 | 0.110918 |
> | 2 | 0.575918 | 0.863924 | 0.757536 |
> | 3 | 0.682722 | 0.774445 | 0.992041 |
>
> I want this:
>
> |   |        A |        B |        C |
> |---+----------+----------+----------|
> | 0 | 0.827817 | 0.664009 | 0.089161 |
> | 1 | 0.170031 | 0.729214 | 0.110918 |
> | 2 | 0.575918 | 0.863924 | 0.757536 |
> | 3 | 0.682722 | 0.774445 | 0.992041 |
>
> I guess that if I start playing around with the python ob module, it
> should be possible to get this working?
>

See the :colnames header argument in the manual.

Best,

>
> Regards,
> Dov
>
> On Mon, Jul 1, 2013 at 8:04 PM, Rasmus <rasmus@gmx.us> wrote:
>> Achim Gratz <Stromeko@nexgo.de> writes:
>>
>>>>> 2. Add to pandas the option of globally influencing the text
>>>>> formatting so that it outputs something more parsable by org-mode.
>>>>
>>>> This sounds promising, if pandas support csv output that will be
>>>> correctly parsed by Org-mode.
>>>
>>> The package already has CSV export, so one could use that.  I don't know
>>> if you could echo the result directly to the output, all examples
>>> revolve around putting the CSV into a file.  For Org, TSV output would
>>> be more natural.
>>
>> Something like:
>>
>> from pandas import DataFrame
>> from numpy.random import rand
>> from sys import stdout
>> df = DataFrame(rand(10,3), columns = list('abc'))
>> df
>> df.to_csv(stdout, sep="\t", header = True, cols=(1,2))
>>
>> I was completely unable to get ob-python working this morning, so I
>> haven't tested it.  I'm using python3, build in python mode and elpy.
>>
>> In any case, the csv route might be better, as Pandas doesn't print
>> the table if it's too big (try changing 10 to 1000 above).
>>
>> --
>> Powered by magic pixies!
>>
>>
>

-- 
Eric Schulte
http://cs.unm.edu/~eschulte

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: org-mode and python pandas
  2013-07-03 14:09         ` Eric Schulte
@ 2015-04-28  8:36           ` Dov Grobgeld
  0 siblings, 0 replies; 9+ messages in thread
From: Dov Grobgeld @ 2015-04-28  8:36 UTC (permalink / raw)
  To: Eric Schulte; +Cc: emacs-orgmode, Rasmus

[-- Attachment #1: Type: text/plain, Size: 5615 bytes --]

I returned to this issue recently and tried to get the ob-python to output
a table with a header, but didn't manage. Here is the code:

#+BEGIN_SRC python :colnames true
return [['','A','B','C'],
        [0,0.628365,0.424279,0.619791],
        [1,0.799666,0.527572,0.132928]]
#+END_SRC

#+RESULTS:
|   |        A |        B |        C |
| 0 | 0.628365 | 0.424279 | 0.619791 |
| 1 | 0.799666 | 0.527572 | 0.132928 |

What I want is:

|   |        A |        B |        C |
|---+----------+----------+----------|
| 0 | 0.628365 | 0.424279 | 0.619791 |
| 1 | 0.799666 | 0.527572 | 0.132928 |

Is there any way to do that besides using the :results raw option?

Thanks!
Dov


On Wed, Jul 3, 2013 at 5:09 PM, Eric Schulte <schulte.eric@gmail.com> wrote:

> Dov Grobgeld <dov.grobgeld@gmail.com> writes:
>
> > Thanks for the answers, but there is still something missing in order
> > to get it to work. Part of it seems to be connected to the python
> > parsing. E.g. the following translation of Eric's sh example doesn't
> > output correctly with python:
> >
> >
> > #+BEGIN_SRC python :results output
> > print """,A,B,C
> > 0,0.628365,0.424279,0.619791
> > 1,0.799666,0.527572,0.132928
> > 2,0.837255,0.138906,0.408233
> > 3,0.388080,0.146212,0.575346
> > """
> > #+END_SRC
> >
> > #+RESULTS:
> > : ,A,B,C
> > : 0,0.628365,0.424279,0.619791
> > : 1,0.799666,0.527572,0.132928
> > : 2,0.837255,0.138906,0.408233
> > : 3,0.388080,0.146212,0.575346
> >
> >
> > #+BEGIN_SRC python :results table
> > return """,A,B,C
> > 0,0.628365,0.424279,0.619791
> > 1,0.799666,0.527572,0.132928
> > 2,0.837255,0.138906,0.408233
> > 3,0.388080,0.146212,0.575346
> > """
> > #+END_SRC
> >
> > #+RESULTS:
> > |
> ,A,B,C\n\n0,0.628365,0.424279,0.619791\n\n1,0.799666,0.527572,0.132928\n\n2,0.837255,0.138906,0.408233\n\n3,0.388080,0.146212,0.575346
> > |
> >
> > It seems that the only way to get a table from python is by outputting
> > a two dimensional python structure:
> >
> > #+BEGIN_SRC python
> > return [[0,0.628365,0.424279,0.619791],
> >         [1,0.799666,0.527572,0.132928]]
> > #+END_SRC
> >
> > #+RESULTS:
> > | 0 | 0.628365 | 0.424279 | 0.619791 |
> > | 1 | 0.799666 | 0.527572 | 0.132928 |
> >
> > This seems quite limiting....
> >
>
> In most cases this is what one wants when returning data from python
> code.  The following elisp defined a "panda" code block, which is just
> like python, only it assumes that the results will be these sort of
> human readable strings instead of python code.
>
>     ;; -*- emacs-lisp -*-
>     (defun org-babel-execute:panda (body params)
>       (let ((results
>              (org-babel-execute:python
>               body (org-babel-merge-params '((:results . "scalar"))
> params))))
>         (org-babel-result-cond (cdr (assoc :result-params params))
>           results
>           (let ((tmp-file (org-babel-temp-file "sh-")))
>             (with-temp-file tmp-file (insert results))
>             (org-babel-import-elisp-from-file tmp-file)))))
>
> With the above evaluated the following works
>
>     #+BEGIN_SRC panda
>     return """,A,B,C
>     0,0.628365,0.424279,0.619791
>     1,0.799666,0.527572,0.132928
>     2,0.837255,0.138906,0.408233
>     3,0.388080,0.146212,0.575346
>     """
>     #+END_SRC
>
>     #+RESULTS:
>     |   |        A |        B |        C |
>     | 0 | 0.628365 | 0.424279 | 0.619791 |
>     | 1 | 0.799666 | 0.527572 | 0.132928 |
>     | 2 | 0.837255 | 0.138906 | 0.408233 |
>     | 3 |  0.38808 | 0.146212 | 0.575346 |
>
> >
> > Another related question is if there is any support for header tables?
> > I.e. instead of this:
> >
> > |   |        A |        B |        C |
> > | 0 | 0.827817 | 0.664009 | 0.089161 |
> > | 1 | 0.170031 | 0.729214 | 0.110918 |
> > | 2 | 0.575918 | 0.863924 | 0.757536 |
> > | 3 | 0.682722 | 0.774445 | 0.992041 |
> >
> > I want this:
> >
> > |   |        A |        B |        C |
> > |---+----------+----------+----------|
> > | 0 | 0.827817 | 0.664009 | 0.089161 |
> > | 1 | 0.170031 | 0.729214 | 0.110918 |
> > | 2 | 0.575918 | 0.863924 | 0.757536 |
> > | 3 | 0.682722 | 0.774445 | 0.992041 |
> >
> > I guess that if I start playing around with the python ob module, it
> > should be possible to get this working?
> >
>
> See the :colnames header argument in the manual.
>
> Best,
>
> >
> > Regards,
> > Dov
> >
> > On Mon, Jul 1, 2013 at 8:04 PM, Rasmus <rasmus@gmx.us> wrote:
> >> Achim Gratz <Stromeko@nexgo.de> writes:
> >>
> >>>>> 2. Add to pandas the option of globally influencing the text
> >>>>> formatting so that it outputs something more parsable by org-mode.
> >>>>
> >>>> This sounds promising, if pandas support csv output that will be
> >>>> correctly parsed by Org-mode.
> >>>
> >>> The package already has CSV export, so one could use that.  I don't
> know
> >>> if you could echo the result directly to the output, all examples
> >>> revolve around putting the CSV into a file.  For Org, TSV output would
> >>> be more natural.
> >>
> >> Something like:
> >>
> >> from pandas import DataFrame
> >> from numpy.random import rand
> >> from sys import stdout
> >> df = DataFrame(rand(10,3), columns = list('abc'))
> >> df
> >> df.to_csv(stdout, sep="\t", header = True, cols=(1,2))
> >>
> >> I was completely unable to get ob-python working this morning, so I
> >> haven't tested it.  I'm using python3, build in python mode and elpy.
> >>
> >> In any case, the csv route might be better, as Pandas doesn't print
> >> the table if it's too big (try changing 10 to 1000 above).
> >>
> >> --
> >> Powered by magic pixies!
> >>
> >>
> >
>
> --
> Eric Schulte
> http://cs.unm.edu/~eschulte
>

[-- Attachment #2: Type: text/html, Size: 8155 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: org-mode and python pandas
@ 2015-04-29  7:12 Dror Atariah
  0 siblings, 0 replies; 9+ messages in thread
From: Dror Atariah @ 2015-04-29  7:12 UTC (permalink / raw)
  To: emacs-orgmode@gnu.org

[-- Attachment #1: Type: text/plain, Size: 4554 bytes --]

Using the tabulate python module it is possible to have the following
"inline" workaround:

------------8<------------8<------------8<------------8<------------8<------------8<------------8<------------
#+BEGIN_SRC python :results raw
import pandas as pd
import numpy as np
from tabulate import tabulate

df = pd.DataFrame(np.random.random((4,3)), columns=['A','B','C'])
print("foo")
return(tabulate(df, headers="keys", tablefmt="orgtbl"))
#+END_SRC

#+RESULTS:
|   |        A |        B |        C |
|---+----------+----------+----------|
| 0 | 0.754799 | 0.492722 | 0.144595 |
| 1 | 0.198475 | 0.417721 | 0.083459 |
| 2 | 0.645011 | 0.444857 | 0.278874 |
| 3 | 0.314883 |   0.7521 | 0.789418 |
------------8<------------8<------------8<------------8<------------8<------------8<------------8<------------

However, this is not optimal, mainly because it pollutes the code block.
Note, that the :results is set to `raw`.

A brief discussion with Dov, yielded the following better workaround:
------------8<------------8<------------8<------------8<----
--------8<------------8<------------8<------------
(setq org-babel-python-wrapper-method "
def main():
%s

res = main()
if 'pandas.core.frame.DataFrame' in str(type(res)):
    from tabulate import tabulate
    out = tabulate(res, headers='keys', tablefmt='orgtbl')
else:
    out = str(res)

open('%s', 'w').write(out)")
------------8<------------8<------------8<------------8<----
--------8<------------8<------------8<------------

This allows a nice output of pandas.DataFrame (again when using :results
raw). Unfortunately, this wrapper has no influence in the case when
:session is used. To that end, it is possible to hack

------------8<------------8<------------8<------------8<----
--------8<------------8<------------8<------------
(defun org-babel-python-evaluate-session
    (session body &optional result-type result-params)
  "Pass BODY to the Python process in SESSION.
If RESULT-TYPE equals 'output then return standard output as a
string.  If RESULT-TYPE equals 'value then return the value of the
last statement in BODY, as elisp."
  (let* ((send-wait (lambda () (comint-send-input nil t) (sleep-for 0 5)))
 (dump-last-value
  (lambda
    (tmp-file pp)
    (mapc
     (lambda (statement) (insert statement) (funcall send-wait))
     (if pp
 (list
  "import pprint"
  (format "open('%s', 'w').write(pprint.pformat(_))"
  (org-babel-process-file-name tmp-file 'noquote)))
       (list (format "_org_tmp = _;

if 'pandas.core.frame.DataFrame' in str(type(_org_tmp)):
    from tabulate import tabulate
    _org_out = tabulate(_org_tmp, headers='keys', tablefmt='orgtbl')
else:
    _org_out = _org_tmp

open('%s', 'w').write(str(_org_out))"
     (org-babel-process-file-name tmp-file
                                                          'noquote)))))))
 (input-body (lambda (body)
       (mapc (lambda (line) (insert line) (funcall send-wait))
     (split-string body "[\r\n]"))
       (funcall send-wait)))
         (results
          (case result-type
            (output
             (mapconcat
              #'org-babel-trim
              (butlast
               (org-babel-comint-with-output
                   (session org-babel-python-eoe-indicator t body)
                 (funcall input-body body)
                 (funcall send-wait) (funcall send-wait)
                 (insert org-babel-python-eoe-indicator)
                 (funcall send-wait))
               2) "\n"))
            (value
             (let ((tmp-file (org-babel-temp-file "python-")))
               (org-babel-comint-with-output
                   (session org-babel-python-eoe-indicator nil body)
                 (let ((comint-process-echoes nil))
                   (funcall input-body body)
                   (funcall dump-last-value tmp-file
                            (member "pp" result-params))
                   (funcall send-wait) (funcall send-wait)
                   (insert org-babel-python-eoe-indicator)
                   (funcall send-wait)))
               (org-babel-eval-read-file tmp-file))))))
    (unless (string= (substring org-babel-python-eoe-indicator 1 -1)
results)
      (org-babel-result-cond result-params
results
        (org-babel-python-table-or-string results)))))
------------8<------------8<------------8<------------8<----
--------8<------------8<------------8<------------

This works, but I would be surprised if this hack meets org-mode's
standards. Nevertheless, maybe someone would find it useful.

What do you think? How can it be improved?

Best,
Dror

[-- Attachment #2: Type: text/html, Size: 12485 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2015-04-29  7:12 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-06-28  5:26 org-mode and python pandas Dov Grobgeld
2013-06-30 23:15 ` Eric Schulte
2013-07-01 16:34   ` Achim Gratz
2013-07-01 17:04     ` Rasmus
2013-07-03  9:15       ` Dov Grobgeld
2013-07-03 10:31         ` Rasmus
2013-07-03 14:09         ` Eric Schulte
2015-04-28  8:36           ` Dov Grobgeld
  -- strict thread matches above, loose matches on Subject: below --
2015-04-29  7:12 Dror Atariah

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).