emacs-orgmode@gnu.org archives
 help / color / mirror / code / Atom feed

* Re: org-mode and python pandas
@ 2015-04-29  7:12 Dror Atariah
  0 siblings, 0 replies; 9+ messages in thread
From: Dror Atariah @ 2015-04-29  7:12 UTC (permalink / raw)
  To: emacs-orgmode@gnu.org

[-- Attachment #1: Type: text/plain, Size: 4554 bytes --]

Using the tabulate python module it is possible to have the following
"inline" workaround:

------------8<------------8<------------8<------------8<------------8<------------8<------------8<------------
#+BEGIN_SRC python :results raw
import pandas as pd
import numpy as np
from tabulate import tabulate

df = pd.DataFrame(np.random.random((4,3)), columns=['A','B','C'])
print("foo")
return(tabulate(df, headers="keys", tablefmt="orgtbl"))
#+END_SRC

#+RESULTS:
|   |        A |        B |        C |
|---+----------+----------+----------|
| 0 | 0.754799 | 0.492722 | 0.144595 |
| 1 | 0.198475 | 0.417721 | 0.083459 |
| 2 | 0.645011 | 0.444857 | 0.278874 |
| 3 | 0.314883 |   0.7521 | 0.789418 |
------------8<------------8<------------8<------------8<------------8<------------8<------------8<------------

However, this is not optimal, mainly because it pollutes the code block.
Note, that the :results is set to `raw`.

A brief discussion with Dov, yielded the following better workaround:
------------8<------------8<------------8<------------8<----
--------8<------------8<------------8<------------
(setq org-babel-python-wrapper-method "
def main():
%s

res = main()
if 'pandas.core.frame.DataFrame' in str(type(res)):
    from tabulate import tabulate
    out = tabulate(res, headers='keys', tablefmt='orgtbl')
else:
    out = str(res)

open('%s', 'w').write(out)")
------------8<------------8<------------8<------------8<----
--------8<------------8<------------8<------------

This allows a nice output of pandas.DataFrame (again when using :results
raw). Unfortunately, this wrapper has no influence in the case when
:session is used. To that end, it is possible to hack

------------8<------------8<------------8<------------8<----
--------8<------------8<------------8<------------
(defun org-babel-python-evaluate-session
    (session body &optional result-type result-params)
  "Pass BODY to the Python process in SESSION.
If RESULT-TYPE equals 'output then return standard output as a
string.  If RESULT-TYPE equals 'value then return the value of the
last statement in BODY, as elisp."
  (let* ((send-wait (lambda () (comint-send-input nil t) (sleep-for 0 5)))
 (dump-last-value
  (lambda
    (tmp-file pp)
    (mapc
     (lambda (statement) (insert statement) (funcall send-wait))
     (if pp
 (list
  "import pprint"
  (format "open('%s', 'w').write(pprint.pformat(_))"
  (org-babel-process-file-name tmp-file 'noquote)))
       (list (format "_org_tmp = _;

if 'pandas.core.frame.DataFrame' in str(type(_org_tmp)):
    from tabulate import tabulate
    _org_out = tabulate(_org_tmp, headers='keys', tablefmt='orgtbl')
else:
    _org_out = _org_tmp

open('%s', 'w').write(str(_org_out))"
     (org-babel-process-file-name tmp-file
                                                          'noquote)))))))
 (input-body (lambda (body)
       (mapc (lambda (line) (insert line) (funcall send-wait))
     (split-string body "[\r\n]"))
       (funcall send-wait)))
         (results
          (case result-type
            (output
             (mapconcat
              #'org-babel-trim
              (butlast
               (org-babel-comint-with-output
                   (session org-babel-python-eoe-indicator t body)
                 (funcall input-body body)
                 (funcall send-wait) (funcall send-wait)
                 (insert org-babel-python-eoe-indicator)
                 (funcall send-wait))
               2) "\n"))
            (value
             (let ((tmp-file (org-babel-temp-file "python-")))
               (org-babel-comint-with-output
                   (session org-babel-python-eoe-indicator nil body)
                 (let ((comint-process-echoes nil))
                   (funcall input-body body)
                   (funcall dump-last-value tmp-file
                            (member "pp" result-params))
                   (funcall send-wait) (funcall send-wait)
                   (insert org-babel-python-eoe-indicator)
                   (funcall send-wait)))
               (org-babel-eval-read-file tmp-file))))))
    (unless (string= (substring org-babel-python-eoe-indicator 1 -1)
results)
      (org-babel-result-cond result-params
results
        (org-babel-python-table-or-string results)))))
------------8<------------8<------------8<------------8<----
--------8<------------8<------------8<------------

This works, but I would be surprised if this hack meets org-mode's
standards. Nevertheless, maybe someone would find it useful.

What do you think? How can it be improved?

Best,
Dror

[-- Attachment #2: Type: text/html, Size: 12485 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

* org-mode and python pandas
@ 2013-06-28  5:26 Dov Grobgeld
  2013-06-30 23:15 ` Eric Schulte
  0 siblings, 1 reply; 9+ messages in thread
From: Dov Grobgeld @ 2013-06-28  5:26 UTC (permalink / raw)
  To: emacs-orgmode

Has anyone used org-mode with the python pandas package? Pandas is in
a certain way an alternative to R, but with the (for me) familiar
syntax of python. See: http://pandas.pydata.org/

Pandas is very much built to be used interactively, and it outputs its
data in space separated tabular format. E.g. in ipython:

In [1]: import pandas as pd
In [2]: import numpy as np

In [3]: pd.DataFrame(np.random.random((4,3)), columns=['A','B','C'])
Out[3]:
          A         B         C
0  0.628365  0.424279  0.619791
1  0.799666  0.527572  0.132928
2  0.837255  0.138906  0.408233
3  0.388080  0.146212  0.575346

Unfortunately this doesn't output as nicely when used from org-mode:

#+BEGIN_SRC python
import pandas as pd
import numpy as np

return pd.DataFrame(np.random.random((4,3)), columns=list('ABC'))
#+END_SRC

#+RESULTS:
: A         B         C
: 0  0.827817  0.664009  0.089161
: 1  0.170031  0.729214  0.110918
: 2  0.575918  0.863924  0.757536
: 3  0.682722  0.774445  0.992041

while I would like to have:

|   |        A |        B |        C |
|---+----------+----------+----------|
| 0 | 0.827817 | 0.664009 | 0.089161 |
| 1 | 0.170031 | 0.729214 | 0.110918 |
| 2 | 0.575918 | 0.863924 | 0.757536 |
| 3 | 0.682722 | 0.774445 | 0.992041 |

The question is how to get this? Here are a few ideas:

1. Write a general filter in the org-mode elisp than uses heuristics
to recognize ascii aligned tables and change these to org-tables.
2. Add to pandas the option of globally influencing the text
formatting so that it outputs something more parsable by org-mode.
3. Create a special language "pandas" that recognize the ascii aligned
tables and saves the need to import pandas and np?
4. And the obvious approach of writing a python function that writes a
org-mode parsable table and always call it as part of the return.

Which is the preferable approach? Any other ideas?

Regards,
Dov

^ permalink raw reply	[flat|nested] 9+ messages in thread

• * Re: org-mode and python pandas
    2013-06-28  5:26 Dov Grobgeld
  @ 2013-06-30 23:15 ` Eric Schulte
    2013-07-01 16:34   ` Achim Gratz
    0 siblings, 1 reply; 9+ messages in thread
  From: Eric Schulte @ 2013-06-30 23:15 UTC (permalink / raw)
    To: Dov Grobgeld; +Cc: emacs-orgmode
  
  Dov Grobgeld <dov.grobgeld@gmail.com> writes:
  
  > Has anyone used org-mode with the python pandas package? Pandas is in
  > a certain way an alternative to R, but with the (for me) familiar
  > syntax of python. See: http://pandas.pydata.org/
  >
  > Pandas is very much built to be used interactively, and it outputs its
  > data in space separated tabular format. E.g. in ipython:
  >
  > In [1]: import pandas as pd
  > In [2]: import numpy as np
  >
  > In [3]: pd.DataFrame(np.random.random((4,3)), columns=['A','B','C'])
  > Out[3]:
  >           A         B         C
  > 0  0.628365  0.424279  0.619791
  > 1  0.799666  0.527572  0.132928
  > 2  0.837255  0.138906  0.408233
  > 3  0.388080  0.146212  0.575346
  >
  > Unfortunately this doesn't output as nicely when used from org-mode:
  >
  > #+BEGIN_SRC python
  > import pandas as pd
  > import numpy as np
  >
  > return pd.DataFrame(np.random.random((4,3)), columns=list('ABC'))
  > #+END_SRC
  >
  > #+RESULTS:
  > : A         B         C
  > : 0  0.827817  0.664009  0.089161
  > : 1  0.170031  0.729214  0.110918
  > : 2  0.575918  0.863924  0.757536
  > : 3  0.682722  0.774445  0.992041
  >
  > while I would like to have:
  >
  > |   |        A |        B |        C |
  > |---+----------+----------+----------|
  > | 0 | 0.827817 | 0.664009 | 0.089161 |
  > | 1 | 0.170031 | 0.729214 | 0.110918 |
  > | 2 | 0.575918 | 0.863924 | 0.757536 |
  > | 3 | 0.682722 | 0.774445 | 0.992041 |
  >
  
  What happens if you add ":results table" to your code block?  Would that
  be sufficient?
  
  >
  > The question is how to get this? Here are a few ideas:
  >
  > 1. Write a general filter in the org-mode elisp than uses heuristics
  > to recognize ascii aligned tables and change these to org-tables.
  
  The default value should be to convert multi-line output to tables, the
  ":results table" option above will force this conversion in case it is
  currently not taking place due to the default header arguments in use.
  
  > 
  > 2. Add to pandas the option of globally influencing the text
  > formatting so that it outputs something more parsable by org-mode.
  
  This sounds promising, if pandas support csv output that will be
  correctly parsed by Org-mode.
  
  > 
  > 3. Create a special language "pandas" that recognize the ascii aligned
  > tables and saves the need to import pandas and np?  4. And the obvious
  > approach of writing a python function that writes a org-mode parsable
  > table and always call it as part of the return.
  >
  > Which is the preferable approach? Any other ideas?
  >
  
  I think a header-argument-based approach would be ideal, I'd look at the
  value of org-babel-default-header-args:python, and read the portion of
  the manual related to the "results" header arguments.
  
  I don't understand multi-line strings in python, but I get the following
  behavior from simple shell script blocks.
  
  #+begin_src sh
    cat <<EOF
              A         B         C
    0  0.628365  0.424279  0.619791
    1  0.799666  0.527572  0.132928
    2  0.837255  0.138906  0.408233
    3  0.388080  0.146212  0.575346
    EOF
  #+end_src
  
  #+RESULTS:
  | A |        B |        C |          |
  | 0 | 0.628365 | 0.424279 | 0.619791 |
  | 1 | 0.799666 | 0.527572 | 0.132928 |
  | 2 | 0.837255 | 0.138906 | 0.408233 |
  | 3 |  0.38808 | 0.146212 | 0.575346 |
  
  #+begin_src sh
    cat <<EOF
    ,A,B,C
    0,0.628365,0.424279,0.619791
    1,0.799666,0.527572,0.132928
    2,0.837255,0.138906,0.408233
    3,0.388080,0.146212,0.575346
    EOF
  #+end_src
  
  #+RESULTS:
  |   |        A |        B |        C |
  | 0 | 0.628365 | 0.424279 | 0.619791 |
  | 1 | 0.799666 | 0.527572 | 0.132928 |
  | 2 | 0.837255 | 0.138906 | 0.408233 |
  | 3 |  0.38808 | 0.146212 | 0.575346 |
  
  Hope this helps,
  
  >
  > Regards,
  > Dov
  >
  
  -- 
  Eric Schulte
  http://cs.unm.edu/~eschulte
  
  ^ permalink raw reply	[flat|nested] 9+ messages in thread

• • * Re: org-mode and python pandas
      2013-06-30 23:15 ` Eric Schulte
    @ 2013-07-01 16:34   ` Achim Gratz
      2013-07-01 17:04     ` Rasmus
      0 siblings, 1 reply; 9+ messages in thread
    From: Achim Gratz @ 2013-07-01 16:34 UTC (permalink / raw)
      To: emacs-orgmode
    
    Eric Schulte writes:
    >> |   |        A |        B |        C |
    >> |---+----------+----------+----------|
    >> | 0 | 0.827817 | 0.664009 | 0.089161 |
    >> | 1 | 0.170031 | 0.729214 | 0.110918 |
    >> | 2 | 0.575918 | 0.863924 | 0.757536 |
    >> | 3 | 0.682722 | 0.774445 | 0.992041 |
    >>
    >
    > What happens if you add ":results table" to your code block?  Would that
    > be sufficient?
    
    The problem is that he's trying to consume human-readable output.  The
    alignment in this output is done with spaces, not tabs (as Org would
    expect).  There are more problems considering that the numbers likely
    lose precision in this format (nota bene: there are other Babel
    languages that have that same problem).
    
    > The default value should be to convert multi-line output to tables, the
    > ":results table" option above will force this conversion in case it is
    > currently not taking place due to the default header arguments in use.
    
    Since there are no tabs in there, it is impossible to skip empty cells
    like the first one (they'd have to be written as "" instead, as would be
    cells that should contain literal spaces).
    
    >> 2. Add to pandas the option of globally influencing the text
    >> formatting so that it outputs something more parsable by org-mode.
    >
    > This sounds promising, if pandas support csv output that will be
    > correctly parsed by Org-mode.
    
    The package already has CSV export, so one could use that.  I don't know
    if you could echo the result directly to the output, all examples
    revolve around putting the CSV into a file.  For Org, TSV output would
    be more natural.
    
    
    Regards,
    Achim.
    -- 
    +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+
    
    Waldorf MIDI Implementation & additional documentation:
    http://Synth.Stromeko.net/Downloads.html#WaldorfDocs
    
    ^ permalink raw reply	[flat|nested] 9+ messages in thread

  • • * Re: org-mode and python pandas
        2013-07-01 16:34   ` Achim Gratz
      @ 2013-07-01 17:04     ` Rasmus
        2013-07-03  9:15       ` Dov Grobgeld
        0 siblings, 1 reply; 9+ messages in thread
      From: Rasmus @ 2013-07-01 17:04 UTC (permalink / raw)
        To: emacs-orgmode
      
      Achim Gratz <Stromeko@nexgo.de> writes:
      
      >>> 2. Add to pandas the option of globally influencing the text
      >>> formatting so that it outputs something more parsable by org-mode.
      >>
      >> This sounds promising, if pandas support csv output that will be
      >> correctly parsed by Org-mode.
      >
      > The package already has CSV export, so one could use that.  I don't know
      > if you could echo the result directly to the output, all examples
      > revolve around putting the CSV into a file.  For Org, TSV output would
      > be more natural.
      
      Something like:
      
      from pandas import DataFrame
      from numpy.random import rand
      from sys import stdout
      df = DataFrame(rand(10,3), columns = list('abc'))
      df
      df.to_csv(stdout, sep="\t", header = True, cols=(1,2))
      
      I was completely unable to get ob-python working this morning, so I
      haven't tested it.  I'm using python3, build in python mode and elpy.
      
      In any case, the csv route might be better, as Pandas doesn't print
      the table if it's too big (try changing 10 to 1000 above).
      
      -- 
      Powered by magic pixies!
      
      ^ permalink raw reply	[flat|nested] 9+ messages in thread

    • • * Re: org-mode and python pandas
          2013-07-01 17:04     ` Rasmus
        @ 2013-07-03  9:15       ` Dov Grobgeld
          2013-07-03 10:31         ` Rasmus
          2013-07-03 14:09         ` Eric Schulte
          0 siblings, 2 replies; 9+ messages in thread
        From: Dov Grobgeld @ 2013-07-03  9:15 UTC (permalink / raw)
          To: Rasmus; +Cc: emacs-orgmode
        
        Thanks for the answers, but there is still something missing in order
        to get it to work. Part of it seems to be connected to the python
        parsing. E.g. the following translation of Eric's sh example doesn't
        output correctly with python:
        
        
        #+BEGIN_SRC python :results output
        print """,A,B,C
        0,0.628365,0.424279,0.619791
        1,0.799666,0.527572,0.132928
        2,0.837255,0.138906,0.408233
        3,0.388080,0.146212,0.575346
        """
        #+END_SRC
        
        #+RESULTS:
        : ,A,B,C
        : 0,0.628365,0.424279,0.619791
        : 1,0.799666,0.527572,0.132928
        : 2,0.837255,0.138906,0.408233
        : 3,0.388080,0.146212,0.575346
        
        
        #+BEGIN_SRC python :results table
        return """,A,B,C
        0,0.628365,0.424279,0.619791
        1,0.799666,0.527572,0.132928
        2,0.837255,0.138906,0.408233
        3,0.388080,0.146212,0.575346
        """
        #+END_SRC
        
        #+RESULTS:
        | ,A,B,C\n\n0,0.628365,0.424279,0.619791\n\n1,0.799666,0.527572,0.132928\n\n2,0.837255,0.138906,0.408233\n\n3,0.388080,0.146212,0.575346
        |
        
        It seems that the only way to get a table from python is by outputting
        a two dimensional python structure:
        
        #+BEGIN_SRC python
        return [[0,0.628365,0.424279,0.619791],
                [1,0.799666,0.527572,0.132928]]
        #+END_SRC
        
        #+RESULTS:
        | 0 | 0.628365 | 0.424279 | 0.619791 |
        | 1 | 0.799666 | 0.527572 | 0.132928 |
        
        This seems quite limiting....
        
        Another related question is if there is any support for header tables?
        I.e. instead of this:
        
        |   |        A |        B |        C |
        | 0 | 0.827817 | 0.664009 | 0.089161 |
        | 1 | 0.170031 | 0.729214 | 0.110918 |
        | 2 | 0.575918 | 0.863924 | 0.757536 |
        | 3 | 0.682722 | 0.774445 | 0.992041 |
        
        I want this:
        
        |   |        A |        B |        C |
        |---+----------+----------+----------|
        | 0 | 0.827817 | 0.664009 | 0.089161 |
        | 1 | 0.170031 | 0.729214 | 0.110918 |
        | 2 | 0.575918 | 0.863924 | 0.757536 |
        | 3 | 0.682722 | 0.774445 | 0.992041 |
        
        I guess that if I start playing around with the python ob module, it
        should be possible to get this working?
        
        Regards,
        Dov
        
        On Mon, Jul 1, 2013 at 8:04 PM, Rasmus <rasmus@gmx.us> wrote:
        > Achim Gratz <Stromeko@nexgo.de> writes:
        >
        >>>> 2. Add to pandas the option of globally influencing the text
        >>>> formatting so that it outputs something more parsable by org-mode.
        >>>
        >>> This sounds promising, if pandas support csv output that will be
        >>> correctly parsed by Org-mode.
        >>
        >> The package already has CSV export, so one could use that.  I don't know
        >> if you could echo the result directly to the output, all examples
        >> revolve around putting the CSV into a file.  For Org, TSV output would
        >> be more natural.
        >
        > Something like:
        >
        > from pandas import DataFrame
        > from numpy.random import rand
        > from sys import stdout
        > df = DataFrame(rand(10,3), columns = list('abc'))
        > df
        > df.to_csv(stdout, sep="\t", header = True, cols=(1,2))
        >
        > I was completely unable to get ob-python working this morning, so I
        > haven't tested it.  I'm using python3, build in python mode and elpy.
        >
        > In any case, the csv route might be better, as Pandas doesn't print
        > the table if it's too big (try changing 10 to 1000 above).
        >
        > --
        > Powered by magic pixies!
        >
        >
        
        ^ permalink raw reply	[flat|nested] 9+ messages in thread

      • • * Re: org-mode and python pandas
            2013-07-03  9:15       ` Dov Grobgeld
          @ 2013-07-03 10:31         ` Rasmus
            2013-07-03 14:09         ` Eric Schulte
            1 sibling, 0 replies; 9+ messages in thread
          From: Rasmus @ 2013-07-03 10:31 UTC (permalink / raw)
            To: emacs-orgmode
          
          Hi Dov,
          
          > Another related question is if there is any support for header tables?
          > I.e. instead of this:
          >
          > |   |        A |        B |        C |
          > | 0 | 0.827817 | 0.664009 | 0.089161 |
          > | 1 | 0.170031 | 0.729214 | 0.110918 |
          > | 2 | 0.575918 | 0.863924 | 0.757536 |
          > | 3 | 0.682722 | 0.774445 | 0.992041 |
          >
          > I want this:
          >
          > |   |        A |        B |        C |
          > |---+----------+----------+----------|
          > | 0 | 0.827817 | 0.664009 | 0.089161 |
          > | 1 | 0.170031 | 0.729214 | 0.110918 |
          > | 2 | 0.575918 | 0.863924 | 0.757536 |
          > | 3 | 0.682722 | 0.774445 | 0.992041 |
          
          This is an issue in R as well, e.g. often I might want to have 
          |A|
          |-|
          |1|
          |2|
          |-|
          |3|
          
          where the last is a summary statistic.  The last hline can be gotten
          as 
          
          #+BEGIN_SRC emacs-lisp :var X=mytabel :exports results
              (let
                  ((L (- (length X) 1)))
                (append
                 (subseq X 0 L)
                 (list 'hline)
                 (subseq X L)))
          #+END_SRC
          
          Perhaps a better way exists?
          
          –Rasmus
          
          -- 
          . . . It begins of course with The Internet.  A Net of Peers.
          
          ^ permalink raw reply	[flat|nested] 9+ messages in thread

        • * Re: org-mode and python pandas
            2013-07-03  9:15       ` Dov Grobgeld
            2013-07-03 10:31         ` Rasmus
          @ 2013-07-03 14:09         ` Eric Schulte
            2015-04-28  8:36           ` Dov Grobgeld
            1 sibling, 1 reply; 9+ messages in thread
          From: Eric Schulte @ 2013-07-03 14:09 UTC (permalink / raw)
            To: Dov Grobgeld; +Cc: emacs-orgmode, Rasmus
          
          Dov Grobgeld <dov.grobgeld@gmail.com> writes:
          
          > Thanks for the answers, but there is still something missing in order
          > to get it to work. Part of it seems to be connected to the python
          > parsing. E.g. the following translation of Eric's sh example doesn't
          > output correctly with python:
          >
          >
          > #+BEGIN_SRC python :results output
          > print """,A,B,C
          > 0,0.628365,0.424279,0.619791
          > 1,0.799666,0.527572,0.132928
          > 2,0.837255,0.138906,0.408233
          > 3,0.388080,0.146212,0.575346
          > """
          > #+END_SRC
          >
          > #+RESULTS:
          > : ,A,B,C
          > : 0,0.628365,0.424279,0.619791
          > : 1,0.799666,0.527572,0.132928
          > : 2,0.837255,0.138906,0.408233
          > : 3,0.388080,0.146212,0.575346
          >
          >
          > #+BEGIN_SRC python :results table
          > return """,A,B,C
          > 0,0.628365,0.424279,0.619791
          > 1,0.799666,0.527572,0.132928
          > 2,0.837255,0.138906,0.408233
          > 3,0.388080,0.146212,0.575346
          > """
          > #+END_SRC
          >
          > #+RESULTS:
          > | ,A,B,C\n\n0,0.628365,0.424279,0.619791\n\n1,0.799666,0.527572,0.132928\n\n2,0.837255,0.138906,0.408233\n\n3,0.388080,0.146212,0.575346
          > |
          >
          > It seems that the only way to get a table from python is by outputting
          > a two dimensional python structure:
          >
          > #+BEGIN_SRC python
          > return [[0,0.628365,0.424279,0.619791],
          >         [1,0.799666,0.527572,0.132928]]
          > #+END_SRC
          >
          > #+RESULTS:
          > | 0 | 0.628365 | 0.424279 | 0.619791 |
          > | 1 | 0.799666 | 0.527572 | 0.132928 |
          >
          > This seems quite limiting....
          >
          
          In most cases this is what one wants when returning data from python
          code.  The following elisp defined a "panda" code block, which is just
          like python, only it assumes that the results will be these sort of
          human readable strings instead of python code.
          
              ;; -*- emacs-lisp -*-
              (defun org-babel-execute:panda (body params)
                (let ((results
                       (org-babel-execute:python
                        body (org-babel-merge-params '((:results . "scalar")) params))))
                  (org-babel-result-cond (cdr (assoc :result-params params))
                    results
                    (let ((tmp-file (org-babel-temp-file "sh-")))
                      (with-temp-file tmp-file (insert results))
                      (org-babel-import-elisp-from-file tmp-file)))))
          
          With the above evaluated the following works
          
              #+BEGIN_SRC panda
              return """,A,B,C
              0,0.628365,0.424279,0.619791
              1,0.799666,0.527572,0.132928
              2,0.837255,0.138906,0.408233
              3,0.388080,0.146212,0.575346
              """
              #+END_SRC
          
              #+RESULTS:
              |   |        A |        B |        C |
              | 0 | 0.628365 | 0.424279 | 0.619791 |
              | 1 | 0.799666 | 0.527572 | 0.132928 |
              | 2 | 0.837255 | 0.138906 | 0.408233 |
              | 3 |  0.38808 | 0.146212 | 0.575346 |
          
          >
          > Another related question is if there is any support for header tables?
          > I.e. instead of this:
          >
          > |   |        A |        B |        C |
          > | 0 | 0.827817 | 0.664009 | 0.089161 |
          > | 1 | 0.170031 | 0.729214 | 0.110918 |
          > | 2 | 0.575918 | 0.863924 | 0.757536 |
          > | 3 | 0.682722 | 0.774445 | 0.992041 |
          >
          > I want this:
          >
          > |   |        A |        B |        C |
          > |---+----------+----------+----------|
          > | 0 | 0.827817 | 0.664009 | 0.089161 |
          > | 1 | 0.170031 | 0.729214 | 0.110918 |
          > | 2 | 0.575918 | 0.863924 | 0.757536 |
          > | 3 | 0.682722 | 0.774445 | 0.992041 |
          >
          > I guess that if I start playing around with the python ob module, it
          > should be possible to get this working?
          >
          
          See the :colnames header argument in the manual.
          
          Best,
          
          >
          > Regards,
          > Dov
          >
          > On Mon, Jul 1, 2013 at 8:04 PM, Rasmus <rasmus@gmx.us> wrote:
          >> Achim Gratz <Stromeko@nexgo.de> writes:
          >>
          >>>>> 2. Add to pandas the option of globally influencing the text
          >>>>> formatting so that it outputs something more parsable by org-mode.
          >>>>
          >>>> This sounds promising, if pandas support csv output that will be
          >>>> correctly parsed by Org-mode.
          >>>
          >>> The package already has CSV export, so one could use that.  I don't know
          >>> if you could echo the result directly to the output, all examples
          >>> revolve around putting the CSV into a file.  For Org, TSV output would
          >>> be more natural.
          >>
          >> Something like:
          >>
          >> from pandas import DataFrame
          >> from numpy.random import rand
          >> from sys import stdout
          >> df = DataFrame(rand(10,3), columns = list('abc'))
          >> df
          >> df.to_csv(stdout, sep="\t", header = True, cols=(1,2))
          >>
          >> I was completely unable to get ob-python working this morning, so I
          >> haven't tested it.  I'm using python3, build in python mode and elpy.
          >>
          >> In any case, the csv route might be better, as Pandas doesn't print
          >> the table if it's too big (try changing 10 to 1000 above).
          >>
          >> --
          >> Powered by magic pixies!
          >>
          >>
          >
          
          -- 
          Eric Schulte
          http://cs.unm.edu/~eschulte
          
          ^ permalink raw reply	[flat|nested] 9+ messages in thread

        • • * Re: org-mode and python pandas
              2013-07-03 14:09         ` Eric Schulte
            @ 2015-04-28  8:36           ` Dov Grobgeld
              0 siblings, 0 replies; 9+ messages in thread
            From: Dov Grobgeld @ 2015-04-28  8:36 UTC (permalink / raw)
              To: Eric Schulte; +Cc: emacs-orgmode, Rasmus
            
            [-- Attachment #1: Type: text/plain, Size: 5615 bytes --]
            
            I returned to this issue recently and tried to get the ob-python to output
            a table with a header, but didn't manage. Here is the code:
            
            #+BEGIN_SRC python :colnames true
            return [['','A','B','C'],
                    [0,0.628365,0.424279,0.619791],
                    [1,0.799666,0.527572,0.132928]]
            #+END_SRC
            
            #+RESULTS:
            |   |        A |        B |        C |
            | 0 | 0.628365 | 0.424279 | 0.619791 |
            | 1 | 0.799666 | 0.527572 | 0.132928 |
            
            What I want is:
            
            |   |        A |        B |        C |
            |---+----------+----------+----------|
            | 0 | 0.628365 | 0.424279 | 0.619791 |
            | 1 | 0.799666 | 0.527572 | 0.132928 |
            
            Is there any way to do that besides using the :results raw option?
            
            Thanks!
            Dov
            
            
            On Wed, Jul 3, 2013 at 5:09 PM, Eric Schulte <schulte.eric@gmail.com> wrote:
            
            > Dov Grobgeld <dov.grobgeld@gmail.com> writes:
            >
            > > Thanks for the answers, but there is still something missing in order
            > > to get it to work. Part of it seems to be connected to the python
            > > parsing. E.g. the following translation of Eric's sh example doesn't
            > > output correctly with python:
            > >
            > >
            > > #+BEGIN_SRC python :results output
            > > print """,A,B,C
            > > 0,0.628365,0.424279,0.619791
            > > 1,0.799666,0.527572,0.132928
            > > 2,0.837255,0.138906,0.408233
            > > 3,0.388080,0.146212,0.575346
            > > """
            > > #+END_SRC
            > >
            > > #+RESULTS:
            > > : ,A,B,C
            > > : 0,0.628365,0.424279,0.619791
            > > : 1,0.799666,0.527572,0.132928
            > > : 2,0.837255,0.138906,0.408233
            > > : 3,0.388080,0.146212,0.575346
            > >
            > >
            > > #+BEGIN_SRC python :results table
            > > return """,A,B,C
            > > 0,0.628365,0.424279,0.619791
            > > 1,0.799666,0.527572,0.132928
            > > 2,0.837255,0.138906,0.408233
            > > 3,0.388080,0.146212,0.575346
            > > """
            > > #+END_SRC
            > >
            > > #+RESULTS:
            > > |
            > ,A,B,C\n\n0,0.628365,0.424279,0.619791\n\n1,0.799666,0.527572,0.132928\n\n2,0.837255,0.138906,0.408233\n\n3,0.388080,0.146212,0.575346
            > > |
            > >
            > > It seems that the only way to get a table from python is by outputting
            > > a two dimensional python structure:
            > >
            > > #+BEGIN_SRC python
            > > return [[0,0.628365,0.424279,0.619791],
            > >         [1,0.799666,0.527572,0.132928]]
            > > #+END_SRC
            > >
            > > #+RESULTS:
            > > | 0 | 0.628365 | 0.424279 | 0.619791 |
            > > | 1 | 0.799666 | 0.527572 | 0.132928 |
            > >
            > > This seems quite limiting....
            > >
            >
            > In most cases this is what one wants when returning data from python
            > code.  The following elisp defined a "panda" code block, which is just
            > like python, only it assumes that the results will be these sort of
            > human readable strings instead of python code.
            >
            >     ;; -*- emacs-lisp -*-
            >     (defun org-babel-execute:panda (body params)
            >       (let ((results
            >              (org-babel-execute:python
            >               body (org-babel-merge-params '((:results . "scalar"))
            > params))))
            >         (org-babel-result-cond (cdr (assoc :result-params params))
            >           results
            >           (let ((tmp-file (org-babel-temp-file "sh-")))
            >             (with-temp-file tmp-file (insert results))
            >             (org-babel-import-elisp-from-file tmp-file)))))
            >
            > With the above evaluated the following works
            >
            >     #+BEGIN_SRC panda
            >     return """,A,B,C
            >     0,0.628365,0.424279,0.619791
            >     1,0.799666,0.527572,0.132928
            >     2,0.837255,0.138906,0.408233
            >     3,0.388080,0.146212,0.575346
            >     """
            >     #+END_SRC
            >
            >     #+RESULTS:
            >     |   |        A |        B |        C |
            >     | 0 | 0.628365 | 0.424279 | 0.619791 |
            >     | 1 | 0.799666 | 0.527572 | 0.132928 |
            >     | 2 | 0.837255 | 0.138906 | 0.408233 |
            >     | 3 |  0.38808 | 0.146212 | 0.575346 |
            >
            > >
            > > Another related question is if there is any support for header tables?
            > > I.e. instead of this:
            > >
            > > |   |        A |        B |        C |
            > > | 0 | 0.827817 | 0.664009 | 0.089161 |
            > > | 1 | 0.170031 | 0.729214 | 0.110918 |
            > > | 2 | 0.575918 | 0.863924 | 0.757536 |
            > > | 3 | 0.682722 | 0.774445 | 0.992041 |
            > >
            > > I want this:
            > >
            > > |   |        A |        B |        C |
            > > |---+----------+----------+----------|
            > > | 0 | 0.827817 | 0.664009 | 0.089161 |
            > > | 1 | 0.170031 | 0.729214 | 0.110918 |
            > > | 2 | 0.575918 | 0.863924 | 0.757536 |
            > > | 3 | 0.682722 | 0.774445 | 0.992041 |
            > >
            > > I guess that if I start playing around with the python ob module, it
            > > should be possible to get this working?
            > >
            >
            > See the :colnames header argument in the manual.
            >
            > Best,
            >
            > >
            > > Regards,
            > > Dov
            > >
            > > On Mon, Jul 1, 2013 at 8:04 PM, Rasmus <rasmus@gmx.us> wrote:
            > >> Achim Gratz <Stromeko@nexgo.de> writes:
            > >>
            > >>>>> 2. Add to pandas the option of globally influencing the text
            > >>>>> formatting so that it outputs something more parsable by org-mode.
            > >>>>
            > >>>> This sounds promising, if pandas support csv output that will be
            > >>>> correctly parsed by Org-mode.
            > >>>
            > >>> The package already has CSV export, so one could use that.  I don't
            > know
            > >>> if you could echo the result directly to the output, all examples
            > >>> revolve around putting the CSV into a file.  For Org, TSV output would
            > >>> be more natural.
            > >>
            > >> Something like:
            > >>
            > >> from pandas import DataFrame
            > >> from numpy.random import rand
            > >> from sys import stdout
            > >> df = DataFrame(rand(10,3), columns = list('abc'))
            > >> df
            > >> df.to_csv(stdout, sep="\t", header = True, cols=(1,2))
            > >>
            > >> I was completely unable to get ob-python working this morning, so I
            > >> haven't tested it.  I'm using python3, build in python mode and elpy.
            > >>
            > >> In any case, the csv route might be better, as Pandas doesn't print
            > >> the table if it's too big (try changing 10 to 1000 above).
            > >>
            > >> --
            > >> Powered by magic pixies!
            > >>
            > >>
            > >
            >
            > --
            > Eric Schulte
            > http://cs.unm.edu/~eschulte
            >
            
            [-- Attachment #2: Type: text/html, Size: 8155 bytes --]
            
            ^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2015-04-29  7:12 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-04-29  7:12 org-mode and python pandas Dror Atariah
  -- strict thread matches above, loose matches on Subject: below --
2013-06-28  5:26 Dov Grobgeld
2013-06-30 23:15 ` Eric Schulte
2013-07-01 16:34   ` Achim Gratz
2013-07-01 17:04     ` Rasmus
2013-07-03  9:15       ` Dov Grobgeld
2013-07-03 10:31         ` Rasmus
2013-07-03 14:09         ` Eric Schulte
2015-04-28  8:36           ` Dov Grobgeld

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).