emacs-orgmode@gnu.org archives
 help / color / mirror / code / Atom feed
* org-mode and python pandas
@ 2013-06-28  5:26 Dov Grobgeld
  2013-06-30 23:15 ` Eric Schulte
  0 siblings, 1 reply; 9+ messages in thread
From: Dov Grobgeld @ 2013-06-28  5:26 UTC (permalink / raw)
  To: emacs-orgmode

Has anyone used org-mode with the python pandas package? Pandas is in
a certain way an alternative to R, but with the (for me) familiar
syntax of python. See: http://pandas.pydata.org/

Pandas is very much built to be used interactively, and it outputs its
data in space separated tabular format. E.g. in ipython:

In [1]: import pandas as pd
In [2]: import numpy as np

In [3]: pd.DataFrame(np.random.random((4,3)), columns=['A','B','C'])
Out[3]:
          A         B         C
0  0.628365  0.424279  0.619791
1  0.799666  0.527572  0.132928
2  0.837255  0.138906  0.408233
3  0.388080  0.146212  0.575346

Unfortunately this doesn't output as nicely when used from org-mode:

#+BEGIN_SRC python
import pandas as pd
import numpy as np

return pd.DataFrame(np.random.random((4,3)), columns=list('ABC'))
#+END_SRC

#+RESULTS:
: A         B         C
: 0  0.827817  0.664009  0.089161
: 1  0.170031  0.729214  0.110918
: 2  0.575918  0.863924  0.757536
: 3  0.682722  0.774445  0.992041

while I would like to have:

|   |        A |        B |        C |
|---+----------+----------+----------|
| 0 | 0.827817 | 0.664009 | 0.089161 |
| 1 | 0.170031 | 0.729214 | 0.110918 |
| 2 | 0.575918 | 0.863924 | 0.757536 |
| 3 | 0.682722 | 0.774445 | 0.992041 |

The question is how to get this? Here are a few ideas:

1. Write a general filter in the org-mode elisp than uses heuristics
to recognize ascii aligned tables and change these to org-tables.
2. Add to pandas the option of globally influencing the text
formatting so that it outputs something more parsable by org-mode.
3. Create a special language "pandas" that recognize the ascii aligned
tables and saves the need to import pandas and np?
4. And the obvious approach of writing a python function that writes a
org-mode parsable table and always call it as part of the return.

Which is the preferable approach? Any other ideas?

Regards,
Dov

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: org-mode and python pandas
@ 2015-04-29  7:12 Dror Atariah
  0 siblings, 0 replies; 9+ messages in thread
From: Dror Atariah @ 2015-04-29  7:12 UTC (permalink / raw)
  To: emacs-orgmode@gnu.org

[-- Attachment #1: Type: text/plain, Size: 4554 bytes --]

Using the tabulate python module it is possible to have the following
"inline" workaround:

------------8<------------8<------------8<------------8<------------8<------------8<------------8<------------
#+BEGIN_SRC python :results raw
import pandas as pd
import numpy as np
from tabulate import tabulate

df = pd.DataFrame(np.random.random((4,3)), columns=['A','B','C'])
print("foo")
return(tabulate(df, headers="keys", tablefmt="orgtbl"))
#+END_SRC

#+RESULTS:
|   |        A |        B |        C |
|---+----------+----------+----------|
| 0 | 0.754799 | 0.492722 | 0.144595 |
| 1 | 0.198475 | 0.417721 | 0.083459 |
| 2 | 0.645011 | 0.444857 | 0.278874 |
| 3 | 0.314883 |   0.7521 | 0.789418 |
------------8<------------8<------------8<------------8<------------8<------------8<------------8<------------

However, this is not optimal, mainly because it pollutes the code block.
Note, that the :results is set to `raw`.

A brief discussion with Dov, yielded the following better workaround:
------------8<------------8<------------8<------------8<----
--------8<------------8<------------8<------------
(setq org-babel-python-wrapper-method "
def main():
%s

res = main()
if 'pandas.core.frame.DataFrame' in str(type(res)):
    from tabulate import tabulate
    out = tabulate(res, headers='keys', tablefmt='orgtbl')
else:
    out = str(res)

open('%s', 'w').write(out)")
------------8<------------8<------------8<------------8<----
--------8<------------8<------------8<------------

This allows a nice output of pandas.DataFrame (again when using :results
raw). Unfortunately, this wrapper has no influence in the case when
:session is used. To that end, it is possible to hack

------------8<------------8<------------8<------------8<----
--------8<------------8<------------8<------------
(defun org-babel-python-evaluate-session
    (session body &optional result-type result-params)
  "Pass BODY to the Python process in SESSION.
If RESULT-TYPE equals 'output then return standard output as a
string.  If RESULT-TYPE equals 'value then return the value of the
last statement in BODY, as elisp."
  (let* ((send-wait (lambda () (comint-send-input nil t) (sleep-for 0 5)))
 (dump-last-value
  (lambda
    (tmp-file pp)
    (mapc
     (lambda (statement) (insert statement) (funcall send-wait))
     (if pp
 (list
  "import pprint"
  (format "open('%s', 'w').write(pprint.pformat(_))"
  (org-babel-process-file-name tmp-file 'noquote)))
       (list (format "_org_tmp = _;

if 'pandas.core.frame.DataFrame' in str(type(_org_tmp)):
    from tabulate import tabulate
    _org_out = tabulate(_org_tmp, headers='keys', tablefmt='orgtbl')
else:
    _org_out = _org_tmp

open('%s', 'w').write(str(_org_out))"
     (org-babel-process-file-name tmp-file
                                                          'noquote)))))))
 (input-body (lambda (body)
       (mapc (lambda (line) (insert line) (funcall send-wait))
     (split-string body "[\r\n]"))
       (funcall send-wait)))
         (results
          (case result-type
            (output
             (mapconcat
              #'org-babel-trim
              (butlast
               (org-babel-comint-with-output
                   (session org-babel-python-eoe-indicator t body)
                 (funcall input-body body)
                 (funcall send-wait) (funcall send-wait)
                 (insert org-babel-python-eoe-indicator)
                 (funcall send-wait))
               2) "\n"))
            (value
             (let ((tmp-file (org-babel-temp-file "python-")))
               (org-babel-comint-with-output
                   (session org-babel-python-eoe-indicator nil body)
                 (let ((comint-process-echoes nil))
                   (funcall input-body body)
                   (funcall dump-last-value tmp-file
                            (member "pp" result-params))
                   (funcall send-wait) (funcall send-wait)
                   (insert org-babel-python-eoe-indicator)
                   (funcall send-wait)))
               (org-babel-eval-read-file tmp-file))))))
    (unless (string= (substring org-babel-python-eoe-indicator 1 -1)
results)
      (org-babel-result-cond result-params
results
        (org-babel-python-table-or-string results)))))
------------8<------------8<------------8<------------8<----
--------8<------------8<------------8<------------

This works, but I would be surprised if this hack meets org-mode's
standards. Nevertheless, maybe someone would find it useful.

What do you think? How can it be improved?

Best,
Dror

[-- Attachment #2: Type: text/html, Size: 12485 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2015-04-29  7:12 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-06-28  5:26 org-mode and python pandas Dov Grobgeld
2013-06-30 23:15 ` Eric Schulte
2013-07-01 16:34   ` Achim Gratz
2013-07-01 17:04     ` Rasmus
2013-07-03  9:15       ` Dov Grobgeld
2013-07-03 10:31         ` Rasmus
2013-07-03 14:09         ` Eric Schulte
2015-04-28  8:36           ` Dov Grobgeld
  -- strict thread matches above, loose matches on Subject: below --
2015-04-29  7:12 Dror Atariah

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).