From mboxrd@z Thu Jan 1 00:00:00 1970 From: Dror Atariah Subject: Re: org-mode and python pandas Date: Wed, 29 Apr 2015 09:12:34 +0200 Message-ID: Mime-Version: 1.0 Content-Type: multipart/alternative; boundary=047d7b86df560d90b10514d7b491 Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:35023) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YnMAf-0005iT-Sp for emacs-orgmode@gnu.org; Wed, 29 Apr 2015 03:12:39 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YnMAe-0002CM-HE for emacs-orgmode@gnu.org; Wed, 29 Apr 2015 03:12:37 -0400 Received: from mail-wi0-x230.google.com ([2a00:1450:400c:c05::230]:37906) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YnMAe-0002CB-5d for emacs-orgmode@gnu.org; Wed, 29 Apr 2015 03:12:36 -0400 Received: by wiun10 with SMTP id n10so54860095wiu.1 for ; Wed, 29 Apr 2015 00:12:34 -0700 (PDT) List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: "emacs-orgmode@gnu.org" --047d7b86df560d90b10514d7b491 Content-Type: text/plain; charset=UTF-8 Using the tabulate python module it is possible to have the following "inline" workaround: ------------8<------------8<------------8<------------8<------------8<------------8<------------8<------------ #+BEGIN_SRC python :results raw import pandas as pd import numpy as np from tabulate import tabulate df = pd.DataFrame(np.random.random((4,3)), columns=['A','B','C']) print("foo") return(tabulate(df, headers="keys", tablefmt="orgtbl")) #+END_SRC #+RESULTS: | | A | B | C | |---+----------+----------+----------| | 0 | 0.754799 | 0.492722 | 0.144595 | | 1 | 0.198475 | 0.417721 | 0.083459 | | 2 | 0.645011 | 0.444857 | 0.278874 | | 3 | 0.314883 | 0.7521 | 0.789418 | ------------8<------------8<------------8<------------8<------------8<------------8<------------8<------------ However, this is not optimal, mainly because it pollutes the code block. Note, that the :results is set to `raw`. A brief discussion with Dov, yielded the following better workaround: ------------8<------------8<------------8<------------8<---- --------8<------------8<------------8<------------ (setq org-babel-python-wrapper-method " def main(): %s res = main() if 'pandas.core.frame.DataFrame' in str(type(res)): from tabulate import tabulate out = tabulate(res, headers='keys', tablefmt='orgtbl') else: out = str(res) open('%s', 'w').write(out)") ------------8<------------8<------------8<------------8<---- --------8<------------8<------------8<------------ This allows a nice output of pandas.DataFrame (again when using :results raw). Unfortunately, this wrapper has no influence in the case when :session is used. To that end, it is possible to hack ------------8<------------8<------------8<------------8<---- --------8<------------8<------------8<------------ (defun org-babel-python-evaluate-session (session body &optional result-type result-params) "Pass BODY to the Python process in SESSION. If RESULT-TYPE equals 'output then return standard output as a string. If RESULT-TYPE equals 'value then return the value of the last statement in BODY, as elisp." (let* ((send-wait (lambda () (comint-send-input nil t) (sleep-for 0 5))) (dump-last-value (lambda (tmp-file pp) (mapc (lambda (statement) (insert statement) (funcall send-wait)) (if pp (list "import pprint" (format "open('%s', 'w').write(pprint.pformat(_))" (org-babel-process-file-name tmp-file 'noquote))) (list (format "_org_tmp = _; if 'pandas.core.frame.DataFrame' in str(type(_org_tmp)): from tabulate import tabulate _org_out = tabulate(_org_tmp, headers='keys', tablefmt='orgtbl') else: _org_out = _org_tmp open('%s', 'w').write(str(_org_out))" (org-babel-process-file-name tmp-file 'noquote))))))) (input-body (lambda (body) (mapc (lambda (line) (insert line) (funcall send-wait)) (split-string body "[\r\n]")) (funcall send-wait))) (results (case result-type (output (mapconcat #'org-babel-trim (butlast (org-babel-comint-with-output (session org-babel-python-eoe-indicator t body) (funcall input-body body) (funcall send-wait) (funcall send-wait) (insert org-babel-python-eoe-indicator) (funcall send-wait)) 2) "\n")) (value (let ((tmp-file (org-babel-temp-file "python-"))) (org-babel-comint-with-output (session org-babel-python-eoe-indicator nil body) (let ((comint-process-echoes nil)) (funcall input-body body) (funcall dump-last-value tmp-file (member "pp" result-params)) (funcall send-wait) (funcall send-wait) (insert org-babel-python-eoe-indicator) (funcall send-wait))) (org-babel-eval-read-file tmp-file)))))) (unless (string= (substring org-babel-python-eoe-indicator 1 -1) results) (org-babel-result-cond result-params results (org-babel-python-table-or-string results))))) ------------8<------------8<------------8<------------8<---- --------8<------------8<------------8<------------ This works, but I would be surprised if this hack meets org-mode's standards. Nevertheless, maybe someone would find it useful. What do you think? How can it be improved? Best, Dror --047d7b86df560d90b10514d7b491 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Using= the tabulate python module it is possible to have the following "inli= ne" workaround:

------------8<------------8<------------8<---------= ---8<------------8<------------8<------------8<------------
#+BEGIN_SRC python :resu= lts raw
import pandas as p= d
import numpy as np
=
from tabulate import tabulate

df =3D pd.DataFrame(np.random.random((4,3)), col= umns=3D['A','B','C'])
print("foo")
return(tabulate(df, headers=3D"keys", tablefmt= =3D"orgtbl"))
#+= END_SRC

#+RESULTS:
| =C2=A0 | =C2=A0 =C2=A0 =C2=A0 =C2=A0A | =C2=A0 =C2= =A0 =C2=A0 =C2=A0B | =C2=A0 =C2=A0 =C2=A0 =C2=A0C |
|---+----------+----------+----------|
| 0 |=C2=A00.754799=C2=A0|=C2=A00.492722=C2=A0| 0.14= 4595 |
| 1 | 0.198475 |=C2= =A00.417721<= /a>=C2=A0| 0.083459 |
| 2 = |=C2=A00.645= 011=C2=A0|=C2=A00.444857=C2=A0|=C2=A00.278874=C2=A0|
| 3 | 0.314883 | =C2=A0 0.7521 |=C2=A00.789418=C2=A0|
------------8<------------8<------= ------8<------------8<------------8<------------8<------------8= <------------

However, this is not optimal, = mainly because it pollutes the code block. Note, that the :results is set t= o `raw`.

A brie= f discussion with Dov, yielded the following better workaround:
<= /div>
------------8<------------8<------------8<------------8<------------8<------------8&l= t;------------8<----= --------
(= setq org-babel-python-wrapper-method "
def main():
%s

res =3D main()
if 'pandas.core.f= rame.DataFrame' in str(type(res)):
=C2=A0 =C2=A0 from tabulat= e import tabulate
=C2=A0 =C2=A0 out =3D tabulate(res, headers=3D&= #39;keys', tablefmt=3D'orgtbl')
else:
=C2= =A0 =C2=A0 out =3D str(res)

open('%s', = 9;w').write(out)")
------------8<------------8<------------8<------------8<----= --------8<----------= --8<------------8<= ;------------

Thi= s allows a nice output of pandas.DataFrame (again when using :results raw).= Unfortunately, this wrapper has no influence in the case when :session is = used. To that end, it is possible to hack=C2=A0

----= --------8<------------8<------------8<------------8<------------8<------------8<------------8<------------
(defun org-babel-python-evaluate-se= ssion
=C2=A0 =C2=A0 (sessi= on body &optional result-type result-params)
=C2=A0 "Pass BODY to the Python process in SESS= ION.
If RESULT-TYPE equals= 'output then return standard output as a
string.=C2=A0 If RESULT-TYPE equals 'value then ret= urn the value of the
last = statement in BODY, as elisp."
=C2=A0 (let* ((send-wait (lambda () (comint-send-input nil t) (sle= ep-for 0 5)))
=C2=A0(dump-last-value
=C2=A0=C2=A0(lambda
=C2=A0=C2=A0 =C2=A0(tmp-file pp)
=C2=A0=C2=A0 =C2=A0(mapc
=C2=A0=C2=A0 = =C2=A0 (lambda (statement) (insert statement) (funcall send-wait))
=C2=A0=C2=A0 =C2=A0 (if pp
=C2=A0(list
=
=C2=A0=C2=A0"import pprint"
= =C2=A0=C2=A0(format "open('%s', 'w').write(pprint.pfor= mat(_))"
=C2=A0=C2=A0(org-babel-process-file-nam= e tmp-file 'noquote)))
=C2=A0=C2=A0 =C2=A0 =C2=A0 (l= ist (format "_org_tmp =3D _; =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0
if 'pandas.core.frame.DataFrame= 9; in str(type(_org_tmp)):
=C2=A0 =C2=A0 from tabulate import tabulate
=C2=A0 =C2=A0 _org_out =3D tabulate(_org_tmp, headers=3D= 'keys', tablefmt=3D'orgtbl')
else:
= =C2=A0 =C2=A0 _org_out =3D _org_tmp

open('%= s', 'w').write(str(_org_out))"
=C2=A0= =C2=A0 =C2=A0 (org-babel-process-file-name tmp-file
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 'noquote)))))))
=C2=A0(input-body (lambda (bo= dy)
=C2=A0=C2=A0 =C2=A0 =C2=A0 (mapc (lambda (line) (in= sert line) (funcall send-wait))
=C2=A0=C2=A0 =C2=A0 (s= plit-string body "[\r\n]"))
=C2=A0=C2=A0 =C2= =A0 =C2=A0 (funcall send-wait)))
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(results
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (case result-typ= e
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 (output
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(mapconcat
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 #'org-babel-trim
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (butlast
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0(org-babel-comint-with-output
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0(session org-babel-python-eoe-indicator t body)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0(funcall input-body body)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0(funcall send-wait) (funcall send-wait)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0(insert org-babel-python-eoe-indicator)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0(funcall send-wait))
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A02) &q= uot;\n"))
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (value
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(let ((tmp-fil= e (org-babel-temp-file "python-")))
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= (org-babel-comint-with-output
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(s= ession org-babel-python-eoe-indicator nil body)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0(let ((comint-process-echoes nil))
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0(funcall input-body body)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0(funcall dump-last-value tmp-file
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (member "pp" result-params= ))
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(funcall send-wait) (funcal= l send-wait)
=C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(insert org-babel-p= ython-eoe-indicator)
=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(funcall = send-wait)))
=C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(org-babel-eval-read-file tmp-fil= e))))))
=C2=A0 =C2=A0 (unl= ess (string=3D (substring org-babel-python-eoe-indicator 1 -1) results)
=C2=A0 =C2=A0 =C2=A0 (org-bab= el-result-cond result-params
results
=C2=A0 =C2=A0 =C2=A0 =C2=A0 (org-babel-python= -table-or-string results)))))
------------8<---= ---------8<---------= ---8<------------8<------------8<------------8<------------8<------------
This works, but I would be surprised if this h= ack meets org-mode's standards. Nevertheless, maybe someone would find = it useful.

What do you think? How can it be improved?

Best,
D= ror
--047d7b86df560d90b10514d7b491--