From mboxrd@z Thu Jan 1 00:00:00 1970 From: =?UTF-8?Q?Ond=C5=99ej_Grover?= Subject: Proposal and RFC for improving ob-python Date: Sat, 5 Dec 2015 18:17:15 +0100 Message-ID: Mime-Version: 1.0 Content-Type: multipart/alternative; boundary=001a11c26650c8aa83052629cc62 Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:38190) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a5GSn-0000Tk-QK for emacs-orgmode@gnu.org; Sat, 05 Dec 2015 12:17:39 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1a5GSm-0004Fu-BV for emacs-orgmode@gnu.org; Sat, 05 Dec 2015 12:17:37 -0500 Received: from mail-lb0-x22c.google.com ([2a00:1450:4010:c04::22c]:35373) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1a5GSm-0004Fn-0H for emacs-orgmode@gnu.org; Sat, 05 Dec 2015 12:17:36 -0500 Received: by lbbed20 with SMTP id ed20so21829337lbb.2 for ; Sat, 05 Dec 2015 09:17:34 -0800 (PST) List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: emacs-orgmode@gnu.org --001a11c26650c8aa83052629cc62 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hello, I've been playing around with the Org-mode Babel framework and I am grateful to all the contributors for making this wonderful library. After some time I noticed that Python support seems a little hacky and inconsistent and after reading through ob-python.el and consulting Python documentation I came up with a proposal for improving it. The ob-ipython project tries to solve this hackiness in a different way by using the client-server infrastructure of IPython/Jupyter. That works quite well too, but my hope is that improving ob-python.el would also make it simpler to use IPython as the python REPL, relying only on the core of the Python language. It essentially boils down to implementing progn-like eval() function in Python which would return the result of the last statement if it is an expression. I have come up with a prototype of such a function by diving into Python scope internals and its AST capabilities. It was written using Org-mode and tangling so it is thoroughly documented and explained, a test suite is included. This interesting exercise made me appreciate Lisp even more. Here it is https://github.com/smartass101/python_block_eval I haven't licensed it yet, because I'm not sure what license would be appropriate if it was used by org-mode. Any suggestions? My proposal is to implement an equivalent of the following bash pseudo code for non session mode python -i << HEREDOC_END ret =3D block_eval(""" """) open().write(str(ret)) HEREDOC_END For session mode it would be even simpler, lines containing HEREDOC above would be dropped and the rest piped directly into the Python REPL. This also means that the 'org_babel_python_eoe' string indicator may not be necessary anymore because end of evaluation would be simply shown by a new line with the primary prompt appearing. The reason why `python -i` (force interactive mode) is used is the IMHO poor design choice in the CPython implementation (and other implementations have a similar issue AFAIK) to have fast but RO access to local variable scope in non-top-level (i.e. functions calling functions) frames/scopes. I tried to hack my way around it in the update_locals_after_eval branch of my repo to no avail, perhaps some Pythonista among you may know a solution. I also favor piping input into `python -i` because it means that a temporary file does not have to be created. As explained in the README.org in my repo, this inconsistency can be worked around by explicitly first evaluating the side-effect-only part of the block and than the last expression with a direct eval() call for each. This makes it longer by 2 lines, but has the advantage of properly handling variable scope and separating side-effects, which could be used to e.g. suppress output. Nevertheless, I think that for Org-mode Babel usage the `python -i` and block_eval() approach suffices, unless someone finds a way to use the advantages of the alternative approach to improve ob-python.el even further. I'm not a skilled Elisp programmer, so I wanted to ask around as to the feasibility of this endeavor or possibly availability of helping hands before I devote more time to this. Kind regards, Ond=C5=99ej Grover --001a11c26650c8aa83052629cc62 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hello,

I've been playing around wit= h the Org-mode Babel framework and I am grateful to all the contributors fo= r making this wonderful library. After some time I noticed that Python supp= ort seems a little hacky and inconsistent and after reading through ob-pyth= on.el and consulting Python documentation I came up with a proposal for imp= roving it.

The ob-ipython project tries to solve t= his hackiness in a different way by using the client-server infrastructure = of IPython/Jupyter. That works quite well too, but my hope is that improvin= g ob-python.el would also make it simpler to use IPython as the python REPL= , relying only on the core of the Python language.

=
It essentially boils down to implementing progn-like eval() function i= n Python which would return the result of the last statement if it is an ex= pression. I have come up with a prototype of such a function by diving into= Python scope internals and its AST capabilities. It was written using Org-= mode and tangling so it is thoroughly documented and explained, a test suit= e is included. This interesting exercise made me appreciate Lisp even more.= Here it is


=
python -i << HEREDOC_END
ret =3D block_eval("&quo= t;"
<CODE BLOCK BODY>
""")
open(<TMP FILE or PIPE>).write(str(ret))
HEREDOC_EN= D

For session mode it would be even simpler,= lines containing HEREDOC above would be dropped and the rest piped directl= y into the Python REPL.=C2=A0

This also means that= the=C2=A0'org_babel_python_eoe' string indicator may not be necess= ary anymore because end of evaluation would be simply shown by a new line w= ith the primary prompt appearing.

The reason why `= python -i` (force interactive mode) is used is the IMHO poor design choice = in the CPython implementation (and other implementations have a similar iss= ue AFAIK) to have fast but RO access to local variable scope in non-top-lev= el (i.e. functions calling functions) frames/scopes. I tried to hack my way= around it in the=C2=A0update_locals_after_eval branch of my repo to no ava= il, perhaps some Pythonista among you may know a solution.

I also favor piping input into `python -i` because it means that a= temporary file does not have to be created.

As ex= plained in the README.org in my repo, this inconsistency can be worked arou= nd by explicitly first evaluating the side-effect-only part of the block an= d than the last expression with a direct eval() call for each. This makes i= t longer by 2 lines, but has the advantage of properly handling variable sc= ope and separating side-effects, which could be used to e.g. suppress outpu= t. Nevertheless, I think that for Org-mode Babel usage the `python -i` and = block_eval() approach suffices, unless someone finds a way to use the advan= tages of the alternative approach to improve ob-python.el even further.

I'm not a skilled Elisp programmer, so I wanted t= o ask around as to the feasibility of this endeavor or possibly availabilit= y of helping hands before I devote more time to this.

<= div>Kind regards,
Ond=C5=99ej Grover
--001a11c26650c8aa83052629cc62--