emacs-orgmode@gnu.org archives
 help / color / mirror / code / Atom feed
* Proposal and RFC for improving ob-python
@ 2015-12-05 17:17 Ondřej Grover
  2015-12-10  7:22 ` Kyle Meyer
  2016-08-29 12:53 ` Greg Minshall
  0 siblings, 2 replies; 12+ messages in thread
From: Ondřej Grover @ 2015-12-05 17:17 UTC (permalink / raw)
  To: emacs-orgmode

[-- Attachment #1: Type: text/plain, Size: 3290 bytes --]

Hello,

I've been playing around with the Org-mode Babel framework and I am
grateful to all the contributors for making this wonderful library. After
some time I noticed that Python support seems a little hacky and
inconsistent and after reading through ob-python.el and consulting Python
documentation I came up with a proposal for improving it.

The ob-ipython project tries to solve this hackiness in a different way by
using the client-server infrastructure of IPython/Jupyter. That works quite
well too, but my hope is that improving ob-python.el would also make it
simpler to use IPython as the python REPL, relying only on the core of the
Python language.

It essentially boils down to implementing progn-like eval() function in
Python which would return the result of the last statement if it is an
expression. I have come up with a prototype of such a function by diving
into Python scope internals and its AST capabilities. It was written using
Org-mode and tangling so it is thoroughly documented and explained, a test
suite is included. This interesting exercise made me appreciate Lisp even
more. Here it is
https://github.com/smartass101/python_block_eval
I haven't licensed it yet, because I'm not sure what license would be
appropriate if it was used by org-mode. Any suggestions?

My proposal is to implement an equivalent of the following bash pseudo code
for non session mode

python -i << HEREDOC_END
ret = block_eval("""
<CODE BLOCK BODY>
""")
open(<TMP FILE or PIPE>).write(str(ret))
HEREDOC_END

For session mode it would be even simpler, lines containing HEREDOC above
would be dropped and the rest piped directly into the Python REPL.

This also means that the 'org_babel_python_eoe' string indicator may not be
necessary anymore because end of evaluation would be simply shown by a new
line with the primary prompt appearing.

The reason why `python -i` (force interactive mode) is used is the IMHO
poor design choice in the CPython implementation (and other implementations
have a similar issue AFAIK) to have fast but RO access to local variable
scope in non-top-level (i.e. functions calling functions) frames/scopes. I
tried to hack my way around it in the update_locals_after_eval branch of my
repo to no avail, perhaps some Pythonista among you may know a solution.

I also favor piping input into `python -i` because it means that a
temporary file does not have to be created.

As explained in the README.org in my repo, this inconsistency can be worked
around by explicitly first evaluating the side-effect-only part of the
block and than the last expression with a direct eval() call for each. This
makes it longer by 2 lines, but has the advantage of properly handling
variable scope and separating side-effects, which could be used to e.g.
suppress output. Nevertheless, I think that for Org-mode Babel usage the
`python -i` and block_eval() approach suffices, unless someone finds a way
to use the advantages of the alternative approach to improve ob-python.el
even further.

I'm not a skilled Elisp programmer, so I wanted to ask around as to the
feasibility of this endeavor or possibly availability of helping hands
before I devote more time to this.

Kind regards,
Ondřej Grover

[-- Attachment #2: Type: text/html, Size: 3786 bytes --]

^ permalink raw reply	[flat|nested] 12+ messages in thread
* Re: Proposal and RFC for improving ob-python
@ 2015-12-07 19:22 Ondřej Grover
  2015-12-07 20:27 ` Kyle Meyer
                   ` (2 more replies)
  0 siblings, 3 replies; 12+ messages in thread
From: Ondřej Grover @ 2015-12-07 19:22 UTC (permalink / raw)
  To: emacs-orgmode

[-- Attachment #1: Type: text/plain, Size: 3945 bytes --]

TL;DR for those that may have been scared off by the length of my previous
email:
I propose a method of improving ob-python.el by using a progn-like eval()
Python function which can wrap and execute source blocks.

If this list is not appropriate for discussion of improvements to
ob-python.el, could you please give me pointers how to reach people that
might be interested in helping and discussing?

Kind regards,
Ondřej Grover

On Sat, Dec 5, 2015 at 6:17 PM, Ondřej Grover <ondrej.grover@gmail.com>
wrote:

> Hello,
>
> I've been playing around with the Org-mode Babel framework and I am
> grateful to all the contributors for making this wonderful library. After
> some time I noticed that Python support seems a little hacky and
> inconsistent and after reading through ob-python.el and consulting Python
> documentation I came up with a proposal for improving it.
>
> The ob-ipython project tries to solve this hackiness in a different way by
> using the client-server infrastructure of IPython/Jupyter. That works quite
> well too, but my hope is that improving ob-python.el would also make it
> simpler to use IPython as the python REPL, relying only on the core of the
> Python language.
>
> It essentially boils down to implementing progn-like eval() function in
> Python which would return the result of the last statement if it is an
> expression. I have come up with a prototype of such a function by diving
> into Python scope internals and its AST capabilities. It was written using
> Org-mode and tangling so it is thoroughly documented and explained, a test
> suite is included. This interesting exercise made me appreciate Lisp even
> more. Here it is
> https://github.com/smartass101/python_block_eval
> I haven't licensed it yet, because I'm not sure what license would be
> appropriate if it was used by org-mode. Any suggestions?
>
> My proposal is to implement an equivalent of the following bash pseudo
> code for non session mode
>
> python -i << HEREDOC_END
> ret = block_eval("""
> <CODE BLOCK BODY>
> """)
> open(<TMP FILE or PIPE>).write(str(ret))
> HEREDOC_END
>
> For session mode it would be even simpler, lines containing HEREDOC above
> would be dropped and the rest piped directly into the Python REPL.
>
> This also means that the 'org_babel_python_eoe' string indicator may not
> be necessary anymore because end of evaluation would be simply shown by a
> new line with the primary prompt appearing.
>
> The reason why `python -i` (force interactive mode) is used is the IMHO
> poor design choice in the CPython implementation (and other implementations
> have a similar issue AFAIK) to have fast but RO access to local variable
> scope in non-top-level (i.e. functions calling functions) frames/scopes. I
> tried to hack my way around it in the update_locals_after_eval branch of my
> repo to no avail, perhaps some Pythonista among you may know a solution.
>
> I also favor piping input into `python -i` because it means that a
> temporary file does not have to be created.
>
> As explained in the README.org in my repo, this inconsistency can be
> worked around by explicitly first evaluating the side-effect-only part of
> the block and than the last expression with a direct eval() call for each.
> This makes it longer by 2 lines, but has the advantage of properly handling
> variable scope and separating side-effects, which could be used to e.g.
> suppress output. Nevertheless, I think that for Org-mode Babel usage the
> `python -i` and block_eval() approach suffices, unless someone finds a way
> to use the advantages of the alternative approach to improve ob-python.el
> even further.
>
> I'm not a skilled Elisp programmer, so I wanted to ask around as to the
> feasibility of this endeavor or possibly availability of helping hands
> before I devote more time to this.
>
> Kind regards,
> Ondřej Grover
>

[-- Attachment #2: Type: text/html, Size: 4665 bytes --]

^ permalink raw reply	[flat|nested] 12+ messages in thread
* Re: Proposal and RFC for improving ob-python
@ 2015-12-10  7:47 Ondřej Grover
  2015-12-11  3:39 ` Kyle Meyer
  0 siblings, 1 reply; 12+ messages in thread
From: Ondřej Grover @ 2015-12-10  7:47 UTC (permalink / raw)
  To: emacs-orgmode

[-- Attachment #1: Type: text/plain, Size: 1682 bytes --]

Thank you for the feedback Kyle.

> As I understand it (and if I'm remembering correctly), there are at
> least three issues with ob-python sessions:

4. session and non-session code differs significantly as you have already
mentioned

> So for non-session code and value results, the main idea is to get rid
> of the need for "return" at the end?
>
Yes, that's the greatest offense to the Python code.

>
> This seems useful to me.  Requiring a trailing "return", which is a
> syntax error when the code is taken by itself, is a quirk of the
> implementation (the code ends up being wrapped in a function to get the
> return value).  Also, it would allow switching from ":results value" to
> ":results output" without needing to remove "return" in order to avoid a
> syntax error.
>
Yes, these would be the obvious benefits.

> But what about when :results != value?  Doesn't your proposal only
> handle returning the last value?
>
You mean :results output ? In that case it could just omit the
"open(...).write(...) " part and capture anything the console prints before
the primary prompt appears again.
Or it could capture stdout into a file, but I think that's needlessly
complicated:
python -i << HEREDOC_END
import sys
sys.stdout = open(<TMP FILE or PIPE>)   # replace stdout with some file
_ = block_eval("""
<CODE BLOCK BODY>
""")
sys.stdout.close()
sys.stdout = sys.__stdout__  # restore stdout
HEREDOC_END

After thinking about this, perhaps it is better to capture the block result
into "_" instead of "ret", because
a) it prevents name clashes
b) it is more in line with what "_" is used for

Kind regards,
Ondřej Grover

[-- Attachment #2: Type: text/html, Size: 2995 bytes --]

^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2016-08-31  2:44 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2015-12-05 17:17 Proposal and RFC for improving ob-python Ondřej Grover
2015-12-10  7:22 ` Kyle Meyer
2016-08-29 12:53 ` Greg Minshall
2016-08-30 19:17   ` John Kitchin
2016-08-31  2:44     ` Greg Minshall
  -- strict thread matches above, loose matches on Subject: below --
2015-12-07 19:22 Ondřej Grover
2015-12-07 20:27 ` Kyle Meyer
2015-12-08  7:43   ` Ondřej Grover
2015-12-07 20:36 ` Thomas S. Dye
2015-12-07 20:51 ` Achim Gratz
2015-12-10  7:47 Ondřej Grover
2015-12-11  3:39 ` Kyle Meyer

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).