Re: Extraneous output from Python code blocks using :session option

[Kyle Meyer <kyle@kyleam.com>]

Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
[...]
> You also need to list new functions and variables in commit message,
> e.g.,

Thanks.  I'll update the commit messages.

> Wouldn't it be better to find the situations above at the source? If may
> be due to some indentation problem at the beginning of the string, in
> which case an `org-trim' within `org-babel-comint-with-output' could
> solve it.

I agree, but I haven't figured out a good way to do this.  As far as I
can tell, it seems like the input sent in the right format.  Instead, it
is empty prompt markers that are causing the issue (see examples at end
for details).

I think there are three separate problems with Python session output
right now.

1. The empty prompt markers from multi-line code mess up the output
   processing and leak into the final output.

   The current patch fixes this in the final combined output, but I've
   realized it doesn't cover all the cases.  (I initially thought the
   markers only leaked into the start of the output, but they can be in
   the middle as well.)

   Plus, as you mention, it seems like it'd be better to fix this more
   upstream.  Since the input seems to be sent in correctly, I think
   this leaves org-babel-comint-with-output.  Specifically, output lines
   that consist of just a prompt (like '>>> ' or '... ') could be
   filtered out before the output lines are joined and returned.
   comint-prompt-regexp could be used for this check.  One problem with
   this is that I don't know if this will have unwanted effects on the
   output for other languages that use org-babel-comint-with-output.

2. org-babel-python-evaluate-session will eat whitespace at the
   beginning of output.

   The leading spaces below would be lost in the results.

   #+begin_src python :session :results output
     print('   <-- leading spaces')
   #+end_src

   Changing org-babel-trim to org-babel-chomp in
   org-babel-python-evaluate-session fixes this.

3. Valid code that has indentation and spaces cannot be sent.

   I think this is just a limitation of the shell, since blank lines are
   used to signal the end of blocks.  One of the patches I sent deals
   with this like python.el does (by using an intermediate file).

   One issue with that patch is whether it should check for blank
   lines *and* indentation, because right now it only checks for
   indentation, meaning that some code is unnecessarily sent through an
   intermediate file.

Below are two examples that explain #1 in more detail.

* Single line

#+begin_src python :session :results output
  3
#+end_src

org-babel-python-evaluate-session splits the input by newlines,
resulting in 3 (no newline) being inserted.  It then sends this input,
followed by a few more empty inputs (not needed here, but required to
finish the block if the statement had indentation), and then inserts and
sends the babel eoe indicator.

This ends up in the Python shell as

    >>> 3
    3
    >>>
    >>>
    >>>
    >>> 'org_babel_python_eoe'
    'org_babel_python_eoe'
    >>>

org-babel-comint-with-output receives then a series of text output,

    "", "3\n>>> ", "", ">>> ", "", ">>> ", "", ">>> ", "",
    "'org_babel_python_eoe'\n>>> "

which it concatenates to

    3
    >>> >>> >>> >>> 'org_babel_python_eoe'
    >>>

It then splits this by comint-prompt-regexp, which includes '>>>'
at the start of a line, resulting in

    ("3\n" ">>> >>> >>> 'org_babel_python_eoe'\n" "")

org-babel-python-evaluate-session receives this, removes the last two
elements, and trims the first, giving the correct output:

#+RESULTS:
: 3


* Multiple lines, no indentation

Here is the same example, but with some problematic lines added to the
beginning.

#+begin_src python :session :results output
  x = 3
  y = 4
  x + y
#+end_src

org-babel-python-evaluate-session splits the input by newlines and sends
x = 3, then x = 4, and then 3, as well as the empty lines and eoe
signal.

This gets ends up in the Python shell as

    x = 3
    >>> y = 4
    >>> x + y
    7
    >>> 
    >>> 
    >>> 
    >>> 'org_babel_python_eoe'
    'org_babel_python_eoe'
    >>> 

org-babel-comint-with-output receives the a series of text output,

    "", ">>> ",  <- From 'x = 3'
    "", ">>> ",  <- From 'y = 4'
    "", "7\n>>> ", "", ">>> ", "", ">>> ", "", ">>> ", ""
    "'org_babel_python_eoe'\n>>> "

which it concatenates to

    >>> >>> 7
    >>> >>> >>> >>> 'org_babel_python_eoe'
    >>>

It then splits this by comint-prompt-regexp, which includes '>>>'
at the start of a line, resulting in

    ("" ">>> 7\n" ">>> >>> >>> 'org_babel_python_eoe'\n" "")

org-babel-python-evaluate-session receives this, removes the last two
elements, and trims the others.  Unlike the output above, this has an
empty line at the beginning due to the additional ">>> " that resulted
from "x = 3", as well as ">>> " before 7 due to the additional ">>> "
that resulted from "y = 4".

#+RESULTS:
: 
: >>> 7

--
Kyle