From: Kyle Meyer <kyle@kyleam.com>
To: Richard Stanton <stanton@haas.berkeley.edu>
Cc: emacs-orgmode@gnu.org, John Kitchin <jkitchin@andrew.cmu.edu>
Subject: Re: Extraneous output from Python code blocks using :session option
Date: Sun, 15 Mar 2015 20:40:02 -0400 [thread overview]
Message-ID: <87lhix6ahp.fsf@kmlap.domain.org> (raw)
In-Reply-To: <87mw3g2dep.fsf@nicolasgoaziou.fr> (Nicolas Goaziou's message of "Sat, 14 Mar 2015 09:26:06 +0100")
Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
[...]
> You also need to list new functions and variables in commit message,
> e.g.,
Thanks. I'll update the commit messages.
> Wouldn't it be better to find the situations above at the source? If may
> be due to some indentation problem at the beginning of the string, in
> which case an `org-trim' within `org-babel-comint-with-output' could
> solve it.
I agree, but I haven't figured out a good way to do this. As far as I
can tell, it seems like the input sent in the right format. Instead, it
is empty prompt markers that are causing the issue (see examples at end
for details).
I think there are three separate problems with Python session output
right now.
1. The empty prompt markers from multi-line code mess up the output
processing and leak into the final output.
The current patch fixes this in the final combined output, but I've
realized it doesn't cover all the cases. (I initially thought the
markers only leaked into the start of the output, but they can be in
the middle as well.)
Plus, as you mention, it seems like it'd be better to fix this more
upstream. Since the input seems to be sent in correctly, I think
this leaves org-babel-comint-with-output. Specifically, output lines
that consist of just a prompt (like '>>> ' or '... ') could be
filtered out before the output lines are joined and returned.
comint-prompt-regexp could be used for this check. One problem with
this is that I don't know if this will have unwanted effects on the
output for other languages that use org-babel-comint-with-output.
2. org-babel-python-evaluate-session will eat whitespace at the
beginning of output.
The leading spaces below would be lost in the results.
#+begin_src python :session :results output
print(' <-- leading spaces')
#+end_src
Changing org-babel-trim to org-babel-chomp in
org-babel-python-evaluate-session fixes this.
3. Valid code that has indentation and spaces cannot be sent.
I think this is just a limitation of the shell, since blank lines are
used to signal the end of blocks. One of the patches I sent deals
with this like python.el does (by using an intermediate file).
One issue with that patch is whether it should check for blank
lines *and* indentation, because right now it only checks for
indentation, meaning that some code is unnecessarily sent through an
intermediate file.
Below are two examples that explain #1 in more detail.
* Single line
#+begin_src python :session :results output
3
#+end_src
org-babel-python-evaluate-session splits the input by newlines,
resulting in 3 (no newline) being inserted. It then sends this input,
followed by a few more empty inputs (not needed here, but required to
finish the block if the statement had indentation), and then inserts and
sends the babel eoe indicator.
This ends up in the Python shell as
>>> 3
3
>>>
>>>
>>>
>>> 'org_babel_python_eoe'
'org_babel_python_eoe'
>>>
org-babel-comint-with-output receives then a series of text output,
"", "3\n>>> ", "", ">>> ", "", ">>> ", "", ">>> ", "",
"'org_babel_python_eoe'\n>>> "
which it concatenates to
3
>>> >>> >>> >>> 'org_babel_python_eoe'
>>>
It then splits this by comint-prompt-regexp, which includes '>>>'
at the start of a line, resulting in
("3\n" ">>> >>> >>> 'org_babel_python_eoe'\n" "")
org-babel-python-evaluate-session receives this, removes the last two
elements, and trims the first, giving the correct output:
#+RESULTS:
: 3
* Multiple lines, no indentation
Here is the same example, but with some problematic lines added to the
beginning.
#+begin_src python :session :results output
x = 3
y = 4
x + y
#+end_src
org-babel-python-evaluate-session splits the input by newlines and sends
x = 3, then x = 4, and then 3, as well as the empty lines and eoe
signal.
This gets ends up in the Python shell as
x = 3
>>> y = 4
>>> x + y
7
>>>
>>>
>>>
>>> 'org_babel_python_eoe'
'org_babel_python_eoe'
>>>
org-babel-comint-with-output receives the a series of text output,
"", ">>> ", <- From 'x = 3'
"", ">>> ", <- From 'y = 4'
"", "7\n>>> ", "", ">>> ", "", ">>> ", "", ">>> ", ""
"'org_babel_python_eoe'\n>>> "
which it concatenates to
>>> >>> 7
>>> >>> >>> >>> 'org_babel_python_eoe'
>>>
It then splits this by comint-prompt-regexp, which includes '>>>'
at the start of a line, resulting in
("" ">>> 7\n" ">>> >>> >>> 'org_babel_python_eoe'\n" "")
org-babel-python-evaluate-session receives this, removes the last two
elements, and trims the others. Unlike the output above, this has an
empty line at the beginning due to the additional ">>> " that resulted
from "x = 3", as well as ">>> " before 7 due to the additional ">>> "
that resulted from "y = 4".
#+RESULTS:
:
: >>> 7
--
Kyle
next prev parent reply other threads:[~2015-03-16 0:40 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-03-11 3:38 Extraneous output from Python code blocks using :session option Richard Stanton
2015-03-11 18:29 ` Kyle Meyer
2015-03-11 18:54 ` Richard Stanton
2015-03-12 15:17 ` John Kitchin
2015-03-12 15:34 ` Richard Stanton
2015-03-12 21:23 ` Kyle Meyer
2015-03-13 6:58 ` Kyle Meyer
2015-03-13 8:21 ` Nicolas Goaziou
2015-03-13 14:40 ` Kyle Meyer
2015-03-14 8:26 ` Nicolas Goaziou
2015-03-16 0:40 ` Kyle Meyer [this message]
2015-03-12 21:21 ` Kyle Meyer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.orgmode.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87lhix6ahp.fsf@kmlap.domain.org \
--to=kyle@kyleam.com \
--cc=emacs-orgmode@gnu.org \
--cc=jkitchin@andrew.cmu.edu \
--cc=stanton@haas.berkeley.edu \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).