From mboxrd@z Thu Jan 1 00:00:00 1970 From: Kyle Meyer Subject: Re: Extraneous output from Python code blocks using :session option Date: Sun, 15 Mar 2015 20:40:02 -0400 Message-ID: <87lhix6ahp.fsf@kmlap.domain.org> References: <1931A590-8B23-4412-86C7-F3571EC466FF@haas.berkeley.edu> <87r3sv2xra.fsf@kyleam.com> <210D96E7-43E3-4438-8401-7841C6612023@haas.berkeley.edu> <87bnjxq59p.fsf@kyleam.com> <87d24dtmck.fsf@kmlap.domain.org> <87ioe5xq8a.fsf@nicolasgoaziou.fr> <877fult0yy.fsf@kmlap.domain.org> <87mw3g2dep.fsf@nicolasgoaziou.fr> Mime-Version: 1.0 Content-Type: text/plain Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:55280) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YXJ5L-0003T2-Qj for emacs-orgmode@gnu.org; Sun, 15 Mar 2015 20:40:49 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YXJ5G-00064Z-IS for emacs-orgmode@gnu.org; Sun, 15 Mar 2015 20:40:47 -0400 Received: from mail-qc0-f173.google.com ([209.85.216.173]:35103) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YXJ0M-0004NY-6K for emacs-orgmode@gnu.org; Sun, 15 Mar 2015 20:35:38 -0400 Received: by qcbkw5 with SMTP id kw5so31358967qcb.2 for ; Sun, 15 Mar 2015 17:35:37 -0700 (PDT) In-Reply-To: <87mw3g2dep.fsf@nicolasgoaziou.fr> (Nicolas Goaziou's message of "Sat, 14 Mar 2015 09:26:06 +0100") List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: Richard Stanton Cc: emacs-orgmode@gnu.org, John Kitchin Nicolas Goaziou wrote: [...] > You also need to list new functions and variables in commit message, > e.g., Thanks. I'll update the commit messages. > Wouldn't it be better to find the situations above at the source? If may > be due to some indentation problem at the beginning of the string, in > which case an `org-trim' within `org-babel-comint-with-output' could > solve it. I agree, but I haven't figured out a good way to do this. As far as I can tell, it seems like the input sent in the right format. Instead, it is empty prompt markers that are causing the issue (see examples at end for details). I think there are three separate problems with Python session output right now. 1. The empty prompt markers from multi-line code mess up the output processing and leak into the final output. The current patch fixes this in the final combined output, but I've realized it doesn't cover all the cases. (I initially thought the markers only leaked into the start of the output, but they can be in the middle as well.) Plus, as you mention, it seems like it'd be better to fix this more upstream. Since the input seems to be sent in correctly, I think this leaves org-babel-comint-with-output. Specifically, output lines that consist of just a prompt (like '>>> ' or '... ') could be filtered out before the output lines are joined and returned. comint-prompt-regexp could be used for this check. One problem with this is that I don't know if this will have unwanted effects on the output for other languages that use org-babel-comint-with-output. 2. org-babel-python-evaluate-session will eat whitespace at the beginning of output. The leading spaces below would be lost in the results. #+begin_src python :session :results output print(' <-- leading spaces') #+end_src Changing org-babel-trim to org-babel-chomp in org-babel-python-evaluate-session fixes this. 3. Valid code that has indentation and spaces cannot be sent. I think this is just a limitation of the shell, since blank lines are used to signal the end of blocks. One of the patches I sent deals with this like python.el does (by using an intermediate file). One issue with that patch is whether it should check for blank lines *and* indentation, because right now it only checks for indentation, meaning that some code is unnecessarily sent through an intermediate file. Below are two examples that explain #1 in more detail. * Single line #+begin_src python :session :results output 3 #+end_src org-babel-python-evaluate-session splits the input by newlines, resulting in 3 (no newline) being inserted. It then sends this input, followed by a few more empty inputs (not needed here, but required to finish the block if the statement had indentation), and then inserts and sends the babel eoe indicator. This ends up in the Python shell as >>> 3 3 >>> >>> >>> >>> 'org_babel_python_eoe' 'org_babel_python_eoe' >>> org-babel-comint-with-output receives then a series of text output, "", "3\n>>> ", "", ">>> ", "", ">>> ", "", ">>> ", "", "'org_babel_python_eoe'\n>>> " which it concatenates to 3 >>> >>> >>> >>> 'org_babel_python_eoe' >>> It then splits this by comint-prompt-regexp, which includes '>>>' at the start of a line, resulting in ("3\n" ">>> >>> >>> 'org_babel_python_eoe'\n" "") org-babel-python-evaluate-session receives this, removes the last two elements, and trims the first, giving the correct output: #+RESULTS: : 3 * Multiple lines, no indentation Here is the same example, but with some problematic lines added to the beginning. #+begin_src python :session :results output x = 3 y = 4 x + y #+end_src org-babel-python-evaluate-session splits the input by newlines and sends x = 3, then x = 4, and then 3, as well as the empty lines and eoe signal. This gets ends up in the Python shell as x = 3 >>> y = 4 >>> x + y 7 >>> >>> >>> >>> 'org_babel_python_eoe' 'org_babel_python_eoe' >>> org-babel-comint-with-output receives the a series of text output, "", ">>> ", <- From 'x = 3' "", ">>> ", <- From 'y = 4' "", "7\n>>> ", "", ">>> ", "", ">>> ", "", ">>> ", "" "'org_babel_python_eoe'\n>>> " which it concatenates to >>> >>> 7 >>> >>> >>> >>> 'org_babel_python_eoe' >>> It then splits this by comint-prompt-regexp, which includes '>>>' at the start of a line, resulting in ("" ">>> 7\n" ">>> >>> >>> 'org_babel_python_eoe'\n" "") org-babel-python-evaluate-session receives this, removes the last two elements, and trims the others. Unlike the output above, this has an empty line at the beginning due to the additional ">>> " that resulted from "x = 3", as well as ">>> " before 7 due to the additional ">>> " that resulted from "y = 4". #+RESULTS: : : >>> 7 -- Kyle