* Extraneous output from Python code blocks using :session option @ 2015-03-11 3:38 Richard Stanton 2015-03-11 18:29 ` Kyle Meyer 0 siblings, 1 reply; 12+ messages in thread From: Richard Stanton @ 2015-03-11 3:38 UTC (permalink / raw) To: emacs-orgmode I'm trying to use the :session option so I can import modules, etc., just once at the beginning of my document, like with am IPython notebook. Unfortunately, the output from these code blocks contains some extraneous characters. For example: #+BEGIN_SRC python :session :results output a = 2 b = 3 c = 4 print 'a= ', a print 'b = ', b print 'a + b = ', a+b #+END_SRC #+RESULTS: : : >>> >>> a= 2 : b = 3 : a + b = 5 How can I stop the production of all those ">" signs (sometimes they're dots), which don't appear if I run the same code block without the :session option? Thanks. Richard Stanton ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Extraneous output from Python code blocks using :session option 2015-03-11 3:38 Extraneous output from Python code blocks using :session option Richard Stanton @ 2015-03-11 18:29 ` Kyle Meyer 2015-03-11 18:54 ` Richard Stanton 2015-03-12 15:17 ` John Kitchin 0 siblings, 2 replies; 12+ messages in thread From: Kyle Meyer @ 2015-03-11 18:29 UTC (permalink / raw) To: Richard Stanton; +Cc: emacs-orgmode [-- Attachment #1: Type: text/plain, Size: 1125 bytes --] Richard Stanton <stanton@haas.berkeley.edu> wrote: > I'm trying to use the :session option so I can import modules, etc., > just once at the beginning of my document, like with am IPython > notebook. Unfortunately, the output from these code blocks contains > some extraneous characters. For example: > > #+BEGIN_SRC python :session :results output > a = 2 > b = 3 > c = 4 > print 'a= ', a > print 'b = ', b > print 'a + b = ', a+b > #+END_SRC > > #+RESULTS: > : > : >>> >>> a= 2 > : b = 3 > : a + b = 5 I also see this behavior. I asked the list some time ago [1], but I didn't follow up beyond the initial email. > How can I stop the production of all those ">" signs (sometimes > they're dots), which don't appear if I run the same code block without > the :session option? I've attached a patch that seems to fix the example you gave and the cases from my earlier email. I don't know enough about babel's internals to know if it is a good way to fix it, but at least it might serve as a quick fix for you until there is a better solution. [1] http://thread.gmane.org/gmane.emacs.orgmode/79014 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 0001-ob-python.el-Strip-leading-session-characters.patch --] [-- Type: text/x-diff, Size: 2137 bytes --] From cfa2783e03152871c0d703a6f79e96254dbe6c44 Mon Sep 17 00:00:00 2001 From: Kyle Meyer <kyle@kyleam.com> Date: Wed, 11 Mar 2015 14:17:09 -0400 Subject: [PATCH] ob-python.el: Strip leading session characters * lisp/ob-python.el (org-babel-python-evaluate-session): Strip leading "..." and ">>>" from session output. (org-babel-python-strip-session-chars): New function. --- lisp/ob-python.el | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-) diff --git a/lisp/ob-python.el b/lisp/ob-python.el index dd3cc66..0267f22 100644 --- a/lisp/ob-python.el +++ b/lisp/ob-python.el @@ -306,16 +306,17 @@ (defun org-babel-python-evaluate-session (results (case result-type (output - (mapconcat - #'org-babel-trim - (butlast - (org-babel-comint-with-output - (session org-babel-python-eoe-indicator t body) - (funcall input-body body) - (funcall send-wait) (funcall send-wait) - (insert org-babel-python-eoe-indicator) - (funcall send-wait)) - 2) "\n")) + (org-babel-python-strip-session-chars + (mapconcat + #'org-babel-chomp + (butlast + (org-babel-comint-with-output + (session org-babel-python-eoe-indicator t body) + (funcall input-body body) + (funcall send-wait) (funcall send-wait) + (insert org-babel-python-eoe-indicator) + (funcall send-wait)) + 2) "\n"))) (value (let ((tmp-file (org-babel-temp-file "python-"))) (org-babel-comint-with-output @@ -339,6 +340,15 @@ (defun org-babel-python-read-string (string) (match-string 1 string) string)) +(defun org-babel-python-strip-session-chars (string) + "Remove leading >>> and ... from Python session output." + (with-temp-buffer + (insert string) + (goto-char (point-min)) + (when (looking-at "\\s-*\\(\\(>>> \\)*\\(\\.\\.\\. \\)*\\)") + (delete-region (match-beginning 1) (match-end 1))) + (buffer-string))) + (provide 'ob-python) -- 2.3.2 [-- Attachment #3: Type: text/plain, Size: 9 bytes --] -- Kyle ^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: Extraneous output from Python code blocks using :session option 2015-03-11 18:29 ` Kyle Meyer @ 2015-03-11 18:54 ` Richard Stanton 2015-03-12 15:17 ` John Kitchin 1 sibling, 0 replies; 12+ messages in thread From: Richard Stanton @ 2015-03-11 18:54 UTC (permalink / raw) To: Kyle Meyer; +Cc: emacs-orgmode > On Mar 11, 2015, at 11:29 AM, Kyle Meyer <kyle@kyleam.com> wrote: > > Richard Stanton <stanton@haas.berkeley.edu> wrote: >> I'm trying to use the :session option so I can import modules, etc., >> just once at the beginning of my document, like with am IPython >> notebook. Unfortunately, the output from these code blocks contains >> some extraneous characters. For example: >> >> #+BEGIN_SRC python :session :results output >> a = 2 >> b = 3 >> c = 4 >> print 'a= ', a >> print 'b = ', b >> print 'a + b = ', a+b >> #+END_SRC >> >> #+RESULTS: >> : >> : >>> >>> a= 2 >> : b = 3 >> : a + b = 5 > > I also see this behavior. I asked the list some time ago [1], but I > didn't follow up beyond the initial email. > >> How can I stop the production of all those ">" signs (sometimes >> they're dots), which don't appear if I run the same code block without >> the :session option? > > I've attached a patch that seems to fix the example you gave and the > cases from my earlier email. I don't know enough about babel's > internals to know if it is a good way to fix it, but at least it might > serve as a quick fix for you until there is a better solution. > > [1] http://thread.gmane.org/gmane.emacs.orgmode/79014 > > From cfa2783e03152871c0d703a6f79e96254dbe6c44 Mon Sep 17 00:00:00 2001 > From: Kyle Meyer <kyle@kyleam.com> > Date: Wed, 11 Mar 2015 14:17:09 -0400 > Subject: [PATCH] ob-python.el: Strip leading session characters > > * lisp/ob-python.el (org-babel-python-evaluate-session): Strip leading > "..." and ">>>" from session output. > (org-babel-python-strip-session-chars): New function. > --- > lisp/ob-python.el | 30 ++++++++++++++++++++---------- > 1 file changed, 20 insertions(+), 10 deletions(-) > > diff --git a/lisp/ob-python.el b/lisp/ob-python.el > index dd3cc66..0267f22 100644 > --- a/lisp/ob-python.el > +++ b/lisp/ob-python.el > @@ -306,16 +306,17 @@ (defun org-babel-python-evaluate-session > (results > (case result-type > (output > - (mapconcat > - #'org-babel-trim > - (butlast > - (org-babel-comint-with-output > - (session org-babel-python-eoe-indicator t body) > - (funcall input-body body) > - (funcall send-wait) (funcall send-wait) > - (insert org-babel-python-eoe-indicator) > - (funcall send-wait)) > - 2) "\n")) > + (org-babel-python-strip-session-chars > + (mapconcat > + #'org-babel-chomp > + (butlast > + (org-babel-comint-with-output > + (session org-babel-python-eoe-indicator t body) > + (funcall input-body body) > + (funcall send-wait) (funcall send-wait) > + (insert org-babel-python-eoe-indicator) > + (funcall send-wait)) > + 2) "\n"))) > (value > (let ((tmp-file (org-babel-temp-file "python-"))) > (org-babel-comint-with-output > @@ -339,6 +340,15 @@ (defun org-babel-python-read-string (string) > (match-string 1 string) > string)) > > +(defun org-babel-python-strip-session-chars (string) > + "Remove leading >>> and ... from Python session output." > + (with-temp-buffer > + (insert string) > + (goto-char (point-min)) > + (when (looking-at "\\s-*\\(\\(>>> \\)*\\(\\.\\.\\. \\)*\\)") > + (delete-region (match-beginning 1) (match-end 1))) > + (buffer-string))) > + > (provide 'ob-python) > > > -- > 2.3.2 > > > — > Kyle Many thanks, Kyle. It would be great if this could be fixed in the main code, but I’ll try your patch today. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Extraneous output from Python code blocks using :session option 2015-03-11 18:29 ` Kyle Meyer 2015-03-11 18:54 ` Richard Stanton @ 2015-03-12 15:17 ` John Kitchin 2015-03-12 15:34 ` Richard Stanton 2015-03-12 21:21 ` Kyle Meyer 1 sibling, 2 replies; 12+ messages in thread From: John Kitchin @ 2015-03-12 15:17 UTC (permalink / raw) To: Kyle Meyer; +Cc: Richard Stanton, emacs-orgmode Nice. I hope this makes it into org-mode one day. In the mean time, I crafted an approach with another hook function that is described here: http://kitchingroup.cheme.cmu.edu/blog/2015/03/12/Making-org-mode-Python-sessions-look-better/ This can be done in an init file. Kyle Meyer writes: > Richard Stanton <stanton@haas.berkeley.edu> wrote: >> I'm trying to use the :session option so I can import modules, etc., >> just once at the beginning of my document, like with am IPython >> notebook. Unfortunately, the output from these code blocks contains >> some extraneous characters. For example: >> >> #+BEGIN_SRC python :session :results output >> a = 2 >> b = 3 >> c = 4 >> print 'a= ', a >> print 'b = ', b >> print 'a + b = ', a+b >> #+END_SRC >> >> #+RESULTS: >> : >> : >>> >>> a= 2 >> : b = 3 >> : a + b = 5 > > I also see this behavior. I asked the list some time ago [1], but I > didn't follow up beyond the initial email. > >> How can I stop the production of all those ">" signs (sometimes >> they're dots), which don't appear if I run the same code block without >> the :session option? > > I've attached a patch that seems to fix the example you gave and the > cases from my earlier email. I don't know enough about babel's > internals to know if it is a good way to fix it, but at least it might > serve as a quick fix for you until there is a better solution. > > [1] http://thread.gmane.org/gmane.emacs.orgmode/79014 -- Professor John Kitchin Doherty Hall A207F Department of Chemical Engineering Carnegie Mellon University Pittsburgh, PA 15213 412-268-7803 @johnkitchin http://kitchingroup.cheme.cmu.edu ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Extraneous output from Python code blocks using :session option 2015-03-12 15:17 ` John Kitchin @ 2015-03-12 15:34 ` Richard Stanton 2015-03-12 21:23 ` Kyle Meyer 2015-03-12 21:21 ` Kyle Meyer 1 sibling, 1 reply; 12+ messages in thread From: Richard Stanton @ 2015-03-12 15:34 UTC (permalink / raw) To: John Kitchin; +Cc: Kyle Meyer, emacs-orgmode This looks great. While we're patching this code, why does having blank lines inside function definitions cause such problems in :session mode? It would be nice if you could just type any valid Python code and have it execute (like you can in an IPython notebook cell) > On Mar 12, 2015, at 8:17 AM, John Kitchin <jkitchin@andrew.cmu.edu> wrote: > > Nice. I hope this makes it into org-mode one day. > > In the mean time, I crafted an approach with another hook function that > is described here: > > http://kitchingroup.cheme.cmu.edu/blog/2015/03/12/Making-org-mode-Python-sessions-look-better/ > > This can be done in an init file. > > Kyle Meyer writes: > >> Richard Stanton <stanton@haas.berkeley.edu> wrote: >>> I'm trying to use the :session option so I can import modules, etc., >>> just once at the beginning of my document, like with am IPython >>> notebook. Unfortunately, the output from these code blocks contains >>> some extraneous characters. For example: >>> >>> #+BEGIN_SRC python :session :results output >>> a = 2 >>> b = 3 >>> c = 4 >>> print 'a= ', a >>> print 'b = ', b >>> print 'a + b = ', a+b >>> #+END_SRC >>> >>> #+RESULTS: >>> : >>> : >>> >>> a= 2 >>> : b = 3 >>> : a + b = 5 >> >> I also see this behavior. I asked the list some time ago [1], but I >> didn't follow up beyond the initial email. >> >>> How can I stop the production of all those ">" signs (sometimes >>> they're dots), which don't appear if I run the same code block without >>> the :session option? >> >> I've attached a patch that seems to fix the example you gave and the >> cases from my earlier email. I don't know enough about babel's >> internals to know if it is a good way to fix it, but at least it might >> serve as a quick fix for you until there is a better solution. >> >> [1] http://thread.gmane.org/gmane.emacs.orgmode/79014 > > -- > Professor John Kitchin > Doherty Hall A207F > Department of Chemical Engineering > Carnegie Mellon University > Pittsburgh, PA 15213 > 412-268-7803 > @johnkitchin > http://kitchingroup.cheme.cmu.edu ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Extraneous output from Python code blocks using :session option 2015-03-12 15:34 ` Richard Stanton @ 2015-03-12 21:23 ` Kyle Meyer 2015-03-13 6:58 ` Kyle Meyer 0 siblings, 1 reply; 12+ messages in thread From: Kyle Meyer @ 2015-03-12 21:23 UTC (permalink / raw) To: Richard Stanton; +Cc: emacs-orgmode, John Kitchin Richard Stanton <stanton@haas.berkeley.edu> wrote: > This looks great. While we're patching this code, why does having > blank lines inside function definitions cause such problems in > :session mode? I think this is because the lines are being sent one by one, so sending the blank line causes an issue, like typing enter with the cursor at _: >>> def abc(): >>> _ ... File "<stdin>", line 2 ^ IndentationError: expected an indented block >>> I'll take a look at fixing this. python.el in Emacs 24 seems to send functions with blank lines over fine, so perhaps I can use it as an example. -- Kyle ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Extraneous output from Python code blocks using :session option 2015-03-12 21:23 ` Kyle Meyer @ 2015-03-13 6:58 ` Kyle Meyer 2015-03-13 8:21 ` Nicolas Goaziou 0 siblings, 1 reply; 12+ messages in thread From: Kyle Meyer @ 2015-03-13 6:58 UTC (permalink / raw) To: Richard Stanton; +Cc: emacs-orgmode, John Kitchin [-- Attachment #1: Type: text/plain, Size: 1049 bytes --] Kyle Meyer <kyle@kyleam.com> wrote: > Richard Stanton <stanton@haas.berkeley.edu> wrote: >> This looks great. While we're patching this code, why does having >> blank lines inside function definitions cause such problems in >> :session mode? > > I think this is because the lines are being sent one by one, so sending > the blank line causes an issue, like typing enter with the cursor at _: > > >>> def abc(): > >>> _ > > ... > File "<stdin>", line 2 > > ^ > IndentationError: expected an indented block > >>> > > I'll take a look at fixing this. python.el in Emacs 24 seems to send > functions with blank lines over fine, so perhaps I can use it as an > example. I've attached two patches. The first one is a proposed way to deal with the indentation issues in sessions. It is very similar to what python.el does for multiline input (use a temporary file and then execute that from the shell). The second is an update of my previous patch to remove shell prompt characters that are leaking into the output. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 0001-ob-python.el-Allow-indented-code-in-sessions.patch --] [-- Type: text/x-diff, Size: 2950 bytes --] From 28f271314f7bff7c54696defe8e451da69cd2d6c Mon Sep 17 00:00:00 2001 From: Kyle Meyer <kyle@kyleam.com> Date: Fri, 13 Mar 2015 02:45:04 -0400 Subject: [PATCH 1/2] ob-python.el: Allow indented code in sessions * lisp/ob-python.el (org-babel-python-evaluate-session): Recognize indented code in session and treat it differently to avoid syntax errors. For session blocks with results set to 'output', go through an intermediate file when there is indented code to prevent indentation-related errors when sending code to the Python shell. For session blocks with results set to 'value', issue an user-error. These (usually) would have resulted in a syntax error in the shell anyway, and the '_' variable can't be used to get the last value here (as it is for non-indented code) because it isn't set when executing the code through a file. --- lisp/ob-python.el | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/lisp/ob-python.el b/lisp/ob-python.el index dd3cc66..8c51679 100644 --- a/lisp/ob-python.el +++ b/lisp/ob-python.el @@ -238,6 +238,14 @@ (defconst org-babel-python-pp-wrapper-method open('%s', 'w').write( pprint.pformat(main()) )") +(defconst org-babel-python-indented-lines-session-method + (concat "fname= '%s'; fh = open(fname); " + "exec(compile(fh.read(), fname, 'exec')); " + "fh.close()") + "Single line to execute indented Python code in session. +%s will be formatted with the file name of the file containing + the code.") + (defun org-babel-python-evaluate (session body &optional result-type result-params preamble) "Evaluate BODY as Python code." @@ -303,6 +311,13 @@ (defun org-babel-python-evaluate-session (mapc (lambda (line) (insert line) (funcall send-wait)) (split-string body "[\r\n]")) (funcall send-wait))) + (indented-p (org-babel-python--indented-p body)) + (body (if indented-p + (let ((tmp-file (org-babel-temp-file "python-"))) + (with-temp-file tmp-file (insert body)) + (format org-babel-python-indented-lines-session-method + tmp-file)) + body)) (results (case result-type (output @@ -317,6 +332,8 @@ (defun org-babel-python-evaluate-session (funcall send-wait)) 2) "\n")) (value + (when indented-p + (user-error "Value output limited to unindented lines with session")) (let ((tmp-file (org-babel-temp-file "python-"))) (org-babel-comint-with-output (session org-babel-python-eoe-indicator nil body) @@ -339,6 +356,13 @@ (defun org-babel-python-read-string (string) (match-string 1 string) string)) +(defun org-babel-python--indented-p (input) + "Return true if any line in INPUT is indented." + (with-temp-buffer + (insert input) + (goto-char (point-min)) + (re-search-forward "^\\s-" nil t))) + (provide 'ob-python) -- 2.3.1 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #3: 0002-ob-python.el-Strip-leading-session-characters.patch --] [-- Type: text/x-diff, Size: 2314 bytes --] From 3d433fac1162f1544fbcfc7e4a8675a258764f34 Mon Sep 17 00:00:00 2001 From: Kyle Meyer <kyle@kyleam.com> Date: Fri, 13 Mar 2015 02:46:38 -0400 Subject: [PATCH 2/2] ob-python.el: Strip leading session characters * lisp/ob-python.el (org-babel-python-evaluate-session): Strip extra leading "..." and ">>>" from session output. --- lisp/ob-python.el | 34 ++++++++++++++++++++++++---------- 1 file changed, 24 insertions(+), 10 deletions(-) diff --git a/lisp/ob-python.el b/lisp/ob-python.el index 8c51679..170b6cd 100644 --- a/lisp/ob-python.el +++ b/lisp/ob-python.el @@ -321,16 +321,17 @@ (defun org-babel-python-evaluate-session (results (case result-type (output - (mapconcat - #'org-babel-trim - (butlast - (org-babel-comint-with-output - (session org-babel-python-eoe-indicator t body) - (funcall input-body body) - (funcall send-wait) (funcall send-wait) - (insert org-babel-python-eoe-indicator) - (funcall send-wait)) - 2) "\n")) + (org-babel-python--strip-session-chars + (mapconcat + #'org-babel-chomp + (butlast + (org-babel-comint-with-output + (session org-babel-python-eoe-indicator t body) + (funcall input-body body) + (funcall send-wait) (funcall send-wait) + (insert org-babel-python-eoe-indicator) + (funcall send-wait)) + 2) "\n"))) (value (when indented-p (user-error "Value output limited to unindented lines with session")) @@ -363,6 +364,19 @@ (defun org-babel-python--indented-p (input) (goto-char (point-min)) (re-search-forward "^\\s-" nil t))) +(defun org-babel-python--strip-session-chars (string) + "Remove leading '>>>' and '...' from Python session output. +`org-babel-comint-with-output' splits by +`comint-prompt-regexp' (which includes '>>>' and '...'), but, in +some situations, these still make it through at the start of the +output string." + (with-temp-buffer + (insert string) + (goto-char (point-min)) + (when (looking-at "\\s-*\n\\(\\(>>> \\)\\|\\(\\.\\.\\. \\)\\)*") + (delete-region (match-beginning 0) (match-end 0))) + (buffer-string))) + (provide 'ob-python) -- 2.3.1 [-- Attachment #4: Type: text/plain, Size: 9 bytes --] -- Kyle ^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: Extraneous output from Python code blocks using :session option 2015-03-13 6:58 ` Kyle Meyer @ 2015-03-13 8:21 ` Nicolas Goaziou 2015-03-13 14:40 ` Kyle Meyer 0 siblings, 1 reply; 12+ messages in thread From: Nicolas Goaziou @ 2015-03-13 8:21 UTC (permalink / raw) To: Kyle Meyer; +Cc: Richard Stanton, emacs-orgmode, John Kitchin Hello, Kyle Meyer <kyle@kyleam.com> writes: > I've attached two patches. The first one is a proposed way to deal with > the indentation issues in sessions. It is very similar to what > python.el does for multiline input (use a temporary file and then > execute that from the shell). The second is an update of my previous > patch to remove shell prompt characters that are leaking into the > output. Thank you. I'll just comment about code, not functionality. > +(defconst org-babel-python-indented-lines-session-method > + (concat "fname= '%s'; fh = open(fname); " > + "exec(compile(fh.read(), fname, 'exec')); " > + "fh.close()") > + "Single line to execute indented Python code in session. > +%s will be formatted with the file name of the file containing > + the code.") ^^^ spurious space > +(defun org-babel-python--indented-p (input) > + "Return true if any line in INPUT is indented." Non-nil if ... > + (with-temp-buffer > + (insert input) > + (goto-char (point-min)) > + (re-search-forward "^\\s-" nil t))) aka (org-string-match-p "^[ \t]" input) > +(defun org-babel-python--strip-session-chars (string) > + "Remove leading '>>>' and '...' from Python session output. > +`org-babel-comint-with-output' splits by > +`comint-prompt-regexp' (which includes '>>>' and '...'), but, in > +some situations, these still make it through at the start of the > +output string." Argument STRING is not explained in the docstring. > + (with-temp-buffer > + (insert string) > + (goto-char (point-min)) > + (when (looking-at "\\s-*\n\\(\\(>>> \\)\\|\\(\\.\\.\\. \\)\\)*") > + (delete-region (match-beginning 0) (match-end 0))) > + (buffer-string))) aka (replace-regexp-in-string "\\`\\s-*\n\\(>>> \\|\\.\\.\\. \\)*" "" string) Regards, -- Nicolas Goaziou ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Extraneous output from Python code blocks using :session option 2015-03-13 8:21 ` Nicolas Goaziou @ 2015-03-13 14:40 ` Kyle Meyer 2015-03-14 8:26 ` Nicolas Goaziou 0 siblings, 1 reply; 12+ messages in thread From: Kyle Meyer @ 2015-03-13 14:40 UTC (permalink / raw) To: Richard Stanton; +Cc: emacs-orgmode, John Kitchin [-- Attachment #1: Type: text/plain, Size: 593 bytes --] Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote: > Hello, > > Kyle Meyer <kyle@kyleam.com> writes: > >> I've attached two patches. The first one is a proposed way to deal with >> the indentation issues in sessions. It is very similar to what >> python.el does for multiline input (use a temporary file and then >> execute that from the shell). The second is an update of my previous >> patch to remove shell prompt characters that are leaking into the >> output. > > Thank you. > > I'll just comment about code, not functionality. Thanks for your comments. I've attached updated patches. [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #2: 0001-ob-python.el-Allow-indented-code-in-sessions.patch --] [-- Type: text/x-diff, Size: 2593 bytes --] From 41aae465038fd789fe92611ecf1fae7e9d08ab5b Mon Sep 17 00:00:00 2001 From: Kyle Meyer <kyle@kyleam.com> Date: Fri, 13 Mar 2015 02:45:04 -0400 Subject: [PATCH 1/2] ob-python.el: Allow indented code in sessions * lisp/ob-python.el (org-babel-python-evaluate-session): Recognize indented code in session and treat it differently to avoid syntax errors. For session blocks with results set to 'output', go through an intermediate file when there is indented code to prevent indentation-related errors when sending code to the Python shell. For session blocks with results set to 'value', issue an user-error. These (usually) would have resulted in a syntax error in the shell anyway, and the '_' variable can't be used to get the last value here (as it is for non-indented code) because it isn't set when executing the code through a file. --- lisp/ob-python.el | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/lisp/ob-python.el b/lisp/ob-python.el index dd3cc66..0dc74ca 100644 --- a/lisp/ob-python.el +++ b/lisp/ob-python.el @@ -238,6 +238,14 @@ (defconst org-babel-python-pp-wrapper-method open('%s', 'w').write( pprint.pformat(main()) )") +(defconst org-babel-python-indented-lines-session-method + (concat "fname= '%s'; fh = open(fname); " + "exec(compile(fh.read(), fname, 'exec')); " + "fh.close()") + "Single line to execute indented Python code in session. +%s will be formatted with the file name of the file containing +the code.") + (defun org-babel-python-evaluate (session body &optional result-type result-params preamble) "Evaluate BODY as Python code." @@ -303,6 +311,13 @@ (defun org-babel-python-evaluate-session (mapc (lambda (line) (insert line) (funcall send-wait)) (split-string body "[\r\n]")) (funcall send-wait))) + (indented-p (org-string-match-p "^[ \t]" body)) + (body (if indented-p + (let ((tmp-file (org-babel-temp-file "python-"))) + (with-temp-file tmp-file (insert body)) + (format org-babel-python-indented-lines-session-method + tmp-file)) + body)) (results (case result-type (output @@ -317,6 +332,8 @@ (defun org-babel-python-evaluate-session (funcall send-wait)) 2) "\n")) (value + (when indented-p + (user-error "Value output limited to unindented lines with session")) (let ((tmp-file (org-babel-temp-file "python-"))) (org-babel-comint-with-output (session org-babel-python-eoe-indicator nil body) -- 2.3.1 [-- Warning: decoded text below may be mangled, UTF-8 assumed --] [-- Attachment #3: 0002-ob-python.el-Strip-leading-session-characters.patch --] [-- Type: text/x-diff, Size: 2161 bytes --] From 28b6a003e4ae1dbb1474350203444e76df9e8572 Mon Sep 17 00:00:00 2001 From: Kyle Meyer <kyle@kyleam.com> Date: Fri, 13 Mar 2015 02:46:38 -0400 Subject: [PATCH 2/2] ob-python.el: Strip leading session characters * lisp/ob-python.el (org-babel-python-evaluate-session): Strip extra leading "..." and ">>>" from session output. --- lisp/ob-python.el | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-) diff --git a/lisp/ob-python.el b/lisp/ob-python.el index 0dc74ca..e976d9c 100644 --- a/lisp/ob-python.el +++ b/lisp/ob-python.el @@ -321,16 +321,17 @@ (defun org-babel-python-evaluate-session (results (case result-type (output - (mapconcat - #'org-babel-trim - (butlast - (org-babel-comint-with-output - (session org-babel-python-eoe-indicator t body) - (funcall input-body body) - (funcall send-wait) (funcall send-wait) - (insert org-babel-python-eoe-indicator) - (funcall send-wait)) - 2) "\n")) + (org-babel-python--strip-session-chars + (mapconcat + #'org-babel-chomp + (butlast + (org-babel-comint-with-output + (session org-babel-python-eoe-indicator t body) + (funcall input-body body) + (funcall send-wait) (funcall send-wait) + (insert org-babel-python-eoe-indicator) + (funcall send-wait)) + 2) "\n"))) (value (when indented-p (user-error "Value output limited to unindented lines with session")) @@ -356,6 +357,15 @@ (defun org-babel-python-read-string (string) (match-string 1 string) string)) +(defun org-babel-python--strip-session-chars (string) + "Remove leading '>>>' and '...' from STRING. +`org-babel-comint-with-output' splits the Python session output +by `comint-prompt-regexp' (which includes '>>>' and '...'), but, +in some situations, these still make it through at the start of +the output string." + (replace-regexp-in-string "\\`\\s-*\n\\(>>> \\|\\.\\.\\. \\)*" "" + string)) + (provide 'ob-python) -- 2.3.1 [-- Attachment #4: Type: text/plain, Size: 9 bytes --] -- Kyle ^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: Extraneous output from Python code blocks using :session option 2015-03-13 14:40 ` Kyle Meyer @ 2015-03-14 8:26 ` Nicolas Goaziou 2015-03-16 0:40 ` Kyle Meyer 0 siblings, 1 reply; 12+ messages in thread From: Nicolas Goaziou @ 2015-03-14 8:26 UTC (permalink / raw) To: Kyle Meyer; +Cc: Richard Stanton, emacs-orgmode, John Kitchin Kyle Meyer <kyle@kyleam.com> writes: > Thanks for your comments. I've attached updated patches. Thanks. > Subject: [PATCH 1/2] ob-python.el: Allow indented code in sessions > > * lisp/ob-python.el (org-babel-python-evaluate-session): Recognize > indented code in session and treat it differently to avoid syntax > errors. You also need to list new functions and variables in commit message, e.g., (org-babel-python-indented-lines-session-method): New variable. > * lisp/ob-python.el (org-babel-python-evaluate-session): Strip extra leading > "..." and ">>>" from session output. Ditto. (org-babel-python--strip-session-chars): New function. > +(defun org-babel-python--strip-session-chars (string) > + "Remove leading '>>>' and '...' from STRING. > +`org-babel-comint-with-output' splits the Python session output > +by `comint-prompt-regexp' (which includes '>>>' and '...'), but, > +in some situations, these still make it through at the start of > +the output string." Wouldn't it be better to find the situations above at the source? If may be due to some indentation problem at the beginning of the string, in which case an `org-trim' within `org-babel-comint-with-output' could solve it. WDYT? Regards, ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Extraneous output from Python code blocks using :session option 2015-03-14 8:26 ` Nicolas Goaziou @ 2015-03-16 0:40 ` Kyle Meyer 0 siblings, 0 replies; 12+ messages in thread From: Kyle Meyer @ 2015-03-16 0:40 UTC (permalink / raw) To: Richard Stanton; +Cc: emacs-orgmode, John Kitchin Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote: [...] > You also need to list new functions and variables in commit message, > e.g., Thanks. I'll update the commit messages. > Wouldn't it be better to find the situations above at the source? If may > be due to some indentation problem at the beginning of the string, in > which case an `org-trim' within `org-babel-comint-with-output' could > solve it. I agree, but I haven't figured out a good way to do this. As far as I can tell, it seems like the input sent in the right format. Instead, it is empty prompt markers that are causing the issue (see examples at end for details). I think there are three separate problems with Python session output right now. 1. The empty prompt markers from multi-line code mess up the output processing and leak into the final output. The current patch fixes this in the final combined output, but I've realized it doesn't cover all the cases. (I initially thought the markers only leaked into the start of the output, but they can be in the middle as well.) Plus, as you mention, it seems like it'd be better to fix this more upstream. Since the input seems to be sent in correctly, I think this leaves org-babel-comint-with-output. Specifically, output lines that consist of just a prompt (like '>>> ' or '... ') could be filtered out before the output lines are joined and returned. comint-prompt-regexp could be used for this check. One problem with this is that I don't know if this will have unwanted effects on the output for other languages that use org-babel-comint-with-output. 2. org-babel-python-evaluate-session will eat whitespace at the beginning of output. The leading spaces below would be lost in the results. #+begin_src python :session :results output print(' <-- leading spaces') #+end_src Changing org-babel-trim to org-babel-chomp in org-babel-python-evaluate-session fixes this. 3. Valid code that has indentation and spaces cannot be sent. I think this is just a limitation of the shell, since blank lines are used to signal the end of blocks. One of the patches I sent deals with this like python.el does (by using an intermediate file). One issue with that patch is whether it should check for blank lines *and* indentation, because right now it only checks for indentation, meaning that some code is unnecessarily sent through an intermediate file. Below are two examples that explain #1 in more detail. * Single line #+begin_src python :session :results output 3 #+end_src org-babel-python-evaluate-session splits the input by newlines, resulting in 3 (no newline) being inserted. It then sends this input, followed by a few more empty inputs (not needed here, but required to finish the block if the statement had indentation), and then inserts and sends the babel eoe indicator. This ends up in the Python shell as >>> 3 3 >>> >>> >>> >>> 'org_babel_python_eoe' 'org_babel_python_eoe' >>> org-babel-comint-with-output receives then a series of text output, "", "3\n>>> ", "", ">>> ", "", ">>> ", "", ">>> ", "", "'org_babel_python_eoe'\n>>> " which it concatenates to 3 >>> >>> >>> >>> 'org_babel_python_eoe' >>> It then splits this by comint-prompt-regexp, which includes '>>>' at the start of a line, resulting in ("3\n" ">>> >>> >>> 'org_babel_python_eoe'\n" "") org-babel-python-evaluate-session receives this, removes the last two elements, and trims the first, giving the correct output: #+RESULTS: : 3 * Multiple lines, no indentation Here is the same example, but with some problematic lines added to the beginning. #+begin_src python :session :results output x = 3 y = 4 x + y #+end_src org-babel-python-evaluate-session splits the input by newlines and sends x = 3, then x = 4, and then 3, as well as the empty lines and eoe signal. This gets ends up in the Python shell as x = 3 >>> y = 4 >>> x + y 7 >>> >>> >>> >>> 'org_babel_python_eoe' 'org_babel_python_eoe' >>> org-babel-comint-with-output receives the a series of text output, "", ">>> ", <- From 'x = 3' "", ">>> ", <- From 'y = 4' "", "7\n>>> ", "", ">>> ", "", ">>> ", "", ">>> ", "" "'org_babel_python_eoe'\n>>> " which it concatenates to >>> >>> 7 >>> >>> >>> >>> 'org_babel_python_eoe' >>> It then splits this by comint-prompt-regexp, which includes '>>>' at the start of a line, resulting in ("" ">>> 7\n" ">>> >>> >>> 'org_babel_python_eoe'\n" "") org-babel-python-evaluate-session receives this, removes the last two elements, and trims the others. Unlike the output above, this has an empty line at the beginning due to the additional ">>> " that resulted from "x = 3", as well as ">>> " before 7 due to the additional ">>> " that resulted from "y = 4". #+RESULTS: : : >>> 7 -- Kyle ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Extraneous output from Python code blocks using :session option 2015-03-12 15:17 ` John Kitchin 2015-03-12 15:34 ` Richard Stanton @ 2015-03-12 21:21 ` Kyle Meyer 1 sibling, 0 replies; 12+ messages in thread From: Kyle Meyer @ 2015-03-12 21:21 UTC (permalink / raw) To: John Kitchin; +Cc: Richard Stanton, emacs-orgmode John Kitchin <jkitchin@andrew.cmu.edu> wrote: > Nice. I hope this makes it into org-mode one day. > > In the mean time, I crafted an approach with another hook function that > is described here: > > http://kitchingroup.cheme.cmu.edu/blog/2015/03/12/Making-org-mode-Python-sessions-look-better/ Good idea. Thanks for the information. -- Kyle ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2015-03-16 0:40 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-03-11 3:38 Extraneous output from Python code blocks using :session option Richard Stanton 2015-03-11 18:29 ` Kyle Meyer 2015-03-11 18:54 ` Richard Stanton 2015-03-12 15:17 ` John Kitchin 2015-03-12 15:34 ` Richard Stanton 2015-03-12 21:23 ` Kyle Meyer 2015-03-13 6:58 ` Kyle Meyer 2015-03-13 8:21 ` Nicolas Goaziou 2015-03-13 14:40 ` Kyle Meyer 2015-03-14 8:26 ` Nicolas Goaziou 2015-03-16 0:40 ` Kyle Meyer 2015-03-12 21:21 ` Kyle Meyer
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).