emacs-orgmode@gnu.org archives
 help / color / mirror / code / Atom feed
* [BUG] ob-python hangs on second start [9.5.4 (release_9.5.4-763-g06373a @ ~/emacs/org-mode/lisp/)]
@ 2023-08-25  4:44 Peter Mao
  2023-08-25  8:52 ` Ihor Radchenko
  0 siblings, 1 reply; 6+ messages in thread
From: Peter Mao @ 2023-08-25  4:44 UTC (permalink / raw)
  To: org-mode-email

[-- Attachment #1: Type: text/plain, Size: 5108 bytes --]

Expectation:  When running ob-python code blocks, I should be able to
kill the python session in the *Python* buffer and run another code
block (or the same one).

Problem:  ob-python works fine on the first execution, but after
`exit()`ing the python session, it hangs without executing the code.
After a `C-g`, the prompt in the *Python* session shows up, but one then
has to re-execute the code block, as none of it has run.

The following three code blocks demonstrate the problem.  I have
bisected the problem, finding 06373a64 as the first bad commit.  I tried
to reproduce the problem on Emacs 28.2, but this commit of Org won't
compile on Emacs 28.2 for me.

#+begin_src elisp :results none
  (org-babel-do-load-languages 'org-babel-load-languages '((python . t)))
  (setq-local org-confirm-babel-evaluate nil)
#+end_src

#+begin_src python :results output
  print("Running first, this one issues a warning but competes
successfully.")
  exit()
#+end_src

#+begin_src python :results output
  print("Running second, this one hangs until user enters C-g.")
  print("None of the code executes.")
  exit()
#+end_src



Emacs  : GNU Emacs 29.1 (build 2, x86_64-pc-linux-gnu, GTK+ Version
3.24.33, cairo version 1.16.0)
 of 2023-07-31
Package: Org mode version 9.5.4 (release_9.5.4-763-g06373a @
/home/peterm/emacs/org-mode/lisp/)

current state:
==============
(setq
 org-link-elisp-confirm-function 'yes-or-no-p
 org-bibtex-headline-format-function #[257 "\300 \236A\207" [:title] 3
"\n\n(fn ENTRY)"]
 org-persist-after-read-hook '(org-element--cache-persist-after-read)
 org-export-before-parsing-hook '(org-attach-expand-links)
 org-cycle-tab-first-hook '(org-babel-hide-result-toggle-maybe
   org-babel-header-arg-expand)
 org-archive-hook '(org-attach-archive-delete-maybe)
 org-cycle-hook '(org-cycle-hide-archived-subtrees
org-cycle-show-empty-lines
 org-cycle-optimize-window-after-visibility-change)
 org-persist-before-read-hook '(org-element--cache-persist-before-read)
 org-mode-hook '(#[0 "\300\301\302\303\304$\207"
  [add-hook change-major-mode-hook org-fold-show-all append
   local]
  5]
#[0 "\300\301\302\303\304$\207"
  [add-hook change-major-mode-hook org-babel-show-result-all
   append local]
  5]
org-babel-result-hide-spec org-babel-hide-all-hashes)
 org-babel-load-languages '((python . t))
 org-confirm-shell-link-function 'yes-or-no-p
 outline-isearch-open-invisible-function 'outline-isearch-open-invisible
 org-agenda-before-write-hook '(org-agenda-add-entry-text)
 org-babel-tangle-lang-exts '(("python" . "py") ("emacs-lisp" . "el")
     ("elisp" . "el"))
 org-src-mode-hook '(org-src-babel-configure-edit-buffer
    org-src-mode-configure-edit-buffer)
 org-confirm-elisp-link-function 'yes-or-no-p
 org-speed-command-hook '(org-speed-command-activate
 org-babel-speed-command-activate)
 org-fold-core-isearch-open-function 'org-fold--isearch-reveal
 org-persist-before-write-hook '(org-element--cache-persist-before-write)
 org-tab-first-hook '(org-babel-hide-result-toggle-maybe
     org-babel-header-arg-expand)
 org-link-shell-confirm-function 'yes-or-no-p
 org-babel-pre-tangle-hook '(save-buffer)
 org-agenda-loop-over-headlines-in-active-region nil
 org-occur-hook '(org-first-headline-recenter)
 org-metadown-hook '(org-babel-pop-to-session-maybe)
 org-link-parameters '(("attachment" :follow org-attach-follow :complete
org-attach-complete-link)
      ("id" :follow org-id-open)
      ("eww" :follow org-eww-open :store org-eww-store-link)
      ("rmail" :follow org-rmail-open :store
org-rmail-store-link)
      ("mhe" :follow org-mhe-open :store org-mhe-store-link)
      ("irc" :follow org-irc-visit :store org-irc-store-link
:export org-irc-export)
      ("info" :follow org-info-open :export org-info-export
:store org-info-store-link :insert-description
org-info-description-as-command)
      ("gnus" :follow org-gnus-open :store
org-gnus-store-link)
      ("docview" :follow org-docview-open :export
org-docview-export :store org-docview-store-link)
      ("bibtex" :follow org-bibtex-open :store
org-bibtex-store-link)
      ("bbdb" :follow org-bbdb-open :export org-bbdb-export
:complete org-bbdb-complete-link :store
org-bbdb-store-link)
      ("w3m" :store org-w3m-store-link)
      ("doi" :follow org-link-doi-open :export
org-link-doi-export)
      ("file+sys") ("file+emacs")
      ("shell" :follow org-link--open-shell)
      ("news" :follow
#[514 "\301\300\302 Q \"\207"
 ["news" browse-url ":"] 6 "\n\n(fn URL ARG)"]
)
      ("mailto" :follow
#[514 "\301\300\302 Q \"\207"
 ["mailto" browse-url ":"] 6 "\n\n(fn URL ARG)"]
)
      ("https" :follow
#[514 "\301\300\302 Q \"\207"
 ["https" browse-url ":"] 6 "\n\n(fn URL ARG)"]
)
      ("http" :follow
#[514 "\301\300\302 Q \"\207"
 ["http" browse-url ":"] 6 "\n\n(fn URL ARG)"]
)
      ("ftp" :follow
#[514 "\301\300\302 Q \"\207" ["ftp" browse-url ":"]
 6 "\n\n(fn URL ARG)"]
)
      ("help" :follow org-link--open-help :store
org-link--store-help)
      ("file" :complete org-link-complete-file)
      ("elisp" :follow org-link--open-elisp))
 org-metaup-hook '(org-babel-load-in-session-maybe)
 )

[-- Attachment #2: Type: text/html, Size: 6557 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [BUG] ob-python hangs on second start [9.5.4 (release_9.5.4-763-g06373a @ ~/emacs/org-mode/lisp/)]
  2023-08-25  4:44 [BUG] ob-python hangs on second start [9.5.4 (release_9.5.4-763-g06373a @ ~/emacs/org-mode/lisp/)] Peter Mao
@ 2023-08-25  8:52 ` Ihor Radchenko
  2023-08-27 21:26   ` Jack Kamm
  0 siblings, 1 reply; 6+ messages in thread
From: Ihor Radchenko @ 2023-08-25  8:52 UTC (permalink / raw)
  To: Peter Mao, Jack Kamm; +Cc: org-mode-email

Peter Mao <peter.mao@gmail.com> writes:

> Expectation:  When running ob-python code blocks, I should be able to
> kill the python session in the *Python* buffer and run another code
> block (or the same one).
>
> Problem:  ob-python works fine on the first execution, but after
> `exit()`ing the python session, it hangs without executing the code.
> After a `C-g`, the prompt in the *Python* session shows up, but one then
> has to re-execute the code block, as none of it has run.

Confirmed.

It looks like `python-shell-first-prompt-hook' does not get triggered
in the described scenario with exit() and we enter infinite loop in the
code below.

    (add-hook
     'python-shell-first-prompt-hook
     #'org-babel-python--setup-session
     nil 'local)))
;; Wait until Python initializes
;; This is more reliable compared to
;; `org-babel-comint-wait-for-output' as python may emit
;; multiple prompts during initialization.
(with-current-buffer py-buffer
  (while (not org-babel-python--initialized)
    (org-babel-comint-wait-for-output py-buffer)))

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [BUG] ob-python hangs on second start [9.5.4 (release_9.5.4-763-g06373a @ ~/emacs/org-mode/lisp/)]
  2023-08-25  8:52 ` Ihor Radchenko
@ 2023-08-27 21:26   ` Jack Kamm
  2023-08-28  0:29     ` Peter Mao
  2023-08-28  9:02     ` Ihor Radchenko
  0 siblings, 2 replies; 6+ messages in thread
From: Jack Kamm @ 2023-08-27 21:26 UTC (permalink / raw)
  To: Ihor Radchenko, Peter Mao; +Cc: org-mode-email

[-- Attachment #1: Type: text/plain, Size: 1087 bytes --]

Ihor Radchenko <yantar92@posteo.net> writes:

> Peter Mao <peter.mao@gmail.com> writes:
>
>> Expectation:  When running ob-python code blocks, I should be able to
>> kill the python session in the *Python* buffer and run another code
>> block (or the same one).
>>
>> Problem:  ob-python works fine on the first execution, but after
>> `exit()`ing the python session, it hangs without executing the code.
>> After a `C-g`, the prompt in the *Python* session shows up, but one then
>> has to re-execute the code block, as none of it has run.
>
> Confirmed.
>
> It looks like `python-shell-first-prompt-hook' does not get triggered
> in the described scenario with exit() and we enter infinite loop in the
> code below.

I think `python-shell-first-prompt-hook' is actually still triggered,
but rather it is `org-babel-comint-wait-for-output' that is hanging
while waiting for startup. However, we can just replace it with a
`sleep-for' instead, because we now wait for
`org-babel-python--initialized' anyways.

See the attached patch. Please let me know if it fixes the problem for
you.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-python-Fix-hanging-on-second-start.patch --]
[-- Type: text/x-patch, Size: 2499 bytes --]

From d5721aa37a399afcd527906e5d9f1b6bce37fdb9 Mon Sep 17 00:00:00 2001
From: Jack Kamm <jackkamm@gmail.com>
Date: Sun, 27 Aug 2023 14:13:15 -0700
Subject: [PATCH] ob-python: Fix hanging on second start

See: https://list.orgmode.org/87ttsnh5bx.fsf@localhost/T/#t

* lisp/ob-python.el (org-babel-python-initiate-session-by-key): Switch
from `org-babel-comint-wait-for-output' to `sleep-for' while waiting
for `org-babel-python--initialized', to prevent hanging on restarted
Python process.
* testing/lisp/test-ob-python.el (test-ob-python/session-restart): New
test for restarting ob-python session process.
---
 lisp/ob-python.el              |  2 +-
 testing/lisp/test-ob-python.el | 19 +++++++++++++++++++
 2 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index b9dab91b4..6b216ce89 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -272,7 +272,7 @@ (defun org-babel-python-initiate-session-by-key (&optional session)
       ;; multiple prompts during initialization.
       (with-current-buffer py-buffer
         (while (not org-babel-python--initialized)
-          (org-babel-comint-wait-for-output py-buffer)))
+          (sleep-for 0 10)))
       (setq org-babel-python-buffers
 	    (cons (cons session py-buffer)
 		  (assq-delete-all session org-babel-python-buffers)))
diff --git a/testing/lisp/test-ob-python.el b/testing/lisp/test-ob-python.el
index 82fbca36e..c11e1d0c2 100644
--- a/testing/lisp/test-ob-python.el
+++ b/testing/lisp/test-ob-python.el
@@ -310,6 +310,25 @@ (ert-deftest test-ob-python/async-local-python-shell ()
 #+end_src"
     (should (org-babel-execute-src-block))))
 
+(ert-deftest test-ob-python/session-restart ()
+  ;; Disable the test on older Emacs as built-in python.el sometimes
+  ;; fail to initialize session.
+  (skip-unless (version<= "28" emacs-version))
+  (should
+   (equal "success"
+          (progn
+            (org-test-with-temp-text "#+begin_src python :session :results output
+print('start')
+#+end_src"
+	                             (org-babel-execute-src-block))
+            (let ((proc (python-shell-get-process)))
+              (python-shell-send-string "exit()")
+              (while (accept-process-output proc)))
+            (org-test-with-temp-text "#+begin_src python :session :results output
+print('success')
+#+end_src"
+	                             (org-babel-execute-src-block))))))
+
 (provide 'test-ob-python)
 
 ;;; test-ob-python.el ends here
-- 
2.41.0


^ permalink raw reply related	[flat|nested] 6+ messages in thread

* Re: [BUG] ob-python hangs on second start [9.5.4 (release_9.5.4-763-g06373a @ ~/emacs/org-mode/lisp/)]
  2023-08-27 21:26   ` Jack Kamm
@ 2023-08-28  0:29     ` Peter Mao
  2023-08-28  9:02     ` Ihor Radchenko
  1 sibling, 0 replies; 6+ messages in thread
From: Peter Mao @ 2023-08-28  0:29 UTC (permalink / raw)
  To: Jack Kamm; +Cc: Ihor Radchenko, org-mode-email

[-- Attachment #1: Type: text/plain, Size: 867 bytes --]

On Sun, Aug 27, 2023 at 2:26 PM Jack Kamm <jackkamm@gmail.com> wrote:

> Ihor Radchenko <yantar92@posteo.net> writes:
>
> > Confirmed.
> >
> > It looks like `python-shell-first-prompt-hook' does not get triggered
> > in the described scenario with exit() and we enter infinite loop in the
> > code below.
>
> I think `python-shell-first-prompt-hook' is actually still triggered,
> but rather it is `org-babel-comint-wait-for-output' that is hanging
> while waiting for startup. However, we can just replace it with a
> `sleep-for' instead, because we now wait for
> `org-babel-python--initialized' anyways.
>
> See the attached patch. Please let me know if it fixes the problem for
> you.
>

Yes, this fixes the problem for me.  I made the change locally to
release_9.6.6 on emacs 29.1.  I have the expected behavior back. thank you!

Peter

[-- Attachment #2: Type: text/html, Size: 1283 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [BUG] ob-python hangs on second start [9.5.4 (release_9.5.4-763-g06373a @ ~/emacs/org-mode/lisp/)]
  2023-08-27 21:26   ` Jack Kamm
  2023-08-28  0:29     ` Peter Mao
@ 2023-08-28  9:02     ` Ihor Radchenko
  2023-08-31  4:52       ` Jack Kamm
  1 sibling, 1 reply; 6+ messages in thread
From: Ihor Radchenko @ 2023-08-28  9:02 UTC (permalink / raw)
  To: Jack Kamm; +Cc: Peter Mao, org-mode-email

Jack Kamm <jackkamm@gmail.com> writes:

> I think `python-shell-first-prompt-hook' is actually still triggered,
> but rather it is `org-babel-comint-wait-for-output' that is hanging
> while waiting for startup. However, we can just replace it with a
> `sleep-for' instead, because we now wait for
> `org-babel-python--initialized' anyways.
>
> See the attached patch. Please let me know if it fixes the problem for
> you.

LGTM! Feel free to push.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [BUG] ob-python hangs on second start [9.5.4 (release_9.5.4-763-g06373a @ ~/emacs/org-mode/lisp/)]
  2023-08-28  9:02     ` Ihor Radchenko
@ 2023-08-31  4:52       ` Jack Kamm
  0 siblings, 0 replies; 6+ messages in thread
From: Jack Kamm @ 2023-08-31  4:52 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: Peter Mao, org-mode-email

Ihor Radchenko <yantar92@posteo.net> writes:

> LGTM! Feel free to push.

Pushed to bugfix (c81dba2fb) and merged to main (b49275acb). Pushed the
unit test as a separate commit to main only (8000b1120).


^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2023-08-31  4:53 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-08-25  4:44 [BUG] ob-python hangs on second start [9.5.4 (release_9.5.4-763-g06373a @ ~/emacs/org-mode/lisp/)] Peter Mao
2023-08-25  8:52 ` Ihor Radchenko
2023-08-27 21:26   ` Jack Kamm
2023-08-28  0:29     ` Peter Mao
2023-08-28  9:02     ` Ihor Radchenko
2023-08-31  4:52       ` Jack Kamm

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).