bug#50514: 28.0.50; org inline call to python src block req C-g to break

bug#50514: 28.0.50; org inline call to python src block req C-g to break

[Ihor Radchenko]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> The bug on the Emacs side was fixed by Augusto's patch, I think, but
> there's a remaining problem in ob-python, so I'm reassigning this bug
> report to org-mode.

Could you please elaborate what the problem is?
There have been multiple issues in this thread and the one previously
reported to Org ML is solved on Emacs side.

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92

bug#50514: 28.0.50; org inline call to python src block req C-g to break

[Lars Ingebrigtsen]

Ihor Radchenko <yantar92@gmail.com> writes:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> The bug on the Emacs side was fixed by Augusto's patch, I think, but
>> there's a remaining problem in ob-python, so I'm reassigning this bug
>> report to org-mode.
>
> Could you please elaborate what the problem is?
> There have been multiple issues in this thread and the one previously
> reported to Org ML is solved on Emacs side.

It's this bit:

> So I found if I extend the sleep-for in ob-python from 10 ms to 90 ms,
> my initialization problem is “fixed”.

The sleep is still 10ms in the in-tree ob-python, at least.

bug#50514: 28.0.50; org inline call to python src block req C-g to break

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 859 bytes --]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>> Could you please elaborate what the problem is?
>> There have been multiple issues in this thread and the one previously
>> reported to Org ML is solved on Emacs side.
>
> It's this bit:
>
>> So I found if I extend the sleep-for in ob-python from 10 ms to 90 ms,
>> my initialization problem is “fixed”.
>
> The sleep is still 10ms in the in-tree ob-python, at least.

10ms magic constant is indeed not great.
However, I am unable to reproduce the original bug on my system.
I tried to draft an alternative patch. See the attached.

Mark, would you mind testing the patch?

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92



[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ob-python-Fix-Emacs-bug-50514-when-Python-loads-slow.patch --]
[-- Type: text/x-patch, Size: 1820 bytes --]

From d5347c688d2fb8037116d82fa7987e9e6690ae66 Mon Sep 17 00:00:00 2001
Message-Id: <d5347c688d2fb8037116d82fa7987e9e6690ae66.1661609451.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Sat, 27 Aug 2022 22:06:11 +0800
Subject: [PATCH] ob-python: Fix Emacs bug#50514 when Python loads slowly

* lisp/ob-python.el (org-babel-python-initiate-session): Wait until
python process is initialized.
(org-babel-python-initiate-session-by-key): Do not rely on 10ms delay
to initialize Python.
---
 lisp/ob-python.el | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index 932aca08e..2ad907bd8 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -197,8 +197,7 @@ (defun org-babel-python-initiate-session-by-key (&optional session)
 	  (setq py-buffer (org-babel-python-with-earmuffs session)))
 	(let ((python-shell-buffer-name
 	       (org-babel-python-without-earmuffs py-buffer)))
-	  (run-python cmd)
-	  (sleep-for 0 10)))
+	  (run-python cmd)))
        ((and (eq 'python-mode org-babel-python-mode)
 	     (fboundp 'py-shell)) ; python-mode.el
 	(require 'python-mode)
@@ -225,8 +224,9 @@ (defun org-babel-python-initiate-session-by-key (&optional session)
 (defun org-babel-python-initiate-session (&optional session _params)
   "Create a session named SESSION according to PARAMS."
   (unless (string= session "none")
-    (org-babel-python-session-buffer
-     (org-babel-python-initiate-session-by-key session))))
+    (let ((session (org-babel-python-session-buffer
+                    (org-babel-python-initiate-session-by-key session)))))
+    (org-babel-comint-wait-for-output session)))
 
 (defvar org-babel-python-eoe-indicator "org_babel_python_eoe"
   "A string to indicate that evaluation has completed.")
-- 
2.35.1

bug#50514: 28.0.50; org inline call to python src block req C-g to break

[Barton, Mark]

[-- Attachment #1: Type: text/plain, Size: 1819 bytes --]



On Aug 27, 2022, at 7:11 AM, Ihor Radchenko <yantar92@gmail.com<mailto:yantar92@gmail.com>> wrote:

---
lisp/ob-python.el | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index 932aca08e..2ad907bd8 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -197,8 +197,7 @@ (defun org-babel-python-initiate-session-by-key (&optional session)
  (setq py-buffer (org-babel-python-with-earmuffs session)))
(let ((python-shell-buffer-name
       (org-babel-python-without-earmuffs py-buffer)))
-   (run-python cmd)
-   (sleep-for 0 10)))
+   (run-python cmd)))
       ((and (eq 'python-mode org-babel-python-mode)
     (fboundp 'py-shell)) ; python-mode.el
(require 'python-mode)
@@ -225,8 +224,9 @@ (defun org-babel-python-initiate-session-by-key (&optional session)
(defun org-babel-python-initiate-session (&optional session _params)
  "Create a session named SESSION according to PARAMS."
  (unless (string= session "none")
-    (org-babel-python-session-buffer
-     (org-babel-python-initiate-session-by-key session))))
+    (let ((session (org-babel-python-session-buffer
+                    (org-babel-python-initiate-session-by-key session)))))
+    (org-babel-comint-wait-for-output session)))

(defvar org-babel-python-eoe-indicator "org_babel_python_eoe"
  "A string to indicate that evaluation has completed.")
--
2.35.1

I get the following in the Messages:
org-babel-comint-wait-for-output: Buffer python-chain does not exist or has no process

Note that python-chain is the name of my session passed to the python blocks as shown in the following line within my org file.

#+PROPERTY: header-args:python :session python-chain :exports results :results raw file :file-ext table

Thanks,

Mark


[-- Attachment #2: Type: text/html, Size: 29021 bytes --]

bug#50514: 28.0.50; org inline call to python src block req C-g to break

[Mark Barton]

[-- Attachment #1: Type: text/plain, Size: 2386 bytes --]



> On Aug 27, 2022, at 11:59 AM, Barton, Mark <Mark.Barton@disney.com> wrote:
> 
>  
> 
>> On Aug 27, 2022, at 7:11 AM, Ihor Radchenko <yantar92@gmail.com> wrote:
>> 
>> ---
>> lisp/ob-python.el | 8 ++++----
>> 1 file changed, 4 insertions(+), 4 deletions(-)
>> 
>> diff --git a/lisp/ob-python.el b/lisp/ob-python.el
>> index 932aca08e..2ad907bd8 100644
>> --- a/lisp/ob-python.el
>> +++ b/lisp/ob-python.el
>> @@ -197,8 +197,7 @@ (defun org-babel-python-initiate-session-by-key (&optional session)
>> 	  (setq py-buffer (org-babel-python-with-earmuffs session)))
>> 	(let ((python-shell-buffer-name
>> 	       (org-babel-python-without-earmuffs py-buffer)))
>> -	  (run-python cmd)
>> -	  (sleep-for 0 10)))
>> +	  (run-python cmd)))
>>        ((and (eq 'python-mode org-babel-python-mode)
>> 	     (fboundp 'py-shell)) ; python-mode.el
>> 	(require 'python-mode)
>> @@ -225,8 +224,9 @@ (defun org-babel-python-initiate-session-by-key (&optional session)
>> (defun org-babel-python-initiate-session (&optional session _params)
>>   "Create a session named SESSION according to PARAMS."
>>   (unless (string= session "none")
>> -    (org-babel-python-session-buffer
>> -     (org-babel-python-initiate-session-by-key session))))
>> +    (let ((session (org-babel-python-session-buffer
>> +                    (org-babel-python-initiate-session-by-key session)))))
>> +    (org-babel-comint-wait-for-output session)))
>> 
>> (defvar org-babel-python-eoe-indicator "org_babel_python_eoe"
>>   "A string to indicate that evaluation has completed.")
>> -- 
>> 2.35.1
> 
> I get the following in the Messages:
> org-babel-comint-wait-for-output: Buffer python-chain does not exist or has no process
> 
> Note that python-chain is the name of my session passed to the python blocks as shown in the following line within my org file.
> 
> #+PROPERTY: header-args:python :session python-chain :exports results :results raw file :file-ext table
> 
> Thanks,
> 
> Mark
> 
I will work on getting more details this weekend. It looked like the buffer passed to org-babel-comint-wait-for-output was missing the "earmuffs". I tried adding the earmuffs with the org function for that but although I got passed that error, I messed up my python session and the following python blocks were not aware of the imports in the first block.

Mark

 

[-- Attachment #2: Type: text/html, Size: 29460 bytes --]

bug#50514: 28.0.50; org inline call to python src block req C-g to break

[Barton, Mark]

[-- Attachment #1: Type: text/plain, Size: 1416 bytes --]


I will work on getting more details this weekend. It looked like the buffer passed to org-babel-comint-wait-for-output was missing the "earmuffs". I tried adding the earmuffs with the org function for that but although I got passed that error, I messed up my python session and the following python blocks were not aware of the imports in the first block.

Mark



I tried something that was suggested last September 14th by Augusto Stoffel. I had not tried it because changing the sleep from 10 to 90 was my workaround.
One thing that might work is to block until the shell is ready with something like

    (run-python)
    (with-current-buffer *the-shell-buffer*
      (while (not python-shell--first-prompt-received)
        (accept-process-output (get-buffer-process (current-buffer)))))

The patch that is working for me now is:

modified   lisp/org/ob-python.el
@@ -198,7 +198,10 @@ org-babel-python-initiate-session-by-key
  (let ((python-shell-buffer-name
        (org-babel-python-without-earmuffs py-buffer)))
   (run-python cmd)
-  (sleep-for 0 10)))
+          (with-current-buffer py-buffer
+            (while (not python-shell--first-prompt-received)
+              (accept-process-output (get-buffer-process (current-buffer)))))
+          ))
        ((and (eq 'python-mode org-babel-python-mode)
      (fboundp 'py-shell)) ; python-mode.el
  (require 'python-mode)


[-- Attachment #2: Type: text/html, Size: 5009 bytes --]

bug#50514: 28.0.50; org inline call to python src block req C-g to break

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 967 bytes --]

"Barton, Mark" <Mark.Barton@disney.com> writes:

> I tried something that was suggested last September 14th by Augusto Stoffel. I had not tried it because changing the sleep from 10 to 90 was my workaround.
> One thing that might work is to block until the shell is ready with something like
>
>     (run-python)
>     (with-current-buffer *the-shell-buffer*
>       (while (not python-shell--first-prompt-received)
>         (accept-process-output (get-buffer-process (current-buffer)))))

Thanks, but it is not a good solution. Org cannot rely on internal
variables inside python.el (python-shell--first-prompt-received is an
internal variable). Internal variables are a subject of change without
notice.

May you instead try the attached alternative patch?

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: v2-0001-ob-python-Fix-Emacs-bug-50514-when-Python-loads-s.patch --]
[-- Type: text/x-patch, Size: 1642 bytes --]

From 06373a64387330d5bbdcbb3417cea7c2638713df Mon Sep 17 00:00:00 2001
Message-Id: <06373a64387330d5bbdcbb3417cea7c2638713df.1661650102.git.yantar92@gmail.com>
From: Ihor Radchenko <yantar92@gmail.com>
Date: Sat, 27 Aug 2022 22:06:11 +0800
Subject: [PATCH v2] ob-python: Fix Emacs bug#50514 when Python loads slowly

* lisp/ob-python.el (org-babel-python-initiate-session-by-key): Do not
rely on 10ms delay to initialize Python.  Wait until python process is
initialized using `org-babel-comint-wait-for-output'.
---
 lisp/ob-python.el | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index 932aca08e..e50a7c2e2 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -197,8 +197,7 @@ (defun org-babel-python-initiate-session-by-key (&optional session)
 	  (setq py-buffer (org-babel-python-with-earmuffs session)))
 	(let ((python-shell-buffer-name
 	       (org-babel-python-without-earmuffs py-buffer)))
-	  (run-python cmd)
-	  (sleep-for 0 10)))
+	  (run-python cmd)))
        ((and (eq 'python-mode org-babel-python-mode)
 	     (fboundp 'py-shell)) ; python-mode.el
 	(require 'python-mode)
@@ -217,6 +216,8 @@ (defun org-babel-python-initiate-session-by-key (&optional session)
 	  (py-shell nil nil t org-babel-python-command py-buffer nil nil t nil)))
        (t
 	(error "No function available for running an inferior Python")))
+      ;; Wait until Python initializes.
+      (org-babel-comint-wait-for-output py-buffer)
       (setq org-babel-python-buffers
 	    (cons (cons session py-buffer)
 		  (assq-delete-all session org-babel-python-buffers)))
-- 
2.35.1

bug#50514: 28.0.50; org inline call to python src block req C-g to break

[Barton, Mark]

[-- Attachment #1: Type: text/plain, Size: 1958 bytes --]



On Aug 27, 2022, at 6:30 PM, Ihor Radchenko <yantar92@gmail.com<mailto:yantar92@gmail.com>> wrote:

* lisp/ob-python.el (org-babel-python-initiate-session-by-key): Do not
rely on 10ms delay to initialize Python.  Wait until python process is
initialized using `org-babel-comint-wait-for-output'.
---
lisp/ob-python.el | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index 932aca08e..e50a7c2e2 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -197,8 +197,7 @@ (defun org-babel-python-initiate-session-by-key (&optional session)
  (setq py-buffer (org-babel-python-with-earmuffs session)))
(let ((python-shell-buffer-name
       (org-babel-python-without-earmuffs py-buffer)))
-   (run-python cmd)
-   (sleep-for 0 10)))
+   (run-python cmd)))
       ((and (eq 'python-mode org-babel-python-mode)
     (fboundp 'py-shell)) ; python-mode.el
(require 'python-mode)
@@ -217,6 +216,8 @@ (defun org-babel-python-initiate-session-by-key (&optional session)
  (py-shell nil nil t org-babel-python-command py-buffer nil nil t nil)))
       (t
(error "No function available for running an inferior Python")))
+      ;; Wait until Python initializes.
+      (org-babel-comint-wait-for-output py-buffer)
      (setq org-babel-python-buffers
    (cons (cons session py-buffer)
  (assq-delete-all session org-babel-python-buffers)))
--
2.35.1

That works for me. I'm using a series of python blocks to create org tables to be inserted in specific places in the document. So each block is named for that reason.

Do we need to worry about the note about not appropriate for large blocks of code or does that not apply in this case with the way you implemented it.
(org-babel-comint-wait-for-output BUFFER)

Wait until output arrives from BUFFER.
Note: this is only safe when waiting for the result of a single
statement (not large blocks of code).

Mark

[-- Attachment #2: Type: text/html, Size: 31975 bytes --]

bug#50514: 28.0.50; org inline call to python src block req C-g to break

[Ihor Radchenko]

"Barton, Mark" <Mark.Barton@disney.com> writes:

> Do we need to worry about the note about not appropriate for large blocks of code or does that not apply in this case with the way you implemented it.
> (org-babel-comint-wait-for-output BUFFER)
>
> Wait until output arrives from BUFFER.
> Note: this is only safe when waiting for the result of a single
> statement (not large blocks of code).

It should not be a problem. org-babel-python-initiate-session-by-key is
creating a new python session---no actual code blocks are passed to
python; just the python command is executed.

Fixed on main via 06373a643.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=06373a64387330d5bbdcbb3417cea7c2638713df

-- 
Ihor Radchenko,
Org mode contributor,
Learn more about Org mode at https://orgmode.org/.
Support Org development at https://liberapay.com/org-mode,
or support my work at https://liberapay.com/yantar92