[BUG] ob-shell async session does not work with multiple blocks running [9.7.16 (9.7.16-ba9734 @ /nix/store/6la93ywyb1h055phgyqba7h7qay60p0m-emacs-packages-deps/share/emacs/site-lisp/elpa/org-9.7.16/)]

[BUG] ob-shell async session does not work with multiple blocks running [9.7.16 (9.7.16-ba9734 @ /nix/store/6la93ywyb1h055phgyqba7h7qay60p0m-emacs-packages-deps/share/emacs/site-lisp/elpa/org-9.7.16/)]

[ParetoOptimal]

Problem: Running two shell source blocks with the same session doesn't
properly extract out results from the sentinel value relating to each
respective source block.

This could be a known issue which might make this an enhancement request
rather than a bug report.

Expected: Running two source blocks at once to properly deal with only
their values from the "foo-comint-indirect" buffer.

Given:

#+begin_src sh :session foo :async
sleep 5
echo "almost done"
sleep 3
echo "done"
#+end_src

#+begin_src sh :session foo :async
sleep 1
echo "almost done"
sleep 6
echo "done"
#+end_src

Then running these right after another using:

M-<			;; beginning-of-buffer
C-c C-v C-n		;; org-babel-next-src-block
C-c C-c			;; org-ctrl-c-ctrl-c
C-c C-v C-n		;; org-babel-next-src-block
C-c C-c			;; org-ctrl-c-ctrl-c

You never get the results for the last source block, but there are 3
stages:

1. After evaluating each block, but before either is done:

#+begin_src sh :session foo :async
sleep 5
echo "almost done"
sleep 3
echo "done"
#+end_src

#+RESULTS:
: 4466c6fe-e7b5-4c14-9cae-d8f066ec6ebf

#+begin_src sh :session foo :async
sleep 1
echo "almost done"
sleep 6
echo "done"
#+end_src

#+RESULTS:
: 17c8a088-82c4-4695-90f6-fe3cba9fcfd7

2. After the first is finished running (notice the clobbered results
pulled in from the other block, this is the bug)

#+begin_src sh :session foo :async
sleep 5
echo "almost done"
sleep 3
echo "done"
#+end_src

#+RESULTS:
: echo 'ob_comint_async_shell_start_17c8a088-82c4-4695-90f6-fe3cba9fcfd7'
: sleep 1
: echo "almost done"
: sleep 6
: echo "done"
: echo 'ob_comint_async_shell_end_17c8a088-82c4-4695-90f6-fe3cba9fcfd7'
: almost done
: done

#+begin_src sh :session foo :async
sleep 1
echo "almost done"
sleep 6
echo "done"
#+end_src

#+RESULTS:
: 17c8a088-82c4-4695-90f6-fe3cba9fcfd7

3. Something is done to fixup the results from the first source block,
but the second source block never gets updated with it's results

#+begin_src sh :session foo :async
sleep 5
echo "almost done"
sleep 3
echo "done"
#+end_src

#+RESULTS:
: almost done
: done

#+begin_src sh :session foo :async
sleep 1
echo "almost done"
sleep 6
echo "done"
#+end_src

#+RESULTS:
: 17c8a088-82c4-4695-90f6-fe3cba9fcfd7

Here is what the foo-comint-indirect buffer looks like:

org_babel_sh_prompt> echo 'ob_comint_async_shell_start_4466c6fe-e7b5-4c14-9cae-d8f066ec6ebf'
sleep 5
echo "almost done"
sleep 3
echo "done"
echo 'ob_comint_async_shell_end_4466c6fe-e7b5-4c14-9cae-d8f066ec6ebf'
ob_comint_async_shell_start_4466c6fe-e7b5-4c14-9cae-d8f066ec6ebf
org_babel_sh_prompt> echo 'ob_comint_async_shell_start_17c8a088-82c4-4695-90f6-fe3cba9fcfd7'
sleep 1
echo "almost done"
sleep 6
echo "done"
echo 'ob_comint_async_shell_end_17c8a088-82c4-4695-90f6-fe3cba9fcfd7'
org_babel_sh_prompt> almost done
org_babel_sh_prompt> org_babel_sh_prompt> done
org_babel_sh_prompt> ob_comint_async_shell_end_4466c6fe-e7b5-4c14-9cae-d8f066ec6ebf
org_babel_sh_prompt> ob_comint_async_shell_start_17c8a088-82c4-4695-90f6-fe3cba9fcfd7
org_babel_sh_prompt> org_babel_sh_prompt> almost done
org_babel_sh_prompt> org_babel_sh_prompt> done
org_babel_sh_prompt> ob_comint_async_shell_end_17c8a088-82c4-4695-90f6-fe3cba9fcfd7
org_babel_sh_prompt>

Thanks for the great feature. Just allowing for long running shell
blocks to not block doing things in emacs was a great addition.


Emacs  : GNU Emacs 29.4 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo version 1.18.2, Xaw3d scroll bars)
Package: Org mode version 9.7.16 (9.7.16-ba9734 @ /nix/store/6la93ywyb1h055phgyqba7h7qay60p0m-emacs-packages-deps/share/emacs/site-lisp/elpa/org-9.7.16/)

Re: [BUG] ob-shell async session does not work with multiple blocks running [9.7.16 (9.7.16-ba9734 @ /nix/store/6la93ywyb1h055phgyqba7h7qay60p0m-emacs-packages-deps/share/emacs/site-lisp/elpa/org-9.7.16/)]

[Ihor Radchenko]

ParetoOptimal <pareto.optimal@mailfence.com> writes:

> Problem: Running two shell source blocks with the same session doesn't
> properly extract out results from the sentinel value relating to each
> respective source block.
>
> This could be a known issue which might make this an enhancement request
> rather than a bug report.
>
> Expected: Running two source blocks at once to properly deal with only
> their values from the "foo-comint-indirect" buffer.
> ...

Confirmed.

Matthew, may you please take a look?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] ob-shell async session does not work with multiple blocks running [9.7.16 (9.7.16-ba9734 @ /nix/store/6la93ywyb1h055phgyqba7h7qay60p0m-emacs-packages-deps/share/emacs/site-lisp/elpa/org-9.7.16/)]

[Matt]

 ---- On Sat, 23 Nov 2024 19:43:05 +0100  Ihor Radchenko  wrote --- 
 > ParetoOptimal pareto.optimal@mailfence.com> writes:
 > 
 > > Problem: Running two shell source blocks with the same session doesn't
 > > properly extract out results from the sentinel value relating to each
 > > respective source block.
 > >
 > > This could be a known issue which might make this an enhancement request
 > > rather than a bug report.
 > >
 > > Expected: Running two source blocks at once to properly deal with only
 > > their values from the "foo-comint-indirect" buffer.
 > > ...
 > 
 > Confirmed.
 > 
 > Matthew, may you please take a look?
 
Thank you for the bug report and for the confirmation.  Unfortunately, I haven't much time at the moment to address this.

It's not clear to me what causes the observed behavior.  

Assigning '1' and '2' to the echoes in each block shows that the results for the second block are placed in the first:

#+begin_src sh :session foo :async
sleep 5
echo "almost done1"
sleep 3
echo "done1"
#+end_src

#+RESULTS:
: almost done2
: done2

#+begin_src sh :session foo :async
sleep 1
echo "almost done2"
sleep 6
echo "done2"
#+end_src

#+RESULTS:
: bebfd924-c3b8-4ad4-aec0-23f322d5af0f

Re: [BUG] ob-shell async session does not work with multiple blocks running [9.7.16 (9.7.16-ba9734 @ /nix/store/6la93ywyb1h055phgyqba7h7qay60p0m-emacs-packages-deps/share/emacs/site-lisp/elpa/org-9.7.16/)]

[Ihor Radchenko]

Matt <matt@excalamus.com> writes:

> It's not clear to me what causes the observed behavior.  
>
> Assigning '1' and '2' to the echoes in each block shows that the results for the second block are placed in the first:
> ...

What happens is related to the way ob-shell insert async indicators at
the beginning/end of the code block. The begin indicator is inserted
synchronously, while the end indicator is inserted _after_ the code
block is evaluated.

In the bug report, what happens is the following:
1. We call the first src bloc, async "begin" indicator is inserted into
   the comint buffer immediately. The rest of the code is still running
2. We call the second src block, another "begin" indicator is inserted
   ...
3. _First_ src block finishes, inserting "end" indicator _after_ the
   rest of the code, capturing both outputs
4. _Second_ src block finishes

This is what I see in the comint buffer

sh-5.2$ PROMPT_COMMAND=;PS1="𒆸 ";PS2=

𒆸 echo 'ob_comint_async_shell_start_2ecc6791-9a1a-4118-b6ea-434e8ea15b8b'
exec 2>&1
sleep 5
echo "almost done1"
sleep 3
echo "done1"
:
echo 'ob_comint_async_shell_end_2ecc6791-9a1a-4118-b6ea-434e8ea15b8b'
ob_comint_async_shell_start_2ecc6791-9a1a-4118-b6ea-434e8ea15b8b
𒆸 𒆸 echo 'ob_comint_async_shell_start_b3810f61-ab61-4895-b0a5-e75133f03a47'
exec 2>&1
sleep 1
echo "almost done2"
sleep 6
echo "done2"
:
echo 'ob_comint_async_shell_end_b3810f61-ab61-4895-b0a5-e75133f03a47'
𒆸 almost done1
𒆸 𒆸 done1
𒆸 𒆸 ob_comint_async_shell_end_2ecc6791-9a1a-4118-b6ea-434e8ea15b8b
𒆸 ob_comint_async_shell_start_b3810f61-ab61-4895-b0a5-e75133f03a47
𒆸 𒆸 𒆸 almost done2
𒆸 𒆸 done2
𒆸 𒆸 ob_comint_async_shell_end_b3810f61-ab61-4895-b0a5-e75133f03a47
𒆸 

-- 
Ihor Radchenko // yantar92,
Org mode maintainer,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: [BUG] ob-shell async session does not work with multiple blocks running [9.7.16 (9.7.16-ba9734 @ /nix/store/6la93ywyb1h055phgyqba7h7qay60p0m-emacs-packages-deps/share/emacs/site-lisp/elpa/org-9.7.16/)]

[Matt]

[-- Attachment #1: Type: text/plain, Size: 158 bytes --]

Running from the latest commit, eebc9be7c, on Emacs on Android, I'm not able to reproduce. 

Did I miss a change between Dec. 10 and today which fixed this?

[-- Attachment #2: Type: text/html, Size: 225 bytes --]

Re: [BUG] ob-shell async session does not work with multiple blocks running [9.7.16 (9.7.16-ba9734 @ /nix/store/6la93ywyb1h055phgyqba7h7qay60p0m-emacs-packages-deps/share/emacs/site-lisp/elpa/org-9.7.16/)]

[Matt]

[-- Attachment #1: Type: text/plain, Size: 415 bytes --]

On December 19, 2024 12:24:48 AM GMT+01:00, Matt <matt@excalamus.com> wrote:
>Running from the latest commit, eebc9be7c, on Emacs on Android, I'm not able to reproduce. 
>
>Did I miss a change between Dec. 10 and today which fixed this?

My bad.  To reproduce the error, the blocks *must* be run using C-c C-c (org-ctrl-c-ctrl-c). Running with org-babel-execute-buffer produces a correct result for each block.

[-- Attachment #2: Type: text/html, Size: 504 bytes --]