Re: [BUG] conda doesn't work in ob-shell sessions AND org-babel-execute-src-block filters characters from :session *shell* output

["Cook, Malcolm" <MEC@stowers.org>]

# -*-  org-confirm-babel-evaluate: nil; -*-

I have been struggling with interrelated issues raised in

 - https://list.orgmode.org/87jznda90u.fsf@localhost/#t
 - https://list.orgmode.org/87le1bc8j3.fsf@localhost/

I expect I am using all the patches offered in addressing these given
my recent build from main.  However, in my hands, I find they still
easily allow for mistakes identifying prompts in code block results.

In this demonstration, I am extending the approach to inquiry begun by Jack in
https://list.orgmode.org/87ttzn1mai.fsf@gmail.com/

#+begin_src emacs-lisp :results raw
   (org-version nil t)
   (emacs-version)
#+end_src

#+RESULTS:
GNU Emacs 31.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo version 1.15.12, Xaw3d scroll bars)
 of 2024-09-04
(Org mode version 9.7.10 (release_9.7.10 @ /home/mec/.local/share/emacs/31.0.50/lisp/org/) GNU Emacs 31.0.50 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo version 1.15.12, Xaw3d scroll bars)
 of 2024-09-04)

#+begin_src emacs-lisp
  (org-babel-do-load-languages
    'org-babel-load-languages
    '((shell . t)))
#+end_src

#+RESULTS:

Here I define two org code blocks I will use repeatedly below:

#+name:test_filter
#+begin_src shell :session *shell* :results output
  printf "a\nb\nc\n>d\n<e\nf>\nggg ggg>\nhhh hhh+\na\n"
#+end_src

#+name:shell_prompt_info
#+begin_src elisp
  (with-current-buffer "*shell*"
    (format "[comint-prompt-regexp]=[%s]\n[org-babel-comint-prompt-regexp-old]=[%s]" comint-prompt-regexp org-babel-comint-prompt-regexp-old))
#+end_src

#+caption: The results looks good - the output apparently is not confused as being prompt.
#+call: test_filter()

#+RESULTS:
: a
: b
: c
: >d
: <e
: f>
: ggg ggg>
: hhh hhh+
: a

#+caption: take a look at the prompt variables.
#+call:shell_prompt_info()

#+RESULTS:
: [comint-prompt-regexp]=[^org_babel_sh_prompt>  *]
: [org-babel-comint-prompt-regexp-old]=[^[^#$%>
: ]*[#$%>] *]

#+caption: check on conda's availabiity & version
#+begin_src shell :session *shell* :results output
conda --version
#+end_src

#+RESULTS:
: conda 24.7.1

#+begin_src shell :session *shell* :results output
conda create --yes --name myenv python=3.9
#+end_src

#+RESULTS:
#+begin_example
... abbreviated...

To activate this environment, use

conda activate myenv

To deactivate an active environment, use

conda deactivate
#+end_example

#+begin_src shell :session *shell* :results output
  conda activate myenv
#+end_src

#+RESULTS:

#+begin_src shell :session *shell* :results output
which python
#+end_src

#+RESULTS:
: /n/projects/mec/SRSCHPC2/local/inst/Mambaforge/24.3.0-0/envs/myenv/bin/python

#+caption: alas, the output of test_filter is changed.  Some lines are gone missing and some are changed.
#+call: test_filter()

#+RESULTS:
: a
: b
: c
: d
: <e
: 
: hhh hhh+
: a

#+caption: Observe the prompts have changed.  Perhaps this is related issue?
#+call:shell_prompt_info()

#+RESULTS:
: [comint-prompt-regexp]=[^[^#$%>
: ]*[#$%>] *]
: [org-babel-comint-prompt-regexp-old]=[^org_babel_sh_prompt>  *]

#+caption:  can we restore by deactivating the environment?
#+begin_src shell :session *shell* :results output
  conda activate  
#+end_src

#+RESULTS:

#+caption: alas, no:
#+call: test_filter()

#+RESULTS:
: a
: b
: c
: d
: <e
: 
: hhh hhh+
: a

#+caption: how about by resetting the prompt
#+begin_src shell :session *shell* :results output
  PROMPT_COMMAND=;PS1="org_babel_sh_prompt> ";PS2=
#+end_src

#+RESULTS:

#+caption: alas, again, no
#+call: test_filter()
#+RESULTS:
: a
: b
: c
: d
: <e
: 
: hhh hhh+
: a

#+caption: perhaps restoring the prompt variables will recover?
#+begin_src elisp
    (with-current-buffer "*shell*"
      (setq-local comint-prompt-regexp "^org_babel_sh_prompt>  *"
      	      org-babel-comint-prompt-regexp-old "[^[^#$%>
]*[#$%>] *"))
#+end_src

#+RESULTS:
: [^[^#$%>
: ]*[#$%>] *

#+caption: YES!
#+call: test_filter()

#+RESULTS:
: a
: b
: c
: >d
: <e
: f>
: ggg ggg>
: hhh hhh+
: a


In the above, I am exclusively allowing org/ob/comint to "own" the shell
buffer, and not interact with it, as recommended earlier by Ivor.

I have tried the above after first calling `(shell)` and find
variations on the above occur.  I would like to be able to 'share' the
*shell* buffer with org/ob/comint but expect resolving the
non-interactive case should possibly lay foundation.

I would additional like to layer in working with remote shells
(e.g. `:dir "/ssh:me@host:~/`) and have tried but this is just
layering in complexity on the localhost case so I'm backing off for
now.

What else can I report or test?