Ihor Radchenko <yantar92@posteo.net> writes:

> Maybe we can simply override `comint-prompt-regexp' as we do in
> ob-shell? The default regexp seems to be too permissive.

I don't think this is a good idea, since this is a deliberate choice by
ESS, which contains explicit commentary that the regexp should not
contain BOL, because in some cases multiple prompts can end up on the
same line [1][2].

After some more thought, rather than overriding `comint-prompt-regexp',
I think it would be better to provide a mechanism to altogether prevent
the prompt removal in `org-babel-comint-async-filter'. There is no need
to remove prompts from Python and R async session evaluation, since I
wrote them in a way that avoids leaking any prompts in the output. And
in both cases, it is easy to come up with examples that work fine in Org
9.6 but get mangled in Org 9.7.

Therefore, I've attached an updated patch that provides such a mechanism
for ob-R and ob-python, reverting them to the Org 9.6 behavior. This is
done through a variable `org-babel-comint-async-remove-prompts-p', which
is set by an optional argument in `org-babel-comint-async-register'.

More generally, I think it is best to avoid doing the prompt removal
when possible, since it is difficult (impossible?) to do it perfectly,
and it can cause many problems. This is why ob-python avoids using
`org-babel-comint-with-output' -- it sources a tmp file rather than
inputting code directly to comint, so that prompts do not leak. I think
non-async R evaluation would benefit from a similar approach, and plan
to propose a patch to make ob-R non-async eval more similar to
ob-python.

[1] https://github.com/emacs-ess/ESS/blob/d60c13a6a347ea7a91ea3408bb464cff0ab4fef6/lisp/ess-r-mode.el#L2538
[2] https://github.com/emacs-ess/ESS/blob/d60c13a6a347ea7a91ea3408bb464cff0ab4fef6/lisp/ess-custom.el#L1829