Re: org-babel-execute-src-block filters characters from :session *shell* output

[Phil <pe@7d.nz>]

* [2024-07-02 22:05] Ihor Radchenko:
> Phil Estival <pe@7d.nz> writes:
> 
>> I'd like to add a few general remarks about *error status*.
>> I'm starting to notice there are not much subprocesses
>> that to do get called through =call-process= with
>> ‘(REAL-DESTINATION ERROR-DESTINATION)’ kept as
>> separate.  You all know how useful stdin/out/err and
>> the return codes are, but they're absent from Babel.
> 
> They are not.
> See `org-babel-eval-error-notify'.
> Not for sessions though.

That's right. The situation is fine when there is
one synchronous command line. And I admit too much
"interactivity" may put astray from strict reproducible
frames.

>> When several commands are given in one batch (what a
>> babel source block is in many cases), they can either be
>> all run
I should rather say "sent to a compiler or interpreter process"
>> until the end or be stopped when an error is
>> met. In both cases, knowing the number of errors met is
>> possible in one shell that do agree with the convention
>> of returning a value for every command (or command line).
>
> May you elaborate?

Source code is given to a compiler one character at a
time. It's also typed in a terminal one character at
a time, and when <return> is hit, a line can be sent to
the underlying interpreter.  Programming environments
often run a parallel evaluator to provide feedback,
completions, suggestions, and so on, to the programmer,
on every keystroke.

Reproducible research will only provides the source:
a text, made of syntactic atoms.
On the host on which this text is read can coexist
numerous evaluators, and while they can virtually be
anything, the basic environment is a syntax colorator
and a command processor that activates only when it is
asked to.

The error management channel is important.  Only a very
small fraction of errors should be silenced, and we
must not have feedback displaying that a function ran
correctly when it did not.

Let's decouple an interactive shell from its
compiler/interpreter (setting aside a Lisp
evaluator that can dodge most of the problems):

In many programming languages the source code can
indeed be assimilated line by line by the
interpreter through an interactive shell.
- It returns nothing or an error after a definition
   or a silent directive, then the prompt.
- One or more values as a result or errors, then the prompt,
   when calling functions and procedures.
- It can expect more input and more lines before
   terminating a statement, returning a secondary
   prompt until that statement completes or fails.

The terminal display is organized from programming
interactions but the separation of channels is needed
for further processing.

1) We should avoid filtering output with regexp acrobatics.
2) Communication with the process should go
    through a function that can display
    inputs and outputs and errors in three different channels
    before sending it to the process.
    Communicating directly to the process should remain
    possible, but it's a short-circuit.

+ In : code block.
    -  Headers and parameters are given upon session
       initialization.
+ Out : program output and exit code.  This we have.

+ Err : file, line, error code, message,
      including the prompt which goes to stderr.
    + The line number where the error occurs
      related to a source block. There may be extra code
      wrapping a block and noweb may complicate
      things even more.
      The situation to avoid is a line number unrelated
      to the edited source block.
      (Weaving and tangling programs is fine for me
      I'm only thinking out loud here).

Example:
Errors goes in one file buffer
and a filter wraps them in blocks.
file:/tmp/babel-uuid/fiction.err
#+begin_err :lang fiction :err 5 :line 2
Parse error: near "?": syntax error
   ? ;
   ^--- error here
#+end_err


Should all commands in a block be run ?
It depends on the language of course, but usually, no,
the execution stops and drops into a backtrace
or raises an exception pointing to the stack frames.
Dash has =set -e= (that doesn't always behave as expected).
Sqlite has =.bail= for noninteractive sessions.

And now here is my problem.
Am I missing something about carriage return ?
A differentiation problem between what happen
when pressing C-q C-j in a shell, and <return> ?

If ob-sql-session concatenates the SQL
commands on one line, sqlite is able to stop on error.
#+begin_src sql-session :engine sqlite :session A
   select 1;  raise;   select 2;
   select 33;
#+end_src

#+RESULTS:
: 1
: Parse error: near "raise": syntax error
:   raise;   select 2; select 33;
:   ^--- error here

If it does not, a newline terminates the command,
the prompt is displayed and the next line is
an other command. Which is not the desirable effect
as there's no way to halt the remaining commands.

#+RESULTS:
: 1
: Parse error: near "raise": syntax error
:   raise;   select 2;
:   ^--- error here
: 33


At last, I would reformulate the above paragraph dealing
with a "batch of commands" like so:

- getting the number of errors met (on a shell that
   would not stop on error)
- and agreeing with the convention of returning a value
   for every command — or command line or function

means: returning [Result,Error] conveyed to different channels.
Quite naturally an interpreter or an interactive shell
implemented like so also return [R,E] for every definitions,
calls and statements as input on its command line.
The prompt is only meant for interactive display,
can be anything and change during the session.
The indication of a command termination being the value
held in Error provided on stderr. The terminal can then
acknowledge the returned value by incrementing counters
displayed into that prompt.

--
Phil Estival