Re: org-babel-execute-src-block filters characters from :session *shell* output

[Ihor Radchenko <yantar92@posteo.net>]

Phil <pe@7d.nz> writes:

> The error management channel is important.  Only a very
> small fraction of errors should be silenced, and we
> must not have feedback displaying that a function ran
> correctly when it did not.

Sure. I completely agree.

> Let's decouple an interactive shell from its
> compiler/interpreter (setting aside a Lisp
> evaluator that can dodge most of the problems):
>
> In many programming languages the source code can
> indeed be assimilated line by line by the
> interpreter through an interactive shell.
> - It returns nothing or an error after a definition
>    or a silent directive, then the prompt.
> - One or more values as a result or errors, then the prompt,
>    when calling functions and procedures.
> - It can expect more input and more lines before
>    terminating a statement, returning a secondary
>    prompt until that statement completes or fails.
>
> The terminal display is organized from programming
> interactions but the separation of channels is needed
> for further processing.
>
> 1) We should avoid filtering output with regexp acrobatics.

It is indeed not ideal. But it is hard to do better.
We simply do not have the means.

Of course, if someone can find a more robust way to decouple errors,
actual evaluation results, prompts (primary, secondary, etc), and the
interpreter echoing the input, it would be a patch we will happily
accept.

> 2) Communication with the process should go
>     through a function that can display
>     inputs and outputs and errors in three different channels
>     before sending it to the process.
>     Communicating directly to the process should remain
>     possible, but it's a short-circuit.

Yes, of course. But how?

> + Err : file, line, error code, message,
>       including the prompt which goes to stderr.
>     + The line number where the error occurs
>       related to a source block. There may be extra code
>       wrapping a block and noweb may complicate
>       things even more.
>       The situation to avoid is a line number unrelated
>       to the edited source block.
>       (Weaving and tangling programs is fine for me
>       I'm only thinking out loud here).

> Example:
> Errors goes in one file buffer
> and a filter wraps them in blocks.
> file:/tmp/babel-uuid/fiction.err
> #+begin_err :lang fiction :err 5 :line 2
> Parse error: near "?": syntax error
>    ? ;
>    ^--- error here
> #+end_err

This would be nice to have for sessions, yes. We already have this for
many non-session code blocks.

> Should all commands in a block be run ?
> It depends on the language of course, but usually, no,
> the execution stops and drops into a backtrace
> or raises an exception pointing to the stack frames.
> Dash has =set -e= (that doesn't always behave as expected).
> Sqlite has =.bail= for noninteractive sessions.

I am not sure if I agree, but your preference is a valid preference some
people might have. Our defaults are to send the whole block, which is an
ok option as well.

> And now here is my problem.
> Am I missing something about carriage return ?
> A differentiation problem between what happen
> when pressing C-q C-j in a shell, and <return> ?
>
> If ob-sql-session concatenates the SQL
> commands on one line, sqlite is able to stop on error.
> #+begin_src sql-session :engine sqlite :session A
>    select 1;  raise;   select 2;
>    select 33;
> #+end_src
>
> #+RESULTS:
> : 1
> : Parse error: near "raise": syntax error
> :   raise;   select 2; select 33;
> :   ^--- error here
>
> If it does not, a newline terminates the command,
> the prompt is displayed and the next line is
> an other command. Which is not the desirable effect
> as there's no way to halt the remaining commands.

This is simply how comint.el buffers work.
C-q C-j will enter literal newline. <return> will send the command
entered so far (including literal newlines) as input stream to the
interpreter. All at once.

The reason why comint does not do line-by-line is simply because it has
limited notion of the interpreter syntax - multiple lines may or may not
be several commands. Think of \ continuation in bash.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>