From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1.migadu.com ([2001:41d0:303:e224::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms13.migadu.com with LMTPS id QBZKKGK+hWZDhAEA62LTzQ:P1 (envelope-from ) for ; Wed, 03 Jul 2024 21:10:58 +0000 Received: from aspmx1.migadu.com ([2001:41d0:303:e224::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1.migadu.com with LMTPS id QBZKKGK+hWZDhAEA62LTzQ (envelope-from ) for ; Wed, 03 Jul 2024 23:10:58 +0200 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=7d.nz header.s=20240212 header.b="Exwp/Hx1"; dmarc=pass (policy=reject) header.from=7d.nz; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1720041058; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=tg3TGgBv+JvmoK7fPCK9P89R32JPSJc+fAQFCSqB7dE=; b=aJQAvxSAfESTPCSMzOBaL/rZT1w6wbvERAo2e+HQC0mR/sXW6vHur4W4w2O2/1X5Z4MIpd Ysdd/QEPNu8fAJwmT5188cCHU9UMr4F8dJQvFALjTw0AD4danhtjd0OpspeJPuT97HYCLH ZMb1A5dZKBJtdgCkv98MEUQdORTZ8s+MdtPz9WEclo4C4X49p8roYVpYg0v68xC6WnQyR4 3PObz2pUUNW7sI/ETq5zcwFDUc/pFMQVHOCMk5bRnFvAJanE8bltL1KkMQkN9txQOBiZmO 0/dngH/6dwkz1nEylsXrWcHIzdKvPj5Uy8rj+GzNX9gJQMyrul/KGy7xFSxmfg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1720041058; a=rsa-sha256; cv=none; b=bUc3xSTcNgJvx5T8UqE23GU8LgAE3GFHVlg9zDmMGDdZU6VxldvNyANqBornflc1z45tkJ fE/XHKvdXNkgA2+EEn5KgNUUHPcFkVsS5S7LrjmcWMqEhWNfuiqWmYWv7l5izKUzRLr6Tb ELLO456NzmOMxWqbT8NGPULNxAiZRKHxcSU01oUJiGhjh7jtywlK5d6Yi3UeZt076m0Jqg 2jZZwK4qC7AINrhJ+C7Tdl4+pRJDbKcFZZB9X0Nm9T6DaQjPjmRbJ++P15eIAFBrysOJ1x CIdLgseJee7G4q8S1eAxPQuiHEr9ajMao6LkoA6l69wNleMxUIlnSkGzogAWNA== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=7d.nz header.s=20240212 header.b="Exwp/Hx1"; dmarc=pass (policy=reject) header.from=7d.nz; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 05F73317A3 for ; Wed, 3 Jul 2024 23:10:58 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sP7EZ-0007y5-Qy; Wed, 03 Jul 2024 17:09:47 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sP7EY-0007xa-1n for emacs-orgmode@gnu.org; Wed, 03 Jul 2024 17:09:46 -0400 Received: from smtp-bc08.mail.infomaniak.ch ([2001:1600:4:17::bc08]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sP7EV-0006BK-2i for emacs-orgmode@gnu.org; Wed, 03 Jul 2024 17:09:45 -0400 Received: from smtp-4-0001.mail.infomaniak.ch (smtp-4-0001.mail.infomaniak.ch [10.7.10.108]) by smtp-3-3000.mail.infomaniak.ch (Postfix) with ESMTPS id 4WDspz53ktzPLt; Wed, 3 Jul 2024 23:09:31 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=7d.nz; s=20240212; t=1720040971; bh=tg3TGgBv+JvmoK7fPCK9P89R32JPSJc+fAQFCSqB7dE=; h=Date:From:Subject:To:Cc:References:In-Reply-To:From; b=Exwp/Hx1pL4tP9iZzZx0EPGa2A+0k8yoSSyuRmK+Dz0qaR0mANufirN/lws8TwABB LTz7Z7RUlGY+17xHukmjJVEnCkgrqVFDfkgnB+POw4SZqKzN2dKVXbYd4nCREEzi3E 944cSr1DB3gKXxWwYN/PiZLysm4f4DOoPWnjYjiwIcHm6WGvN6bPRQD0CU+OUhVP5u SoDfovFTOgIZRfIJc/N1PMaMkM6gBFb4ikveR4bUBy8gF0ZLlqonfuWjo1F0EJRwr7 47xQPpXxtDqdr/a3/Y+j+xw5exnz5MtN2QRgAC+ob2XxidnMM/B2VJVGl91rAi7IZo Cg4hqErchtMEA== Received: from unknown by smtp-4-0001.mail.infomaniak.ch (Postfix) with ESMTPA id 4WDspz1zNbzQnt; Wed, 3 Jul 2024 23:09:31 +0200 (CEST) Message-ID: Date: Wed, 3 Jul 2024 23:07:45 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird From: Phil Subject: Re: org-babel-execute-src-block filters characters from :session *shell* output To: Ihor Radchenko Cc: emacs-orgmode@gnu.org References: <87ikybk44c.fsf@localhost> <87o782gx7o.fsf@localhost> <87h6dra1ut.fsf@localhost> <874j9al08g.fsf@localhost> <5d3c8cce-b79d-43c2-baaa-459fea1ed3d9@7d.nz> <87bk3fa7ap.fsf@localhost> Content-Language: en-US In-Reply-To: <87bk3fa7ap.fsf@localhost> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Infomaniak-Routing: alpha Received-SPF: pass client-ip=2001:1600:4:17::bc08; envelope-from=pe@7d.nz; helo=smtp-bc08.mail.infomaniak.ch X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: emacs-orgmode-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Queue-Id: 05F73317A3 X-Migadu-Scanner: mx12.migadu.com X-Migadu-Spam-Score: -7.68 X-Spam-Score: -7.68 X-TUID: SvlFesOmirfp * [2024-07-02 22:05] Ihor Radchenko: > Phil Estival writes: > >> I'd like to add a few general remarks about *error status*. >> I'm starting to notice there are not much subprocesses >> that to do get called through =call-process= with >> ‘(REAL-DESTINATION ERROR-DESTINATION)’ kept as >> separate. You all know how useful stdin/out/err and >> the return codes are, but they're absent from Babel. > > They are not. > See `org-babel-eval-error-notify'. > Not for sessions though. That's right. The situation is fine when there is one synchronous command line. And I admit too much "interactivity" may put astray from strict reproducible frames. >> When several commands are given in one batch (what a >> babel source block is in many cases), they can either be >> all run I should rather say "sent to a compiler or interpreter process" >> until the end or be stopped when an error is >> met. In both cases, knowing the number of errors met is >> possible in one shell that do agree with the convention >> of returning a value for every command (or command line). > > May you elaborate? Source code is given to a compiler one character at a time. It's also typed in a terminal one character at a time, and when is hit, a line can be sent to the underlying interpreter. Programming environments often run a parallel evaluator to provide feedback, completions, suggestions, and so on, to the programmer, on every keystroke. Reproducible research will only provides the source: a text, made of syntactic atoms. On the host on which this text is read can coexist numerous evaluators, and while they can virtually be anything, the basic environment is a syntax colorator and a command processor that activates only when it is asked to. The error management channel is important. Only a very small fraction of errors should be silenced, and we must not have feedback displaying that a function ran correctly when it did not. Let's decouple an interactive shell from its compiler/interpreter (setting aside a Lisp evaluator that can dodge most of the problems): In many programming languages the source code can indeed be assimilated line by line by the interpreter through an interactive shell. - It returns nothing or an error after a definition or a silent directive, then the prompt. - One or more values as a result or errors, then the prompt, when calling functions and procedures. - It can expect more input and more lines before terminating a statement, returning a secondary prompt until that statement completes or fails. The terminal display is organized from programming interactions but the separation of channels is needed for further processing. 1) We should avoid filtering output with regexp acrobatics. 2) Communication with the process should go through a function that can display inputs and outputs and errors in three different channels before sending it to the process. Communicating directly to the process should remain possible, but it's a short-circuit. + In : code block. - Headers and parameters are given upon session initialization. + Out : program output and exit code. This we have. + Err : file, line, error code, message, including the prompt which goes to stderr. + The line number where the error occurs related to a source block. There may be extra code wrapping a block and noweb may complicate things even more. The situation to avoid is a line number unrelated to the edited source block. (Weaving and tangling programs is fine for me I'm only thinking out loud here). Example: Errors goes in one file buffer and a filter wraps them in blocks. file:/tmp/babel-uuid/fiction.err #+begin_err :lang fiction :err 5 :line 2 Parse error: near "?": syntax error ? ; ^--- error here #+end_err Should all commands in a block be run ? It depends on the language of course, but usually, no, the execution stops and drops into a backtrace or raises an exception pointing to the stack frames. Dash has =set -e= (that doesn't always behave as expected). Sqlite has =.bail= for noninteractive sessions. And now here is my problem. Am I missing something about carriage return ? A differentiation problem between what happen when pressing C-q C-j in a shell, and ? If ob-sql-session concatenates the SQL commands on one line, sqlite is able to stop on error. #+begin_src sql-session :engine sqlite :session A select 1; raise; select 2; select 33; #+end_src #+RESULTS: : 1 : Parse error: near "raise": syntax error : raise; select 2; select 33; : ^--- error here If it does not, a newline terminates the command, the prompt is displayed and the next line is an other command. Which is not the desirable effect as there's no way to halt the remaining commands. #+RESULTS: : 1 : Parse error: near "raise": syntax error : raise; select 2; : ^--- error here : 33 At last, I would reformulate the above paragraph dealing with a "batch of commands" like so: - getting the number of errors met (on a shell that would not stop on error) - and agreeing with the convention of returning a value for every command — or command line or function means: returning [Result,Error] conveyed to different channels. Quite naturally an interpreter or an interactive shell implemented like so also return [R,E] for every definitions, calls and statements as input on its command line. The prompt is only meant for interactive display, can be anything and change during the session. The indication of a command termination being the value held in Error provided on stderr. The terminal can then acknowledge the returned value by incrementing counters displayed into that prompt. -- Phil Estival