From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1.migadu.com ([2001:41d0:403:58f0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id KGUjIsjz0WVHYAAA62LTzQ:P1 (envelope-from ) for ; Sun, 18 Feb 2024 13:10:48 +0100 Received: from aspmx1.migadu.com ([2001:41d0:403:58f0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1.migadu.com with LMTPS id KGUjIsjz0WVHYAAA62LTzQ (envelope-from ) for ; Sun, 18 Feb 2024 13:10:48 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=excalamus.com header.s=zmail header.b=bBUgdvrH; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org"; dmarc=none; arc=pass ("zohomail.com:s=zohoarc:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1708258248; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=PxmwaVZOQ9gBmqgPISNWDnIdbN23cp3QKsJR3Ibjvxc=; b=f1blEJI9YEANnzczXUGqNMHRUf7oIBQCrS3pkXn7ZmwxKhlW9DGAMxIYyOYc+SWx5/errU S7o8ukNqIWKCe8M7HBIPQM4QAD8RAo/GJ9RnzWsNfPN3aPo0NGTV2jJMtnYmW6znCM7TkX i6+AwaTYAgtQ49Xs4k+3Drb5An3r+S9p+PmJEEUbEROkvgjgyhn4exZ1BZ/zfbMn6ykqZg mmfn0n8680rD4tokCxfcK9FDLgw9/ZKiJ1VioABk9TCQoQJrHQ/gIis3j1tNfz4YN87xs2 o5X3sygAg7UVdcyG6WkdK2DgcnRFR2ChN/omFemahWihJ/lSfy84NUBudu33vw== ARC-Authentication-Results: i=2; aspmx1.migadu.com; dkim=pass header.d=excalamus.com header.s=zmail header.b=bBUgdvrH; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org"; dmarc=none; arc=pass ("zohomail.com:s=zohoarc:i=1") ARC-Seal: i=2; s=key1; d=yhetil.org; t=1708258248; a=rsa-sha256; cv=pass; b=sNtli0vYeGRU3UU0ZV25ZbW/gT2JsMiZ6D9TP2faXSNfefGyLBFDVVfKgaTZkmnksVde1N iw/QOTtew2YXPLU1eSXpP/eXBAZ6nuHDLIgwXKjdr1YRSbMfrgAsKXM8DMhgSNEh7sX6qi Dk9D4AohR52NkTVjmlB860Ej+SCULCMKZsMUmCaOlcDDsNGuJWW3C5SkMs2oWwTcUOmyh0 LQpuDpB57u/yB3WfCtzFOQaDn6V2f3jcnH7OvMy9HAuQBr4bzgGuSxUcRyC5OvYuTM0wu6 y/E3fQcXbpE9qOkZDDSpor7zxbW7cyfmkw667Fr6hxMkd2gAbxQQJm5/kmAvTg== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 2B6623D654 for ; Sun, 18 Feb 2024 13:10:48 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rbfzL-0000kv-D6; Sun, 18 Feb 2024 07:09:43 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rbfzJ-0000ke-O3 for emacs-orgmode@gnu.org; Sun, 18 Feb 2024 07:09:41 -0500 Received: from sender4-pp-f112.zoho.com ([136.143.188.112]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rbfzH-00018t-NJ for emacs-orgmode@gnu.org; Sun, 18 Feb 2024 07:09:41 -0500 ARC-Seal: i=1; a=rsa-sha256; t=1708258176; cv=none; d=zohomail.com; s=zohoarc; b=Imw810ppJnMlEJ5HcxmIw4Gv79pyw1sSh+3u1nUa9ubApWEAzRffHJHat8sGy753fqcG4GcRYSXXGYNIJDrvZdRMVsBjKQUZ+xGm/57IyIhY0+4lIAoelvzXUZWkV01vuQb/Rp5w0GLoj9o7D8NxjTZJl1Q0hCcYC5BxepeFMIs= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1708258176; h=Content-Type:Cc:Cc:Date:Date:From:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:Subject:To:To:Message-Id:Reply-To; bh=PxmwaVZOQ9gBmqgPISNWDnIdbN23cp3QKsJR3Ibjvxc=; b=hwyVN3iW+hvm+/WHt6UbykC8TpvRQFyyvU6xGQ27Fii4WZapwt383dugRqvguD1QTLmNyMzTdJe14poKr57dLLCCaMxi0h1j4kwO6Pg1DCiSgROzUuS22xzhyaE9707etqLWNTBxFhYz1TXoKDOTePesxookyM2dnr1CPHNd8/M= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=excalamus.com; spf=pass smtp.mailfrom=matt@excalamus.com; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1708258176; s=zmail; d=excalamus.com; i=matt@excalamus.com; h=Date:Date:From:From:To:To:Cc:Cc:Message-Id:Message-Id:In-Reply-To:References:Subject:Subject:MIME-Version:Content-Type:Reply-To; bh=PxmwaVZOQ9gBmqgPISNWDnIdbN23cp3QKsJR3Ibjvxc=; b=bBUgdvrHvHQ3ZvlbHUQYBnJz3DWeuFfV6zAkdqbTasVnQQHH8oPtPnaKHdo9m7fD 9wfbbDNS/L+2FM8kDJxAJ1SGkVm4L/YC23OKsw9uZkx2Vho9LaEqJDUKDYIQM6DU65c NfoL6PRtuWus9xrioo7QR8eUEKiOYl9kIv7Yp2tU= Received: from mail.zoho.com by mx.zohomail.com with SMTP id 1708258174815619.0702314824484; Sun, 18 Feb 2024 04:09:34 -0800 (PST) Date: Sun, 18 Feb 2024 13:09:34 +0100 From: Matt To: "Ihor Radchenko" Cc: "emacs-orgmode" Message-Id: <18dbc1f273c.11687295c1395973.3345700621594100711@excalamus.com> In-Reply-To: <87ttmn9fg0.fsf@localhost> References: <18d753c1e8a.cfb3e1921191837.5665565128507976741@excalamus.com> <87ttmn9fg0.fsf@localhost> Subject: Re: [BUG] Prompt appears in async shell results MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_4553725_1335109661.1708258174780" Importance: Medium User-Agent: Zoho Mail X-Mailer: Zoho Mail X-Zoho-Virus-Status: 1 X-Zoho-AV-Stamp: zmail-av-1.1.0/208.195.26 Received-SPF: pass client-ip=136.143.188.112; envelope-from=matt@excalamus.com; helo=sender4-pp-f112.zoho.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: emacs-orgmode-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN X-Migadu-Spam-Score: -5.92 X-Spam-Score: -5.92 X-Migadu-Queue-Id: 2B6623D654 X-Migadu-Scanner: mx11.migadu.com X-TUID: gD92JqCtPgFA ------=_Part_4553725_1335109661.1708258174780 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable ---- On Mon, 05 Feb 2024 15:00:18 +0100 Ihor Radchenko wrote ---=20 =20 > Yes. The right fix would be extracting the filter from > `org-babel-comint-with-output' and re-using it. This message documents my progress so that I may switch focus to Bruno's pa= tches in "Asynchronous blocks for everything" attention (https://list.orgmo= de.org/65cfa0d8.050a0220.cb569.ce34@mx.google.com/T/#u). Attached is a failed attempt at extracting the filter in =3Dorg-babel-comin= t-with-output=3D. It tries to extract the filter more-or-less directly: 1. take the filter code from =3Dorg-babel-comint-with-output=3D and put it into a separate function, =3Dorg-babel-comint-process-output-filter=3D 2. call =3Dorg-babel-comint-process-output-filter=3D from =3Dorg-babel-comint-with-output=3D and =3Dorg-babel-comint-async-filter=3D The unmodified =3Dorg-babel-comint-with-output=3D has a comment that says, = "Filter out prompts". This is misleading. The filter code does two things= : removes prompts *and* removes echoed input. The problem is the filter which removes echoes uses the body of the source = block. It's unclear how to give =3Dorg-babel-comint-async-filter=3D the bl= ock body. =3Dorg-babel-comint-async-filter=3D is a =3Dcomint-output-filter= -function=3D which receives a single input, "a string containing the text a= s originally inserted" in the comint buffer. Thoughts: - Split prompt filtering and input echoing into two filters + this seems to imply a =3D-hook=3D or =3D-functions=3D type implementati= on + where could input echo filter go? Where has access to the block body? - creating a generic prompt filter duplicates =3Dob-shell-async-chunk-call= back=3D or, more fundamentally, =3Dorg-babel-comint-async-chunk-callback=3D - What would it take to consolidate output filtering? In addition to promp= t filtering and input echo filtering, ob-shell filters the =3Dorg-babel-sh-= eoe-indicator=3D. I'm sure there's other filtering that happens on block o= utput. Wouldn't it be nice if that were in a single place, like right befo= re results are inserted? Please feel free to provide feedback and suggestions. -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at=C2=A0https://liberapay.com/org-mode ------=_Part_4553725_1335109661.1708258174780 Content-Type: application/octet-stream; name=v01-refactor-filter.diff Content-Transfer-Encoding: 7bit X-ZM_AttachId: 139275509747810000 Content-Disposition: attachment; filename=v01-refactor-filter.diff diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el index 1ec84e865..fd80880e9 100644 --- a/lisp/ob-comint.el +++ b/lisp/ob-comint.el @@ -74,6 +74,35 @@ This is useful when prompt unexpectedly changes." (setq comint-prompt-regexp org-babel-comint-prompt-regexp-old org-babel-comint-prompt-regexp-old tmp)))) +(defun org-babel-comint-process-output-filter (from-string &optional to-remove remove-echo) + "Remove string FROM-STRING from TO-REMOVE. + +TO-REMOVE defaults to `comint-prompt-regexp'. REMOVE-ECHO +removes echoed commands some shells may print and defaults to t." + (let* ((to-remove (or to-remove comint-prompt-regexp)) + (remove-echo (or remove-echo t))) + ;; remove TO-REMOVE from FROM-STRING + (while (string-match-p to-remove from-string) + (setq from-string + (replace-regexp-in-string + (format + "\\(?:%s\\)?\\(?:%s\\)[ \t]*" + "org-babel-comint-process-output-filter-separator\n" + to-remove) + "org-babel-comint-process-output-filter-separator\n" + from-string))) + ;; remove echo + (when (and remove-echo from-string ; <-- FIXME from-string needs to be block body + (string-match + (replace-regexp-in-string + "\n" "[\r\n]+" (regexp-quote (or from-string ""))) + from-string)) + (setq from-string (substring from-string (match-end 0)))) + (delete "" (split-string + from-string + "org-babel-comint-process-output-filter-separator\n")) + )) + (defmacro org-babel-comint-with-output (meta &rest body) "Evaluate BODY in BUFFER and return process output. Will wait until EOE-INDICATOR appears in the output, then return @@ -139,31 +168,35 @@ or user `keyboard-quit' during execution of body." (goto-char (process-mark (get-buffer-process (current-buffer)))) (insert dangling-text) - ;; Filter out prompts. - (while (string-match-p comint-prompt-regexp string-buffer) - (setq string-buffer - (replace-regexp-in-string - ;; Sometimes, we get multiple agglomerated - ;; prompts together in a single output: - ;; "prompt prompt prompt output" - ;; Or even "promptprompt ...>. - ;; Remove them progressively, so that - ;; possible "^" in the prompt regexp gets to - ;; work as we remove the heading prompt - ;; instance. - (format "\\(?:%s\\)?\\(?:%s\\)[ \t]*" ,org-babel-comint-prompt-separator comint-prompt-regexp) - ,org-babel-comint-prompt-separator - string-buffer))) - ;; remove echo'd FULL-BODY from input - (when (and ,remove-echo ,full-body - (string-match - (replace-regexp-in-string - "\n" "[\r\n]+" (regexp-quote (or ,full-body ""))) - string-buffer)) - (setq string-buffer (substring string-buffer (match-end 0)))) - (delete "" (split-string - string-buffer - ,org-babel-comint-prompt-separator)))))) + (org-babel-comint-process-output-filter string-buffer) + + ;; ;; Filter out prompts. + ;; (while (string-match-p comint-prompt-regexp string-buffer) + ;; (setq string-buffer + ;; (replace-regexp-in-string + ;; ;; Sometimes, we get multiple agglomerated + ;; ;; prompts together in a single output: + ;; ;; "prompt prompt prompt output" + ;; ;; Or even "promptprompt ...>. + ;; ;; Remove them progressively, so that + ;; ;; possible "^" in the prompt regexp gets to + ;; ;; work as we remove the heading prompt + ;; ;; instance. + ;; (format "\\(?:%s\\)?\\(?:%s\\)[ \t]*" ,org-babel-comint-prompt-separator comint-prompt-regexp) + ;; ,org-babel-comint-prompt-separator + ;; string-buffer))) + ;; ;; remove echo'd FULL-BODY from input + ;; (when (and ,remove-echo ,full-body + ;; (string-match + ;; (replace-regexp-in-string + ;; "\n" "[\r\n]+" (regexp-quote (or ,full-body ""))) + ;; string-buffer)) + ;; (setq string-buffer (substring string-buffer (match-end 0)))) + ;; (delete "" (split-string + ;; string-buffer + ;; ,org-babel-comint-prompt-separator)) + + )))) (defun org-babel-comint-input-command (buffer cmd) "Pass CMD to BUFFER. @@ -324,8 +357,9 @@ STRING contains the output originally inserted into the comint buffer." until (and (equal (match-string 1) "start") (equal (match-string 2) uuid)) finally return (+ 1 (match-end 0))))) - ;; Apply callback to clean up the result - (res-str (funcall org-babel-comint-async-chunk-callback + ;; Apply filter to clean up the result + (res-str ;(funcall org-babel-comint-async-chunk-callback + (org-babel-comint-process-output-filter res-str-raw))) ;; Search for uuid in associated org-buffers to insert results (cl-loop for buf in org-buffers ------=_Part_4553725_1335109661.1708258174780--