From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id iNDQIW5G92WnigAA62LTzQ:P1 (envelope-from ) for ; Sun, 17 Mar 2024 20:37:18 +0100 Received: from aspmx1.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1.migadu.com with LMTPS id iNDQIW5G92WnigAA62LTzQ (envelope-from ) for ; Sun, 17 Mar 2024 20:37:18 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=excalamus.com header.s=zmail header.b=fLs1DuA7; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org"; dmarc=none; arc=pass ("zohomail.com:s=zohoarc:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1710704238; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=V1Irk9+6pzlw1chSCgpF1iAadZrUqGhKH/1ttDIzNB4=; b=eshKZczB/gaOqnN9elHVNpWL+NFOZb+M8DDJmr8YaVqwlFYBF/bc8/9MxMF4rEk4VDqaca 8QAJEnZVYsOlIduSzhSRE5veUHkmTDhrcVBzysskh+I0exWMI8QLWmlPvXgcrQ8RpK8HJ9 MsJ5+m/kq8FjwWZKQVYoETZCO12hMBOQfHVTtHqRIet9liIR22TekwVxebZm/ZLsX7IEhk h3OOSI8CgnBsnLHmorgGQLHtE2ZxSMakEzcO6/8SfthVe8PsTwAgs4ZKl1PB5IU/q2fd7C TgbI6KMSuMLQaJcA4IBgs7MOAEurztjGdx3JfIk4WH4jarsM1tIkUhiRxmzZkg== ARC-Authentication-Results: i=2; aspmx1.migadu.com; dkim=pass header.d=excalamus.com header.s=zmail header.b=fLs1DuA7; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org"; dmarc=none; arc=pass ("zohomail.com:s=zohoarc:i=1") ARC-Seal: i=2; s=key1; d=yhetil.org; t=1710704238; a=rsa-sha256; cv=pass; b=KAZdxxvfgbI6yMVkNRS7rw700AnQ5haICzbVbVmMyDlpnkdQE5U09OGGYjjThUlkn5MDOp FxEu5Xe1IzgOohsYkZxDmd1j1FobNwKb6ECyKKsmiM3aKPfqs0hzILSgk+22x6Mytq37ND jaVL89ZASW8U911KnJkkh5yoRnucFmGby+gpPrk3OrGNSvYKBIOZETSrHBPRfKx7pMMTUd rDW2zK20KA7kb5jm0pz4RzzpkNPgQbrkZLZC06dilU7m9olJDgaRf/XaOvo6TLt5rtY9Oa KpwaQZ/z6/UWyBNNweuOCV/gqMZLlmlMp0c+alLj/C+V6mxTDpREgEg9MEQqRA== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 1D02544440 for ; Sun, 17 Mar 2024 20:37:18 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rlwJT-0003tU-87; Sun, 17 Mar 2024 15:36:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rlwJS-0003tH-KV for emacs-orgmode@gnu.org; Sun, 17 Mar 2024 15:36:54 -0400 Received: from sender4-pp-f112.zoho.com ([136.143.188.112]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rlwJQ-0007aL-MJ for emacs-orgmode@gnu.org; Sun, 17 Mar 2024 15:36:54 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1710704209; cv=none; d=zohomail.com; s=zohoarc; b=fTKYmJOtx5wkB7v4TfdFAkW3wVXLkH2nF5TINYZX9Hpq0FNDUrqgkrQXHXP6lBhSHgYFC2kqKwhxsfdH1fHgbThD+KSeCV+sn2bNHskqJLyocUou2bQYOe0eUUnH6MLdoXND+jlQls1zVsolqZa9raFFKemv6srv8IJ9i2eRBCU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1710704209; h=Content-Type:Cc:Cc:Date:Date:From:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:Subject:To:To:Message-Id:Reply-To; bh=V1Irk9+6pzlw1chSCgpF1iAadZrUqGhKH/1ttDIzNB4=; b=Ja5OL0VRzsYFRuCaes268Ydo1fLnlUu0ofW3xavsxZIqFwoDjz7nwOK/jfcN35S1a2ebhP+hB5Hx9SQnd0EHb5HLHDvSGIZdSi/B1YleV+t6lMN6QgrwQzbVni8ceU9rt9hoqy8MXYLv3eXZ20epe9Gai9U/Ri/fiB3QU8DIGTY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=excalamus.com; spf=pass smtp.mailfrom=matt@excalamus.com; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1710704209; s=zmail; d=excalamus.com; i=matt@excalamus.com; h=Date:Date:From:From:To:To:Cc:Cc:Message-Id:Message-Id:In-Reply-To:References:Subject:Subject:MIME-Version:Content-Type:Reply-To; bh=V1Irk9+6pzlw1chSCgpF1iAadZrUqGhKH/1ttDIzNB4=; b=fLs1DuA7a3qXW5t6q58VFTE89vx69+fVi+saLuoHkkEV0C3QC/906LbPD0cWFTW/ xhEldWXUUTk4mUUE9GZxDZFmGOyonffWVboNnwmwhcqOaeD3zswCHcQviiaTXOsVmdz pGnKdqoW57ZCA9JGWnMWZKXh9/eqK+aKWS7q0u9E= Received: from mail.zoho.com by mx.zohomail.com with SMTP id 1710704206311586.656012101867; Sun, 17 Mar 2024 12:36:46 -0700 (PDT) Date: Sun, 17 Mar 2024 20:36:46 +0100 From: Matt To: "Ihor Radchenko" Cc: "emacs-orgmode" Message-Id: <18e4deaa1bd.ff0f87fa694858.5955779335024266818@excalamus.com> In-Reply-To: <87ttm4g14s.fsf@localhost> References: <18d753c1e8a.cfb3e1921191837.5665565128507976741@excalamus.com> <87ttmn9fg0.fsf@localhost> <18dbc1f273c.11687295c1395973.3345700621594100711@excalamus.com> <87ttm4g14s.fsf@localhost> Subject: Re: [BUG] Prompt appears in async shell results MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_2243495_1715428140.1710704206269" Importance: Medium User-Agent: Zoho Mail X-Mailer: Zoho Mail X-Zoho-Virus-Status: 1 X-Zoho-AV-Stamp: zmail-av-1.1.0/210.683.81 Received-SPF: pass client-ip=136.143.188.112; envelope-from=matt@excalamus.com; helo=sender4-pp-f112.zoho.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: emacs-orgmode-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -8.61 X-Spam-Score: -8.61 X-Migadu-Queue-Id: 1D02544440 X-Migadu-Scanner: mx13.migadu.com X-TUID: 69TteuKA4/mt ------=_Part_2243495_1715428140.1710704206269 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable ---- On Mon, 19 Feb 2024 12:07:55 +0100 Ihor Radchenko wrote --- > Matt matt@excalamus.com> writes: > > > - Split prompt filtering and input echoing into two filters > > + this seems to imply a =3D-hook=3D or =3D-functions=3D type impleme= ntation > > Not necessarily. You may just split the filter into (1) prompt > remover; (2) body remover. The filter does not need to interact with > comint buffer and may simply be provided a regexp/string to be removed. > Then, the caller will be responsible to supply prompt regexp/body. Attached are patches which split the prompt and echo filtering into separat= e functions. These are then used to refactor =3Dorg-babel-comint-with-outp= ut=3D and to remove the prompt from async results. I also wrote tests, one= for the reported bug and two for the filter functions. Tests for the filt= er functions are in a new file, 'test-ob-comint.el'. Two thoughts: 1. Can people please test that the changes to =3Dorg-babel-comint-with-outp= ut=3D haven't broken other languages? The filter, which I extracted from =3Dorg-babel-comint-with-output=3D, work= s according to the test I wrote. The test is based on ob-shell output. Ho= wever, =3Dorg-babel-comint-with-output=3D is used by other languages. I wa= sn't sure if echoes showed up for other languages and I had no examples fro= m other languages to work from. The filter is untested for anything but sh= ell, aside from running 'make test' (which returns no unexpected errors). = If no echo argument is given, the filter simply returns the string it would= otherwise try to remove the echo from. So, I suspect the worst outcome wo= uld be that echoes might start showing up in output. But when have my pred= ictions ever been right? :) 2. Legacy code woes I'll start another thread to rant, er, discuss. -- Matt Trzcinski Emacs Org contributor (ob-shell) Learn more about Org mode at https://orgmode.org Support Org development at=C2=A0https://liberapay.com/org-mode ------=_Part_2243495_1715428140.1710704206269 Content-Type: application/octet-stream; name=0001-testing-lisp-test-ob-comint.el-Make-test-for-prompt-.patch Content-Transfer-Encoding: base64 X-ZM_AttachId: 139299970062700120 Content-Disposition: attachment; filename=0001-testing-lisp-test-ob-comint.el-Make-test-for-prompt-.patch RnJvbSA2NmNjNTE0ZWIyN2E5YzUwZmIwMTI4ZmEzMGU1NWI3NjA5YmI5YjE4IE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBNYXR0aGV3IFRyemNpbnNraSA8bWF0dEBleGNhbGFtdXMuY29t PgpEYXRlOiBTdW4sIDE3IE1hciAyMDI0IDE2OjQ3OjIyICswMTAwClN1YmplY3Q6IFtQQVRDSCAx LzddIHRlc3RpbmcvbGlzcC90ZXN0LW9iLWNvbWludC5lbDogTWFrZSB0ZXN0IGZvciBwcm9tcHQK IGZpbHRlcgoKKiB0ZXN0LW9iLWNvbWludC5lbDogIE1ha2UgbmV3IGZpbGUgZm9yIGNvbWludCB0 ZXN0cy4KKHRlc3Qtb3JnLWJhYmVsLWNvbWludC9wcm9tcHQtZmlsdGVyLXJlbW92ZXMtcHJvbXB0 KTogVGVzdCB0aGF0IHRoZQpwcm9tcHQgaXMgcmVtb3ZlZCBmcm9tIHByb2Nlc3MgYnVmZmVyIG91 dHB1dC4KLS0tCiB0ZXN0aW5nL2xpc3AvdGVzdC1vYi1jb21pbnQuZWwgfCA1NSArKysrKysrKysr KysrKysrKysrKysrKysrKysrKysrKysrCiAxIGZpbGUgY2hhbmdlZCwgNTUgaW5zZXJ0aW9ucygr KQogY3JlYXRlIG1vZGUgMTAwNjQ0IHRlc3RpbmcvbGlzcC90ZXN0LW9iLWNvbWludC5lbAoKZGlm ZiAtLWdpdCBhL3Rlc3RpbmcvbGlzcC90ZXN0LW9iLWNvbWludC5lbCBiL3Rlc3RpbmcvbGlzcC90 ZXN0LW9iLWNvbWludC5lbApuZXcgZmlsZSBtb2RlIDEwMDY0NAppbmRleCAwMDAwMDAwMDAuLjkx OTNlZjhmMwotLS0gL2Rldi9udWxsCisrKyBiL3Rlc3RpbmcvbGlzcC90ZXN0LW9iLWNvbWludC5l bApAQCAtMCwwICsxLDU1IEBACis7OzsgdGVzdC1vYi1jb21pbnQuZWwgIC0qLSBsZXhpY2FsLWJp bmRpbmc6IHQ7IC0qLQorCis7OyBDb3B5cmlnaHQgKGMpIDIwMjQgTWF0dGhldyBUcnpjaW5za2kK Kzs7IEF1dGhvcnM6IE1hdHRoZXcgVHJ6Y2luc2tpCisKKzs7IFRoaXMgZmlsZSBpcyBub3QgcGFy dCBvZiBHTlUgRW1hY3MuCisKKzs7IFRoaXMgcHJvZ3JhbSBpcyBmcmVlIHNvZnR3YXJlOyB5b3Ug Y2FuIHJlZGlzdHJpYnV0ZSBpdCBhbmQvb3IgbW9kaWZ5Cis7OyBpdCB1bmRlciB0aGUgdGVybXMg b2YgdGhlIEdOVSBHZW5lcmFsIFB1YmxpYyBMaWNlbnNlIGFzIHB1Ymxpc2hlZCBieQorOzsgdGhl IEZyZWUgU29mdHdhcmUgRm91bmRhdGlvbiwgZWl0aGVyIHZlcnNpb24gMyBvZiB0aGUgTGljZW5z ZSwgb3IKKzs7IChhdCB5b3VyIG9wdGlvbikgYW55IGxhdGVyIHZlcnNpb24uCisKKzs7IFRoaXMg cHJvZ3JhbSBpcyBkaXN0cmlidXRlZCBpbiB0aGUgaG9wZSB0aGF0IGl0IHdpbGwgYmUgdXNlZnVs LAorOzsgYnV0IFdJVEhPVVQgQU5ZIFdBUlJBTlRZOyB3aXRob3V0IGV2ZW4gdGhlIGltcGxpZWQg d2FycmFudHkgb2YKKzs7IE1FUkNIQU5UQUJJTElUWSBvciBGSVRORVNTIEZPUiBBIFBBUlRJQ1VM QVIgUFVSUE9TRS4gIFNlZSB0aGUKKzs7IEdOVSBHZW5lcmFsIFB1YmxpYyBMaWNlbnNlIGZvciBt b3JlIGRldGFpbHMuCisKKzs7IFlvdSBzaG91bGQgaGF2ZSByZWNlaXZlZCBhIGNvcHkgb2YgdGhl IEdOVSBHZW5lcmFsIFB1YmxpYyBMaWNlbnNlCis7OyBhbG9uZyB3aXRoIHRoaXMgcHJvZ3JhbS4g IElmIG5vdCwgc2VlIDxodHRwczovL3d3dy5nbnUub3JnL2xpY2Vuc2VzLz4uCisKKwwKKzs7OyBD b21tZW50OgorCis7OyBTZWUgdGVzdGluZy9SRUFETUUgZm9yIGhvdyB0byBydW4gdGVzdHMuCisK KwwKKzs7OyBSZXF1aXJlbWVudHM6CisKKwwKKzs7OyBDb2RlOgorKGVydC1kZWZ0ZXN0IHRlc3Qt b3JnLWJhYmVsLWNvbWludC9wcm9tcHQtZmlsdGVyLXJlbW92ZXMtcHJvbXB0ICgpCisgICJUZXN0 IHRoYXQgcHJvbXB0IGlzIGFjdHVhbGx5IHJlbW92ZWQuIgorICAobGV0KiAoKHByb21wdCAib3Jn X2JhYmVsX3NoX3Byb21wdD4gIikKKyAgICAgICAgIChyZXN1bHRzICJvcmdfYmFiZWxfc2hfcHJv bXB0PiBlY2hvICdvYl9jb21pbnRfYXN5bmNfc2hlbGxfc3RhcnRfZDc4YWM0OWYtZGM4YS00YzM5 LTgyN2MtYzkzMjI1NDg0ZDU5JworIyBwcmludCBtZXNzYWdlCitlY2hvIFwiaGVsbG8gd29ybGRc IgorZWNobyAnb2JfY29taW50X2FzeW5jX3NoZWxsX2VuZF9kNzhhYzQ5Zi1kYzhhLTRjMzktODI3 Yy1jOTMyMjU0ODRkNTknCitvYl9jb21pbnRfYXN5bmNfc2hlbGxfc3RhcnRfZDc4YWM0OWYtZGM4 YS00YzM5LTgyN2MtYzkzMjI1NDg0ZDU5CitvcmdfYmFiZWxfc2hfcHJvbXB0PiBvcmdfYmFiZWxf c2hfcHJvbXB0PiBcImhlbGxvIHdvcmxkXCIKK29yZ19iYWJlbF9zaF9wcm9tcHQ+IG9iX2NvbWlu dF9hc3luY19zaGVsbF9lbmRfZDc4YWM0OWYtZGM4YS00YzM5LTgyN2MtYzkzMjI1NDg0ZDU5Citv cmdfYmFiZWxfc2hfcHJvbXB0PiAiKSkKKyAgICAoc2hvdWxkIChzdHJpbmc9CisgICAgICAgICAg ICAgKG9yZy10cmltIChzdHJpbmctam9pbiAobWFwY2FyICMnb3JnLXRyaW0gKG9yZy1iYWJlbC1j b21pbnQtLXByb21wdC1maWx0ZXIgcmVzdWx0cyBwcm9tcHQpKSAiXG4iKSAiXG4iKQorICAgICAg ICAgICAgICJlY2hvICdvYl9jb21pbnRfYXN5bmNfc2hlbGxfc3RhcnRfZDc4YWM0OWYtZGM4YS00 YzM5LTgyN2MtYzkzMjI1NDg0ZDU5JworIyBwcmludCBtZXNzYWdlCitlY2hvIFwiaGVsbG8gd29y bGRcIgorZWNobyAnb2JfY29taW50X2FzeW5jX3NoZWxsX2VuZF9kNzhhYzQ5Zi1kYzhhLTRjMzkt ODI3Yy1jOTMyMjU0ODRkNTknCitvYl9jb21pbnRfYXN5bmNfc2hlbGxfc3RhcnRfZDc4YWM0OWYt ZGM4YS00YzM5LTgyN2MtYzkzMjI1NDg0ZDU5CitcImhlbGxvIHdvcmxkXCIKK29iX2NvbWludF9h c3luY19zaGVsbF9lbmRfZDc4YWM0OWYtZGM4YS00YzM5LTgyN2MtYzkzMjI1NDg0ZDU5IikpKSkK KworCisocHJvdmlkZSAndGVzdC1vYi1jb21pbnQpCisKKzs7OyB0ZXN0LW9iLWNvbWludC5lbCBl bmRzIGhlcmUKLS0gCjIuNDEuMAoK ------=_Part_2243495_1715428140.1710704206269 Content-Type: application/octet-stream; name=0002-lisp-ob-comint.el-Create-comint-prompt-filter.patch Content-Transfer-Encoding: 7bit X-ZM_AttachId: 139299970062740000 Content-Disposition: attachment; filename=0002-lisp-ob-comint.el-Create-comint-prompt-filter.patch >From ed5f3e358f2e32c12d3487de9ab17797370eaf81 Mon Sep 17 00:00:00 2001 From: Matthew Trzcinski Date: Sun, 17 Mar 2024 16:54:33 +0100 Subject: [PATCH 2/7] lisp/ob-comint.el: Create comint prompt filter * lisp/ob-comint.el (org-babel-comint--prompt-filter): Extract prompt filtering logic from `org-babel-comint-with-output' into a new function. --- lisp/ob-comint.el | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el index 1ec84e865..882e19289 100644 --- a/lisp/ob-comint.el +++ b/lisp/ob-comint.el @@ -74,6 +74,19 @@ This is useful when prompt unexpectedly changes." (setq comint-prompt-regexp org-babel-comint-prompt-regexp-old org-babel-comint-prompt-regexp-old tmp)))) +(defun org-babel-comint--prompt-filter (string &optional prompt-regexp) + "Remove PROMPT-REGEXP from STRING. + +PROMPT-REGEXP defaults to `comint-prompt-regexp'." + (let* ((prompt-regexp (or prompt-regexp comint-prompt-regexp)) + (separator "org-babel-comint--prompt-filter-separator\n")) + (while (string-match-p prompt-regexp string) + (setq string + (replace-regexp-in-string + (format "\\(?:%s\\)?\\(?:%s\\)[ \t]*" separator prompt-regexp) + separator string))) + (delete "" (split-string string separator)))) + (defmacro org-babel-comint-with-output (meta &rest body) "Evaluate BODY in BUFFER and return process output. Will wait until EOE-INDICATOR appears in the output, then return -- 2.41.0 ------=_Part_2243495_1715428140.1710704206269 Content-Type: application/octet-stream; name=0003-testing-lisp-test-ob-comint.el-Make-test-for-echo-fi.patch Content-Transfer-Encoding: 7bit X-ZM_AttachId: 139299970062760000 Content-Disposition: attachment; filename=0003-testing-lisp-test-ob-comint.el-Make-test-for-echo-fi.patch >From 44bccab228b9dee80672b3a3dd9d27b2cc26c4b3 Mon Sep 17 00:00:00 2001 From: Matthew Trzcinski Date: Sun, 17 Mar 2024 16:56:43 +0100 Subject: [PATCH 3/7] testing/lisp/test-ob-comint.el: Make test for echo filter * test-ob-comint.el: (test-org-babel-comint/echo-filter-removes-echo): Test that echoed input is removed from process buffer output. --- testing/lisp/test-ob-comint.el | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/testing/lisp/test-ob-comint.el b/testing/lisp/test-ob-comint.el index 9193ef8f3..8f6def8ca 100644 --- a/testing/lisp/test-ob-comint.el +++ b/testing/lisp/test-ob-comint.el @@ -49,6 +49,26 @@ ob_comint_async_shell_start_d78ac49f-dc8a-4c39-827c-c93225484d59 \"hello world\" ob_comint_async_shell_end_d78ac49f-dc8a-4c39-827c-c93225484d59")))) +(ert-deftest test-org-babel-comint/echo-filter-removes-echo () + "Test that echo is actually removed." + (let* ((echo "echo 'ob_comint_async_shell_start_d78ac49f-dc8a-4c39-827c-c93225484d59' +# print message +echo \"hello world\" +echo 'ob_comint_async_shell_end_d78ac49f-dc8a-4c39-827c-c93225484d59'") + (result "org_babel_sh_prompt> echo 'ob_comint_async_shell_start_d78ac49f-dc8a-4c39-827c-c93225484d59' +# print message +echo \"hello world\" +echo 'ob_comint_async_shell_end_d78ac49f-dc8a-4c39-827c-c93225484d59' +ob_comint_async_shell_start_d78ac49f-dc8a-4c39-827c-c93225484d59 +org_babel_sh_prompt> org_babel_sh_prompt> \"hello world\" +org_babel_sh_prompt> ob_comint_async_shell_end_d78ac49f-dc8a-4c39-827c-c93225484d59 +org_babel_sh_prompt> ")) + (should (string= + (org-babel-comint--echo-filter result echo) + "\nob_comint_async_shell_start_d78ac49f-dc8a-4c39-827c-c93225484d59 +org_babel_sh_prompt> org_babel_sh_prompt> \"hello world\" +org_babel_sh_prompt> ob_comint_async_shell_end_d78ac49f-dc8a-4c39-827c-c93225484d59 +org_babel_sh_prompt> ")))) (provide 'test-ob-comint) -- 2.41.0 ------=_Part_2243495_1715428140.1710704206269 Content-Type: application/octet-stream; name=0004-lisp-ob-comint.el-Create-comint-echo-filter.patch Content-Transfer-Encoding: 7bit X-ZM_AttachId: 139299970062770000 Content-Disposition: attachment; filename=0004-lisp-ob-comint.el-Create-comint-echo-filter.patch >From c629cb7041d25480cba82812ba5da41329f4e435 Mon Sep 17 00:00:00 2001 From: Matthew Trzcinski Date: Sun, 17 Mar 2024 16:58:52 +0100 Subject: [PATCH 4/7] lisp/ob-comint.el: Create comint echo filter * lisp/ob-comint.el (org-babel-comint--echo-filter): Extract echo filtering logic from `org-babel-comint-with-output' into a new function. --- lisp/ob-comint.el | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el index 882e19289..e8d8e7609 100644 --- a/lisp/ob-comint.el +++ b/lisp/ob-comint.el @@ -87,6 +87,15 @@ PROMPT-REGEXP defaults to `comint-prompt-regexp'." separator string))) (delete "" (split-string string separator)))) +(defun org-babel-comint--echo-filter (string &optional echo) + "Remove ECHO from STRING." + (and echo string + (string-match + (replace-regexp-in-string "\n" "[\r\n]+" (regexp-quote echo)) + string) + (setq string (substring string (match-end 0)))) + string) + (defmacro org-babel-comint-with-output (meta &rest body) "Evaluate BODY in BUFFER and return process output. Will wait until EOE-INDICATOR appears in the output, then return -- 2.41.0 ------=_Part_2243495_1715428140.1710704206269 Content-Type: application/octet-stream; name=0005-lisp-ob-comint.el-Refactor-org-babel-comint-with-out.patch Content-Transfer-Encoding: 7bit X-ZM_AttachId: 139299970062780000 Content-Disposition: attachment; filename=0005-lisp-ob-comint.el-Refactor-org-babel-comint-with-out.patch >From f9821b7ff50aeae096be2ebb55f6d920c0afa1e0 Mon Sep 17 00:00:00 2001 From: Matthew Trzcinski Date: Sun, 17 Mar 2024 18:24:43 +0100 Subject: [PATCH 5/7] lisp/ob-comint.el: Refactor `org-babel-comint-with-output' * lisp/ob-comint.el (org-babel-comint-with-output): Replace logic for prompt and echo filtering with `org-babel-comint--prompt-filter' and `org-babel-comint--echo-filter'. Delete `org-babel-comint-prompt-separator' variable and move related comment to `org-babel-comint--prompt-filter'. --- lisp/ob-comint.el | 39 +++++++++------------------------------ 1 file changed, 9 insertions(+), 30 deletions(-) diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el index e8d8e7609..d13aacccc 100644 --- a/lisp/ob-comint.el +++ b/lisp/ob-comint.el @@ -79,6 +79,9 @@ This is useful when prompt unexpectedly changes." PROMPT-REGEXP defaults to `comint-prompt-regexp'." (let* ((prompt-regexp (or prompt-regexp comint-prompt-regexp)) + ;; We need newline in case if we do progressive replacement + ;; of agglomerated comint prompts with `comint-prompt-regexp' + ;; containing ^. (separator "org-babel-comint--prompt-filter-separator\n")) (while (string-match-p prompt-regexp string) (setq string @@ -112,12 +115,7 @@ or user `keyboard-quit' during execution of body." (let ((buffer (nth 0 meta)) (eoe-indicator (nth 1 meta)) (remove-echo (nth 2 meta)) - (full-body (nth 3 meta)) - (org-babel-comint-prompt-separator - ;; We need newline in case if we do progressive replacement - ;; of agglomerated comint prompts with `comint-prompt-regexp' - ;; containing ^. - "org-babel-comint-prompt-separator\n")) + (full-body (nth 3 meta))) `(org-babel-comint-in-buffer ,buffer (let* ((string-buffer "") (comint-output-filter-functions @@ -161,31 +159,12 @@ or user `keyboard-quit' during execution of body." (goto-char (process-mark (get-buffer-process (current-buffer)))) (insert dangling-text) + ;; remove echo'd FULL-BODY from input + (and ,remove-echo ,full-body + (setq string-buffer (org-babel-comint--echo-filter string-buffer ,full-body))) + ;; Filter out prompts. - (while (string-match-p comint-prompt-regexp string-buffer) - (setq string-buffer - (replace-regexp-in-string - ;; Sometimes, we get multiple agglomerated - ;; prompts together in a single output: - ;; "prompt prompt prompt output" - ;; Or even "promptprompt ...>. - ;; Remove them progressively, so that - ;; possible "^" in the prompt regexp gets to - ;; work as we remove the heading prompt - ;; instance. - (format "\\(?:%s\\)?\\(?:%s\\)[ \t]*" ,org-babel-comint-prompt-separator comint-prompt-regexp) - ,org-babel-comint-prompt-separator - string-buffer))) - ;; remove echo'd FULL-BODY from input - (when (and ,remove-echo ,full-body - (string-match - (replace-regexp-in-string - "\n" "[\r\n]+" (regexp-quote (or ,full-body ""))) - string-buffer)) - (setq string-buffer (substring string-buffer (match-end 0)))) - (delete "" (split-string - string-buffer - ,org-babel-comint-prompt-separator)))))) + (org-babel-comint--prompt-filter string-buffer))))) (defun org-babel-comint-input-command (buffer cmd) "Pass CMD to BUFFER. -- 2.41.0 ------=_Part_2243495_1715428140.1710704206269 Content-Type: application/octet-stream; name=0006-testing-lisp-test-ob-shell.el-Test-async-prompt-remo.patch Content-Transfer-Encoding: 7bit X-ZM_AttachId: 139299970062800000 Content-Disposition: attachment; filename=0006-testing-lisp-test-ob-shell.el-Test-async-prompt-remo.patch >From 4f943d1e72c1e32f02723a53d4d8f5ee201dfbbb Mon Sep 17 00:00:00 2001 From: Matthew Trzcinski Date: Sun, 17 Mar 2024 18:27:11 +0100 Subject: [PATCH 6/7] testing/lisp/test-ob-shell.el: Test async prompt removal * testing/lisp/test-ob-shell.el (test-ob-shell/session-async-results): Create test verifying bug report that shell prompt appears in async results. --- testing/lisp/test-ob-shell.el | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/testing/lisp/test-ob-shell.el b/testing/lisp/test-ob-shell.el index 879555af0..8cebd8467 100644 --- a/testing/lisp/test-ob-shell.el +++ b/testing/lisp/test-ob-shell.el @@ -129,6 +129,30 @@ echo 2 (if (should (string= ": 1\n: 2\n" (buffer-substring-no-properties (point) (point-max)))) (kill-buffer session-name))))) +(ert-deftest test-ob-shell/session-async-results () + "Test that async evaluation removes prompt from results." + (let* ((session-name "test-ob-shell/session-async-results") + (kill-buffer-query-functions nil) + (start-time (current-time)) + (wait-time (time-add start-time 3)) + uuid-placeholder) + (org-test-with-temp-text + (concat "#+begin_src sh :session " session-name " :async t +# print message +echo \"hello world\" +#+end_src") + (setq uuid-placeholder (org-trim (org-babel-execute-src-block))) + (catch 'too-long + (while (string-match uuid-placeholder (buffer-string)) + (progn + (sleep-for 0.01) + (when (time-less-p wait-time (current-time)) + (throw 'too-long (ert-fail "Took too long to get result from callback")))))) + (search-forward "#+results") + (beginning-of-line 2) + (if (should (string= ": hello world\n" (buffer-substring-no-properties (point) (point-max)))) + (kill-buffer session-name))))) + (ert-deftest test-ob-shell/generic-uses-no-arrays () "Test generic serialization of array into a single string." (org-test-with-temp-text -- 2.41.0 ------=_Part_2243495_1715428140.1710704206269 Content-Type: application/octet-stream; name=0007-lisp-ob-comint.el-Fix-prompt-appearing-in-async-shel.patch Content-Transfer-Encoding: 7bit X-ZM_AttachId: 139299970062810010 Content-Disposition: attachment; filename=0007-lisp-ob-comint.el-Fix-prompt-appearing-in-async-shel.patch >From abe14703390e9ae56829a118c3d4e488443f6845 Mon Sep 17 00:00:00 2001 From: Matthew Trzcinski Date: Sun, 17 Mar 2024 19:04:39 +0100 Subject: [PATCH 7/7] lisp/ob-comint.el: Fix prompt appearing in async shell results * lisp/ob-comint.el (org-babel-comint-async-filter): Call prompt `org-babel-comint--prompt-filter' Reported-by: "Matthew Trzcinski" Link: https://list.orgmode.org/18d753c1e8a.cfb3e1921191837.5665565128507976741@excalamus.com/ --- lisp/ob-comint.el | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el index d13aacccc..f2251892a 100644 --- a/lisp/ob-comint.el +++ b/lisp/ob-comint.el @@ -325,9 +325,10 @@ STRING contains the output originally inserted into the comint buffer." until (and (equal (match-string 1) "start") (equal (match-string 2) uuid)) finally return (+ 1 (match-end 0))))) - ;; Apply callback to clean up the result - (res-str (funcall org-babel-comint-async-chunk-callback - res-str-raw))) + ;; Remove prompt + (res-promptless (org-trim (string-join (mapcar #'org-trim (org-babel-comint--prompt-filter res-str-raw)) "\n") "\n")) + ;; Apply user callback + (res-str (funcall org-babel-comint-async-chunk-callback res-promptless))) ;; Search for uuid in associated org-buffers to insert results (cl-loop for buf in org-buffers until (with-current-buffer buf -- 2.41.0 ------=_Part_2243495_1715428140.1710704206269--