From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0.migadu.com ([2001:41d0:403:58f0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms13.migadu.com with LMTPS id OF+NHEH9HmcnJwEAqHPOHw:P1 (envelope-from ) for ; Mon, 28 Oct 2024 02:56:01 +0000 Received: from aspmx1.migadu.com ([2001:41d0:403:58f0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0.migadu.com with LMTPS id OF+NHEH9HmcnJwEAqHPOHw (envelope-from ) for ; Mon, 28 Oct 2024 03:56:01 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=fpoQ6h1v; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1730084161; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=Zw2EZhg7W0VUMIZBikGpmkWb1YlhmPGMO5HVg7yo8MY=; b=XQb3CTvCs6RqzdxZF9nufvs07VFZIRrUKe9CjQKUCP4uFID6jYUxUbFWdhKDfmglFmwzHb NzQf2kgc4VHLDbJieLcBbqDkoxO/x/yPtmXHRexGAI3WviZkuT0Mq+3RhA0wl/bYKgcF/Q b81fysGJPVorFhYtmIiIjEXEbgcfffaC9Ml5lhsqXWJlzWbkhPJG6iWfAEjlgzg1PAuYNZ 4kMx4AJRewqghDQvhOJFJktpF/hVuK1xM93IxKrLHqOQN+VJz1TJ+is3WuzWpWi+rgjuEz /pRYZdw4PJ8JKhtE6kwlzhVUS2Ejo8+wiyNDB4b2Yyl7tC+uGXDTgnHh+H3rqg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=fpoQ6h1v; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=key1; d=yhetil.org; t=1730084161; a=rsa-sha256; cv=none; b=quT0hkBuKzW3l4YlmWwmwxrYCDei036w9qZ36pScbqwpcOHgVAreUhR9jumHenrdmm0P50 M3nICcgTrSMLkoCfP/Fg/St0eP8GuAYm/8zOBBt2wJyJP/c+kp5sn9trvRxMlGOnd4chfI +SR71dMvcLZUOPVi7QQ17vq0sp+lSVqIxFV6Phj4g9TKUzpohHYbco/dLEFyKmGT57ALSU FzXpzJ+3ZMz/KzKRrvI8H68om2INMSAdiZuddhNV6/yIVXACT09GcAibh0oKy1Y8zofc2s QsOYqhrLz4LUKvSI9ppLVI+W27cdqaiGylbqNKi7yyHZQiuy9MHj3JI2Y+aPiA== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 08C2D42E72 for ; Mon, 28 Oct 2024 03:56:00 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t5Fub-0001vS-U9; Sun, 27 Oct 2024 22:55:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t5FuY-0001uv-87 for emacs-orgmode@gnu.org; Sun, 27 Oct 2024 22:55:18 -0400 Received: from mail-oo1-xc35.google.com ([2607:f8b0:4864:20::c35]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1t5FuW-00040f-2C for emacs-orgmode@gnu.org; Sun, 27 Oct 2024 22:55:17 -0400 Received: by mail-oo1-xc35.google.com with SMTP id 006d021491bc7-5eb5f07410aso1699623eaf.0 for ; Sun, 27 Oct 2024 19:55:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1730084114; x=1730688914; darn=gnu.org; h=mime-version:message-id:date:references:in-reply-to:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=Zw2EZhg7W0VUMIZBikGpmkWb1YlhmPGMO5HVg7yo8MY=; b=fpoQ6h1visQNUOoNdL9iUBFNfLwdnyyWW3Hg4XEo1c6I3UPJlpQ3wttSR6mGXGEfRT 4Z1tY81k1jFksh8lzB1sinCbpN+av1yqKxKZ9Zg72bWwcRu0X3629ntyjOY7ImDDXlKz Vk1O8NFIJoVeLGDuljgCj6FZo/8HKOEE4nUEePwaXrDNyYna45PHd5H2er6LCzlouV3P Ecg+yMMk1H8siQnCm2vu379eiiXRNLPTiZoMf+KEswbMcNQl6mzp+ak/9yszb30JXwDr CxjW67qQ6ZF/fscYoHrP7PFBu2MAo2pdYVdfmmfGBLt3pJI6maOlPX/VudS+r4LRIBi2 Rsgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730084114; x=1730688914; h=mime-version:message-id:date:references:in-reply-to:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Zw2EZhg7W0VUMIZBikGpmkWb1YlhmPGMO5HVg7yo8MY=; b=TzJxiYE0qR4ZpAWnb5NCemzcIZn8AIdah3wsI5GGnAVa4Yj0x6fapNoVqvYjgm+wgm 8lcsBk8nYWpf7VWmknjjQTdfPPDnPrZVywFYtCfcI+7+o60L2qkBYGQ1yp9D66BKpZ4U lPhzo6iWfuun2jOIYMNz5mwfNPOwyJ8/Wi/AEXpLYM29a8/TIyf40x/BC5voQsCld1LC s3fjby6LLm6lL9+gOAfjR0P7oPMaZI8DMhOq9d78yriutX+jvE/OEYQUkCjM4EuhiN19 jsqCYH/ss+8UHSv20t63h+JuIDX/tgCQ5oPetGYRl9fcOXnroPvYPZD+rq8U5xzkNTWH hYTg== X-Gm-Message-State: AOJu0YwKzC9t/1uoysQBmbP9XKDF6fStflefCRFf3grHW6Nn4hSfTBXq PHDsIihyNisvIBjyS00NEN5ZxTVPBY/t63Spo9+Y60b61rpg2ZiP X-Google-Smtp-Source: AGHT+IEm+IqOhbcK/CVtG0O1aipRNGb7xjri/BO5+FmLfpa799EGr8twlZMHwy9R8svPaAIdbmohuQ== X-Received: by 2002:a05:6870:a10b:b0:288:6779:31a3 with SMTP id 586e51a60fabf-29051ddca2emr4647557fac.45.1730084114031; Sun, 27 Oct 2024 19:55:14 -0700 (PDT) Received: from localhost ([198.27.183.102]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72057921601sm4723063b3a.41.2024.10.27.19.55.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 27 Oct 2024 19:55:13 -0700 (PDT) From: Jack Kamm To: Ihor Radchenko Cc: emacs-orgmode@gnu.org, matt@excalamus.com, jeremiejuste@gmail.com Subject: Re: [PATCH] Async sessions: Fix prompt removal regression in ob-R In-Reply-To: <87ed4812ij.fsf@localhost> References: <87setrqs4z.fsf@gmail.com> <87wmiqigfp.fsf@localhost> <87wmi9etku.fsf@gmail.com> <87h698wml2.fsf@localhost> <87ed4b45r7.fsf@gmail.com> <875xpnrubg.fsf@localhost> <8734koert4.fsf@gmail.com> <87ed4812ij.fsf@localhost> Date: Sun, 27 Oct 2024 19:55:12 -0700 Message-ID: <87ldy9gcmn.fsf@gmail.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Received-SPF: pass client-ip=2607:f8b0:4864:20::c35; envelope-from=jackkamm@gmail.com; helo=mail-oo1-xc35.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: emacs-orgmode-bounces+larch=yhetil.org@gnu.org X-Migadu-Country: US X-Migadu-Flow: FLOW_IN X-Migadu-Scanner: mx11.migadu.com X-Migadu-Spam-Score: -1.52 X-Spam-Score: -1.52 X-Migadu-Queue-Id: 08C2D42E72 X-TUID: rMwD9YHdp3JT --=-=-= Content-Type: text/plain Ihor Radchenko writes: >> + (t (error (format "Unrecognized prompt handling behavior %s" >> + (symbol-name prompt-handling)))))) > > I think that `symbol-name' is unnecessary here. > It will lead to cryptic error if PROMPT-HANDLING happens to be something > that is not symbol (for example, a number) > >> +*** ~org-babel-comint-async-register~: Added argument to specify prompt handling > > Maybe "New optional argument controlling prompt handling" > >> +The new argument ~prompt-handling~ allows Babel languages to specify >> +how prompts should be handled when passing output to >> +~org-babel-comint-async-chunk-callback~. If equal to >> +~filter-prompts~, prompts are removed beforehand, similar to the > > similar -> same (or did I miss something?) Thanks for the suggestions, all of which I agree with. I've updated the patches accordingly, and also rebased them to most recent versions of bugfix/main. --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename=0001-Disable-async-prompt-removal-in-ob-R-python.patch >From 16dfc0d4500ab941333a9e705b2ddac85c30efdc Mon Sep 17 00:00:00 2001 From: Jack Kamm Date: Sun, 22 Sep 2024 13:48:45 -0700 Subject: [PATCH] Disable async prompt removal in ob-R,python * lisp/ob-comint.el (org-babel-comint-async-remove-prompts-p): New variable to disable prompt removal in async output. (org-babel-comint-async-filter): Check `org-babel-comint-async-remove-prompts-p' before calling `org-babel-comint--prompt-filter'. (org-babel-comint-async-register): Added argument for whether prompts should be removed from async output. * lisp/ob-python.el (org-babel-python-async-evaluate-session): Set option to inhibit prompt removal when registering async evaluators. * lisp/ob-R.el (ob-session-async-org-babel-R-evaluate-session): Set option to inhibit prompt removal when registering async evaluators. * testing/lisp/test-ob-R.el (test-ob-R/async-prompt-filter): Test for over-aggressive prompt removal. --- lisp/ob-R.el | 3 ++- lisp/ob-comint.el | 34 ++++++++++++++++++++++++++++------ lisp/ob-python.el | 3 ++- testing/lisp/test-ob-R.el | 28 ++++++++++++++++++++++++++++ 4 files changed, 60 insertions(+), 8 deletions(-) diff --git a/lisp/ob-R.el b/lisp/ob-R.el index 8074496f8..e9a1ce768 100644 --- a/lisp/ob-R.el +++ b/lisp/ob-R.el @@ -486,7 +486,8 @@ (defun ob-session-async-org-babel-R-evaluate-session session (current-buffer) "^\\(?:[>.+] \\)*\\[1\\] \"ob_comint_async_R_\\(start\\|end\\|file\\)_\\(.+\\)\"$" 'org-babel-chomp - 'ob-session-async-R-value-callback) + 'ob-session-async-R-value-callback + 'disable-prompt-filtering) (cl-case result-type (value (let ((tmp-file (org-babel-temp-file "R-"))) diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el index 764927af7..be5b41331 100644 --- a/lisp/ob-comint.el +++ b/lisp/ob-comint.el @@ -239,6 +239,9 @@ (defvar-local org-babel-comint-async-chunk-callback nil comint process. It should return a string that will be passed to `org-babel-insert-result'.") +(defvar-local org-babel-comint-async-remove-prompts-p t + "Whether prompts should be detected and removed from async output.") + (defvar-local org-babel-comint-async-dangling nil "Dangling piece of the last process output, as a string. Used when `org-babel-comint-async-indicator' is spread across multiple @@ -326,10 +329,16 @@ (defun org-babel-comint-async-filter (string) until (and (equal (match-string 1) "start") (equal (match-string 2) uuid)) finally return (+ 1 (match-end 0))))) - ;; Remove prompt - (res-promptless (org-trim (string-join (mapcar #'org-trim (org-babel-comint--prompt-filter res-str-raw)) "\n") "\n")) ;; Apply user callback - (res-str (funcall org-babel-comint-async-chunk-callback res-promptless))) + (res-str (funcall org-babel-comint-async-chunk-callback + (if org-babel-comint-async-remove-prompts-p + (org-trim (string-join + (mapcar #'org-trim + (org-babel-comint--prompt-filter + res-str-raw)) + "\n") + t) + res-str-raw)))) ;; Search for uuid in associated org-buffers to insert results (cl-loop for buf in org-buffers until (with-current-buffer buf @@ -350,18 +359,31 @@ (defun org-babel-comint-async-filter (string) (defun org-babel-comint-async-register (session-buffer org-buffer indicator-regexp - chunk-callback file-callback) + chunk-callback file-callback + &optional prompt-handling) "Set local org-babel-comint-async variables in SESSION-BUFFER. ORG-BUFFER is added to `org-babel-comint-async-buffers' if not present. `org-babel-comint-async-indicator', `org-babel-comint-async-chunk-callback', and `org-babel-comint-async-file-callback' are set to -INDICATOR-REGEXP, CHUNK-CALLBACK, and FILE-CALLBACK -respectively." +INDICATOR-REGEXP, CHUNK-CALLBACK, and FILE-CALLBACK respectively. +PROMPT-HANDLING may be either of the symbols `filter-prompts', in +which case prompts matching `comint-prompt-regexp' are filtered +from output before it is passed to CHUNK-CALLBACK, or +`disable-prompt-filtering', in which case this behavior is +disabled. For backward-compatibility, the default value of `nil' +is equivalent to `filter-prompts'." (org-babel-comint-in-buffer session-buffer (setq org-babel-comint-async-indicator indicator-regexp org-babel-comint-async-chunk-callback chunk-callback org-babel-comint-async-file-callback file-callback) + (setq org-babel-comint-async-remove-prompts-p + (cond + ((eq prompt-handling 'disable-prompt-filtering) nil) + ((eq prompt-handling 'filter-prompts) t) + ((eq prompt-handling nil) t) + (t (error (format "Unrecognized prompt handling behavior %s" + prompt-handling))))) (unless (memq org-buffer org-babel-comint-async-buffers) (setq org-babel-comint-async-buffers (cons org-buffer org-babel-comint-async-buffers))) diff --git a/lisp/ob-python.el b/lisp/ob-python.el index f881918c7..9975e83be 100644 --- a/lisp/ob-python.el +++ b/lisp/ob-python.el @@ -538,7 +538,8 @@ (defun org-babel-python-async-evaluate-session (org-babel-comint-async-register session (current-buffer) "ob_comint_async_python_\\(start\\|end\\|file\\)_\\(.+\\)" - 'org-babel-chomp 'org-babel-python-async-value-callback) + 'org-babel-chomp 'org-babel-python-async-value-callback + 'disable-prompt-filtering) (pcase result-type (`output (let ((uuid (org-id-uuid))) diff --git a/testing/lisp/test-ob-R.el b/testing/lisp/test-ob-R.el index 9ffbf3afd..05b91afd6 100644 --- a/testing/lisp/test-ob-R.el +++ b/testing/lisp/test-ob-R.el @@ -316,6 +316,34 @@ (org-test-with-temp-text-in-file (string= (concat text result) (buffer-string))))))) +(ert-deftest test-ob-R/async-prompt-filter () + "Test that async evaluation doesn't remove spurious prompts and leading indentation." + (let* (ess-ask-for-ess-directory + ess-history-file + org-confirm-babel-evaluate + (session-name "*R:test-ob-R/session-async-results*") + (kill-buffer-query-functions nil) + (start-time (current-time)) + (wait-time (time-add start-time 3)) + uuid-placeholder) + (org-test-with-temp-text + (concat "#+begin_src R :session " session-name " :async t :results output +table(c('ab','ab','c',NA,NA), useNA='always') +#+end_src") + (setq uuid-placeholder (org-trim (org-babel-execute-src-block))) + (catch 'too-long + (while (string-match uuid-placeholder (buffer-string)) + (progn + (sleep-for 0.01) + (when (time-less-p wait-time (current-time)) + (throw 'too-long (ert-fail "Took too long to get result from callback")))))) + (search-forward "#+results") + (beginning-of-line 2) + (when (should (re-search-forward "\ +:\\([ ]+ab\\)[ ]+c[ ]+[ ]* +:\\([ ]+2\\)[ ]+1[ ]+2")) + (should (equal (length (match-string 1)) (length (match-string 2)))) + (kill-buffer session-name))))) (provide 'test-ob-R) -- 2.46.2 --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename=0001-NEWS-entry-for-the-new-argument-of-org-babel-comint-.patch >From a4e66d0f66e04983d75d93ac84acf96d6e41d870 Mon Sep 17 00:00:00 2001 From: Jack Kamm Date: Mon, 21 Oct 2024 20:22:00 -0700 Subject: [PATCH] NEWS entry for the new argument of org-babel-comint-async-register The optional argument was added on bugfix branch, but we are adding this extra NEWS entry on main. See also: https://list.orgmode.org/875xpnrubg.fsf@localhost/T/#m179d313e1db284ff28eb4098129bb49418824a25 --- etc/ORG-NEWS | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS index 5d421172f..de4f11b25 100644 --- a/etc/ORG-NEWS +++ b/etc/ORG-NEWS @@ -186,6 +186,24 @@ accept the INFO channel and return a string. This makes it possible to dynamically generate the content of the resulting ~~ tag in the resulting HTML document. +*** ~org-babel-comint-async-register~: New optional argument controlling prompt handling + +The new argument ~prompt-handling~ allows Babel languages to specify +how prompts should be handled when passing output to +~org-babel-comint-async-chunk-callback~. If equal to +~filter-prompts~, prompts are removed beforehand, same as the +default behavior of ~org-babel-comint-with-output~. If equal to +~disable-prompt-filtering~, then the prompt filtering is skipped. If +unset, then the default behavior is the same as ~filter-prompts~ for +backwards compatibility. + +Prompt filtering is needed for some Babel languages, such as ob-shell, +which leave extra prompts in the output as a side effect of +evaluation. However other Babel languages, like ob-python, don't +leave extra prompts after evaluation, and skipping the prompt +filtering can be more robust for such languages (as this avoids +removing false positive prompts). + ** Miscellaneous *** Org mode no longer prevents =flyspell= from spell-checking inside =LOGBOOK= drawers -- 2.46.2 --=-=-=--