From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms13.migadu.com with LMTPS id kI9jGM3qFWdfMgEAe85BDQ:P1 (envelope-from ) for ; Mon, 21 Oct 2024 05:46:53 +0000 Received: from aspmx1.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2.migadu.com with LMTPS id kI9jGM3qFWdfMgEAe85BDQ (envelope-from ) for ; Mon, 21 Oct 2024 07:46:53 +0200 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=H4BsgJKy; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1729489613; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=zqMGK0LNkxIy2otUsyJUlWDWTnjy1iH+3EHkhU3iurs=; b=HOm352+8ggDdlTi+KDo41f0sU/sqg8Z7m1mbgvRmcMzBtn/ZNw0pFa13+TaVOjcZKWwfk7 I95xTex84Ah/8NM0RIANf8kjfgQDdtP2+JUYiToQtVj8Fka9uTUV/ORFcGE+zLOBfJe7/c ZvgZ3OE60QKKPdlSOLeg255nQBfI4DGBzKn+n+ouT9hije15W1McJRjPxp1qXr5knK3ci8 IsIrixtOCq4q/44uP+IOjTfmd16C6o+Abexe3uhdTw40uPEI7+xstgYrZKw0+Bf3cCw6KX P/7e5NZo2NoAy9pyV0To/Uz3RA7yFqFAb4/noB57odpcfaw5JFgvrM+cYGDkjQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1729489613; a=rsa-sha256; cv=none; b=q+dhWvvGU7Oh3GVuIt0WawQ6IaRtQj47ngtpaDvH/qziHpIMCnBOiw+szG5dmHYgiGHor8 gaAnb+Gt3iFZthDspH5blU2742+vAjVawRdXmwwUOjDsca4z/x5GNchRb/t7dkjjsp/KKQ 05FTLPW9JwR2zEKIF1tdP49hQ0z+nkwBOvU87si9O9z5qrCVdrXUhFa539FQdBVvJwKqSw 8T0sH+1SIW5/iW1wRe/9mFL5IZ782CErt4W2oAcvSfWy38kUl/6Dg3vu1gkf2CMB31kCcY GKB0jC2wGErPXygQpFY8ugw/UAba4z/7orJPF6YO8Myx0vakpHaGjLDCVhsYrw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=H4BsgJKy; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gmail.com Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 9CD6284B2A for ; Mon, 21 Oct 2024 07:46:52 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1t2lET-0002mw-Pk; Mon, 21 Oct 2024 01:45:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1t2lEF-0002mL-St for emacs-orgmode@gnu.org; Mon, 21 Oct 2024 01:45:29 -0400 Received: from mail-pg1-x530.google.com ([2607:f8b0:4864:20::530]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1t2lEC-00040s-U0 for emacs-orgmode@gnu.org; Mon, 21 Oct 2024 01:45:19 -0400 Received: by mail-pg1-x530.google.com with SMTP id 41be03b00d2f7-7db908c9c83so2509653a12.2 for ; Sun, 20 Oct 2024 22:45:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1729489515; x=1730094315; darn=gnu.org; h=mime-version:message-id:date:references:in-reply-to:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=zqMGK0LNkxIy2otUsyJUlWDWTnjy1iH+3EHkhU3iurs=; b=H4BsgJKyrukTTZCB3nATJ0aUilJFUmF6MT2gHJ8/a098nHNOKmpaffeiRNmzjBWUzY Lvmiejt6gLIzkC3wSJDzVPX5XWxmj9h90NIHpPqP0aTVVb/AtsDd+9XIjN+Io0P2fIUA nm7zCjkPtzrP1AoUy/IvJzm0mKBLyJCFOZI+7AhoekahnLPn9NI2pc4O9Gnp9QTr1TwQ oIdvWTxl0x3d/un7qBHok+Nn/HJvxgNFEMYdMZ8YK0iCicE6g6q8IgqiqhUIM6Xrp1CI nUcKjT0C++gQK8lgTlttj4KwCW235hR2aEWHajo8LHit2uFLuyVL9ZfQQwgLx1wX7Q/u XAcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729489515; x=1730094315; h=mime-version:message-id:date:references:in-reply-to:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zqMGK0LNkxIy2otUsyJUlWDWTnjy1iH+3EHkhU3iurs=; b=UCOoGlV1ArAetfsTH09ksdHm2PjTRvCpPXoyhHg9DZ6CJ0cY6UvqjjM/L/6Xy+4811 9rHvvFsfBCH2ZCBQiRNbKuPb5w/HUNvsax0pHImBixaFYT9LcipHbnwyI0LnMC0xP9iS UdDjnKLi7pNuDHrOTaCYwu8R8KUek6Di7/GvdCFOG22yKadwI5gJ7kmLvdhm2p+CSB50 MtLroCcRvbgVS5SZ/2H6dxYuDZbW2vlSDZq/4MJ8hHJv33YcDp0kPJYzvq87IvABPWK+ IO3HnPxiTGFatJ5L8gcUDrZhB2D4v3OOR0ykGQ4rYPnC//79gYB36xOcqtg6d72sald8 uQgQ== X-Gm-Message-State: AOJu0YzgWxZLJz8gDQtEIqCDbIFMoHWA8kXaenFiYT1yHTMPxjsjXRKS t+N02k1pVSd/HLOqiIfcPKsT0Do8SO2OrTmYFBPVpWUeVOdE+I3D X-Google-Smtp-Source: AGHT+IE81/ovamIKTycjKZ8G5TO59a7mXZX1/kQSHqcFdQ4LT2IwOKcnK35RzSd+utvaL/OISjGFFQ== X-Received: by 2002:a17:90a:be09:b0:2e2:e6c8:36a7 with SMTP id 98e67ed59e1d1-2e5618d0cbdmr11774773a91.31.1729489514495; Sun, 20 Oct 2024 22:45:14 -0700 (PDT) Received: from localhost ([198.27.183.102]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2e5ad4ee3b0sm2596079a91.42.2024.10.20.22.45.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 20 Oct 2024 22:45:13 -0700 (PDT) From: Jack Kamm To: Ihor Radchenko Cc: emacs-orgmode@gnu.org, jeremiejuste@gmail.com Subject: Re: [PATCH] ob-comint,R,python: Options for more robust non-async session output In-Reply-To: <87v7xowod1.fsf@localhost> References: <87h6993y6q.fsf@gmail.com> <87v7xowod1.fsf@localhost> Date: Sun, 20 Oct 2024 22:45:12 -0700 Message-ID: <878qui3t7b.fsf@gmail.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Received-SPF: pass client-ip=2607:f8b0:4864:20::530; envelope-from=jackkamm@gmail.com; helo=mail-pg1-x530.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: emacs-orgmode-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -4.82 X-Spam-Score: -4.82 X-Migadu-Queue-Id: 9CD6284B2A X-Migadu-Scanner: mx13.migadu.com X-TUID: IlZxMRHwnKBg --=-=-= Content-Type: text/plain Ihor Radchenko writes: > Previously, the point was always at process-mark when BODY is > called. Now, it may not be the case. It may cause problems. On the one hand -- I'm not sure I agree that it would cause problems. IMO setting point to process mark goes hand-in-hand with cutting the dangling text -- so that text can be inserted into the comint buffer at process mark. And the default behavior remains to cut dangling text and move point to process mark, so nothing should change for those languages that rely on this behavior. OTOH -- I'm not sure the option to skip cutting dangling is really necessary anymore. While the ob-python-style approach (which relies on `process-send-string', or more specifically `python-shell-send-string') doesn't require moving point or cutting the dangling text -- it doesn't seem to affect anything either, since BODY is wrapped in `save-excursion' (via `org-babel-comint-in-buffer'). In an earlier version of this patch I did see some weird behavior that I thought was due to cutting the dangling text, but I can't reproduce it anymore. So for now, I think it's better to just remove the NO-SAVE-DANGLING option. If we find we need this option later for some reason, we can figure out the details about point/process-mark then. I've reattached the patch with this option removed, and also rebased onto latest main. --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename=0001-Add-options-to-skip-extra-processing-in-org-babel-co.patch >From 629788caa86773199fc06af1797cf8fd67a7884a Mon Sep 17 00:00:00 2001 From: Jack Kamm Date: Thu, 17 Oct 2024 17:33:47 -0700 Subject: [PATCH] Add options to skip extra processing in org-babel-comint-with-output This patch adds options to org-babel-comint-with-output to skip prompt removal. Allowing individual languages to handle this cleanup can be more robust than relying on the generic ob-comint implementation. This allows ob-python to switch back to using `org-babel-comint-with-output' rather than its own bespoke reimplementation, reducing code duplication. Furthermore, this adds a new implementation of ob-R non-async session output evaluation, that is similar to the ob-python approach in that it avoids leaking prompts, rather than relying on the cleanup from `org-babel-comint-with-output'. A test is added to test-ob-R.el to demonstrate the improved robustness of the new approach; previously, this test would fail due to a false positive prompt, but now passes. * lisp/ob-comint.el (org-babel-comint-with-output): Add new arguments to prevent extra processing for prompt cleanup. Also, search for the end-of-execution sentinel within the collected output rather than the comint buffer, which allows for evaluation via `process-send-string' (or similar functions like `python-shell-send-string' and `'ess-send-string') rather than directly entering the code block into the comint buffer. * lisp/ob-python.el (org-babel-python-send-string): Switch to using `org-babel-comint-with-output', rather than bespoke reimplementation. * lisp/ob-R.el (ess-send-string): Declare external function. (org-babel-R-evaluate-session): New implementation of output evaluation that avoids leaking prompts, by writing the code block to a tmp file and then sourcing it. * testing/lisp/test-ob-R.el (test-ob-r/session-output-with->-bol): New test for robustness against false positive prompts at the beginning of a line. --- lisp/ob-R.el | 33 +++++++++++----------------- lisp/ob-comint.el | 46 ++++++++++++++++++++++----------------- lisp/ob-python.el | 26 ++++++---------------- testing/lisp/test-ob-R.el | 12 ++++++++++ 4 files changed, 58 insertions(+), 59 deletions(-) diff --git a/lisp/ob-R.el b/lisp/ob-R.el index 481212202..fb29590e0 100644 --- a/lisp/ob-R.el +++ b/lisp/ob-R.el @@ -42,6 +42,8 @@ (declare-function ess-make-buffer-current "ext:ess-inf" ()) (declare-function ess-eval-buffer "ext:ess-inf" (vis)) (declare-function ess-wait-for-process "ext:ess-inf" (&optional proc sec-prompt wait force-redisplay)) +(declare-function ess-send-string "ext:ess-inf" + (process string &optional visibly message type)) (defvar ess-current-process-name) ; ess-custom.el (defvar ess-local-process-name) ; ess-custom.el @@ -448,26 +450,17 @@ (defun org-babel-R-evaluate-session (org-babel-import-elisp-from-file tmp-file '(16))) column-names-p))) (output - (mapconcat - 'org-babel-chomp - (butlast - (delq nil - (mapcar - (lambda (line) (when (> (length line) 0) line)) - (mapcar - (lambda (line) ;; cleanup extra prompts left in output - (if (string-match - "^\\([>+.]\\([ ][>.+]\\)*[ ]\\)" - (car (split-string line "\n"))) - (substring line (match-end 1)) - line)) - (with-current-buffer session - (let ((comint-prompt-regexp (concat "^" comint-prompt-regexp))) - (org-babel-comint-with-output (session org-babel-R-eoe-output) - (insert (mapconcat 'org-babel-chomp - (list body org-babel-R-eoe-indicator) - "\n")) - (inferior-ess-send-input)))))))) "\n")))) + (let ((tmp-src-file (org-babel-temp-file "R-"))) + (with-temp-file tmp-src-file + (insert (concat + (org-babel-chomp body) "\n" org-babel-R-eoe-indicator))) + (with-current-buffer session + (org-babel-comint-with-output + (session org-babel-R-eoe-output nil nil t) + (ess-send-string (get-buffer-process (current-buffer)) + (format "source('%s', echo=F, print.eval=T)" + (org-babel-process-file-name + tmp-src-file 'noquote))))))))) (defun org-babel-R-process-value-result (result column-names-p) "R-specific processing of return value. diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el index b88ac445a..a21c50ff1 100644 --- a/lisp/ob-comint.el +++ b/lisp/ob-comint.el @@ -105,11 +105,15 @@ (defmacro org-babel-comint-with-output (meta &rest body) "Evaluate BODY in BUFFER and return process output. Will wait until EOE-INDICATOR appears in the output, then return all process output. If REMOVE-ECHO and FULL-BODY are present and -non-nil, then strip echo'd body from the returned output. META -should be a list containing the following where the last two -elements are optional. +non-nil, then strip echo'd body from the returned output. If +NO-CLEANUP-PROMPT is nil, prompts are detected in the output, and +the returned value is a list of the output split on the prompt +positions; if non-nil, suppress that behavior, and just return a +single string of all the output up to EOE-INDICATOR. META should +be a list containing the following where the last three elements +are optional. - (BUFFER EOE-INDICATOR REMOVE-ECHO FULL-BODY) + (BUFFER EOE-INDICATOR REMOVE-ECHO FULL-BODY NO-CLEANUP-PROMPT) This macro ensures that the filter is removed in case of an error or user `keyboard-quit' during execution of body." @@ -117,7 +121,8 @@ (defmacro org-babel-comint-with-output (meta &rest body) (let ((buffer (nth 0 meta)) (eoe-indicator (nth 1 meta)) (remove-echo (nth 2 meta)) - (full-body (nth 3 meta))) + (full-body (nth 3 meta)) + (no-cleanup-prompt (nth 4 meta))) `(org-babel-comint-in-buffer ,buffer (let* ((string-buffer "") (comint-output-filter-functions @@ -125,7 +130,7 @@ (defmacro org-babel-comint-with-output (meta &rest body) (setq string-buffer (concat string-buffer text))) comint-output-filter-functions)) dangling-text) - ;; got located, and save dangling text + ;; got located, and save dangling text (goto-char (process-mark (get-buffer-process (current-buffer)))) (let ((start (point)) (end (point-max))) @@ -135,13 +140,11 @@ (defmacro org-babel-comint-with-output (meta &rest body) ,@body ;; wait for end-of-evaluation indicator (let ((start-time (current-time))) - (while (progn - (goto-char comint-last-input-end) - (not (save-excursion - (and (re-search-forward - (regexp-quote ,eoe-indicator) nil t) - (re-search-forward - comint-prompt-regexp nil t))))) + (while (not (save-excursion + (and (string-match + (regexp-quote ,eoe-indicator) string-buffer) + (string-match + comint-prompt-regexp string-buffer)))) (accept-process-output (get-buffer-process (current-buffer)) org-babel-comint-fallback-regexp-threshold) @@ -152,21 +155,24 @@ (defmacro org-babel-comint-with-output (meta &rest body) (goto-char comint-last-input-end) (save-excursion (and - (re-search-forward - (regexp-quote ,eoe-indicator) nil t) - (re-search-forward - org-babel-comint-prompt-regexp-fallback nil t))))) + (string-match + (regexp-quote ,eoe-indicator) string-buffer) + (string-match + org-babel-comint-prompt-regexp-fallback string-buffer))))) (org-babel-comint--set-fallback-prompt)))) ;; replace cut dangling text - (goto-char (process-mark (get-buffer-process (current-buffer)))) + (goto-char (process-mark (get-buffer-process (current-buffer)))) (insert dangling-text) ;; remove echo'd FULL-BODY from input (and ,remove-echo ,full-body (setq string-buffer (org-babel-comint--echo-filter string-buffer ,full-body))) - ;; Filter out prompts. - (org-babel-comint--prompt-filter string-buffer))))) + (if ,no-cleanup-prompt + (save-match-data + (string-match (regexp-quote ,eoe-indicator) string-buffer) + (org-babel-chomp (substring string-buffer 0 (match-beginning 0)))) + (org-babel-comint--prompt-filter string-buffer)))))) (defun org-babel-comint-input-command (buffer cmd) "Pass CMD to BUFFER. diff --git a/lisp/ob-python.el b/lisp/ob-python.el index 8a3c24f70..ceade40ee 100644 --- a/lisp/ob-python.el +++ b/lisp/ob-python.el @@ -451,31 +451,19 @@ (defun org-babel-python-evaluate-external-process (defun org-babel-python-send-string (session body) "Pass BODY to the Python process in SESSION. Return output." - (with-current-buffer session - (let* ((string-buffer "") - (comint-output-filter-functions - (cons (lambda (text) (setq string-buffer - (concat string-buffer text))) - comint-output-filter-functions)) - (body (format "\ + (org-babel-comint-with-output + ((org-babel-session-buffer:python session) + org-babel-python-eoe-indicator + nil nil t) + (python-shell-send-string (format "\ try: %s except: raise finally: print('%s')" - (org-babel-python--shift-right body 4) - org-babel-python-eoe-indicator))) - (let ((python-shell-buffer-name - (org-babel-python-without-earmuffs session))) - (python-shell-send-string body)) - ;; same as `python-shell-comint-end-of-output-p' in emacs-25.1+ - (while (not (and (python-shell-comint-end-of-output-p string-buffer) - (string-match - org-babel-python-eoe-indicator - string-buffer))) - (accept-process-output (get-buffer-process (current-buffer)))) - (org-babel-chomp (substring string-buffer 0 (match-beginning 0)))))) + (org-babel-python--shift-right body 4) + org-babel-python-eoe-indicator)))) (defun org-babel-python-evaluate-session (session body &optional result-type result-params graphics-file) diff --git a/testing/lisp/test-ob-R.el b/testing/lisp/test-ob-R.el index 0d291bf54..b8dcaa973 100644 --- a/testing/lisp/test-ob-R.el +++ b/testing/lisp/test-ob-R.el @@ -126,6 +126,18 @@ (ert-deftest test-ob-r/output-with-<> () )))) +(ert-deftest test-ob-r/session-output-with->-bol () + "make sure prompt-like strings are well formatted, even when at beginning of line." + (let (ess-ask-for-ess-directory ess-history-file) + (should (string="abc +def>