From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id yFyfDl5ErWKE2QAAbAwnHQ (envelope-from ) for ; Sat, 18 Jun 2022 05:19:58 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id wC+EDl5ErWKOOwEAauVa8A (envelope-from ) for ; Sat, 18 Jun 2022 05:19:58 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 5C9A99FC5 for ; Sat, 18 Jun 2022 05:19:57 +0200 (CEST) Received: from localhost ([::1]:58394 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1o2P07-0001Mu-Nz for larch@yhetil.org; Fri, 17 Jun 2022 23:19:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:59320) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1o2Oza-0001Mk-Sy for emacs-orgmode@gnu.org; Fri, 17 Jun 2022 23:19:22 -0400 Received: from mail-pg1-x529.google.com ([2607:f8b0:4864:20::529]:35821) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1o2OzY-0003G6-Mq for emacs-orgmode@gnu.org; Fri, 17 Jun 2022 23:19:22 -0400 Received: by mail-pg1-x529.google.com with SMTP id 193so2699768pgc.2 for ; Fri, 17 Jun 2022 20:19:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:in-reply-to:references:date:message-id :mime-version; bh=cXuE+xDB9MxJ+tGG2V7NCgdOvqGvoyRz5gdbrxfbaxM=; b=SVqxlSfELbIypaycFced+OQrnv4YBB30ofrgUn+mT1XQYiU2sit/S8xOuUyP4LBu8A c9pttS9+COhXbik5oRt3jtUqgJQMw4ogjyaGjf2hiNgnXOi1qgp7iunMgPW2fUoPmC60 Rinj+G7Iztqbyyclzm+X1A0EU+OW7A8FuW5h82e++9QHY86I1v8UxKJCyPYYJBkXRXfp sFbz4Ktc3yDBIeXpgS13MBmYuA+gxZmCUx5Hd+Th6Kgef/J5OxTbQFTVXjWg9x2Y0S5i yEvUsxhollfLd+eGg0SlCMRjHRiByS6+x8amtYHLeo73yBlCOJ0CMBGeBXckQeyJVEXM sQFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version; bh=cXuE+xDB9MxJ+tGG2V7NCgdOvqGvoyRz5gdbrxfbaxM=; b=iCHn5SpmtcP/rC4/4ZyoHsuvJ3VIOGgMXA786cy0n4LcLGy+sjuBTtsfY2HHk5N6GJ 7v8Xa1o+BD7hBVW/JV/bZE2zHEykDzusqu4cu/Y6R8HhqaAElelARPmo+6nrLEXn3fBw yGCDO0ozsUnR0B6FYKoJsm6zwD9dRXrN8kA+XKQEq00sZ7S/fhxEvEPpfs9SgNJzbYH4 NUSMIE9xVKJsE+iOTS/SM4/GOWi+vuUbGPwCegptoj1gNtMZhhhFc7HU6dJ7fTFcLq8y 8Wal6WN7aw4P9l3sHXkX78V7Yc/xhZgP02H8mt9uG4tgrCTV4hY+AZC/q+13aqCGUUll dt5A== X-Gm-Message-State: AJIora/qVvDW/4+CL9CCd0a+uVlVdk11JVUU37yMHecq6VBl1D+hIGNS HtNn1y7QM7WNreTQxKgWkjQ= X-Google-Smtp-Source: AGRyM1ulzOMC/dJzA8VXboc0tHgy6j/+f6CPVDyw1AeuhEpzP9mBX2vSogFqVMmYBsxTRpbX4l3nIw== X-Received: by 2002:aa7:82ca:0:b0:51b:cf43:d00a with SMTP id f10-20020aa782ca000000b0051bcf43d00amr13289562pfn.58.1655522359133; Fri, 17 Jun 2022 20:19:19 -0700 (PDT) Received: from localhost ([204.44.110.111]) by smtp.gmail.com with ESMTPSA id jf12-20020a170903268c00b0016a0ac06424sm1408870plb.51.2022.06.17.20.19.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Jun 2022 20:19:18 -0700 (PDT) From: Ihor Radchenko To: Felix Freeman Cc: emacs-orgmode@gnu.org, kaushal.modi@gmail.com, nick@nicksavage.ca Subject: [PATCH] babel output seems to drop anything before % (in session) In-Reply-To: References: Date: Sat, 18 Jun 2022 11:20:26 +0800 Message-ID: <87pmj6wtc5.fsf@localhost> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Received-SPF: pass client-ip=2607:f8b0:4864:20::529; envelope-from=yantar92@gmail.com; helo=mail-pg1-x529.google.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: "Emacs-orgmode" X-Migadu-Flow: FLOW_IN X-Migadu-To: larch@yhetil.org X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1655522397; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=cXuE+xDB9MxJ+tGG2V7NCgdOvqGvoyRz5gdbrxfbaxM=; b=u1XZUKRj2fBO+vmyYVFsdbsoT9ejXZEqhzpiCiFzQSZDDPA/RgVA4ihLtY6bQ3MKjFlI2j YODW1XbFYZaJgfNbb5mFjmFJZZbTeHxGFzNYCDHT4qFWNuHKVz97XkYehqBJ4YN5n6X2w0 lAxYqdpcO0HV3FnRjU2IsZeRcDTkns7vL2hH8rqyaGKn8pzxiuTRqEglUm7k/OfnXI3yVW Bb4ljkz5hhuPnM+mUD+GVHMaBZ8pp53aqLe8yMPDtOKtZ8BNUzUOG3o0OTLo+8GvKbPXCy e99ibjM/f6aXRCWn/IJkuo2FvvAWJi5t5ZSmygyM/GkcEozt9KdJNEFgVzwLoQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1655522397; a=rsa-sha256; cv=none; b=MbdxNzFLY36gU9Fu4+vuwCUuivjoNbUpiukE5ZfvaTN2z9vVR3fKRHaaQp5F8S6pC20i4a MXRV7mzlSq+9p9i5wZTWnfL4eVyDCuj/VuccRRxUmBzEkULOXui1BSV8j9tMfBJzwbnCi6 dGGtfa4Pe8DEgtfxLsOmxgE/nbAbDLtLuMuMZtdgq677ntPGc4vrdm85rLA9NsfynUznec bJqndIzZNV2NPeNFPYVUbEWMbw7bZ3TASfHjCskVw9oCPd7Mny92uS/HubMasyzKUohv9E cSXNcGmx6Au0NFGS/1MaG2sNJdbOzSQu6rtEaJHEa00YnzYBHNkOAxo8krsfbw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=SVqxlSfE; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -1.79 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=SVqxlSfE; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 5C9A99FC5 X-Spam-Score: -1.79 X-Migadu-Scanner: scn0.migadu.com X-TUID: 9RDLeFZRniL3 --=-=-= Content-Type: text/plain "Felix Freeman" writes: > Today I stumbed upon the same bug. > > Sadly I have no idea how to fix it, but thought it would be nice to post > about it here so the bug doesn't get lost in the sands of time. I finally managed to get some working fix. At least it kind of works on my system using bash. The idea is changing shell prompt to something unique when we initialize the session. See the attached patch. Please test this on "sh" "zsh" "fish" "csh" "ash" "dash" "ksh" "mksh" and "posh". Especially on fish, csh, and posh (which I believe refers to PowerShell). Also, doing PS1="blabla > " in bash may not always work. At least it does not work with https://github.com/nojhan/liquidprompt So, more reliable ways to change prompt are welcome. Best, Ihor --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename=0001-Add-disabled-test-for-in-babel-shell-output.patch >From 141f810a658d652e1cb3a147c5f71659f62df86f Mon Sep 17 00:00:00 2001 Message-Id: <141f810a658d652e1cb3a147c5f71659f62df86f.1655522152.git.yantar92@gmail.com> From: Daniele Pizzolli Date: Wed, 5 May 2021 17:00:28 +0200 Subject: [PATCH 1/2] Add disabled test for % in babel shell output --- testing/lisp/test-ob-shell.el | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/testing/lisp/test-ob-shell.el b/testing/lisp/test-ob-shell.el index 2f346f699..e81058ac7 100644 --- a/testing/lisp/test-ob-shell.el +++ b/testing/lisp/test-ob-shell.el @@ -106,6 +106,35 @@ (ert-deftest ob-shell/simple-list () "#+BEGIN_SRC sh :results output :var l='(1 2)\necho ${l}\n#+END_SRC" (org-trim (org-babel-execute-src-block)))))) +(ert-deftest ob-shell/percent-simple () + "Test percent in output." + (should + (equal "one 0% two\ntree 0% four" + (org-test-with-temp-text + "#+BEGIN_SRC bash :results output verbatim\necho one 0% two\necho tree 0% four\n#+END_SRC" + (org-trim (org-babel-execute-src-block))))) + (should + (equal "five 0% two\ntree 0% four" + (org-test-with-temp-text + "#+BEGIN_SRC bash :results output verbatim\necho five 0% two\necho tree 0% four\n#+END_SRC" + (org-trim (org-babel-execute-src-block)))))) + +(ert-deftest ob-shell/percent-session () + "Test percent in output for session. + +At the second iteration the string before % is dropped." + :expected-result :failed + (should + (equal "one 0% two\ntree 0% four" + (org-test-with-temp-text + "#+BEGIN_SRC bash :results output verbatim :session sess\necho one 0% two\necho tree 0% four\n#+END_SRC" + (org-trim (org-babel-execute-src-block))))) + (should + (equal "five 0% six\nseven 0% eight" + (org-test-with-temp-text + "#+BEGIN_SRC bash :results output verbatim :session sess\necho five 0% six\necho seven 0% eight\n#+END_SRC" + (org-trim (org-babel-execute-src-block)))))) + (provide 'test-ob-shell) ;;; test-ob-shell.el ends here -- 2.35.1 --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename=0002-ob-shell-Fix-output-containing-strings-matching-comi.patch >From 4f7a1369e72b22836cb74931c7c86e845155a729 Mon Sep 17 00:00:00 2001 Message-Id: <4f7a1369e72b22836cb74931c7c86e845155a729.1655522152.git.yantar92@gmail.com> In-Reply-To: <141f810a658d652e1cb3a147c5f71659f62df86f.1655522152.git.yantar92@gmail.com> References: <141f810a658d652e1cb3a147c5f71659f62df86f.1655522152.git.yantar92@gmail.com> From: Ihor Radchenko Date: Sat, 18 Jun 2022 11:11:12 +0800 Subject: [PATCH 2/2] ob-shell: Fix output containing strings matching `comint-prompt-regexp' * lisp/ob-comint.el (org-babel-comint-wait-for-output): Do not rely on `face-at-point' returning non-nil. * lisp/ob-shell.el (org-babel-shell-set-prompt-commands): New constant holding shell-specific commands to change prompt. (org-babel-prompt-command): New variable holding command to be user to set distinguishable prompt. (org-babel-shell-initialize): Set `org-babel-prompt-command' according to shell name. (org-babel-sh-prompt): New variable holding default shell prompt. (org-babel-sh-initiate-session): Change the default prompt to `org-babel-sh-prompt' and alter `comint-prompt-regexp' to match it tightly. * testing/lisp/test-ob-shell.el (ob-shell/percent-session): Do not expect failure in the fixed test. Fixes https://list.orgmode.org/CKK9TULBP2BG.2UITT31YJV03J@laptop/T/#mc8e3ca2f5f1b9a94040a68b4c6201234b209041c --- lisp/ob-comint.el | 4 +--- lisp/ob-shell.el | 42 ++++++++++++++++++++++++++++++----- testing/lisp/test-ob-shell.el | 4 ++-- 3 files changed, 39 insertions(+), 11 deletions(-) diff --git a/lisp/ob-comint.el b/lisp/ob-comint.el index 427aba341..1d3e24607 100644 --- a/lisp/ob-comint.el +++ b/lisp/ob-comint.el @@ -124,9 +124,7 @@ (defun org-babel-comint-wait-for-output (buffer) (while (progn (goto-char comint-last-input-end) (not (and (re-search-forward comint-prompt-regexp nil t) - (goto-char (match-beginning 0)) - (string= (face-name (face-at-point)) - "comint-highlight-prompt")))) + (goto-char (match-beginning 0))))) (accept-process-output (get-buffer-process buffer))))) (defun org-babel-comint-eval-invisibly-and-wait-for-file diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el index c25941a44..f80783653 100644 --- a/lisp/ob-shell.el +++ b/lisp/ob-shell.el @@ -42,6 +42,23 @@ (declare-function orgtbl-to-generic "org-table" (table params)) (defvar org-babel-default-header-args:shell '()) (defvar org-babel-shell-names) +(defconst org-babel-shell-set-prompt-commands + '(("fish" . "function fish_prompt\n\techo \"%s\"\nend") + ("csh" . "set prompt=\"%s\"") + ("posh" . "function prompt { \"%s\" }") + (t . "PS1=\"%s\"")) + "Alist assigning shells with their prompt setting command. + +Each element of the alist associates a shell type from +`org-babel-shell-names' with a template used to create a command to +change the default prompt. The template is an argument to `format' +that will be called with a single additional argument: prompt string. + +The fallback association template is defined in (t . \"template\") +alist element.") + +(defvar org-babel-prompt-command) + (defun org-babel-shell-initialize () "Define execution functions associated to shell names. This function has to be called whenever `org-babel-shell-names' @@ -51,7 +68,10 @@ (defun org-babel-shell-initialize () (eval `(defun ,(intern (concat "org-babel-execute:" name)) (body params) ,(format "Execute a block of %s commands with Babel." name) - (let ((shell-file-name ,name)) + (let ((shell-file-name ,name) + (org-babel-prompt-command + (or (alist-get ,name org-babel-shell-set-prompt-commands) + (alist-get t org-babel-shell-set-prompt-commands)))) (org-babel-execute:shell body params)))) (eval `(defalias ',(intern (concat "org-babel-variable-assignments:" name)) 'org-babel-variable-assignments:shell @@ -206,6 +226,13 @@ (defun org-babel-sh-var-to-string (var &optional sep hline) (mapconcat echo-var var "\n")) (t (funcall echo-var var))))) +(defvar org-babel-sh-eoe-indicator "echo 'org_babel_sh_eoe'" + "String to indicate that evaluation has completed.") +(defvar org-babel-sh-eoe-output "org_babel_sh_eoe" + "String to indicate that evaluation has completed.") +(defvar org-babel-sh-prompt "org_babel_sh_prompt> " + "String to set prompt in session shell.") + (defun org-babel-sh-initiate-session (&optional session _params) "Initiate a session named SESSION according to PARAMS." (when (and session (not (string= session "none"))) @@ -213,17 +240,20 @@ (defun org-babel-sh-initiate-session (&optional session _params) (or (org-babel-comint-buffer-livep session) (progn (shell session) + ;; Set unique prompt for easier analysis of the output. + (org-babel-comint-wait-for-output (current-buffer)) + (org-babel-comint-input-command + (current-buffer) + (format org-babel-prompt-command org-babel-sh-prompt)) + (setq-local comint-prompt-regexp + (concat "^" (regexp-quote org-babel-sh-prompt) + " *")) ;; Needed for Emacs 23 since the marker is initially ;; undefined and the filter functions try to use it without ;; checking. (set-marker comint-last-output-start (point)) (get-buffer (current-buffer))))))) -(defvar org-babel-sh-eoe-indicator "echo 'org_babel_sh_eoe'" - "String to indicate that evaluation has completed.") -(defvar org-babel-sh-eoe-output "org_babel_sh_eoe" - "String to indicate that evaluation has completed.") - (defun org-babel-sh-evaluate (session body &optional params stdin cmdline) "Pass BODY to the Shell process in BUFFER. If RESULT-TYPE equals `output' then return a list of the outputs diff --git a/testing/lisp/test-ob-shell.el b/testing/lisp/test-ob-shell.el index e81058ac7..2bfc29228 100644 --- a/testing/lisp/test-ob-shell.el +++ b/testing/lisp/test-ob-shell.el @@ -122,8 +122,8 @@ (ert-deftest ob-shell/percent-simple () (ert-deftest ob-shell/percent-session () "Test percent in output for session. -At the second iteration the string before % is dropped." - :expected-result :failed +Percent is matching the default value of `comint-prompt-regexp', which +can cause issues with analysis." (should (equal "one 0% two\ntree 0% four" (org-test-with-temp-text -- 2.35.1 --=-=-=--