From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id GFkQGi0OvF8ZUgAA0tVLHw (envelope-from ) for ; Mon, 23 Nov 2020 19:31:57 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id 8FX5FS0OvF/7GwAAB5/wlQ (envelope-from ) for ; Mon, 23 Nov 2020 19:31:57 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 548199404FB for ; Mon, 23 Nov 2020 19:31:56 +0000 (UTC) Received: from localhost ([::1]:54350 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1khHZ5-0002w6-3K for larch@yhetil.org; Mon, 23 Nov 2020 14:31:55 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:48430) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1khHWZ-0002ur-00 for emacs-orgmode@gnu.org; Mon, 23 Nov 2020 14:29:19 -0500 Received: from mail-pg1-x536.google.com ([2607:f8b0:4864:20::536]:41939) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1khHWW-0000aP-Nd for emacs-orgmode@gnu.org; Mon, 23 Nov 2020 14:29:18 -0500 Received: by mail-pg1-x536.google.com with SMTP id s63so5309320pgc.8 for ; Mon, 23 Nov 2020 11:29:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:mime-version; bh=bPq/VyDCouhFyCCoO0byBMOgHh8WLS63ou3qtrsqBTc=; b=WILUBXEw3hqsWk/J9TXaT2Rway0K7bdw578/tsRiYLNxgJNgQWqIw01CKUYqt4xkaf 3Dk4LldY9eQ6i2nTozkIe1tnGb9hfEtKdmE1hZDlMkqkPBQNXh9n6HAc5AIsjVqJ36Dz D68cgReSjwPw5Fk46bzLdIxVaepPgnrafqk+KQ2XIoUL0YDJNRLevxkwbhGsPRV4/UtD 9MbUA4A/11fl3UgSX8x9D+kElTm1KSHBTVEJelhaYK+jcbQ+Nr+B96uUo4V0a7tcvF03 TbBx4q67yuWcQi83LwHRI95/1WBCnkFpEcpyAiLCAVK1IxQudnjVmiyrhGys7mpbdAmW 6kZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:mime-version; bh=bPq/VyDCouhFyCCoO0byBMOgHh8WLS63ou3qtrsqBTc=; b=XWKRFFaRHKUN5q28v8g4pTxcVZlh1blwDziW6JyegtMEdgmr2brTkQ3zpXTGaDOIaM 6Am23djvY3ZLHaT/aOAeBUi2ymmye+e+WYQTQTaDNiYRSa3Kq9rF5rzM4aIq+Jar7m6J aWp/qpEFI4D26pfW/ObUgm8wAcbodkghUw+lRPJSlu1sEPT0vLKp298X1ayz2gmjJLxC jRFv+jn5bjRyTC+MdX20xiLj9TpWQPuztuuQ+Kavh7edo8QW9Knzs9JiLZLF3HWF307a vFMlNI/1VKmmBWgLAKS3yRfRTLqbl3rIEkSLwXsFAR8chPxBcWu1PO2dD8R69zdr8q3I ObeQ== X-Gm-Message-State: AOAM5317nv8Er5kXuFE4aahH2faxrcI+P7lE0tvymRgEAUdmbHDR2c7c DQssrGk+z6mRFbmECNSx35ZsVutv24o= X-Google-Smtp-Source: ABdhPJyNtVRQJ+vmvxoElxj/xOawCLZ0lbCe0uMccmiyothWgRaVQ9ZnuH1O/oTZaNu23OcmNVegNQ== X-Received: by 2002:a17:90b:4d0e:: with SMTP id mw14mr511870pjb.92.1606159754579; Mon, 23 Nov 2020 11:29:14 -0800 (PST) Received: from localhost (ip72-194-51-80.sb.sd.cox.net. [72.194.51.80]) by smtp.gmail.com with ESMTPSA id y3sm187093pjb.18.2020.11.23.11.29.13 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Nov 2020 11:29:13 -0800 (PST) From: Karthik Chikmagalur To: emacs-orgmode@gnu.org Subject: Bug: Org-babel output malformed for MATLAB source blocks running in a session [9.3 (release_9.3 @ /usr/share/emacs/27.1/lisp/org/)] Date: Mon, 23 Nov 2020 11:29:07 -0800 Message-ID: <86zh37vpmk.fsf@gmail.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Received-SPF: pass client-ip=2607:f8b0:4864:20::536; envelope-from=karthikchikmagalur@gmail.com; helo=mail-pg1-x536.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: "Emacs-orgmode" X-Scanner: ns3122888.ip-94-23-21.eu Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=WILUBXEw; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of emacs-orgmode-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=emacs-orgmode-bounces@gnu.org X-Spam-Score: -1.71 X-TUID: 41y+gEwX9HRl --=-=-= Content-Type: text/plain Remember to cover the basics, that is, what you expected to happen and what in fact did happen. You don't know how to make a good report? See https://orgmode.org/manual/Feedback.html#Feedback Your bug report will be posted to the Org mailing list. ------------------------------------------------------------------------ Org-babel's handling of MATLAB block output with the =:results session= argument is broken. Here is some sample code along with the expected result: #+begin_src matlab :results output :session *MATLAB* a = 3; b = 4; c = a + b #+end_src #+RESULTS: : c = : : 7 However here is the actual result: #+begin_src matlab :results output :session *MATLAB* a = 3; b = 4; c = a + b #+end_src #+RESULTS: #+begin_example a = 3; b = 4; c = a + b c = 7 'org_babel_eoe' ans = 'org_babel_eoe' #+end_example There are two separate problems: 1. The =org-babel-octave-eoe-indicator= is not being stripped. 2. The comint input is being echoed in the output. #1 is easy to fix. The problem is in the function =org-babel-octave-evaluate-session=, in the line: #+begin_src emacs-lisp (cdr (reverse (delq "" (mapcar #'org-strip-quotes (mapcar #'org-trim raw))))) #+end_src Where empty lines in the output are being removed with =delq=. =delq= compares with =eq= instead of =equal=, which fails on blank lines. Replacing this with =delete= works fine. #2 is a much trickier problem to solve. Here is the body of the input as it appears in the MATLAB session: #+begin_example >> a = 3; b = 4; c = a + b 'org_babel_eoe' a = 3; >> b = 4; >> c = a + b c = 7 >> 'org_babel_eoe' ans = 'org_babel_eoe' >> #+end_example Here `>>' is the shell prompt, which is absent from the raw comint output as given to org by comint-mode. =matlab-shell= does not work like other comint shells in that it doesn't echo bulk (/i.e./ multi-line) input all at once. Instead, it echoes each line of input and follows it with its output. This interspersal of input and output is causing =org-babel-comint-with-output=, the function (actually macro) responsible for removing the comint input echo text from the raw comint output, to fail. This macro assumes that the raw comint output looks like #+begin_example <> <> #+end_example as you can see from this code from =org-babel-comint-with-output=: #+begin_src emacs-lisp (when (and ,remove-echo ,full-body (string-match (replace-regexp-in-string "\n" "[\r\n]+" (regexp-quote (or ,full-body ""))) string-buffer)) (setq string-buffer (substring string-buffer (match-end 0)))) #+end_src To fix this, either =org-babel-octave-evaluate-session= or =org-babel-comint-with-output= needs to be modified. This problem is local to MATLAB, it does not happen with GNU Octave, which shares most of its org-babel code with MATLAB's. So I wrote a patch to the former that does additional line-by-line processing on the raw comint output to detect and remove the echoed input. (Patch is attached.) However while this fixes the problem it's not a robust solution. -Karthik --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename=ob-octave.el.patch Content-Description: patch to ob-octave.el --- ob-octave.el 2020-11-23 11:22:01.473682045 -0800 +++ ob-octave-new.el 2020-11-23 11:10:07.961900383 -0800 @@ -187,6 +187,7 @@ (org-babel-process-file-name tmp-file 'noquote))) (org-babel-octave-import-elisp-from-file tmp-file)))))) + (defun org-babel-octave-evaluate-session (session body result-type &optional matlabp) "Evaluate BODY in SESSION." @@ -237,12 +238,31 @@ (`output (setq results (if matlabp - (cdr (reverse (delq "" (mapcar #'org-strip-quotes + (cdr (reverse (delete "" (mapcar #'org-strip-quotes (mapcar #'org-trim raw))))) (cdr (member org-babel-octave-eoe-output (reverse (mapcar #'org-strip-quotes (mapcar #'org-trim raw))))))) - (mapconcat #'identity (reverse results) "\n"))))) + ;; This kludge is to remove the input lines from the output. Because of + ;; the special way that MATLAB processes bulk comint output (the output + ;; of each line follows that line) the macro + ;; `org-babel-comint-with-output' cannot remove the echoed commands. The + ;; following handles this manually, by splitting both the original input + ;; (`BODY') and full output (`RESULTS') on newlines, comparing them line + ;; by line and removing all lines in BODY from RESULTS. Note that RESULTS + ;; is already a list of strings so additional care is needed. + (let* ((body-lines (split-string body "\n+")) + (result-lines (flatten-list + (mapcar + (lambda (entry) (reverse (split-string entry "\n"))) + results)))) + (mapconcat + #'identity + (reverse (cl-remove-if + (lambda (line) (member line body-lines)) + result-lines)) "\n") + ))))) + (defun org-babel-octave-import-elisp-from-file (file-name) "Import data from FILE-NAME. --=-=-= Content-Type: text/plain Emacs : GNU Emacs 27.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.22, cairo version 1.17.3) of 2020-08-28 Package: Org mode version 9.3 (release_9.3 @ /usr/share/emacs/27.1/lisp/org/) current state: ============== (setq org-src-mode-hook '(org-src-babel-configure-edit-buffer org-src-mode-configure-edit-buffer) org-link-shell-confirm-function 'yes-or-no-p org-metadown-hook '(org-babel-pop-to-session-maybe) org-clock-out-hook '(org-clock-remove-empty-clock-drawer) org-mode-hook '(#[0 "\300\301\302\303\304$\207" [add-hook change-major-mode-hook org-show-all append local] 5] #[0 "\300\301\302\303\304$\207" [add-hook change-major-mode-hook org-babel-show-result-all append local] 5] org-babel-result-hide-spec org-babel-hide-all-hashes) org-archive-hook '(org-attach-archive-delete-maybe) org-confirm-elisp-link-function 'yes-or-no-p org-agenda-before-write-hook '(org-agenda-add-entry-text) org-metaup-hook '(org-babel-load-in-session-maybe) org-bibtex-headline-format-function #[257 "\300\236A\207" [:title] 3 "\n\n(fn ENTRY)"] org-babel-pre-tangle-hook '(save-buffer) org-tab-first-hook '(org-babel-hide-result-toggle-maybe org-babel-header-arg-expand) org-occur-hook '(org-first-headline-recenter) org-cycle-hook '(org-cycle-hide-archived-subtrees org-cycle-show-empty-lines org-optimize-window-after-visibility-change) org-speed-command-hook '(org-speed-command-activate org-babel-speed-command-activate) org-confirm-shell-link-function 'yes-or-no-p org-link-parameters '(("attachment" :follow org-attach-open-link :export org-attach-export-link :complete org-attach-complete-link) ("id" :follow org-id-open) ("eww" :follow eww :store org-eww-store-link) ("rmail" :follow org-rmail-open :store org-rmail-store-link) ("mhe" :follow org-mhe-open :store org-mhe-store-link) ("irc" :follow org-irc-visit :store org-irc-store-link :export org-irc-export) ("info" :follow org-info-open :export org-info-export :store org-info-store-link) ("gnus" :follow org-gnus-open :store org-gnus-store-link) ("docview" :follow org-docview-open :export org-docview-export :store org-docview-store-link) ("bibtex" :follow org-bibtex-open :store org-bibtex-store-link) ("bbdb" :follow org-bbdb-open :export org-bbdb-export :complete org-bbdb-complete-link :store org-bbdb-store-link) ("w3m" :store org-w3m-store-link) ("file+sys") ("file+emacs") ("shell" :follow org-link--open-shell) ("news" :follow #[257 "\301\300\302Q!\207" ["news" browse-url ":"] 5 "\n\n(fn URL)"]) ("mailto" :follow #[257 "\301\300\302Q!\207" ["mailto" browse-url ":"] 5 "\n\n(fn URL)"]) ("https" :follow #[257 "\301\300\302Q!\207" ["https" browse-url ":"] 5 "\n\n(fn URL)"]) ("http" :follow #[257 "\301\300\302Q!\207" ["http" browse-url ":"] 5 "\n\n(fn URL)"]) ("ftp" :follow #[257 "\301\300\302Q!\207" ["ftp" browse-url ":"] 5 "\n\n(fn URL)"]) ("help" :follow org-link--open-help) ("file" :complete org-link-complete-file) ("elisp" :follow org-link--open-elisp) ("doi" :follow org-link--open-doi)) org-link-elisp-confirm-function 'yes-or-no-p ) --=-=-=--