From mboxrd@z Thu Jan 1 00:00:00 1970 From: Felipe Lema Subject: Re: correct remote path handling Date: Wed, 26 Feb 2020 22:58:25 -0300 Message-ID: <1993557.tO6IXrBGSL@lenovo-x200> References: <1873694.uLqJpLN6Zz@linux-q1p8> <8736ayy85n.fsf@gmail.com> <87zhd6fpj6.fsf@gnu.org> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="nextPart4377444.z2nPX1k6b1" Content-Transfer-Encoding: 7Bit Return-path: Received: from eggs.gnu.org ([2001:470:142:3::10]:38701) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1j78Rr-0001o9-P7 for emacs-orgmode@gnu.org; Wed, 26 Feb 2020 20:58:49 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1j78Rp-0002wk-Fy for emacs-orgmode@gnu.org; Wed, 26 Feb 2020 20:58:47 -0500 In-Reply-To: <87zhd6fpj6.fsf@gnu.org> List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane-mx.org@gnu.org Sender: "Emacs-orgmode" To: Bastien Cc: Jack Kamm , emacs-orgmode@gnu.org This is a multi-part message in MIME format. --nextPart4377444.z2nPX1k6b1 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" Hey, y'all I found out that ob-shell was more broken than I thought so I had to upgrade my sword and get a new shield to fix it. Tramp support for ob-python and ob-shell look OK now, although still no tests. I'll wait until this review is done until I start working on the tests. Will probably take a while because I'm thinking of some cross-section code that will do the a set of tests (all current? some marked? some new instead?) using a remote directory. Using `ssh localhost` should suffice, but should these remote tests be done by default? It's most likely that everyone running tests will require a passwordless setup. While I've automatized this for a separate package, it may be bothersome to someone else (see [1]). Anyway, I hope the patch file is correct now. I messed up my mirror repo and I /think/ I got it right. BTW I just got a mail that FSF papers are approved now. Gards, Felipe [1]: https://github.com/FelipeLema/emacs-counsel-gtags/blame/ 5d2a8c2c2d358e374a576cf8a3a67f7997a8839b/.travis.yml#L6 On Tuesday, 25 February 2020 16:14:05 -03 Bastien wrote: > Hi Felipe and Jack, > > > Felipe Lema writes: > >> I've signed the necessary papers from (to?) the FSF involving org > >> mode, so I'm ready on my side to add tests and maybe add support > >> for other tramp-related stuff. > > I've checked and the papers are not yet processed by the FSF. > > > Great, we should add you to the list of copyrighted contributors: > > https://orgmode.org/worg/org-contribute.html > > Done - https://orgmode.org/worg/org-contribute.html#org7c578f2 > > Let's iterate on this patch and get it ready for 9.5. > > Thanks, --nextPart4377444.z2nPX1k6b1 Content-Disposition: attachment; filename="tramp-support.patch" Content-Transfer-Encoding: quoted-printable Content-Type: text/x-patch; charset="utf-8"; name="tramp-support.patch" fix tramp support for ob-shell & ob-python * ob-python.el (org-babel-python-evaluate-session): use file path local to = (maybe) remote process. * ob-shell.el (org-babel-execute:shell, org-babel-sh-evaluate): call-proces= s-=E2=80=A6=E2=86=92 process-file-=E2=80=A6 (tramp aware), wrap :session bl= ock in a function, separate "ob overhead" from "eval block", add documentat= ion. diff --git a/lisp/ob-python.el b/lisp/ob-python.el index dbcfac08d..1afea9cb3 100644 =2D-- a/lisp/ob-python.el +++ b/lisp/ob-python.el @@ -327,7 +327,8 @@ last statement in BODY, as elisp." "python-"))) (with-temp-file tmp-src-file (insert body)) (format org-babel-python--exec-tmpfile =2D tmp-src-file)) + (org-babel-local-file-name + tmp-src-file))) body))) (mapconcat #'org-trim @@ -345,9 +346,10 @@ last statement in BODY, as elisp." "python-"))) (with-temp-file tmp-src-file (insert body)) (format org-babel-python--eval-ast =2D tmp-src-file)))) =2D (org-babel-comint-with-output =2D (session org-babel-python-eoe-indicator nil body) + (org-babel-local-file-name + tmp-src-file))))) + (org-babel-comint-with-output + (session org-babel-python-eoe-indicator nil body) (let ((comint-process-echoes nil)) (funcall input-body body) (dolist diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el index 347ffedd1..66cdfb94c 100644 =2D-- a/lisp/ob-shell.el +++ b/lisp/ob-shell.el @@ -82,12 +82,17 @@ This function is called by `org-babel-execute-src-block= '." (value-is-exit-status (member "value" (cdr (assq :result-params params)))) (cmdline (cdr (assq :cmdline params))) + (shebang (cdr (assq :shebang params))) + (padline (not (equal "no" (cdr (assq :padline params))))) + (result-params (cdr (assq :result-params params))) (full-body (concat (org-babel-expand-body:generic body params (org-babel-variable-assignments:shell params)) (when value-is-exit-status "\necho $?")))) (org-babel-reassemble-table =2D (org-babel-sh-evaluate session full-body params stdin cmdline) + (org-babel-sh-evaluate session full-body + stdin cmdline shebang value-is-exit-status padline + result-params) (org-babel-pick-name (cdr (assq :colname-names params)) (cdr (assq :colnames params))) (org-babel-pick-name @@ -206,76 +211,114 @@ var of the same value." "String to indicate that evaluation has completed.") (defvar org-babel-sh-eoe-output "org_babel_sh_eoe" "String to indicate that evaluation has completed.") +(defvar org-babel-sh-block-function-name "org_babel_block" + "Name of the shell function that will hold the code to be executed.") =20 =2D(defun org-babel-sh-evaluate (session body &optional params stdin cmdlin= e) =2D "Pass BODY to the Shell process in BUFFER. =2DIf RESULT-TYPE equals `output' then return a list of the outputs =2Dof the statements in BODY, if RESULT-TYPE equals `value' then =2Dreturn the value of the last statement in BODY." =2D (let* ((shebang (cdr (assq :shebang params))) =2D (value-is-exit-status =2D (member "value" (cdr (assq :result-params params)))) =2D (results =2D (cond =2D ((or stdin cmdline) ; external shell script w/STDIN =2D (let ((script-file (org-babel-temp-file "sh-script-")) =2D (stdin-file (org-babel-temp-file "sh-stdin-")) =2D (padline (not (string=3D "no" (cdr (assq :padline params)))))) =2D (with-temp-file script-file =2D (when shebang (insert shebang "\n")) =2D (when padline (insert "\n")) =2D (insert body)) =2D (set-file-modes script-file #o755) =2D (with-temp-file stdin-file (insert (or stdin ""))) =2D (with-temp-buffer =2D (call-process-shell-command =2D (concat (if shebang script-file =2D (format "%s %s" shell-file-name script-file)) =2D (and cmdline (concat " " cmdline))) =2D stdin-file =2D (current-buffer)) =2D (buffer-string)))) =2D (session ; session evaluation =2D (mapconcat =2D #'org-babel-sh-strip-weird-long-prompt =2D (mapcar =2D #'org-trim =2D (butlast =2D (org-babel-comint-with-output =2D (session org-babel-sh-eoe-output t body) =2D (dolist (line (append (split-string (org-trim body) "\n") =2D (list org-babel-sh-eoe-indicator))) =2D (insert line) =2D (comint-send-input nil t) =2D (while (save-excursion =2D (goto-char comint-last-input-end) =2D (not (re-search-forward =2D comint-prompt-regexp nil t))) =2D (accept-process-output =2D (get-buffer-process (current-buffer)))))) =2D 2)) =2D "\n")) +(defun org-babel-sh-evaluate (session body + &optional stdin cmdline + shebang value-is-exit-status padline + result-params) + "Pass BODY to the Shell process in SESSION. + +Optional arguments: + +- Send STDIN as stdin in. +- Extra commandline arguments (such as -f -x -v=E2=80=A6) in CMDLINE. +- Use SHEBANG as interpreter for block (such as \"#!/usr/bin/bash\") +- When VALUE-IS-EXIT-STATUS, returns the exit status value as a string. +- Add an extra end-of-line when PADLINE +- forward RESULT-PARAMS to `org-babel-result-cond'." + (let* ((results + (cond + ((or stdin cmdline) ; external shell script w/STDIN + (let ((script-file (org-babel-temp-file "sh-script-")) + (stdin-file (org-babel-temp-file "sh-stdin-"))) + (with-temp-file script-file + (when shebang (insert shebang "\n")) + (when padline (insert "\n")) + (insert body)) + (set-file-modes script-file #o755) + (with-temp-file stdin-file (insert (or stdin ""))) + (with-temp-buffer + (process-file-shell-command + (concat + (let ((local-script-file (org-babel-local-file-name + script-file))) + (if shebang local-script-file + (format "%s %s" shell-file-name + local-script-file))) + (and cmdline (concat " " cmdline))) + stdin-file + (current-buffer)) + (buffer-string)))) + (session ; session evaluation + (let* ((block-output-lines + (let ((fun-body + (format "%s(){\n%s\n}\n%s" + org-babel-sh-block-function-name + ;; function block + (concat + body + "\n" + org-babel-sh-eoe-indicator) ;; mark eoe + ;; mark "function has been input" + org-babel-sh-eoe-indicator))) + (cl-flet ((send-and-wait () + (comint-send-input nil t) + (while (save-excursion + (goto-char comint-last-input-end) + (not (re-search-forward + comint-prompt-regexp nil t))) + (accept-process-output + (get-buffer-process (current-buffer)))))) + ;; define a function with the code we want to eval + (org-babel-comint-with-output + (session org-babel-sh-eoe-output t fun-body) + (insert fun-body) + (send-and-wait)) ;; wait until function code is accepted + ;; now, actually eval the code by calling it as a function + (org-babel-comint-with-output + (session org-babel-sh-eoe-output t + org-babel-sh-block-function-name) + (insert org-babel-sh-block-function-name) + (send-and-wait))))) + (block-output-lines-sans-marker-sans-prompt + (thread-first + block-output-lines + (car) ;; (list ouput-str prompt-str) + (split-string "\n" t) ;; string lines + (butlast 1))) ;; remove last line with prompt + (block-output-clean-lines + (mapcar #'org-trim + block-output-lines-sans-marker-sans-prompt)) + (block-output + (mapconcat + #'org-babel-sh-strip-weird-long-prompt + block-output-clean-lines "\n"))) + block-output)) ;; External shell script, with or without a predefined =2D ;; shebang. =2D ((org-string-nw-p shebang) =2D (let ((script-file (org-babel-temp-file "sh-script-")) =2D (padline (not (equal "no" (cdr (assq :padline params)))))) =2D (with-temp-file script-file =2D (insert shebang "\n") =2D (when padline (insert "\n")) =2D (insert body)) =2D (set-file-modes script-file #o755) =2D (org-babel-eval script-file ""))) =2D (t (org-babel-eval shell-file-name (org-trim body)))))) + ;; shebang. + ((org-string-nw-p shebang) + (let ((script-file (org-babel-temp-file "sh-script-"))) + (with-temp-file script-file + (insert shebang "\n") + (when padline (insert "\n")) + (insert body)) + (set-file-modes script-file #o755) + ;; (maybe remotely) run this script as a command + (org-babel-eval (org-babel-local-file-name script-file) ""))) + (t (org-babel-eval + shell-file-name + (org-trim body)))))) (when value-is-exit-status + ;; last line is the output of "echo $?" (setq results (car (reverse (split-string results "\n" t))))) (when results =2D (let ((result-params (cdr (assq :result-params params)))) =2D (org-babel-result-cond result-params + (org-babel-result-cond result-params results (let ((tmp-file (org-babel-temp-file "sh-"))) (with-temp-file tmp-file (insert results)) =2D (org-babel-import-elisp-from-file tmp-file))))))) + (org-babel-import-elisp-from-file tmp-file)))))) =20 (defun org-babel-sh-strip-weird-long-prompt (string) "Remove prompt cruft from a string of shell output." --nextPart4377444.z2nPX1k6b1--