Re: correct remote path handling

[Felipe Lema <felipelema@mortemale.org>]

[-- Attachment #1: Type: text/plain, Size: 1676 bytes --]

Hey, y'all

I found out that ob-shell was more broken than I thought so I had to upgrade 
my sword and get a new shield to fix it.

Tramp support for ob-python and ob-shell look OK now, although still no tests. 
I'll wait until this review is done until I start working on the tests. Will 
probably take a while because I'm thinking of some cross-section code that 
will do the a set of tests (all current? some marked? some new instead?) using 
a remote directory.

Using `ssh localhost` should suffice, but should these remote tests be done by 
default? It's most likely that everyone running tests will require a 
passwordless setup. While I've automatized this for a separate package, it may 
be bothersome to someone else (see [1]).

Anyway, I hope the patch file is correct now. I messed up my mirror repo and I 
/think/ I got it right.

BTW I just got a mail that FSF papers are approved now.

Gards, Felipe

[1]: https://github.com/FelipeLema/emacs-counsel-gtags/blame/
5d2a8c2c2d358e374a576cf8a3a67f7997a8839b/.travis.yml#L6

On Tuesday, 25 February 2020 16:14:05 -03 Bastien wrote:
> Hi Felipe and Jack,
> 
> > Felipe Lema <felipelema@mortemale.org> writes:
> >> I've signed the necessary papers from (to?) the FSF involving org
> >> mode, so I'm ready on my side to add tests and maybe add support
> >> for other tramp-related stuff.
> 
> I've checked and the papers are not yet processed by the FSF.
> 
> > Great, we should add you to the list of copyrighted contributors:
> > https://orgmode.org/worg/org-contribute.html
> 
> Done -  https://orgmode.org/worg/org-contribute.html#org7c578f2
> 
> Let's iterate on this patch and get it ready for 9.5.
> 
> Thanks,


[-- Attachment #2: tramp-support.patch --]
[-- Type: text/x-patch, Size: 9929 bytes --]

fix tramp support for ob-shell & ob-python

* ob-python.el (org-babel-python-evaluate-session): use file path local to (maybe) remote process.
* ob-shell.el (org-babel-execute:shell, org-babel-sh-evaluate): call-process-…→ process-file-… (tramp aware), wrap :session block in a function, separate "ob overhead" from "eval block", add documentation.

diff --git a/lisp/ob-python.el b/lisp/ob-python.el
index dbcfac08d..1afea9cb3 100644
--- a/lisp/ob-python.el
+++ b/lisp/ob-python.el
@@ -327,7 +327,8 @@ last statement in BODY, as elisp."
 						  "python-")))
 			       (with-temp-file tmp-src-file (insert body))
 			       (format org-babel-python--exec-tmpfile
-				       tmp-src-file))
+				       (org-babel-local-file-name
+					tmp-src-file)))
 			   body)))
 	       (mapconcat
 		#'org-trim
@@ -345,9 +346,10 @@ last statement in BODY, as elisp."
 					      "python-")))
 			   (with-temp-file tmp-src-file (insert body))
 			   (format org-babel-python--eval-ast
-				   tmp-src-file))))
-               (org-babel-comint-with-output
-                   (session org-babel-python-eoe-indicator nil body)
+				   (org-babel-local-file-name
+				    tmp-src-file)))))
+	       (org-babel-comint-with-output
+		   (session org-babel-python-eoe-indicator nil body)
                  (let ((comint-process-echoes nil))
                    (funcall input-body body)
 		   (dolist
diff --git a/lisp/ob-shell.el b/lisp/ob-shell.el
index 347ffedd1..66cdfb94c 100644
--- a/lisp/ob-shell.el
+++ b/lisp/ob-shell.el
@@ -82,12 +82,17 @@ This function is called by `org-babel-execute-src-block'."
 	 (value-is-exit-status
 	  (member "value" (cdr (assq :result-params params))))
 	 (cmdline (cdr (assq :cmdline params)))
+	 (shebang (cdr (assq :shebang params)))
+	 (padline (not (equal "no" (cdr (assq :padline params)))))
+	 (result-params (cdr (assq :result-params params)))
          (full-body (concat
 		     (org-babel-expand-body:generic
 		      body params (org-babel-variable-assignments:shell params))
 		     (when value-is-exit-status "\necho $?"))))
     (org-babel-reassemble-table
-     (org-babel-sh-evaluate session full-body params stdin cmdline)
+     (org-babel-sh-evaluate session full-body
+			    stdin cmdline shebang value-is-exit-status padline
+			    result-params)
      (org-babel-pick-name
       (cdr (assq :colname-names params)) (cdr (assq :colnames params)))
      (org-babel-pick-name
@@ -206,76 +211,114 @@ var of the same value."
   "String to indicate that evaluation has completed.")
 (defvar org-babel-sh-eoe-output "org_babel_sh_eoe"
   "String to indicate that evaluation has completed.")
+(defvar org-babel-sh-block-function-name "org_babel_block"
+  "Name of the shell function that will hold the code to be executed.")
 
-(defun org-babel-sh-evaluate (session body &optional params stdin cmdline)
-  "Pass BODY to the Shell process in BUFFER.
-If RESULT-TYPE equals `output' then return a list of the outputs
-of the statements in BODY, if RESULT-TYPE equals `value' then
-return the value of the last statement in BODY."
-  (let* ((shebang (cdr (assq :shebang params)))
-	 (value-is-exit-status
-	  (member "value" (cdr (assq :result-params params))))
-	 (results
-	  (cond
-	   ((or stdin cmdline)	       ; external shell script w/STDIN
-	    (let ((script-file (org-babel-temp-file "sh-script-"))
-		  (stdin-file (org-babel-temp-file "sh-stdin-"))
-		  (padline (not (string= "no" (cdr (assq :padline params))))))
-	      (with-temp-file script-file
-		(when shebang (insert shebang "\n"))
-		(when padline (insert "\n"))
-		(insert body))
-	      (set-file-modes script-file #o755)
-	      (with-temp-file stdin-file (insert (or stdin "")))
-	      (with-temp-buffer
-		(call-process-shell-command
-		 (concat (if shebang script-file
-			   (format "%s %s" shell-file-name script-file))
-			 (and cmdline (concat " " cmdline)))
-		 stdin-file
-		 (current-buffer))
-		(buffer-string))))
-	   (session			; session evaluation
-	    (mapconcat
-	     #'org-babel-sh-strip-weird-long-prompt
-	     (mapcar
-	      #'org-trim
-	      (butlast
-	       (org-babel-comint-with-output
-		   (session org-babel-sh-eoe-output t body)
-		 (dolist (line (append (split-string (org-trim body) "\n")
-				       (list org-babel-sh-eoe-indicator)))
-		   (insert line)
-		   (comint-send-input nil t)
-		   (while (save-excursion
-			    (goto-char comint-last-input-end)
-			    (not (re-search-forward
-				  comint-prompt-regexp nil t)))
-		     (accept-process-output
-		      (get-buffer-process (current-buffer))))))
-	       2))
-	     "\n"))
+(defun org-babel-sh-evaluate (session body
+				      &optional stdin cmdline
+				      shebang value-is-exit-status padline
+				      result-params)
+  "Pass BODY to the Shell process in SESSION.
+
+Optional arguments:
+
+- Send STDIN as stdin in.
+- Extra commandline arguments (such as -f -x -v…) in CMDLINE.
+- Use SHEBANG as interpreter for block (such as \"#!/usr/bin/bash\")
+- When VALUE-IS-EXIT-STATUS, returns the exit status value as a string.
+- Add an extra end-of-line when PADLINE
+- forward RESULT-PARAMS to `org-babel-result-cond'."
+  (let* ((results
+          (cond
+           ((or stdin cmdline)         ; external shell script w/STDIN
+            (let ((script-file (org-babel-temp-file "sh-script-"))
+                  (stdin-file (org-babel-temp-file "sh-stdin-")))
+              (with-temp-file script-file
+                (when shebang (insert shebang "\n"))
+                (when padline (insert "\n"))
+                (insert body))
+              (set-file-modes script-file #o755)
+              (with-temp-file stdin-file (insert (or stdin "")))
+              (with-temp-buffer
+                (process-file-shell-command
+                 (concat
+                  (let ((local-script-file (org-babel-local-file-name
+                                            script-file)))
+                    (if shebang local-script-file
+                      (format "%s %s" shell-file-name
+                              local-script-file)))
+                  (and cmdline (concat " " cmdline)))
+                 stdin-file
+                 (current-buffer))
+                (buffer-string))))
+           (session                     ; session evaluation
+	    (let* ((block-output-lines
+		    (let ((fun-body
+			   (format "%s(){\n%s\n}\n%s"
+				    org-babel-sh-block-function-name
+				    ;; function block
+				    (concat
+				     body
+				     "\n"
+				     org-babel-sh-eoe-indicator) ;; mark eoe
+				    ;; mark "function has been input"
+				    org-babel-sh-eoe-indicator)))
+		      (cl-flet ((send-and-wait ()
+					       (comint-send-input nil t)
+					       (while (save-excursion
+							(goto-char comint-last-input-end)
+							(not (re-search-forward
+							      comint-prompt-regexp nil t)))
+						 (accept-process-output
+						  (get-buffer-process (current-buffer))))))
+			;; define a function with the code we want to eval
+			(org-babel-comint-with-output
+			    (session org-babel-sh-eoe-output t fun-body)
+			  (insert fun-body)
+			  (send-and-wait)) ;; wait until function code is accepted
+			;; now, actually eval the code by calling it as a function
+			(org-babel-comint-with-output
+			    (session org-babel-sh-eoe-output t
+				     org-babel-sh-block-function-name)
+			  (insert org-babel-sh-block-function-name)
+			  (send-and-wait)))))
+		   (block-output-lines-sans-marker-sans-prompt
+		    (thread-first
+			block-output-lines
+		      (car) ;; (list ouput-str prompt-str)
+		      (split-string "\n" t) ;; string lines
+		      (butlast 1))) ;; remove last line with prompt
+		   (block-output-clean-lines
+		    (mapcar #'org-trim
+			    block-output-lines-sans-marker-sans-prompt))
+		   (block-output
+		    (mapconcat
+		     #'org-babel-sh-strip-weird-long-prompt
+		     block-output-clean-lines "\n")))
+	      block-output))
 	   ;; External shell script, with or without a predefined
-	   ;; shebang.
-	   ((org-string-nw-p shebang)
-	    (let ((script-file (org-babel-temp-file "sh-script-"))
-		  (padline (not (equal "no" (cdr (assq :padline params))))))
-	      (with-temp-file script-file
-		(insert shebang "\n")
-		(when padline (insert "\n"))
-		(insert body))
-	      (set-file-modes script-file #o755)
-	      (org-babel-eval script-file "")))
-	   (t (org-babel-eval shell-file-name (org-trim body))))))
+           ;; shebang.
+           ((org-string-nw-p shebang)
+            (let ((script-file (org-babel-temp-file "sh-script-")))
+              (with-temp-file script-file
+                (insert shebang "\n")
+                (when padline (insert "\n"))
+                (insert body))
+              (set-file-modes script-file #o755)
+              ;; (maybe remotely) run this script as a command
+              (org-babel-eval (org-babel-local-file-name script-file) "")))
+           (t (org-babel-eval
+               shell-file-name
+               (org-trim body))))))
     (when value-is-exit-status
+      ;; last line is the output of "echo $?"
       (setq results (car (reverse (split-string results "\n" t)))))
     (when results
-      (let ((result-params (cdr (assq :result-params params))))
-        (org-babel-result-cond result-params
+      (org-babel-result-cond result-params
           results
           (let ((tmp-file (org-babel-temp-file "sh-")))
             (with-temp-file tmp-file (insert results))
-            (org-babel-import-elisp-from-file tmp-file)))))))
+            (org-babel-import-elisp-from-file tmp-file))))))
 
 (defun org-babel-sh-strip-weird-long-prompt (string)
   "Remove prompt cruft from a string of shell output."