[PATCH] Support #+include-ing URLs

[PATCH] Support #+include-ing URLs

[Timothy]

[-- Attachment #1.1: Type: text/plain, Size: 218 bytes --]

Hi All,

This is just a little patchset to treat `#+include: URL' the same way as
`#+setupfile: URL'. All the usual `#+include:' bells and whistles (`::*Heading',
`:lines', etc.) work as normal.

All the best,
Timothy

[-- Attachment #1.2: Type: text/html, Size: 3326 bytes --]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0002-org-lint-don-t-complain-about-include-URL.patch --]
[-- Type: text/x-patch, Size: 4172 bytes --]

From df0b382e43cf44860247fafd14bd2932fe3ed026 Mon Sep 17 00:00:00 2001
From: TEC <tec@tecosaur.com>
Date: Sun, 5 Jun 2022 22:28:39 +0800
Subject: [PATCH 2/2] org-lint: don't complain about #+include URL

* lisp/org-lint.el (org-lint-wrong-include-link-parameter): When the
included file is a URL, skip the usual file checks.
---
 lisp/org-lint.el | 67 ++++++++++++++++++++++++------------------------
 1 file changed, 34 insertions(+), 33 deletions(-)

diff --git a/lisp/org-lint.el b/lisp/org-lint.el
index cce6fddbd..1e0dba4a0 100644
--- a/lisp/org-lint.el
+++ b/lisp/org-lint.el
@@ -649,39 +649,40 @@ (defun org-lint-wrong-include-link-parameter (ast)
   (org-element-map ast 'keyword
     (lambda (k)
       (when (equal (org-element-property :key k) "INCLUDE")
-	(let* ((value (org-element-property :value k))
-	       (path
-		(and (string-match "^\\(\".+\"\\|\\S-+\\)[ \t]*" value)
-		     (save-match-data
-		       (org-strip-quotes (match-string 1 value))))))
-	  (if (not path)
-	      (list (org-element-property :post-affiliated k)
-		    "Missing location argument in INCLUDE keyword")
-	    (let* ((file (org-string-nw-p
-			  (if (string-match "::\\(.*\\)\\'" path)
-			      (substring path 0 (match-beginning 0))
-			    path)))
-		   (search (and (not (equal file path))
-				(org-string-nw-p (match-string 1 path)))))
-	      (if (and file
-		       (not (file-remote-p file))
-		       (not (file-exists-p file)))
-		  (list (org-element-property :post-affiliated k)
-			"Non-existent file argument in INCLUDE keyword")
-		(let* ((visiting (if file (find-buffer-visiting file)
-				   (current-buffer)))
-		       (buffer (or visiting (find-file-noselect file)))
-		       (org-link-search-must-match-exact-headline t))
-		  (unwind-protect
-		      (with-current-buffer buffer
-			(when (and search
-				   (not (ignore-errors
-					  (org-link-search search nil t))))
-			  (list (org-element-property :post-affiliated k)
-				(format
-				 "Invalid search part \"%s\" in INCLUDE keyword"
-				 search))))
-		    (unless visiting (kill-buffer buffer))))))))))))
+        (let* ((value (org-element-property :value k))
+               (path
+                (and (string-match "^\\(\".+\"\\|\\S-+\\)[ \t]*" value)
+                     (save-match-data
+                       (org-strip-quotes (match-string 1 value))))))
+          (if (not path)
+              (list (org-element-property :post-affiliated k)
+                    "Missing location argument in INCLUDE keyword")
+            (let* ((file (org-string-nw-p
+                          (if (string-match "::\\(.*\\)\\'" path)
+                              (substring path 0 (match-beginning 0))
+                            path)))
+                   (search (and (not (equal file path))
+                                (org-string-nw-p (match-string 1 path)))))
+              (unless (org-url-p file)
+                (if (and file
+                         (not (file-remote-p file))
+                         (not (file-exists-p file)))
+                    (list (org-element-property :post-affiliated k)
+                          "Non-existent file argument in INCLUDE keyword")
+                  (let* ((visiting (if file (find-buffer-visiting file)
+                                     (current-buffer)))
+                         (buffer (or visiting (find-file-noselect file)))
+                         (org-link-search-must-match-exact-headline t))
+                    (unwind-protect
+                        (with-current-buffer buffer
+                          (when (and search
+                                     (not (ignore-errors
+                                            (org-link-search search nil t))))
+                            (list (org-element-property :post-affiliated k)
+                                  (format
+                                   "Invalid search part \"%s\" in INCLUDE keyword"
+                                   search))))
+                      (unless visiting (kill-buffer buffer)))))))))))))
 
 (defun org-lint-obsolete-include-markup (ast)
   (let ((regexp (format "\\`\\(?:\".+\"\\|\\S-+\\)[ \t]+%s"
-- 
2.36.1


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: 0001-ox-Support-include-ing-URLs.patch --]
[-- Type: text/x-patch, Size: 4174 bytes --]

From 2e2189c451c8e8bbd8461a626fe63e5e97dd4a07 Mon Sep 17 00:00:00 2001
From: TEC <tec@tecosaur.com>
Date: Sun, 5 Jun 2022 22:25:22 +0800
Subject: [PATCH 1/2] ox: Support #+include-ing URLs

* lisp/ox.el (org-export--prepare-file-contents,
org-export--inclusion-absolute-lines): Replace instances of
`(insert-file-contents FILE)' with `(insert (org-file-contents FILE))',
as in `org--collect-keywords-1'.
(org-export-expand-include-keyword): Tweak to accept a URL as FILE, and
not perform the standard "file exists and is readable" check.

* etc/ORG-NEWS: Mention this change in behaviour.
---
 etc/ORG-NEWS |  3 +++
 lisp/ox.el   | 22 +++++++++++++---------
 2 files changed, 16 insertions(+), 9 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 35af94f92..397cb668c 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -219,6 +219,9 @@ blocks to LaTeX. This requires the =fvextra=, =float=, and (by
 default, but not necessarily) =tcolorbox= LaTeX packages be
 installed. It uses Emacs' font-lock information, and so tends to
 produce results superior to Minted or Listings.
+*** Support for =#+include=-ing URLs
+
+=#+include: FILE= will now accept URLs as the file.
 
 ** New functions and changes in function arguments
 
diff --git a/lisp/ox.el b/lisp/ox.el
index 9a8e63046..fc0ee671f 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -3229,15 +3229,18 @@ (defun org-export-expand-include-keyword (&optional included dir footnotes)
 				       value)
 			 (prog1
 			     (save-match-data
-			       (let ((matched (match-string 1 value)))
+			       (let ((matched (match-string 1 value))
+                                     stripped)
 				 (when (string-match "\\(::\\(.*?\\)\\)\"?\\'"
 						     matched)
 				   (setq location (match-string 2 matched))
 				   (setq matched
 					 (replace-match "" nil nil matched 1)))
-				 (expand-file-name (org-strip-quotes matched)
-						   dir)))
-			   (setq value (replace-match "" nil nil value)))))
+                                 (setq stripped (org-strip-quotes matched))
+                                 (if (org-url-p stripped)
+                                     stripped
+                                   (expand-file-name stripped dir))))
+                           (setq value (replace-match "" nil nil value)))))
 		   (only-contents
 		    (and (string-match ":only-contents *\\([^: \r\t\n]\\S-*\\)?"
 				       value)
@@ -3273,7 +3276,7 @@ (defun org-export-expand-include-keyword (&optional included dir footnotes)
 	      (delete-region (point) (line-beginning-position 2))
 	      (cond
 	       ((not file) nil)
-	       ((not (file-readable-p file))
+	       ((and (not (org-url-p file)) (not (file-readable-p file)))
 		(error "Cannot include file %s" file))
 	       ;; Check if files has already been parsed.  Look after
 	       ;; inclusion lines too, as different parts of the same
@@ -3319,8 +3322,9 @@ (defun org-export-expand-include-keyword (&optional included dir footnotes)
 			 includer-file)))
 		     (org-export-expand-include-keyword
 		      (cons (list file lines) included)
-		      (file-name-directory file)
-		      footnotes)
+                      (unless (org-url-p file)
+                        (file-name-directory file))
+                      footnotes)
 		     (buffer-string)))))
 		;; Expand footnotes after all files have been
 		;; included.  Footnotes are stored at end of buffer.
@@ -3343,7 +3347,7 @@ (defun org-export--inclusion-absolute-lines (file location only-contents lines)
 Return a string of lines to be included in the format expected by
 `org-export--prepare-file-contents'."
   (with-temp-buffer
-    (insert-file-contents file)
+    (insert (org-file-contents file))
     (unless (eq major-mode 'org-mode)
       (let ((org-inhibit-startup t)) (org-mode)))
     (condition-case err
@@ -3448,7 +3452,7 @@ (defun org-export--prepare-file-contents
 Optional argument INCLUDER is the file name where the inclusion
 is to happen."
   (with-temp-buffer
-    (insert-file-contents file)
+    (insert (org-file-contents file))
     (when lines
       (let* ((lines (split-string lines "-"))
 	     (lbeg (string-to-number (car lines)))
-- 
2.36.1

Re: [PATCH] Support #+include-ing URLs

[Max Nikulin]

On 05/06/2022 21:32, Timothy wrote:
> 
> This is just a little patchset to treat #+include: URL the same way as 
> #+setupfile: URL. All the usual #+include: bells and whistles 
> (::*Heading, :lines, etc.) work as normal.

Is it possible to disable fetching remote files by some setting? If I 
remember correctly, e.g. XML engines can disable processing of remote 
entities as a security measure.

However some src block can flip such setting, so the only way is to run 
the process in a separate network namespace...

Re: [PATCH] Support #+include-ing URLs

[Fraga, Eric]

On Sunday,  5 Jun 2022 at 22:01, Max Nikulin wrote:
> Is it possible to disable fetching remote files by some setting? 

+1

I would not want automatic retrieval of URLs without confirmation.

-- 
: Eric S Fraga, with org release_9.5.4-521-g1105da in Emacs 29.0.50

Re: [PATCH] Support #+include-ing URLs

[Timothy]

[-- Attachment #1: Type: text/plain, Size: 682 bytes --]

Hi Eric,

>> Is it possible to disable fetching remote files by some setting?
>
> I would not want automatic retrieval of URLs without confirmation.

This already occurs with #+setupfile.

I think this is a good point to raise, but a slightly separate one as it
concerns the pre-existing behaviour of org-file-contents.

A new patch set introducing a defcustom with three values (t, prompt, nil) and
accompanying modification to org-file-contents (it’s the only use of
`url-retrieve-synchronously' other than org-feed I see) would probably be a decent
idea. If nobody makes such a patch in the next week or so I’ll take a punt at
this.

All the best,
Timothy

Re: [PATCH] Support #+include-ing URLs

[Fraga, Eric]

On Tuesday,  7 Jun 2022 at 19:27, Timothy wrote:
> This already occurs with #+setupfile.

Ah, interesting.  I had not realised this.  It definitely would be nice
to provide confirmation for setup files as well, then.

thank you,
eric

-- 
: Eric S Fraga, with org release_9.5.4-521-g1105da in Emacs 29.0.50

Re: [PATCH] Support #+include-ing URLs

[Timothy]

Hi All,

> This is just a little patchset to treat #+include: URL the same way as
> #+setupfile: URL. All the usual #+include: bells and whistles
> (::*Heading, :lines, etc.) work as normal.

Since it's been a week and nobody has raised any objections to this
functionality (though it has provoked a proposed change to
org-file-contents), I'm going to tentatively merge this and prepare a
patch for org-file-contents to add download settings.

All the best,
Timothy