Re: [patch, ox] #+INCLUDE resolves links

[Rasmus <rasmus@gmx.us>]

[-- Attachment #1: Type: text/plain, Size: 5293 bytes --]

Hi,

Thanks for the comments.  I hope I addressed the previous comments and
did not introduce new reasons bugs.
I added tests.

Comments on comments follow.

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

>> Okay, I hope I got it now.  It's a rather forgiving regexp in terms of
>> mistakes.  Is that OK?
>
> Please no ":only-contents yes", ":only-contents true", ":only-contents
> of_course!" in the regexp. If :only-contents is followed by anything but
> nil or another keyword, its value is non-nil. See below.

Good catch; I added explicit support for

     ¡?[oO][fF][-_]?[cC][Oo][uU][rR][sS][\.!¡]?

in the regexp!  ¡Gotta catch 'em all!

> The sentence is not complete. Also, it should be something like "If you
> set @code{:only-contents} property to a non-nil value, only...".
>> [...]
> This is not true anymore about the drawers. This should be merged with
> the previous phrase to avoid duplicating "@code{:only-contents}" (e.g.,
> only the contents of the matched element are inserted, without any
> planning line or property drawer).

Fixed this and other documentation bugs — hopefully.  Let me know if
it's clear.

>> +			    (let ((matched (save-match-data
>> +					     (org-split-string
>> + (org-remove-double-quotes (match-string 1 value)) "::"))))
>
> There's no reason to use `org-split-string' here since you only want to
> match the last "::". You can use the same regexp used by
> "org-element.el", i.e.
>
>   (when (string-match "::\\(.*\\)\\'" value)
>     (setq location (match-string 1 value)
>           value (replace-match "" nil nil value)))


 Custom_ID is very flexible.  I've use a similar regexp.

>> +		 (only-contents
>> +		  (and (string-match
>> + ":only-?contents?[[:space:]]*\"?\\(t\\|true\\|yes\\)?\"?"
>> +			value)
>> +		       (prog1 (and (match-string 1 value) t)
>> +			 (setq value (replace-match "" nil nil value)))))
>
>   (only-contents
>    (and (string-match ":only-contents +\\([^: \r\t\n]\\S-*\\)" value)
>         (org-not-nil (match-string 1 value))))

I have removed flexibility in speling.

>> +      (narrow-to-region
>> +       (org-element-property
>> +	(if only-contents :contents-begin :begin) (org-element-at-point))
>> + (org-element-property (if only-contents :contents-end :end)
>> (org-element-at-point))))
>
>   (let ((element (org-element-at-point)))
>     (let ((contents-beg
>            (and only-contents
>                 (org-element-property :contents-begin element))))
>       (narrow-to-region
>        (or contents-beg (org-element-property :begin element))
>        (org-element-property (if contents-beg :contents-end :end) element))))

Just out of curiosity, what is an example of a element that can be
named and does not have a :contents-begin?

>> +    (when only-contents
>> +      ;; skip drawers and property-drawers
>> +      ;; these are removed as needed in `org-export--prepare-file-contents'
>> +      ;; TODO: How to actually do this?  Only line numbers are send to
>> +      ;; `org-export--prepare-file-contents'.  split in two?
>> +      (goto-char (point-min))
>> +      (org-skip-whitespace)
>> +      (beginning-of-line)
>> +      (let ((element (org-element-at-point)))
>> +	(while (memq (org-element-type element) '(drawer property-drawer))
>> +	  (goto-char (org-element-property :end element))
>> +	  (setq element (org-element-at-point)))))
>
> Regular drawers are not expected to be skipped. Also, the following
> should be better
>
>   (when (and only-contents
>              (memq (org-element-type element) '(headline inlinetask)))
>     (goto-char (point-min))
>     (when (org-looking-at-p org-planning-line-re) (forward-line))
>     (when (looking-at org-property-drawer-re) (goto-char (match-end 0)))
>     (unless (bolp) (forward-line)))
>
> This should be obviously included within the previous `let'.

Okay, there's a lot of improvements in that suggestion.  However, it
misses this case which created using only "official" shortcuts

     * head
     SCHEDULED: <2014-09-28 sun>
     :LOGBOOK:
     - Note taken on [2014-09-28 sat 12:21] \\
       a drawer
     :END:
     :PROPERTIES:
     :CUSTOM_ID: h
     :END:

The patch handles something like this now cf. the last test.

>> +    (apply (lambda (beg end) (format "%s-%s" beg end))
>> +	   ;; `line-number-at-pos' returns the narrowed line-number
>> + (mapcar 'line-number-at-pos (prog1 (list (point-min) (point-max))
>> (widen))))))
>
> This is inefficient because `line-number-at-pos' will start counting
> twice from line 1.
>
>   (goto-char beg)
>   (widen)
>   (let ((start-line (line-number-at-pos)))
>     (format "%d-%d"
>             start-line
>             (+ start-line
>                (let ((c 0)) (while (< (point) end) (incf c) (forward-line)) c))))
>
> I didn't check, there may an off-by-one error. Anyway, all this needs
> tests.

Fine with me.  It's a bit less elegant IMO, but you are right.  I had
to do it slightly differently since the line number needs to appear
irrespective of whether lines are included in the call initially.
That being said, I could have very well overlooked some obvioues way
of doing it.

—Rasmus

-- 
Hooray!

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ox-Allow-file-links-with-INCLUDE-keyword.patch --]
[-- Type: text/x-diff, Size: 12731 bytes --]

From 0c0b4a4b9ea0db7ce3f8c08c9a2947db85c54470 Mon Sep 17 00:00:00 2001
From: Rasmus <rasmus@gmx.us>
Date: Sun, 28 Sep 2014 21:05:17 +0200
Subject: [PATCH] ox: Allow file-links with #+INCLUDE-keyword

* org.el (org-edit-special): Handle file-links for INCLUDE.
* ox.el (org-export--prepare-file-contents): Handle links and
add option no-heading.
* ox.el (org-export-expand-include-keyword): Resolve headline
links and add option :only-contents.
* orgguide.texi (Include files)
org.texi (Include files): Updated.
* testing/examples/include.org: New examples.
* test-ox.el (test-org-export/expand-include): New tests.
---
 doc/org.texi                 |  18 +++++++
 doc/orgguide.texi            |   9 +++-
 lisp/org.el                  |   8 +--
 lisp/ox.el                   | 117 +++++++++++++++++++++++++++++++++++++++----
 testing/examples/include.org |  25 +++++++++
 testing/lisp/test-ox.el      |  57 ++++++++++++++++++++-
 6 files changed, 218 insertions(+), 16 deletions(-)

diff --git a/doc/org.texi b/doc/org.texi
index 7d98d51..5a9c0c5 100644
--- a/doc/org.texi
+++ b/doc/org.texi
@@ -10008,6 +10008,24 @@ to use the obvious defaults.
 #+INCLUDE: "~/.emacs" :lines "10-"    @r{Include lines from 10 to EOF}
 @end example
 
+Finally, you may use a file-link, see @ref{search option in file links}, to
+extract an object as matched by @code{org-link-search}@footnote{Note that
+@code{org-link-search-must-match-exact-headline} is locally bound to non-nil.
+Therefore, @code{org-link-search} only matches headlines and named
+elements.}.  If the @code{:only-contents} property is non-nil, only the
+contents of the requested element will be included, omitting any
+property-drawers, planning-lines, attributes, captions etc.  The
+@code{:lines} keyword operates locally with respect to the requested element.
+Some examples:
+
+@example
+#+INCLUDE: "./paper.org::#theory" :only-contents t
+   @r{Include the body of the heading with the custom id @code{theory}}
+#+INCLUDE: "./paper.org::mytable"  @r{Include named element.}
+#+INCLUDE: "./paper.org::*conclusion" :lines 1-20
+   @r{Include the first 20 lines of the headline named conclusion.}
+@end example
+
 @table @kbd
 @kindex C-c '
 @item C-c '
diff --git a/doc/orgguide.texi b/doc/orgguide.texi
index ca8e052..4feeaca 100644
--- a/doc/orgguide.texi
+++ b/doc/orgguide.texi
@@ -2264,8 +2264,13 @@ include your @file{.emacs} file, you could use:
 The optional second and third parameter are the markup (i.e., @samp{example}
 or @samp{src}), and, if the markup is @samp{src}, the language for formatting
 the contents.  The markup is optional, if it is not given, the text will be
-assumed to be in Org mode format and will be processed normally. @kbd{C-c '}
-will visit the included file.
+assumed to be in Org mode format and will be processed normally.  File-links
+will be interpreted as well:
+@smallexample
+#+INCLUDE: "./otherfile.org::#my_custom_id" :only-contents t
+@end smallexample
+@noindent
+@kbd{C-c '} will visit the included file.
 
 @node Embedded @LaTeX{},  , Include files, Markup
 @section Embedded @LaTeX{}
diff --git a/lisp/org.el b/lisp/org.el
index b09e72d..c70e44a 100755
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -20522,9 +20522,11 @@ Otherwise, return a user error."
 		       session params))))))
       (keyword
        (if (member (org-element-property :key element) '("INCLUDE" "SETUPFILE"))
-           (find-file-other-window
-            (org-remove-double-quotes
-             (car (org-split-string (org-element-property :value element)))))
+           (org-open-link-from-string
+	    (format "[[%s]]"
+		    (expand-file-name
+		     (org-remove-double-quotes
+		      (car (org-split-string (org-element-property :value element)))))))
          (user-error "No special environment to edit here")))
       (table
        (if (eq (org-element-property :type element) 'table.el)
diff --git a/lisp/ox.el b/lisp/ox.el
index 59091fc..ed1024b 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -3321,13 +3321,25 @@ paths."
 	  ;; Extract arguments from keyword's value.
 	  (let* ((value (org-element-property :value element))
 		 (ind (org-get-indentation))
+		 location
 		 (file (and (string-match
 			     "^\\(\".+?\"\\|\\S-+\\)\\(?:\\s-+\\|$\\)" value)
-			    (prog1 (expand-file-name
-				    (org-remove-double-quotes
-				     (match-string 1 value))
-				    dir)
+			    (prog1
+				(save-match-data
+				  (let ((matched (match-string 1 value)))
+				    (when (string-match "\\(::\\(.*?\\)\\)\"?\\'" matched)
+				      (setq location (match-string 2 matched))
+				      (setq matched
+					    (replace-match "" nil nil  matched 1)))
+				    (expand-file-name
+				     (org-remove-double-quotes
+				      matched)
+				     dir)))
 			      (setq value (replace-match "" nil nil value)))))
+		 (only-contents
+		  (and (string-match ":only-contents +\\([^: \r\t\n]\\S-*\\)" value)
+		       (prog1 (org-not-nil (match-string 1 value))
+			 (setq value (replace-match "" nil nil value)))))
 		 (lines
 		  (and (string-match
 			":lines +\"\\(\\(?:[0-9]+\\)?-\\(?:[0-9]+\\)?\\)\""
@@ -3387,17 +3399,88 @@ paths."
 	       (t
 		(insert
 		 (with-temp-buffer
-		   (let ((org-inhibit-startup t)) (org-mode))
-		   (insert
-		    (org-export--prepare-file-contents
-		     file lines ind minlevel
-		     (or (gethash file file-prefix)
-			 (puthash file (incf current-prefix) file-prefix))))
+		   (let ((org-inhibit-startup t)
+			 (lines
+			  (if location
+			      (org-export--inclusion-absolute-lines
+			       file  location only-contents lines)
+			    lines)))
+		     (org-mode)
+		     (insert
+		      (org-export--prepare-file-contents
+		       file lines ind minlevel
+		       (or (gethash file file-prefix)
+			   (puthash file (incf current-prefix) file-prefix)))))
 		   (org-export-expand-include-keyword
 		    (cons (list file lines) included)
 		    (file-name-directory file))
 		   (buffer-string)))))))))))))
 
+(defun org-export--inclusion-absolute-lines (file location only-contents lines)
+  "Resolve absolute lines for an included file with file-link.
+
+FILE is string file-name of the file to include.  LOCATION is a
+string name within FILE to be included (located via
+`org-link-search').  If ONLY-CONTENTS is non-nil only the
+contents of the named element will be included, as determined
+Org-Element.  If LINES is non-nil only those lines are included.
+
+Return a string of lines to be included in the format expected by
+`org-export--prepare-file-contents'."
+  (with-temp-buffer
+    (insert-file-contents file)
+    (unless (eq major-mode 'org-mode)
+      (let ((org-inhibit-startup t)) (org-mode)))
+    (condition-case err
+	;; Enforce consistent search.
+	(let ((org-link-search-must-match-exact-headline t))
+	  (org-link-search location))
+      (error
+       (error (format "%s for %s::%s" (error-message-string err) file location))))
+    (let* ((element (org-element-at-point))
+	   (contents-begin
+	    (and only-contents (org-element-property :contents-begin element))))
+      (narrow-to-region
+       (or contents-begin (org-element-property :begin element))
+       (org-element-property (if contents-begin :contents-end :end) element))
+      (when (and only-contents
+		 (memq (org-element-type element) '(headline inlinetask)))
+	;; skip planning line and property-drawer.  If a normal drawer
+	;; precedes a property-drawer both will be included.
+	;; Remaining property-drawers are removed as needed in
+	;; `org-export--prepare-file-contents'
+	(goto-char (point-min))
+	(when (org-looking-at-p org-planning-line-re) (forward-line))
+	(when (looking-at org-property-drawer-re) (goto-char (match-end 0)))
+	(unless (bolp) (forward-line))
+	(narrow-to-region (point) (point-max))))
+    (when lines
+      (org-skip-whitespace)
+      (beginning-of-line)
+      (let* ((lines (split-string lines "-"))
+	     (lbeg (string-to-number (car lines)))
+	     (lend (string-to-number (cadr lines)))
+	     (beg (if (zerop lbeg) (point-min)
+		    (goto-char (point-min))
+		    (forward-line (1- lbeg))
+		    (point)))
+	     (end (if (zerop lend) (point-max)
+		    (goto-char beg)
+		    (forward-line (1-  lend))
+		    (point))))
+	(narrow-to-region beg end)))
+    (let ((end (point-max)))
+      (goto-char (point-min))
+      (widen)
+      (let ((start-line (line-number-at-pos)))
+	(format "%d-%d"
+		start-line
+		(save-excursion
+		  (+ start-line
+		     (let ((counter 0))
+		       (while (< (point) end) (incf counter) (forward-line))
+		       counter))))))))
+
 (defun org-export--prepare-file-contents (file &optional lines ind minlevel id)
   "Prepare the contents of FILE for inclusion and return them as a string.
 
@@ -3444,6 +3527,20 @@ with footnotes is included in a document."
     (skip-chars-backward " \r\t\n")
     (forward-line)
     (delete-region (point) (point-max))
+    ;; Remove property-drawers after drawers.
+    (when (or ind minlevel)
+      (unless (eq major-mode 'org-mode)
+    	(let ((org-inhibit-startup t)) (org-mode)))
+      (goto-char (point-min))
+      (when (looking-at org-drawer-regexp)
+	(goto-char (match-end 0))
+	(search-forward-regexp org-drawer-regexp)
+	(forward-line 1)
+	(beginning-of-line))
+      (when (looking-at org-property-drawer-re)
+	(delete-region (match-beginning 0) (match-end 0))
+	(beginning-of-line))
+      (delete-region (point) (save-excursion (and (org-skip-whitespace) (point)))))
     ;; If IND is set, preserve indentation of include keyword until
     ;; the first headline encountered.
     (when ind
diff --git a/testing/examples/include.org b/testing/examples/include.org
index 186facb..c04c942 100644
--- a/testing/examples/include.org
+++ b/testing/examples/include.org
@@ -8,3 +8,28 @@ Small Org file with an include keyword.
 
 * Heading
 body
+
+* Another heading
+:PROPERTIES:
+:CUSTOM_ID: ah
+:END:
+1
+2
+3
+
+* A headline with a table
+:PROPERTIES:
+:CUSTOM_ID: ht
+:END:
+#+CAPTION: a table
+#+NAME: tbl
+| 1 |
+
+* drawer-headline
+:LOGBOOK:
+drawer
+:END:
+:PROPERTIES:
+:CUSTOM_ID: dh
+:END:
+content
diff --git a/testing/lisp/test-ox.el b/testing/lisp/test-ox.el
index 4af3510..4706ec2 100644
--- a/testing/lisp/test-ox.el
+++ b/testing/lisp/test-ox.el
@@ -918,7 +918,62 @@ Footnotes[fn:1], [fn:test] and [fn:inline:anonymous footnote].
 		(org-export-expand-include-keyword)
 		(org-element-map (org-element-parse-buffer)
 		    'footnote-reference
-		  (lambda (ref) (org-element-property :label ref))))))))))))
+		  (lambda (ref) (org-element-property :label ref)))))))))))
+  ;; If only-contents is non-nil only include contents of element
+  (should
+   (equal
+    "body\n"
+    (org-test-with-temp-text
+	(format "#+INCLUDE: \"%s/examples/include.org::*Heading\" :only-contents t" org-test-dir)
+      (org-export-expand-include-keyword)
+      (buffer-string))))
+  ;; Headings can be included via CUSTOM_ID
+  (should
+   (org-test-with-temp-text
+	(format "#+INCLUDE: \"%s/examples/include.org::#ah\"" org-test-dir)
+     (org-export-expand-include-keyword)
+     (goto-char (point-min))
+     (looking-at "* Another heading")))
+  ;; Named objects can be included
+  (should
+   (equal
+    "| 1 |\n"
+    (org-test-with-temp-text
+	(format "#+INCLUDE: \"%s/examples/include.org::tbl\" :only-contents t" org-test-dir)
+      (org-export-expand-include-keyword)
+      (buffer-string))))
+  ;; Including non-existing elements should result in an error
+  (should-error
+   (org-test-with-temp-text
+	(format "#+INCLUDE: \"%s/examples/include.org::*non-existing heading\"" org-test-dir)
+     (org-export-expand-include-keyword)))
+  ;; Lines work relatively to an included element
+  (should
+   (equal
+    "2\n3\n"
+    (org-test-with-temp-text
+	(format "#+INCLUDE: \"%s/examples/include.org::#ah\" :only-contents t :lines \"2-3\"" org-test-dir)
+      (org-export-expand-include-keyword)
+      (buffer-string))))
+  ;; Properties should be dropped from headlines
+  (should
+   (equal
+    (org-test-with-temp-text
+	(format "#+INCLUDE: \"%s/examples/include.org::#ht\" :only-contents t" org-test-dir)
+      (org-export-expand-include-keyword)
+      (buffer-string))
+    (org-test-with-temp-text
+	(format "#+INCLUDE: \"%s/examples/include.org::tbl\"" org-test-dir)
+      (org-export-expand-include-keyword)
+      (buffer-string))))
+  ;; Properties should be dropped, drawers should not be
+  (should
+   (equal
+    ":LOGBOOK:\ndrawer\n:END:\ncontent\n"
+    (org-test-with-temp-text
+	(format "#+INCLUDE: \"%s/examples/include.org::#dh\" :only-contents t" org-test-dir)
+      (org-export-expand-include-keyword)
+      (buffer-string)))))
 
 (ert-deftest test-org-export/expand-macro ()
   "Test macro expansion in an Org buffer."
-- 
2.1.1