Re: [patch, ox] #+INCLUDE resolves links

[Rasmus <rasmus@gmx.us>]

[-- Attachment #1: Type: text/plain, Size: 6978 bytes --]

Hi,

Okay, I hope I got a better patch here.  I do — for sure — present
another dreadfully long email!  Let's start.

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

>>> #+INCLUDE: file.org::#custom_id :noheadline :lines "3-"
>
> Is it `:only-contents' or `:no-headline'? Also ":kwd1 :kbd2 value" is
> usually a shortcut for ":kwd1 nil :kbd2 value" (at least in export
> attributes). Your example is thus confusing, you should include the
> expected value.
>
>   #+INCLUDE: "file.org::#custom_id" :only-contents t :lines "3-"

Well it's only-contents now.  Why?  It's more precise in terms of the
org-element terminology and it makes more sense when you include, say,
a table.

>> +elements.}.  If the keyword @code{:only-contents} is used, only the
>> contents
>> +of the element in included.  For headlines, drawers and properties
>                   ^^
>
>> +assumed to be in Org mode format and will be processed normally.
>> File-links
>> +will be interpret as well:
>            ^^^^^^^^^

Sorry about that.  Fixed.

>>  ;;; ox.el --- Generic Export Engine for Org Mode
>> -
>>  ;; Copyright (C) 2012-2014 Free Software Foundation, Inc.
>
> You can remove this chunk.

As above.  [I should somehow disable whitespace cleanup when in source repos].

>> +		 (only-contents
>> +		  (and (string-match
>> + ":\\(only-?contents?[[:space:]]*\\(?:'t\\|true\\|yes\\)?\\)"
>> value)
>
> This should be ":only-contents t" or ":only-contents nil".
> ":only-contents" alone can be tolerated as a shortcut for
> ":only-contents nil", but that's all.

Okay, I hope I got it now.  It's a rather forgiving regexp in terms of
mistakes.  Is that OK?


>> +		       (prog1 t
>> + (setq value (replace-match "" nil nil value)))))
>
> Since `replace-match' cannot return nil here, you can remove

I did it in another way in the new patch now since to allow for nil
values.

>   (prog1 t ...)
>
> wrapper. If you insist on ONLY-CONTENTS being t, then

No, it's changed.

>> + (org-export--prepare-file-contents file location only-contents
>> lines))))
>
> Couldn't location, only-contents and lines be merged into a single
> argument? At the moment, you are either short-circuiting or breaking
> guard against circular inclusions (which relies on a combination of
> file-name and lines).

I try to do that now.  So the arguments of
`org-export--prepare-file-contents' are now unchanged compared to
master.

Note that lonely property drawers are now unconditionally discard if
they are in the beginning of the buffer — including just after an
initial drawer.

> IOW, each include keyword could be defined as a triplet of file name,
> beginning and ending global positions. You could implement a helper
> function to translate FILE LOCATION and ONLY-CONTENTS into this
> triplet,
> which would then be passed to `org-export--prepare-file-contents'.

I just pass a string of line numbers like before.

>> -(defun org-export--prepare-file-contents (file &optional lines ind minlevel id)
>> +(defun org-export--prepare-file-contents (file &optional location only-contents lines ind minlevel id)
>>    "Prepare the contents of FILE for inclusion and return them as a string.
>>
>> +When optional argument LOCATION is a string the matching element
>> +identified using `org-link-search' is returned.  Note that
>> +`org-link-search-must-match-exact-headline' is locally set to
>> +non-nil.  When ONLY-CONTENTS is non-nil only the contents of the
>> +matched element in included.  If LOCATION is a headline and
>> +ONLY-CONTENTS is non-nil, drawers and property-drawers
>> +immediately following the first headline are also removed.
>> +
>>  When optional argument LINES is a string specifying a range of
>>  lines, include only those lines.
>>
>> @@ -3420,6 +3437,26 @@ This is useful to avoid conflicts when more than one Org file
>>  with footnotes is included in a document."
>>    (with-temp-buffer
>>      (insert-file-contents file)
>> +    (org-mode)
>
> You cannot enforce `org-mode' as the current major mode since you can
> include other file types.
>
>> +    (when location
>> +      (condition-case err
>> + ;; enforce consistency in search.
>> + (let ((org-link-search-must-match-exact-headline t))
>> +	    (org-link-search location))
>> +	;; helpful error messages
>> + (error (error (format "%s for %s::%s"
>> + (error-message-string err) file location))))
>> +      (narrow-to-region
>> +       (org-element-property
>> + (if only-contents :contents-begin :begin) (org-element-at-point))
>> + (org-element-property (if only-contents :contents-end :end)
>> (org-element-at-point)))
>> + ;; get rid of drawers and properties
>> +      (when only-contents
>> + (let ((element (org-element-at-point)))
>> + (while (member (org-element-type element) '(drawer
>> property-drawer))
>> + (delete-region (org-element-property :begin element)
>> + (org-element-property :end element))
>> + (setq element (org-element-at-point))))))
>
> This could be handled when building the triplet. However, please do
> not
> skip drawers (property drawers are fine), as you cannot tell what the
> contents are.

OK.

>>      (when lines
>>        (let* ((lines (split-string lines "-"))
>>  	     (lbeg (string-to-number (car lines)))
>> @@ -3495,7 +3532,7 @@ with footnotes is included in a document."
>>      (org-element-normalize-string (buffer-string))))
>>
>>  (defun org-export-execute-babel-code ()
>> -  "Execute every Babel code in the visible part of current buffer."
>> +  "ExecUte every Babel code in the visible part of current buffer."
>
> You can remove this chunk too.

Sorry.

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Rasmus <rasmus@gmx.us> writes:
>
>>> You cannot enforce `org-mode' as the current major mode since you can
>>> include other file types.
>>
>> But then I can't use org-element-at-point:
>>
>> (with-temp-buffer
>>   (text-mode) (insert "* test\nmy txt") (goto-char (point-min)) (org-element-at-point))
>>
>> (with-temp-buffer
>>   (org-mode) (insert "* test\nmy txt") (goto-char (point-min)) (org-element-at-point))
>
> `org-export--prepare-file-contents' is not called with the same number
> of arguments when it is an Org file. You can activate `org-mode' if you
> are in this situation (look for (when ind ...) and (when minlevel ...)
> in the function).

I hope it's OK now.  In the new `org-export--inclusion-absolute-lines'
I trigger org-mode when a location is given (e.g. file::#h3).  This
should be OK as only org files should be specified with this sort of
link (I believe) and further it's currently only called in the end of
`org-export-expand-include-keyword' when we know it's an org file.

It doesn't make sense to apply this to source files, right?

I will document the functions better before a final patch.

Thanks,
Rasmus

--
Don't panic!!!

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ox-Allow-file-links-with-INCLUDE-keyword.patch --]
[-- Type: text/x-diff, Size: 10238 bytes --]

From 29892a30446c58c42736b426d7385783d73579be Mon Sep 17 00:00:00 2001
From: Rasmus <rasmus@gmx.us>
Date: Tue, 23 Sep 2014 22:16:45 +0200
Subject: [PATCH] ox: Allow file-links with #+INCLUDE-keyword

* org.el (org-edit-special): Handle file-links for INCLUDE.
* ox.el (org-export--prepare-file-contents): Handle links and
add option no-heading.
* ox.el (org-export-expand-include-keyword): Resolve headline
links and add option :only-contents.
* orgguide.texi (Include files)
org.texi (Include files): Updated.
---
 doc/org.texi      |  18 +++++++++
 doc/orgguide.texi |   9 ++++-
 lisp/org.el       |   8 ++--
 lisp/ox.el        | 115 ++++++++++++++++++++++++++++++++++++++++++++++--------
 4 files changed, 129 insertions(+), 21 deletions(-)

diff --git a/doc/org.texi b/doc/org.texi
index 7d98d51..c3dd1c1 100644
--- a/doc/org.texi
+++ b/doc/org.texi
@@ -10008,6 +10008,24 @@ to use the obvious defaults.
 #+INCLUDE: "~/.emacs" :lines "10-"    @r{Include lines from 10 to EOF}
 @end example
 
+Finally, you may use a file-link, see @ref{External links}, to extract an
+object as matched by @code{org-link-search}@footnote{Note that
+@code{org-link-search-must-match-exact-headline} is locally bound to non-nil.
+Therefore, @code{org-link-search} only matches headlines and named
+elements.}.  If the keyword @code{:only-contents} is used, only the contents
+of the included element.  For headlines, drawers and properties
+immediately following the headline will not be included when using
+@code{:only-contents}.  The @code{:lines} keyword is local to the
+element in question.  Some examples:
+
+@example
+#+INCLUDE: "./paper.org::#theory" :only-contents
+   @r{Include the body of the heading with the custom id @code{theory}}
+#+INCLUDE: "./paper.org::mytable"  @r{Include tabel with name and caption.}
+#+INCLUDE: "./paper.org::*conclusion" :lines 1-20
+   @r{Include the first 20 lines of the headline named conclusion.}
+@end example
+
 @table @kbd
 @kindex C-c '
 @item C-c '
diff --git a/doc/orgguide.texi b/doc/orgguide.texi
index ca8e052..8f0098e 100644
--- a/doc/orgguide.texi
+++ b/doc/orgguide.texi
@@ -2264,8 +2264,13 @@ include your @file{.emacs} file, you could use:
 The optional second and third parameter are the markup (i.e., @samp{example}
 or @samp{src}), and, if the markup is @samp{src}, the language for formatting
 the contents.  The markup is optional, if it is not given, the text will be
-assumed to be in Org mode format and will be processed normally. @kbd{C-c '}
-will visit the included file.
+assumed to be in Org mode format and will be processed normally.  File-links
+will be interpreted as well:
+@smallexample
+#+INCLUDE: "./otherfile.org::#my_custom_id" :no-contents
+@end smallexample
+@noindent
+@kbd{C-c '} will visit the included file.
 
 @node Embedded @LaTeX{},  , Include files, Markup
 @section Embedded @LaTeX{}
diff --git a/lisp/org.el b/lisp/org.el
index 4ffe1e8..86a1bf9 100755
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -20520,9 +20520,11 @@ Otherwise, return a user error."
 		       session params))))))
       (keyword
        (if (member (org-element-property :key element) '("INCLUDE" "SETUPFILE"))
-           (find-file-other-window
-            (org-remove-double-quotes
-             (car (org-split-string (org-element-property :value element)))))
+           (org-open-link-from-string
+	    (format "[[%s]]"
+		    (expand-file-name
+		     (org-remove-double-quotes
+		      (car (org-split-string (org-element-property :value element)))))))
          (user-error "No special environment to edit here")))
       (table
        (if (eq (org-element-property :type element) 'table.el)
diff --git a/lisp/ox.el b/lisp/ox.el
index f01f951..122e62a 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -3321,13 +3321,24 @@ paths."
 	  ;; Extract arguments from keyword's value.
 	  (let* ((value (org-element-property :value element))
 		 (ind (org-get-indentation))
+		 location
 		 (file (and (string-match
 			     "^\\(\".+?\"\\|\\S-+\\)\\(?:\\s-+\\|$\\)" value)
-			    (prog1 (expand-file-name
-				    (org-remove-double-quotes
-				     (match-string 1 value))
-				    dir)
-			      (setq value (replace-match "" nil nil value)))))
+			    (let ((matched (save-match-data
+					     (org-split-string
+					      (org-remove-double-quotes (match-string 1 value)) "::"))))
+			      (setq location (car-safe (cdr-safe matched)))
+			      (prog1 (expand-file-name
+				      (car matched)
+				      dir)
+				(setq value (replace-match "" nil nil value))))))
+
+		 (only-contents
+		  (and (string-match
+			":only-?contents?[[:space:]]*\"?\\(t\\|true\\|yes\\)?\"?"
+			value)
+		       (prog1 (and (match-string 1 value) t)
+			 (setq value (replace-match "" nil nil value)))))
 		 (lines
 		  (and (string-match
 			":lines +\"\\(\\(?:[0-9]+\\)?-\\(?:[0-9]+\\)?\\)\""
@@ -3370,34 +3381,92 @@ paths."
 		(insert
 		 (let ((ind-str (make-string ind ? ))
 		       (arg-str (if (stringp src-args)
-				  (format " %s" src-args)
-				""))
+				    (format " %s" src-args)
+				  ""))
 		       (contents
 			(org-escape-code-in-string
-			 (org-export--prepare-file-contents file lines))))
+			 (org-export--prepare-file-contents file nil nil lines))))
 		   (format "%s#+BEGIN_%s%s\n%s%s#+END_%s\n"
 			   ind-str block arg-str contents ind-str block))))
 	       ((stringp block)
 		(insert
 		 (let ((ind-str (make-string ind ? ))
 		       (contents
-			 (org-export--prepare-file-contents file lines)))
+			(org-export--prepare-file-contents file lines)))
 		   (format "%s#+BEGIN_%s\n%s%s#+END_%s\n"
 			   ind-str block contents ind-str block))))
 	       (t
 		(insert
 		 (with-temp-buffer
-		   (let ((org-inhibit-startup t)) (org-mode))
-		   (insert
-		    (org-export--prepare-file-contents
-		     file lines ind minlevel
-		     (or (gethash file file-prefix)
-			 (puthash file (incf current-prefix) file-prefix))))
+		   (let ((org-inhibit-startup t)
+			 (lines (org-export--inclusion-absolute-lines
+				 file  location only-contents lines)))
+		     (org-mode)
+		     (insert
+		      (org-export--prepare-file-contents
+		       file lines ind minlevel
+		       (or (gethash file file-prefix)
+			   (puthash file (incf current-prefix) file-prefix)))))
 		   (org-export-expand-include-keyword
 		    (cons (list file lines) included)
 		    (file-name-directory file))
 		   (buffer-string)))))))))))))
 
+(defun org-export--inclusion-absolute-lines (file &optional location only-contents lines)
+  "Resolve absolute lines for an INCLUDEd file given arguments.
+
+Inputs are the FILE to be included, the LOCATION in FILE, whether
+to ONLY-CONTENTS should be included and lines.
+
+Return absolute lines as a string."
+  (with-temp-buffer
+    (insert-file-contents file)
+    (when location
+      ;; locations are only defined for org files so
+      ;; OK to start org-mode.
+      (condition-case err
+	  ;; enforce consistency in search.
+	  (let ((org-link-search-must-match-exact-headline t))
+	    (org-link-search location))
+	;; helpful error messages
+	(error
+	 (error (format "%s for %s::%s" (error-message-string err) file location))))
+      (org-mode)
+      (narrow-to-region
+       (org-element-property
+	(if only-contents :contents-begin :begin) (org-element-at-point))
+       (org-element-property (if only-contents :contents-end :end) (org-element-at-point))))
+    (when only-contents
+      ;; skip drawers and property-drawers
+      ;; these are removed as needed in `org-export--prepare-file-contents'
+      ;; TODO: How to actually do this?  Only line numbers are send to
+      ;; `org-export--prepare-file-contents'.  split in two?
+      (goto-char (point-min))
+      (org-skip-whitespace)
+      (beginning-of-line)
+      (let ((element (org-element-at-point)))
+	(while (memq (org-element-type element) '(drawer property-drawer))
+	  (goto-char (org-element-property :end element))
+	  (setq element (org-element-at-point)))))
+    (when lines
+      (org-skip-whitespace)
+      (beginning-of-line)
+      (let* ((lines (split-string lines "-"))
+	     (lbeg (string-to-number (car lines)))
+	     (lend (string-to-number (cadr lines)))
+	     (beg (if (zerop lbeg) (point-min)
+		    (goto-char (point-min))
+		    (forward-line (1- lbeg))
+		    (point)))
+	     (end (if (zerop lend) (point-max)
+		    (goto-char (point-min))
+		    (forward-line (1- lend))
+		    (point))))
+	(narrow-to-region beg end)))
+    (apply (lambda (beg end) (format "%s-%s" beg end))
+	   ;; `line-number-at-pos' returns the narrowed line-number
+	   (mapcar 'line-number-at-pos (prog1  (list (point-min) (point-max)) (widen))))))
+
 (defun org-export--prepare-file-contents (file &optional lines ind minlevel id)
   "Prepare the contents of FILE for inclusion and return them as a string.
 
@@ -3419,7 +3488,8 @@ each footnote definition and reference if FILE is an Org file.
 This is useful to avoid conflicts when more than one Org file
 with footnotes is included in a document."
   (with-temp-buffer
-    (insert-file-contents file)
+   (switch-to-buffer-other-window (current-buffer))
+   (insert-file-contents file)
     (when lines
       (let* ((lines (split-string lines "-"))
 	     (lbeg (string-to-number (car lines)))
@@ -3449,6 +3519,19 @@ with footnotes is included in a document."
     (when ind
       (unless (eq major-mode 'org-mode)
 	(let ((org-inhibit-startup t)) (org-mode)))
+      ;; make sure that there is not an immediate "lonely"
+      ;; property-drawer.  Normal drawers are OK but property-drawers
+      ;; may follow normal drawers.
+      (goto-char (point-min))
+      (let ((element (org-element-at-point)))
+	(while (memq (org-element-type element) '(drawer property-drawer))
+	  ;; entering here the first element is not a heading so it's
+	  ;; safe to get rid of property-drawers.
+	  (if (eq (org-element-type element) 'property-drawer)
+	      (delete-region (org-element-property :begin element)
+			     (org-element-property :end element))
+	  (goto-char (org-element-property :end element)))
+	  (setq element (org-element-at-point))))
       (goto-char (point-min))
       (let ((ind-str (make-string ind ? )))
 	(while (not (or (eobp) (looking-at org-outline-regexp-bol)))
-- 
2.1.0