[PATCH] Add new keyword :coding for #+include directive

[PATCH] Add new keyword :coding for #+include directive

[Pierre Téchoueyres]

Hello org's developpers,

I want to propose the attached patch which allow you to specify an
optionnal `:coding' keyword to the `#+INCLUDE:' directive.

This allow you to specify something like
#+begin_example
,#+INCLUDE: "myfile.cmd" src cmd :coding "cp850-dos"
#+end_example
Which allow you to have different encoding for your various sources
files.

This allow me to include localised Microsoft Windows batch sources
inside my utf-8-unix org-files.

Thanks in advance for your remarks.

Pierre

Re: [PATCH] Add new keyword :coding for #+include directive

[Pierre Téchoueyres]

[-- Attachment #1: Type: text/plain, Size: 561 bytes --]


Hello org's developpers,
Sorry, I forgot the patch.
so here is the whole mail + patch:

I want to propose the attached patch which allow you to specify an
optionnal `:coding' keyword to the `#+INCLUDE:' directive.

This allow you to specify something like
#+begin_example
,#+INCLUDE: "myfile.cmd" src cmd :coding "cp850-dos"
#+end_example
Which allow you to have different encoding for your various sources
files.

This allow me to include localised Microsoft Windows batch sources
inside my utf-8-unix org-files.

Thanks in advance for your remarks.

Pierre


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: add-keyword-to-include --]
[-- Type: text/x-patch, Size: 6517 bytes --]

From 5200ea1fed3f13f429615baa0f196f512ca47fe7 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Pierre=20T=C3=A9choueyres?= <pierre.techoueyres@free.fr>
Date: Mon, 16 Apr 2018 21:20:17 +0200
Subject: [PATCH] Add new keyword :coding for #+include directive

* lisp/ox.el (org-export-expand-include-keyword): Add new keyword
  `:coding' for specify the file encoding whith the `#+include:'
  directive.

This allow to use somting like :
when your org-file is encoded in utf-8.
---
 etc/ORG-NEWS |  10 ++++-
 lisp/ox.el   | 117 +++++++++++++++++++++++++++++++----------------------------
 2 files changed, 71 insertions(+), 56 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 013c7b139..7a526d737 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -108,6 +108,14 @@ You can use =ob-scala.el= as packaged in scala-mode, available from the
 MELPA repository.
 
 ** New features
+*** New keyword for ~#+include:~ directive
+Add ~:coding "codign-system"~ keyword to allow include of files from
+different codign system than the main org-file.
+For example:
+#+begin_example
+,#+INCLUDE: "myfile.cmd" src cmd :coding "cp850-dos"
+#+end_example
+
 *** iCalendar export uses inheritance for TIMEZONE and LOCATION properties
 Both these properties can be inherited during iCalendar export,
 depending on the value of ~org-use-property-inheritance~.
@@ -514,7 +522,7 @@ want to take over maintenance of this compatibility, please contact
 our mailing list.
 
 *** New syntax for export blocks
-
+    
 Export blocks are explicitly marked as such at the syntax level to
 disambiguate their parsing from special blocks.  The new syntax is
 
diff --git a/lisp/ox.el b/lisp/ox.el
index ea7d1dc81..9423ae090 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -3326,6 +3326,12 @@ storing and resolving footnotes.  It is created automatically."
 			  value)
 			 (prog1 (match-string 1 value)
 			   (setq value (replace-match "" nil nil value)))))
+		   (coding
+		    (intern (or (and (string-match
+				      ":coding +\"\\([^\"]+\\)\"" value)
+				     (prog1 (match-string 1 value)
+				       (setq value (replace-match "" nil nil value))))
+				(symbol-name buffer-file-coding-system))))
 		   (env (cond
 			 ((string-match "\\<example\\>" value) 'literal)
 			 ((string-match "\\<export\\(?: +\\(.*\\)\\)?" value)
@@ -3348,63 +3354,64 @@ storing and resolving footnotes.  It is created automatically."
 			       (match-string 1 value))))
 	      ;; Remove keyword.
 	      (delete-region (point) (line-beginning-position 2))
-	      (cond
-	       ((not file) nil)
-	       ((not (file-readable-p file))
-		(error "Cannot include file %s" file))
-	       ;; Check if files has already been parsed.  Look after
-	       ;; inclusion lines too, as different parts of the same
-	       ;; file can be included too.
-	       ((member (list file lines) included)
-		(error "Recursive file inclusion: %s" file))
-	       (t
+	      (let ((coding-system-for-read coding))
 		(cond
-		 ((eq env 'literal)
-		  (insert
-		   (let ((ind-str (make-string ind ?\s))
-			 (arg-str (if (stringp args) (format " %s" args) ""))
-			 (contents
-			  (org-escape-code-in-string
-			   (org-export--prepare-file-contents file lines))))
-		     (format "%s#+BEGIN_%s%s\n%s%s#+END_%s\n"
-			     ind-str block arg-str contents ind-str block))))
-		 ((stringp block)
-		  (insert
-		   (let ((ind-str (make-string ind ?\s))
-			 (contents
-			  (org-export--prepare-file-contents file lines)))
-		     (format "%s#+BEGIN_%s\n%s%s#+END_%s\n"
-			     ind-str block contents ind-str block))))
+		 ((not file) nil)
+		 ((not (file-readable-p file))
+		  (error "Cannot include file %s" file))
+		 ;; Check if files has already been parsed.  Look after
+		 ;; inclusion lines too, as different parts of the same
+		 ;; file can be included too.
+		 ((member (list file lines) included)
+		  (error "Recursive file inclusion: %s" file))
 		 (t
-		  (insert
-		   (with-temp-buffer
-		     (let ((org-inhibit-startup t)
-			   (lines
-			    (if location
-				(org-export--inclusion-absolute-lines
-				 file location only-contents lines)
-			      lines)))
-		       (org-mode)
-		       (insert
-			(org-export--prepare-file-contents
-			 file lines ind minlevel
-			 (or
-			  (gethash file file-prefix)
-			  (puthash file (cl-incf current-prefix) file-prefix))
-			 footnotes)))
-		     (org-export-expand-include-keyword
-		      (cons (list file lines) included)
-		      (file-name-directory file)
-		      footnotes)
-		     (buffer-string)))))
-		;; Expand footnotes after all files have been
-		;; included.  Footnotes are stored at end of buffer.
-		(unless included
-		  (org-with-wide-buffer
-		   (goto-char (point-max))
-		   (maphash (lambda (k v)
-			      (insert (format "\n[fn:%s] %s\n" k v)))
-			    footnotes))))))))))))
+		  (cond
+		   ((eq env 'literal)
+		    (insert
+		     (let ((ind-str (make-string ind ?\s))
+			   (arg-str (if (stringp args) (format " %s" args) ""))
+			   (contents
+			    (org-escape-code-in-string
+			     (org-export--prepare-file-contents file lines))))
+		       (format "%s#+BEGIN_%s%s\n%s%s#+END_%s\n"
+			       ind-str block arg-str contents ind-str block))))
+		   ((stringp block)
+		    (insert
+		     (let ((ind-str (make-string ind ?\s))
+			   (contents
+			    (org-export--prepare-file-contents file lines)))
+		       (format "%s#+BEGIN_%s\n%s%s#+END_%s\n"
+			       ind-str block contents ind-str block))))
+		   (t
+		    (insert
+		     (with-temp-buffer
+		       (let ((org-inhibit-startup t)
+			     (lines
+			      (if location
+				  (org-export--inclusion-absolute-lines
+				   file location only-contents lines)
+				lines)))
+			 (org-mode)
+			 (insert
+			  (org-export--prepare-file-contents
+			   file lines ind minlevel
+			   (or
+			    (gethash file file-prefix)
+			    (puthash file (cl-incf current-prefix) file-prefix))
+			   footnotes)))
+		       (org-export-expand-include-keyword
+			(cons (list file lines) included)
+			(file-name-directory file)
+			footnotes)
+		       (buffer-string)))))
+		  ;; Expand footnotes after all files have been
+		  ;; included.  Footnotes are stored at end of buffer.
+		  (unless included
+		    (org-with-wide-buffer
+		     (goto-char (point-max))
+		     (maphash (lambda (k v)
+				(insert (format "\n[fn:%s] %s\n" k v)))
+			      footnotes)))))))))))))
 
 (defun org-export--inclusion-absolute-lines (file location only-contents lines)
   "Resolve absolute lines for an included file with file-link.
-- 
2.14.3

Re: [PATCH] Add new keyword :coding for #+include directive

[Nicolas Goaziou]

Hello,

pierre.techoueyres@free.fr (Pierre Téchoueyres) writes:

> I want to propose the attached patch which allow you to specify an
> optionnal `:coding' keyword to the `#+INCLUDE:' directive.

Thank you.

> This allow you to specify something like
>
> #+begin_example
> ,#+INCLUDE: "myfile.cmd" src cmd :coding "cp850-dos"
> #+end_example

The quotes are not necessary. AFAICT, coding systems do not contain
spaces.

> Which allow you to have different encoding for your various sources
> files.
>
> This allow me to include localised Microsoft Windows batch sources
> inside my utf-8-unix org-files.

Is it really an Org problem? E.g., couldn't you put a coding: cookie in
your ".cmd" file? IMO, the coding system depends on the includee, not
the includer.

WDYT?

Regards,

-- 
Nicolas Goaziou

Re: [PATCH] Add new keyword :coding for #+include directive

[Pierre Téchoueyres]

[-- Attachment #1: Type: text/plain, Size: 361 bytes --]

Hello,

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Hello,
> ...
>> This allow you to specify something like
>>
>> #+begin_example
>> ,#+INCLUDE: "myfile.cmd" src cmd :coding "cp850-dos"
>> #+end_example
>
> The quotes are not necessary. AFAICT, coding systems do not contain
> spaces.
Ok, I've reworked the rexexp to suppress them. New patch attached


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: include-with-coding --]
[-- Type: text/x-patch, Size: 6525 bytes --]

> From fcf191842bea64442d69a0fcfa927a046d8fbd92 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Pierre=20T=C3=A9choueyres?= <pierre.techoueyres@free.fr>
Date: Mon, 16 Apr 2018 21:20:17 +0200
Subject: [PATCH] Add new keyword :coding for #+include directive

* lisp/ox.el (org-export-expand-include-keyword): Add new keyword
  `:coding' for specify the file encoding whith the `#+include:'
  directive.

This allow to use somting like :
when your org-file is encoded in utf-8.
---
 etc/ORG-NEWS |  10 ++++-
 lisp/ox.el   | 117 +++++++++++++++++++++++++++++++----------------------------
 2 files changed, 71 insertions(+), 56 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 013c7b139..f285b7337 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -108,6 +108,14 @@ You can use =ob-scala.el= as packaged in scala-mode, available from the
 MELPA repository.
 
 ** New features
+*** New keyword for ~#+include:~ directive
+Add ~:coding "codign-system"~ keyword to allow include of files from
+different codign system than the main org-file.
+For example:
+#+begin_example
+,#+INCLUDE: "myfile.cmd" src cmd :coding cp850-dos
+#+end_example
+
 *** iCalendar export uses inheritance for TIMEZONE and LOCATION properties
 Both these properties can be inherited during iCalendar export,
 depending on the value of ~org-use-property-inheritance~.
@@ -514,7 +522,7 @@ want to take over maintenance of this compatibility, please contact
 our mailing list.
 
 *** New syntax for export blocks
-
+    
 Export blocks are explicitly marked as such at the syntax level to
 disambiguate their parsing from special blocks.  The new syntax is
 
diff --git a/lisp/ox.el b/lisp/ox.el
index ea7d1dc81..f4c5660ff 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -3326,6 +3326,12 @@ storing and resolving footnotes.  It is created automatically."
 			  value)
 			 (prog1 (match-string 1 value)
 			   (setq value (replace-match "" nil nil value)))))
+		   (coding
+		    (intern (or (and (string-match
+				      ":coding +\\<\\([a-z0-9\\-]+\\)\\>" value)
+				     (prog1 (match-string 1 value)
+				       (setq value (replace-match "" nil nil value))))
+				(symbol-name buffer-file-coding-system))))
 		   (env (cond
 			 ((string-match "\\<example\\>" value) 'literal)
 			 ((string-match "\\<export\\(?: +\\(.*\\)\\)?" value)
@@ -3348,63 +3354,64 @@ storing and resolving footnotes.  It is created automatically."
 			       (match-string 1 value))))
 	      ;; Remove keyword.
 	      (delete-region (point) (line-beginning-position 2))
-	      (cond
-	       ((not file) nil)
-	       ((not (file-readable-p file))
-		(error "Cannot include file %s" file))
-	       ;; Check if files has already been parsed.  Look after
-	       ;; inclusion lines too, as different parts of the same
-	       ;; file can be included too.
-	       ((member (list file lines) included)
-		(error "Recursive file inclusion: %s" file))
-	       (t
+	      (let ((coding-system-for-read coding))
 		(cond
-		 ((eq env 'literal)
-		  (insert
-		   (let ((ind-str (make-string ind ?\s))
-			 (arg-str (if (stringp args) (format " %s" args) ""))
-			 (contents
-			  (org-escape-code-in-string
-			   (org-export--prepare-file-contents file lines))))
-		     (format "%s#+BEGIN_%s%s\n%s%s#+END_%s\n"
-			     ind-str block arg-str contents ind-str block))))
-		 ((stringp block)
-		  (insert
-		   (let ((ind-str (make-string ind ?\s))
-			 (contents
-			  (org-export--prepare-file-contents file lines)))
-		     (format "%s#+BEGIN_%s\n%s%s#+END_%s\n"
-			     ind-str block contents ind-str block))))
+		 ((not file) nil)
+		 ((not (file-readable-p file))
+		  (error "Cannot include file %s" file))
+		 ;; Check if files has already been parsed.  Look after
+		 ;; inclusion lines too, as different parts of the same
+		 ;; file can be included too.
+		 ((member (list file lines) included)
+		  (error "Recursive file inclusion: %s" file))
 		 (t
-		  (insert
-		   (with-temp-buffer
-		     (let ((org-inhibit-startup t)
-			   (lines
-			    (if location
-				(org-export--inclusion-absolute-lines
-				 file location only-contents lines)
-			      lines)))
-		       (org-mode)
-		       (insert
-			(org-export--prepare-file-contents
-			 file lines ind minlevel
-			 (or
-			  (gethash file file-prefix)
-			  (puthash file (cl-incf current-prefix) file-prefix))
-			 footnotes)))
-		     (org-export-expand-include-keyword
-		      (cons (list file lines) included)
-		      (file-name-directory file)
-		      footnotes)
-		     (buffer-string)))))
-		;; Expand footnotes after all files have been
-		;; included.  Footnotes are stored at end of buffer.
-		(unless included
-		  (org-with-wide-buffer
-		   (goto-char (point-max))
-		   (maphash (lambda (k v)
-			      (insert (format "\n[fn:%s] %s\n" k v)))
-			    footnotes))))))))))))
+		  (cond
+		   ((eq env 'literal)
+		    (insert
+		     (let ((ind-str (make-string ind ?\s))
+			   (arg-str (if (stringp args) (format " %s" args) ""))
+			   (contents
+			    (org-escape-code-in-string
+			     (org-export--prepare-file-contents file lines))))
+		       (format "%s#+BEGIN_%s%s\n%s%s#+END_%s\n"
+			       ind-str block arg-str contents ind-str block))))
+		   ((stringp block)
+		    (insert
+		     (let ((ind-str (make-string ind ?\s))
+			   (contents
+			    (org-export--prepare-file-contents file lines)))
+		       (format "%s#+BEGIN_%s\n%s%s#+END_%s\n"
+			       ind-str block contents ind-str block))))
+		   (t
+		    (insert
+		     (with-temp-buffer
+		       (let ((org-inhibit-startup t)
+			     (lines
+			      (if location
+				  (org-export--inclusion-absolute-lines
+				   file location only-contents lines)
+				lines)))
+			 (org-mode)
+			 (insert
+			  (org-export--prepare-file-contents
+			   file lines ind minlevel
+			   (or
+			    (gethash file file-prefix)
+			    (puthash file (cl-incf current-prefix) file-prefix))
+			   footnotes)))
+		       (org-export-expand-include-keyword
+			(cons (list file lines) included)
+			(file-name-directory file)
+			footnotes)
+		       (buffer-string)))))
+		  ;; Expand footnotes after all files have been
+		  ;; included.  Footnotes are stored at end of buffer.
+		  (unless included
+		    (org-with-wide-buffer
+		     (goto-char (point-max))
+		     (maphash (lambda (k v)
+				(insert (format "\n[fn:%s] %s\n" k v)))
+			      footnotes)))))))))))))
 
 (defun org-export--inclusion-absolute-lines (file location only-contents lines)
   "Resolve absolute lines for an included file with file-link.
-- 
2.14.3


[-- Attachment #3: Type: text/plain, Size: 411 bytes --]



> ...
> Is it really an Org problem? E.g., couldn't you put a coding: cookie in
> your ".cmd" file? IMO, the coding system depends on the includee, not
> the includer.

I tend to aggree with you that TRTDT is to put cookies or something
inside the included file. But :
  a) This seem to not work as expected (see exemples joinned).
  b) Sometimes you can't modify the included file (ex: remote file access).


[-- Attachment #4: test.org --]
[-- Type: text/x-org, Size: 374 bytes --]

# -*- coding: utf-8 -*-
#+title: test with different encodings.

* Pierre Téchoueyres
  bad
  #+include: "./file.1.txt" src bat
  or
  #+include: "./file.2.txt" src sh
 

  and good ? 
  #+include: "./file.1.txt" src bat :coding cp850-dos
  or
  #+include: "./file.2.txt" src sh :coding iso-8859-15-unix
 

# Local Variables:
# coding: utf-8-unix
# End:

[-- Attachment #5: file.1.txt --]
[-- Type: text/plain, Size: 133 bytes --]

::- -*- coding: cp850-dos -*-
@echo off

echo Hello Pierre Téchoueyres

rem Local Variables:
rem coding: cp850-dos
rem End:

[-- Attachment #6: file.2.txt --]
[-- Type: text/plain, Size: 140 bytes --]

#!/bin/sh # -*- coding: iso-8859-15-unix -*-

echo "Hello Pierre Téchoueyres"

# Local Variables:
# coding: iso-8859-15-unix
# End:

[-- Attachment #7: Type: text/plain, Size: 101 bytes --]


For a I can try to jump into the rabbit hole and find a solution (but
 for now I'm lost)


Regards,

Re: [PATCH] Add new keyword :coding for #+include directive

[Pierre Téchoueyres]

[-- Attachment #1: Type: text/plain, Size: 701 bytes --]

pierre.techoueyres@free.fr (Pierre Téchoueyres) writes:

Hello,
> ...
>> ...
>> Is it really an Org problem? E.g., couldn't you put a coding: cookie in
>> your ".cmd" file? IMO, the coding system depends on the includee, not
>> the includer.
>
> I tend to aggree with you that TRTDT is to put cookies or something
> inside the included file. But :
>   a) This seem to not work as expected (see exemples joinned).
>   b) Sometimes you can't modify the included file (ex: remote file access).
> ...
> For a I can try to jump into the rabbit hole and find a solution (but
>  for now I'm lost)

I did my homework : found a fix for a) and rebased previous patch on master.

Regards,


[-- Attachment #2: Add-new-keyword-coding-for-include-directive --]
[-- Type: text/x-patch, Size: 6420 bytes --]

From 679fa184dff97f1a5ff617144dc7d87e95b8ccd9 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Pierre=20T=C3=A9choueyres?= <pierre.techoueyres@free.fr>
Date: Sat, 21 Apr 2018 00:31:10 +0200
Subject: [PATCH] Add new keyword :coding for #+include directive

* lisp/ox.el (org-export-expand-include-keyword): Add new keyword
  `:coding' for specify the file encoding whith the `#+include:'
  directive.

This allow to use somting like :
  #+include: "./myfile" :coding cp850-dos
when your org-file is encoded in utf-8 for example.
---
 etc/ORG-NEWS |   7 ++++
 lisp/ox.el   | 121 +++++++++++++++++++++++++++++++----------------------------
 2 files changed, 71 insertions(+), 57 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index abebe08fe..8b717d8f3 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -121,6 +121,13 @@ now sort according to the locale’s collation rules instead of by
 code-point.
 
 ** New features
+*** New keyword for ~#+include:~ directive
+Add ~:coding "codign-system"~ keyword to allow include of files from
+different codign system than the main org-file.
+For example:
+#+begin_example
+,#+INCLUDE: "myfile.cmd" src cmd :coding cp850-dos
+#+end_example
 *** Add ~:results link~ support for Babel
 
 With this output format, create a link to the file specified in
diff --git a/lisp/ox.el b/lisp/ox.el
index 5a83ae01d..e75030ffb 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -3298,6 +3298,12 @@ storing and resolving footnotes.  It is created automatically."
 				  (org-unbracket-string "\"" "\"" matched)
 				  dir)))
 			   (setq value (replace-match "" nil nil value)))))
+		   (coding
+		    (intern (or (and (string-match
+				      ":coding +\\<\\([a-z0-9\\-]+\\)\\>" value)
+				     (prog1 (match-string 1 value)
+				       (setq value (replace-match "" nil nil value))))
+				(symbol-name coding-system-for-read))))
 		   (only-contents
 		    (and (string-match ":only-contents *\\([^: \r\t\n]\\S-*\\)?"
 				       value)
@@ -3331,65 +3337,66 @@ storing and resolving footnotes.  It is created automatically."
 			       (match-string 1 value))))
 	      ;; Remove keyword.
 	      (delete-region (point) (line-beginning-position 2))
-	      (cond
-	       ((not file) nil)
-	       ((not (file-readable-p file))
-		(error "Cannot include file %s" file))
-	       ;; Check if files has already been parsed.  Look after
-	       ;; inclusion lines too, as different parts of the same
-	       ;; file can be included too.
-	       ((member (list file lines) included)
-		(error "Recursive file inclusion: %s" file))
-	       (t
+	      (let ((coding-system-for-read coding))
 		(cond
-		 ((eq env 'literal)
-		  (insert
-		   (let ((ind-str (make-string ind ?\s))
-			 (arg-str (if (stringp args) (format " %s" args) ""))
-			 (contents
-			  (org-escape-code-in-string
-			   (org-export--prepare-file-contents file lines))))
-		     (format "%s#+BEGIN_%s%s\n%s%s#+END_%s\n"
-			     ind-str block arg-str contents ind-str block))))
-		 ((stringp block)
-		  (insert
-		   (let ((ind-str (make-string ind ?\s))
-			 (contents
-			  (org-export--prepare-file-contents file lines)))
-		     (format "%s#+BEGIN_%s\n%s%s#+END_%s\n"
-			     ind-str block contents ind-str block))))
+		 ((not file) nil)
+		 ((not (file-readable-p file))
+		  (error "Cannot include file %s" file))
+		 ;; Check if files has already been parsed.  Look after
+		 ;; inclusion lines too, as different parts of the same
+		 ;; file can be included too.
+		 ((member (list file lines) included)
+		  (error "Recursive file inclusion: %s" file))
 		 (t
-		  (insert
-		   (with-temp-buffer
-		     (let ((org-inhibit-startup t)
-			   (lines
-			    (if location
-				(org-export--inclusion-absolute-lines
-				 file location only-contents lines)
-			      lines)))
-		       (org-mode)
-		       (insert
-			(org-export--prepare-file-contents
-			 file lines ind minlevel
-			 (or (gethash file file-prefix)
-			     (puthash file
-				      (cl-incf current-prefix)
-				      file-prefix))
-			 footnotes
-			 includer-file)))
-		     (org-export-expand-include-keyword
-		      (cons (list file lines) included)
-		      (file-name-directory file)
-		      footnotes)
-		     (buffer-string)))))
-		;; Expand footnotes after all files have been
-		;; included.  Footnotes are stored at end of buffer.
-		(unless included
-		  (org-with-wide-buffer
-		   (goto-char (point-max))
-		   (maphash (lambda (k v)
-			      (insert (format "\n[fn:%s] %s\n" k v)))
-			    footnotes))))))))))))
+		  (cond
+		   ((eq env 'literal)
+		    (insert
+		     (let ((ind-str (make-string ind ?\s))
+			   (arg-str (if (stringp args) (format " %s" args) ""))
+			   (contents
+			    (org-escape-code-in-string
+			     (org-export--prepare-file-contents file lines))))
+		       (format "%s#+BEGIN_%s%s\n%s%s#+END_%s\n"
+			       ind-str block arg-str contents ind-str block))))
+		   ((stringp block)
+		    (insert
+		     (let ((ind-str (make-string ind ?\s))
+			   (contents
+			    (org-export--prepare-file-contents file lines)))
+		       (format "%s#+BEGIN_%s\n%s%s#+END_%s\n"
+			       ind-str block contents ind-str block))))
+		   (t
+		    (insert
+		     (with-temp-buffer
+		       (let ((org-inhibit-startup t)
+			     (lines
+			      (if location
+				  (org-export--inclusion-absolute-lines
+				   file location only-contents lines)
+				lines)))
+			 (org-mode)
+			 (insert
+			  (org-export--prepare-file-contents
+			   file lines ind minlevel
+			   (or (gethash file file-prefix)
+			       (puthash file
+					(cl-incf current-prefix)
+					file-prefix))
+			   footnotes
+			   includer-file)))
+		       (org-export-expand-include-keyword
+			(cons (list file lines) included)
+			(file-name-directory file)
+			footnotes)
+		       (buffer-string)))))
+		  ;; Expand footnotes after all files have been
+		  ;; included.  Footnotes are stored at end of buffer.
+		  (unless included
+		    (org-with-wide-buffer
+		     (goto-char (point-max))
+		     (maphash (lambda (k v)
+				(insert (format "\n[fn:%s] %s\n" k v)))
+			      footnotes)))))))))))))
 
 (defun org-export--inclusion-absolute-lines (file location only-contents lines)
   "Resolve absolute lines for an included file with file-link.
-- 
2.14.3


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #3: Correctly-convert-encoding-of-included-files --]
[-- Type: text/x-patch, Size: 1071 bytes --]

From fbd854f566d47729f7dcc0f304b537890a6eec0c Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Pierre=20T=C3=A9choueyres?= <pierre.techoueyres@free.fr>
Date: Sat, 21 Apr 2018 00:19:10 +0200
Subject: [PATCH] Correctly convert encoding of included files

* lisp/ox.el (org-export--prepare-file-contents): convert temporary
  buffer encoding to org buffer's encoding.
---
 lisp/ox.el | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/lisp/ox.el b/lisp/ox.el
index 5a83ae01d..a41d4d8d0 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -3481,7 +3481,10 @@ the included document.
 Optional argument INCLUDER is the file name where the inclusion
 is to happen."
   (with-temp-buffer
-    (insert-file-contents file)
+    (let ((org-buffer-coding-system buffer-file-coding-system))
+      (insert-file-contents file)
+      (unless (eq org-buffer-coding-system buffer-file-coding-system)
+	(set-buffer-file-coding-system org-buffer-coding-system)))
     (when lines
       (let* ((lines (split-string lines "-"))
 	     (lbeg (string-to-number (car lines)))
-- 
2.14.3

Re: [PATCH] Add new keyword :coding for #+include directive

[Nicolas Goaziou]

Hello,

pierre.techoueyres@free.fr (Pierre Téchoueyres) writes:

> I did my homework : found a fix for a) and rebased previous patch on
> master.

Thank you. Some comments follow.

> * lisp/ox.el (org-export-expand-include-keyword): Add new keyword
>   `:coding' for specify the file encoding whith the `#+include:'
>   directive.
>
> This allow to use somting like :
>   #+include: "./myfile" :coding cp850-dos
> when your org-file is encoded in utf-8 for example.

Nitpick: Org file

>  ** New features
> +*** New keyword for ~#+include:~ directive
> +Add ~:coding "codign-system"~ keyword to allow include of files from

=~:coding coding-system=

> +different codign system than the main org-file.

Org file.

> +For example:
> +#+begin_example
> +,#+INCLUDE: "myfile.cmd" src cmd :coding cp850-dos
> +#+end_example

Note that we are freezing Org 9.2 right now. This will go into master
once 9.2 is out.

> +		   (coding
> +		    (intern (or (and (string-match
> +				      ":coding +\\<\\([a-z0-9\\-]+\\)\\>" value)
> +				     (prog1 (match-string 1 value)
> +				       (setq value (replace-match "" nil nil value))))
> +				(symbol-name coding-system-for-read))))

                   (coding
		    (or (and (string-match ":coding +\\([a-z0-9\\-]+\\)" value)
                             (prog1 (intern (match-string 1 value))
                               (setq value (replace-match "" nil nil value))))
                        coding-system-for-read))

In Lisp, symbol constituents are all words plus $&*+-_<>

We might want to update the regexp accordingly.

> +Subject: [PATCH] Correctly convert encoding of included files
> +
> +* lisp/ox.el (org-export--prepare-file-contents): convert temporary
> +  buffer encoding to org buffer's encoding.

Org buffer's encoding.

Could you send an updated patch?

Regards,

-- 
Nicolas Goaziou                                                0x80A93738

Re: [PATCH] Add new keyword :coding for #+include directive

[Pierre Téchoueyres]

[-- Attachment #1: Type: text/plain, Size: 206 bytes --]

Hello Nicolas,
I think I've corrected all points. You'll find new versions attached.

Would you mind consider to include the patch for the detection of
encoding with the #+include keyword in 9.2 release ?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: fix-org-export--prepare-file-contents --]
[-- Type: text/x-patch, Size: 1099 bytes --]

From e4d6cf8f9959781c682738a4b9e5ea8ae5747b6b Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Pierre=20T=C3=A9choueyres?= <pierre.techoueyres@free.fr>
Date: Sat, 21 Apr 2018 00:19:10 +0200
Subject: [PATCH] Manage correctly the encoding of files with #+include:
 directive

* lisp/ox.el (org-export--prepare-file-contents): convert the encoding
  of temporary buffer to Org buffer's encoding.
---
 lisp/ox.el | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/lisp/ox.el b/lisp/ox.el
index 5a83ae01d..a41d4d8d0 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -3481,7 +3481,10 @@ the included document.
 Optional argument INCLUDER is the file name where the inclusion
 is to happen."
   (with-temp-buffer
-    (insert-file-contents file)
+    (let ((org-buffer-coding-system buffer-file-coding-system))
+      (insert-file-contents file)
+      (unless (eq org-buffer-coding-system buffer-file-coding-system)
+	(set-buffer-file-coding-system org-buffer-coding-system)))
     (when lines
       (let* ((lines (split-string lines "-"))
 	     (lbeg (string-to-number (car lines)))
-- 
2.14.3


[-- Attachment #3: include-new-directive --]
[-- Type: text/x-patch, Size: 6469 bytes --]

From 7a67b6633461ff4d3bce69d5e781e36f54bf8b9d Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Pierre=20T=C3=A9choueyres?= <pierre.techoueyres@free.fr>
Date: Sat, 21 Apr 2018 00:31:10 +0200
Subject: [PATCH] Add new keyword :coding for #+include directive

* lisp/ox.el (org-export-expand-include-keyword): Add new keyword
  `:coding' for specify the file encoding whith the `#+include:'
  directive.

This allow to use somting like :
  #+include: "./myfile" :coding cp850-dos
when your Org file is encoded in utf-8 for example.
---
 etc/ORG-NEWS |   7 ++++
 lisp/ox.el   | 121 +++++++++++++++++++++++++++++++----------------------------
 2 files changed, 71 insertions(+), 57 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index abebe08fe..99350ee84 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -121,6 +121,13 @@ now sort according to the locale’s collation rules instead of by
 code-point.
 
 ** New features
+*** New keyword for ~#+include:~ directive
+Add =:coding codign-system= keyword to allow include of files from
+different codign system than the main Org file.
+For example:
+#+begin_example
+,#+INCLUDE: "myfile.cmd" src cmd :coding cp850-dos
+#+end_example
 *** Add ~:results link~ support for Babel
 
 With this output format, create a link to the file specified in
diff --git a/lisp/ox.el b/lisp/ox.el
index 5a83ae01d..d4241720c 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -3298,6 +3298,12 @@ storing and resolving footnotes.  It is created automatically."
 				  (org-unbracket-string "\"" "\"" matched)
 				  dir)))
 			   (setq value (replace-match "" nil nil value)))))
+		   (coding
+		    (intern (or (and (string-match
+				      ":coding[[:space:]]+\\_<\\(\\(?:\\sw\\|\\$\\|&\\|\\*\\|\\+\\|-\\|_\\|<\\|>\\)+\\)\\_>" value)
+				     (prog1 (match-string 1 value)
+				       (setq value (replace-match "" nil nil value))))
+				(symbol-name coding-system-for-read))))
 		   (only-contents
 		    (and (string-match ":only-contents *\\([^: \r\t\n]\\S-*\\)?"
 				       value)
@@ -3331,65 +3337,66 @@ storing and resolving footnotes.  It is created automatically."
 			       (match-string 1 value))))
 	      ;; Remove keyword.
 	      (delete-region (point) (line-beginning-position 2))
-	      (cond
-	       ((not file) nil)
-	       ((not (file-readable-p file))
-		(error "Cannot include file %s" file))
-	       ;; Check if files has already been parsed.  Look after
-	       ;; inclusion lines too, as different parts of the same
-	       ;; file can be included too.
-	       ((member (list file lines) included)
-		(error "Recursive file inclusion: %s" file))
-	       (t
+	      (let ((coding-system-for-read coding))
 		(cond
-		 ((eq env 'literal)
-		  (insert
-		   (let ((ind-str (make-string ind ?\s))
-			 (arg-str (if (stringp args) (format " %s" args) ""))
-			 (contents
-			  (org-escape-code-in-string
-			   (org-export--prepare-file-contents file lines))))
-		     (format "%s#+BEGIN_%s%s\n%s%s#+END_%s\n"
-			     ind-str block arg-str contents ind-str block))))
-		 ((stringp block)
-		  (insert
-		   (let ((ind-str (make-string ind ?\s))
-			 (contents
-			  (org-export--prepare-file-contents file lines)))
-		     (format "%s#+BEGIN_%s\n%s%s#+END_%s\n"
-			     ind-str block contents ind-str block))))
+		 ((not file) nil)
+		 ((not (file-readable-p file))
+		  (error "Cannot include file %s" file))
+		 ;; Check if files has already been parsed.  Look after
+		 ;; inclusion lines too, as different parts of the same
+		 ;; file can be included too.
+		 ((member (list file lines) included)
+		  (error "Recursive file inclusion: %s" file))
 		 (t
-		  (insert
-		   (with-temp-buffer
-		     (let ((org-inhibit-startup t)
-			   (lines
-			    (if location
-				(org-export--inclusion-absolute-lines
-				 file location only-contents lines)
-			      lines)))
-		       (org-mode)
-		       (insert
-			(org-export--prepare-file-contents
-			 file lines ind minlevel
-			 (or (gethash file file-prefix)
-			     (puthash file
-				      (cl-incf current-prefix)
-				      file-prefix))
-			 footnotes
-			 includer-file)))
-		     (org-export-expand-include-keyword
-		      (cons (list file lines) included)
-		      (file-name-directory file)
-		      footnotes)
-		     (buffer-string)))))
-		;; Expand footnotes after all files have been
-		;; included.  Footnotes are stored at end of buffer.
-		(unless included
-		  (org-with-wide-buffer
-		   (goto-char (point-max))
-		   (maphash (lambda (k v)
-			      (insert (format "\n[fn:%s] %s\n" k v)))
-			    footnotes))))))))))))
+		  (cond
+		   ((eq env 'literal)
+		    (insert
+		     (let ((ind-str (make-string ind ?\s))
+			   (arg-str (if (stringp args) (format " %s" args) ""))
+			   (contents
+			    (org-escape-code-in-string
+			     (org-export--prepare-file-contents file lines))))
+		       (format "%s#+BEGIN_%s%s\n%s%s#+END_%s\n"
+			       ind-str block arg-str contents ind-str block))))
+		   ((stringp block)
+		    (insert
+		     (let ((ind-str (make-string ind ?\s))
+			   (contents
+			    (org-export--prepare-file-contents file lines)))
+		       (format "%s#+BEGIN_%s\n%s%s#+END_%s\n"
+			       ind-str block contents ind-str block))))
+		   (t
+		    (insert
+		     (with-temp-buffer
+		       (let ((org-inhibit-startup t)
+			     (lines
+			      (if location
+				  (org-export--inclusion-absolute-lines
+				   file location only-contents lines)
+				lines)))
+			 (org-mode)
+			 (insert
+			  (org-export--prepare-file-contents
+			   file lines ind minlevel
+			   (or (gethash file file-prefix)
+			       (puthash file
+					(cl-incf current-prefix)
+					file-prefix))
+			   footnotes
+			   includer-file)))
+		       (org-export-expand-include-keyword
+			(cons (list file lines) included)
+			(file-name-directory file)
+			footnotes)
+		       (buffer-string)))))
+		  ;; Expand footnotes after all files have been
+		  ;; included.  Footnotes are stored at end of buffer.
+		  (unless included
+		    (org-with-wide-buffer
+		     (goto-char (point-max))
+		     (maphash (lambda (k v)
+				(insert (format "\n[fn:%s] %s\n" k v)))
+			      footnotes)))))))))))))
 
 (defun org-export--inclusion-absolute-lines (file location only-contents lines)
   "Resolve absolute lines for an included file with file-link.
-- 
2.14.3


[-- Attachment #4: Type: text/plain, Size: 10 bytes --]


Regards,

Re: [PATCH] Add new keyword :coding for #+include directive

[Nicolas Goaziou]

Hello,

pierre.techoueyres@free.fr (Pierre Téchoueyres) writes:

> I think I've corrected all points. You'll find new versions attached.

Thank you.

> Would you mind consider to include the patch for the detection of
> encoding with the #+include keyword in 9.2 release ?

This patch is still missing some small parts for proper integration,
namely documentation, and, if possible, a couple of tests. Besides, 9.2
branch is supposedly frozen.

Granted, it doesn't seem too harmful, but is there any strong reason to
integrate it in Org 9.2 (assuming documentation is ready)?

> +		   (coding
> +		    (intern (or (and (string-match
> +				      ":coding[[:space:]]+\\_<\\(\\(?:\\sw\\|\\$\\|&\\|\\*\\|\\+\\|-\\|_\\|<\\|>\\)+\\)\\_>" value)
> +				     (prog1 (match-string 1 value)
> +				       (setq value (replace-match "" nil nil value))))
> +				(symbol-name coding-system-for-read))))

I suggested a refactoring that you didn't integrate: it seems wasteful
to call `intern' on the return value of `symbol-name'.

Besides, my suggestion about the regexp was wrong. We shouldn't make the
syntax foolproof. I think

  ":coding +\\(\\S-+\\)"

is enough actually. Sorry about sending you in the wrong track.

Regards,

-- 
Nicolas Goaziou

Re: [PATCH] Add new keyword :coding for #+include directive

[Pierre Téchoueyres]

[-- Attachment #1: Type: text/plain, Size: 1699 bytes --]


Hello,
Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Hello,
>
> pierre.techoueyres@free.fr (Pierre Téchoueyres) writes:
>
>> I think I've corrected all points. You'll find new versions attached.
>
> Thank you.
>
>> Would you mind consider to include the patch for the detection of
>> encoding with the #+include keyword in 9.2 release ?
>
> This patch is still missing some small parts for proper integration,
> namely documentation, and, if possible, a couple of tests. Besides, 9.2
> branch is supposedly frozen.

I argree for the documentation and tests (but I have to admit I don't
know how to add them).

> Granted, it doesn't seem too harmful, but is there any strong reason to
> integrate it in Org 9.2 (assuming documentation is ready)?


I think I wasn't clear enough : I had hope you will only include the
part which correct the decoding of include keyword, not the whole two
patchs. I think the former is simply a bug fixes. 


>
>> +		   (coding
>> +		    (intern (or (and (string-match
>> +
>> ":coding[[:space:]]+\\_<\\(\\(?:\\sw\\|\\$\\|&\\|\\*\\|\\+\\|-\\|_\\|<\\|>\\)+\\)\\_>"
>> value)
>> +				     (prog1 (match-string 1 value)
>> +				       (setq value (replace-match "" nil nil value))))
>> +				(symbol-name coding-system-for-read))))
>
> I suggested a refactoring that you didn't integrate: it seems wasteful
> to call `intern' on the return value of `symbol-name'.
>
> Besides, my suggestion about the regexp was wrong. We shouldn't make the
> syntax foolproof. I think
>
>   ":coding +\\(\\S-+\\)"
>
> is enough actually. Sorry about sending you in the wrong track.
>
> Regards,

Here is a new amended patch.


[-- Attachment #2: include-new-directive --]
[-- Type: text/x-patch, Size: 6395 bytes --]

From bb9d154ba06d2fd3608bf6877c0b765a932b38e7 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Pierre=20T=C3=A9choueyres?= <pierre.techoueyres@free.fr>
Date: Sat, 21 Apr 2018 00:31:10 +0200
Subject: [PATCH] Add new keyword :coding for #+include directive

* lisp/ox.el (org-export-expand-include-keyword): Add new keyword
  `:coding' for specify the file encoding whith the `#+include:'
  directive.

This allow to use somting like :
  #+include: "./myfile" :coding cp850-dos
when your Org file is encoded in utf-8 for example.
---
 etc/ORG-NEWS |   7 ++++
 lisp/ox.el   | 121 +++++++++++++++++++++++++++++++----------------------------
 2 files changed, 71 insertions(+), 57 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index abebe08fe..99350ee84 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -121,6 +121,13 @@ now sort according to the locale’s collation rules instead of by
 code-point.
 
 ** New features
+*** New keyword for ~#+include:~ directive
+Add =:coding codign-system= keyword to allow include of files from
+different codign system than the main Org file.
+For example:
+#+begin_example
+,#+INCLUDE: "myfile.cmd" src cmd :coding cp850-dos
+#+end_example
 *** Add ~:results link~ support for Babel
 
 With this output format, create a link to the file specified in
diff --git a/lisp/ox.el b/lisp/ox.el
index 5a83ae01d..dd454ef42 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -3298,6 +3298,12 @@ storing and resolving footnotes.  It is created automatically."
 				  (org-unbracket-string "\"" "\"" matched)
 				  dir)))
 			   (setq value (replace-match "" nil nil value)))))
+		   (coding
+		    (or (and (string-match
+			      ":coding +\\_<\\(\\S-+\\)\\_>" value)
+			     (prog1 (intern (match-string 1 value))
+			       (setq value (replace-match "" nil nil value))))
+			coding-system-for-read))
 		   (only-contents
 		    (and (string-match ":only-contents *\\([^: \r\t\n]\\S-*\\)?"
 				       value)
@@ -3331,65 +3337,66 @@ storing and resolving footnotes.  It is created automatically."
 			       (match-string 1 value))))
 	      ;; Remove keyword.
 	      (delete-region (point) (line-beginning-position 2))
-	      (cond
-	       ((not file) nil)
-	       ((not (file-readable-p file))
-		(error "Cannot include file %s" file))
-	       ;; Check if files has already been parsed.  Look after
-	       ;; inclusion lines too, as different parts of the same
-	       ;; file can be included too.
-	       ((member (list file lines) included)
-		(error "Recursive file inclusion: %s" file))
-	       (t
+	      (let ((coding-system-for-read coding))
 		(cond
-		 ((eq env 'literal)
-		  (insert
-		   (let ((ind-str (make-string ind ?\s))
-			 (arg-str (if (stringp args) (format " %s" args) ""))
-			 (contents
-			  (org-escape-code-in-string
-			   (org-export--prepare-file-contents file lines))))
-		     (format "%s#+BEGIN_%s%s\n%s%s#+END_%s\n"
-			     ind-str block arg-str contents ind-str block))))
-		 ((stringp block)
-		  (insert
-		   (let ((ind-str (make-string ind ?\s))
-			 (contents
-			  (org-export--prepare-file-contents file lines)))
-		     (format "%s#+BEGIN_%s\n%s%s#+END_%s\n"
-			     ind-str block contents ind-str block))))
+		 ((not file) nil)
+		 ((not (file-readable-p file))
+		  (error "Cannot include file %s" file))
+		 ;; Check if files has already been parsed.  Look after
+		 ;; inclusion lines too, as different parts of the same
+		 ;; file can be included too.
+		 ((member (list file lines) included)
+		  (error "Recursive file inclusion: %s" file))
 		 (t
-		  (insert
-		   (with-temp-buffer
-		     (let ((org-inhibit-startup t)
-			   (lines
-			    (if location
-				(org-export--inclusion-absolute-lines
-				 file location only-contents lines)
-			      lines)))
-		       (org-mode)
-		       (insert
-			(org-export--prepare-file-contents
-			 file lines ind minlevel
-			 (or (gethash file file-prefix)
-			     (puthash file
-				      (cl-incf current-prefix)
-				      file-prefix))
-			 footnotes
-			 includer-file)))
-		     (org-export-expand-include-keyword
-		      (cons (list file lines) included)
-		      (file-name-directory file)
-		      footnotes)
-		     (buffer-string)))))
-		;; Expand footnotes after all files have been
-		;; included.  Footnotes are stored at end of buffer.
-		(unless included
-		  (org-with-wide-buffer
-		   (goto-char (point-max))
-		   (maphash (lambda (k v)
-			      (insert (format "\n[fn:%s] %s\n" k v)))
-			    footnotes))))))))))))
+		  (cond
+		   ((eq env 'literal)
+		    (insert
+		     (let ((ind-str (make-string ind ?\s))
+			   (arg-str (if (stringp args) (format " %s" args) ""))
+			   (contents
+			    (org-escape-code-in-string
+			     (org-export--prepare-file-contents file lines))))
+		       (format "%s#+BEGIN_%s%s\n%s%s#+END_%s\n"
+			       ind-str block arg-str contents ind-str block))))
+		   ((stringp block)
+		    (insert
+		     (let ((ind-str (make-string ind ?\s))
+			   (contents
+			    (org-export--prepare-file-contents file lines)))
+		       (format "%s#+BEGIN_%s\n%s%s#+END_%s\n"
+			       ind-str block contents ind-str block))))
+		   (t
+		    (insert
+		     (with-temp-buffer
+		       (let ((org-inhibit-startup t)
+			     (lines
+			      (if location
+				  (org-export--inclusion-absolute-lines
+				   file location only-contents lines)
+				lines)))
+			 (org-mode)
+			 (insert
+			  (org-export--prepare-file-contents
+			   file lines ind minlevel
+			   (or (gethash file file-prefix)
+			       (puthash file
+					(cl-incf current-prefix)
+					file-prefix))
+			   footnotes
+			   includer-file)))
+		       (org-export-expand-include-keyword
+			(cons (list file lines) included)
+			(file-name-directory file)
+			footnotes)
+		       (buffer-string)))))
+		  ;; Expand footnotes after all files have been
+		  ;; included.  Footnotes are stored at end of buffer.
+		  (unless included
+		    (org-with-wide-buffer
+		     (goto-char (point-max))
+		     (maphash (lambda (k v)
+				(insert (format "\n[fn:%s] %s\n" k v)))
+			      footnotes)))))))))))))
 
 (defun org-export--inclusion-absolute-lines (file location only-contents lines)
   "Resolve absolute lines for an included file with file-link.
-- 
2.14.3


[-- Attachment #3: Type: text/plain, Size: 42 bytes --]



Regards,

-- 
Pierre Téchoueyres

Re: [PATCH] Add new keyword :coding for #+include directive

[Pierre Téchoueyres]

Hello Nicolas,
Did you have time to review the patches ?


pierre.techoueyres@free.fr (Pierre Téchoueyres) writes:

> Hello,
> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>
>> Hello,
>>
>> pierre.techoueyres@free.fr (Pierre Téchoueyres) writes:
>>
>>> I think I've corrected all points. You'll find new versions attached.
>>
>> Thank you.
>>
>>> Would you mind consider to include the patch for the detection of
>>> encoding with the #+include keyword in 9.2 release ?
>>
>> This patch is still missing some small parts for proper integration,
>> namely documentation, and, if possible, a couple of tests. Besides, 9.2
>> branch is supposedly frozen.
>
> I argree for the documentation and tests (but I have to admit I don't
> know how to add them).
>
>> Granted, it doesn't seem too harmful, but is there any strong reason to
>> integrate it in Org 9.2 (assuming documentation is ready)?
>
>
> I think I wasn't clear enough : I had hope you will only include the
> part which correct the decoding of include keyword, not the whole two
> patchs. I think the former is simply a bug fixes. 
>
>
>>
>>> +		   (coding
>>> +		    (intern (or (and (string-match
>>> +
>>> ":coding[[:space:]]+\\_<\\(\\(?:\\sw\\|\\$\\|&\\|\\*\\|\\+\\|-\\|_\\|<\\|>\\)+\\)\\_>"
>>> value)
>>> +				     (prog1 (match-string 1 value)
>>> +				       (setq value (replace-match "" nil nil value))))
>>> +				(symbol-name coding-system-for-read))))
>>
>> I suggested a refactoring that you didn't integrate: it seems wasteful
>> to call `intern' on the return value of `symbol-name'.
>>
>> Besides, my suggestion about the regexp was wrong. We shouldn't make the
>> syntax foolproof. I think
>>
>>   ":coding +\\(\\S-+\\)"
>>
>> is enough actually. Sorry about sending you in the wrong track.
>>
>> Regards,
>
> Here is a new amended patch.
>
>
>
>
> Regards,

Re: [PATCH] Add new keyword :coding for #+include directive

[Nicolas Goaziou]

Hello,

pierre.techoueyres@free.fr (Pierre Téchoueyres) writes:

> Hello Nicolas,
> Did you have time to review the patches ?

Sorry for the delay, I have been sidetracked.

I admit I don't fully understand your bugfix patch, i.e., "[PATCH]
Correctly convert encoding of included files".

For the record, here is the change:

   (with-temp-buffer
-    (insert-file-contents file)
+    (let ((org-buffer-coding-system buffer-file-coding-system))
+      (insert-file-contents file)
+      (unless (eq org-buffer-coding-system buffer-file-coding-system)
+	(set-buffer-file-coding-system org-buffer-coding-system)))


You pretend `org-buffer-coding-system' is storing coding-system from the
Org buffer, but the let-binding happens from within `with-temp-buffer'.
So the coding system comes from the temporary buffer instead. 

Also, `insert-file-contents' is not supposed to change coding system, or
is it? So when would the `unless' be triggered?

Could you explain a bit the issue you are fixing here?

Thank you.

Regards,

-- 
Nicolas Goaziou                                                0x80A93738

Re: [PATCH] Add new keyword :coding for #+include directive

[Pierre Téchoueyres]

[-- Attachment #1: Type: text/plain, Size: 2269 bytes --]

Hello,
And sorry for the delay.


Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Hello,
>
> pierre.techoueyres@free.fr (Pierre Téchoueyres) writes:
>
>> Hello Nicolas,
>> Did you have time to review the patches ?
>
> Sorry for the delay, I have been sidetracked.
>
> I admit I don't fully understand your bugfix patch, i.e., "[PATCH]
> Correctly convert encoding of included files".
>
> For the record, here is the change:
>
>    (with-temp-buffer
> -    (insert-file-contents file)
> +    (let ((org-buffer-coding-system buffer-file-coding-system))
> +      (insert-file-contents file)
> +      (unless (eq org-buffer-coding-system buffer-file-coding-system)
> +	(set-buffer-file-coding-system org-buffer-coding-system)))
>
>
> You pretend `org-buffer-coding-system' is storing coding-system from the
> Org buffer, but the let-binding happens from within `with-temp-buffer'.
> So the coding system comes from the temporary buffer instead. 

Yes, that's true, but the coding system of the temporary buffer is
inherited of the one of the org file (At least it's what I've found in
my tests). But my be should I add a comment to explain that.

>
> Also, `insert-file-contents' is not supposed to change coding system, or
> is it? So when would the `unless' be triggered?
>

For the tests I've made and the examples I had sent, it seem to change
the coding system.

> Could you explain a bit the issue you are fixing here?
>
> Thank you.
>
> Regards,

I hoped the example Ive had sent was clear enough.

I'm using org on an Windows machine where the default encoding seem to
be iso-latin-1-dos.
But I edit my org files in utf-8 encoding and of course I want to
include some files as source files for reference. Thoses files are
sometime encoded wit an different coding system (espetially cmd files
with comments in french : cp850).

I use principally those coding  ystems :
- cmd : cp850
- org : utf-8 or iso8859-15
- sql : window1252-dos

But when I include thoses files their content isn't correctly inserted,
especially the accents.
Hope I'me clearer now.


I reattach the examples, but note that  the cmd.txt or sh.txt extensions
are only there to avoid my mail to be wiped out.

Regards,

[-- Attachment #2: test.org --]
[-- Type: text/x-org, Size: 383 bytes --]

# -*- coding: utf-8 -*-
#+title: test with different encodings.

* Pierre Téchoueyres
  bad
  #+include: "./file.cmd.txt" src bat
  or
  #+include: "./file.sh.txt" src sh

  and good ? 
  #+include: "./file.cmd.txt" src bat :coding iso-8859-15-dos
  or
  #+include: "./file.sh.txt" src sh :coding iso-8859-15-unix
 

# Local Variables:
# coding: utf-8-unix
# End:

[-- Attachment #3: file.cmd --]
[-- Type: text/plain, Size: 120 bytes --]

@echo off

echo Hello Pierre TÚchoueyres ñ

rem Local Variables:
rem coding: cp850-dos
rem mode: bat
rem End:

[-- Attachment #4: file.sh --]
[-- Type: text/plain, Size: 140 bytes --]

#!/bin/sh # -*- coding: iso-8859-15-unix -*-

echo "Hello Pierre Téchoueyres"

# Local Variables:
# coding: iso-8859-15-unix
# End:

Re: [PATCH] Add new keyword :coding for #+include directive

[Nicolas Goaziou]

Hello,

pierre.techoueyres@free.fr (Pierre Téchoueyres) writes:

> I reattach the examples, but note that  the cmd.txt or sh.txt extensions
> are only there to avoid my mail to be wiped out.

I must be missing something. When I install both patches, and I export
your example to, e.g., UTF-8, I get the following document, where coding
systems are not resolved.

--8<---------------cut here---------------start------------->8---
1 Pierre Téchoueyres
════════════════════

  bad
  ┌────
  │ @echo off
  │ 
  │ echo Hello Pierre T├Üchoueyres ├▒
  │ 
  │ rem Local Variables:
  │ rem coding: cp850-dos
  │ rem mode: bat
  │ rem End:
  └────
  or
  ┌────
  │ #!/bin/sh # -*- coding: iso-8859-15-unix -*-
  │ 
  │ echo "Hello Pierre Téchoueyres"
  │ 
  │ # Local Variables:
  │ # coding: iso-8859-15-unix
  │ # End:
  └────

  and good ?
  ┌────
  │ @echo off
  │ 
  │ echo Hello Pierre TÚchoueyres ñ
  │ 
  │ rem Local Variables:
  │ rem coding: cp850-dos
  │ rem mode: bat
  │ rem End:
  └────
  or
  ┌────
  │ #!/bin/sh # -*- coding: iso-8859-15-unix -*-
  │ 
  │ echo "Hello Pierre Téchoueyres"
  │ 
  │ # Local Variables:
  │ # coding: iso-8859-15-unix
  │ # End:
  └────
--8<---------------cut here---------------end--------------->8---

Regards,

-- 
Nicolas Goaziou                                                0x80A93738

Re: [PATCH] Add new keyword :coding for #+include directive

[Pierre Téchoueyres]

[-- Attachment #1: Type: text/plain, Size: 656 bytes --]

Hello Nicolas,
it's really strange.
I've checked it another time some minutes ago and, on my machine, this
works.
I've used the following command to do the test :
for emacs 25.3
#+begin_src sh
emacs -Q -L lisp -L ~/.emacs.d/elpa-25/htmlize-20180412.1244 -l ~/.emacs.d/elpa-25/htmlize-20180412.1244/htmlize-autoloads.el testing/examples/test.org
#+end_src

for Emacs 26.1 (934bb475b9)
#+begin_src sh
emacs -Q -L lisp -L ~/.emacs.d/elpa-25/htmlize-20180412.1244 -l ~/.emacs.d/elpa-25/htmlize-20180412.1244/htmlize-autoloads.el testing/examples/test.org
#+end_src

And I still obtain the expected results : see the text attached file
(encoded in utf-8-unix).

[-- Attachment #2: test.txt --]
[-- Type: text/plain, Size: 1245 bytes --]

		                                    
		     TEST WITH DIFFERENT ENCODINGS.

			   Pierre TÚchoueyres
		                                    


Table of Contents
                 

1. Pierre TÚchoueyres





1 Pierre TÚchoueyres
                    

  bad
       
    @echo off
    
    echo Hello Pierre T┌choueyres ±
    
    rem Local Variables:
    rem coding: cp850-dos
    rem mode: bat
    rem End:
       
  or (only without
  0001-Manage-correctly-the-encoding-of-files-with-include-.patch
  applied)
       
    #!/bin/sh # -*- coding: iso-8859-15-unix -*-
    
    echo "Hello Pierre TÚchoueyres"
    
    # Local Variables:
    # coding: iso-8859-15-unix
    # End:
       

  and good ?
       
    @echo off
    
    echo Hello Pierre TÚchoueyres  
    
    rem Local Variables:
    rem coding: cp850-dos
    rem mode: bat
    rem End:
       
  or
       
    #!/bin/sh # -*- coding: iso-8859-15-unix -*-
    
    echo "Hello Pierre TÚchoueyres"
    
    # Local Variables:
    # coding: iso-8859-15-unix
    # End:
       

       
    Org mode version 9.1.13 (release_9.1.13-763-gcd25c7 @ /home/ptechoueyres/Travail/VCS/emacs-org-mode/lisp/)
       

[-- Attachment #3: Type: text/plain, Size: 1870 bytes --]


So I don't really understand what's appening.

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Hello,
>
> pierre.techoueyres@free.fr (Pierre Téchoueyres) writes:
>
>> I reattach the examples, but note that  the cmd.txt or sh.txt extensions
>> are only there to avoid my mail to be wiped out.
>
> I must be missing something. When I install both patches, and I export
> your example to, e.g., UTF-8, I get the following document, where coding
> systems are not resolved.
>
> 1 Pierre Téchoueyres
> ════════════════════
>
>   bad
>   ┌────
>   │ @echo off
>   │ 
>   │ echo Hello Pierre T├Üchoueyres ├▒
>   │ 
>   │ rem Local Variables:
>   │ rem coding: cp850-dos
>   │ rem mode: bat
>   │ rem End:
>   └────
>   or
>   ┌────
>   │ #!/bin/sh # -*- coding: iso-8859-15-unix -*-
>   │ 
>   │ echo "Hello Pierre Téchoueyres"
>   │ 
>   │ # Local Variables:
>   │ # coding: iso-8859-15-unix
>   │ # End:
>   └────
>
>   and good ?
>   ┌────
>   │ @echo off
>   │ 
>   │ echo Hello Pierre TÚchoueyres ñ
>   │ 
>   │ rem Local Variables:
>   │ rem coding: cp850-dos
>   │ rem mode: bat
>   │ rem End:
>   └────
>   or
>   ┌────
>   │ #!/bin/sh # -*- coding: iso-8859-15-unix -*-
>   │ 
>   │ echo "Hello Pierre Téchoueyres"
>   │ 
>   │ # Local Variables:
>   │ # coding: iso-8859-15-unix
>   │ # End:
>   └────
>
> Regards,

-- 
| Pierre Téchoueyres -  Appt. 150 Bat. B
| Tel : 05 56 42 53 22  Res. Le FLORE              ___
|       09 51 40 74 43  32 rue Marcelin Berthelot  ,,,
| Port: 06 77 11 64 05  33200 CAUDERAN            (o o)
| mailto:Pierre.Techoueyres@free.fr         ---ooO-(_)-Ooo---

Re: [PATCH] Add new keyword :coding for #+include directive

[Nicolas Goaziou]

Hello,

pierre.techoueyres@free.fr (Pierre Téchoueyres) writes:

> I've used the following command to do the test :
> for emacs 25.3
>
> #+begin_src sh
> emacs -Q -L lisp -L ~/.emacs.d/elpa-25/htmlize-20180412.1244 -l ~/.emacs.d/elpa-25/htmlize-20180412.1244/htmlize-autoloads.el testing/examples/test.org
> #+end_src

With the command above, you're not exporting the file using "ox.el".
What are you testing? Could you use a command that exports the file
(preferably using ASCII (utf-8) export back-end)?

> 		     TEST WITH DIFFERENT ENCODINGS.
>
> 			   Pierre TÚchoueyres
                                  ^^^
                                  see

Regards,

-- 
Nicolas Goaziou

Re: [PATCH] Add new keyword :coding for #+include directive

[Pierre Téchoueyres]

Hello Nicolas,
I'm sorry for the delay, this problem drove me crazy for a few days.
So I started again from the beginning and, of course, I am now unable to
replicate my original test case.
So, I think you could delete the
0001-Manage-the-encoding-of-files-with-include-.patch patch.
I am really sorry for the inconvenience.

I hope we can go ahead with the other patch to add a ":coding" keyword
to the include directive.

To resume the goal:
The include keyword works well with encodings like cp850 if, and only
if, there are some indications of the encoding in the file (as local
variables for example). But sometimes, there is no such information and
you can not modify the file (for various reasons. ex: the file is
generated automatically).

Once again, I'm sorry for the disturbance.
Regards,

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Hello,
>
> pierre.techoueyres@free.fr (Pierre Téchoueyres) writes:
>
>> I've used the following command to do the test :
>> for emacs 25.3
>>
>> #+begin_src sh
>> emacs -Q -L lisp -L ~/.emacs.d/elpa-25/htmlize-20180412.1244 -l
>> ~/.emacs.d/elpa-25/htmlize-20180412.1244/htmlize-autoloads.el
>> testing/examples/test.org
>> #+end_src
>
> With the command above, you're not exporting the file using "ox.el".
> What are you testing? Could you use a command that exports the file
> (preferably using ASCII (utf-8) export back-end)?
>
>> 		     TEST WITH DIFFERENT ENCODINGS.
>>
>> 			   Pierre TÚchoueyres
>                                   ^^^
>                                   see
>
> Regards,

Re: [PATCH] Add new keyword :coding for #+include directive

[Nicolas Goaziou]

Hello,

pierre.techoueyres@free.fr (Pierre Téchoueyres) writes:

> Hello Nicolas,
> I'm sorry for the delay, this problem drove me crazy for a few days.
> So I started again from the beginning and, of course, I am now unable to
> replicate my original test case.
> So, I think you could delete the
> 0001-Manage-the-encoding-of-files-with-include-.patch patch.
> I am really sorry for the inconvenience.

No problem. It also drove me crazy.

> I hope we can go ahead with the other patch to add a ":coding" keyword
> to the include directive.
>
> To resume the goal:
> The include keyword works well with encodings like cp850 if, and only
> if, there are some indications of the encoding in the file (as local
> variables for example). But sometimes, there is no such information and
> you can not modify the file (for various reasons. ex: the file is
> generated automatically).

I applied the first patch in "next" branch, with a slight refactoring.

Thank you.

Regards,

-- 
Nicolas Goaziou                                                0x80A93738

Re: [PATCH] Add new keyword :coding for #+include directive

[Pierre Téchoueyres]

Hello nicolas,
Many thanks for your patience and for applying my patch.
I've seen you didn't apply the part which added an entry in ORG-NEWS
file. Was this deliberate or just an side effect of your refactoring ?
Or is this unneeded ? 
Again thanks for your patience.

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> ...
> I applied the first patch in "next" branch, with a slight refactoring.
>
> Thank you.
>
> Regards,

Re: [PATCH] Add new keyword :coding for #+include directive

[Nicolas Goaziou]

Hello,

pierre.techoueyres@free.fr (Pierre Téchoueyres) writes:

> I've seen you didn't apply the part which added an entry in ORG-NEWS
> file. Was this deliberate or just an side effect of your refactoring ?
> Or is this unneeded ? 
> Again thanks for your patience.

I had to remove that change or your patch wouldn't apply. I forgot to
apply it back separately. This is now done. Thank you.

Regards,

-- 
Nicolas Goaziou                                                0x80A93738