[Bug] Export Coding System

[Bug] Export Coding System

[Achim Gratz]

I've ran into this before, but I guess I wasn't able to clearly explain what the
issue was at the time... so let's try again.

I'm exporting an Org file in UTF-8 to LaTeX.  Unless I switch the coding system
in Emacs from "default" to "UTF-8" (which has the side effect that any new
buffer will have UTF-8 coding, which is usually not what I want), the LaTeX
buffer gets created with ISO8859-1/latin-1 encoding (which is dead wrong,
because several characters in the document are in fact not representable in that
encoding).  Changing the option "Org Export Latex Coding System" to utf-8
doesn't change how the LaTeX buffer gets created, but it will then helpfully ask
when its time to save the buffer if I want to save it as UTF-8 (no, I want it
created with UTF-8, not changed to a different encoding on save).  That salvages
a few characters, but the larger damage of \usepackage[latin1]{inputenc} doesn't
get changed of course.

The coding system of any export buffers should by default follow the Org buffer
they are created from and any changes to the coding system stipulated by export
configuration must be applied before the buffer coding system gets used to make
any decisions on how to export things.


Regards,
Achim.

Re: [Bug] Export Coding System

[Nicolas Goaziou]

Hello,

Achim Gratz <Stromeko@NexGo.DE> writes:

> I'm exporting an Org file in UTF-8 to LaTeX.  Unless I switch the coding system
> in Emacs from "default" to "UTF-8" (which has the side effect that any new
> buffer will have UTF-8 coding, which is usually not what I want), the LaTeX
> buffer gets created with ISO8859-1/latin-1 encoding (which is dead wrong,
> because several characters in the document are in fact not representable in that
> encoding).  Changing the option "Org Export Latex Coding System" to utf-8
> doesn't change how the LaTeX buffer gets created, but it will then helpfully ask
> when its time to save the buffer if I want to save it as UTF-8 (no, I want it
> created with UTF-8, not changed to a different encoding on save).  That salvages
> a few characters, but the larger damage of \usepackage[latin1]{inputenc} doesn't
> get changed of course.
>
> The coding system of any export buffers should by default follow the Org buffer
> they are created from and any changes to the coding system stipulated by export
> configuration must be applied before the buffer coding system gets used to make
> any decisions on how to export things.

IIUC, there is no such thing as a coding system associated to a buffer.
A coding system only kicks in when doing some I/O operation.

Anyway, with the same context described above, what's the return value
for:

  (detect-coding-string (org-export-as 'latex) t)

in the buffer you want to export?


Regards,

-- 
Nicolas Goaziou

Re: [Bug] Export Coding System

[Achim Gratz]

Nicolas Goaziou <n.goaziou <at> gmail.com> writes:
> IIUC, there is no such thing as a coding system associated to a buffer.
> A coding system only kicks in when doing some I/O operation.

The coding system should be associated with the file the buffer is visiting, but
a fresh buffer still shows a coding system indicator in the modeline even if it
is not (yet) associated with a file.  That coding system always seems to be the
default coding system as provided by the language environment.

> Anyway, with the same context described above, what's the return value
> for:
> 
>   (detect-coding-string (org-export-as 'latex) t)

Depends on what language environment is set to, but with the default setting of
my Emacs (German) it becomes iso-latin-1, independently of what the coding
system in the original Org buffer was.

I think that the export buffer coding system should be explicitly set (via
buffer-file-coding-system, which is automatically buffer-local) to copy the
coding of the parent buffer (or the coding specified via export options if
anything like that exists) so that the default choice of the language
environment doesn't kick in.


Regards,
Achim.

Re: [Bug] Export Coding System

[Nicolas Goaziou]

Achim Gratz <Stromeko@NexGo.DE> writes:

> Nicolas Goaziou <n.goaziou <at> gmail.com> writes:
>> IIUC, there is no such thing as a coding system associated to a buffer.
>> A coding system only kicks in when doing some I/O operation.
>
> The coding system should be associated with the file the buffer is visiting, but
> a fresh buffer still shows a coding system indicator in the modeline even if it
> is not (yet) associated with a file.  That coding system always seems to be the
> default coding system as provided by the language environment.
>
>> Anyway, with the same context described above, what's the return value
>> for:
>> 
>>   (detect-coding-string (org-export-as 'latex) t)
>
> Depends on what language environment is set to, but with the default setting of
> my Emacs (German) it becomes iso-latin-1, independently of what the coding
> system in the original Org buffer was.

In this case, it should be `utf-8', shouldn't it?

> I think that the export buffer coding system should be explicitly set (via
> buffer-file-coding-system, which is automatically buffer-local) to copy the
> coding of the parent buffer (or the coding specified via export options if
> anything like that exists) so that the default choice of the language
> environment doesn't kick in.

Still trying to understand: is the coding system wrong when you export
to a file, to a (temporary) buffer, or both?

Note that `org-export-to-file' use `coding-system-for-write', which
overrides `buffer-file-coding-system'. So this variable is probably
irrelevant here.


Regards,

-- 
Nicolas Goaziou

Re: [Bug] Export Coding System

[Achim Gratz]

Nicolas Goaziou writes:
>> Depends on what language environment is set to, but with the default setting of
>> my Emacs (German) it becomes iso-latin-1, independently of what the coding
>> system in the original Org buffer was.
>
> In this case, it should be `utf-8', shouldn't it?

I want it to be utf-8, but it isn't.  The course of events is apparently
that when a new buffer gets created it will have the default coding
system.  You are then apparently asking that buffer for the coding
system (always the default) and set inputenc accordingly, but should
have asked the parent Org buffer.  Or so I think, a few details are
probably still wrong.

>> I think that the export buffer coding system should be explicitly set (via
>> buffer-file-coding-system, which is automatically buffer-local) to copy the
>> coding of the parent buffer (or the coding specified via export options if
>> anything like that exists) so that the default choice of the language
>> environment doesn't kick in.
>
> Still trying to understand: is the coding system wrong when you export
> to a file, to a (temporary) buffer, or both?

Both.

> Note that `org-export-to-file' use `coding-system-for-write', which
> overrides `buffer-file-coding-system'. So this variable is probably
> irrelevant here.

No it is not irrelevant, it simply gets set too late in the game: it
asks for the new coding system when it is time to save the buffer, while
the content of the buffer has been cobbled together while assuming a
different coding system.  The only way I know (from browsing the
documentation) to override the coding system for a buffer that does not
yet have a file association is to set that variable directly,
preferrably directly after the buffer is created.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: [Bug] Export Coding System

[Nicolas Goaziou]

[-- Attachment #1: Type: text/plain, Size: 1907 bytes --]

Achim Gratz <Stromeko@nexgo.de> writes:

> Nicolas Goaziou writes:
>>> Depends on what language environment is set to, but with the default setting of
>>> my Emacs (German) it becomes iso-latin-1, independently of what the coding
>>> system in the original Org buffer was.
>>
>> In this case, it should be `utf-8', shouldn't it?
>
> I want it to be utf-8, but it isn't.  The course of events is apparently
> that when a new buffer gets created it will have the default coding
> system.  You are then apparently asking that buffer for the coding
> system (always the default) and set inputenc accordingly, but should
> have asked the parent Org buffer.  Or so I think, a few details are
> probably still wrong.
>
>>> I think that the export buffer coding system should be explicitly set (via
>>> buffer-file-coding-system, which is automatically buffer-local) to copy the
>>> coding of the parent buffer (or the coding specified via export options if
>>> anything like that exists) so that the default choice of the language
>>> environment doesn't kick in.
>>
>> Still trying to understand: is the coding system wrong when you export
>> to a file, to a (temporary) buffer, or both?
>
> Both.
>
>> Note that `org-export-to-file' use `coding-system-for-write', which
>> overrides `buffer-file-coding-system'. So this variable is probably
>> irrelevant here.
>
> No it is not irrelevant, it simply gets set too late in the game: it
> asks for the new coding system when it is time to save the buffer, while
> the content of the buffer has been cobbled together while assuming a
> different coding system.  The only way I know (from browsing the
> documentation) to override the coding system for a buffer that does not
> yet have a file association is to set that variable directly,
> preferrably directly after the buffer is created.

Does the following patch fix the problem?


Regards,

-- 
Nicolas Goaziou

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Fix encoding problem --]
[-- Type: text/x-patch, Size: 1237 bytes --]

From ed16f38854c197e8b31607bd32622d00e47fe10c Mon Sep 17 00:00:00 2001
From: Nicolas Goaziou <n.goaziou@gmail.com>
Date: Fri, 22 Feb 2013 23:07:04 +0100
Subject: [PATCH] ox: Fix coding system error

* lisp/ox.el (org-export--generate-copy-script): Clone
  `buffer-file-coding-system' when creating a buffer copy.

This patches makes sure the output will share the same encoding as the
original buffer.
---
 lisp/ox.el | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/lisp/ox.el b/lisp/ox.el
index efce29d..65e5acd 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -2729,10 +2729,13 @@ another buffer, effectively cloning the original buffer there."
 			(val (cdr entry)))
 		    (and (not (eq var 'org-font-lock-keywords))
 			 (or (memq var
-				   '(major-mode default-directory
-						buffer-file-name outline-level
-						outline-regexp
-						buffer-invisibility-spec))
+				   '(major-mode
+				     default-directory
+				     buffer-file-name
+				     buffer-file-coding-system
+				     outline-level
+				     outline-regexp
+				     buffer-invisibility-spec))
 			     (string-match "^\\(org-\\|orgtbl-\\)"
 					   (symbol-name var)))
 			 ;; Skip unreadable values, as they cannot be
-- 
1.8.1.4

Re: [Bug] Export Coding System

[Achim Gratz]

Nicolas Goaziou writes:
>> No it is not irrelevant, it simply gets set too late in the game: it
>> asks for the new coding system when it is time to save the buffer, while
>> the content of the buffer has been cobbled together while assuming a
>> different coding system.  The only way I know (from browsing the
>> documentation) to override the coding system for a buffer that does not
>> yet have a file association is to set that variable directly,
>> preferrably directly after the buffer is created.
>
> Does the following patch fix the problem?

It fixes the problem when the Org buffer encoding is identical to the
encoding for the LaTeX file.  It doesn't fix the problem when I want to
export the file in a different encoding (for instance by customizing Org
Export Latex Coding System), that would require a second step of
re-setting the buffer-file-coding-system before using it to determine
what to write into the inputenc option.  Other exporters like HTML
probably also have a way (or should have it) to determine the target
coding system independently from the input coding system, so this is not
a LaTeX only exercise.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptation for Waldorf microQ V2.22R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: [Bug] Export Coding System

[Nicolas Goaziou]

[-- Attachment #1: Type: text/plain, Size: 406 bytes --]

Achim Gratz <Stromeko@nexgo.de> writes:

> It doesn't fix the problem when I want to export the file in
> a different encoding (for instance by customizing Org Export Latex
> Coding System), that would require a second step of re-setting the
> buffer-file-coding-system before using it to determine what to write
> into the inputenc option.

Would the following patch work?


Regards,

-- 
Nicolas Goaziou

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ox-latex-Set-inputenc-option-according-to-org-export.patch --]
[-- Type: text/x-patch, Size: 931 bytes --]

From 2f2591719cc41e7312401ce7368b4bbbdde2c8ad Mon Sep 17 00:00:00 2001
From: Nicolas Goaziou <n.goaziou@gmail.com>
Date: Sat, 23 Feb 2013 11:29:45 +0100
Subject: [PATCH] ox-latex: Set inputenc option according to
 `org-export-coding-system'

* lisp/ox-latex.el (org-latex-guess-inputenc): Set inputenc option
  according to `org-export-coding-system'.
---
 lisp/ox-latex.el | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lisp/ox-latex.el b/lisp/ox-latex.el
index cbeaa4e..4047aa2 100644
--- a/lisp/ox-latex.el
+++ b/lisp/ox-latex.el
@@ -946,7 +946,7 @@ when specified inputenc option is \"AUTO\".
 Return the new header, as a string."
   (let* ((cs (or (ignore-errors
 		   (latexenc-coding-system-to-inputenc
-		    buffer-file-coding-system))
+		    (or org-export-coding-system buffer-file-coding-system)))
 		 "utf8")))
     (if (not cs) header
       ;; First translate if that is requested.
-- 
1.8.1.4

Re: [Bug] Export Coding System

[Achim Gratz]

Nicolas Goaziou writes:
> Would the following patch work?

Not for LaTeX export AFAICS, ox-latex doesn't seem to use
org-export-coding-system.  But that'd arguably be a bug in ox-latex,
then (more specifically org-latex-guess-inputenc), which is easily fixed
after that patch has been applied.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: [Bug] Export Coding System

[Nicolas Goaziou]

Achim Gratz <Stromeko@nexgo.de> writes:

> Nicolas Goaziou writes:
>> Would the following patch work?
>
> Not for LaTeX export AFAICS, ox-latex doesn't seem to use
> org-export-coding-system.  But that'd arguably be a bug in ox-latex,
> then (more specifically org-latex-guess-inputenc),

I don't understand your answer.

My default coding system is utf-8 and inputenc option is AUTO. When
I eval:

  (setq org-export-coding-system 'iso-latin-1)

and export some test buffer to LaTeX, with the suggested patch, I get:

  \usepackage[latin1]{inputenc}

So it looks like `org-export-coding-system' variable is taken into
consideration, doesn't it?

> which is easily fixed after that patch has been applied.

What fix are you talking about?

For now, I apply the previous patch.


Regards,

-- 
Nicolas Goaziou

Re: [Bug] Export Coding System

[Achim Gratz]

Nicolas Goaziou writes:
>> Not for LaTeX export AFAICS, ox-latex doesn't seem to use
>> org-export-coding-system.  But that'd arguably be a bug in ox-latex,
>> then (more specifically org-latex-guess-inputenc),
>
> I don't understand your answer.
>
> My default coding system is utf-8 and inputenc option is AUTO. When
> I eval:
>
>   (setq org-export-coding-system 'iso-latin-1)

The LaTeX exporter used to have a custom variable
org-export-latex-coding-system for this.  This variable is not evaluated
anymore but still visible in customize due to the fact that the new
exporter didn't override the definition from the old exporter.  So it's
no wonder that nothing changes when I customize away on this variable.

The only new exporter that defines its own coding system is currently
the HTML exporter.

> and export some test buffer to LaTeX, with the suggested patch, I get:
>
>   \usepackage[latin1]{inputenc}
>
> So it looks like `org-export-coding-system' variable is taken into
> consideration, doesn't it?

Yes, but I was looking at the example in the HTML exporter that would
prefer its own coding system over the general export coding system and
expected to see something similar.

> What fix are you talking about?

Introducing and using a defcustom specifically for LaTeX export (if
anything at all).

> For now, I apply the previous patch.

Please apply the second one, too.


Now I'll have to go and see what twisted ways it takes for the old
exporter defcustoms to show up in customize, they shouldn't have been
there at all.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Factory and User Sound Singles for Waldorf rackAttack:
http://Synth.Stromeko.net/Downloads.html#WaldorfSounds

Re: [Bug] Export Coding System

[Nicolas Goaziou]

Achim Gratz <Stromeko@nexgo.de> writes:

> Introducing and using a defcustom specifically for LaTeX export (if
> anything at all).

I don't think it is really important, considering you can also bind it
buffer-wise. On the other hand, a back-end independent variable is much
more easy to handle.

> Now I'll have to go and see what twisted ways it takes for the old
> exporter defcustoms to show up in customize, they shouldn't have been
> there at all.

As I explained in my announcement for new export framework, this is due
to customize autoloads from bundled Org (cus-load.el). As soon as you
call customize, these old friends are loaded.


Regards,

-- 
Nicolas Goaziou

Re: [Bug] Export Coding System

[Achim Gratz]

Nicolas Goaziou writes:
>> Now I'll have to go and see what twisted ways it takes for the old
>> exporter defcustoms to show up in customize, they shouldn't have been
>> there at all.
>
> As I explained in my announcement for new export framework, this is due
> to customize autoloads from bundled Org (cus-load.el). As soon as you
> call customize, these old friends are loaded.

Welp, and its all done with symbol properties so it would really be fun
to weed these out.  Great.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2:
http://Synth.Stromeko.net/Downloads.html#WaldorfSDada

Re: [Bug] Export Coding System

[Achim Gratz]

Nicolas Goaziou <n.goaziou <at> gmail.com> writes:
> Does the following patch fix the problem?

I've just had a chance to test this on the system where the problem originally
showed up (Cygwin with the current 24.3 pre-release).  On this system, the
buffer-file-coding-system is not copied and the (wrong) default takes over
again.  I'm puzzled, I can't see how this happens...


Regards,
Achim.

Re: [Bug] Export Coding System

[Achim Gratz]

Achim Gratz writes:
> I've just had a chance to test this on the system where the problem originally
> showed up (Cygwin with the current 24.3 pre-release).  On this system, the
> buffer-file-coding-system is not copied and the (wrong) default takes over
> again.  I'm puzzled, I can't see how this happens...

Also tested it with NT-Emacs on the same system and it also doesn't copy
all variables to the export buffer.  I can't find anything wrong with
the macro, however, if I run the code there directly it works.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

SD adaptations for KORG EX-800 and Poly-800MkII V0.9:
http://Synth.Stromeko.net/Downloads.html#KorgSDada

Re: [Bug] Export Coding System

[Nicolas Goaziou]

Hello,

Achim Gratz <Stromeko@nexgo.de> writes:

> Achim Gratz writes:
>> I've just had a chance to test this on the system where the problem originally
>> showed up (Cygwin with the current 24.3 pre-release).  On this system, the
>> buffer-file-coding-system is not copied and the (wrong) default takes over
>> again.  I'm puzzled, I can't see how this happens...
>
> Also tested it with NT-Emacs on the same system and it also doesn't copy
> all variables to the export buffer.  I can't find anything wrong with
> the macro, however, if I run the code there directly it works.

Can you reproduce the problem in both synchronous and asynchronous
export?


Regards,

-- 
Nicolas Goaziou