org-html-use-unicode-chars breaks source code blocks

org-html-use-unicode-chars breaks source code blocks

[Vladimir Alexiev]

Hi! 
I've set org-html-use-unicode-chars since I want ox-html to leave IRIs as IRIs.
But this has another undesired effect: it breaks <URL> references in code,
since it doesn't escape the brackets.

Eg this:

#+BEGIN_SRC Turtle
@prefix aat:   <http://vocab.getty.edu/aat/>.
#+END_SRC

results in the URL being invisible in the exported HTML.
The fault is here:

(defun org-html-final-function (contents backend info)
  ...
    (when org-html-use-unicode-chars
      (require 'mm-url)
      (mm-url-decode-entities))

previous code carefully escaped the entities in org-html-protect-char-alist, 
only for mm-url-decode-entities to unescape them.

http://article.gmane.org/gmane.emacs.orgmode/94742 is somewhat related.

Re: org-html-use-unicode-chars breaks source code blocks

[Rasmus]

[-- Attachment #1: Type: text/plain, Size: 477 bytes --]

Hi,

Vladimir Alexiev <vladimir.alexiev@ontotext.com> writes:

> I've set org-html-use-unicode-chars since I want ox-html to leave IRIs as IRIs.
> But this has another undesired effect: it breaks <URL> references in code,
> since it doesn't escape the brackets.

I think this should only apply to entities.  Any reason to do it on the
whole output?  Nicolas?

This patch makes that change.

Rasmus

-- 
This message is brought to you by the department of redundant departments

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ox-html-Only-translate-entities-to-UTF-8.patch --]
[-- Type: text/x-diff, Size: 1940 bytes --]

From 535366ec1e1819c73bb038712a19f5e1be0a51b7 Mon Sep 17 00:00:00 2001
From: Rasmus <rasmus@gmx.us>
Date: Tue, 4 Aug 2015 19:12:00 +0200
Subject: [PATCH 1/4] ox-html: Only translate entities to UTF-8

* ox-html.el (org-html-final-function): Do not check
 :html-use-unicode-chars.
 (org-html-entity): Check :html-use-unicode-chars
 (org-html-use-unicode-chars): Update docstring.

Reported-by: Vladimir Alexiev <vladimir.alexiev@ontotext.com>
<http://permalink.gmane.org/gmane.emacs.orgmode/99451>
---
 lisp/ox-html.el | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)

diff --git a/lisp/ox-html.el b/lisp/ox-html.el
index 2c13bf6..c329b72 100644
--- a/lisp/ox-html.el
+++ b/lisp/ox-html.el
@@ -609,10 +609,10 @@ Warning: non-nil may break indentation of source code blocks."
   :type 'boolean)
 
 (defcustom org-html-use-unicode-chars nil
-  "Non-nil means to use unicode characters instead of HTML entities."
+  "Non-nil means to use unicode characters for org-entities instead of HTML codes."
   :group 'org-export-html
-  :version "24.4"
-  :package-version '(Org . "8.0")
+  :version "25.1"
+  :package-version '(Org . "8.3")
   :type 'boolean)
 
 ;;;; Drawers
@@ -2359,7 +2359,9 @@ holding contextual information.  See `org-export-data'."
   "Transcode an ENTITY object from Org to HTML.
 CONTENTS are the definition itself.  INFO is a plist holding
 contextual information."
-  (org-element-property :html entity))
+  (if (plist-get info :html-use-unicode-chars)
+      (org-element-property :utf-8 entity)
+    (org-element-property :html entity)))
 
 ;;;; Example Block
 
@@ -3500,9 +3502,6 @@ contextual information."
     (set-auto-mode t)
     (if (plist-get info :html-indent)
 	(indent-region (point-min) (point-max)))
-    (when (plist-get info :html-use-unicode-chars)
-      (require 'mm-url)
-      (mm-url-decode-entities))
     (buffer-substring-no-properties (point-min) (point-max))))
 
 \f
-- 
2.5.0

Re: org-html-use-unicode-chars breaks source code blocks

[Nicolas Goaziou]

Hello,

Rasmus <rasmus@gmx.us> writes:

> Vladimir Alexiev <vladimir.alexiev@ontotext.com> writes:
>
>> I've set org-html-use-unicode-chars since I want ox-html to leave IRIs as IRIs.
>> But this has another undesired effect: it breaks <URL> references in code,
>> since it doesn't escape the brackets.
>
> I think this should only apply to entities.  Any reason to do it on the
> whole output?  Nicolas?

It was introduced in e8742b78e0a982a7fca0bf25b4f3551be58660ef. I'm not
sure about the intent of this variable but I tend to think it is about
beautification of the output. As a consequence, it isn't meant to apply
to Org entities specifically.

However, as you noticed, it is not subtle enough to apply
`mm-url-decode-entities' on the full output. It needs to be applied
piece-wise wherever that makes sense. `org-html-entity' is one case.
Maybe `org-html-plain-text' for another one.


Regards,

-- 
Nicolas Goaziou

Re: org-html-use-unicode-chars breaks source code blocks

[Rasmus]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Hello,
>
> Rasmus <rasmus@gmx.us> writes:
>
>> Vladimir Alexiev <vladimir.alexiev@ontotext.com> writes:
>>
>>> I've set org-html-use-unicode-chars since I want ox-html to leave IRIs as IRIs.
>>> But this has another undesired effect: it breaks <URL> references in code,
>>> since it doesn't escape the brackets.
>>
>> I think this should only apply to entities.  Any reason to do it on the
>> whole output?  Nicolas?
>
> It was introduced in e8742b78e0a982a7fca0bf25b4f3551be58660ef. I'm not
> sure about the intent of this variable but I tend to think it is about
> beautification of the output. As a consequence, it isn't meant to apply
> to Org entities specifically.
>
> However, as you noticed, it is not subtle enough to apply
> `mm-url-decode-entities' on the full output. It needs to be applied
> piece-wise wherever that makes sense. `org-html-entity' is one case.
> Maybe `org-html-plain-text' for another one.

OK.  I added it to plain-text as well.  What is an example of a plain-text
that would need beautification?

Should we apply it to snippets as well?  In the spirit of "beautification"
it would make sense, but it could also seem like a bad choice.

Rasmus

-- 
Slowly unravels in a ball of yarn and the devil collects it

Re: org-html-use-unicode-chars breaks source code blocks

[Nicolas Goaziou]

> OK.  I added it to plain-text as well.  What is an example of a plain-text
> that would need beautification?

To tell the truth, I don't know why we need beautification in the first
place. Bastien introduced it, so he may be able to answer.

> Should we apply it to snippets as well?  In the spirit of "beautification"
> it would make sense, but it could also seem like a bad choice.

I think it makes sense to apply it to snippets, indeed, but see above.

Regards,

Re: org-html-use-unicode-chars breaks source code blocks

[Rasmus]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

>> OK.  I added it to plain-text as well.  What is an example of a plain-text
>> that would need beautification?
>
> To tell the truth, I don't know why we need beautification in the first
> place. Bastien introduced it, so he may be able to answer.

To this extend; me neither.  But Vladimir uses it to "leave IRIs as IRIs"
(I don't know what this means).

>> Should we apply it to snippets as well?  In the spirit of "beautification"
>> it would make sense, but it could also seem like a bad choice.
>
> I think it makes sense to apply it to snippets, indeed, but see above.

My initial reaction was to kill it as well.  But I might feel like this a
bit to often (I feel the same way about headline keywords like COMMENT).

Rasmus

-- 
9000!

Re: org-html-use-unicode-chars breaks source code blocks

[Andreas Leha]

Hi,

[ deleted: discussion on beatification ]

>
> My initial reaction was to kill it as well.  But I might feel like this a
> bit to often (I feel the same way about headline keywords like COMMENT).

There has been repeated 'bashing' of the COMMENT keyword lately on this
list.  Let me just raise a voice in defence.  I do not mind the syntax
too much, but the functionality of commenting a whole subtree without
loosing the outline functionality is really handy.  Especially also in
distinction to the equally handy :noexport: tag.

So, even if there is probably not a high risk for the COMMENT keyword to
be dropped I just wanted to express my support for it.

Regards,
Andreas

Re: org-html-use-unicode-chars breaks source code blocks

[Sebastien Vauban]

Andreas Leha <andreas.leha-A1rZ2h3LdSKGMSlLMZIubhS11BummzK+@public.gmane.org> writes:
> [ deleted: discussion on beatification ]
>
>> My initial reaction was to kill it as well.  But I might feel like this a
>> bit to often (I feel the same way about headline keywords like COMMENT).
>
> There has been repeated 'bashing' of the COMMENT keyword lately on this
> list.  Let me just raise a voice in defence.  I do not mind the syntax
> too much, but the functionality of commenting a whole subtree without
> loosing the outline functionality is really handy.  Especially also in
> distinction to the equally handy :noexport: tag.
>
> So, even if there is probably not a high risk for the COMMENT keyword to
> be dropped I just wanted to express my support for it.

+1

Both COMMENT and :noexport: are necessary, for achieving different tasks.

Best regards,
  Seb

-- 
Sebastien Vauban

Re: org-html-use-unicode-chars breaks source code blocks

[Bastien Guerry]

Andreas Leha <andreas.leha@med.uni-goettingen.de> writes:

> So, even if there is probably not a high risk for the COMMENT keyword to
> be dropped I just wanted to express my support for it.

COMMENT will stay, for sure.

-- 
 Bastien

Re: org-html-use-unicode-chars breaks source code blocks

[Bastien Guerry]

Hi,

I removed `org-html-use-unicode-chars'.

Thanks,

-- 
 Bastien

Re: org-html-use-unicode-chars breaks source code blocks

[Brady Trainor]

Speaking of COMMENT, I had noticed some strange behavior if I have a state like COMMENTED_OUT. 

If I have a header like =#+TODO: TODO COMMENTED_OUT | DONE=, and cycle through state with S-<right arrow>, it gets pretty wonky. For now, I simply use COMMENT when I have a section I want to consider as commented out, but ideally this can behavior can be different? 

Always true, but I haven't been on the mailing list for a spell, so, big thanks to all who contribute to this software. 

Bastien Guerry <bzg@gnu.org> writes:

> Andreas Leha <andreas.leha@med.uni-goettingen.de> writes:
>
>> So, even if there is probably not a high risk for the COMMENT keyword to
>> be dropped I just wanted to express my support for it.
>
> COMMENT will stay, for sure.

-- 
Brady

Re: org-html-use-unicode-chars breaks source code blocks

[Nicolas Goaziou]

Hello,

Brady Trainor <algebrat@uw.edu> writes:

> Speaking of COMMENT, I had noticed some strange behavior if I have a state like COMMENTED_OUT. 
>
> If I have a header like =#+TODO: TODO COMMENTED_OUT | DONE=, and cycle
> through state with S-<right arrow>, it gets pretty wonky.

Could you elaborate a bit? I cannot reproduce anything suspicious except
a minor fontification glitch.


Regards,

-- 
Nicolas Goaziou

Re: org-html-use-unicode-chars breaks source code blocks

[Brady Trainor]

Hello,

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Hello,
>
> Brady Trainor <algebrat@uw.edu> writes:
>
>> Speaking of COMMENT, I had noticed some strange behavior if I have a
>> state like COMMENTED_OUT.
>>
>> If I have a header like =#+TODO: TODO COMMENTED_OUT | DONE=, and cycle
>> through state with S-<right arrow>, it gets pretty wonky.
>
> Could you elaborate a bit? I cannot reproduce anything suspicious except
> a minor fontification glitch.
>
>
> Regards,

Ah, I should have checked 8.3... This is on 8.2.10 that it was doing this. In 8.2.10, as states were passed through, I could see several different all-caps states in one headline at once, though only one would be highlighted.

I guess on 8.3, it is just that COMMENT of COMMENTED_OUT would be highlighted, leaving ED_OUT unhighlighted. I am guessing that is what you are seeing. 

--
Brady

Re: org-html-use-unicode-chars breaks source code blocks

[Rasmus]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Hello,
>
> Brady Trainor <algebrat@uw.edu> writes:
>
>> Speaking of COMMENT, I had noticed some strange behavior if I have a state like COMMENTED_OUT. 
>>
>> If I have a header like =#+TODO: TODO COMMENTED_OUT | DONE=, and cycle
>> through state with S-<right arrow>, it gets pretty wonky.
>
> Could you elaborate a bit? I cannot reproduce anything suspicious except
> a minor fontification glitch.

I only saw the fontification error as well, which can be fixed in
org-set-font-lock-defaults.  AFAICT, you did not fix this, right?

Rasmus

-- 
The right to be left alone is a human right