Mismatch in url escaping between org and exported html

Mismatch in url escaping between org and exported html

[Mark Janssen]

[-- Attachment #1: Type: text/plain, Size: 698 bytes --]

Hello list,

If I insert a http:// link containing question marks, the verbatim link
being inserted in the org document has the question mark escaped.

For example the link

http://mpcjanssen.nl/fossil/simpletask/tktview?name=ee0504fc8c

is actually inserted as:

[[http://mpcjanssen.nl/fossil/simpletask/tktview?name%3Dee0504fc8c][bug]]

This works fine when clicking the link from emacs, but breaks when
exporting to html. Then it is exported as:

<a href="http://mpcjanssen.nl/fossil/simpletask/tktview?name%3Dee0504fc8c
">bug</a>

This will not work.

Is this an error in the html exporter or in org mode itself? I would expect
the link to be included verbatim in the .org file.

Regards,
Mark

[-- Attachment #2: Type: text/html, Size: 1280 bytes --]

Re: Mismatch in url escaping between org and exported html

[Mark Janssen]

[-- Attachment #1: Type: text/plain, Size: 317 bytes --]

On Sun, Feb 9, 2014 at 9:56 PM, Mark Janssen <mpc.janssen@gmail.com> wrote:

> Hello list,
>
> If I insert a http:// link containing question marks, the verbatim link
> being inserted in the org document has the question mark escaped.
>
>
>
Seems (setq org-url-hexify-p nil) will give me the required behavior.

Mark

[-- Attachment #2: Type: text/html, Size: 800 bytes --]

Re: Mismatch in url escaping between org and exported html

[Bastien]

Hi Mark,

Mark Janssen <mpc.janssen@gmail.com> writes:

> If I insert a http:// link containing question marks, the verbatim
> link being inserted in the org document has the question mark
> escaped.

I can't reproduce this with latest stable or unstable Org version.

What version of Org and Emacs are you using?

M-x org-version RET
M-x emacs-version RET

Thanks,

-- 
 Bastien

Re: Mismatch in url escaping between org and exported html

[Mark Janssen]

[-- Attachment #1.1: Type: text/plain, Size: 1098 bytes --]

On Mon, Feb 10, 2014 at 12:06 PM, Bastien <bzg@gnu.org> wrote:

> Hi Mark,
>
> Mark Janssen <mpc.janssen@gmail.com> writes:
>
> > If I insert a http:// link containing question marks, the verbatim
> > link being inserted in the org document has the question mark
> > escaped.
>
> I can't reproduce this with latest stable or unstable Org version.
>
>
I shouldn't have posted this to the mailing list at the time I did :(.
Apologies for any confusion I have caused.
 The problem is with escaping of an equals sign not the question mark.

I have attached a demo org and a resulting html file. Note that in the html
file, the url is http://test/test?name%3Dme which is wrong. It should be
http://test/test?name=me

Running C-c C-ehh on that org file gives the attached html. I have started
emacs as:

emacs.exe -Q -l minimal.el test.org

;;minimal.el
(require 'package)
(package-initialize)
(require 'org)


> What version of Org and Emacs are you using?
>
>
(org-version)
"8.2.5h"

(emacs-version)
"GNU Emacs 24.3.1 (i386-mingw-nt6.1.7601)
 of 2013-03-17 on MARVIN"




> Thanks,
>
> --
>  Bastien
>

[-- Attachment #1.2: Type: text/html, Size: 2456 bytes --]

[-- Attachment #2: test.html --]
[-- Type: text/html, Size: 5266 bytes --]

[-- Attachment #3: test.org --]
[-- Type: application/octet-stream, Size: 36 bytes --]

[[http://test/test?name%3Dme][me]]

[-- Attachment #4: minimal.el --]
[-- Type: application/octet-stream, Size: 78 bytes --]

(require 'package)
(package-initialize)
(require 'org)
(require 'ox-html)

Re: Mismatch in url escaping between org and exported html

[Bastien]

Hi Mark,

it has been fixed in the master branch by Rick, I've cherry-picked the
change in the maint branch so that it will be part of the next minor
release.

Thanks,

-- 
 Bastien

Re: Mismatch in url escaping between org and exported html

[Mark Janssen]

[-- Attachment #1: Type: text/plain, Size: 309 bytes --]

On Tue, Feb 11, 2014 at 11:10 AM, Bastien <bzg@gnu.org> wrote:

> Hi Mark,
>
> it has been fixed in the master branch by Rick, I've cherry-picked the
> change in the maint branch so that it will be part of the next minor
> release.
>
> Thanks,
>
>
Hi Bastien and Rick,

Thanks for the great support.

--
Mark

[-- Attachment #2: Type: text/html, Size: 772 bytes --]

Re: Mismatch in url escaping between org and exported html

[Michael Brand]

[-- Attachment #1: Type: text/plain, Size: 874 bytes --]

Hi Bastien

From the commit of Rick Frankel for org-html-link (reformatted):

#+BEGIN_SRC emacs-lisp
  (org-link-escape
   (org-link-unescape (concat type ":" raw-path))
   org-link-escape-chars-browser)
#+END_SRC

I would like to discourage the use of the constant
org-link-escape-chars-browser in favor of the function
org-link-escape-browser that should use, and finally can be replaced
with, the function url-encode-url available since Emacs 24.3.1. The
patch attached for review will fix an Org link like e. g.
[[http://lists.gnu.org/archive/cgi-bin/namazu.cgi?idxname=emacs-orgmode&query=%252Bsubject:"Release+8.2"]]
with Emacs 24.3.1 now also for the Org export to HTML.

Some time ago I added url-encode-url already to the function
org-open-at-point. As this is not in maint and the attached patch
depends on it I intend to apply the latter only on master.

Michael

[-- Attachment #2: 0001-Fix-escaping-of-more-links-in-html-export.patch.txt --]
[-- Type: text/plain, Size: 5546 bytes --]

From 64b202f4a07dd7a1d927563df3c2df089c727aff Mon Sep 17 00:00:00 2001
From: Michael Brand <michael.ch.brand@gmail.com>
Date: Tue, 11 Feb 2014 20:00:26 +0100
Subject: [PATCH] Fix escaping of more links in HTML export

* lisp/org.el (org-link-escape-chars): Extend docstring.
(org-link-escape-chars-browser): Mention in docstring that it will
become a candidate for removal.
(org-link-escape-browser): Mention in docstring that it will become a
candidate for removal.
(org-open-at-point): Move `url-encode-url' and comments into
`org-link-escape-browser'.

* lisp/ox-html.el (org-html-link): Make use of
`org-link-escape-browser' like `org-open-at-point'.
---
 lisp/org.el     | 60 +++++++++++++++++++++++++++++----------------------------
 lisp/ox-html.el |  5 ++---
 2 files changed, 33 insertions(+), 32 deletions(-)

diff --git a/lisp/org.el b/lisp/org.el
index dfb0517..e87c930 100644
--- a/lisp/org.el
+++ b/lisp/org.el
@@ -9811,14 +9811,17 @@ according to FMT (default from `org-email-link-description-format')."
 (defconst org-link-escape-chars
   ;;%20 %2B %3B %3D %5B %5D
   '(?\  ?\+ ?\; ?\= ?\[ ?\])
-  "List of characters that should be escaped in link.
+  "List of characters that should be escaped in a link when stored to Org.
 This is the list that is used for internal purposes.")
 
 (defconst org-link-escape-chars-browser
   ;;%20 %22
   '(?\  ?\")
-  "List of escapes for characters that are problematic in links.
-This is the list that is used before handing over to the browser.")
+  "List of characters to be escaped before handing over to the browser.
+If you consider using this constant then you probably want to use
+the function `org-link-escape-browser' instead.  See there why
+this constant is a candidate to be removed once Org drops support
+for Emacs 24.1 and 24.2.")
 
 (defun org-link-escape (text &optional table merge)
   "Return percent escaped representation of TEXT.
@@ -9847,11 +9850,27 @@ If optional argument MERGE is set, merge TABLE into
        (char-to-string char))) text ""))
 
 (defun org-link-escape-browser (text)
-  (if (org-string-match-p
-       (concat "[[:nonascii:]" org-link-escape-chars-browser "]")
-       text)
-      (org-link-escape text org-link-escape-chars-browser)
-    text))
+  "Escape some characters before handing over to the browser.
+This function is a candidate to be removed together with the
+constant `org-link-escape-chars-browser' once Org drops support
+for Emacs 24.1 and 24.2.  All calls to this function will have to
+be replaced with `url-encode-url' which is available since Emacs
+24.3.1."
+  ;; Example with the Org link
+  ;; [[http://lists.gnu.org/archive/cgi-bin/namazu.cgi?idxname=emacs-orgmode&query=%252Bsubject:"Release+8.2"]]
+  ;; to open the browser with +subject:"Release 8.2" filled into the
+  ;; query field: In this case the variable TEXT contains the
+  ;; unescaped [...]=%2Bsubject:"Release+8.2".  Then `url-encode-url'
+  ;; converts correctly to [...]=%2Bsubject:%22Release+8.2%22 or
+  ;; `org-link-escape' with `org-link-escape-chars-browser' converts
+  ;; wrongly to [...]=%252Bsubject:%22Release+8.2%22.
+  (if (fboundp 'url-encode-url)
+      (url-encode-url text)
+    (if (org-string-match-p
+	 (concat "[[:nonascii:]" org-link-escape-chars-browser "]")
+	 text)
+	(org-link-escape text org-link-escape-chars-browser)
+      text)))
 
 (defun org-link-unescape (str)
   "Unhex hexified Unicode strings as returned from the JavaScript function
@@ -10576,29 +10595,12 @@ application the system uses for this file type."
 	      (apply cmd (nreverse args1))))
 
 	   ((member type '("http" "https" "ftp" "news"))
-	    ;; In the example of the http Org link
-	    ;; [[http://lists.gnu.org/archive/cgi-bin/namazu.cgi?idxname=emacs-orgmode&query=%252Bsubject:"Release+8.2"]]
-	    ;; to open a browser with +subject:"Release 8.2" in the
-	    ;; query field the variable `path' contains
-	    ;; [...]=%2Bsubject:"Release+8.2", `url-encode-url'
-	    ;; converts correct to [...]=%2Bsubject:%22Release+8.2%22
-	    ;; and `org-link-escape-browser' converts wrong to
-	    ;; [...]=%252Bsubject:%22Release+8.2%22.
-	    ;;
-	    ;; `url-encode-url' is available since Emacs 24.3.1 and
-	    ;; `org-link-escape-browser' can be removed altogether
-	    ;; once Org drops support for Emacs 24.1 and 24.2.
-	    (browse-url (funcall (if (fboundp 'url-encode-url)
-				     #'url-encode-url
-				   #'org-link-escape-browser)
-				 (concat type ":" path))))
+	    (browse-url (org-link-escape-browser
+			 (concat type ":" path))))
 
 	   ((string= type "doi")
-	    ;; See comments for type http above
-	    (browse-url (funcall (if (fboundp 'url-encode-url)
-				     #'url-encode-url
-				   #'org-link-escape-browser)
-				 (concat org-doi-server-url path))))
+	    (browse-url (org-link-escape-browser
+			 (concat org-doi-server-url path))))
 
 	   ((member type '("message"))
 	    (browse-url (concat type ":" path)))
diff --git a/lisp/ox-html.el b/lisp/ox-html.el
index 5ceea71..a8c924f 100644
--- a/lisp/ox-html.el
+++ b/lisp/ox-html.el
@@ -2718,9 +2718,8 @@ INFO is a plist holding contextual information.  See
 	 (path
 	  (cond
 	   ((member type '("http" "https" "ftp" "mailto"))
-	    (org-link-escape
-	     (org-link-unescape
-	      (concat type ":" raw-path)) org-link-escape-chars-browser))
+	    (org-link-escape-browser
+	     (org-link-unescape (concat type ":" raw-path))))
 	   ((string= type "file")
 	    ;; Treat links to ".org" files as ".html", if needed.
 	    (setq raw-path
-- 
1.7.12.4 (Apple Git-37)

Re: Mismatch in url escaping between org and exported html

[Bastien]

Hi Michael,

Michael Brand <michael.ch.brand@gmail.com> writes:

> Some time ago I added url-encode-url already to the function
> org-open-at-point. As this is not in maint and the attached patch
> depends on it I intend to apply the latter only on master.

Looks good, feel free to apply it on master, thanks!

-- 
 Bastien

Re: Mismatch in url escaping between org and exported html

[Mark Janssen]

[-- Attachment #1: Type: text/plain, Size: 849 bytes --]

On Sun, Feb 9, 2014 at 11:07 PM, Mark Janssen <mpc.janssen@gmail.com> wrote:

>
>
>>
> Seems (setq org-url-hexify-p nil) will give me the required behavior.
>
> Mark
>
>
This didn't work, so I investigated a bit further. Because the = sign is
included in org-link-escape-chars, the org-url-hexify-p value will not make
anu difference.

So the problem is still that equals signs are hexified in org-links.
Reading =org.el= and seeing the =org-link-escape-chars= constant this seems
to be expected behavior.

So I think the issue is that the html exporter doesn't unhexify the equals
sign in the link. As a result the link in the html doesn't work.

e.g.

The link:

http://test/test?name=me

Is translated to:

http://test/test?name%3Dme

In the org file

And that link is also included in the HTML where I would have expected the
equals sign again.

[-- Attachment #2: Type: text/html, Size: 2346 bytes --]