Incorrect hexification in URLs in LaTeX Export

Incorrect hexification in URLs in LaTeX Export

[R. Michael Weylandt <michael.weylandt@gmail.com>]

I've tried this with Org 7.9.3 and 8.2.5h to the same result:

<---------------------->
#+TITLE: Test
* One
Here is a [[http://google.com/search?q=orgmode][link]]
<---------------------->

Exporting to HTML doesn't transform the link but exporting to LaTeX results in the (non-working) http://google.com/search?%3Dorgmode

Is there a reason for this behavior and, if so, a way to work around it?

RFC 3986 2.2 explicitly says URLs may include `=` and =url-encode-url= doesn't change the link in question.

I've played with org-url-hexify-p and read past ML discussions, but they seem primarily concerned with characters which should not appear in URIs. 

Thanks,
Michael

Re: Incorrect hexification in URLs in LaTeX Export

[Andreas Leha]

"R. Michael Weylandt <michael.weylandt@gmail.com>"
<michael.weylandt@gmail.com> writes:

> I've tried this with Org 7.9.3 and 8.2.5h to the same result:
>
> <---------------------->
> #+TITLE: Test
> * One
> Here is a [[http://google.com/search?q=orgmode][link]]
> <---------------------->
>
> Exporting to HTML doesn't transform the link but exporting to LaTeX results in the (non-working) http://google.com/search?%3Dorgmode
>
> Is there a reason for this behavior and, if so, a way to work around it?
>
> RFC 3986 2.2 explicitly says URLs may include `=` and =url-encode-url= doesn't change the link in question.
>
> I've played with org-url-hexify-p and read past ML discussions, but
> they seem primarily concerned with characters which should not appear
> in URIs.
>
> Thanks,
> Michael



Hi Michael,

I have recently been bitten by this as well.  Based on a block
post [fn:1], I now have this in my .emacs as a work-around:

--8<---------------cut here---------------start------------->8---
(defun al-link-filter (contents backend info)
  (let ((contents (replace-regexp-in-string "\\\\#\\+name:.*$" "" contents)));; old and unrelated
      (replace-regexp-in-string "%3D" "=" contents)))
(add-to-list 'org-export-filter-final-output-functions 'al-link-filter)
--8<---------------cut here---------------end--------------->8---

It seems to work for me.

Regards,
Andreas



Footnotes:

[fn:1] http://irreal.org/blog/?p=2175

Org-link-escape-chars (was Incorrect hexification in URLs in LaTeX Export)

[R. Michael Weylandt <michael.weylandt@gmail.com>]

On Feb 28, 2014, at 15:26, Andreas Leha <andreas.leha@med.uni-goettingen.de> wrote:

> "R. Michael Weylandt <michael.weylandt@gmail.com>"
> <michael.weylandt@gmail.com> writes:
> 
>> I've tried this with Org 7.9.3 and 8.2.5h to the same result:
>> 
>> <---------------------->
>> #+TITLE: Test
>> * One
>> Here is a [[http://google.com/search?q=orgmode][link]]
>> <---------------------->
>> 
>> Exporting to HTML doesn't transform the link but exporting to LaTeX results in the (non-working) http://google.com/search?%3Dorgmode
>> 
>> Is there a reason for this behavior and, if so, a way to work around it?
>> 
>> RFC 3986 2.2 explicitly says URLs may include `=` and =url-encode-url= doesn't change the link in question.
>> 
>> I've played with org-url-hexify-p and read past ML discussions, but
>> they seem primarily concerned with characters which should not appear
>> in URIs.
>> 
>> Thanks,
>> Michael
> 
> 
> 
> Hi Michael,
> 
> I have recently been bitten by this as well.  Based on a block
> post [fn:1], I now have this in my .emacs as a work-around:
> 
> --8<---------------cut here---------------start------------->8---
> (defun al-link-filter (contents backend info)
>  (let ((contents (replace-regexp-in-string "\\\\#\\+name:.*$" "" contents)));; old and unrelated
>      (replace-regexp-in-string "%3D" "=" contents)))
> (add-to-list 'org-export-filter-final-output-functions 'al-link-filter)
> --8<---------------cut here---------------end---------------

Hi Andreas,

Thanks for this. I had time to dig further and it seems that the 'problem'  is the default value of org-link-escape-chars, which explicitly escapes = as well as other valid URL characters. 

Perhaps someone more familiar with this code could explain the choice these escapes. 

Michael

[Subject updated because the problem isn't LaTeX specific in 8.2.5h, only in 7.9.3]

Re: Org-link-escape-chars (was Incorrect hexification in URLs in LaTeX Export)

[Simon Thum]

This seems to be a question of objective. Do you want to encode, i.e. 
maintain some reversible original in an url no matter what, or do you 
want to fix url's which wouldn't otherwise be legal? In the latter case, 
the question mark should probably be retained.

On 03/03/2014 09:06 PM, R. Michael Weylandt <michael.weylandt@gmail.com> 
wrote:
>
>
> On Feb 28, 2014, at 15:26, Andreas Leha <andreas.leha@med.uni-goettingen.de> wrote:
>
>> "R. Michael Weylandt <michael.weylandt@gmail.com>"
>> <michael.weylandt@gmail.com> writes:
>>
>>> I've tried this with Org 7.9.3 and 8.2.5h to the same result:
>>>
>>> <---------------------->
>>> #+TITLE: Test
>>> * One
>>> Here is a [[http://google.com/search?q=orgmode][link]]
>>> <---------------------->
>>>
>>> Exporting to HTML doesn't transform the link but exporting to LaTeX results in the (non-working) http://google.com/search?%3Dorgmode
>>>
>>> Is there a reason for this behavior and, if so, a way to work around it?
>>>
>>> RFC 3986 2.2 explicitly says URLs may include `=` and =url-encode-url= doesn't change the link in question.
>>>
>>> I've played with org-url-hexify-p and read past ML discussions, but
>>> they seem primarily concerned with characters which should not appear
>>> in URIs.
>>>
>>> Thanks,
>>> Michael
>>
>>
>>
>> Hi Michael,
>>
>> I have recently been bitten by this as well.  Based on a block
>> post [fn:1], I now have this in my .emacs as a work-around:
>>
>> --8<---------------cut here---------------start------------->8---
>> (defun al-link-filter (contents backend info)
>>   (let ((contents (replace-regexp-in-string "\\\\#\\+name:.*$" "" contents)));; old and unrelated
>>       (replace-regexp-in-string "%3D" "=" contents)))
>> (add-to-list 'org-export-filter-final-output-functions 'al-link-filter)
>> --8<---------------cut here---------------end---------------
>
> Hi Andreas,
>
> Thanks for this. I had time to dig further and it seems that the 'problem'  is the default value of org-link-escape-chars, which explicitly escapes = as well as other valid URL characters.
>
> Perhaps someone more familiar with this code could explain the choice these escapes.
>
> Michael
>
> [Subject updated because the problem isn't LaTeX specific in 8.2.5h, only in 7.9.3]
>
>

Re: Org-link-escape-chars (was Incorrect hexification in URLs in LaTeX Export)

[R. Michael Weylandt]

On Tue, Mar 4, 2014 at 3:45 PM, Simon Thum <simon.thum@gmx.de> wrote:
> This seems to be a question of objective. Do you want to encode, i.e.
> maintain some reversible original in an url no matter what, or do you want
> to fix url's which wouldn't otherwise be legal? In the latter case, the
> question mark should probably be retained.
>

I believe the former. If the user types in a working link, the
exporter shouldn't break it.

This could be fixed by sprinkling org-url-decode through various
backends, but that suggests to me that the problem may be "upstream."

Michael

Re: Org-link-escape-chars (was Incorrect hexification in URLs in LaTeX Export)

[Andreas Leha]

"R. Michael Weylandt" <michael.weylandt@gmail.com> writes:

> On Tue, Mar 4, 2014 at 3:45 PM, Simon Thum <simon.thum@gmx.de> wrote:
>> This seems to be a question of objective. Do you want to encode, i.e.
>> maintain some reversible original in an url no matter what, or do you want
>> to fix url's which wouldn't otherwise be legal? In the latter case, the
>> question mark should probably be retained.
>>

I am not sure, that I understand what you say here.  But in case you
want to convince me, that exporting non-working links when the user
supplies working links is a feature and not a bug, you will have a hard
time ;-)

- Andreas

>
> I believe the former. If the user types in a working link, the
> exporter shouldn't break it.
>
> This could be fixed by sprinkling org-url-decode through various
> backends, but that suggests to me that the problem may be "upstream."
>
> Michael

Re: Org-link-escape-chars (was Incorrect hexification in URLs in LaTeX Export)

[Simon Thum]

Hehe ;) No, I was merely suggesting that maybe someone tried too hard to 
fix non-working urls becauase of other use cases.

Luckily it seems fixed now.


On 03/06/2014 10:17 AM, Andreas Leha wrote:
> "R. Michael Weylandt" <michael.weylandt@gmail.com> writes:
>
>> On Tue, Mar 4, 2014 at 3:45 PM, Simon Thum <simon.thum@gmx.de> wrote:
>>> This seems to be a question of objective. Do you want to encode, i.e.
>>> maintain some reversible original in an url no matter what, or do you want
>>> to fix url's which wouldn't otherwise be legal? In the latter case, the
>>> question mark should probably be retained.
>>>
>
> I am not sure, that I understand what you say here.  But in case you
> want to convince me, that exporting non-working links when the user
> supplies working links is a feature and not a bug, you will have a hard
> time ;-)
>
> - Andreas
>
>>
>> I believe the former. If the user types in a working link, the
>> exporter shouldn't break it.
>>
>> This could be fixed by sprinkling org-url-decode through various
>> backends, but that suggests to me that the problem may be "upstream."
>>
>> Michael
>
>
>

Re: Incorrect hexification in URLs in LaTeX Export

[Bastien]

"R. Michael Weylandt <michael.weylandt@gmail.com>"
<michael.weylandt@gmail.com> writes:

> I've tried this with Org 7.9.3 and 8.2.5h to the same result:
>
> <---------------------->
> #+TITLE: Test
> * One
> Here is a [[http://google.com/search?q=orgmode][link]]
> <---------------------->
>
> Exporting to HTML doesn't transform the link but exporting to LaTeX
> results in the (non-working) http://google.com/search?%3Dorgmode

I think this is now fixed in master.  Can anyone confirm this?

-- 
 Bastien

Re: Incorrect hexification in URLs in LaTeX Export

[Andreas Leha]

Hi Bastien,

Bastien <bzg@gnu.org> writes:

> "R. Michael Weylandt <michael.weylandt@gmail.com>"
> <michael.weylandt@gmail.com> writes:
>
>> I've tried this with Org 7.9.3 and 8.2.5h to the same result:
>>
>> <---------------------->
>> #+TITLE: Test
>> * One
>> Here is a [[http://google.com/search?q=orgmode][link]]
>> <---------------------->
>>
>> Exporting to HTML doesn't transform the link but exporting to LaTeX
>> results in the (non-working) http://google.com/search?%3Dorgmode
>
> I think this is now fixed in master.  Can anyone confirm this?


Thanks for looking into this.  It seems the proposed fix is working at
most partly.  Or maybe I get something wrong?


Following is the test file from above in an extended version.

The first link is exported correctly (I do not know whether it has been
before).

I opened the link in firefox, copied the address from firefox and
inserted into the Org file via 'C-c C-l', which gives me the second
link.

The second link is not clickable in the resulting pdf.


--8<---------------cut here---------------start------------->8---
#+TITLE: Test
* One
Here is a [[http://google.com/search?q=orgmode][link]]

This one is the same, but inserted via C-c C-l and the system clipboard.
[[https://encrypted.google.com/search?q%3Dorgmode][link]]

#+LaTeX_header: \usepackage{hyperref}
--8<---------------cut here---------------end--------------->8---


Regards,
Andreas

Re: Incorrect hexification in URLs in LaTeX Export

[Bastien]

Hi Andreas,

Andreas Leha <andreas.leha@med.uni-goettingen.de> writes:

> The second link is not clickable in the resulting pdf.

This should be fixed now, thanks.

-- 
 Bastien

Re: Incorrect hexification in URLs in LaTeX Export

[R. Michael Weylandt <michael.weylandt@gmail.com>]

On Mar 18, 2014, at 11:41, Bastien <bzg@gnu.org> wrote:

> Hi Andreas,
> 
> Andreas Leha <andreas.leha@med.uni-goettingen.de> writes:
> 
>> The second link is not clickable in the resulting pdf.
> 
> This should be fixed now, thanks.
> 

Hi Bastien,

I just tried with 35f27a1fe and my issue is fixed. (The one about the hexifying of a valid character. I.e., if (fundamental-mode) shows a valid link, the resulting export is also valid)

Can't comment on Andreas's issue about unescaping text when it's given to org already escaped. 

Thanks,
Michael 

Re: Incorrect hexification in URLs in LaTeX Export

[R. Michael Weylandt <michael.weylandt@gmail.com>]

On Mar 18, 2014, at 17:24, "R. Michael Weylandt" <michael.weylandt@gmail.com> wrote:

> Can't comment on Andreas's issue about unescaping text when it's given to org already escaped. 

Hi Bastien,

TLDR: remove ?\= from org-link-escape-chars.

I looked at this again and I think I've stumbled on another small issue relating to hex escaping. 

If I type a link with an equals sign (=) into the link prompt of org-insert-link, the link inserted into my buffer has a %3D instead of the valid =. 

Tracing this, it goes through org-make-link-string, which hands off to org-link-escape which uses the defconst org-link-escape-chars as the table of values to escape. Org-link-escape-chars appears to only be used within org-link-escape at this time.

I think = is escaped because it may have conflicted with the verbatim syntax at some point, but that doesn't appear to be an issue any more. 

Is there a reason not to remove it?

Re: Incorrect hexification in URLs in LaTeX Export

[Bastien]

Hi Michael,

"R. Michael Weylandt <michael.weylandt@gmail.com>"
<michael.weylandt@gmail.com> writes:

> TLDR: remove ?\= from org-link-escape-chars.

Done (in master.)

I'm copying David since he's the author of this commit:
http://orgmode.org/w/?p=org-mode.git;a=commitdiff;h=1a68b6

David, sorry to jump in, but are there any reason why the
characters + ; and = where escaped in this commit?

Thanks in advance,

-- 
 Bastien

Re: Incorrect hexification in URLs in LaTeX Export

[David Maus]

Hi all,

At Sun, 25 May 2014 07:56:15 +0200,
Bastien wrote:
> 
> Hi Michael,
> 
> "R. Michael Weylandt <michael.weylandt@gmail.com>"
> <michael.weylandt@gmail.com> writes:
> 
> > TLDR: remove ?\= from org-link-escape-chars.
> 
> Done (in master.)
> 
> I'm copying David since he's the author of this commit:
> http://orgmode.org/w/?p=org-mode.git;a=commitdiff;h=1a68b6
> 
> David, sorry to jump in, but are there any reason why the
> characters + ; and = where escaped in this commit?
> 

I think the only reason was that these characters already had been
escaped in `org-link-escape-chars'.

The commit removed the special rules for the letters with
diacritics. The previous commit (0c4bb0e) introduced an algorithm that
covered non-ASCII characters in general, thus special rules for
letters with diacritics where no longer necessary.

I can only speculate why they were escaped in the first place: + ; and
= do have special meaning in HTTP URIs and IIRC Org did not draw a
strict distinction between escaping for internal purposes and escaping
of HTTP URIs.

Best,
  -- David

> Thanks in advance,
> 
> -- 
>  Bastien

Re: Incorrect hexification in URLs in LaTeX Export

[Bastien]

Hi David,

David Maus <dmaus@ictsoc.de> writes:

> I can only speculate why they were escaped in the first place: + ; and
> = do have special meaning in HTTP URIs and IIRC Org did not draw a
> strict distinction between escaping for internal purposes and escaping
> of HTTP URIs.

Yes, that's certainly it -- thanks for enlightening me on this.

+ ; and = are not the only reserved characters, cf. RFC 3986 mentions
# @ and others that we don't escape.  So I think we're good here.

Best,

-- 
 Bastien

Re: Incorrect hexification in URLs in LaTeX Export

[David Maus]

Followup: There has been a discussion about hex-escaping last year
with some back-and-forth on the topic of link escaping:

http://thread.gmane.org/gmane.emacs.orgmode/74983/focus=75002

It's quite a muddy area.

Best,
  -- David

At Sun, 25 May 2014 09:09:50 +0200,
David Maus wrote:
> 
> Hi all,
> 
> At Sun, 25 May 2014 07:56:15 +0200,
> Bastien wrote:
> > 
> > Hi Michael,
> > 
> > "R. Michael Weylandt <michael.weylandt@gmail.com>"
> > <michael.weylandt@gmail.com> writes:
> > 
> > > TLDR: remove ?\= from org-link-escape-chars.
> > 
> > Done (in master.)
> > 
> > I'm copying David since he's the author of this commit:
> > http://orgmode.org/w/?p=org-mode.git;a=commitdiff;h=1a68b6
> > 
> > David, sorry to jump in, but are there any reason why the
> > characters + ; and = where escaped in this commit?
> > 
> 
> I think the only reason was that these characters already had been
> escaped in `org-link-escape-chars'.
> 
> The commit removed the special rules for the letters with
> diacritics. The previous commit (0c4bb0e) introduced an algorithm that
> covered non-ASCII characters in general, thus special rules for
> letters with diacritics where no longer necessary.
> 
> I can only speculate why they were escaped in the first place: + ; and
> = do have special meaning in HTTP URIs and IIRC Org did not draw a
> strict distinction between escaping for internal purposes and escaping
> of HTTP URIs.
> 
> Best,
>   -- David
> 
> > Thanks in advance,
> > 
> > -- 
> >  Bastien

Re: Incorrect hexification in URLs in LaTeX Export

[Bastien]

Hi David,

David Maus <dmaus@ictsoc.de> writes:

> It's quite a muddy area.

Yes, I can see this.  I'm all for the status quo right now, as long
as there are no real bugs behind the current code.  Removing chars
from `org-link-escape-chars' won't fix anything deepl, but it may
help clearing the waters.

Thanks again,

-- 
 Bastien