ox.html causes w3c xhtml validation

ox.html causes w3c xhtml validation

[Colin Baxter]

Publishing with Org mode version 9.3.6 (release_9.3.6-419-gd9b1fb) I get
an validation error using http://validator.w3.org/check. The error is:

----- Begin -----
Warning Line 205, Column 74:
cannot generate system identifier for general entity "dn"

…rn:btih:1f739d935676111cfff4b4693e3816e664797050&dn=gpl-3.0.txt GPL-v3-or-Later

    An entity reference was found in the document, but there is no
    reference by that name defined. Often this is caused by misspelling
    the reference name, unencoded ampersands, or by leaving off the
    trailing semicolon (;). The most common cause of this error is
    unencoded ampersands in URLs as described by the WDG in "Ampersands
    in URLs".

----- End -------

The cause is ox-html, which has the following lines

./lisp/ox-html.el:235:// @license
magnet:?xt=urn:btih:1f739d935676111cfff4b4693e3816e664797050&dn=gpl-3.0.txt
GPL-v3-or-Later

./lisp/ox-html.el:513:// @license
magnet:?xt=urn:btih:1f739d935676111cfff4b4693e3816e664797050&dn=gpl-3.0.txt
GPL-v3-or-Later

./lisp/ox-html.el:518:// @license
magnet:?xt=urn:btih:1f739d935676111cfff4b4693e3816e664797050&dn=gpl-3.0.txt
GPL-v3-or-Later


I think the culprit is commit 68fa5e589f00c8d5b4f7f0dc70be6ebe59238bb8,
11 Feb.

Is it possible to fix this?

Best wishes,


Colin Baxter
www.Colin-Baxter.com

Re: ox.html causes w3c xhtml validation

[Colin Baxter]

There is no error using org-mode built in the emacs-28.0.50 so it does
look as if commit 68fa5e589f00c8d5b4f7f0dc70be6ebe59238bb8 is the culprit.

Is it only me using  http://validator.w3.org/check?

Best wishes,

-- 
Colin Baxter
www.Colin-Baxter.com

Re: ox.html causes w3c xhtml validation

[Kyle Meyer]

Colin Baxter <m43cap@yandex.com> writes:

> ----- Begin -----
> Warning Line 205, Column 74:
> cannot generate system identifier for general entity "dn"
>
> …rn:btih:1f739d935676111cfff4b4693e3816e664797050&dn=gpl-3.0.txt GPL-v3-or-Later
>
>     An entity reference was found in the document, but there is no
>     reference by that name defined. Often this is caused by misspelling
>     the reference name, unencoded ampersands, or by leaving off the
>     trailing semicolon (;). The most common cause of this error is
>     unencoded ampersands in URLs as described by the WDG in "Ampersands
>     in URLs".
>
> ----- End -------
>
> The cause is ox-html, which has the following lines
>
> ./lisp/ox-html.el:235:// @license
> magnet:?xt=urn:btih:1f739d935676111cfff4b4693e3816e664797050&dn=gpl-3.0.txt
> GPL-v3-or-Later
[...]
> I think the culprit is commit 68fa5e589f00c8d5b4f7f0dc70be6ebe59238bb8,
> 11 Feb.
>
> Is it possible to fix this?

From what I gather from briefly searching around (mostly based on this
information [0]), I don't think there's an easy fix for inline
javascript.

I wonder if Arne [+cc], who suggested the change [1], knows of a solution.

[0]: https://mrcoles.com/blog/how-use-amersands-html-encode/
[1]: https://lists.gnu.org/archive/html/emacs-orgmode/2019-11/msg00030.html

Re: ox.html causes w3c xhtml validation

[Colin Baxter]

Dear Kyle,

Thanks for your reply.

>>>>> Kyle Meyer <kyle@kyleam.com> writes:

    > Colin Baxter <m43cap@yandex.com> writes:
    >> ----- Begin ----- Warning Line 205, Column 74: cannot generate
    >> system identifier for general entity "dn"
    >> 
    >> …rn:btih:1f739d935676111cfff4b4693e3816e664797050&dn=gpl-3.0.txt
    >> GPL-v3-or-Later
    >> 
    >> An entity reference was found in the document, but there is no
    >> reference by that name defined. Often this is caused by
    >> misspelling the reference name, unencoded ampersands, or by
    >> leaving off the trailing semicolon (;). The most common cause of
    >> this error is unencoded ampersands in URLs as described by the
    >> WDG in "Ampersands in URLs".
    >> 
    >> ----- End -------
    >> 
    >> The cause is ox-html, which has the following lines
    >> 
    >> ./lisp/ox-html.el:235:// @license
    >> magnet:?xt=urn:btih:1f739d935676111cfff4b4693e3816e664797050&dn=gpl-3.0.txt
    >> GPL-v3-or-Later
    > [...]
    >> I think the culprit is commit
    >> 68fa5e589f00c8d5b4f7f0dc70be6ebe59238bb8, 11 Feb.
    >> 
    >> Is it possible to fix this?

    > From what I gather from briefly searching around (mostly based on
    > this information [0]), I don't think there's an easy fix for
    > inline javascript.

In my opinion, if it can't be fixed then the changes should be
removed. Surely, we cannot have an org-mode that knowingly
exports/publishes something that causes a validation error!

    > I wonder if Arne [+cc], who suggested the change [1], knows of a
    > solution.

    > [0]: https://mrcoles.com/blog/how-use-amersands-html-encode/ [1]:
    > https://lists.gnu.org/archive/html/emacs-orgmode/2019-11/msg00030.html

Best wishes,

Colin.

-- 
Colin Baxter
www.Colin-Baxter.com

Re: ox.html causes w3c xhtml validation

[Adam Porter]

Colin Baxter <m43cap@yandex.com> writes:

> In my opinion, if it can't be fixed then the changes should be
> removed. Surely, we cannot have an org-mode that knowingly
> exports/publishes something that causes a validation error!

Looking at the error message, the fix might be very simple:

  The most common cause of this error is unencoded ampersands in URLs as
  described by the WDG in "Ampersands in URLs".

Re: ox.html causes w3c xhtml validation

[Kyle Meyer]

Adam Porter <adam@alphapapa.net> writes:

> Colin Baxter <m43cap@yandex.com> writes:
>
>> In my opinion, if it can't be fixed then the changes should be
>> removed. Surely, we cannot have an org-mode that knowingly
>> exports/publishes something that causes a validation error!
>
> Looking at the error message, the fix might be very simple:
>
>   The most common cause of this error is unencoded ampersands in URLs as
>   described by the WDG in "Ampersands in URLs".

Hmm, perhaps it is as simple as encoding the ampersand.  That won't work
with inline javascript, if I'm understanding correctly the link I gave
in my reply.  *But*, despite being embedded javascript, the librejs
magnet link happens within a comment, so I suppose it really just comes
down to how librejs treats it.  Poking around in librejs (525e3a5), it
seems it is clever and will s/&amp;/&/:

    // Match by link on first parameter (legacy)
    if (licenses[key]["Magnet link"] === first.replace("&amp;","&") ||
        licenses[key]["URL"] === first.replace("&amp;","&")) {
          return [true, `Recognized license: "${licenses[key]['Name']}".`];

Colin, could you try exporting with the change below and see if that
resolves your validation issues?  It'd also be great if you could check
whether librejs is still happy after that change.

-- >8 --
diff --git a/lisp/ox-html.el b/lisp/ox-html.el
index 9a1dd44b5..5c2b766e9 100644
--- a/lisp/ox-html.el
+++ b/lisp/ox-html.el
@@ -232,7 +232,7 @@ (defconst org-html-special-string-regexps
 
 (defconst org-html-scripts
   "<script type=\"text/javascript\">
-// @license magnet:?xt=urn:btih:1f739d935676111cfff4b4693e3816e664797050&dn=gpl-3.0.txt GPL-v3-or-Later
+// @license magnet:?xt=urn:btih:1f739d935676111cfff4b4693e3816e664797050&amp;dn=gpl-3.0.txt GPL-v3-or-Later
 <!--/*--><![CDATA[/*><!--*/
      function CodeHighlightOn(elem, id)
      {
@@ -510,12 +510,12 @@ (defcustom org-html-infojs-options
 
 (defcustom org-html-infojs-template
   "<script type=\"text/javascript\" src=\"%SCRIPT_PATH\">
-// @license magnet:?xt=urn:btih:1f739d935676111cfff4b4693e3816e664797050&dn=gpl-3.0.txt GPL-v3-or-Later
+// @license magnet:?xt=urn:btih:1f739d935676111cfff4b4693e3816e664797050&amp;dn=gpl-3.0.txt GPL-v3-or-Later
 // @license-end
 </script>
 
 <script type=\"text/javascript\">
-// @license magnet:?xt=urn:btih:1f739d935676111cfff4b4693e3816e664797050&dn=gpl-3.0.txt GPL-v3-or-Later
+// @license magnet:?xt=urn:btih:1f739d935676111cfff4b4693e3816e664797050&amp;dn=gpl-3.0.txt GPL-v3-or-Later
 <!--/*--><![CDATA[/*><!--*/
 %MANAGER_OPTIONS
 org_html_manager.setup();  // activate after the parameters are set

Re: ox.html causes w3c xhtml validation

[Arne Babenhauserheide]

Dear Kyle, Dear Colin,

Colin Baxter <m43cap@yandex.com> writes:
>     >> ./lisp/ox-html.el:235:// @license
>     >> magnet:?xt=urn:btih:1f739d935676111cfff4b4693e3816e664797050&dn=gpl-3.0.txt
>     >> GPL-v3-or-Later
>     > [...]
>     >> I think the culprit is commit
>     >> 68fa5e589f00c8d5b4f7f0dc70be6ebe59238bb8, 11 Feb.

>     > From what I gather from briefly searching around (mostly based on
>     > this information [0]), I don't think there's an easy fix for
>     > inline javascript.
>
> In my opinion, if it can't be fixed then the changes should be
> removed. Surely, we cannot have an org-mode that knowingly
> exports/publishes something that causes a validation error!
>
>     > I wonder if Arne [+cc], who suggested the change [1], knows of a
>     > solution.

Firstoff: I can reproduce this by using an xhtml header:

    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
    <html xmlns="http://www.w3.org/1999/xhtml">

And second: I just verified that LibreJS still recognizes the license
when we escape the & as &amp;

I would suggest the following:

diff --git a/lisp/ox-html.el b/lisp/ox-html.el
index 9a1dd44b5..5c2b766e9 100644
--- a/lisp/ox-html.el
+++ b/lisp/ox-html.el
@@ -232,7 +232,7 @@ property on the headline itself.")
 
 (defconst org-html-scripts
   "<script type=\"text/javascript\">
-// @license magnet:?xt=urn:btih:1f739d935676111cfff4b4693e3816e664797050&dn=gpl-3.0.txt GPL-v3-or-Later
+// @license magnet:?xt=urn:btih:1f739d935676111cfff4b4693e3816e664797050&amp;dn=gpl-3.0.txt GPL-v3-or-Later
 <!--/*--><![CDATA[/*><!--*/
      function CodeHighlightOn(elem, id)
      {
@@ -510,12 +510,12 @@ means to use the maximum value consistent with other options."
 
 (defcustom org-html-infojs-template
   "<script type=\"text/javascript\" src=\"%SCRIPT_PATH\">
-// @license magnet:?xt=urn:btih:1f739d935676111cfff4b4693e3816e664797050&dn=gpl-3.0.txt GPL-v3-or-Later
+// @license magnet:?xt=urn:btih:1f739d935676111cfff4b4693e3816e664797050&amp;dn=gpl-3.0.txt GPL-v3-or-Later
 // @license-end
 </script>
 
 <script type=\"text/javascript\">
-// @license magnet:?xt=urn:btih:1f739d935676111cfff4b4693e3816e664797050&dn=gpl-3.0.txt GPL-v3-or-Later
+// @license magnet:?xt=urn:btih:1f739d935676111cfff4b4693e3816e664797050&amp;dn=gpl-3.0.txt GPL-v3-or-Later
 <!--/*--><![CDATA[/*><!--*/
 %MANAGER_OPTIONS
 org_html_manager.setup();  // activate after the parameters are set

Thank you for CC'ing me!

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein
ohne es zu merken

Re: ox.html causes w3c xhtml validation

[Kyle Meyer]

Arne Babenhauserheide <arne_bab@web.de> writes:

> And second: I just verified that LibreJS still recognizes the license
> when we escape the & as &amp;
>
> I would suggest the following:
>
> diff --git a/lisp/ox-html.el b/lisp/ox-html.el

Great, thanks the suggestion and verifying the LibreJS recognizes it.
That it matches what I ended up landing on a bit downstream (sorry, it
looks like along the way you got dropped from the cc):

  https://lists.gnu.org/archive/html/emacs-orgmode/2020-03/msg00153.html

I'll apply the follow-up patch that you sent.

Thanks again.

Re: ox.html causes w3c xhtml validation

[Colin Baxter]

>>>>> Kyle Meyer <kyle@kyleam.com> writes:

    > Adam Porter <adam@alphapapa.net> writes:
    >> Colin Baxter <m43cap@yandex.com> writes:
    >> 
    >>> In my opinion, if it can't be fixed then the changes should be
    >>> removed. Surely, we cannot have an org-mode that knowingly
    >>> exports/publishes something that causes a validation error!
    >> 
    >> Looking at the error message, the fix might be very simple:
    >> 
    >> The most common cause of this error is unencoded ampersands in
    >> URLs as described by the WDG in "Ampersands in URLs".

    > Hmm, perhaps it is as simple as encoding the ampersand.  That
    > won't work with inline javascript, if I'm understanding correctly
    > the link I gave in my reply.  *But*, despite being embedded
    > javascript, the librejs magnet link happens within a comment, so I
    > suppose it really just comes down to how librejs treats it.
    > Poking around in librejs (525e3a5), it seems it is clever and will
    > s/&amp;/&/:

    >     // Match by link on first parameter (legacy) if
    > (licenses[key]["Magnet link"] === first.replace("&amp;","&") ||
    > licenses[key]["URL"] === first.replace("&amp;","&")) { return
    > [true, `Recognized license: "${licenses[key]['Name']}".`];

    > Colin, could you try exporting with the change below and see if
    > that resolves your validation issues?  It'd also be great if you
    > could check whether librejs is still happy after that change.

Ok, I'll try later today and report back.

Thanks again.

Best wishes,


-- 
Colin Baxter
www.Colin-Baxter.com
---------------------------------------------------------------------
GnuPG fingerprint: 68A8 799C 0230 16E7 BF68  2A27 BBFA 2492 91F5 41C8
---------------------------------------------------------------------
Since mathematicians have invaded the theory of relativity, I do not
understand it myself. A. Einstein

Re: ox.html causes w3c xhtml validation

[Colin Baxter]

>>>>> Kyle Meyer <kyle@kyleam.com> writes:

    > Adam Porter <adam@alphapapa.net> writes:
    >> Colin Baxter <m43cap@yandex.com> writes:
    >> 
    >>> In my opinion, if it can't be fixed then the changes should be
    >>> removed. Surely, we cannot have an org-mode that knowingly
    >>> exports/publishes something that causes a validation error!
    >> 
    >> Looking at the error message, the fix might be very simple:
    >> 
    >> The most common cause of this error is unencoded ampersands in
    >> URLs as described by the WDG in "Ampersands in URLs".

    > Hmm, perhaps it is as simple as encoding the ampersand.  That
    > won't work with inline javascript, if I'm understanding correctly
    > the link I gave in my reply.  *But*, despite being embedded
    > javascript, the librejs magnet link happens within a comment, so I
    > suppose it really just comes down to how librejs treats it.
    > Poking around in librejs (525e3a5), it seems it is clever and will
    > s/&amp;/&/:

    >     // Match by link on first parameter (legacy) if
    > (licenses[key]["Magnet link"] === first.replace("&amp;","&") ||
    > licenses[key]["URL"] === first.replace("&amp;","&")) { return
    > [true, `Recognized license: "${licenses[key]['Name']}".`];

    > Colin, could you try exporting with the change below and see if
    > that resolves your validation issues?  It'd also be great if you
    > could check whether librejs is still happy after that change.

I've now applied the patch to my local org-mode and exported the
file "index.org". It seems to work because I now get "This document was
successfully checked as XHTML 1.0 Strict!" when validating by uploading
the html file "index.html" to http://validator.w3.org/.

As to librejs, I don't actually use js on my web-site so apart from the
js license information in the html header there is no js on the
page. However, I've opened my "index.html" file with icecat (v52.3.0 32
bit) and get the following output:

---------- Begin -----

List of accepted JavaScript in file:///home/redknight/public_html/index.html

    LibreJS did not allow the execution of any scripts on this page: '
        There may be no scripts on this page (check source, C-u)
        The inline and on-page JavaScript code may not be free and/or
        may not have proper license information and external scripts (if
        present) may have been removed by default. 
        External scripts may not be free and/or may not have proper
        licensing and are not part of the whitelist of free JavaScript
        libraries. 

List of blocked JavaScript in file:///home/redknight/public_html/index.html

    LibreJS did not block any scripts on this page:
        There may be no scripts on this page (check source, C-u).
        All the scripts on this page may be trivial and/or free.
        You may have whitelisted this domain name or url from the
        preferences (Type about:addons in your location bar to check) 
        You may have clicked the "allow all scripts" button, which
        causes LibreJS to load all JavaScript on a page regardless of
        whether it is free, trivial, nontrivial or nonfree. This policy
        is effective for the entire duration of a Firefox session. 
        If for any reason you think LibreJS should have blocked
        JavaScript code on this page, please report this issue to:
        bug-librejs@gnu.org
        
---------- End -----

I hope this helps - and thanks

Best wishes,

-- 
Colin Baxter
www.Colin-Baxter.com

Re: ox.html causes w3c xhtml validation

[Kyle Meyer]

Hi Colin,

Colin Baxter <m43cap@yandex.com> writes:

>>>>>> Kyle Meyer <kyle@kyleam.com> writes:

>     > Colin, could you try exporting with the change below and see if
>     > that resolves your validation issues?  It'd also be great if you
>     > could check whether librejs is still happy after that change.
>
> I've now applied the patch to my local org-mode and exported the
> file "index.org". It seems to work because I now get "This document was
> successfully checked as XHTML 1.0 Strict!" when validating by uploading
> the html file "index.html" to http://validator.w3.org/.
>
> As to librejs, I don't actually use js on my web-site so apart from the
> js license information in the html header there is no js on the
> page. However, I've opened my "index.html" file with icecat (v52.3.0 32
> bit) and get the following output:

Thanks for reporting back.  Arne sent an equivalent patch (see sibling
thread), verifying that LibreJS worked with it.  I applied that in
661696036 (ox-html: escape & in license magnets, 2020-03-15), so we
should be all set.

Re: ox.html causes w3c xhtml validation

[Colin Baxter]

Dear Kyle,
>>>>> Kyle Meyer <kyle@kyleam.com> writes:

    > Thanks for reporting back.  Arne sent an equivalent patch (see
    > sibling thread), verifying that LibreJS worked with it.  I applied
    > that in 661696036 (ox-html: escape & in license magnets,
    > 2020-03-15), so we should be all set.

I've done a fresh git-pull of org-mode and all works well again. Thank
you and Arne for your work in resolving the issue.

Best wishes,

Colin.