[BUG] The documentation webserver gives 404s [9.5.4 (release_9.5.4-3-g6dc785 @ <censored>)]

[BUG] The documentation webserver gives 404s [9.5.4 (release_9.5.4-3-g6dc785 @ <censored>)]

[Rudolf Adamkovič]

Hello again!

I noticed that the Org documentation server gives 404 Not Found for a
large number of links published all over the internet because (1) it
parses URLs case-sensitively and (2) the case has changed at some point.
I stumble upon such 404s errors daily.  See, for instance, the link to
Column Groups documentation in this Stack Overflow answer:

https://stackoverflow.com/a/8570307/1306956.

I know that Org does not compete for popularity, but for the people
getting familiar with it, this issue might cause a lot of headaches for
no good reason.

Rudy

Emacs : GNU Emacs 29.0.50 (build 20, aarch64-apple-darwin21.5.0, NS
 appkit-2113.50 Version 12.4 (Build 21F79)) of 2022-07-02 Package: Org
 mode version 9.5.4 (release_9.5.4-3-g6dc785 @ <censored>)
-- 
"Logic is a science of the necessary laws of thought, without which no
employment of the understanding and the reason takes place."
-- Immanuel Kant, 1785

Rudolf Adamkovič <salutis@me.com> [he/him]
Studenohorská 25
84103 Bratislava
Slovakia

Re: [BUG] The documentation webserver gives 404s [9.5.4 (release_9.5.4-3-g6dc785 @ <censored>)]

[Ihor Radchenko]

Rudolf Adamkovič <salutis@me.com> writes:

> I noticed that the Org documentation server gives 404 Not Found for a
> large number of links published all over the internet because (1) it
> parses URLs case-sensitively and (2) the case has changed at some point.
> I stumble upon such 404s errors daily.  See, for instance, the link to
> Column Groups documentation in this Stack Overflow answer:
>
> https://stackoverflow.com/a/8570307/1306956.
>
> I know that Org does not compete for popularity, but for the people
> getting familiar with it, this issue might cause a lot of headaches for
> no good reason.

Thanks for the heads up!
I am CCing Tim Cross. He is currently doing some work on our server. He
is more likely to know how to fix this.

Best,
Ihor

Re: [BUG] The documentation webserver gives 404s [9.5.4 (release_9.5.4-3-g6dc785 @ <censored>)]

[Tim Cross]

Ihor Radchenko <yantar92@gmail.com> writes:

> Rudolf Adamkovič <salutis@me.com> writes:
>
>> I noticed that the Org documentation server gives 404 Not Found for a
>> large number of links published all over the internet because (1) it
>> parses URLs case-sensitively and (2) the case has changed at some point.
>> I stumble upon such 404s errors daily.  See, for instance, the link to
>> Column Groups documentation in this Stack Overflow answer:
>>
>> https://stackoverflow.com/a/8570307/1306956.
>>
>> I know that Org does not compete for popularity, but for the people
>> getting familiar with it, this issue might cause a lot of headaches for
>> no good reason.
>
> Thanks for the heads up!
> I am CCing Tim Cross. He is currently doing some work on our server. He
> is more likely to know how to fix this.
>

Well I'm not sure what is the correct 'fix'. As far as I am aware, the
file part of URI are case sensitive and always have been. The web server
does not have a simple configuration switch to turn on case insensitive
URLs. If we want the web server to treat them as case insensitive, then
we have to add costly regular expression rewrite rules to the URLs sent to the
server. We would also need to verify this won't break anything - it is
possible some links depend on the server treating URLs as case
sensitive. 

It isn't clear to me what the chain of events have been here. Is it that
we changed the case on our server or is it simply people have put out
link which arfe wrong?

Is this the result of some change in the way the HTML manual is
generated or a change in how org generates HTML or is this just the
result of people posting the wrong URL?

BTW this is not something I have the access to change as it would
involve changes to the actual web server configuration. It isn't
something which can be 'fixed' at the Org level. 

Re: [BUG] The documentation webserver gives 404s [9.5.4 (release_9.5.4-3-g6dc785 @ <censored>)]

[Ihor Radchenko]

Tim Cross <theophilusx@gmail.com> writes:

>>> I noticed that the Org documentation server gives 404 Not Found for a
>>> large number of links published all over the internet because (1) it
>>> parses URLs case-sensitively and (2) the case has changed at some point.
>>> I stumble upon such 404s errors daily.  See, for instance, the link to
>>> Column Groups documentation in this Stack Overflow answer:
>>>
>>> https://stackoverflow.com/a/8570307/1306956.
>> ...
>> Thanks for the heads up!
>> I am CCing Tim Cross. He is currently doing some work on our server. He
>> is more likely to know how to fix this.
>
> Well I'm not sure what is the correct 'fix'. As far as I am aware, the
> file part of URI are case sensitive and always have been. The web server
> does not have a simple configuration switch to turn on case insensitive
> URLs. If we want the web server to treat them as case insensitive, then
> we have to add costly regular expression rewrite rules to the URLs sent to the
> server. We would also need to verify this won't break anything - it is
> possible some links depend on the server treating URLs as case
> sensitive. 

> ...
> BTW this is not something I have the access to change as it would
> involve changes to the actual web server configuration. It isn't
> something which can be 'fixed' at the Org level. 

Then, I am also CCing Bastien.
As an idea, the "fix" can be creating symlinks to alternative file
names. But we first need to figure out the difference between old/new
naming schemes.

> It isn't clear to me what the chain of events have been here. Is it that
> we changed the case on our server or is it simply people have put out
> link which arfe wrong?
>
> Is this the result of some change in the way the HTML manual is
> generated or a change in how org generates HTML or is this just the
> result of people posting the wrong URL?

I have a suspicion that it might have something to do with texinfo
versions. AFAIU, Org manual pages are generated using makeinfo and it is
makeinfo that it responsible for the file name generation. (see
https://www.gnu.org/software/texinfo/manual/texinfo/texinfo.html#HTML-Splitting)
Probably, the naming scheme or some configuration variables have changed
in different versions of texinfo.

Best,
Ihor

Re: [BUG] The documentation webserver gives 404s [9.5.4 (release_9.5.4-3-g6dc785 @ <censored>)]

[Max Nikulin]

> 
>>>> I noticed that the Org documentation server gives 404 Not Found for a
>>>> large number of links published all over the internet because (1) it
>>>> parses URLs case-sensitively and (2) the case has changed at some point.
>>>> I stumble upon such 404s errors daily.  See, for instance, the link to
>>>> Column Groups documentation in this Stack Overflow answer:
>>>>
>>>> https://stackoverflow.com/a/8570307/1306956.

On 03/07/2022 21:01, Ihor Radchenko wrote:
> 
> Then, I am also CCing Bastien.
> As an idea, the "fix" can be creating symlinks to alternative file
> names. But we first need to figure out the difference between old/new
> naming schemes.

Previous discussion of the issue:

Ori. Re: Possible bug report: URL capitalization in online manual. Sun, 
9 Feb 2020 13:49:04 -0500
https://list.orgmode.org/CACyCJqX3xrJsctFEmTNUp6HwzGE1M83KGnc9atYt_LUpgk69Bw@mail.gmail.com/

Re: [BUG] The documentation webserver gives 404s [9.5.4 (release_9.5.4-3-g6dc785 @ <censored>)]

[Tim Cross]

Max Nikulin <manikulin@gmail.com> writes:

>> 
>>>>> I noticed that the Org documentation server gives 404 Not Found for a
>>>>> large number of links published all over the internet because (1) it
>>>>> parses URLs case-sensitively and (2) the case has changed at some point.
>>>>> I stumble upon such 404s errors daily.  See, for instance, the link to
>>>>> Column Groups documentation in this Stack Overflow answer:
>>>>>
>>>>> https://stackoverflow.com/a/8570307/1306956.
>
> On 03/07/2022 21:01, Ihor Radchenko wrote:
>> Then, I am also CCing Bastien.
>> As an idea, the "fix" can be creating symlinks to alternative file
>> names. But we first need to figure out the difference between old/new
>> naming schemes.
>
> Previous discussion of the issue:
>
> Ori. Re: Possible bug report: URL capitalization in online manual. Sun, 9 Feb 2020
> 13:49:04 -0500
> https://list.orgmode.org/CACyCJqX3xrJsctFEmTNUp6HwzGE1M83KGnc9atYt_LUpgk69Bw@mail.gmail.com/


Thanks Max, that helped fill in some of the blanks and confirms some of
what I assumed.

Seems that the change to letter case was a deliberate change, as was
removal of the old pages without upper case letters following the
initial uppercase letter in the filename.

Making the server case insensitive is not straight-forward. Essentially,
we would need to add a regular expression based rewrite rule OR we would
need to add copies or perhaps setup symbolic links. While all possible,
it will add to maintenance overhead and will likely be a source for
future bugs.

I do have an alternative suggestion which may help. Given that the
'broken' URLs are actually from external links to old documentation
which has been removed, what we could do is create a more informative
404 page. Once users are on the 'real' site, the case issue does not
exist. It is only a problem due to outdated URLs on external sites like
stack overflow. Instead of just saying 404 Not Found, the page could say
the requested URL was not found and is likely a link to old outdated doc
documentation. The page could include a link to the main orgmode
page. This would be a fairly simple 'fix' that would improve user
experience to some degree. 

Re: [BUG] The documentation webserver gives 404s [9.5.4 (release_9.5.4-3-g6dc785 @ <censored>)]

[Ihor Radchenko]

Tim Cross <theophilusx@gmail.com> writes:

> I do have an alternative suggestion which may help. Given that the
> 'broken' URLs are actually from external links to old documentation
> which has been removed, what we could do is create a more informative
> 404 page. Once users are on the 'real' site, the case issue does not
> exist. It is only a problem due to outdated URLs on external sites like
> stack overflow. Instead of just saying 404 Not Found, the page could say
> the requested URL was not found and is likely a link to old outdated doc
> documentation. The page could include a link to the main orgmode
> page. This would be a fairly simple 'fix' that would improve user
> experience to some degree. 

This sounds like a good idea.
I am not sure if it is feasible, but the 404 page could also provide
suggestions based on similar existing links. I have something like
https://list.orgmode.org/orgmode/83tu89b7pr.fsf@gnu.org/ in mind.

Best,
Ihor

Re: [BUG] The documentation webserver gives 404s [9.5.4 (release_9.5.4-3-g6dc785 @ <censored>)]

[Tim Cross]

Ihor Radchenko <yantar92@gmail.com> writes:

> Tim Cross <theophilusx@gmail.com> writes:
>
>> I do have an alternative suggestion which may help. Given that the
>> 'broken' URLs are actually from external links to old documentation
>> which has been removed, what we could do is create a more informative
>> 404 page. Once users are on the 'real' site, the case issue does not
>> exist. It is only a problem due to outdated URLs on external sites like
>> stack overflow. Instead of just saying 404 Not Found, the page could say
>> the requested URL was not found and is likely a link to old outdated doc
>> documentation. The page could include a link to the main orgmode
>> page. This would be a fairly simple 'fix' that would improve user
>> experience to some degree. 
>
> This sounds like a good idea.
> I am not sure if it is feasible, but the 404 page could also provide
> suggestions based on similar existing links. I have something like
> https://list.orgmode.org/orgmode/83tu89b7pr.fsf@gnu.org/ in mind.
>

Sorry, I don't understand the relevance of that link (it seems to be to
a discussion about GC size?).

Yes, you should be able to use JS to examine the URL which failed and
transform it by making the first character of each word in the url
filename upper case. Essentially do what was proposed with the rewrite
rules, but which wouldn't have the same performance hit as it would only
be triggered when incorrect URLs were entered and wold run in the browser.

The only downside is it wouldn't work with non JS based browsers (like
eww). If the server had support for server side scripting (perl, ruby,
etc) it could be done so that it would work with all browsers by doing
the rendering server side.

Re: [BUG] The documentation webserver gives 404s [9.5.4 (release_9.5.4-3-g6dc785 @ <censored>)]

[Ihor Radchenko]

Tim Cross <theophilusx@gmail.com> writes:

>> This sounds like a good idea.
>> I am not sure if it is feasible, but the 404 page could also provide
>> suggestions based on similar existing links. I have something like
>> https://list.orgmode.org/orgmode/83tu89b7pr.fsf@gnu.org/ in mind.
>
> Sorry, I don't understand the relevance of that link (it seems to be to
> a discussion about GC size?).

The link I provided is pointing to a thread, which is not in Org ML.
Yet, list.orgmode.org is able to detect messages in other mailing lists
:)

For me, the page in the link looks like:

Message-ID: <83tu89b7pr.fsf@gnu.org>
found in another inbox:

https://yhetil.org/emacs-devel/83tu89b7pr.fsf@gnu.org/

Perhaps try an external site:

https://marc.info/?i=83tu89b7pr.fsf@gnu.org
https://www.mail-archive.com/search?l=mid&q=83tu89b7pr.fsf@gnu.org
nntp://news.gmane.io/83tu89b7pr.fsf@gnu.org
https://lists.debian.org/msgid-search/83tu89b7pr.fsf@gnu.org
https://docs.FreeBSD.org/cgi/mid.cgi?db=mid&id=83tu89b7pr.fsf@gnu.org
https://www.w3.org/mid/83tu89b7pr.fsf@gnu.org
http://www.postgresql.org/message-id/83tu89b7pr.fsf@gnu.org
https://lists.debconf.org/cgi-lurker/keyword.cgi?doc-url=/lurker&format=en.html&query=id:83tu89b7pr.fsf@gnu.org

> Yes, you should be able to use JS to examine the URL which failed and
> transform it by making the first character of each word in the url
> filename upper case. Essentially do what was proposed with the rewrite
> rules, but which wouldn't have the same performance hit as it would only
> be triggered when incorrect URLs were entered and wold run in the browser.
>
> The only downside is it wouldn't work with non JS based browsers (like
> eww). If the server had support for server side scripting (perl, ruby,
> etc) it could be done so that it would work with all browsers by doing
> the rendering server side.

If JS is the only option it is still better to have it and fall back to
more generic page if JS is not supported. Of course, given that the JS
code is not going to be too complex and not going to require too much
maintenance.

Best,
Ihor

Re: [BUG] The documentation webserver gives 404s [9.5.4 (release_9.5.4-3-g6dc785 @ <censored>)]

[Max Nikulin]

On 04/07/2022 19:58, Ihor Radchenko wrote:
> 
> If JS is the only option it is still better to have it and fall back to
> more generic page if JS is not supported. Of course, given that the JS
> code is not going to be too complex and not going to require too much
> maintenance.

It is possible to perform a kind of fuzzy matching on the server, but I 
am unsure if Bastien wouldn't mind a dynamic 404 page.

The manual and the short guide may have their own 404 pages explaining 
that some page is not available any more and offering table of contents.

However it seems, Bastien earlier configured a set of rewrite rules 
mapping old file names with more lower case letters to new ones. In my 
opinion it is the best option and it should be restored. List of files 
may be committed to git (either Org or site) to detect changes later and 
to add new mappings when some file disappears.

Re: [BUG] The documentation webserver gives 404s [9.5.4 (release_9.5.4-3-g6dc785 @ <censored>)]

[Tim Cross]

Max Nikulin <manikulin@gmail.com> writes:

>
> However it seems, Bastien earlier configured a set of rewrite rules mapping old file names
> with more lower case letters to new ones. In my opinion it is the best option and it
> should be restored. List of files may be committed to git (either Org or site) to detect
> changes later and to add new mappings when some file disappears.


Are you sure about that. There is nothing along these lines in the nginx
config file that I can see. Also, my reading following that thread you
provided earlier was that Bastien thought the overhead to manage such
lists was too high?

I guess we need to wait for Bastien to indicate his preference. I find
things a little challenging as there seems to be a fair bit of
outdated/conflicting information scattered through various files/notes. 

At this point, my vote is to just do a basic updated 404 page that
points to the index.html page for orgmode.org so that users have
something to follow and not a big ugly 404 Not Found. The rest we can
consider later.

Re: [BUG] The documentation webserver gives 404s [9.5.4 (release_9.5.4-3-g6dc785 @ <censored>)]

[Max Nikulin]

On 05/07/2022 05:37, Tim Cross wrote:
> Max Nikulin writes:
>>
>> However it seems, Bastien earlier configured a set of rewrite rules mapping old file names
>> with more lower case letters to new ones. In my opinion it is the best option and it
>> should be restored. List of files may be committed to git (either Org or site) to detect
>> changes later and to add new mappings when some file disappears.
> 
> 
> Are you sure about that. There is nothing along these lines in the nginx
> config file that I can see. Also, my reading following that thread you
> provided earlier was that Bastien thought the overhead to manage such
> lists was too high?

Bastien. Re: Possible bug report: URL capitalization in online manual. 
Mon, 10 Feb 2020 07:48:21 +0100 
https://list.orgmode.org/874kvzdjka.fsf@gnu.org/
> I've now installed the rewrite rules on the server and all these links
> are redirecting correctly. Given the amount of Org documentation links
> living out there, that's really a good idea.

I did not check it that time, so I can not be really sure. The rules 
might be lost during migration to deployment using SourceHut. My 
expectation is "301 Moved Permanently" redirection for obsolete file names.

I do not think, a hundred of redirection rules gives significant 
overhead. I attributed Bastien's "too much" to maintaining documentation 
for several versions of Org. A part of original problem (if I get it 
correctly) was mix of old files survived from earlier versions of the 
manual and current ones. Certainly removing of outdated files was a 
proper step.

Tim Cross:
> At this point, my vote is to just do a basic updated 404 page that
> points to the index.html page for orgmode.org

For the manual and for the guide directories I still consider links to 
the table of contents (or even full table of contents) on 404 pages as a 
better variant.

Re: [BUG] The documentation webserver gives 404s [9.5.4 (release_9.5.4-3-g6dc785 @ <censored>)]

[Tim Cross]

Max Nikulin <manikulin@gmail.com> writes:

> On 05/07/2022 05:37, Tim Cross wrote:
>> Max Nikulin writes:
>>>
>>> However it seems, Bastien earlier configured a set of rewrite rules mapping old file names
>>> with more lower case letters to new ones. In my opinion it is the best option and it
>>> should be restored. List of files may be committed to git (either Org or site) to detect
>>> changes later and to add new mappings when some file disappears.
>> Are you sure about that. There is nothing along these lines in the nginx
>> config file that I can see. Also, my reading following that thread you
>> provided earlier was that Bastien thought the overhead to manage such
>> lists was too high?
>
> Bastien. Re: Possible bug report: URL capitalization in online manual. Mon, 10 Feb 2020
> 07:48:21 +0100 https://list.orgmode.org/874kvzdjka.fsf@gnu.org/
>> I've now installed the rewrite rules on the server and all these links
>> are redirecting correctly. Given the amount of Org documentation links
>> living out there, that's really a good idea.
>
> I did not check it that time, so I can not be really sure. The rules might be lost during
> migration to deployment using SourceHut. My expectation is "301 Moved Permanently"
> redirection for obsolete file names.
>
> I do not think, a hundred of redirection rules gives significant overhead. I attributed
> Bastien's "too much" to maintaining documentation for several versions of Org. A part of
> original problem (if I get it correctly) was mix of old files survived from earlier
> versions of the manual and current ones. Certainly removing of outdated files was a proper
> step.
>
> Tim Cross:
>> At this point, my vote is to just do a basic updated 404 page that
>> points to the index.html page for orgmode.org
>
> For the manual and for the guide directories I still consider links to the table of
> contents (or even full table of contents) on 404 pages as a better variant.

I don't really care what specific page is linked to - just that we keep
it simple and have a link for an initial quick fix. Only reason I
mentioned the index.html page is that it has links to everything - worg,
git, manual, mail list etc. It is possible that people get to 404 via
links other than to the manual. 

If you or someone else wants to do something more sophisticated, I'd say
go for it.  In the meantime, I will see if I can put together a default
404 page. However, I think only Bastien has the necessary access to
install it.

Re: [BUG] The documentation webserver gives 404s [9.5.4 (release_9.5.4-3-g6dc785 @ <censored>)]

[Max Nikulin]

On 05/07/2022 09:54, Tim Cross wrote:
> Max Nikulin writes:
>> Bastien. Re: Possible bug report: URL capitalization in online manual. Mon, 10 Feb 2020
>> 07:48:21 +0100 https://list.orgmode.org/874kvzdjka.fsf@gnu.org/
>>> I've now installed the rewrite rules on the server and all these links
>>> are redirecting correctly. Given the amount of Org documentation links
>>> living out there, that's really a good idea.

Tim, is there any chance that Bastien forgot to send you some included 
file for nginx configuration? However, the site ignores this file even 
it exists somewhere on filesystem.

>> Tim Cross:
>>> At this point, my vote is to just do a basic updated 404 page that
>>> points to the index.html page for orgmode.org
>>
>> For the manual and for the guide directories I still consider links to the table of
>> contents (or even full table of contents) on 404 pages as a better variant.
> 
> I don't really care what specific page is linked to - just that we keep
> it simple and have a link for an initial quick fix. Only reason I
> mentioned the index.html page is that it has links to everything - worg,
> git, manual, mail list etc. It is possible that people get to 404 via
> links other than to the manual.

I do not have ready to use snippet, but I am sure it should be possible 
to set specific 404 pages for particular directories like the manual and 
the guide. Of course, these pages, besides links to table of contents, 
may have another link to index.html or a navigation header shared with 
other pages on the site.

Re: [BUG] The documentation webserver gives 404s [9.5.4 (release_9.5.4-3-g6dc785 @ <censored>)]

[Tim Cross]

Max Nikulin <manikulin@gmail.com> writes:

> On 05/07/2022 09:54, Tim Cross wrote:
>> Max Nikulin writes:
>>> Bastien. Re: Possible bug report: URL capitalization in online manual. Mon, 10 Feb 2020
>>> 07:48:21 +0100 https://list.orgmode.org/874kvzdjka.fsf@gnu.org/
>>>> I've now installed the rewrite rules on the server and all these links
>>>> are redirecting correctly. Given the amount of Org documentation links
>>>> living out there, that's really a good idea.
>
> Tim, is there any chance that Bastien forgot to send you some included file for nginx
> configuration? However, the site ignores this file even it exists somewhere on filesystem.
>

anything is possible. In a 'standard' nginx configuration, you would
typically have all the config related to s specific site in one
file. However, that is just convention, not a rule. 

>>> Tim Cross:
>>>> At this point, my vote is to just do a basic updated 404 page that
>>>> points to the index.html page for orgmode.org
>>>
>>> For the manual and for the guide directories I still consider links to the table of
>>> contents (or even full table of contents) on 404 pages as a better variant.
>> I don't really care what specific page is linked to - just that we keep
>> it simple and have a link for an initial quick fix. Only reason I
>> mentioned the index.html page is that it has links to everything - worg,
>> git, manual, mail list etc. It is possible that people get to 404 via
>> links other than to the manual.
>
> I do not have ready to use snippet, but I am sure it should be possible to set specific
> 404 pages for particular directories like the manual and the guide. Of course, these
> pages, besides links to table of contents, may have another link to index.html or a
> navigation header shared with other pages on the site.

Many things are possible. However, I'm trying very hard not to be overly
distracted by this issue, which I think is fairly insignificant. My
objective is to do as little as possible at this time because I want to
get back to cleanup of worg. It is quite possible that work will also
have some influence on the rest of the org web space. 

If you or others want to look at the 404 page issue, please do so. For
right now, I'll just do a basic 404 page which Bastien can install
easily. If you or others come up with something more sophisticated, it
can be used to replace the simple basic version, otherwise we can
revisit this later if necessary. 

Re: [BUG] The documentation webserver gives 404s [9.5.4 (release_9.5.4-3-g6dc785 @ <censored>)]

[Ihor Radchenko]

Tim Cross <theophilusx@gmail.com> writes:

> If you or others want to look at the 404 page issue, please do so. For
> right now, I'll just do a basic 404 page which Bastien can install
> easily. If you or others come up with something more sophisticated, it
> can be used to replace the simple basic version, otherwise we can
> revisit this later if necessary. 

+1
Having a simple 404 page is already an improvement.
There is no urgent need to prioritize further complexities just yet.
At least, there is no need to distract Tim from doing much more
important job of other WORG improvements.

If someone else is willing to allocate time and look into this issue,
feel free to reply here. We can then arrange access to the relevant
configs.

Best,
Ihor

Re: [BUG] The documentation webserver gives 404s [9.5.4 (release_9.5.4-3-g6dc785 @ <censored>)]

[Bastien]

Sorry I'm late to fixing this.

I reinstalled the rewrite rules on the server and I configured nginx
to use the errors pages that Tim already created.

We should be covered for most problems, but let me know if we need to
do more.

Thanks!

Ihor Radchenko <yantar92@gmail.com> writes:

> If someone else is willing to allocate time and look into this issue,
> feel free to reply here. We can then arrange access to the relevant
> configs.

-- 
 Bastien