ox-publish: Some starting problems

ox-publish: Some starting problems

[c.buhtz]

Hello together,

I was able to generate HTML files of my org-roam (v2) nodes. But I have 
some problems and need to be pointed into the right direction.

1.
Is this mailing list the right place to discuss ox-publish related 
topics? The ox-publish.el only contains two e-mail addresses?

2.
Because ox-publish is skipping "unmodified files" I deleted all HTML 
files between the publish-runs just to be sure that there are no side 
effects. But now when my output directory is totally empty ox-publish 
still keeps saying that it is skipping the unmodified files. So nothing 
is generated anymore. How can I handle that? Maybe resetting the 
"unmodified" flag somewhere? Or shouldn't ox-publish not only check for 
modifications but if there is something earlier generated content exist?

3.
I use (setq org-export-with-broken-links t) with and without 
":with-broken-links "mark"" to prevend ox-publish stopping when there 
are broken links. I swear and I also checked that there are only a few 
of them. But in the HTML output all links are gone. No links. No text 
for the links.
I tried to reproduce this in a minimal example with two new nodes. But 
for them the links are generated.
The links (of all nodes!) are working very well inside of org-roam.
Any idea how could go on with diagnosis here?
First I thought ox-publish has a problem with org-roam-v2 id-links. But 
the two new created nodes just for testing where working well. The older 
nodes (just 2 months old) also using id-links and where created with the 
same Emacs and org-roam version.

4.
ox-publish ignores newlines. Two short lines in the org-file becomes one 
line in the HTML content.

5.
I am not totally new to Emacs but quit fresh. So I have two questions 
about how to handle the documentation and help informations here.
a) I tried to "descrbie" the mode "ox-publish". But Emacs only offers me 
an "describe-mode" which describe all currently existing nodes. There is 
nothing like "describe-mode MODENAME". How can I handle that?
b) I another thread I was pointed to 
https://orgmode.org/manual/Publishing-options.html . There is no 
description about the options they are only listed. This does not help 
me. Maybe there are some useful options that could solve some of my 
problems. But without description I am not able to decide or know.

6.
Is there a way to integrated backlinks into output?

Re: ox-publish: Some starting problems

[c.buhtz]

Some additional information.

Am 09.03.2022 09:55 schrieb c.buhtz@posteo.jp:
> 2.
> Because ox-publish is skipping "unmodified files"
> [..]
> nothing is generated anymore.

Doing this forces ox-publish to generated everything
     M: (org-publish "projectname" t)


> 3.
> [..]
> in the HTML output all links are gone. No links. No text
> for the links.
> I tried to reproduce this in a minimal example with two new nodes. But
> for them the links are generated.
> [..]

I need to correct myself here. After forcing new generation of HTML 
content the links in the new test nodes also gone.
I am confused. I need to do further investigation here and create a 
minimal reproducible setup to find out the steps to reproduce this 
behavior.
Should I then open a bug report? Do I understand 
https://orgmode.org/worg/org-issues.html correct that there is no 
"usual" bug tracker but just he mailing list where I have to post bugs?

> 4.
> ox-publish ignores newlines. Two short lines in the org-file becomes
> one line in the HTML content.

I am still not sure but it looks like that this does fix that problem.
     (setq org-export-preserve-breaks t)

Re: ox-publish: Some starting problems

[Max Nikulin]

On 09/03/2022 16:41, c.buhtz@posteo.jp wrote:
> Should I then open a bug report? Do I understand 
> https://orgmode.org/worg/org-issues.html correct that there is no 
> "usual" bug tracker but just he mailing list where I have to post bugs?

There is a kind of dashboard at https://updates.orgmode.org Someone need 
to confirm the bug report before it will appear there.

Accordingly to https://orgmode.org/org.html#Feedback
you may use
    M-x org-submit-bug-report <RET>
or just to send description of the problem to this list.

There is a subpackage for Org Mode at https://debbugs.gnu.org, but the 
message will be anyway redirected to this mail list. Bugs reported 
against Emacs instead of Org Mode have greater chance to remain 
unnoticed for some period of time. Answers sent to the mail list 
directly instead of the bug email address will not appear in debbugs 
archives. Notice the following as well:

https://list.orgmode.org/87v9kavoms.fsf@gnu.org/
Bastien. Re: issue tracker? Mon, 01 Jun 2020 16:36:59 +0200
 > I'd like to keep this mailing list as the central place to discuss
 > everything about Org, including bug reports.

Re: ox-publish: Some starting problems

[Max Nikulin]

On 09/03/2022 15:55, c.buhtz wrote:
> 
> 1.
> Is this mailing list the right place to discuss ox-publish related 
> topics? The ox-publish.el only contains two e-mail addresses?

ox-publish.el belongs to the "core" part of Org mode, so this mail list 
is an appropriate place. There is a chance to get response from the Org 
maintainers. I do not follow stackexchange and reddit topics so I can 
tell nothing concerning probability to get some answer there.

> 3.
> I use (setq org-export-with-broken-links t) with and without 
> ":with-broken-links "mark"" to prevend ox-publish stopping when there 
> are broken links. I swear and I also checked that there are only a few 
> of them. But in the HTML output all links are gone. No links. No text 
> for the links.

If you insist on setq than try
    (setq org-export-with-broken-links 'mark)
without :with-broken-links. You can get correct value using easy 
customization interface. It does not matter for 
`org-export-with-broken-links', but some custom variables have :set 
property, so the following may be generally better
    (custom-set-variables
     '(org-export-with-broken-links 'mark))

When specifying the publish project option, do not add quotes around 
"mark", the string (unlike the symbol) is considered as t. Unfortunately 
no warning is issued.

> I tried to reproduce this in a minimal example with two new nodes. But 
> for them the links are generated.

It seems, changing project options or global variables does not lead to 
updating of the files if the sources have not modified.

> First I thought ox-publish has a problem with org-roam-v2 id-links. But 
> the two new created nodes just for testing where working well. The older 
> nodes (just 2 months old) also using id-links and where created with the 
> same Emacs and org-roam version.

There is a known problem with id links. They may be broken if they lead 
to another file:
inkbottle. org-id with ox-html. Sat, 14 Aug 2021 00:28:35 +0200
https://list.orgmode.org/4617246.m1MCmUpgFQ@pluto/

> 4.
> ox-publish ignores newlines. Two short lines in the org-file becomes one 
> line in the HTML content.

I suppose, it is unrelated to ox-publish, it is behavior of ox 
(org-export) and org-element parser. Paragraphs are separated by empty 
lines. Likely you need some markup, e.g.
    #+begin_example
    ...
    #+end_example
block.

> 5.
> I am not totally new to Emacs but quit fresh. So I have two questions 
> about how to handle the documentation and help informations here.
> a) I tried to "descrbie" the mode "ox-publish". But Emacs only offers me 
> an "describe-mode" which describe all currently existing nodes. There is 
> nothing like "describe-mode MODENAME". How can I handle that?

org-mode is a mode, ox-publish maybe a package (actually unsure). 
Unfortunately e.g. `describe-package' C-h P org shows built-in version 
of org for me even though git main HEAD is actually loaded. It extracts 
some comments from the top of the source file.

> b) I another thread I was pointed to 
> https://orgmode.org/manual/Publishing-options.html . There is no 
> description about the options they are only listed. This does not help 
> me. Maybe there are some useful options that could solve some of my 
> problems. But without description I am not able to decide or know.

That page specifies related global variable, so you can try 
`describe-variable' C-h v with the related name.

I do not know if you are familiar with GNU "info" 
https://www.gnu.org/software/emacs/manual/html_node/info/ that allows to 
read e.g. Org Manual in Emacs. There is a variable index 
https://orgmode.org/manual/Variable-Index.html but this case the entry 
leads to the "Export Settings" section already known to you.

Sometimes it is more convenient to search in the manual variant exported 
as a single HTML page https://orgmode.org/org.html

The last resort is reading source files.

> 6.
> Is there a way to integrated backlinks into output?

Do you mean something specific to org-roam?

Re: ox-publish: Some starting problems

[c.buhtz]

Dear Max,

thank's a lot for your help and your patience with a newbie.

Am 09.03.2022 16:32 schrieb Max Nikulin:
>> 3.
>> I use (setq org-export-with-broken-links t) with and without 
>> ":with-broken-links "mark"" to prevend ox-publish stopping when there 
>> are broken links. I swear and I also checked that there are only a few 
>> of them. But in the HTML output all links are gone. No links. No text 
>> for the links.
> 
> If you insist on setq than try
>    (setq org-export-with-broken-links 'mark)
> without :with-broken-links. You can get correct value using easy
> customization interface. It does not matter for
> `org-export-with-broken-links', but some custom variables have :set
> property, so the following may be generally better
>    (custom-set-variables
>     '(org-export-with-broken-links 'mark))

Do you mean that (setq org-export-with-broken-links 'mark) is the same 
as :with-broken-links mark? This are just two different ways to set the 
same thing?
How do I know as a newbie? ;)

Why using custom-set variables here? Is there something wrong with just 
doing
(org-export-with-broken-links t)
?

>> I tried to reproduce this in a minimal example with two new nodes. But 
>> for them the links are generated.
> 
> It seems, changing project options or global variables does not lead
> to updating of the files if the sources have not modified.

I do not understand. Do you a see a solution for the problem?

> There is a known problem with id links. They may be broken if they
> lead to another file:
> inkbottle. org-id with ox-html. Sat, 14 Aug 2021 00:28:35 +0200
> https://list.orgmode.org/4617246.m1MCmUpgFQ@pluto/

Great to know. Is there a way to monitor (subscribe) to the "dashboard"? 
A bugtracker where I can not track a bug is not a bugtracker. :D
OK, bugs going to the mailing list. But am I allowed to attach extra 
files (zip) to my mails to the mailinglist? Some bug reports need 
attached files. But I learned and agree that it is rude to attach files 
to a mailing list.

Back to the bug.
The bug-thread is hard for me to understand because of a lot of internal 
details. What is the current essence? Currently there is no way to use 
ox-publish with org-roam-v2 generated org files which use IDs as links?
But as I told sometimes the links are generated.

Btw: Also in my minimal working example I was not able to surly 
reproduce the problem. Sometimes it works sometimes not. I also deleted 
~/.emacs.d/org-roam.db, ~/.org-timestamps and ~/.emacs-d/.orgid-indexes 
(I forget the correct name for the latter).

> org-mode is a mode, ox-publish maybe a package (actually unsure).
I thought it was a separate thing because of "(require ox-publish)". 
Again: How do I as a newbie know?

> That page specifies related global variable, so you can try
> `describe-variable' C-h v with the related name.

I know that and I did. But this is waste of time when I have to do this 
for each of the variable in that big list.

>> 6.
>> Is there a way to integrated backlinks into output?
> 
> Do you mean something specific to org-roam?

I am not sure but maybe yes. ;)
I can do C-c n l which invokes org-roam-buffer-toogle and shows me an 
extra buffer with a list of all nodes (org-files?) which are linking to 
the current open node.

Re: ox-publish: Some starting problems

[chris]

[-- Attachment #1: Type: text/plain, Size: 3634 bytes --]

Hi Christian,
I've received the following in my personal email box, and I think it is related to this thread, 
so I paste it here:
On Thursday, 10 March 2022 06:54:22 CET c.buhtz@posteo.jp wrote:
> Hello together,
> 
> I do not understand all details. But it seems to me that currently
> ox-publish is not fully capable to generate a linked together HTML
> files out of org-roam-v2 generated org files because of the ID-linking
> of org-roam-v2.
> 
> In my tests it sometimes work and sometimes not. I was not able to
> reproduce this. While writing to the list about that problem I was
> pointed to that bug report here.
> 
> Kind
> Christian
And I think that I understand the question, and the general setting related to it.

So, on the one hand you have `org-mode/org-id` links like `[[id:32ba80a3-1982-4f43-becb-
b0e346a91b0d][hello]]`, which seem the thing to do in the 21st century, the most sensible 
thing to do IMO, I only use those, `org-roam` only use those.

When with `emacs` the links are followed/resolved through a database. For example, the 
database can be updated through functions like `org-id-update-id-locations`, part of 
https://github.com/tkf/org-mode/blob/master/lisp/org-id.el[1].

However the use of them is controversial because some deem them as not human friendly 
enough. So I suppose guidelines should be defined for anything to happen, but I'm not 
sure people would agree on what to do. And internal ID links, resolved locally through 
reading in a database, should be thoroughly translated to something accurate and 
consistent at export-time, and that seems a lot.

I myself use `org-roam` and `ID`; considering the difficulties of exporting to html through 
emacs/org-mode, I just gave up.
Thanks,
Chris
PS: I think you are doing an awesome job at trying to have all that working.


On Wednesday, 9 March 2022 17:39:46 CET c.buhtz@posteo.jp wrote:
> Dear Max,
> 
> thank's a lot for your help and your patience with a newbie.
> 
> Am 09.03.2022 16:32 schrieb Max Nikulin:
> >> 3.
> >> I use (setq org-export-with-broken-links t) with and without
> >> ":with-broken-links "mark"" to prevend ox-publish stopping when there
> >> are broken links. I swear and I also checked that there are only a few
> >> of them. But in the HTML output all links are gone. No links. No text
> >> for the links.
> > 
> > If you insist on setq than try
> > 
> >    (setq org-export-with-broken-links 'mark)
> > 
> > without :with-broken-links. You can get correct value using easy
> > customization interface. It does not matter for
> > `org-export-with-broken-links', but some custom variables have :set
> > property, so the following may be generally better
> > 
> >    (custom-set-variables
> >    
> >     '(org-export-with-broken-links 'mark))
> 
> Do you mean that (setq org-export-with-broken-links 'mark) is the same
> as :with-broken-links mark? This are just two different ways to set the
> same thing?
> How do I know as a newbie? ;)
> 
> Why using custom-set variables here? Is there something wrong with just
> doing
> (org-export-with-broken-links t)
> ?
> 
> >> I tried to reproduce this in a minimal example with two new nodes. But
> >> for them the links are generated.
> > 
> > It seems, changing project options or global variables does not lead
> > to updating of the files if the sources have not modified.
> 
> I do not understand. Do you a see a solution for the problem?
> 
> > There is a known problem with id links. They may be broken if they
> > lead to another file:
> > inkbottle. org-id with ox-html. Sat, 14 Aug 2021 00:28:35 +0200
> > https://list.orgmode.org/4617246.m1MCmUpgFQ@pluto/

[-- Attachment #2: Type: text/html, Size: 14494 bytes --]

Re: ox-publish: Some starting problems

[Max Nikulin]

On 09/03/2022 23:39, c.buhtz@posteo.jp wrote:
> Am 09.03.2022 16:32 schrieb Max Nikulin:
> 
> Do you mean that (setq org-export-with-broken-links 'mark) is the same 
> as :with-broken-links mark? This are just two different ways to set the 
> same thing?
> How do I know as a newbie? ;)

I suggested you to have a look at the `org-export-options-alist' 
definition in ox.el in another thread. It is reasonable to have a 
per-file setting (that you do not like), a per-project one, and a global 
variable that determines behavior when neither project nor file specify 
desired value.

I recommended to set only one of them (at the beginning) to avoid 
confusion due to e.g. typos when it is uncertain which value is really 
active.

> Why using custom-set variables here? Is there something wrong with just 
> doing
> (org-export-with-broken-links t)

Likely you mean

    (setq org-export-with-broken-links t)

You were trying to set incorrect value with quotes around "mark". Easy 
customization interface helps to avoid such problems. There are custom 
variables with :set attribute that may behave incorrectly with `setq'.

>> There is a known problem with id links. They may be broken if they
>> lead to another file:
>> inkbottle. org-id with ox-html. Sat, 14 Aug 2021 00:28:35 +0200
>> https://list.orgmode.org/4617246.m1MCmUpgFQ@pluto/
> 
> Back to the bug.
> The bug-thread is hard for me to understand because of a lot of internal 
> details. What is the current essence? Currently there is no way to use 
> ox-publish with org-roam-v2 generated org files which use IDs as links?
> But as I told sometimes the links are generated.

I can say nothing concerning org-roam. I can reproduce the problem with 
pure org-publish, but I have not tried to find a workaround. I am using 
CUSTOM_ID links with some mnemonics in anchors (and I am aware that such 
links become broken when a heading is refiled to another file).

> Btw: Also in my minimal working example I was not able to surly 
> reproduce the problem. Sometimes it works sometimes not. I also deleted 
> ~/.emacs.d/org-roam.db, ~/.org-timestamps and ~/.emacs-d/.orgid-indexes 
> (I forget the correct name for the latter).

Org-publish regenerates files only for the sources that were modified 
(by default). Unchanged files may have links to the updated ones. 
Originally ox (org-export) assigns random anchors, but org-publish 
caches them in some file to avoid broken links after partial updates. It 
may affect behavior for consecutive runs.

>> org-mode is a mode, ox-publish maybe a package (actually unsure).
> I thought it was a separate thing because of "(require ox-publish)". 
> Again: How do I as a newbie know?

org-mode as a major mode consists of a lot of packages. See e.g. 
https://www.gnu.org/software/emacs/manual/html_node/emacs/Modes.html

>> That page specifies related global variable, so you can try
>> `describe-variable' C-h v with the related name.
> 
> I know that and I did. But this is waste of time when I have to do this 
> for each of the variable in that big list.

When you are reading info "(org) Publishing options" 
https://orgmode.org/manual/Publishing-options.html, it is assumed that 
you are already familiar with info "(org) Export Settings" 
https://orgmode.org/manual/Export-Settings.html earlier section. When 
you are reading the manual inside Emacs, you can hit C-h v having cursor 
on the variable name. The only issue is that the package should be 
loaded (M-x load-library RET ox RET).

>>> 6.
>>> Is there a way to integrated backlinks into output?
>>
>> Do you mean something specific to org-roam?
> 
> I am not sure but maybe yes. ;)
> I can do C-c n l which invokes org-roam-buffer-toogle and shows me an 
> extra buffer with a list of all nodes (org-files?) which are linking to 
> the current open node.

It is better to discuss with the org-roam community or the developers.

Re: ox-publish: Some starting problems

[Nick Dokos]

[Answers to some of the questions inline]

c.buhtz@posteo.jp writes:

> 1.  Is this mailing list the right place to discuss ox-publish
> related topics? The ox-publish.el only contains two e-mail
> addresses?
>
Yes.

> 2.  Because ox-publish is skipping "unmodified files" I deleted all
> HTML files between the publish-runs just to be sure that there are
> no side effects. But now when my output directory is totally empty
> ox-publish still keeps saying that it is skipping the unmodified
> files. So nothing is generated anymore. How can I handle that? Maybe
> resetting the "unmodified" flag somewhere? Or shouldn't ox-publish
> not only check for modifications but if there is something earlier
> generated content exist?
>
Org mode records the time when a file was published and compares it to
the last modification time to decide whether to publish or not. It
does not have a dependency mechanism like `make' to figure things
out. As you found out, you can force it by calling it with a `t'
argument for the `force' parameter. You can do this interactively with
`C-u M-x org-publish'. See the "Triggering Publication" section of the
manual:

  (info "(org)Triggering Publication")

That section also points you to the variable
`org-publish-use-timestamps-flag': if you set that to `nil', no
timestamp checking is done.

The timestamps are kept in the directory named by the variable
`org-publish-timestamp-directory' whose default value is
"~/.org-timestamps/".

> 4.  ox-publish ignores newlines. Two short lines in the org-file
> becomes one line in the HTML content.
>

That is how HTML export (and publishing) works: the browser does its
own layout of HTML content and you have no say in that (except for
`verbatim' things that are pre-formatted).

> 5.  I am not totally new to Emacs but quit fresh. So I have two
> questions about how to handle the documentation and help
> informations here.  a) I tried to "descrbie" the mode
> "ox-publish". But Emacs only offers me an "describe-mode" which
> describe all currently existing nodes. There is nothing like
> "describe-mode MODENAME". How can I handle that?

There is no such mode: `ox-publish' is (part of) the name of the file
that contains the code that is used for publishing - in particular
it contains the code of the function `org-publish'. You can get
the doc string of the function with `C-h f org-publish'.

> b) I another thread I was pointed to
> https://orgmode.org/manual/Publishing-options.html . There is no
> description about the options they are only listed. This does not
> help me. Maybe there are some useful options that could solve some
> of my problems. But without description I am not able to decide or
> know.
>

Read the first paragraph of that section carefully:

,----
| The property list can be used to set many export options for the HTML
| and LaTeX exporters.  In most cases, these properties correspond to
| user variables in Org.  The table below lists these properties along
| with the variable they belong to.  See the documentation string for
| the respective variable for details.
`----

So to find out e.g. what the option ‘:archived-trees’ does, you have
to ask for the doc string of the variable ‘org-export-with-archived-trees’.
You can do that with `C-h v org-export-with-archived-trees' or more easily
by putting the cursor on the name and typing `C-h v RET`.

-- 
Nick

"There are only two hard problems in computer science: cache
invalidation, naming things, and off-by-one errors." -Martin Fowler