multipage html output

multipage html output

[Orm Finnendahl]

Hi,

 after my clunky publishing chain from org to gitbook with multipage
page output broke down recently I finally decided to tackle adding an
export backend for multipage html output to org-export.

It is done now and mainly working. The backend uses all the
funcionality of the ox html exporter, only slightly modifying the code
in places where it is necessary for multipage output. In addition I
tried to make it as general, as possible to enable adding other
multipage backends (like for md output) easily.

Before sharing it I thought it might be a good idea to think about
integrating it properly/officially into org. I would be willing to
provide the code, docs, patches, etc.

There are a couple of decisions to make (should it be integrated as an
option into the html output backend or should it be a separate backend
altogether?  What options concerning footnotes, toc, etc. should be
provided?  etc...) and this mail is basically asking about how to
proceed.

My questions:

- Is there widespread interest to fully integrate it into org mode?

- If so, whom should I contact, or is it expected that I just go ahead
  and supply merge requests?

I'm a bit hesitant putting in the extra work of fully integrating it
without approval by the maintainers to go ahead.

In case someone wants to take a peek at the current state of the code
you can check out my github repository here:

https://github.com/ormf/ox-html-multipage

Be aware and warned that the code is in constant flux, not finalized
and there still are some open questions for me what would be the best
way to integrate the code into the old export engine, like whether
adding optional args to the transcoding functions or using properties
in the info channel, etc... Once it is finalized, the current single
page html export will work exactly as before (it already does, but
while checking it out I am modifying the html templates for the
multipage navigation, toc, etc.)

Hope to hear from you, especially if the maintainers are reading this.

--
Orm

Re: multipage html output

[Dr. Arne Babenhauserheide]

[-- Attachment #1: Type: text/plain, Size: 440 bytes --]

Orm Finnendahl <orm.finnendahl@selma.hfmdk-frankfurt.de> writes:

> https://github.com/ormf/ox-html-multipage

Do I understand it right, that this exports a single org file into
multiple HTML files in the html subfolder?

In the interest of making it possible to build upon the code, can you
make the license GPL v2.0 *or later*?

Best wishes,
Arne
-- 
Unpolitisch sein
heißt politisch sein,
ohne es zu merken.
draketo.de

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 1125 bytes --]

Re: multipage html output

[Christian Moe]

Orm Finnendahl writes:

> Hi,
>
>  after my clunky publishing chain from org to gitbook with multipage
> page output broke down recently I finally decided to tackle adding an
> export backend for multipage html output to org-export.
>
> (... snip ...)
>
> - Is there widespread interest to fully integrate it into org mode?

It would be nice to have.

Conceptually, I'd see it as fitting into org-publish, perhaps, rather
than as an exporter? With org-publish-project-alist as a convenient
place to set up various options?

Yours,
Christian

Re: multipage html output

[Ihor Radchenko]

Christian Moe <mail@christianmoe.com> writes:

>>  after my clunky publishing chain from org to gitbook with multipage
>> page output broke down recently I finally decided to tackle adding an
>> export backend for multipage html output to org-export.
>>
>> (... snip ...)
>>
>> - Is there widespread interest to fully integrate it into org mode?
>
> It would be nice to have.
>
> Conceptually, I'd see it as fitting into org-publish, perhaps, rather
> than as an exporter? With org-publish-project-alist as a convenient
> place to set up various options?

Not really. ox-publish is more about exporting multiple input
.org/non-.org files into outputs.

I'd rather see this kind of feature being a part of ox.el - an option to
export one .org to many smaller files. Currently, we only have an option
to export one .org (or part of it) to a single string/file. (And then,
ox-odt has to try various kludges to make things work as expected with
.odt, which consist of multiple files under the hood).

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: multipage html output

[Christian Moe]

Ihor Radchenko writes:

> Christian Moe <mail@christianmoe.com> writes:
>
>>>  after my clunky publishing chain from org to gitbook with multipage
>>> page output broke down recently I finally decided to tackle adding an
>>> export backend for multipage html output to org-export.
>>>
>>> (... snip ...)
>>>
>>> - Is there widespread interest to fully integrate it into org mode?
>>
>> It would be nice to have.
>>
>> Conceptually, I'd see it as fitting into org-publish, perhaps, rather
>> than as an exporter? With org-publish-project-alist as a convenient
>> place to set up various options?
>
> Not really. ox-publish is more about exporting multiple input
> .org/non-.org files into outputs.

I was thinking in terms of purpose: organizing export of multiple
outputs to be published together. It does that with multiple inputs
because, as you say, one-to-one export is the option we currently have.

> I'd rather see this kind of feature being a part of ox.el - an option to
> export one .org to many smaller files. Currently, we only have an option
> to export one .org (or part of it) to a single string/file. (And then,
> ox-odt has to try various kludges to make things work as expected with
> .odt, which consist of multiple files under the hood).

Yes, I suppose the code for multipage export belongs on the ox.el
level. And then one would want to be able to use it out of the box
without necessarily having to configure a publishing project, just
relying on sensible defaults. So I take that back.

(There might be some considerations for ox-publish when using
multipage/chunked export *inside* a publishing project, e.g. regarding
which levels of output to include in a sitemap, but that's for another
day.)

Yours,
Christian

Re: multipage html output

[Rudolf Adamkovič]

Orm Finnendahl <orm.finnendahl@selma.hfmdk-frankfurt.de> writes:

> - Is there widespread interest to fully integrate it into org mode?

Definitely. :)

Rudy
-- 
"It is no paradox to say that in our most theoretical moods we may be
nearest to our most practical applications."  --- Alfred North
Whitehead, 1861-1947

Rudolf Adamkovič <rudolf@adamkovic.org> [he/him]
http://adamkovic.org

Re: multipage html output

[Orm Finnendahl]

Hi,

Am Mittwoch, den 03. Juli 2024 um 11:05:39 Uhr (+0000) schrieb Ihor
Radchenko:
> 
> Not really. ox-publish is more about exporting multiple input
> .org/non-.org files into outputs.
> 
> I'd rather see this kind of feature being a part of ox.el - an option to
> export one .org to many smaller files. Currently, we only have an option
> to export one .org (or part of it) to a single string/file. (And then,
> ox-odt has to try various kludges to make things work as expected with
> .odt, which consist of multiple files under the hood).

 that is/was my intention: Basically there was only a very small
change to ox.el necessary to make it work (it's mentioned in the
comment on top of ox-multipage-html in my github repository):

Currently `org-export-as' combines parsing the org document into a
global parse tree with all additional options applied and serializing
that into the final output target format. My code simply splits the
code sections of these tasks into two separate functions, which are
called by org-export-as, `org-export--collect-tree-info' and
`org-export--transcode-headline'. The advantage of this approach is
that it is fully compatible with the prior code, but gives the
necessary flexibility to the backend export code to split up the
global parse tree before serializing.

The multipage html backend (ox-html-multipage.el) takes care of
generating the global parse tree with org-export--headline, divides
that tree into the subtrees of the individual pages, then calls the
serializing function for each of the subtrees and writes the results
to file. Is that along the lines of what you meant?

In the meantime I thought about the proposed backend. Maybe it's a
good idea to integrate the single page *and* the multipage backend
into one backend altogether: The Backend *always* produces multipage
output, but you can define the level at which the pages are split with
an #+OPTION: in the org file. Setting the default level to 0 if the
option is not set will generate the exact same output as the old
backend without breaking anything for anybody. I'm quite sure it'll
work and as I said it's mainly done and wouldn't require a lot of
work.

What do you think?

--
Orm

Re: multipage html output

[Ihor Radchenko]

Orm Finnendahl <orm.finnendahl@selma.hfmdk-frankfurt.de> writes:

>> I'd rather see this kind of feature being a part of ox.el - an option to
>> export one .org to many smaller files. Currently, we only have an option
>> to export one .org (or part of it) to a single string/file. (And then,
>> ox-odt has to try various kludges to make things work as expected with
>> .odt, which consist of multiple files under the hood).
>
>  that is/was my intention: Basically there was only a very small
> change to ox.el necessary to make it work (it's mentioned in the
> comment on top of ox-multipage-html in my github repository):
>
> Currently `org-export-as' combines parsing the org document into a
> global parse tree with all additional options applied and serializing
> that into the final output target format. My code simply splits the
> code sections of these tasks into two separate functions, which are
> called by org-export-as, `org-export--collect-tree-info' and
> `org-export--transcode-headline'. The advantage of this approach is
> that it is fully compatible with the prior code, but gives the
> necessary flexibility to the backend export code to split up the
> global parse tree before serializing.

This makes sense.

Although, multipage export may imply two different things:
1. An ability to produce multiple pages from parts of the original Org
   file.
2. An ability to produce multiple pages from a single part of Org file.
   For example, consider an Org document with images exported to
   ODT. The images should be stored alongside XML content file and
   referenced from there. So, export produces multiple files from the
   same document/subtree.
   
Your approach only addresses (1), but not (2).

That said, even having (1) is a welcome improvement.

> The multipage html backend (ox-html-multipage.el) takes care of
> generating the global parse tree with org-export--headline, divides
> that tree into the subtrees of the individual pages, then calls the
> serializing function for each of the subtrees and writes the results
> to file. Is that along the lines of what you meant?

Yes, but we also need to carefully discuss the rules how the full parse
tree is separated into subtrees. Your proof of concept code hard-codes
these rules.

> In the meantime I thought about the proposed backend. Maybe it's a
> good idea to integrate the single page *and* the multipage backend
> into one backend altogether: The Backend *always* produces multipage
> output, but you can define the level at which the pages are split with
> an #+OPTION: in the org file. Setting the default level to 0 if the
> option is not set will generate the exact same output as the old
> backend without breaking anything for anybody. I'm quite sure it'll
> work and as I said it's mainly done and wouldn't require a lot of
> work.

1. Most of the existing backends are written to produce a single
   page. So, our design of ox.el part should be able to handle
   those. What you proposed (calling the same backend on pre-split parse
   tree) sounds good in this context.

2. Some backends, as you proposed, may target multipage export from the
   very beginning. So, we need to provide some way for the backend (in
   org-export-define*-backend) to specify that it wants to split the
   original parse tree. I imagine some kind of option with default
   values configured via backend, but optionally overwritten by user
   settings/in-buffer keywords.

3. Your suggestion to add a new export option for splitting based on
   headline level is one idea.

   Another idea is to split out subtrees with :EXPORT_FILE_NAME:
   property.

4. One possible extra feature might be exporting only a part of the
   original Org file to separate pages. Say, only pages with specific
   tag. The whole original Org file is also exported, replacing the
   split-out parts with, for example, links. This will generalize
   "index" pages from ox-publish.

5. We need to consider the rules used to generate export file names.
   Currently, we choose between :EXPORT_FILE_NAME: property,
   #+EXPORT_FILE_NAME: keyword, and the original file name.

   As I see in your code, you also introduced deriving file name from
   the headline title.

6. I can see people flipping between exporting the whole document and
   multipage document. We probably need some kind of easy switch in M-x
   org-export-dispatch to choose how to export.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: multipage html output

[Orm Finnendahl]

Hi Ihor,

 thanks for your time to study the code and your very valuable input,
much appreciated!

Am Donnerstag, den 04. Juli 2024 um 11:41:35 Uhr (+0000) schrieb Ihor
Radchenko:
>
> 2. An ability to produce multiple pages from a single part of Org file.
>    For example, consider an Org document with images exported to
>    ODT. The images should be stored alongside XML content file and
>    referenced from there. So, export produces multiple files from the
>    same document/subtree.
>    
> Your approach only addresses (1), but not (2).

Sure. I'm not at all familiar with the peculiarities of other output
backends, but see your point. If you can give any hints or have any
ideas *how* we could find general rules for separating the subtrees,
which cover foreseeable use cases, or devise a flexible mechanism for
doing so, I'd be glad to help setting them up and implementing them. I
definitely agree, the code should be as general as possible while
providing complete backward compatibility.

> 1. Most of the existing backends are written to produce a single
>    page. So, our design of ox.el part should be able to handle
>    those. What you proposed (calling the same backend on pre-split parse
>    tree) sounds good in this context.

Ok.

> 2. Some backends, as you proposed, may target multipage export from the
>    very beginning. So, we need to provide some way for the backend (in
>    org-export-define*-backend) to specify that it wants to split the
>    original parse tree. I imagine some kind of option with default
>    values configured via backend, but optionally overwritten by user
>    settings/in-buffer keywords.

I'll look into that and maybe I can come up with something. I was
hesitant to propose anything as I tried to stay as limited as possible
and not get too deep into changing things. If you have suggestions,
please let me know.

> 3. Your suggestion to add a new export option for splitting based on
>    headline level is one idea.
> 
>    Another idea is to split out subtrees with :EXPORT_FILE_NAME:
>    property.

I'm not sure I fully understand what you mean: Do you mean specifying
different :EXPORT_FILE_NAME: properties throughout the same document
and then export accordingly?

> 4. One possible extra feature might be exporting only a part of the
>    original Org file to separate pages. Say, only pages with specific
>    tag. The whole original Org file is also exported, replacing the
>    split-out parts with, for example, links. This will generalize
>    "index" pages from ox-publish.

Very nice idea! MAybe along these lines is that I thought about
"Master" org files which combine different documentations by linking
to them in some sort of top menu which is included on every page of
all these documentations and then being able to generate a single
documentation without having to recompile everything. But for now I'd
prefer to first get it working and then think about such extensions (I
have more ideas for different extensions and "plugins" which could be
useful). It shouldn't be too hard to implement at a later point and
probably also wouldn't need a complete rewrite.

> 5. We need to consider the rules used to generate export file names.
>    Currently, we choose between :EXPORT_FILE_NAME: property,
>    #+EXPORT_FILE_NAME: keyword, and the original file name.
> 
>    As I see in your code, you also introduced deriving file name from
>    the headline title.

Exactly. I wanted to make sure, the file names are sorted correctly,
are unique and the title is relatable to the section it names on the
directory level. I also thought about making it user-configurable, but
first wanted to implement a working solution.

> 6. I can see people flipping between exporting the whole document and
>    multipage document. We probably need some kind of easy switch in M-x
>    org-export-dispatch to choose how to export.

Sure, that is the disadvantage of my proposal to make everything a
"multipage" document. Another disadvantage is that when the user
chooses to open the final document or display it in a buffer the user
can't choose whether to only open/display one page or every exported
page. In most circumstances it should be advisable to just
open/display the first page. We can also just add a switch between
single-page and multipage, with multipage always just exporting to
file, but that also has disadvantages.

As the code I proposed is encapsulated in the html backend and not
spreading all over the place, I will now first go ahead to finalize
the existing code to a fully working setup. ASFAICT adapting that to
other needs shouldn't require a complete rewrite. And I might be
around for a while ;-)

--
Orm

Re: multipage html output

[Ihor Radchenko]

Orm Finnendahl <orm.finnendahl@selma.hfmdk-frankfurt.de> writes:

> Sure. I'm not at all familiar with the peculiarities of other output
> backends, but see your point. If you can give any hints or have any
> ideas *how* we could find general rules for separating the subtrees,
> which cover foreseeable use cases, or devise a flexible mechanism for
> doing so, I'd be glad to help setting them up and implementing them. I
> definitely agree, the code should be as general as possible while
> providing complete backward compatibility.

I think that the easiest would be adding a new option to
`org-export-options-alist' - it is already extendable for individual
backends and allows users to tweak things via in-buffer keywords,
properties, variables, and export options.

The most generic rule would be some kind of function that takes AST
node as input and returns whether that node should be going to a separate
file or not, and if yes, tell (1) which export backend to use to export
that subtree to a file (may as well allow exporting to different
formats, while we are at it); (2) what are the export parameters to be
used for that export, (possibly) including the file path.

Then, in addition to the most generic (and most flexible) "rule being an
Elisp function", we can allow some simplified semantics to define rules.

The semantics should probably give a couple of toggles to customize:
(1) which subtrees are selected for export; (2) which export backend is
used (3) how their file names are generated; (4) (optional) how they are
represented when exporting the whole original file; e.g. whether to put
links to exported files in place of their subtrees; (5) (optional) how
the original file is represented in the exported subtrees; e.g. whether
to put backlink to parent file

The subtree selection may boil down to the usual TAGS matcher (or
function), as described in "11.3.3 Matching tags and properties" section
of the manual. This will cover the previously discussed separation based
on headline level, a tag, or a property.

The export backend selection may be realized by allowing multiple rules
with each rule defining selection/backend/file name/....

In terms of the value semantics in Elisp, I am thinking about something
re-using backend definition format:

(setq org-export-pages
      '(:selector "LEVEL=2+blog+TODO=DONE"
        :backend html
         ;; completely remove the exported subtree is original document
         ;; is being exported.
        :page-transcoder nil
         ;; or :page-transcoder #'org-export-page-as-heading-with-link
        :export-file-name "%{TITLE}-%{page-number}" ;; or some other kind of template syntax
        )

       '(:selector a-function-accepting-ast-node
         :source-backend any 
         :backend
         (:parent html ;; `org-export-define-derived-backend'-like semantics
          :options-alist
          ;; Do not export private headings in HTML pages.
          ((:exclude-tags "EXCLUDE_TAGS" nil (cons "private" org-export-exclude-tags) split))))

        '(:selector "+export_ascii_page"
          :source-backend html ; only use this rule when exporting to html
          :backend
          (:parent ascii
           ((template .
              (lambda (contents info)
                (format "Paged out from %s\n%s"
                   (plist-get
                     ;; INFO channel for parent document
                     (plist-get info :page-source)
                     :title)
                   (org-ascii-template contents info)))))))))

>> 2. Some backends, as you proposed, may target multipage export from the
>>    very beginning. So, we need to provide some way for the backend (in
>>    org-export-define*-backend) to specify that it wants to split the
>>    original parse tree. I imagine some kind of option with default
>>    values configured via backend, but optionally overwritten by user
>>    settings/in-buffer keywords.
>
> I'll look into that and maybe I can come up with something. I was
> hesitant to propose anything as I tried to stay as limited as possible
> and not get too deep into changing things. If you have suggestions,
> please let me know.

One way could be simply adding an option like :selector above to the
backend definition. Then, it will be used as default selector:

(setq org-export-pages
  (:selector default :backend html) ; export pages to html with default selector
)

or even

(setq org-export-pages
  (:backend html) ; export pages to html with default selector
)

or just

;; export using the same target backend as selected in the export menu
(setq org-export-pages t)
;; (setq org-export-pages nil) - existing single page export
;; (setq org-export-pages 'only-pages) - only export pages, ignore original file

>> 3. Your suggestion to add a new export option for splitting based on
>>    headline level is one idea.
>> 
>>    Another idea is to split out subtrees with :EXPORT_FILE_NAME:
>>    property.
>
> I'm not sure I fully understand what you mean: Do you mean specifying
> different :EXPORT_FILE_NAME: properties throughout the same document
> and then export accordingly?

Yes. It is re-using the existing idea with subtree export

13.2 Export Settings

‘EXPORT_FILE_NAME’
     The name of the output file to be generated.  Otherwise, Org
     generates the file name based on the buffer name and the extension
     based on the backend format.

If a subtree has that property set, it is used as output file name.
Since there is usually no reason to set this property unless you also
want to export subtree to individual file, it makes sense to use this as
selector for what to export as pages.

Example:

#+TITLE: Index document

* Emacs notes
** Emacs blog post #1
:PROPERTIES:
:EXPORT_FILE_NAME: my-first-post
:END:
...
** Fleeting note at [2024-06-20 Thu 22:16]
Some notes, no need to export them.

* Personal notes
** Personal blog post #1
:PROPERTIES:
:EXPORT_FILE_NAME: private/personal-post-trial
:END:
...

>> 6. I can see people flipping between exporting the whole document and
>>    multipage document. We probably need some kind of easy switch in M-x
>>    org-export-dispatch to choose how to export.
>
> Sure, that is the disadvantage of my proposal to make everything a
> "multipage" document. Another disadvantage is that when the user
> chooses to open the final document or display it in a buffer the user
> can't choose whether to only open/display one page or every exported
> page. In most circumstances it should be advisable to just
> open/display the first page. We can also just add a switch between
> single-page and multipage, with multipage always just exporting to
> file, but that also has disadvantages.

What to open is a minor detail, really. It can be worked out any moment
we need to. The most sensible default, IMHO, it to open dired with the
containing directory with all the exported pages.

> As the code I proposed is encapsulated in the html backend and not
> spreading all over the place, I will now first go ahead to finalize
> the existing code to a fully working setup. ASFAICT adapting that to
> other needs shouldn't require a complete rewrite. And I might be
> around for a while ;-)

I advice against doing this.
While reading your code, I saw that you used some html-specific
functions for modifications in ox.el. If you start by modifying ox.el in
Org git repo directly, simply doing "make compile" will warn about
instances of using functions not defined in ox.el.
Another advantage of editing the ox.el and using Org repository is that
you can run "make test" any time and see if you managed to break Org :)

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>