From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2.migadu.com ([2001:41d0:700:3204::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms8.migadu.com with LMTPS id aBfYNOgvjGXe0QAAkFu2QA (envelope-from ) for ; Wed, 27 Dec 2023 15:08:40 +0100 Received: from aspmx1.migadu.com ([2001:41d0:303:e224::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2.migadu.com with LMTPS id WCD0LugvjGWuKAEAe85BDQ (envelope-from ) for ; Wed, 27 Dec 2023 15:08:40 +0100 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=posteo.net header.s=2017 header.b=Uqpbgv7b; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=posteo.net ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1703686120; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=XnjBPxk+bf1AUbKKP8le9oFmMUmLlKkhEDt2B08JwpM=; b=I19CIv49E40rxyYEh6Niaus9ZNKCthDb3tqL/OWibD2gHcxJEsrEMJCYZF5HLpwwYXtZJf sm/m2MLWNVZgeS7XYPwMFwIFzXLjdmOSatkz445dbfnLb/pVkChCcw7GeMcLUcmXE8yf7i lgjr6lmFif9iwlvbrnwJn00mQiMQO9j7ytOx79CiGPF/EueBSx3p1aqRw4VYryi2UoRRRY cnjhLNLLQnX1HU+xjoqzt81qj8IUWaq2km7IlRzwPeAn7kDJOK/L16f4PXStudAJfP8lIR r9A8snhUtduc+zQMXRY32Jp7nuFgvHjKsTITR8yIi5gH1CPPVaPoC51K/f/kfQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1703686120; a=rsa-sha256; cv=none; b=lpZyfEIn8zhkF2KTIQvosEdIw9IDykVHTXVOPcqv85z3GR55jOZO5x7KdOdSP8BlWvdcDB /HK4vGnIjIs1LPlY8ItrNN4w04PFp0ewIkq8a8aWgYnzC/ZHohpS4NO5byUN9chDvwAmSK xHY9s4L/ZEX7x8pOnRSzZVh4zJOduazGyvD1sIS9IV6cIBvRZUqgoVfYLz9+kH0Z61vL8l ehazBUDw1ZaCVu9psETGl8SlrhpTYI9hJ7GfcQi2sy3TzSlZt7Pb3mIQylSxKLKYvXa6xB pr9ZXzngadJEDiQtFpyGSXUthgJRHCCTDPIVINE3/d9g72wpsXlVV2QyIkBbow== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=posteo.net header.s=2017 header.b=Uqpbgv7b; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=posteo.net Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 8C4E81924B for ; Wed, 27 Dec 2023 15:08:40 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1rIU9M-0007kg-1d; Wed, 27 Dec 2023 08:40:44 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rIU9K-0007kP-7D for emacs-orgmode@gnu.org; Wed, 27 Dec 2023 08:40:42 -0500 Received: from mout02.posteo.de ([185.67.36.66]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1rIU9H-0004GI-Cs for emacs-orgmode@gnu.org; Wed, 27 Dec 2023 08:40:41 -0500 Received: from submission (posteo.de [185.67.36.169]) by mout02.posteo.de (Postfix) with ESMTPS id 1C3BF240101 for ; Wed, 27 Dec 2023 14:40:35 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1703684436; bh=zVKl/XU4kgoEUGSgwrZ4uk19zrBHAebCANeiWce1Sso=; h=From:To:Subject:Date:Message-ID:MIME-Version:From; b=Uqpbgv7bJx5vWwPDmruhNvDgsWKmh6HjH3UM0gI2q5GoZJkMdsrZu1NCapBFPoaT3 1IZZ8ZHSHSYBojnQH5+GCsI8oVHW5IbXdgzOR/aYxk8LiaVYsTt87t1br7mkBqYe+G Rz0lmgJrve0k9EUsZG89P8fTQ+C8M9yEMt7hwfMA8Mbn9hWQ1ZwBepOAuk/uEnprm+ e80HoVq63xlDRbeZTfFuwQtk3gu1UaRfGebdTd8/U0HH6Etg6cBb3dD+ZGdsbPDqC0 KwHpYfA4oZcfrbmY62n79hbfaEHpMmJzHJptv7qNXe+Y1fG58LnolKtDp7co7bp0lK flbvyaFlb572g== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4T0XpC1nG3z9rxD; Wed, 27 Dec 2023 14:40:35 +0100 (CET) From: Ihor Radchenko To: emacs-orgmode@gnu.org, Thomas S. Dye , Karthik Chikmagalur , Matt Subject: [PATCH v2] org-manual: Describe export process flow In-Reply-To: <87wmt1dp1c.fsf@localhost> References: <87wmt1dp1c.fsf@localhost> Date: Wed, 27 Dec 2023 13:43:42 +0000 Message-ID: <87le9fep69.fsf@localhost> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Received-SPF: pass client-ip=185.67.36.66; envelope-from=yantar92@posteo.net; helo=mout02.posteo.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H5=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: emacs-orgmode-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -9.46 X-Spam-Score: -9.46 X-Migadu-Queue-Id: 8C4E81924B X-Migadu-Scanner: mx12.migadu.com X-TUID: xxnt1sI8p50C --=-=-= Content-Type: text/plain Ihor Radchenko writes: > I'd like to add a new section to Org mode manual. > The new section will describe all the steps performed by Org export > process. This should hopefully create a more clear picture on how > various export hooks and filters are used. > > The patch is attached. > I'd appreciate feedback from people not familiar with ox.el. Thank you all for the feedback! I am attaching revised version of the patch with most of the comments addressed. I will put more detailed responses inline. "Thomas S. Dye" writes: > I'm not too familiar with ox.el. > > I edited mostly to use an active voice. I put author queries in > parentheses. I haven't paid attention to manual formatting > conventions. > > IMO, more links would likely be helpful. > > * Suggested revision > ... I believe that I have addressed everything you commented on. Karthik Chikmagalur writes: > - When exporting a sub-tree, at what stage of the export process is the > buffer narrowed to the sub-tree? I added a clarification on subtree export now. > - Are "inner" and "outer" templates described in the manual, and if they > are could you add a link to those sections when mentioning them in > this summary? I only found references to the plist properties > BEAMER_INNER_THEME etc. This is internal terminology. I changed the wording, expanding on what inner and outer template do. Matt writes: > Here are all the hooks and functions for org-export (via =C-h v > org-export--hooks TAB= and =C-h v org-export--function TAB=). I see > 59 of them. > ... > * Feedback 1: > How are the functions not present in the patch handled? - I fixed the obsolete variable names. - `org-export-stack-mode-hook' is not directly relevant to the export process - it is for asynchronous export listing buffer - Syntax-specific filters are applied according to the corresponding Org syntax element. I tried to make it more clear. - Special filters, like `org-export-before-parsing-functions' are described separately. I think I have mentioned all of them. > I would write out "src" as "source". Do we have an official way to > refer to source blocks? For example, we standardize on "Org": > https://git.savannah.gnu.org/cgit/emacs/org-mode.git/tree/doc/Documentation_Standards.org#n47 We use "source block", "code block", and "source code block" across the manual. Not "src block" though. I went with "code block" in the patch. I am not sure if it is necessary to standardize the above three terms. They are all equally understandable I believe. > Remove the trailing period or add periods to all the others. I tend > leave the period of the last sentence of a list. I'm not sure of a > style guide that recommends one or the other. Maybe someone knows > what's "right"? No idea. We do not have a consistency in the manual either. I went with ";" as Thomas suggested. > Maybe use "converted" instead of "transcoded"? I'm a native speaker > but I wonder if "converted" is a simpler word for people who aren't. "transcoded" is closer to what we use in the code. I tried to use "converted" in more general description, but still hint on the internal terminology. --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename=v2-0001-doc-org-manual.org-Fix-some-obsolete-variable-nam.patch >From efee8fb5e8aca473b1b80aacc2b38951421225cc Mon Sep 17 00:00:00 2001 Message-ID: From: Ihor Radchenko Date: Wed, 27 Dec 2023 14:23:29 +0100 Subject: [PATCH v2 1/2] doc/org-manual.org: Fix some obsolete variable names * doc/org-manual.org (Export hooks): Use the new `org-export-before-processing-functions' and `org-export-before-parsing-functions' instead of their obsolete aliases. --- doc/org-manual.org | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/doc/org-manual.org b/doc/org-manual.org index 23f250fa7..b35a83434 100644 --- a/doc/org-manual.org +++ b/doc/org-manual.org @@ -16397,12 +16397,14 @@ *** Export hooks :END: #+vindex: org-export-before-processing-hook +#+vindex: org-export-before-processing-functions #+vindex: org-export-before-parsing-hook The export process executes two hooks before the actual exporting -begins. The first hook, ~org-export-before-processing-hook~, runs -before any expansions of macros, Babel code, and include keywords in -the buffer. The second hook, ~org-export-before-parsing-hook~, runs -before the buffer is parsed. +begins. The first hook, ~org-export-before-processing-functions~, +runs before any expansions of macros, Babel code, and include keywords +in the buffer. The second hook, +~org-export-before-parsing-functions~, runs before the buffer is +parsed. Functions added to these hooks are called with a single argument: the export backend actually used, as a symbol. You may use them for @@ -16421,7 +16423,7 @@ *** Export hooks ;; the docstring of `org-map-entries' for details. (setq org-map-continue-from (point))))) -(add-hook 'org-export-before-parsing-hook #'my-headline-removal) +(add-hook 'org-export-before-parsing-functions #'my-headline-removal) #+end_src *** Filters -- 2.42.0 --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename=v2-0002-doc-org-manual.org-Describe-export-flow.patch >From 704eda63d6abf847efcd79cd97e3560ba4729921 Mon Sep 17 00:00:00 2001 Message-ID: <704eda63d6abf847efcd79cd97e3560ba4729921.1703683799.git.yantar92@posteo.net> In-Reply-To: References: From: Ihor Radchenko Date: Tue, 26 Dec 2023 15:15:23 +0100 Subject: [PATCH v2 2/2] doc/org-manual.org: Describe export flow * doc/org-manual.org (Summary of the export process): Explain how the export process is handled in Org mode. --- doc/org-manual.org | 149 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 149 insertions(+) diff --git a/doc/org-manual.org b/doc/org-manual.org index b35a83434..12e7d031a 100644 --- a/doc/org-manual.org +++ b/doc/org-manual.org @@ -16505,6 +16505,155 @@ *** Defining filters for individual files ,#+END_SRC #+end_example +*** Summary of the export process +:PROPERTIES: +:UNNUMBERED: notoc +:END: + +Org mode export is a multi-step process that works on a temporary copy +of the buffer. On high-level, the export process consists of 4 major +steps: + +1. Process the temporary copy, making necessary changes to the buffer + text; + +2. Parse the buffer, converting plain Org markup into abstract syntax + tree (AST); + +3. Convert the AST to text, as prescribed by the selected export + backend; + +4. Post-process the resulting exported text. + + +#+texinfo: @noindent +Process temporary copy of the source Org buffer [fn::Unless +otherwise specified, each step of the export process only operates on +the accessible portion of the buffer. When subtree export is selected +(see [[*The Export Dispatcher]]), the buffer is narrowed to the body of +the selected subtree, so that the rest of the buffer text, except +export keywords, does not contribute to the export output.]: + +1. Execute ~org-export-before-processing-functions~ (see [[*Export hooks]]); + +2. Expand =#+include= keywords in the whole buffer (see + [[*Include Files]]); + +3. Remove commented subtrees in the whole buffer (see [[*Comment + Lines]]); + +4. Replace macros in the whole buffer (see [[*Macro Replacement]]); + +5. When ~org-export-use-babel~ is non-nil (default), process code + blocks: + + - Leave code blocks inside archived subtrees (see [[*Internal + archiving]]) as is; + + - Evaluate all the other code blocks according to code block + headers (see [[*Limit code block evaluation]]); + + - Remove code, results of evaluation, both, or neither according + to =:exports= header argument (see [[*Exporting Code Blocks]]). + + +#+texinfo: @noindent +Parse the temporary buffer, creating AST: + +1. Execute ~org-export-before-parsing-functions~ (see [[*Export hooks]]). + The hook functions may still modify the buffer; + +2. Calculate export option values according to subtree-specific export + settings, in-buffer keywords, =#+BIND= keywords, and buffer-local + and global customization. The whole buffer is considered; + +3. Determine contributing bibliographies and record them into export + options (see [[*Citations]]). The whole buffer is considered; + +4. Execute ~org-export-filter-options-functions~; + +5. Parse the accessible portion of the temporary buffer to generate + AST. The AST is a nested list of lists representing Org syntax + elements (see [[https://orgmode.org/worg/dev/org-element-api.html][Org Element API]] for more details): + + : (org-data ... + : (heading + : (section + : (paragraph (plain-text) (bold (plain-text)))) + : (heading) + : (heading (section ...)))) + + Past this point, modifications in the temporary buffer copy no + longer affect export; Org export works only with the AST; + +6. Remove elements that will not be exported from the AST: + + - Headings according to =SELECT_TAGS= and =EXCLUDE_TAGS= export + keywords, and =task=, =inline=, =arch= export options (see + [[*Export Settings]]); + + - Comments; + + - Clocks, drawers, fixed-width environments, footnotes, LaTeX + environments and fragments, node properties, planning lines, + property drawers, statistics cookies, timestamps, timestamps, + etc according to =#+OPTIONS= keyword (see [[*Export Settings]]); + + - Table rows containing width and alignment markers (see [[*Column + Width and Alignment]]); + + - Table columns containing recalc marks (see [[*Advanced features]]). + +7. Expand environment variables in file link AST nodes, according to + the =expand-links= export option (see [[*Export Settings]]); + +8. Execute ~org-export-filter-parse-tree-functions~. These + functions can modify AST by side effect; + +9. Replace citation AST nodes and =#+print_bibliography= keyword AST + nodes as prescribed by the selected citation export processor + (see [[*Citation export processors]]). + + +#+texinfo: @noindent +Convert the AST to text by traversing the AST nodes, depth-first: + +1. Convert the leaf nodes (without children) to text as prescribed + by "transcoders" in the selected export backend + [fn:: See transcoders and ~:translate-alist~ in the docstrings + of ~org-export-define-backend~ and ~org-export-define-derived-backend~.]; + +2. Pass the converted nodes through the corresponding export + filters (see [[*Filters]]); + +3. Concatenate all the converted child nodes to produce parent + node contents; + +4. Convert the nodes with children to text, passing the nodes + themselves and their contents to the corresponding transcoders + and then to export filters (see [[*Filters]]). + + +#+texinfo: @noindent +Post-process the exported text: + + 1. Post-process the converted AST, as prescribed by the export + backend. [fn:: See ~inner-template~ in the docstring of ~org-export-define-backend~.] + This step usually adds generated content (like Table of Contents) + to the exported text; + + 2. Execute ~org-export-filter-body-functions~; + + 3. Unless body-only export is selected (see [[*The Export Dispatcher]]), + add the necessary metadata to the final document, as prescribed + by the export backend. Examples: Document author/title; HTML + headers/footers; LaTeX preamble; + + 4. Add bibliography metadata, as prescribed by the citation export + processor; + + 5. Execute ~org-export-filter-final-output-functions~. + *** Extending an existing backend :PROPERTIES: :UNNUMBERED: notoc -- 2.42.0 --=-=-= Content-Type: text/plain -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at . Support Org development at , or support my work at --=-=-=--