Re: Org Babel "swallows" table column groups

[Ihor Radchenko <yantar92@posteo.net>]

[-- Attachment #1: Type: text/plain, Size: 900 bytes --]

Ihor Radchenko <yantar92@posteo.net> writes:

>>   #+BEGIN_SRC emacs-lisp
>>   (list (list 1 2) (list "/" "<>") 'hline (list 3 4) (list 5 6))
>>   #+END_SRC
>>
>> Org Babel outputs
>> ...
>> with the second element of the list
>>
>>   (list "/" "<>")
>>
>> swallowed, without a word.
>>
>> Why would Org Babel do this?
>>
>> And, how can one output tables with column groups?
>
> This is because of how `orgtbl-to-generic' is implemented. It is taking
> pieces from the full ox.el exporter, hard-coding certain things. For
> example, it always removes special table lines:
>
>     ;; Since we are going to export using a low-level mechanism,
>     ;; ignore special column and special rows manually.

I refactored `orgtbl-to-generic', so that it does not have to duplicate
`org-export-as'. Now, things should be more consistent with the normal
export.

May you try the attached tentative patch set?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-ox-New-custom-option-to-disable-macro-replacement.patch --]
[-- Type: text/x-patch, Size: 4113 bytes --]

From f1ff68920c417343fda4c5a6450567d703ccf9b6 Mon Sep 17 00:00:00 2001
Message-ID: <f1ff68920c417343fda4c5a6450567d703ccf9b6.1720440129.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Mon, 8 Jul 2024 13:52:32 +0200
Subject: [PATCH 1/2] ox: New custom option to disable macro replacement

* lisp/ox.el (org-export-replace-macros): New custom option
controlling macro replacement.
(org-export--annotate-info): Honor it, except when processing inline
code block results and their {{{results...}}} macro.
* etc/ORG-NEWS (Allow disabling macro replacement during export):
Announce the new option.
* doc/org-manual.org (Macro Replacement):
(Summary of the export process): Document the new option.
---
 doc/org-manual.org | 11 +++++++----
 etc/ORG-NEWS       |  9 +++++++++
 lisp/ox.el         | 13 +++++++++++--
 3 files changed, 27 insertions(+), 6 deletions(-)

diff --git a/doc/org-manual.org b/doc/org-manual.org
index d30c18e0c..5f5104f91 100644
--- a/doc/org-manual.org
+++ b/doc/org-manual.org
@@ -12682,9 +12682,11 @@ ** Macro Replacement
 #+cindex: @samp{MACRO}, keyword
 
 #+vindex: org-export-global-macros
-Macros replace text snippets during export.  Macros are defined
-globally in ~org-export-global-macros~, or document-wise with the
-following syntax:
+#+vindex: org-export-replace-macros
+Macros replace text snippets during export[fn::The macro replacement
+can be disabled by setting ~org-export-replace-macros~ to nil (default
+is t).].  Macros are defined globally in ~org-export-global-macros~,
+or document-wise with the following syntax:
 
 : #+MACRO: name   replacement text; $1, $2 are arguments
 
@@ -16702,7 +16704,8 @@ *** Summary of the export process
 3. Remove commented subtrees in the whole buffer (see [[*Comment
    Lines]]);
 
-4. Replace macros in the whole buffer (see [[*Macro Replacement]]);
+4. Replace macros in the whole buffer (see [[*Macro Replacement]]),
+   unless ~org-export-replace-macros~ is nil;
 
 5. When ~org-export-use-babel~ is non-nil (default), process code
    blocks:
diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index 0c3b14128..dcd324115 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -43,6 +43,15 @@ or newer.
 # adding new customizations, or changing the interpretation of the
 # existing customizations.
 
+*** Allow disabling macro replacement during export
+
+New custom option ~org-export-replace-macros~ controls whether Org
+mode replaces macros in the buffer before export.  Set it to nil to
+disable macro replacement.
+
+This variable has no effect on the ={{{results...}}}= macros for inline
+code block results.
+
 *** Allow headline/olp target in ~org-capture-templates~ to be a function/variable
 
 The variable ~org-capture-templates~ accepts a target specification as
diff --git a/lisp/ox.el b/lisp/ox.el
index 6fa21be90..902c9f089 100644
--- a/lisp/ox.el
+++ b/lisp/ox.el
@@ -860,6 +860,14 @@ (defcustom org-export-expand-links t
   :package-version '(Org . "9.7")
   :type 'boolean)
 
+(defcustom org-export-replace-macros t
+  "When non-nil, replace macros before export.
+This variable does not affect {{{results}}} macros when processing
+code block results."
+  :group 'org-export-general
+  :package-version '(Org . "9.8")
+  :type 'boolean)
+
 (defcustom org-export-snippet-translation-alist nil
   "Alist between export snippets backends and exporter backends.
 
@@ -3048,8 +3056,9 @@ (defun org-export--annotate-info (backend info &optional subtreep visible-only e
                         (org-export-backend-name backend))
     (org-export-expand-include-keyword nil nil nil nil (plist-get info :expand-links))
     (org-export--delete-comment-trees)
-    (org-macro-initialize-templates org-export-global-macros)
-    (org-macro-replace-all org-macro-templates parsed-keywords)
+    (when org-export-replace-macros
+      (org-macro-initialize-templates org-export-global-macros)
+      (org-macro-replace-all org-macro-templates parsed-keywords))
     ;; Refresh buffer properties and radio targets after previous
     ;; potentially invasive changes.
     (org-set-regexps-and-options)
-- 
2.45.2


[-- Attachment #3: 0002-orgtbl-to-generic-Retain-special-rows-in-code-block-.patch --]
[-- Type: text/x-patch, Size: 10632 bytes --]

From 3e0a5164661153d41a89ce984c9282273c54a3de Mon Sep 17 00:00:00 2001
Message-ID: <3e0a5164661153d41a89ce984c9282273c54a3de.1720440129.git.yantar92@posteo.net>
In-Reply-To: <f1ff68920c417343fda4c5a6450567d703ccf9b6.1720440129.git.yantar92@posteo.net>
References: <f1ff68920c417343fda4c5a6450567d703ccf9b6.1720440129.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Mon, 8 Jul 2024 13:54:14 +0200
Subject: [PATCH 2/2] orgtbl-to-generic: Retain special rows in code block
 table output
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* lisp/org-table.el (orgtbl--skip):
(orgtbl--skipcols): New helper functions.
(orgtbl-to-generic): Use `org-export-as' machinery to setup table
export instead of duplicating ox.el internals.  Retain special rows in
tables when exporting to Org.  Org export is used by ob-core to format
the code block output and will thus retain special rows.
* testing/lisp/test-org-table.el (test-org-table/to-generic): Adjust
test to expect special rows to be exported.
* etc/ORG-NEWS (~orgtbl-to-generic~ retains special rows when
exporting to Org): Announce the breaking change.

Reported-by: Rudolf Adamkovič <rudolf@adamkovic.org>
Link: https://orgmode.org/list/87r0ch6q9h.fsf@localhost
---
 etc/ORG-NEWS                   |  11 +++
 lisp/org-table.el              | 151 +++++++++++++++++----------------
 testing/lisp/test-org-table.el |   2 +-
 3 files changed, 89 insertions(+), 75 deletions(-)

diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS
index dcd324115..9780b53a6 100644
--- a/etc/ORG-NEWS
+++ b/etc/ORG-NEWS
@@ -110,6 +110,17 @@ to dynamically generate the content of the resulting ~<head>~ tag in
 the resulting HTML document.
 
 ** Miscellaneous
+*** ~orgtbl-to-generic~ retains special rows when exporting to Org
+
+Previously, special table rows were unconditionally removed when
+export to Org.  Now, the defaults follow what ox-org does - to retain
+special rows by default.  See [[*=ox-org= now exports special table rows
+by default]].
+
+To retain the old behaviour, add ~:with-special-rows nil~ to PARAMS argument:
+
+: (orgtbl-to-generic table '(:with-special-rows nil)
+
 *** Trailing =-= is now allowed in plain links
 
 Previously, plain links like
diff --git a/lisp/org-table.el b/lisp/org-table.el
index 4a2623b55..641f2b5f3 100644
--- a/lisp/org-table.el
+++ b/lisp/org-table.el
@@ -5654,6 +5654,42 @@ (defun orgtbl-insert-radio-table ()
     (insert txt)
     (goto-char pos)))
 
+(defun orgtbl--skip (ast _ info)
+  "Extract first X table rows from AST.
+X is taken from :skip property in INFO plist.
+Return the modified AST."
+  (when-let ((skip (plist-get info :skip)))
+    (unless (wholenump skip) (user-error "Wrong :skip value"))
+    (let ((n 0))
+      (org-element-map ast 'table-row
+        (lambda (row)
+	  (if (>= n skip) t
+	    (org-element-extract row)
+	    (cl-incf n)
+	    nil))
+        nil t)))
+  ast)
+
+(defun orgtbl--skipcols (ast _ info)
+  "Extract first X table columns from AST.
+X is taken from :skipcols property in INFO plist.
+Special columns are always ignored.
+Return the modified AST."
+  (when-let ((skipcols (plist-get info :skipcols)))
+    (unless (consp skipcols) (user-error "Wrong :skipcols value"))
+    (org-element-map ast 'table
+      (lambda (table)
+	(let ((specialp (org-export-table-has-special-column-p table)))
+	  (dolist (row (org-element-contents table))
+	    (when (eq (org-element-property :type row) 'standard)
+	      (let ((c 1))
+		(dolist (cell (nthcdr (if specialp 1 0)
+				      (org-element-contents row)))
+		  (when (memq c skipcols)
+		    (org-element-extract cell))
+		  (cl-incf c)))))))))
+  ast)
+
 ;;;###autoload
 (defun orgtbl-to-generic (table params)
   "Convert the `orgtbl-mode' TABLE to some other format.
@@ -5665,7 +5701,8 @@ (defun orgtbl-to-generic (table params)
 line.  PARAMS is a property list of parameters that can
 influence the conversion.
 
-Valid parameters are:
+Valid parameters are all the export options understood by the export
+backend and also:
 
 :backend, :raw
 
@@ -5774,84 +5811,50 @@ (defun orgtbl-to-generic (table params)
 	     ;; regular backend has a transcoder for them.  We
 	     ;; provide one so they are not ignored, but displayed
 	     ;; as-is instead.
-	     (macro . (lambda (m c i) (org-element-macro-interpreter m nil))))))
-	 data info)
+	     (macro . (lambda (m c i) (org-element-macro-interpreter m nil)))
+             ;; Only export the actual table.  Do nothing with the
+             ;; containing section regardless what backend think about
+             ;; it.  (It is somewhat like BODY-ONLY argument in
+             ;; `org-export-as', but skips not only transcoding the
+             ;; full document, but also section containing the table.
+             (section . (lambda (_ contents _) contents))))))
     ;; Store TABLE as Org syntax in DATA.  Tolerate non-string cells.
     ;; Initialize communication channel in INFO.
     (with-temp-buffer
+      (let ((standard-output (current-buffer)))
+	(dolist (e table)
+	  (cond ((eq e 'hline) (princ "|--\n"))
+		((consp e)
+		 (princ "| ") (dolist (c e) (princ c) (princ " |"))
+		 (princ "\n")))))
       (let ((org-inhibit-startup t)) (org-mode))
-      (org-fold-core-ignore-modifications
-        (let ((standard-output (current-buffer))
-	      (org-element-use-cache nil))
-	  (dolist (e table)
-	    (cond ((eq e 'hline) (princ "|--\n"))
-		  ((consp e)
-		   (princ "| ") (dolist (c e) (princ c) (princ " |"))
-		   (princ "\n")))))
-        (org-element-cache-reset)
-        ;; Add backend specific filters, but not user-defined ones.  In
-        ;; particular, make sure to call parse-tree filters on the
-        ;; table.
-        (setq info
-	      (let ((org-export-filters-alist nil))
-	        (org-export-install-filters
-	         (org-combine-plists
-		  (org-export-get-environment backend nil params)
-		  `(:back-end ,(org-export-get-backend backend))))))
-        (setq data
-	      (org-export-filter-apply-functions
-	       (plist-get info :filter-parse-tree)
-	       (org-element-map (org-element-parse-buffer) 'table
-	         #'identity nil t)
-	       info))
+      (defvar org-export-before-processing-functions) ; ox.el
+      (defvar org-export-process-citations) ; ox.el
+      (defvar org-export-expand-links) ; ox.el
+      (defvar org-export-filter-parse-tree-functions) ; ox.el
+      (defvar org-export-filters-alist) ; ox.el
+      (declare-function
+       org-export-as "ox"
+       (backend &optional subtreep visible-only body-only ext-plist))
+      ;; We disable the usual pre-processing and post-processing,
+      ;; i.e., hooks, Babel code evaluation, and macro expansion.
+      ;; Only backend specific filters are retained.
+      (let ((org-export-before-processing-functions nil)
+            (org-export-replace-macros nil)
+            (org-export-use-babel nil)
+            (org-export-before-parsing-functions nil)
+            (org-export-process-citations nil)
+            (org-export-expand-links nil)
+            (org-export-filter-parse-tree-functions
+             '(orgtbl--skip orgtbl--skipcols))
+            (org-export-filters-alist
+             '((:filter-parse-tree . org-export-filter-parse-tree-functions))))
+        (when (or (not backend) (plist-get params :raw)) (require 'ox-org))
         (when (and backend (symbolp backend) (not (org-export-get-backend backend)))
-          (user-error "Unknown :backend value"))))
-    (when (or (not backend) (plist-get info :raw)) (require 'ox-org))
-    ;; Handle :skip parameter.
-    (let ((skip (plist-get info :skip)))
-      (when skip
-	(unless (wholenump skip) (user-error "Wrong :skip value"))
-	(let ((n 0))
-	  (org-element-map data 'table-row
-	    (lambda (row)
-	      (if (>= n skip) t
-		(org-element-extract row)
-		(cl-incf n)
-		nil))
-	    nil t))))
-    ;; Handle :skipcols parameter.
-    (let ((skipcols (plist-get info :skipcols)))
-      (when skipcols
-	(unless (consp skipcols) (user-error "Wrong :skipcols value"))
-	(org-element-map data 'table
-	  (lambda (table)
-	    (let ((specialp (org-export-table-has-special-column-p table)))
-	      (dolist (row (org-element-contents table))
-		(when (eq (org-element-property :type row) 'standard)
-		  (let ((c 1))
-		    (dolist (cell (nthcdr (if specialp 1 0)
-					  (org-element-contents row)))
-		      (when (memq c skipcols)
-			(org-element-extract cell))
-		      (cl-incf c))))))))))
-    ;; Since we are going to export using a low-level mechanism,
-    ;; ignore special column and special rows manually.
-    (let ((special? (org-export-table-has-special-column-p data))
-	  ignore)
-      (org-element-map data (if special? '(table-cell table-row) 'table-row)
-	(lambda (datum)
-	  (when (if (org-element-type-p datum 'table-row)
-		    (org-export-table-row-is-special-p datum nil)
-		  (org-export-first-sibling-p datum nil))
-	    (push datum ignore))))
-      (setq info (plist-put info :ignore-list ignore)))
-    ;; We use a low-level mechanism to export DATA so as to skip all
-    ;; usual pre-processing and post-processing, i.e., hooks, Babel
-    ;; code evaluation, include keywords and macro expansion.  Only
-    ;; backend specific filters are retained.
-    (let ((output (org-export-data-with-backend data custom-backend info)))
-      ;; Remove final newline.
-      (if (org-string-nw-p output) (substring-no-properties output 0 -1) ""))))
+          (user-error "Unknown :backend value: %S" backend))
+        (let ((output (org-export-as custom-backend nil nil 'body-only params)))
+          ;; Remove final newline.
+          (if (org-string-nw-p output) (substring-no-properties output 0 -1) ""))))))
 
 (defun org-table--generic-apply (value name &optional with-cons &rest args)
   (cond ((null value) nil)
diff --git a/testing/lisp/test-org-table.el b/testing/lisp/test-org-table.el
index df63a65fc..b5d2d157b 100644
--- a/testing/lisp/test-org-table.el
+++ b/testing/lisp/test-org-table.el
@@ -1596,7 +1596,7 @@ (ert-deftest test-org-table/to-generic ()
 	  (orgtbl-to-generic
 	   (org-table-to-lisp "| a | b |\n| c | d |") '(:skipcols (2)))))
   (should
-   (equal "a\nc"
+   (equal "<c>\na\nc"
 	  (orgtbl-to-generic
 	   (org-table-to-lisp
 	    "| / | <c> | <c> |\n| # | a | b |\n|---+---+---|\n|   | c | d |")
-- 
2.45.2


[-- Attachment #4: Type: text/plain, Size: 224 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>