From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0.migadu.com ([2001:41d0:303:e224::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms13.migadu.com with LMTPS id 0E2CGbryi2YCVAEAqHPOHw:P1 (envelope-from ) for ; Mon, 08 Jul 2024 14:07:54 +0000 Received: from aspmx1.migadu.com ([2001:41d0:303:e224::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0.migadu.com with LMTPS id 0E2CGbryi2YCVAEAqHPOHw (envelope-from ) for ; Mon, 08 Jul 2024 16:07:54 +0200 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=posteo.net header.s=2017 header.b=JpWObBb9; dmarc=pass (policy=none) header.from=posteo.net; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1720447674; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=mkpxmYx/Z4v+ynqlehFjMVUTugB3nN7YQ65Y3axQ904=; b=H66afuWG0XwZqmQLakvfAcz88oWlSa/p7IiIJF/gGLagS/ywyxXcEHtMRYc3ptM+3Z8nMp qHu2dI4R3AYDpEX62UH90PMb3ifabnLX6OrrLnW6mhTKXn7cpGLQKg+oXalYI40UzxxL8g kEXBn7HPFYlf9Kt7EchWVbVMghidkshFsIXUPYi9VskYRUvZuuXY1bYIHtxq86FZp40aMI IF9qId8UOq1otI2jdDGvnPd+HrCLyvpn3UuLQlvkwBjPLqfIClvN85TxrY/ofpfwA8aMJO S6Of9C8kv5n7pcRa5M7ZH0MEjnaGrgnLX+wgN9jOLuVhrEVMmWXhfDrNAmqCKQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1720447674; a=rsa-sha256; cv=none; b=kFhStInGkLLFwqAOvOlRWBwxNtuxBzZ7qh2a9q6aPf61veDGhzVbtfUJT1FAtxRczNhKs5 bgWE0ifIVzh5m7U57endIA2v8Q0XKtHiosr9LbOX+YQJyfZCQj/Z5wCX3A/5+15Myy9on2 pVSwuiY9C3r/7w1bbCbKk3ieJ4thyk1Zf7te1PXARuhHybQmThZquueIKAeU4gpmkvhexU /MvHpvbKcxlQPBeh1cxXJcZA5rD6PXwRRJBJ87gVTk96uE0G58cJM6paG2E9OINdgy14vd Xl/Z0LIZXhKJSFGz38BjOx8JTZWwg0Db//4QxxBNowU/zp13G2Qq6Hp1ncW8kg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=posteo.net header.s=2017 header.b=JpWObBb9; dmarc=pass (policy=none) header.from=posteo.net; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 1C9551B3E5 for ; Mon, 8 Jul 2024 16:07:54 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sQn6Y-0007fR-7R; Mon, 08 Jul 2024 08:04:26 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sQn6S-0007Wn-Dn for emacs-orgmode@gnu.org; Mon, 08 Jul 2024 08:04:20 -0400 Received: from mout01.posteo.de ([185.67.36.65]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sQn6O-0004CS-8e for emacs-orgmode@gnu.org; Mon, 08 Jul 2024 08:04:20 -0400 Received: from submission (posteo.de [185.67.36.169]) by mout01.posteo.de (Postfix) with ESMTPS id 15494240027 for ; Mon, 8 Jul 2024 14:04:11 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=posteo.net; s=2017; t=1720440253; bh=sTmroFdAhE8VOCGAMajkVB2yNGNjbCzlWFPmgYOiW1s=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:Content-Type: From; b=JpWObBb94wx9iNyeh0GQAKHE8qBd5xK3x2C+IaJQMKWqsm1KSlJXOZEfIFAuqoDUE lmYpiuT29ti/ldJiNJqS+8Gc9lDGNrjYnPKlc8Q5eJRer85l+lSJ6p8eHVk9OxYYcu K3Eh+cD0mYH3bhPEmtv8BmdGB6frnAhO61ftWQwehtGWy1EFtbX+G30RPIwVhkKO1x oZgB7XwHJUhjXuReLgdLBGQ7UnyRsC/K3aBHgPQdV/gKMRHgHilPAhLbRrVxAyuM5h H9ZANmWfrP41pInRTTxL2x2sqVYjFh9UZ0ZV4wdMR70oU7bK+Z6cJz8nLEeSM1Ok55 EGzOe+Xmyn3LQ== Received: from customer (localhost [127.0.0.1]) by submission (posteo.de) with ESMTPSA id 4WHjTQ5G2Vz6tvr; Mon, 8 Jul 2024 14:04:10 +0200 (CEST) From: Ihor Radchenko To: Rudolf =?utf-8?Q?Adamkovi=C4=8D?= Cc: emacs-orgmode@gnu.org Subject: Re: Org Babel "swallows" table column groups In-Reply-To: <87r0ch6q9h.fsf@localhost> References: <87r0ch6q9h.fsf@localhost> Date: Mon, 08 Jul 2024 12:05:38 +0000 Message-ID: <871q441431.fsf@localhost> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Received-SPF: pass client-ip=185.67.36.65; envelope-from=yantar92@posteo.net; helo=mout01.posteo.de X-Spam_score_int: -43 X-Spam_score: -4.4 X-Spam_bar: ---- X-Spam_report: (-4.4 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_MED=-2.3, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: emacs-orgmode-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Queue-Id: 1C9551B3E5 X-Migadu-Scanner: mx12.migadu.com X-Migadu-Spam-Score: -8.12 X-Spam-Score: -8.12 X-TUID: iRGhKH2UaHmh --=-=-= Content-Type: text/plain Ihor Radchenko writes: >> #+BEGIN_SRC emacs-lisp >> (list (list 1 2) (list "/" "<>") 'hline (list 3 4) (list 5 6)) >> #+END_SRC >> >> Org Babel outputs >> ... >> with the second element of the list >> >> (list "/" "<>") >> >> swallowed, without a word. >> >> Why would Org Babel do this? >> >> And, how can one output tables with column groups? > > This is because of how `orgtbl-to-generic' is implemented. It is taking > pieces from the full ox.el exporter, hard-coding certain things. For > example, it always removes special table lines: > > ;; Since we are going to export using a low-level mechanism, > ;; ignore special column and special rows manually. I refactored `orgtbl-to-generic', so that it does not have to duplicate `org-export-as'. Now, things should be more consistent with the normal export. May you try the attached tentative patch set? --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename=0001-ox-New-custom-option-to-disable-macro-replacement.patch >From f1ff68920c417343fda4c5a6450567d703ccf9b6 Mon Sep 17 00:00:00 2001 Message-ID: From: Ihor Radchenko Date: Mon, 8 Jul 2024 13:52:32 +0200 Subject: [PATCH 1/2] ox: New custom option to disable macro replacement * lisp/ox.el (org-export-replace-macros): New custom option controlling macro replacement. (org-export--annotate-info): Honor it, except when processing inline code block results and their {{{results...}}} macro. * etc/ORG-NEWS (Allow disabling macro replacement during export): Announce the new option. * doc/org-manual.org (Macro Replacement): (Summary of the export process): Document the new option. --- doc/org-manual.org | 11 +++++++---- etc/ORG-NEWS | 9 +++++++++ lisp/ox.el | 13 +++++++++++-- 3 files changed, 27 insertions(+), 6 deletions(-) diff --git a/doc/org-manual.org b/doc/org-manual.org index d30c18e0c..5f5104f91 100644 --- a/doc/org-manual.org +++ b/doc/org-manual.org @@ -12682,9 +12682,11 @@ ** Macro Replacement #+cindex: @samp{MACRO}, keyword #+vindex: org-export-global-macros -Macros replace text snippets during export. Macros are defined -globally in ~org-export-global-macros~, or document-wise with the -following syntax: +#+vindex: org-export-replace-macros +Macros replace text snippets during export[fn::The macro replacement +can be disabled by setting ~org-export-replace-macros~ to nil (default +is t).]. Macros are defined globally in ~org-export-global-macros~, +or document-wise with the following syntax: : #+MACRO: name replacement text; $1, $2 are arguments @@ -16702,7 +16704,8 @@ *** Summary of the export process 3. Remove commented subtrees in the whole buffer (see [[*Comment Lines]]); -4. Replace macros in the whole buffer (see [[*Macro Replacement]]); +4. Replace macros in the whole buffer (see [[*Macro Replacement]]), + unless ~org-export-replace-macros~ is nil; 5. When ~org-export-use-babel~ is non-nil (default), process code blocks: diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS index 0c3b14128..dcd324115 100644 --- a/etc/ORG-NEWS +++ b/etc/ORG-NEWS @@ -43,6 +43,15 @@ or newer. # adding new customizations, or changing the interpretation of the # existing customizations. +*** Allow disabling macro replacement during export + +New custom option ~org-export-replace-macros~ controls whether Org +mode replaces macros in the buffer before export. Set it to nil to +disable macro replacement. + +This variable has no effect on the ={{{results...}}}= macros for inline +code block results. + *** Allow headline/olp target in ~org-capture-templates~ to be a function/variable The variable ~org-capture-templates~ accepts a target specification as diff --git a/lisp/ox.el b/lisp/ox.el index 6fa21be90..902c9f089 100644 --- a/lisp/ox.el +++ b/lisp/ox.el @@ -860,6 +860,14 @@ (defcustom org-export-expand-links t :package-version '(Org . "9.7") :type 'boolean) +(defcustom org-export-replace-macros t + "When non-nil, replace macros before export. +This variable does not affect {{{results}}} macros when processing +code block results." + :group 'org-export-general + :package-version '(Org . "9.8") + :type 'boolean) + (defcustom org-export-snippet-translation-alist nil "Alist between export snippets backends and exporter backends. @@ -3048,8 +3056,9 @@ (defun org-export--annotate-info (backend info &optional subtreep visible-only e (org-export-backend-name backend)) (org-export-expand-include-keyword nil nil nil nil (plist-get info :expand-links)) (org-export--delete-comment-trees) - (org-macro-initialize-templates org-export-global-macros) - (org-macro-replace-all org-macro-templates parsed-keywords) + (when org-export-replace-macros + (org-macro-initialize-templates org-export-global-macros) + (org-macro-replace-all org-macro-templates parsed-keywords)) ;; Refresh buffer properties and radio targets after previous ;; potentially invasive changes. (org-set-regexps-and-options) -- 2.45.2 --=-=-= Content-Type: text/x-patch; charset=utf-8 Content-Disposition: inline; filename=0002-orgtbl-to-generic-Retain-special-rows-in-code-block-.patch Content-Transfer-Encoding: quoted-printable >From 3e0a5164661153d41a89ce984c9282273c54a3de Mon Sep 17 00:00:00 2001 Message-ID: <3e0a5164661153d41a89ce984c9282273c54a3de.1720440129.git.yantar= 92@posteo.net> In-Reply-To: References: From: Ihor Radchenko Date: Mon, 8 Jul 2024 13:54:14 +0200 Subject: [PATCH 2/2] orgtbl-to-generic: Retain special rows in code block table output MIME-Version: 1.0 Content-Type: text/plain; charset=3DUTF-8 Content-Transfer-Encoding: 8bit * lisp/org-table.el (orgtbl--skip): (orgtbl--skipcols): New helper functions. (orgtbl-to-generic): Use `org-export-as' machinery to setup table export instead of duplicating ox.el internals. Retain special rows in tables when exporting to Org. Org export is used by ob-core to format the code block output and will thus retain special rows. * testing/lisp/test-org-table.el (test-org-table/to-generic): Adjust test to expect special rows to be exported. * etc/ORG-NEWS (~orgtbl-to-generic~ retains special rows when exporting to Org): Announce the breaking change. Reported-by: Rudolf Adamkovi=C4=8D Link: https://orgmode.org/list/87r0ch6q9h.fsf@localhost --- etc/ORG-NEWS | 11 +++ lisp/org-table.el | 151 +++++++++++++++++---------------- testing/lisp/test-org-table.el | 2 +- 3 files changed, 89 insertions(+), 75 deletions(-) diff --git a/etc/ORG-NEWS b/etc/ORG-NEWS index dcd324115..9780b53a6 100644 --- a/etc/ORG-NEWS +++ b/etc/ORG-NEWS @@ -110,6 +110,17 @@ to dynamically generate the content of the resulting ~= ~ tag in the resulting HTML document. =20 ** Miscellaneous +*** ~orgtbl-to-generic~ retains special rows when exporting to Org + +Previously, special table rows were unconditionally removed when +export to Org. Now, the defaults follow what ox-org does - to retain +special rows by default. See [[*=3Dox-org=3D now exports special table ro= ws +by default]]. + +To retain the old behaviour, add ~:with-special-rows nil~ to PARAMS argume= nt: + +: (orgtbl-to-generic table '(:with-special-rows nil) + *** Trailing =3D-=3D is now allowed in plain links =20 Previously, plain links like diff --git a/lisp/org-table.el b/lisp/org-table.el index 4a2623b55..641f2b5f3 100644 --- a/lisp/org-table.el +++ b/lisp/org-table.el @@ -5654,6 +5654,42 @@ (defun orgtbl-insert-radio-table () (insert txt) (goto-char pos))) =20 +(defun orgtbl--skip (ast _ info) + "Extract first X table rows from AST. +X is taken from :skip property in INFO plist. +Return the modified AST." + (when-let ((skip (plist-get info :skip))) + (unless (wholenump skip) (user-error "Wrong :skip value")) + (let ((n 0)) + (org-element-map ast 'table-row + (lambda (row) + (if (>=3D n skip) t + (org-element-extract row) + (cl-incf n) + nil)) + nil t))) + ast) + +(defun orgtbl--skipcols (ast _ info) + "Extract first X table columns from AST. +X is taken from :skipcols property in INFO plist. +Special columns are always ignored. +Return the modified AST." + (when-let ((skipcols (plist-get info :skipcols))) + (unless (consp skipcols) (user-error "Wrong :skipcols value")) + (org-element-map ast 'table + (lambda (table) + (let ((specialp (org-export-table-has-special-column-p table))) + (dolist (row (org-element-contents table)) + (when (eq (org-element-property :type row) 'standard) + (let ((c 1)) + (dolist (cell (nthcdr (if specialp 1 0) + (org-element-contents row))) + (when (memq c skipcols) + (org-element-extract cell)) + (cl-incf c))))))))) + ast) + ;;;###autoload (defun orgtbl-to-generic (table params) "Convert the `orgtbl-mode' TABLE to some other format. @@ -5665,7 +5701,8 @@ (defun orgtbl-to-generic (table params) line. PARAMS is a property list of parameters that can influence the conversion. =20 -Valid parameters are: +Valid parameters are all the export options understood by the export +backend and also: =20 :backend, :raw =20 @@ -5774,84 +5811,50 @@ (defun orgtbl-to-generic (table params) ;; regular backend has a transcoder for them. We ;; provide one so they are not ignored, but displayed ;; as-is instead. - (macro . (lambda (m c i) (org-element-macro-interpreter m nil)))))) - data info) + (macro . (lambda (m c i) (org-element-macro-interpreter m nil))) + ;; Only export the actual table. Do nothing with the + ;; containing section regardless what backend think about + ;; it. (It is somewhat like BODY-ONLY argument in + ;; `org-export-as', but skips not only transcoding the + ;; full document, but also section containing the table. + (section . (lambda (_ contents _) contents)))))) ;; Store TABLE as Org syntax in DATA. Tolerate non-string cells. ;; Initialize communication channel in INFO. (with-temp-buffer + (let ((standard-output (current-buffer))) + (dolist (e table) + (cond ((eq e 'hline) (princ "|--\n")) + ((consp e) + (princ "| ") (dolist (c e) (princ c) (princ " |")) + (princ "\n"))))) (let ((org-inhibit-startup t)) (org-mode)) - (org-fold-core-ignore-modifications - (let ((standard-output (current-buffer)) - (org-element-use-cache nil)) - (dolist (e table) - (cond ((eq e 'hline) (princ "|--\n")) - ((consp e) - (princ "| ") (dolist (c e) (princ c) (princ " |")) - (princ "\n"))))) - (org-element-cache-reset) - ;; Add backend specific filters, but not user-defined ones. In - ;; particular, make sure to call parse-tree filters on the - ;; table. - (setq info - (let ((org-export-filters-alist nil)) - (org-export-install-filters - (org-combine-plists - (org-export-get-environment backend nil params) - `(:back-end ,(org-export-get-backend backend)))))) - (setq data - (org-export-filter-apply-functions - (plist-get info :filter-parse-tree) - (org-element-map (org-element-parse-buffer) 'table - #'identity nil t) - info)) + (defvar org-export-before-processing-functions) ; ox.el + (defvar org-export-process-citations) ; ox.el + (defvar org-export-expand-links) ; ox.el + (defvar org-export-filter-parse-tree-functions) ; ox.el + (defvar org-export-filters-alist) ; ox.el + (declare-function + org-export-as "ox" + (backend &optional subtreep visible-only body-only ext-plist)) + ;; We disable the usual pre-processing and post-processing, + ;; i.e., hooks, Babel code evaluation, and macro expansion. + ;; Only backend specific filters are retained. + (let ((org-export-before-processing-functions nil) + (org-export-replace-macros nil) + (org-export-use-babel nil) + (org-export-before-parsing-functions nil) + (org-export-process-citations nil) + (org-export-expand-links nil) + (org-export-filter-parse-tree-functions + '(orgtbl--skip orgtbl--skipcols)) + (org-export-filters-alist + '((:filter-parse-tree . org-export-filter-parse-tree-function= s)))) + (when (or (not backend) (plist-get params :raw)) (require 'ox-org)) (when (and backend (symbolp backend) (not (org-export-get-backend = backend))) - (user-error "Unknown :backend value")))) - (when (or (not backend) (plist-get info :raw)) (require 'ox-org)) - ;; Handle :skip parameter. - (let ((skip (plist-get info :skip))) - (when skip - (unless (wholenump skip) (user-error "Wrong :skip value")) - (let ((n 0)) - (org-element-map data 'table-row - (lambda (row) - (if (>=3D n skip) t - (org-element-extract row) - (cl-incf n) - nil)) - nil t)))) - ;; Handle :skipcols parameter. - (let ((skipcols (plist-get info :skipcols))) - (when skipcols - (unless (consp skipcols) (user-error "Wrong :skipcols value")) - (org-element-map data 'table - (lambda (table) - (let ((specialp (org-export-table-has-special-column-p table))) - (dolist (row (org-element-contents table)) - (when (eq (org-element-property :type row) 'standard) - (let ((c 1)) - (dolist (cell (nthcdr (if specialp 1 0) - (org-element-contents row))) - (when (memq c skipcols) - (org-element-extract cell)) - (cl-incf c)))))))))) - ;; Since we are going to export using a low-level mechanism, - ;; ignore special column and special rows manually. - (let ((special? (org-export-table-has-special-column-p data)) - ignore) - (org-element-map data (if special? '(table-cell table-row) 'table-ro= w) - (lambda (datum) - (when (if (org-element-type-p datum 'table-row) - (org-export-table-row-is-special-p datum nil) - (org-export-first-sibling-p datum nil)) - (push datum ignore)))) - (setq info (plist-put info :ignore-list ignore))) - ;; We use a low-level mechanism to export DATA so as to skip all - ;; usual pre-processing and post-processing, i.e., hooks, Babel - ;; code evaluation, include keywords and macro expansion. Only - ;; backend specific filters are retained. - (let ((output (org-export-data-with-backend data custom-backend info))) - ;; Remove final newline. - (if (org-string-nw-p output) (substring-no-properties output 0 -1) "= ")))) + (user-error "Unknown :backend value: %S" backend)) + (let ((output (org-export-as custom-backend nil nil 'body-only par= ams))) + ;; Remove final newline. + (if (org-string-nw-p output) (substring-no-properties output 0 -= 1) "")))))) =20 (defun org-table--generic-apply (value name &optional with-cons &rest args) (cond ((null value) nil) diff --git a/testing/lisp/test-org-table.el b/testing/lisp/test-org-table.el index df63a65fc..b5d2d157b 100644 --- a/testing/lisp/test-org-table.el +++ b/testing/lisp/test-org-table.el @@ -1596,7 +1596,7 @@ (ert-deftest test-org-table/to-generic () (orgtbl-to-generic (org-table-to-lisp "| a | b |\n| c | d |") '(:skipcols (2))))) (should - (equal "a\nc" + (equal "\na\nc" (orgtbl-to-generic (org-table-to-lisp "| / | | |\n| # | a | b |\n|---+---+---|\n| | c | d |") --=20 2.45.2 --=-=-= Content-Type: text/plain -- Ihor Radchenko // yantar92, Org mode contributor, Learn more about Org mode at . Support Org development at , or support my work at --=-=-=--