Change behavior of "org-babel-tangle-publish" from org 9.6 to 9.7 -- deleting source files and not tangling properly

Change behavior of "org-babel-tangle-publish" from org 9.6 to 9.7 -- deleting source files and not tangling properly

[Gilles Marait]

[-- Attachment #1: Type: text/plain, Size: 2942 bytes --]

Hello everyone,

I encountered a change of behavior since org version 9.7 than removes
my source files instead of tangling them at the correct location, using
"org-babel-tangle-publish".

I have not found in the changelog any description of something that
could explain this change, so I report it here.

I'll try to show a minimal example of this issue.

* Description

I have the following file structure.

.
├── org
│   └── Code.org
├── publish.el
└── src
    └── code.cpp

In the "org" directory, I have my org file that describes my code in
literate programming. It's content is:

#+begin_src c++ :tangle ../src/code.cpp 
int a = 3;
#+end_src

The expected behavior is to generate the file "code.cpp" in the src
directory with the tangle mechanism. In this example code.cpp only
contains the line:

int a = 3;

To generate the sources from the org files, I use org-publish. The
code of publish.el is:

  (require 'package)
  (package-initialize)
  (require 'org)
  (require 'ox-publish)
  (setq org-publish-project-alist
	(list
	 (list "generate-source-code"
	       :base-directory "./org"
	       :base-extension "org"
	       :recursive t
	       :publishing-function '(org-babel-tangle-publish)
	       :publishing-directory "."
	       :auto-sitemap nil)))
  (provide 'publish)
  ;;; publish.el ends here

To use the publish from a script I use the following command:

  emacs --batch --load publish.el --eval '(org-publish "generate-source-code")'

* With org-mode 9.6

It works as expected, the code in "Code.cpp" is tangled into "src/code.cpp".

* With org-mode 9.7

I can see "code.cpp" is tangled at the root directory "./code.cpp",
but the file "src/code.cpp" that existed previously is deleted.

* Additional information

emacs version: "29.4"

org-version:
- Test with version 9.7.10 (commit 5ebd973e09d7a46d056f1cd9b5efc44e878cce9c)
- Test with version 9.6.30 (commit 071c6e986c424d2e496be7d0815d6e9cd83ae4e6)

If I use directly "org-babel-tangle-file" I don't have the problem,
the file in tangled to the right location:

emacs --batch --eval "(require 'org)" --eval '(org-babel-tangle-file "./org/Code.org")'

* Bonus question

As a guix user, I use contained environments as much as possible. In
the case described above, I generate an environment without git. I get
the following warning:

Error: (file-missing "Searching for program" "No such file or directory" "git")

It does not have any consequence on the tangling process, but I was
wondering if it was expected that my org-publish process depends on
git ? Would there be any benefit to add it to my environment ?

* Repository

I created a repository with some scripts to reproduce the error, using
guix to control the environment.

https://gitlab.inria.fr/gmarait/debug-org-9.7 


Thank you and best regards, 
Gilles Marait 

[-- Attachment #2: Type: text/html, Size: 3104 bytes --]

Re: Change behavior of "org-babel-tangle-publish" from org 9.6 to 9.7 -- deleting source files and not tangling properly

[Ihor Radchenko]

Gilles Marait <gilles.marait@inria.fr> writes:

> I encountered a change of behavior since org version 9.7 than removes
> my source files instead of tangling them at the correct location, using
> "org-babel-tangle-publish".
>
> I have not found in the changelog any description of something that
> could explain this change, so I report it here.
> ...

I suspect that the regression is after
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=478576749d

May I know if you end up with _two_ code.cpp files in Org 9.6 - one in
"." directory and one in src/ directory?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Change behavior of "org-babel-tangle-publish" from org 9.6 to 9.7 -- deleting source files and not tangling properly

[Gilles Marait]

Hi Igor,

Thanks for your answer. 
Yes, with org 9.6, I end up with two code.cpp as your said, one in "." and one in "src".

I think the line you're pointing to is indeed the problem

----- Mail original -----
> De: "Ihor Radchenko" <yantar92@posteo.net>
> À: "Gilles Marait" <gilles.marait@inria.fr>
> Cc: "emacs-orgmode" <emacs-orgmode@gnu.org>, "emmanuel agullo" <emmanuel.agullo@inria.fr>
> Envoyé: Lundi 9 Septembre 2024 19:40:27
> Objet: Re: Change behavior of "org-babel-tangle-publish" from org 9.6 to 9.7 -- deleting source files and not tangling
> properly

> Gilles Marait <gilles.marait@inria.fr> writes:
> 
>> I encountered a change of behavior since org version 9.7 than removes
>> my source files instead of tangling them at the correct location, using
>> "org-babel-tangle-publish".
>>
>> I have not found in the changelog any description of something that
>> could explain this change, so I report it here.
>> ...
> 
> I suspect that the regression is after
> https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=478576749d
> 
> May I know if you end up with _two_ code.cpp files in Org 9.6 - one in
> "." directory and one in src/ directory?
> 
> --
> Ihor Radchenko // yantar92,
> Org mode contributor,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>

Re: Change behavior of "org-babel-tangle-publish" from org 9.6 to 9.7 -- deleting source files and not tangling properly

[Ihor Radchenko]

Gilles Marait <gilles.marait@inria.fr> writes:

> Hi Igor,
>
> Thanks for your answer. 
> Yes, with org 9.6, I end up with two code.cpp as your said, one in "." and one in "src".
>
> I think the line you're pointing to is indeed the problem

This is a bit tricky.

The new behavior is not exactly a bug - you just relied upon
undocumented behavior of `org-babel-tangle-publish' not cleaning up the
tangled files.

On the other hand, I can see tangled files being removed during
publishing as unexpected - what if the same set of files is used for
actual tangling (e.g. of config files). In such scenario, the configs
will be moved, potentially breaking workflows.

IMHO, it makes more sense to preserve _and also document_ (in the
docstring) the old behavior. We just need to make sure that the bug
fixed in 478576749d does not re-surface - when publishing directory is
"." the old code failed trying to copy tangled file into self.

I think that we can simply call `org-publish-attachment' on every
tangled file in `org-babel-tangle-publish' - `org-publish-attachment'
takes care about the situation when the tangled file is already in the
publishing dir.

Would you be interested to submit a patch fixing the issue as I described?

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Change behavior of "org-babel-tangle-publish" from org 9.6 to 9.7 -- deleting source files and not tangling properly

[Ihor Radchenko]

[-- Attachment #1: Type: text/plain, Size: 623 bytes --]

Ihor Radchenko <yantar92@posteo.net> writes:

> IMHO, it makes more sense to preserve _and also document_ (in the
> docstring) the old behavior. We just need to make sure that the bug
> fixed in 478576749d does not re-surface - when publishing directory is
> "." the old code failed trying to copy tangled file into self.
>
> I think that we can simply call `org-publish-attachment' on every
> tangled file in `org-babel-tangle-publish' - `org-publish-attachment'
> takes care about the situation when the tangled file is already in the
> publishing dir.

Gilles, may you please check if the attached patch works for you?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-babel-tangle-publish-Keep-the-original-tangled-f.patch --]
[-- Type: text/x-patch, Size: 1935 bytes --]

From 778a29dcc64e7f400f94145adf01dca67f5588f4 Mon Sep 17 00:00:00 2001
Message-ID: <778a29dcc64e7f400f94145adf01dca67f5588f4.1734270889.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Sun, 15 Dec 2024 14:52:39 +0100
Subject: [PATCH] org-babel-tangle-publish: Keep the original tangled files

* lisp/ob-tangle.el (org-babel-tangle-publish): When publishing
tangled files, do not remove the original versions of these files.
Keeping was the old behaviour that some users relied upon.  Removing
the tangled files may also be unexpected when they were tangled
outside publishing, during normal user workflows.  Update the
docstring to explicitly state that the tangled files are copied.

Reported-by: Gilles Marait <gilles.marait@inria.fr>
Link: https://orgmode.org/list/878qvtkwhn.fsf@localhost
---
 lisp/ob-tangle.el | 12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)

diff --git a/lisp/ob-tangle.el b/lisp/ob-tangle.el
index fe143b039f..9554a3226d 100644
--- a/lisp/ob-tangle.el
+++ b/lisp/ob-tangle.el
@@ -227,13 +227,11 @@ (defun org-babel-tangle-file (file &optional target-file lang-re)
              (org-babel-tangle nil target-file lang-re)))))
 
 (defun org-babel-tangle-publish (_ filename pub-dir)
-  "Tangle FILENAME and place the results in PUB-DIR."
-  (unless (file-exists-p pub-dir)
-    (make-directory pub-dir t))
-  (setq pub-dir (file-name-as-directory pub-dir))
-  ;; Rename files to avoid copying to same file when publishing to ./
-  ;; `copy-file' would throw an error when copying file to self.
-  (mapc (lambda (el) (rename-file el pub-dir t))
+  "Tangle FILENAME and copy the tangled file to PUB-DIR."
+  (require 'ox-publish)
+  (declare-function org-publish-attachment "ox-publish"
+                    (plist filename pub-dir))
+  (mapc (lambda (el) (org-publish-attachment nil el pub-dir))
         (org-babel-tangle-file filename)))
 
 ;;;###autoload
-- 
2.47.1


[-- Attachment #3: Type: text/plain, Size: 223 bytes --]


-- 
Ihor Radchenko // yantar92,
Org mode maintainer,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Change behavior of "org-babel-tangle-publish" from org 9.6 to 9.7 -- deleting source files and not tangling properly

[Gilles Marait]

[-- Attachment #1: Type: text/plain, Size: 1575 bytes --]

Hello Ihor,

Yes, it works perfectly.
Thank you very much for the patch. I don't think I would have been able
to do it properly with my limited knowledge of elisp.

Gilles

----- Mail original -----
> De: "Ihor Radchenko" <yantar92@posteo.net>
> À: "Gilles Marait" <gilles.marait@inria.fr>
> Cc: "emacs-orgmode" <emacs-orgmode@gnu.org>, "emmanuel agullo" <emmanuel.agullo@inria.fr>
> Envoyé: Dimanche 15 Décembre 2024 14:55:44
> Objet: Re: Change behavior of "org-babel-tangle-publish" from org 9.6 to 9.7 -- deleting source files and not tangling
> properly

> Ihor Radchenko <yantar92@posteo.net> writes:
> 
>> IMHO, it makes more sense to preserve _and also document_ (in the
>> docstring) the old behavior. We just need to make sure that the bug
>> fixed in 478576749d does not re-surface - when publishing directory is
>> "." the old code failed trying to copy tangled file into self.
>>
>> I think that we can simply call `org-publish-attachment' on every
>> tangled file in `org-babel-tangle-publish' - `org-publish-attachment'
>> takes care about the situation when the tangled file is already in the
>> publishing dir.
> 
> Gilles, may you please check if the attached patch works for you?
> 
> 
> 
> [Documents
> texte:0001-org-babel-tangle-publish-Keep-the-original-tangled-f.patch]
> 
> 
> --
> Ihor Radchenko // yantar92,
> Org mode maintainer,
> Learn more about Org mode at <https://orgmode.org/>.
> Support Org development at <https://liberapay.com/org-mode>,
> or support my work at <https://liberapay.com/yantar92>

[-- Attachment #2: Type: text/x-patch, Size: 1935 bytes --]

From 778a29dcc64e7f400f94145adf01dca67f5588f4 Mon Sep 17 00:00:00 2001
Message-ID: <778a29dcc64e7f400f94145adf01dca67f5588f4.1734270889.git.yantar92@posteo.net>
From: Ihor Radchenko <yantar92@posteo.net>
Date: Sun, 15 Dec 2024 14:52:39 +0100
Subject: [PATCH] org-babel-tangle-publish: Keep the original tangled files

* lisp/ob-tangle.el (org-babel-tangle-publish): When publishing
tangled files, do not remove the original versions of these files.
Keeping was the old behaviour that some users relied upon.  Removing
the tangled files may also be unexpected when they were tangled
outside publishing, during normal user workflows.  Update the
docstring to explicitly state that the tangled files are copied.

Reported-by: Gilles Marait <gilles.marait@inria.fr>
Link: https://orgmode.org/list/878qvtkwhn.fsf@localhost
---
 lisp/ob-tangle.el | 12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)

diff --git a/lisp/ob-tangle.el b/lisp/ob-tangle.el
index fe143b039f..9554a3226d 100644
--- a/lisp/ob-tangle.el
+++ b/lisp/ob-tangle.el
@@ -227,13 +227,11 @@ (defun org-babel-tangle-file (file &optional target-file lang-re)
              (org-babel-tangle nil target-file lang-re)))))
 
 (defun org-babel-tangle-publish (_ filename pub-dir)
-  "Tangle FILENAME and place the results in PUB-DIR."
-  (unless (file-exists-p pub-dir)
-    (make-directory pub-dir t))
-  (setq pub-dir (file-name-as-directory pub-dir))
-  ;; Rename files to avoid copying to same file when publishing to ./
-  ;; `copy-file' would throw an error when copying file to self.
-  (mapc (lambda (el) (rename-file el pub-dir t))
+  "Tangle FILENAME and copy the tangled file to PUB-DIR."
+  (require 'ox-publish)
+  (declare-function org-publish-attachment "ox-publish"
+                    (plist filename pub-dir))
+  (mapc (lambda (el) (org-publish-attachment nil el pub-dir))
         (org-babel-tangle-file filename)))
 
 ;;;###autoload
-- 
2.47.1

Re: Change behavior of "org-babel-tangle-publish" from org 9.6 to 9.7 -- deleting source files and not tangling properly

[Ihor Radchenko]

Gilles Marait <gilles.marait@inria.fr> writes:

> Yes, it works perfectly.
> Thank you very much for the patch. I don't think I would have been able
> to do it properly with my limited knowledge of elisp.

Fixed, on bugfix.
https://git.savannah.gnu.org/cgit/emacs/org-mode.git/commit/?id=778a29dcc6

-- 
Ihor Radchenko // yantar92,
Org mode maintainer,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>