Re: [PATCH] lisp/ob-octave.el, was [PATCH] rfc: using ert-deftest with side-effects

[Leo Butler <Leo.Butler@umanitoba.ca>]

[-- Attachment #1: Type: text/plain, Size: 1126 bytes --]

On Sat, Jan 07 2023, Leo Butler <Leo.Butler@umanitoba.ca> wrote:

> On Fri, Jan 06 2023, Ihor Radchenko <yantar92@posteo.net> wrote:
>
>>
>> Leo Butler <Leo.Butler@umanitoba.ca> writes:
>>
>>>> https://builds.sr.ht/~bzg/job/914954
>>>> 2 unexpected results:
>>>>    FAILED  ob-octave/graphics-file  ((should-not (get-buffer "*Org-Babel
>>>>    Error Output*")) :form (get-buffer "*Org-Babel Error Output*") :value
>>>>    #<killed buffer>) 
>>>>    FAILED  ob-octave/graphics-file-space  ((should-not (get-buffer
>>>>    "*Org-Babel Error Output*")) :form (get-buffer "*Org-Babel Error
>>>>    Output*") :value #<killed buffer>) 
>>>>
>>>> As you can see *Org-Babel Error Output* buffer does not exist when
>>>> running the test.
>>>>
>>>> Leo, could you please take a look?
>>>
>>> An earlier test is creating that *Org Babel Error Output* buffer.
>
> I will try to look into improving the tests so that we can trap the test(s)
> that is(are) creating that error buffer.

See the attachment. There are four test failures that are currently
untrapped. I also see the `buffer-live-p' bug. 

Leo


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: org-tests.org --]
[-- Type: text/x-org; name="org-tests.org", Size: 14725 bytes --]

#+AUTHOR: Leo Butler
#+DATE: 10 Jan 2023
#+TITLE: False positives in ~org~ tests
#+STARTUP: show2levels

* Summary
Applying the [[patch-to--org-test-with-temp-text]] reveals 4 tests that fail, but which are passed with the unpatched ~org-test-with-temp-text~ macro.
The next sections show the test failures. References:
- *Org-Babel Error Output* :: [[https://orgmode.org/list/87bknh5nva.fsf@localhosthttps://orgmode.org/list/87bknh5nva.fsf@localhost]]
- buffer-live-p :: [[https://debbugs.gnu.org/cgi/bugreport.cgi?bug=60626]]
  
#+name: patch-to--org-test-with-temp-text
#+begin_example
diff --git a/testing/org-test.el b/testing/org-test.el
index 22ac60670..7909f36cc 100644
--- a/testing/org-test.el
+++ b/testing/org-test.el
@@ -196,7 +196,18 @@ otherwise place the point at the beginning of the inserted text."
 	   (insert inside-text)
 	   (goto-char (point-min))))
        (font-lock-ensure (point-min) (point-max))
-       ,@body)))
+       (prog1
+           (progn ,@body)
+         (unwind-protect
+             (should-not (buffer-live-p (get-buffer "*Org-Babel Error Output*")))
+           (when (get-buffer "*Org-Babel Error Output*")
+             (message "Detected: *Org-Babel Error Output*")
+             (message "Contents:")
+             (message (with-current-buffer "*Org-Babel Error Output*"
+                        (buffer-substring-no-properties (point-min) (point-max))))
+             (message "End:")
+             (kill-buffer "*Org-Babel Error Output*"))))
+         )))
 
 (defmacro org-test-with-temp-text-in-file (text &rest body)
   "Run body in a temporary file buffer with Org mode as the active mode.
   #+end_example
* Command
The tests are run in-place like so:
#+begin_src sh :exports none :results raw drawer
make test-dirty
#+end_src

* Test Failures
** Failure 1
#+begin_example
Detected: *Org-Babel Error Output*
Contents:
/tmp/tmp-orgtest/fortran-src-ym6m0X.F90:5:17:

    5 | write (*, ’(i2)’), nint(s(1,2))
      |                 1
Warning: Legacy Extension: Comma before i/o item list at (1)
[ Babel evaluation exited with code 0 ]
/tmp/tmp-orgtest/fortran-src-EYVgCk.F90:5:17:

    5 | write (*, ’(i2)’), nint(s(2,3))
      |                 1
Warning: Legacy Extension: Comma before i/o item list at (1)
[ Babel evaluation exited with code 0 ]
/tmp/tmp-orgtest/fortran-src-SwTuHS.F90:5:20:

    5 | write (*, ’(3f5.2)’), s
      |                    1
Warning: Legacy Extension: Comma before i/o item list at (1)
[ Babel evaluation exited with code 0 ]
/tmp/tmp-orgtest/fortran-src-xnH1cO.F90:5:20:

    5 | write (*, ’(2f5.2)’), s
      |                    1
Warning: Legacy Extension: Comma before i/o item list at (1)
[ Babel evaluation exited with code 0 ]
End:
Test ob-java/args-quoted-string backtrace:
  ert-fail(((should-not (buffer-live-p (get-buffer "*Org-Babel Error O
  (if (not (unwind-protect (setq value-1015 (apply fn-1013 args-1014))
  (let (form-description-1017) (if (not (unwind-protect (setq value-10
  (let ((value-1015 'ert-form-evaluation-aborted-1016)) (let (form-des
  (let* ((fn-1013 #'buffer-live-p) (args-1014 (condition-case err (let
  (unwind-protect (let* ((fn-1013 #'buffer-live-p) (args-1014 (conditi
  (prog1 (progn (let* ((fn-1008 #'string=) (args-1009 (condition-case 
  (progn (org-mode) (let ((point (string-match "<point>" inside-text))
  (unwind-protect (progn (org-mode) (let ((point (string-match "<point
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn
  (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current
  (let ((inside-text (if (stringp "#+begin_src java :dir 'nil :results
  (let ((lexical-binding t)) (let ((inside-text (if (stringp "#+begin_
  (closure (t) nil (let ((lexical-binding t)) (let ((inside-text (if (
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name ob-java/args-quoted-string :documenta
  ert-run-or-rerun-test(#s(ert--stats :selector "\\(org\\|ob\\|ox\\...
  ert-run-tests("\\(org\\|ob\\|ox\\)" #f(compiled-function (event-type
  ert-run-tests-batch("\\(org\\|ob\\|ox\\)")
  ert-run-tests-batch-and-exit("\\(org\\|ob\\|ox\\)")
  (let ((org-id-track-globally t) (org-test-selector (if org-test-sele
  org-test-run-batch-tests("\\(org\\|ob\\|ox\\)")
  command-line-1(("--eval" "(setq vc-handled-backends nil org-startup-
  command-line()
  normal-top-level()
Test ob-java/args-quoted-string condition:
    (ert-test-failed
     ((should-not
       (buffer-live-p
	(get-buffer "*Org-Babel Error Output*")))
      :form
      (buffer-live-p #<killed buffer>)
      :value t))
   FAILED    71/1116  ob-java/args-quoted-string (0.313545 sec)
#+end_example
** Failure 2
#+begin_example
Detected: *Org-Babel Error Output*
Contents:
[ Babel evaluation exited with code 1 ]
[ Babel evaluation exited with code 2 ]
End:
Test ob-shell/remote-with-stdin-or-cmdline backtrace:
  ert-fail(((should-not (buffer-live-p (get-buffer "*Org-Babel Error O
  (if (not (unwind-protect (setq value-2299 (apply fn-2297 args-2298))
  (let (form-description-2301) (if (not (unwind-protect (setq value-22
  (let ((value-2299 'ert-form-evaluation-aborted-2300)) (let (form-des
  (let* ((fn-2297 #'buffer-live-p) (args-2298 (condition-case err (let
  (unwind-protect (let* ((fn-2297 #'buffer-live-p) (args-2298 (conditi
  (prog1 (progn (org-trim (org-babel-execute-src-block))) (unwind-prot
  (progn (org-mode) (let ((point (string-match "<point>" inside-text))
  (unwind-protect (progn (org-mode) (let ((point (string-match "<point
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn
  (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current
  (let ((inside-text (if (stringp (mapconcat #'identity (list "#+name:
  (let* ((result (let ((inside-text (if (stringp ...) (mapconcat ... .
  (let ((default-directory (or (plist-get spec :dir) default-directory
  (let ((spec (car --dolist-tail--))) (let ((default-directory (or (pl
  (while --dolist-tail-- (let ((spec (car --dolist-tail--))) (let ((de
  (let ((--dolist-tail-- (cons nil (cons (list ':dir remote-dir) (cons
  (closure (t) (remote-dir) (let ((--dolist-tail-- (cons nil (cons (li
  funcall((closure (t) (remote-dir) (let ((--dolist-tail-- (cons nil (
  (let ((tramp-methods (cons '("mock" (tramp-login-program "sh") (tram
  (cond (env-def (funcall body env-def)) ((eq system-type 'windows-nt)
  (let ((env-def (getenv "REMOTE_TEMPORARY_FILE_DIRECTORY"))) (cond (e
  org-test-with-tramp-remote-dir--worker((closure (t) (remote-dir) (le
  (let ((lexical-binding t)) (let* ((fn-2292 #'not) (args-2293 (condit
  (closure (t) nil (let ((lexical-binding t)) (let* ((fn-2292 #'not) (
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name ob-shell/remote-with-stdin-or-cmdline
  ert-run-or-rerun-test(#s(ert--stats :selector "\\(org\\|ob\\|ox\\...
  ert-run-tests("\\(org\\|ob\\|ox\\)" #f(compiled-function (event-type
  ert-run-tests-batch("\\(org\\|ob\\|ox\\)")
  ert-run-tests-batch-and-exit("\\(org\\|ob\\|ox\\)")
  (let ((org-id-track-globally t) (org-test-selector (if org-test-sele
  org-test-run-batch-tests("\\(org\\|ob\\|ox\\)")
  command-line-1(("--eval" "(setq vc-handled-backends nil org-startup-
  command-line()
  normal-top-level()
Test ob-shell/remote-with-stdin-or-cmdline condition:
    (ert-test-failed
     ((should-not
       (buffer-live-p
	(get-buffer "*Org-Babel Error Output*")))
      :form
      (buffer-live-p #<killed buffer>)
      :value t))
   FAILED   190/1116  ob-shell/remote-with-stdin-or-cmdline (0.013323 sec)
#+end_example
** Failure 3
#+begin_example
Detected: *Org-Babel Error Output*
Contents:
[ Babel evaluation exited with code 2 ]
End:
Test ob-tangle/jump-to-org backtrace:
  ert--should-signal-hook(ert-test-failed (((should-not (buffer-live-p
  ert-fail(((should-not (buffer-live-p (get-buffer "*Org-Babel Error O
  (if (not (unwind-protect (setq value-2414 (apply fn-2412 args-2413))
  (let (form-description-2416) (if (not (unwind-protect (setq value-24
  (let ((value-2414 'ert-form-evaluation-aborted-2415)) (let (form-des
  (let* ((fn-2412 #'buffer-live-p) (args-2413 (condition-case err (let
  (unwind-protect (let* ((fn-2412 #'buffer-live-p) (args-2413 (conditi
  (prog1 (progn (org-babel-tangle-jump-to-org) (buffer-string)) (unwin
  (progn (org-mode) (let ((point (string-match "<point>" inside-text))
  (unwind-protect (progn (org-mode) (let ((point (string-match "<point
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn
  (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current
  (let ((inside-text (if (stringp (format ";; [[file:%s][H:1]]\n<point
  (let ((file (buffer-file-name))) (let ((inside-text (if (stringp (fo
  (progn (let ((file (buffer-file-name))) (let ((inside-text (if (stri
  (progn (setq buffer (find-file file)) (if (re-search-forward "<point
  (unwind-protect (progn (setq buffer (find-file file)) (if (re-search
  (let ((file (make-temp-file "org-test")) (inside-text (if (stringp "
  (list "* H\n#+begin_src emacs-lisp\n1\n#+end_src" (let ((file (make-
  (let ((signal-hook-function #'ert--should-signal-hook)) (list "* H\n
  (condition-case err (let ((signal-hook-function #'ert--should-signal
  (let* ((fn-2417 #'equal) (args-2418 (condition-case err (let ((signa
  (let ((org-file-apps '((t . emacs)))) (let* ((fn-2417 #'equal) (args
  (let ((lexical-binding t)) (let ((org-file-apps '((t . emacs)))) (le
  (closure (t) nil (let ((lexical-binding t)) (let ((org-file-apps '((
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name ob-tangle/jump-to-org :documentation 
  ert-run-or-rerun-test(#s(ert--stats :selector "\\(org\\|ob\\|ox\\...
  ert-run-tests("\\(org\\|ob\\|ox\\)" #f(compiled-function (event-type
  ert-run-tests-batch("\\(org\\|ob\\|ox\\)")
  ert-run-tests-batch-and-exit("\\(org\\|ob\\|ox\\)")
  (let ((org-id-track-globally t) (org-test-selector (if org-test-sele
  org-test-run-batch-tests("\\(org\\|ob\\|ox\\)")
  command-line-1(("--eval" "(setq vc-handled-backends nil org-startup-
  command-line()
  normal-top-level()
Test ob-tangle/jump-to-org condition:
    (ert-test-failed
     ((should-not
       (buffer-live-p
	(get-buffer "*Org-Babel Error Output*")))
      :form
      (buffer-live-p #<killed buffer>)
      :value t))
   FAILED   206/1116  ob-tangle/jump-to-org (0.007505 sec)
#+end_example
** Failure 4
#+begin_example
Detected: *Org-Babel Error Output*
Contents:
ls: cannot access ’NoSuchFileOrDirectory.txt’: No such file or directory
[ Babel evaluation exited with code 2 ]
End:
Test test-ob/allow-spaces-around-=-in-var-specs backtrace:
  ert-fail(((should-not (buffer-live-p (get-buffer "*Org-Babel Error O
  (if (not (unwind-protect (setq value-3233 (apply fn-3231 args-3232))
  (let (form-description-3235) (if (not (unwind-protect (setq value-32
  (let ((value-3233 'ert-form-evaluation-aborted-3234)) (let (form-des
  (let* ((fn-3231 #'buffer-live-p) (args-3232 (condition-case err (let
  (unwind-protect (let* ((fn-3231 #'buffer-live-p) (args-3232 (conditi
  (prog1 (progn (let* ((fn-3226 #'=) (args-3227 (condition-case err (l
  (progn (org-mode) (let ((point (string-match "<point>" inside-text))
  (unwind-protect (progn (org-mode) (let ((point (string-match "<point
  (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn
  (let ((temp-buffer (generate-new-buffer " *temp*" t))) (save-current
  (let ((inside-text (if (stringp "#+begin_src emacs-lisp :var a = 1 b
  (let ((lexical-binding t)) (let ((inside-text (if (stringp "#+begin_
  (closure (t) nil (let ((lexical-binding t)) (let ((inside-text (if (
  ert--run-test-internal(#s(ert--test-execution-info :test #s(ert-test
  ert-run-test(#s(ert-test :name test-ob/allow-spaces-around-=-in-var-
  ert-run-or-rerun-test(#s(ert--stats :selector "\\(org\\|ob\\|ox\\...
  ert-run-tests("\\(org\\|ob\\|ox\\)" #f(compiled-function (event-type
  ert-run-tests-batch("\\(org\\|ob\\|ox\\)")
  ert-run-tests-batch-and-exit("\\(org\\|ob\\|ox\\)")
  (let ((org-id-track-globally t) (org-test-selector (if org-test-sele
  org-test-run-batch-tests("\\(org\\|ob\\|ox\\)")
  command-line-1(("--eval" "(setq vc-handled-backends nil org-startup-
  command-line()
  normal-top-level()
Test test-ob/allow-spaces-around-=-in-var-specs condition:
    (ert-test-failed
     ((should-not
       (buffer-live-p
	(get-buffer "*Org-Babel Error Output*")))
      :form
      (buffer-live-p #<killed buffer>)
      :value t))
   FAILED   315/1116  test-ob/allow-spaces-around-=-in-var-specs (0.003319 sec)
#+end_example
** Test Summary
#+begin_example
Ran 1116 tests, 1102 results as expected, 4 unexpected, 10 skipped (2023-01-10 11:36:40-0600, 64.394581 sec)
2 expected failures

4 unexpected results:
   FAILED  ob-java/args-quoted-string  ((should-not (buffer-live-p (get-buffer "*Org-Babel Error Output*"))) :form (buffer-live-p #<killed buffer>) :value t)
   FAILED  ob-shell/remote-with-stdin-or-cmdline  ((should-not (buffer-live-p (get-buffer "*Org-Babel Error Output*"))) :form (buffer-live-p #<killed buffer>) :value t)
   FAILED  ob-tangle/jump-to-org  ((should-not (buffer-live-p (get-buffer "*Org-Babel Error Output*"))) :form (buffer-live-p #<killed buffer>) :value t)
   FAILED  test-ob/allow-spaces-around-=-in-var-specs  ((should-not (buffer-live-p (get-buffer "*Org-Babel Error Output*"))) :form (buffer-live-p #<killed buffer>) :value t)

10 skipped results:
  SKIPPED  org-missing-dependency/test-ob-R  ((skip-unless nil) :form nil :value nil)
  SKIPPED  org-missing-dependency/test-ob-clojure  ((skip-unless nil) :form nil :value nil)
  SKIPPED  org-missing-dependency/test-ob-eshell  ((skip-unless nil) :form nil :value nil)
  SKIPPED  org-missing-dependency/test-ob-julia  ((skip-unless nil) :form nil :value nil)
  SKIPPED  org-missing-dependency/test-ob-lua  ((skip-unless nil) :form nil :value nil)
  SKIPPED  org-missing-dependency/test-ob-ruby  ((skip-unless nil) :form nil :value nil)
  SKIPPED  org-missing-dependency/test-ob-scheme  ((skip-unless nil) :form nil :value nil)
  SKIPPED  org-missing-dependency/test-ob-sql  ((skip-unless nil) :form nil :value nil)
  SKIPPED  org-missing-dependency/test-ob-sqlite  ((skip-unless nil) :form nil :value nil)
  SKIPPED  org-missing-dependency/test-org-attach-git  ((skip-unless nil) :form nil :value nil)

make: *** [mk/targets.mk:100: test-dirty] Error 1
#+end_example