Regression tests and stderr

Regression tests and stderr

[Sebastien Vauban]

Hello,

FYI, I just updated, and ran the tests, looking more carefully at the results
once again.

The trailer of what's displayed in the terminal is here:

--8<---------------cut here---------------start------------->8---
       passed  102/112  test-org/accumulated-properties-in-drawers
       passed  103/112  test-org/org-link-escape-ascii-character
       passed  104/112  test-org/org-link-escape-ascii-ctrl-character
       passed  105/112  test-org/org-link-escape-custom-table
       passed  106/112  test-org/org-link-escape-custom-table-merge
       passed  107/112  test-org/org-link-escape-multibyte-character
       passed  108/112  test-org/org-link-escape-url-with-escaped-char
       passed  109/112  test-org/org-link-unescape-ascii-character
       passed  110/112  test-org/org-link-unescape-ascii-ctrl-character
       passed  111/112  test-org/org-link-unescape-ascii-extended-char
       passed  112/112  test-org/org-link-unescape-multibyte-character

    Ran 112 tests, 111 results as expected, 1 unexpected (2012-03-01 22:03:59+0100)
    11 expected failures

    1 unexpected results:
       FAILED  test-org-export/fuzzy-links
--8<---------------cut here---------------end--------------->8---

Pretty good. One failure, though, which I'll try to investigate (later on).

What's weird is the output that comes _after_ the above summary:

--8<---------------cut here---------------start------------->8---
    (a (quote 1))

    (a (quote 2))

    (it (quote 1))

    (it (quote 4))

    (results (quote (4 3 2 1)))

    [...]

    (bar (quote 2))

    (foo (quote 1))

    (bar (quote 2))
    sva@MEDIACENTER$
--8<---------------cut here---------------end--------------->8---

Weird flushes, quite disturbing for analyzing the output of the tests.

I've discovered the same symptoms when using `print' or `princ' functions in
code that's run in batch mode. They come after calls to `message'.

What's even more annoying is that all the output of the testing -- but the
above trailer which comes after the test summary -- is sent to `/dev/stderr'.

What I tried to get is normal, informational messages, sent to `/dev/stdout',
and real errors (like test failures) sent to `/dev/stderr'.

Sadly, this seems to be a limit of Emacs itself:

   "In batch mode, Emacs does not display the text being edited, and the standard
    terminal interrupt characters such as C-z and C-c have their usual effect.
    Emacs functions that normally print a message in the echo area will print to
    either the standard output stream (stdout) or the standard error stream
    (stderr) instead. (To be precise, functions like prin1, princ and print print
    to stdout, while message and error print to stderr.) Functions that normally
    read keyboard input from the minibuffer take their input from the terminal's
    standard input stream (stdin) instead."

    http://www.gnu.org/software/emacs/manual/html_node/emacs/Initial-Options.html

That's a pity, because it really limits (IMHO) the usage of such "script mode"
of Emacs. We can't afford rewriting all calls to `message' into calls to
`princ (format ...)'. Plus, doing so, moves messages in a non-chronological
way (like the "weird flushes" above).

Any workaround in sight?

Best regards,
  Seb

-- 
Sebastien Vauban

Re: Regression tests and stderr

[Achim Gratz]

"Sebastien Vauban"
<wxhgmqzgwmuf@spammotel.com> writes:
> Pretty good. One failure, though, which I'll try to investigate (later on).

That's a new test from Nicolas which seems to need some testing
itself... :-)

> What's weird is the output that comes _after_ the above summary:

Let me guess, you are on a Windows console or running cmd in a shell
buffer?  Then the order of output between STDOUT / STDERR is not
predictable since by default both streams will be buffered.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Samples for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra

Re: Regression tests and stderr

[Sebastien Vauban]

Hi Achim,

Achim Gratz wrote:
> "Sebastien Vauban"
> <wxhgmqzgwmuf-geNee64TY+gS+FvcfC7Uqw@public.gmane.org> writes:
>> Pretty good. One failure, though, which I'll try to investigate (later on).
>
> That's a new test from Nicolas which seems to need some testing
> itself... :-)

Just saw your reporting as well... Thanks for pointing out anyway.

>> What's weird is the output that comes _after_ the above summary:
>
> Let me guess, you are on a Windows console or running cmd in a shell
> buffer?  Then the order of output between STDOUT / STDERR is not
> predictable since by default both streams will be buffered.

On Windows, yes. But in a MinTTY terminal with Cygwin.

I never have seen mixed (delayed) streams with other applications, but they
may be less verbose: exit on error, for example, while here a lot of output is
written anyway.

The delay is a problem. The fact that normal messages (vs only calls to
`error') are sent to `stderr' is another big, I think. We can't grep through
errors only, or check for some file to be empty (when redirecting stderr to a
file), or...

Best regards,
  Seb

-- 
Sebastien Vauban