From: Bruno Barbier <brubar.cs@gmail.com>
To: emacs-orgmode@gnu.org
Subject: [PATCH] Add tests for ob-haskell (GHCi)
Date: Sun, 19 Mar 2023 10:12:50 +0100 [thread overview]
Message-ID: <6416d214.5d0a0220.d9c1.54aa@mx.google.com> (raw)
[-- Attachment #1: Type: text/plain, Size: 1059 bytes --]
Hi all,
I've collected some tests about ob-haskell, using GHCi.
This patch is not ready for final review. It's a preliminary version.
These tests are about GHCi, and only about GHCi (that's why I named
the file 'test-ob-haskell-ghci.el').
Some tests are revealing bugs; they are flagged with:
:expected-result :failed
All the tests will randomly fail; that is an ob-haskell bug (see
the command 'test-ob-haskell-ghci.el' to make them fail).
Some tests might be questionable, revealing more design choices and/or
backend capabilities than the expected behavior.
Solving the random failure bug should be a priority, in my opinion, as
it makes the testing process unreliable. Adding support for sessions
would also be a good thing, so that one test doesn't contaminate others.
FTR, here are the tools that I have used:
| emacs-version | 29.0.60 |
| org-version | main@4cad6c8ea (Mar 16 2023) |
| haskell-mode | master@20d4e23 (Mar 4 2023) |
| ghci | 9.0.2 |
Bruno
[-- Attachment #2: Add tests for haskell (GHCi) --]
[-- Type: text/x-patch, Size: 14254 bytes --]
From c085fac2fcb429f7e643df8e4fff3a4ae1665d07 Mon Sep 17 00:00:00 2001
From: Bruno BARBIER <brubar.cs@gmail.com>
Date: Fri, 18 Nov 2022 20:14:20 +0100
Subject: [PATCH] ob-haskell: Add tests for GHCi
testing/lisp/test-ob-haskell-ghci.el: New file.
---
testing/lisp/test-ob-haskell-ghci.el | 428 +++++++++++++++++++++++++++
1 file changed, 428 insertions(+)
create mode 100644 testing/lisp/test-ob-haskell-ghci.el
diff --git a/testing/lisp/test-ob-haskell-ghci.el b/testing/lisp/test-ob-haskell-ghci.el
new file mode 100644
index 000000000..4b1e4669b
--- /dev/null
+++ b/testing/lisp/test-ob-haskell-ghci.el
@@ -0,0 +1,428 @@
+;;; test-ob-haskell-ghci.el --- tests for ob-haskell.el GHCi -*- lexical-binding: t; -*-
+
+;; Copyright (c) 2023 Free Software Foundation, Inc.
+
+;; Authors: Bruno BARBIER <brubar.cs@gmail.com>
+
+;; This program is free software; you can redistribute it and/or modify
+;; it under the terms of the GNU General Public License as published by
+;; the Free Software Foundation, either version 3 of the License, or
+;; (at your option) any later version.
+
+;; This program is distributed in the hope that it will be useful,
+;; but WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+;; GNU General Public License for more details.
+
+;; You should have received a copy of the GNU General Public License
+;; along with this program. If not, see <https://www.gnu.org/licenses/>.
+
+;;; Commentary:
+;;
+
+;;;; Useful references
+;;
+;; - https://orgmode.org/worg/org-contrib/babel/languages/lang-compat.html
+
+;;;; FIXME: Random failures
+;;
+;; To increase the chances of failure when running tests, you can use this command line:
+;;
+;; (for I in 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10 0 1 2 3 4 5 6 7 8 9 10; do make 'BTEST_OB_LANGUAGES=haskell' BTEST_RE='haskell' test-dirty & done) 2>&1 | grep FAILED
+;;
+
+;;;; Status
+;;
+;; All the tests should succeed (except for random failures); those
+;; flagged with ":expected-result :failed" are known
+;; limitations/bugs. Tested with (2023-03-18):
+;;
+;; | emacs-version | 29.0.60 |
+;; | org-version | main@4cad6c8ea (Mar 16 2023) |
+;; | haskell-mode | master@20d4e23 (Mar 4 2023) |
+;; | ghci | 9.0.2 |
+
+
+;;; Code:
+;;
+
+(require 'org-test "../testing/org-test")
+(org-test-for-executable "ghci")
+(unless (featurep 'haskell-mode)
+ (signal 'missing-test-dependency "haskell-mode"))
+
+
+;;; Helpers
+;;
+
+(defun test-ob-haskell-ghci--with-global-session-worker (todo)
+ "See `test-ob-haskell-ghci--with-global-session-worker'."
+ (when (get-buffer "*haskell*")
+ (error "A buffer named '*haskell*' exists. Can't safely test haskell blocks"))
+ (unwind-protect (funcall todo)
+ ;; Kill the "*haskell*" buffer to not pollute other tests.
+ (when-let ((hb (get-buffer "*haskell*")))
+ (with-current-buffer hb
+ (let ((kill-buffer-query-functions nil)
+ (kill-buffer-hook nil))
+ (kill-buffer hb))))))
+
+(defmacro test-ob-haskell-ghci-with-global-session (&rest body)
+ "Eval BODY in a new session, then destroy the session.
+The library ob-haskell doesn't implement session yet. It will
+always use a buffer named \"*haskell*\". We kill that buffer
+after the source block execution. To be safe, we fail if such a
+buffer already exists."
+ `(test-ob-haskell-ghci--with-global-session-worker (lambda () ,@body)))
+
+(defun test-ob-haskell-ghci (args content &optional preamble unprotected)
+ "Execute the code block CONTENT in a new GHCi session; return the result.
+Add ARGS to the code block argument line. Insert PREAMBLE
+before the code block. When UNPROTECTED is non-nil, don't control
+which session is used (i.e. don't call
+`test-ob-haskell-ghci--with-global-session-worker')."
+ (when (listp content)
+ (setq content (string-join content "\n")))
+ (unless preamble
+ (setq preamble ""))
+ (let ((todo (lambda ()
+ (org-test-with-temp-text
+ (concat preamble "\n" "#+begin_src haskell :compile no "
+ args "\n" "<point>" content "\n#+end_src")
+ (org-babel-execute-src-block)))))
+ (if unprotected (funcall todo)
+ (test-ob-haskell-ghci-with-global-session (funcall todo)))))
+
+
+;;; Tests
+
+
+;;;; Temporary tests
+;;
+
+(ert-deftest ob-haskell/check-version ()
+ "Check GHCi version.
+Only tested with GHCi 9.0.2."
+ (should (equal '("linux" "ghc" 9) (test-ob-haskell-ghci "" "
+import System.Info
+import Data.Version
+(os, compilerName, (versionBranch fullCompilerVersion)!!0)
+"))))
+
+
+;;;; Hello Worlds.
+;;
+
+(ert-deftest ob-haskell/hello-world-value-pure ()
+ (should (equal "Hello World!"
+ (test-ob-haskell-ghci "" "\"Hello World!\""))))
+
+(ert-deftest ob-haskell/hello-world-value-IO ()
+ (should (equal "Hello World!"
+ (test-ob-haskell-ghci "" "return \"Hello World!\""))))
+
+(ert-deftest ob-haskell/hello-world-output ()
+ (should (equal "Hello World!"
+ (test-ob-haskell-ghci ":results output" "putStrLn \"Hello World!\""))))
+
+(ert-deftest ob-haskell/hello-world-output-nothing ()
+ :expected-result :failed
+ (should (equal ""
+ (test-ob-haskell-ghci ":results output" "return \"Hello World!\""))))
+
+(ert-deftest ob-haskell/hello-world-output-multilines ()
+ :expected-result :failed
+ (should (equal "Hello World!"
+ (test-ob-haskell-ghci ":results output" "
+:{
+main :: IO ()
+main = putStrLn \"Hello World!\"
+:}
+
+main
+"))))
+
+;;;; Sessions
+;;
+
+(ert-deftest ob-haskell/sessions-must-not-share-variables ()
+ "Sessions must not share variables."
+ :expected-result :failed
+ (test-ob-haskell-ghci-with-global-session
+ (test-ob-haskell-ghci ":session s1" "x=2" nil :unprotected)
+ (should (equal 2 (test-ob-haskell-ghci ":session s1" "x" nil :unprotected)))
+ (test-ob-haskell-ghci ":session s2" "x=3" nil :unprotected)
+ (should-not (equal 3 (test-ob-haskell-ghci ":session s1" "x" nil :unprotected)))
+ ))
+
+(ert-deftest ob-haskell/no-session-means-one-shot-sessions ()
+ "When no session, use a new session."
+ :expected-result :failed
+ (test-ob-haskell-ghci-with-global-session
+ (test-ob-haskell-ghci "" "x=2" nil :unprotected)
+ (should-not (equal 2 (test-ob-haskell-ghci "" "x" nil :unprotected)))))
+
+
+;;;; Values
+;;
+
+(ert-deftest ob-haskell/value-is-the-last-expression ()
+ "Return the value of the last expression."
+ (should (equal 3 (test-ob-haskell-ghci "" '("1" "1+1" "1+1+1"))))
+ (should (equal 3 (test-ob-haskell-ghci "" '("x=1" "y=1+1" "x+y")))))
+
+(ert-deftest ob-haskell/value-is-the-last-expression-2 ()
+ "Return the value of the last expression."
+ (should (equal 7 (test-ob-haskell-ghci "" "
+putStrLn \"a string\"
+return \"useless\"
+3+4
+"))))
+
+
+
+(ert-deftest ob-haskell/eval-numbers ()
+ "Evaluation of numbers."
+ (should (equal 7 (test-ob-haskell-ghci "" "7")))
+ (should (equal 7.5 (test-ob-haskell-ghci "" "7.5")))
+ (should (equal 10.0 (test-ob-haskell-ghci "" "10::Double")))
+ (should (equal 10 (test-ob-haskell-ghci "" "10::Int"))))
+
+
+(ert-deftest ob-haskell/eval-strings ()
+ "Evaluation of strings."
+ (should (equal "a string" (test-ob-haskell-ghci "" "\"a string\""))))
+
+
+;;;; Local variables
+(ert-deftest ob-haskell/let-one-line ()
+ "Local definitions on one line."
+ (should (equal 6 (test-ob-haskell-ghci "" "let { x=2; y=3 } in x*y"))))
+
+(ert-deftest ob-haskell/let-multilines-1 ()
+ "Local definitions on multiple lines."
+ :expected-result :failed
+ (should (equal 6 (test-ob-haskell-ghci "" "
+:{
+ let { x=2
+ ; y=3
+ }
+ in x*y
+:}
+"))))
+
+(ert-deftest ob-haskell/let-multilines-2 ()
+ "Local definitions on multiple lines, relying on indentation."
+ :expected-result :failed
+ (should (equal 6 (test-ob-haskell-ghci "" "
+:{
+ let x=2
+ y=3
+ in x*y
+:}
+"))))
+
+;;;; Declarations with multiple lines.
+(ert-deftest ob-haskell/decl-multilines-1 ()
+ "A multiline declaration, then use it."
+ (should (equal 3 (test-ob-haskell-ghci "" "
+:{
+let length' [] = 0
+ length' (_:l) = 1 + length' l
+:}
+length' [1,2,3]
+"))))
+
+(ert-deftest ob-haskell/decl-multilines-2 ()
+ "A multiline declaration, then use it."
+ (should (equal 5 (test-ob-haskell-ghci "" "
+:{
+length' :: [a] -> Int
+length' [] = 0
+length' (_:l) = 1 + length' l
+:}
+
+length' [1..5]
+"))))
+
+
+(ert-deftest ob-haskell/primes ()
+ "From haskell.org."""
+ :expected-result :failed
+ (should (equal '(2 3 5 7 11 13 17 19 23 29)
+ (test-ob-haskell-ghci "" "
+:{
+primes = filterPrime [2..] where
+ filterPrime (p:xs) =
+ p : filterPrime [x | x <- xs, x `mod` p /= 0]
+:}
+
+take 10 primes
+"))))
+
+;;;; Lists
+;;
+
+(ert-deftest ob-haskell/a-simple-list ()
+ "Evaluation of list of values."
+ (should (equal '(1 2 3) (test-ob-haskell-ghci "" "[1,2,3]"))))
+
+
+(ert-deftest ob-haskell/2D-lists ()
+ "Evaluation of nested lists into a table."
+ (should (equal '((1 2 3) (4 5 6))
+ (test-ob-haskell-ghci "" "[[1..3], [4..6]]"))))
+
+(ert-deftest ob-haskell/2D-lists-multilines ()
+ "Evaluation of nested lists into a table, as multilines."
+ :expected-result :failed
+ (should (equal '((1 2 3) (4 5 6))
+ (test-ob-haskell-ghci "" "
+:{
+[ [1..3]
+, [4..6]
+, [7..9]
+]
+:}
+"))))
+
+
+;;;; Data tables
+;;
+
+(ert-deftest ob-haskell/int-table-data ()
+ "From worg: int-table-data."
+ (should (equal 10 (test-ob-haskell-ghci ":var t=int-table-data"
+ "sum [sum r | r <- t]"
+ "#+name: int-table-data
+ | 1 | 2 |
+ | 3 | 4 |"))))
+
+(ert-deftest ob-haskell/float-table-data ()
+ "From worg: float-table-data."
+ (should (equal 11.0 (test-ob-haskell-ghci ":var t=float-table-data"
+ "sum [sum r | r <- t]"
+ "#+name: float-table-data
+ | 1.1 | 2.2 |
+ | 3.3 | 4.4 |"))))
+
+(ert-deftest ob-haskell/string-table-data ()
+ "From worg: string-table-data."
+ (should (equal "abcd" (test-ob-haskell-ghci ":var t=string-table-data"
+ "concat [concat r | r <- t]"
+ "#+name: string-table-data
+ | a | b |
+ | c | d |"))))
+
+
+;;;; Not define errors
+;;
+
+(ert-deftest ob-haskell/not-defined ()
+ "Evaluation of undefined variables."
+ (should (string-match "Variable not in scope"
+ (test-ob-haskell-ghci "" "notDefined :: IO Int"))))
+
+(ert-deftest ob-haskell/not-defined-then-defined-1 ()
+ "Evaluation of undefined variables.
+This is a valid haskell source, but, invalid when entered one
+line at a time in GHCi."
+ (let ((r (test-ob-haskell-ghci "" "
+v :: Int
+v = 4
+")))
+ (should (and r (string-match "Variable not in scope" r)))))
+
+(ert-deftest ob-haskell/not-defined-then-defined-1-fixed ()
+ "Like not-defined-then-defined-1, but using the mutiline marks."
+ :expected-result :failed
+ (let ((r (test-ob-haskell-ghci "" "
+:{
+ v :: Int
+ v = 4
+:}
+")))
+ (should (eq nil r))))
+
+(ert-deftest ob-haskell/not-defined-then-defined-1-fixed-2 ()
+ "Like not-defined-then-defined-1, but using one line."
+ (should (eq nil (test-ob-haskell-ghci "" "v = 4 :: Int"))))
+
+
+
+(ert-deftest ob-haskell/not-defined-then-defined-2 ()
+ "Evaluation of undefined variables, followed by a correct one."
+ ;; ghci output is:
+ ;; | <interactive>:2:1-4: error:
+ ;; | • Variable not in scope: main :: IO ()
+ ;; | • Perhaps you meant ‘min’ (imported from Prelude)
+ ;; | Hello, World!
+ ;; and ob-haskell just reports the last line "Hello, World!".
+ (should (string-match "Variable not in scope"
+ (test-ob-haskell-ghci ":results output" "
+main :: IO ()
+main = putStrLn \"Hello, World!\"
+main
+"))))
+
+;;;; Imports
+;;
+
+(ert-deftest ob-haskell/import ()
+ "Import and use library."
+ (should (equal 65 (test-ob-haskell-ghci "" "
+import Data.IORef
+r <- newIORef 65
+readIORef r
+"))))
+
+(ert-deftest ob-haskell/import-with-vars ()
+ "Import and use library with vars."
+ (should (equal 65 (test-ob-haskell-ghci ":var x=65" "
+import Data.IORef
+r <- newIORef x
+readIORef r
+"))))
+
+;;;; What is the result?
+;;
+
+(ert-deftest ob-haskell/results-value-1 ()
+ "Don't confuse output and values: nothing."
+ (should (equal nil (test-ob-haskell-ghci ":results value" "return ()"))))
+
+(ert-deftest ob-haskell/results-value-2 ()
+ "Don't confuse output and values: a list."
+ (should (equal '(1 2) (test-ob-haskell-ghci ":results value" "return [1,2]"))))
+
+(ert-deftest ob-haskell/results-value-3 ()
+ "Don't confuse output and values: nothing."
+ :expected-result :failed
+ (should (equal nil (test-ob-haskell-ghci ":results value" "putStrLn \"3\""))))
+
+(ert-deftest ob-haskell/results-value-4 ()
+ "Don't confuse output and values: nothing."
+ :expected-result :failed
+ (should (equal nil (test-ob-haskell-ghci ":results value" "
+putStrLn \"3\"
+return ()
+"))))
+
+
+;;;; GHCi commands
+;;
+
+(ert-deftest ob-haskell/ghci-type ()
+ "The ghci meta command ':type'."
+ (should (equal "3 :: Num p => p"
+ (test-ob-haskell-ghci ":results output" ":type 3"))))
+
+(ert-deftest ob-haskell/ghci-info ()
+ "The ghci meta command ':info' ."
+ (should (equal "repeat :: a -> [a] -- Defined in ‘GHC.List’"
+ (test-ob-haskell-ghci ":results output" ":info repeat"))))
+
+
+(provide 'test-ob-haskell-ghci)
+
+;;; test-ob-haskell-ghci.el ends here
--
2.39.1
next reply other threads:[~2023-03-19 9:13 UTC|newest]
Thread overview: 26+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-03-19 9:12 Bruno Barbier [this message]
2023-03-19 10:20 ` [PATCH] Add tests for ob-haskell (GHCi) Ihor Radchenko
2023-03-19 10:28 ` Ihor Radchenko
2023-03-19 10:32 ` Bruno Barbier
2023-03-22 10:16 ` Ihor Radchenko
2023-03-24 10:36 ` Ihor Radchenko
2023-03-25 10:01 ` Bruno Barbier
2023-03-26 9:09 ` Ihor Radchenko
2023-03-26 9:40 ` Bruno Barbier
2023-03-26 9:46 ` Ihor Radchenko
[not found] ` <notmuch-sha1-0807e1720f829950d42ef560bc30e56bd152766c>
2023-05-07 8:50 ` Bruno Barbier
2023-05-07 9:18 ` Ruijie Yu via General discussions about Org-mode.
2023-05-07 11:15 ` Bruno Barbier
2023-05-08 10:59 ` Ihor Radchenko
2023-05-21 7:40 ` Bruno Barbier
2023-06-02 8:44 ` Ihor Radchenko
2023-08-10 12:51 ` Ihor Radchenko
2023-08-25 19:10 ` Bruno Barbier
2023-09-07 14:21 ` Bruno Barbier
2023-09-08 8:23 ` Ihor Radchenko
2023-09-08 9:49 ` Bruno Barbier
2023-03-23 10:35 ` Ihor Radchenko
2023-03-23 21:01 ` ParetoOptimalDev
2023-03-23 21:30 ` ParetoOptimalDev
2023-03-24 10:40 ` Ihor Radchenko
2023-03-26 3:27 ` ParetoOptimalDev
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.orgmode.org/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=6416d214.5d0a0220.d9c1.54aa@mx.google.com \
--to=brubar.cs@gmail.com \
--cc=emacs-orgmode@gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://git.savannah.gnu.org/cgit/emacs/org-mode.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).