Repository navigation
Support cabal test-suite with shelltestrunner #11
Description
Activity
Sounds nice! Would you like to make a start on it ?
At the moment, the whole project is in one file.
I would need to split it into an executable and a library.
Do you think it makes sense ? Is it something acceptable ?
- Sure, that’s fine.
FWIW, I do basically the same thing now (this is my ShellTests.hs):
module Main where import System.Process ( CreateProcess( env ), createProcess, shell, waitForProcess ) import System.Exit ( exitWith ) main :: IO () main = do -- Get a CreateProcess object corresponding to our shell command. let createproc = shell "shelltest test/shell/*.test" -- But clear its environment before running the command. let empty_env_createproc = createproc { env = Just [] } -- Ignore stdin/stdout/stderr... (_,_,_,hproc) <- createProcess empty_env_createproc -- Now run the ProcessHandle and exit with its result. result <- waitForProcess hproc exitWith result
Granted that yours looks a lot nicer. Depending on libraries with cabal works a lot better than depending on executables, too. For example, hackage can't tell you that my project depends on "the shelltest executable being in
$PATH".Thanks for this idea back in 2017. Are you still using it ? Is there anything we need to do here ?
Perhaps add as a reproducible cookbook example to the https://github.com/simonmichael/shelltestrunner/blob/master/README.md ?
Yep, still using it, but struggling to remember what this issue is about =)
I think all the original report is asking for is a way to "do what the shelltest program would do," but within a library function. If there was a
shelltestfunction provided by a library underTest.ShellTest, then you could use the sample code in the first comment rather than the real code in my previous comment.Here's a sketch:
- Refactor the bulk of shelltest's
mainfunction intoTest.ShellTest.shelltest - Update shelltest's
mainto call that new function. - Ship
Test.ShellTestas a public library.
Then anyone who wants to use shelltest as part of a cabal test suite can call the library function instead of the executable (which is a bit more trouble).
- Refactor the bulk of shelltest's
Ah, so make shelltestrunner usable as a haskel library ? That I can understand, thanks for clarifying. I agree this could be useful.
In general, an executable can be used in a testsuite by
cabal testif it is declared as build dependency.
For an example, testing "myself" (agda2lagda) usinggoldplate, see: https://github.com/andreasabel/agda2lagda/blob/93d317c87bddf973e3fa6abd5bf24dcfa9a797f0/agda2lagda.cabal#L104-L115So, add
build-tool-depends: shelltestrunner:shelltestrunnerto the testsuite stanza. Then you cansystem "shelltestrunner" ...in the Main of the testsuite. (Example: https://github.com/andreasabel/agda2lagda/blob/93d317c87bddf973e3fa6abd5bf24dcfa9a797f0/test/Tests.hs#L5)Reacted by Simon MichaelAh, build-tool-depends does indeed solve half of the problem. I am somehow just learning about it today. But keep in mind that this issue predates cabal-2.0 =)
Using
system "shelltestrunner"isn't a great solution though. It's not really portable -- you have to feed it a command that will work on every possible shell, including Windows' cmd.exe -- and it's also a bit inefficient in that it creates a whole new shell process (sourcing.rcfiles and whatnot) to launch the command.With shelltestrunner, I usually have multiple
.testfiles in some subdirectory oftests/. What I'd like to do is to use a portable solution to find them all...import System.FilePath.Find ((==?), always, extension, find) find_sources :: IO [FilePath] find_sources = find always (extension ==? ".test") "tests/shell"
and then test them within the same process, without involving an OS shell at all, e.g. (in pseudocode):
main :: IO () main = do sources <- find_sources shelltest $ sources
Reacted by Andreas AbelAh, build-tool-depends does indeed solve half of the problem.
This was premature ☠️
The "configure" phase happily proceeds when the required
build-tool-dependsare missing. In this case...$ runghc Setup.hs configure --enable-tests -v3 ... Searching for shelltestrunner in path. Cannot find shelltestrunner on the pathreturns success (zero) on the command-line, rather than dying immediately as it would if a required library were absent.
$ runghc Setup.hs configure --enable-tests -v3I have not tested this since I am always using the
cabalcommand. Is there an issue on this problem in the cabal issue tracker?I have not tested this since I am always using the
cabalcommand. Is there an issue on this problem in the cabal issue tracker?This one is similar in spirit: haskell/cabal#5464
When I develop executables as part of a Haskell project, I have to do and repeat some manual tests to ensure that the executable behaviour is correct (e.g. status code, error message, output ...). This is tedious, and I think it could be automated nicely with shelltestrunner.
My proposal would be to support shelltestrunner in Cabal file as a test-suite.
Example of a Cabal test-suite with shelltestrunner support:
test/shelltest.hs
Shelltestrunner test files would be gathered in test/shelltest.
Shelltestrunner configuration could be set directly in shelltest.hs or in an external configuration file.