Pyret-style testing

[jackfirth]

In the Pyret language, tests may be included within the definition of a function using a where block:

fun sum(l):
  cases (List) l:
    | empty => 0
    | link(first, rest) => first + sum(rest)
  end
where:
  sum([list: ]) is 0
  sum([list: 1, 2, 3]) is 6
end

This runs the two sum() calls in the where block as test cases. This even works for nested functions, with nested test cases run whenever the enclosing function's test cases result in a call to the enclosing function:

fun make-adder(num1 :: Number):
  fun result(num2 :: Number):
    num1 + num2
  where:
    result(5) is num1 + 5
    result(10) is num1 + 10
  end
  result
where:
  make-adder(3)(6) is 9
  make-adder(4)(2) is 6
end

example originally from this mailing list discussion

Here, the nested result function's test cases are run twice: once each when make-adder(3)
and make-adder(4) are called while testing the enclosing make-adder function. I'd like something like this for Racket, but there's a few things I'm unsure of:

1. Would this work with test submodules? Using module+ test avoids runtime dependencies on testing frameworks and I'd like a solution to this problem to preserve that feature.
2. Should this be test-framework-agnostic or only work with RackUnit? Should it be a main-distribution package? If not, should RackUnit recommend it's use or otherwise endorse it?
3. What should failure output for a nested test case look like?
4. Many nested tests can result in a combinatoric explosion of test cases. Pyret by default doesn't run all cases. Should a Racket implementation do something similar? How would a test writer control this behavior?
5. Is this the best way to handle tests for nested functions? What if inner tests aren't run because outer tests don't call them? Would it encourage clearer code to simply disallow tests for nested functions, since Racketeers typically choose modules over nested functions for organization anyway?

Discussion highly welcome, cc @mfelleisen

[sorawee]

One possibility: create a syntax parameter, say test, that in regular mode will disappear, but when a certain condition is met (e.g., some environment variable is set), will splice its body into the enclosing form. Then we can write something like this?

(test (require rackunit))

(define (make-adder num-1)
  (define (result num-2)
    (+ num-1 num-2))
  (test
    (check-equal (result 5) (+ num-1 5))
    (check-equal (result 10) (+ num-1 10)))
  result)
(test
  (check-equal ((make-adder 3) 6) 9)
  (check-equal ((make-adder 4) 2) 6))

The problem is, something will need to provide test. Racket probably should not do that. But if rackunit does that then the entire rackunit will be needed at runtime too...

[mfelleisen]

I just spoke with BenL (one of the major compiler writers) to clarify the semantics of nested tests in Pyret. A nested test is run every time the function is run (and apparently there is no way to turn those off). A top-level test is dealt with like something in module+-test. Library tests are thus ignored if code imports a library.

What this means philosophically and technically is that these tests are contracts. In the past I have suggested to think of contracts as generalizations of predicate tests and of the latter as generalizations of unit tests. Of course when A generalizes B, A can express B (functionally at least, if not non-functional properties such as "only when module+ test is required").

I conjecture that we could compile nested tests into Racket contracts locally, possibly with a macro, with a good syntax design, and that should be done.

;; - - -

It cannot work with test submodules for a number of reasons. The most important one is that the nested tests close over locally defined vars, including function parameters. So you would need some form of lambda lifting and protocol between tests and functions. So the "cannot" means, at a minimum, that this sounds way too expensive.

Given the above analysis, I would say this should go into the contract package not the rackunit one.

Finally this is the first time I see a reason to be able to run off contracts (partially). :-)

;; - - -

@rfindler @chrdimo