Allow users to filter tests using command-line arguments

[cfinegan]

This pull request changes the behavior of the define-check macro so that it examines the current-command-line-arguments, which it interprets as a set of names, files, and line numbers to run. More specific information about the filtering behavior is available in the docs under "Filtering Tests with Command-line arguments."

[jeapostrophe]

What if not numberable string

[cfinegan]

Fixed this so it ignore invalid line numbers.

[jeapostrophe]

@jackfirth 's recent changes to the data structures broke your in-development version, so please update your fork and fix it up. See the Travis CI log for the problem.

[bennn]

Do test-case and test-suite set the name field?

(So, if I have (test-case "foo" ....) will raco test foo file.rkt run only that test case?)

[bennn]

Ooops just realized I need to write raco test ++arg foo file.rkt.

It would be good to have an example of raco test .... in these docs.

[bennn]

file number

line number

[cfinegan]

Thanks, fixed this.

[rmculpepper]

I don't think this feature should be built into rackunit. If you want to conditionally execute parts of your test suite, say so with when and unless and perhaps a DSL built on top of those features. Here's a sketch of one possible form:

(filtering-begin
  #:name "name"
  #:tags '(...)
  #:when ...expression...
  body ...)

The filtering-begin macro could use the name, tags, when-condition, etc (together with information from the command-line arguments or environment variables) to decide whether to execute its body. You could even have it wrap each body expression in something that checked the line number.

---

Some other comments:

The granularity for selective execution should be test cases, not checks. A check might guard another check or an action that shouldn't be executed if the first check fails. In contrast, test cases are supposed to be independent.

A library should probably not be parsing command-line arguments, and especially not with such generic names. An environment variable is more idiomatic for adjusting library behavior.

In any case, the command-line arguments (or environment variable) should not be re-parsed/processed each time a check executes. Instead, the library should define another parameter (current-execution-filter?) and initialize that parameter once when the library is loaded.

If this must be tied into rackunit, it should be implemented via the existing current-test-case-around parameter. Add a library (rackunit/filter-execution?) that defines and initializes current-execution-filter, modifies current-test-case-around to consult it, etc.

[samth]

I disagree with @rmculpepper here. Handling selective execution is a big part of making a nice-to-use testing system, and command line parsing is a very effective way of doing this.

My experience with py.test has been really positive. This feature is built in there, and you just provide a simple command line argument to specify it.

[jeapostrophe]

My thoughts, @rmculpepper

Re: check vs test-case --- I think that most Rackunit users don't use test-case and just put a bunch of check-equal?, etc in their test submodule.

Re: parsing & reparsing --- I think your idea is fine. Personally, I would implement it with a global variable and a cache, rather than a parameter. @cfinegan can you do @rmculpepper 's idea?

Re: command line vs env variable --- I think that they are orthogonal and I was looking for something with good mouth feel. My preference was "raco test file --- name" but @mflatt objected so we have "raco test ++arg name file" which feels better to the taste than "PLT_RACKUNIT_FILTER=name raco test file"

Are you blocking the change or are you convinced in any way?

[rmculpepper]

I'm unconvinced. I'll write a more detailed response later tonight.

[rmculpepper]

I have two main problems with the current design and implementation.

1. it doesn't avoid much execution (not useful)
2. it breaks tests that already use command-line arguments (like mine)

First, because the decision of whether to execute or not happens within individual checks and after the evaluation of their arguments, very little evaluation is actually avoided. Suppose we have the following tests, and suppose the we have indicated that we want to run only the first one:

(check-equal? (expensive-computation 1 2) 3)
(check-equal? (expensive-computation 3 4) 7)
(test-case "resource test"
  (define r (create-resource))
  (check-equal? (expensive-computation 3 4)
                (access-resource 'a)))

The second expensive computation still gets run. The resource r still gets created and accessed, and the third expensive computation still gets run. If that's the intended behavior, then it would be much better to just implement a filter on the reporting sent to stderr. Otherwise, the right granularity for selective execution is the test case, not the check.

Second, the use of command-line arguments breaks tests that already use command-line arguments, such as the db tests. I introduced an error into the db tests and ran racket -l tests/db/all-tests sl pg (that is, execute the tests with sqlite3 and postgresql drivers). I got back "138 successes, 0 failures ... 98 successes, 0 failures..." because the presence of command-line arguments caused rackunit to skip all of the checks inside the test cases.

My recommendation is to build a completely separate filtered execution macro; I don't think it belongs in the rackunit core. (But possibly as a rackunit library.) Wrapping the form around test cases or top-level checks or sequences thereof solves the first problem, and making it opt-in (as opposed to changing the meaning of existing rackunit programs) helps with the second.

[jackfirth]

I agree with @rmculpepper. Having said that, @samth could you open an issue describing what sort of test filtering you'd like for the Typed Racket tests (I assume TR is your motivation here?)

I think if we had a concrete use case it'd be much easier to work out the right API for this.

[mfelleisen]

@samth 1. I use rackunit a lot and I don’t understand why a library should parse command-line args. Can you explain? 2. When I ran Hell a couple of years ago, I discovered the Python testing and running facilities. Matthew’s reaction to my write-up from back then was just write: sure we need something like this but let me think about it. — And then he came up with submodules, which are far superior and general to Python’s approach. I am hoping we can get to a similar place here.