Captured output not displayed for functions from the outer test

[pganssle]

Normally when a function fails, the captured output from the whole function is printed in the output results, but with subtests, pytest is only outputting the captured logs and output from the specific subtest that failed. This makes some sense, but I think it would make more sense to show the captured output from the parts of the test not under a subtest. For example:

def my_function():
    print("This function is critical!")
    return 3, 4

def test_with_subtests(subtests):
    a, b = my_function()

    with subtests.test(a):
        assert a == 5

    with subtests.test(b):
        assert b == 6

This doesn't print anything, but the logs from my_function may be important, since the results of that function are used in the subtests.

I think it would make sense to display output from anything in the enclosing scope of a given subtest, so for example:

def test_with_subtests(subtests):
    print(1)
    for i in range(5):
        with subtests.test(i):
            print(f"Subtest: {i}")
            assert i < 4

    with subtests.test("Nested"):
        print("Nested")
        with subtests.test("Failing"):
            print("Fail")
            assert False

        with subtests.test("Succeeding"):
           print("Success")
           assert True

I would want this to show

1
4
Nested
Fail

Or if the output is not combined at the test level, then for subtest 4, I'd want:

1
4

And for subest "Nested/Failing" I'd want:

1
Nested
Fail

This is somewhat related to #11 — I think tests should probably be considered failures if any subtests nested under them fail, and this is an example of one reason it makes sense to think of things that way.

[nicoddemus]

Thanks @pganssle for the report.

I think tests should probably be considered failures if any subtests nested under them fail, and this is an example of one reason it makes sense to think of things that way.

Yeah, I agree.

The initial reasoning was to mirror how unittest handles subtests, but I think we can have different output with the subtests fixture, to better match how pytest users expect things to work.