pytest -x doesn't break on first failing subtest

[pganssle]

I have noticed that when running pytest -x (i.e. stop after the first failure) on a test with failing subtests, the entire test will run - the test won't abort after the first failing subtest. For example, with a test like this:

import pytest

def test_with_subtests(subtests):
    for i in range(10):
        with subtests.test(msg="Message", i=i):
            assert i == 0

The short test summary for pytest -x shows 9 failures instead of 1:

$ pytest -x
(tmp) [/tmp/tmp.1B06JKxL9V]$ pytest -x tests.py
================================================== test session starts ===================================================
platform linux -- Python 3.8.1, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
rootdir: /tmp/tmp.1B06JKxL9V
plugins: subtests-0.3.0
collected 1 item                                                                                                         

tests.py .FFFFFFFFF.                                                                                               [100%]

< skipping failures >
================================================ short test summary info =================================================
FAILED tests.py::test_with_subtests - assert 1 == 0
FAILED tests.py::test_with_subtests - assert 2 == 0
FAILED tests.py::test_with_subtests - assert 3 == 0
FAILED tests.py::test_with_subtests - assert 4 == 0
FAILED tests.py::test_with_subtests - assert 5 == 0
FAILED tests.py::test_with_subtests - assert 6 == 0
FAILED tests.py::test_with_subtests - assert 7 == 0
FAILED tests.py::test_with_subtests - assert 8 == 0
FAILED tests.py::test_with_subtests - assert 9 == 0
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 9 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================== 9 failed, 1 passed in 0.08s ===============================================

I've been using pytest-subtests quite extensively with the reference implementation for PEP 615, and some tests have dozens of subtests (e.g. this one, where the subtest is basically used for parametrization), and I'd like to be able to quickly stop the tests and get short feedback if something is broken, but instead I get dozens of failure messages.

I assume there's a related but possibly trickier question of when you set --max-failures to some number other than 1, are you counting subtests or are you counting top-level tests, but I only ever use -x to mean "stop immediately on first failure", and this is defeating my intuitions on that use case.

[Celeborn2BeAlive]

Same issue here, it makes it quite hard to iterate on a single large test having many small subtests (most of my subtests are just a single assert line)