Support marker keyword arguments in marker expression (test selection via -m)

[lovetheguitar]

What's the problem this feature will solve?

It's possible to create custom markers like so:

@pytest.mark.my_device_under_test

This marker can the be used to de/select tests via marker expressions, e.g.:
pytest -m "not my_device_under_test"

It is also possible to define keyword arguments for custom markers, e.g.:

@pytest.mark.my_device_under_test(with_serial="1234")

Now, it would be amazing to select tests further depending on their keyword arguments + values, e.g.:

pytest -m "my_device_under_test(with_serial="1234)"

So only tests with that specific marker + that specific keyword argument+value would be run.

Describe the solution you'd like

Marker keyword argument selection in pytest's marker expression.

Alternative Solutions

• There was a nose test plugin, which supported something similar here. This could be ported to pytest but comes with it's own attr decorator.

• A pytest plugin could probably do this in an extra step as well, but would somehow need to disable pytest's internal marker selection plugin here as it would collide/fail otherwise.

Additional context

When skimming through the code I saw that @bluetech implemented a custom parsers some years ago. I'm not very familiar with the topic, as in how much effort it would be support such feature. I'd be interested to contribute though.

[bluetech]

The feature request makes sense to me, I think it would be great to allow matching on mark arguments (there are two kinds -- positional and keyword arguments), and something like the function call syntax seems natural for this.

For the match expression grammar, it shouldn't pose a big challenge to add. We would need to decide whether the argument values (the 1234 in your example) can be just idents (simple), or more complex, e.g. to support my_device_under_test(with_serial=1234 or 5678) etc., which would be more complex.

The bigger implementation issue would be the evaluation of the expression. Currently, to optimize the evaluation, we compile the expression to a python AST, then eval the AST with an environment ("locals") where every ident returns either True (if matches) or False. This will need to be extended to support the function-call syntax. As a quick thought, we could replace the locals hack with injecting some $match symbol into the eval environment which has the form matcher(name, *args: str, **kwargs: str) (assuming we only allow idents).

[nicoddemus]

If we are to implement this, I propose we keep it simple, as it would be possible to also write my_device_under_test(with_serial=1234) or my_device_under_test(with_serial=5678).

[The-Compiler]

I can see the appeal of this, and I agree in theory it makes sense to keep things close to a Python-like syntax.

However, at the same time, I'm also worried this might be a bottomless pit. Where do we stop in terms of types of arguments we support? Allowing my_device_under_test(with_serial=1234) or use_case("ABC-1234") seems obvious (and would indeed cover things I've seen various companies implement custom arguments/filtering via plugin hooks for).

However, what about, say, enums? Surely there's an argument to be made for enums, as in size(Size.end2end). But then where is Size coming from? I'm playing devil's advocate here, but if we allow enums, why wouldn't we also allow, I don't know, custom dataclass instances (in the sense of device(Multimeter(...)))? Etc. etc., and then at some point we're back to essentially reimplementing the eval() solution we had at some point.

Maybe though, we could limit this to int / str / bool / None or somesuch, or perhaps what ast.literal_eval supports. But still, while I can absolutely see the need and usefulness, it seems very tempting to take this too far over time.

[ckutlu]

Running into this thought myself, I was wondering if we could instead consider a bit of a tangent to avoid pitfalls mentioned by @The-Compiler. How about introducing a new interface called pytest.mark.annotated(*args: str)?

It would look like:

@pytest.mark.annotated("annotation1", "annotation2", ...)
def test_my_function(...):
    pass

The selection can be with a slightly specialized syntax:

pytest -m [annotation1]

which can be combined with other non-annotated markers if needed:

pytest -m "awesome and [annotation1] and [annotation2]"

This is surely limiting, but maybe that's not a bad thing. For my use case, this would help with adding bug number/jira ticket annotations to test cases.