Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

signals_allclose doesn't plot allclose_tolerances #1638

Open
hunse opened this issue Sep 10, 2020 · 2 comments
Open

signals_allclose doesn't plot allclose_tolerances #1638

hunse opened this issue Sep 10, 2020 · 2 comments

Comments

@hunse
Copy link
Collaborator

hunse commented Sep 10, 2020

Describe the bug

allclose_tolerances allow backends to override default allclose tolerances for specific tests. These can work with signals_allclose by making it use the allclose fixture rather than np.allclose, but it still uses its default tolerances for plotting. This means that in the plots, the error can look like it's outside tolerances, while the test passes because the error is within the overridden tolerances.

Expected behavior

The allclose fixture supports different tolerances each time it's called, by passing a list to allclose_tolerances. Supporting this seems difficult. More straightforward would be to use _get_allclose_overrides (from pytest-allclose) in signals_allclose, and just use the first (or last?) tolerance if applicable. (I think all our tests that use signals_allclose just have one allclose comparison.)

@tbekolay
Copy link
Member

I wonder if we should move the stuff in signals_allclose into the pytest-allclose repo? I.e. give an easy way to generate those plots by passing plot=True or something like that to allclose. If we're going to touch that code anyway, that is.

@hunse
Copy link
Collaborator Author

hunse commented Sep 11, 2020

I like that idea. It definitely seems like a more sensible place for that code to live.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

2 participants