Saving multiple figures in one test

[gopalakrishna]

I need to test multiple figures that are generated by a method for correctness. Using @pytest.mark.mpl_image_compare does not enable me to do that as it does not save multiple figures. Any way to do that?

Right now my thought is having dummy tests that return these figures individually from a shared variable and validate it one by one in different tests

[ConorMacBride]

This plugin can only compare one figure per test. However, you should be able to use @pytest.mark.parametrize to run the same test function multiple times with different input parameters, and compare to each of their figures.

Your solution is another common way to solve this, as you can define a function such as create_figure(...) which generates and returns a figure. Inside individual test functions you can then do return create_figure(...) to test the specific figure. Here's an example of something like this.

[gopalakrishna]

Thanks for the reply. I am trying something like the following

`

class TestPCA:
    @classmethod
    def setup_class(cls):
        pca.run_pca_on_csv("./myfile.csv")
        for i in plt.get_fignums():
            plt.figure(i)
            plt.savefig('figure%d.png' % i)

    @pytest.mark.mpl_image_compare(baseline_dir='baseline_images')
    def test_first_plot(self):
        return plt.figure(1)

    @pytest.mark.mpl_image_compare(baseline_dir='baseline_images')
    def test_second_plot(self):
        return plt.figure(2)

    @pytest.mark.mpl_image_compare(baseline_dir='baseline_images')
    def test_third_plot(self):
        return plt.figure(3)

`

And, I see that savefig call saves all the images. But, when I run

pytest -s --mpl-generate-path=baseline_images

there is no image captured. I can see that the 3 tests are picked by pytest

` platform win32 -- Python 3.10.2, pytest-7.0.0, pluggy-1.0.0

Matplotlib: 3.5.1

Freetype: 2.6.1

rootdir: D:\Work\python_test_automation, configfile: setup.cfg, testpaths: test

plugins: mpl-0.13

collected 3 items

`

If I move the method test_(first/second/third)_plot outside the class the baseline images are captured. Sorry, to deviate from the topic of the issue but I am trying to get this working for the first time.

[ConorMacBride]

The figures may not be available in the tests due to this:

pytest-mpl/pytest_mpl/plugin.py

Lines 617 to 622 in e1bf8dd

	# In some cases, for example if setup_method is used,
	# original appears to belong to an instance of the test
	# class that is not the same as args[0], and args[0] is the
	# one that has the correct attributes set up from setup_method
	# so we ignore original.__self__ and use args[0] instead.
	fig = original.__func__(*args, **kwargs)

Not 100% sure as I don't often use test classes. Does seem like a bug though.

Instead of the setup_class, a pytest fixture outside of the class which returns a list of Figure objects should hopefully work. (And scope it to the class or module so it doesn't recreate all the figures for every test method.)

[gopalakrishna]

Thanks for the reply. This is a bug indeed.

For my immediate need, I am using matplotlib.test now. There is another bug(or maybe intended feature) that all the figures are closed that does not allow me to use cached figures across tests. Matplotlib.tests allows specifying multiple images to compare against hence using that for now.

[ConorMacBride]

You're welcome. The core Matplotlib decorator is probably easiest if a single method/function produces multiple figures. This plugin is designed for one figure returned per test function, which makes reporting the results and generating a HTML report easier. The closing of figures is intended for convenient memory management, although in theory there could be an option to disable this if needed.

For future reference, I just had a quick look to see what the issue was. Looks like it's caused by how figure numbers are scoped in Matplotlib. It may be because the active figure manager when setup_class is run is not the same as when test_*_plot are run. (Possibly due to the comment I linked to earlier.) Manually linking them in a figs attribute seems to solve this for me. The following code should therefore work.

class TestPCA:
    @classmethod
    def setup_class(cls):
        pca.run_pca_on_csv("./myfile.csv")
        cls.figs = [plt.figure(i) for i in (1, 2, 3)]

    @pytest.mark.mpl_image_compare(baseline_dir='baseline_images')
    def test_first_plot(self):
        return self.figs[0]

    @pytest.mark.mpl_image_compare(baseline_dir='baseline_images')
    def test_second_plot(self):
        return self.figs[1]

    @pytest.mark.mpl_image_compare(baseline_dir='baseline_images')
    def test_third_plot(self):
        return self.figs[2]