What is our contract around iris.tests

[lbdreyer]

I recently discovered that there are libraries that depend on Iris tests, i.e. they use iris as part of their code, but they also use some of the code from iris.tests in their own testing e.g. inherit from iris.tests.IrisTest.

This is of concern as I am not aware of any rules/guidance/contract about backwards compatibility around tests. I would have thought that semantic versioning only applied to the code. So perhaps it is just the case that we have no formal contract determiningin what will happen to testing code and we are free to change it as we wish.

I wanted to raise this as I know there are plans for changes to Iris' testing and we should consider how this may impact users. In particular I would be interested in your thoughts on:

1. Do semantic versioning rules apply to iris.tests
2. If not, how should we manage the upcoming testing changes (assuming they are significant), we could:
   a) Continue as planned and just make changes as needed for the Iris project
   b) Add a note in the whats new to warn of changes, but still make them anyway
   c) Try to identify libraries that are using Iris' tests and start up conversation about how this change should be managed.

Whatever we choose, it may also make sense to add something to the docs to explain what the contract around iris.tests is.

Another thing worth mentioning is that there are other packages that provide some testing utilities. For example numpy provides some test support through numpy.testing. These appear to be under some versioning contract as I have seen deprecation notices in this code (for example here). And xarray offer a similar thing with xarray.testing. We could consider offering a similar option to our users (iris.testing) and keep that separate to (iris.tests)?

[trexfeathers]

Two ideas that I personally like, in order of preference:

1. Stop shipping the tests within Iris, and find a case-specific way to run the tests for testing the UK Met Office stacks.
   (Don't ship the tests with iris #3416)
2. Rename to iris._tests, which is considered the Pythonic way to label something we are free to change at any time.

[lbdreyer]

It sounds like you are in favour of iris.tests not being used by users. I very much agree since I don't even consider it part of Iris' API (we don't document it!) so there shouldn't be a compatibility contract. However, I think we first need to establish what Iris' contract around tests are before coming up with solutions. Notably it was Iris core devs that built on top of iris.tests in one of the projects affected, in the first place, so it was Iris who initially gave this a greenlight!

With regards to

Stop shipping the tests within Iris, and find a case-specific way to run the tests for testing the UK Met Office stacks.

Albeit a guaranteed way of blocking any user from using them(!) I think that is not the right approach to solving this particular problem. Were it not for the recent discussions on #3416, I doubt this idea would be being proposed so I'm concerned by the influence it is having on this discussion. Users using iris.tests may be a contributing factor to the decision on #3416 (but if so it should be discussed on that issue), but it is a strange way to solve this particular issue. Of note, the two examples I give above (xarray and numpy) both included the testing module (which is documented) and a tests module (which is not). If the contract is clear as to what a user can depend on, similar to how we manage the experimental module which is part of the API but not open to change, then that should be enough

[trexfeathers]

Yeah this is starting to sound pretty gnarly - picking apart which bits we actually intended people to use. I'm familiar with some of the obviously public bits of NumPy testing, and there is a clear distinction from just 'stuff they use in their tests'.

[lbdreyer]

A little update:
So far I have come across 10 12 repositories that are importing from iris.tests. Then they appear to have varying degrees of dependency on it, but this was more than I was expecting

I have also had a chat with one of the projects and they are interested in dropping their dependency on iris.tests

[pp-mo]

it was Iris core devs that built on top of iris.tests in one of the projects affected

Can we name names, do you have a list ?

I am aware of iris-grib, and also ncdata of course.
From a technical point of view, I think it should be quite possible to break those 2 dependencies, at least.
Though in the case of iris-grib, only by cloning the IrisTest code initially -- which would be messy but not particularly impractical.

[lbdreyer]

Can we name names, do you have a list ?

Unfortunately I was only looking at private repos, when I first raised this PR.

I am aware of iris-grib, and also ncdata of course.

I had not considered iris-grib, nor ncdata. So now we are on 12 😬

by cloning the IrisTest code initially

This is the suggestion I have made so far for projects that are worried they won't have time to migrate should we break things!