Version Locking Dependencies

[rwuebker]

Hello,

This might be an odd general question, but I noticed the dependencies aren't version locked. Why is this?

I had some issue with a scipy release that caused my cvxpy to fail on import. I think if the dependencies were version locked this wouldn't be an issue. Just curious.

Thanks for any replies!
Rick

[rileyjmurray]

Hi Rick! Thanks for coming by. I don't think we've had a big discussion about this per se. I can share some thoughts on the topic off-hand.

First and foremost, we can't specify exact versions for each dependency because that would create "dependency hell" for many cvxpy users. This means the question is really one of what upper and lower bounds we enforce for dependencies.

I believe we already specify lower bounds for all dependencies and that this question is really about guarding against breaking changes from new releases (e.g., of SciPy). I.e., I think your question is about how we set upper bounds in the versions of our dependencies.

There are some instances where we explicitly enforce upper bounds on solvers. For example, our interfaces to GLOP and PDLP have an upper bound on Google's ortools. That said, we don't set upper bounds for NumPy and SciPy. That hasn't been a big problem because the maintainers for those libraries have a big commitment to backward compatibility. The few times that a NumPy or SciPy release has unexpectedly broken CVXPY, we just scrambled to release a fix ASAP.

I feel like it would be good for us to reduce the risk of such scrambling. It's not clear to me what those steps should be. I see two uncontroversial things we can do to get started:

1. Enforce upper bounds on the major versions of NumPy and SciPy (the x in x.y.z). Even if these libraries strictly adhered to semantic versioning (which is not the case), it would be fair game for such a version change to break the API. So we should protect against that.
2. Do a better job of addressing deprecation warnings in our code. There have been deprecation warnings about scipy.sparse and scipy.sparse.spmatrix for a long time. I'm not totally certain, but I think that this week's SciPy release wouldn't have broken CVXPY if we had addressed those notices.

Anyway, that's what I've got for now. I do think this is a good conversation to have, so thanks again for initiating it.

Ping @cvxpy/maintainers. Thoughts?

[sjschneider]

This is a great discussion. We scrambled to fix build problems when this issue came up.

For what it’s worth, I don’t think that Option 1 to enforce upper bounds is a good idea unless there is a known incompatibility. More often than not nothing breaks with major version upgrades and so introducing a limit means that you potentially introduce an unnecessary constraint. I think the answer is in addressing those deprecation warnings.

[rileyjmurray]

Fair enough, @sjschneider! I think Option 1 is also unnecessary if we go with the route Philipp proposed.

[phschiele]

Another option to consider is to add a scheduled workflow that uses pre-released versions of numpy and SciPy.
Perhaps we would have caught this issue already after 1.11.0rc1 was released, which would have given us almost a month's lead time to address it.

[rileyjmurray]

@phschiele I think this is a great idea!