Future of Tianshou

[BFAnas]

Hi everyone,

I'm writing to start a discussion about the future of Tianshou. I've been a big fan of this repo for a while now, and I've been really impressed with the quality of the code and the helpfulness of the community. However, lately the repo has been uncharacteristically underperforming. There are some stale pull requests, and it seems like @Trinkle23897 the dynamo of the library has been busy lately with other projects.

I'm wondering if we can start a discussion about what we can do to help the repo get back on track. From my part I'm open to helping to review pull requests, and be more engaged in the issues section.

Thanks.

[Trinkle23897]

Would you like to fix current broken test on master? I'm still having tons of work tbd in openai

[BFAnas]

Yes for sure!

[BFAnas]

Done! please check the pull request when you have some time.

[Trinkle23897]

tysm!

[MischaPanch]

Hi, I would like to join in the discussion! I have also asked a while ago about deprecating python below 3.8, as gymnasium doesn't support it (and I generally think it's a good idea).

As I got no answer at that time, I started working on a fork. I like tianshou and would like to keep it alive!

So far I have done:

1. Added protocol types for different Batch types, allowing more transparency, IDE completions, and generally clearer code
2. Added literal types to better annotate string interfaces
3. Extracted a bunch of code from scripts to new methods in utils to reduce code duplication
4. Added support for jsonargparse and configuration in yaml. This is a big one, as with this and the new utils most scripts become almost identical, differing only by configuration. It also allows default configurations and hierarchical config. This should make differences between tuned benchmark examples and different algorithms much clearer (so far only adjusted the mujoco_ppo example, others will follow)
5. Removed some code duplication, unused params, and did some improvements/fixes on docstrings
6. I'm in the process of adding support for pytorch-lightning and probably also keras-core for some of the algorithms/optimization loops. The latter is particularly interesting, as moving to keras-core would make things framework independent, so users could e.g. use jax instead of pytorch (should be a sexy feature when finished).

My colleagues and I will further work on tianshou, we plan including more algorithms (like advantage weighted regression and implicit Q learning). We would love to merge the changes back to the original repo. Probably one at a time makes sense for easier reviewing.

What do you think? My company does quite some RL and we are committed to open source, so we could devote significant time to extending and improving tianshou. I have worked with SB3 and rllib in the past, and tianshou has a much better foundation for the future, in my opinion. torchRL is also out there, but it essentially has no high level interfaces and is not user friendly at all.

Would love to hear from you!

The fork is here:
https://github.com/aai-institute/tianshou

[MischaPanch]

Btw, I would also like to move the build to poetry, if you don't mind

[MischaPanch]

One question for the future is whether Batch, one of the cornerstones of the library, should be replaced by torch's new TensorDict. From what I see, the scopes and functionalities of the two are very similar. Moving to TensorDict would make tianshou compatible with torchRL and other projects, as TensorDict is here to stay. It's quite a lot of work to do that though and would require a new major release

[BFAnas]

I think it's a good idea, Batch is awesome, I have a current pull request #903 for having custom keys support in the __getitem__ and add methods of Batch, so one can add seamlessly any new key they want. I think this would have worked out of the box for TensorDict. However I wonder if adopting TensorDict as the main data structure will go against the other point of being framework independent?

[MischaPanch]

Maybe there is some way to make it work, I would need to think a lot and try some jax. It's for a further future though, I think, there are more pressing issues to be addressed first.

I am generally not sure what is more attractive - better integration with torch-based libraries or framework independence. For me personally the latter is not particularly important

[MischaPanch]

Seems like a bright future is feasible :). I started committing stuff and going through the issues, colleagues of mine will do the same soon. Thanks to @Trinkle23897 for the responsiveness!

@BFAnas feel free to message me if you want to collaborate on something, e.g. using tensordict. Now is probably a good time, since big changes are happening anyway (deprecating older python, some changes in trainer interfaces etc). One could probably even make things backwards compatible by keeping Batch around, letting it inherit from TensorDict and logging a deprecation warning in the constructor

[BFAnas]

Nice to hear, Tianshou is hands down the best RL library I used and its good to know that there’s a will to keep it that way. I m currently working on implementing some algorithms RBVE, CalQL and distributional versions of SAC and CQL. I m open to collaborate on TensorDict to see what it can bring to Tianshou and maybe start with working on a simple replay buffer class that supports it and test it on a simple algorithm such as DQN to get a feel on how it would impact the current code.

[Trinkle23897]

I'd say something controversial: torch_rl is largely based on tianshou's design and doesn't mention tianshou in the codebase (I saw there are some mentions in their arxiv paper). When torch_rl came up, some people discussed it with me and said, "Why is it so similar to tianshou", so I personally have a negative impression of it. Not to mention in 2021/2022 we submitted a pytorch ecosystem request to include tianshou, but we got no response.

I'm open to include TensorDict but please keep Batch implementation in the codebase instead of deleting it or making it inherit from TensorDict. We can polish Batch implementation, or switch to optree (which is compatible and faster)

[MischaPanch]

I also had a negative impression of torch_rl and it does seem close to tianshou in the parts where it's good. I am particularly concerned about the way torch_rl is "marketed", since in my opinion it's barely a usable RL library at all - rather a collection of utilities for helping with design of algos, but not providing implementations themselves. Their examples of algos are huge scripts that don't contain reusable code. I would never want this kind of complexity and potential for errors for a real-world use case, which is why I immediately chose tianshou over torch_rl.

We can discuss separately what to do about TensorDict, I wanna try some stuff out. Note that tensordict is its own library, independent of torch_rl. If there is a way to delegate maintenance of a batch-like object to a separate group of people, it might be worth thinking over. In any case, that's not urgent, as Batch is working just fine

[nuance1979]

Sorry I'm late to this discussion but I completely agree with @Trinkle23897 and would argue against replacing Batch with TensorDict. From my experience, a project with little traction like TensorDict has a much bigger risk of going obsolete than tianshou.Batch. I find it unwise to take up that risk for barely any gain. If there is a good feature in TensorDict that we like, we can simply add it to Batch.

[MischaPanch]

While I'm no longer arguing to include tensordict now, or to base batch on it, my general idea is that code that is officially backed by a team at pytorch has a good chance of long-term support. I guess we will see down the line what happens to torch-rl and tensordict. If they are here to stay, I believe it would be good to be compatible, or even to offload some functionalities to them in the long run (if it doesn't cost much effort and brings benefits), as it harnesses synergies and almost has the effect of bringing a dedicated team from meta to support a part of tianshou code.

I don't think OSS should be about competition, in the end we all just want a good RL library. If there are synergies with torch-rl, they should at least be explored.

In any case, nothing in this direction is going to happen in the next months, as there are far more important things to work on now. Maybe later we have a reason to revisit this conversation

[nuance1979]

I understand your point and my argument is not based on competition or spite. My concern is exactly about long-term support. I have doubts about tensordict being supported long term by pytorch or meta. TensorDict is currently hosted under https://github.com/pytorch-labs, not https://github.com/pytorch. My guess is pytorch-labs is for experimental features or some kind of incubator or corporate politics.

IMHO, the decision boils down to cost benefit analysis. I am in favor of doing some exploration to have better estimates of cost and benefit.