Use Atari-5 for future benchmarking of discrete RL

[nuance1979]

• I have marked all applicable categories:
  • exception-raising bug
  • RL algorithm bug
  • documentation request (i.e. "X is missing from the documentation.")
  • new feature request
  • design request (i.e. "X should be changed to Y.")
• I have visited the source website
• I have searched through the issue tracker for duplicates
• I have mentioned version numbers, operating system and environment, where applicable:

  import tianshou, gymnasium as gym, torch, numpy, sys
  print(tianshou.__version__, gym.__version__, torch.__version__, numpy.__version__, sys.version, sys.platform)

I came across the Atari-5 paper, which convinced me that Atari-5 (Battle Zone, Double Dunk, Name This Game, Phoenix, Q*Bert) is a better subset of ALE games in terms of benchmarking efficiency than our current selection, which is approximately DQN-7 (Beam Rider, Breakout, Enduro, Pong, Q*Bert, Seaquest, S. Invaders). We can keep Pong as a sanity check so my proposal becomes Atari-5 + Pong.

TL; DR of the paper: by evaluating an algorithm on Atari-5, you get a fairly close estimation of its performance on the whole 57 ALE games.

I haven't tried it with Tianshou myself so not sure if there will be any unforeseen issue. Just want to throw it out there and seek your feedback.

[MischaPanch]

I think it's a very good idea, thanks for bringing it up! One of the important issues in the next two months will be establishing reliable regression tests for all algos and run them on each release. Testing on the Atari-5 will likely be incorporated into that