[RLlib] Upgrade to gymnasium 1.0.0a2 and ale_py 0.9.0. #45328

sven1977 · 2024-05-14T09:22:55Z

Upgrade RLlib to gymnasium 1.0.0a2.

Reason:

We require some bug fixes in gymnasium that only exist in 1.0.0a1/2 (not in 0.29.1) that allow us to make use of their vectorized sync and async environments in RLlib's new EnvRunners.

Why are these changes needed?

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: sven1977 <svenmika1977@gmail.com>

sven1977 · 2024-05-14T09:24:45Z

python/requirements/ml/rllib-test-requirements.txt

@@ -3,7 +3,6 @@
 # Environment adapters.
 # ---------------------
 # Atari
-gymnasium==0.28.1


Since gymnasium is already part of the main Ray requirements.txt file, we won't need this here anymore.

Signed-off-by: sven1977 <svenmika1977@gmail.com>

sven1977 · 2024-05-14T12:58:34Z

cc: @pseudo-rnd-thoughts @jkterry1
Congrats on gymnasium 1.0!! This is super exciting. :)

…ade_gymnasium_to_1_0_0a1

Signed-off-by: sven1977 <svenmika1977@gmail.com>

rllib/env/single_agent_env_runner.py

Signed-off-by: Sven Mika <sven@anyscale.io>

sven1977 · 2024-05-14T14:04:44Z

rllib/env/single_agent_env_runner.py

@@ -249,6 +249,8 @@ def _sample_timesteps(
                    observation=obs[env_index],
                    infos=infos[env_index],
                )
+            self._was_terminated = [False for _ in range(self.num_envs)]


This is completely new auto-reset logic of gymnasium 1.0. The sub-env only gets reset'd upon the next(!) step call (with a fake reward of 0.0 and term/trunc=guaranteed False; and the obs/infos being the reset-obs/infos).
This is actually good for us as we should always do the env-to-module connector pass (even after the last timestep with the terminal obs in the Episodes list) to make sure the user - in case they are writing to the episode - gets a chance to also alter the final obs.

simonsays1980

LGTM.

simonsays1980 · 2024-05-14T14:08:47Z

rllib/env/single_agent_env_runner.py

@@ -88,7 +88,7 @@ def __init__(self, config: AlgorithmConfig, **kwargs):
            #  actually hold the spaces for a single env, but for boxes the
            #  shape is (1, 1) which brings a problem with the action dists.
            #  shape=(1,) is expected.
-            module_spec.action_space = self.env.envs[0].action_space
+            module_spec.action_space = self.env.single_action_space


Sweet. This is now gone.

simonsays1980 · 2024-05-14T14:21:34Z

rllib/env/single_agent_env_runner.py

                    eps += 1

-                    episodes[env_index].add_env_step(
-                        infos[env_index].pop("final_observation"),


Okay, i.e. with gymnasium>=1.0.0 the final_observation is gone and instead a regular observartion will be returned?

Correct, the final observation is returned in the actual obs. The reset obs, you only get on the next(!) call to step, together with a dummy reward of 0.0.

Signed-off-by: sven1977 <svenmika1977@gmail.com>

…to upgrade_gymnasium_to_1_0_0a1 # Conflicts: # rllib/env/single_agent_env_runner.py

…ade_gymnasium_to_1_0_0a1

Signed-off-by: sven1977 <svenmika1977@gmail.com>

pseudo-rnd-thoughts

Hi, we have just released Gymnasium alpha 2, would you be able to test with gymnasium>=1.0.0a1? This would help check compatibility

…ade_gymnasium_to_1_0_0a1 Signed-off-by: sven1977 <svenmika1977@gmail.com> # Conflicts: # rllib/env/single_agent_env_runner.py

Signed-off-by: sven1977 <svenmika1977@gmail.com>

sven1977 · 2024-05-31T11:28:52Z

Hey @pseudo-rnd-thoughts , yes, we are in the process of wrapping this up. Thanks so much! Now that Atari is supported, I don't see any issues anymore holding us back to support 1.0.0a2 in RLlib's new stack. We'll let you know, if we still find any issues with the API. Very exciting! :)

pseudo-rnd-thoughts · 2024-05-31T12:15:28Z

Hey @pseudo-rnd-thoughts , yes, we are in the process of wrapping this up. Thanks so much! Now that Atari is supported, I don't see any issues anymore holding us back to support 1.0.0a2 in RLlib's new stack. We'll let you know, if we still find any issues with the API. Very exciting! :)

Amazing, that is very exciting to hear

wip

9cb5160

Signed-off-by: sven1977 <svenmika1977@gmail.com>

sven1977 requested review from richardliaw, ericl and edoakes as code owners May 14, 2024 09:22

sven1977 assigned richardliaw and edoakes May 14, 2024

sven1977 commented May 14, 2024

View reviewed changes

sven1977 added 4 commits May 14, 2024 11:58

fixes

5678569

Signed-off-by: sven1977 <svenmika1977@gmail.com>

fixes

f750ac3

Signed-off-by: sven1977 <svenmika1977@gmail.com>

fixes

d2a36b3

Signed-off-by: sven1977 <svenmika1977@gmail.com>

fix

3c90cc7

Signed-off-by: sven1977 <svenmika1977@gmail.com>

sven1977 requested review from avnishn, ArturNiederfahrenhorst, maxpumperla, kouroshHakha and simonsays1980 as code owners May 14, 2024 11:38

edoakes approved these changes May 14, 2024

View reviewed changes

sven1977 added 3 commits May 14, 2024 15:01

Merge branch 'master' of https://github.com/ray-project/ray into upgr…

2bb745a

…ade_gymnasium_to_1_0_0a1

LINT

7921430

Signed-off-by: sven1977 <svenmika1977@gmail.com>

LINT

639698a

Signed-off-by: sven1977 <svenmika1977@gmail.com>

sven1977 assigned simonsays1980 May 14, 2024

sven1977 commented May 14, 2024

View reviewed changes

rllib/env/single_agent_env_runner.py Outdated Show resolved Hide resolved

Apply suggestions from code review

bdda97c

Signed-off-by: Sven Mika <sven@anyscale.io>

sven1977 commented May 14, 2024

View reviewed changes

simonsays1980 approved these changes May 14, 2024

View reviewed changes

sven1977 added 4 commits May 14, 2024 18:44

fixes

cf8f554

Signed-off-by: sven1977 <svenmika1977@gmail.com>

Merge remote-tracking branch 'origin/upgrade_gymnasium_to_1_0_0a1' in…

a67302a

…to upgrade_gymnasium_to_1_0_0a1 # Conflicts: # rllib/env/single_agent_env_runner.py

Merge branch 'master' of https://github.com/ray-project/ray into upgr…

82b1638

…ade_gymnasium_to_1_0_0a1

wip

4d36a44

Signed-off-by: sven1977 <svenmika1977@gmail.com>

fixes

773dbdf

Signed-off-by: sven1977 <svenmika1977@gmail.com>

sven1977 mentioned this pull request May 15, 2024

[Question] Rough timeline for Atari (ALE) support for gymnasium>=1.0. Farama-Foundation/Gymnasium#1049

Closed

pseudo-rnd-thoughts reviewed May 21, 2024

View reviewed changes

pseudo-rnd-thoughts mentioned this pull request May 21, 2024

Projects updated to v1.0.0 alphas Farama-Foundation/Gymnasium#944

Open

20 tasks

sven1977 added 2 commits May 31, 2024 12:18

Merge branch 'master' of https://github.com/ray-project/ray into upgr…

45dcf93

…ade_gymnasium_to_1_0_0a1 Signed-off-by: sven1977 <svenmika1977@gmail.com> # Conflicts: # rllib/env/single_agent_env_runner.py

Update to gymnasium 1.0.0a2 and new ale_py (which now supports Atari)

7f6fd9d

Signed-off-by: sven1977 <svenmika1977@gmail.com>

sven1977 changed the title ~~[RLlib] Upgrade to gymnasium 1.0.0a1.~~ [RLlib] Upgrade to gymnasium 1.0.0a2 and ale_py 0.9.0. May 31, 2024

Update to gymnasium 1.0.0a2 and new ale_py (which now supports Atari)

8946a00

Signed-off-by: sven1977 <svenmika1977@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RLlib] Upgrade to gymnasium 1.0.0a2 and ale_py 0.9.0. #45328

[RLlib] Upgrade to gymnasium 1.0.0a2 and ale_py 0.9.0. #45328

sven1977 commented May 14, 2024 •

edited

sven1977 May 14, 2024

sven1977 commented May 14, 2024 •

edited

sven1977 May 14, 2024

simonsays1980 left a comment

simonsays1980 May 14, 2024

simonsays1980 May 14, 2024

sven1977 May 14, 2024

pseudo-rnd-thoughts left a comment

sven1977 commented May 31, 2024

pseudo-rnd-thoughts commented May 31, 2024

[RLlib] Upgrade to gymnasium 1.0.0a2 and ale_py 0.9.0. #45328

Are you sure you want to change the base?

[RLlib] Upgrade to gymnasium 1.0.0a2 and ale_py 0.9.0. #45328

Conversation

sven1977 commented May 14, 2024 • edited

Why are these changes needed?

Related issue number

Checks

sven1977 May 14, 2024

Choose a reason for hiding this comment

sven1977 commented May 14, 2024 • edited

sven1977 May 14, 2024

Choose a reason for hiding this comment

simonsays1980 left a comment

Choose a reason for hiding this comment

simonsays1980 May 14, 2024

Choose a reason for hiding this comment

simonsays1980 May 14, 2024

Choose a reason for hiding this comment

sven1977 May 14, 2024

Choose a reason for hiding this comment

pseudo-rnd-thoughts left a comment

Choose a reason for hiding this comment

sven1977 commented May 31, 2024

pseudo-rnd-thoughts commented May 31, 2024

sven1977 commented May 14, 2024 •

edited

sven1977 commented May 14, 2024 •

edited