Questions on Adding Humanoids in PointNav/ObjNav tasks in Habitat Environment

[Sawyer-Wu]

Habitat-Lab and Habitat-Sim versions

Habitat-Lab: nightly
Habitat-Sim: master

❓ Questions and Help

Hi, Thank you for your brilliant work on this simulator.

I am currently working on adding humanoids into tasks such as PointNav and ObjNav with the simulator to better simulate dynamic real-world scenarios. My ultimate goal is to successfully train and test this setup using the habitat-baseline framework. However, I have encountered a series of challenges that I seek your assistance with.

Here is a detailed description of my progress and inquiries:

1. Completed Tasks:

• I have successfully added a humanoid to the mp3d_example scene in habitat-sim by following the instructions in /test/test_humanoid. This involved using KinematicHumanoid, HumanoidRearrangeController, and I have implemented continuous control.
• Following the tutorials in /example/humanoids_tutorial.ipynb and examples/tutorials/articulated_agents_tutorial.ipynb, I managed to include a humanoid with RearrangeSim and also controlled it with RearrangeEnvironment.

2. My Attempts:

• I attempted to write a function to initialize a humanoid within habitat-sim and tried to modify the _worker_env() function in /habitat/core/vector_env.py. My approach was to extract the "sim" (_sim = env.env.env.env._env.sim) from the "env" (env = EnvCountEpisodeWrapper(EnvObsDictWrapper(env_fn(*env_fn_args)))) and initialize and reset the humanoid using the methods from /test/test_humanoid. Although this seemed to work, I encountered issues with the network parameters resulting in a matrix of NaN values during training, which interrupts the training process. Despite being able to resume from a checkpoint, I was unable to resolve this issue and suspect that my approach may not be logically sound, leading me to explore methods within habitat-lab.
• During this period, I noticed updates to the repo's examples/tutorials/humanoids_tutorial.ipynb and examples/tutorials/articulated_agents_tutorial.ipynb, which demonstrate the recommended methods for including humanoids. However, I found that these methods are deeply tied to RearrangeSim and RearrangeEnvironment. When using only RearrangeSim, I successfully changed the sim_cfg.scene to a scene from mp3d_example. However, for reinforcement learning training within the simulator, I am required to use Env(from habitat.core.env import Env).
• When attempting to use RearrangeEnvironment, I faced numerous challenges. Initially, changing only the sim_cfg.scene yielded no changes, leading me to believe it might be due to DatasetConfig(). However, I do not have rearrange episode datasets for hm3d. I also attempted to change the SimulatorConfig, TaskConfig, and other configurations to values related to PointNav, but I could not add a humanoid, and the third_rgb_sensor did not display correctly. I suspect that humanoid and third_rgb_sensor are deeply tied to RearrangeSim and RearrangeEnvironment, but I have not found concrete evidence.

I am curious about how to correctly integrate humanoids into the PointNav task using datasets like hm3d. Is it necessary to redefine a PointNavSim and PointNavEnvironment similar to RearrangeSim and RearrangeEnvironment? I would greatly appreciate any guidance you could provide.

Additionally, I have a question regarding the tutorial examples/tutorials/articulated_agents_tutorial.ipynb. In the Multi-Agent Interaction section, after copying the config, the second_agent's head_rgb seems to exhibit anomalies. I am wondering if this might be a bug?

Thank you in advance for your time and assistance.

[xavierpuigf]

Hi, will look at these rgb anomalies, thanks for the note! Let me try to address the config for navigation issue here:

The tutorial does not include defining tasks with multi-agents. This is something we will probably add in a different tutorial. Will try to think what is the easiest way to explain this.

The only thing you need to support humans when defining a task (even PointNav) is the agents, actions and measurements. See for example in social_nav, which is very similar to what you want to do. The social nav config calls here, the hssd_human_spot_social_nav.yaml, which defines a spot agent, a human agent (lines 1-12) and the actions (up to line 20).

For pointnav, you could do something very similar. The difficulty is in following the hydra configs...
If you look at pointnav it calls /benchmark/nav/pointnav: pointnav_gibson which is here. You can see how it defines a simualtor:agents:main_agent. This also happens for object nav. What you need to do is to modify the config (in the first case the pointnav_gibson.yaml so that instead of main_agent, it defines agent_0 and agent_1. Then you can make agent_1 be the humanoid just like in hssd_human_spot_social_nav.yaml.

Let me know if that helps or you reach out other blockers!

[Sawyer-Wu]

Hi, @xavierpuigf. Thank you for your reply. I have previously tried a similar method to what you mentioned, and I have tried again. Below is my test code and config:
The test code is as follows:

import os
import git
from matplotlib import pyplot as plt
import habitat

repo = git.Repo(".", search_parent_directories=True)
dir_path = repo.working_tree_dir
data_path = os.path.join(dir_path, "data")
os.chdir(dir_path)

if __name__ == "__main__":
    config = habitat.get_config(
        config_path=os.path.join(
            dir_path,
            "habitat-lab/habitat/config/benchmark/nav/pointnav/pointnav_gibson_test.yaml",
        ),
    )
    
    try:
        env.close()
    except NameError:
        pass
    env = habitat.Env(config=config)

Config file(pointnav_gibson_test.yaml) is as follows:

# @package _global_

defaults:
  - pointnav_base_wsy
  - /habitat/dataset/pointnav: gibson
  
  - /habitat/simulator/agents@habitat.simulator.agents.agent_0: spot
  - /habitat/simulator/sim_sensors@habitat.simulator.agents.agent_0.sim_sensors.rgb_sensor: rgb_sensor
  - /habitat/simulator/agents@habitat.simulator.agents.agent_1: human
  - /habitat/simulator/sensor_setups@habitat.simulator.agents.agent_1: rgbd_head_agent

  # - /habitat/task/actions@habitat.task.actions.agent_0_base_velocity: base_velocity_non_cylinder
  # - /habitat/task/actions@habitat.task.actions.agent_1_base_velocity: base_velocity
  # - /habitat/task/actions@habitat.task.actions.agent_1_rearrange_stop: rearrange_stop
  # - /habitat/task/actions@habitat.task.actions.agent_1_pddl_apply_action: pddl_apply_action
  # - /habitat/task/actions@habitat.task.actions.agent_1_oracle_nav_action: oracle_nav_action
  # - /habitat/task/actions@habitat.task.actions.agent_1_oracle_nav_randcoord_action: oracle_nav_action

  - _self_

habitat:
  environment:
    max_episode_steps: 500
  simulator:
    agents_order:
      - agent_0
      - agent_1

Visualization code is as follows:

obs = env.reset()
valid_key_list = ["rgb", 'head_rgb', 'head_depth']
valid_num = 0
print(obs.keys())
for ind, name in enumerate(obs.keys()):
    if name not in valid_key_list:
        continue
    valid_num += 1
_, ax = plt.subplots(1,valid_num)

for ind, name in enumerate(obs.keys()):
    if name not in valid_key_list:
        continue
    ax[ind].imshow(obs[name])
    ax[ind].set_axis_off()
    ax[ind].set_title(name)

As shown in the visualization results, you can see that even though I added the spot agent and humanoid agent, their sensors show the same location, and after performing multiple env.reset(), I did not see the added agent models. Therefore, I wonder if there might be an issue with this method? If I use Rearrangesim, this problem does not occur.

Additionally, I encounter problems when trying to add actions in the config above. The error message is too long so I put it into the attached file error.txt.

My purpose is to add some moving models to train the main_agent’s dynamic obstacle avoidance capabilities. Maybe it's not necessarily to define tasks with multi-agents? In fact, I only want the main agent to be trained, and the rest can just serve as dynamic obstacles, but I hope they can walk freely in the environment, like moving with OracleNavActionConfig. Could you please tell me, if I define tasks with multi-agents, will all added agents be involved in the task training?

Thank you for your patient explanation. If there is anything wrong with my approach, please point it out. Thank you.

[xavierpuigf]

That makes sense! Will try to help, some of the blockers you see here will be useful as we update the simulator to be more flexible/easier to use.

I think the easiest thing would be to look at the social navigation task and configs, since it is pretty much the same task you are trying to do here, and defines humans that walk around the scene.

The main issue you have here is that your simulator does not have an agent_manager. The agent manager is what deals with multi-agent. It is also what will allow you to have per-agent observations (not that once you use that manager, the observations will be prepended by the agent name, e.g. agent_0_rgb, agent_0_head_depth. So your issue is not as much about humans but more about multi-agents.

You can see in rearrange_sim where the agent manager is defined: https://github.com/facebookresearch/habitat-lab/blob/main/habitat-lab/habitat/tasks/rearrange/rearrange_sim.py#L118

What I would recommend you is to either:

1. Add agent manager in your sim
2. Make your sim be a subclass of rearrange_sim, so that it inherits the agent manager
3. start with the social navigation task and modify that to include the dataset you want to work with.

I hope this is useful! Happy to keep on this thread.

[Sawyer-Wu]

That makes sense! Will try to help, some of the blockers you see here will be useful as we update the simulator to be more flexible/easier to use.

I think the easiest thing would be to look at the social navigation task and configs, since it is pretty much the same task you are trying to do here, and defines humans that walk around the scene.

The main issue you have here is that your simulator does not have an agent_manager. The agent manager is what deals with multi-agent. It is also what will allow you to have per-agent observations (not that once you use that manager, the observations will be prepended by the agent name, e.g. agent_0_rgb, agent_0_head_depth. So your issue is not as much about humans but more about multi-agents.

You can see in rearrange_sim where the agent manager is defined: https://github.com/facebookresearch/habitat-lab/blob/main/habitat-lab/habitat/tasks/rearrange/rearrange_sim.py#L118

What I would recommend you is to either:

1. Add agent manager in your sim
2. Make your sim be a subclass of rearrange_sim, so that it inherits the agent manager
3. start with the social navigation task and modify that to include the dataset you want to work with.

I hope this is useful! Happy to keep on this thread.

Thans a lot! I think these suggestions will be very helpful. I will let you know if there are any developments!