[hitl][hab-llm] New controller to work with habitat-llm Agents #1936

zephirefaith · 2024-05-01T18:17:36Z

Motivation and Context

This PR creates LLMController based on SingleAgentBaselineController which houses basic init, reset and act code for calling habitat-llm agents.

Confused: Do I need to update SingleAgentAccessMgr class as well to be compatible with habitat-agents related to habitat-llm? The previously mentioned class just houses a lot of extra code which is useless in given setup. In habitat-llm setup the action is directly sent from policies running wherever habitat-llm code is running, without needing any policy setup or inference from SingleAgentAccessMgr class.
SingleAgentAccessMgr is also not needed, as inference comes from habitat-llm policies directly.

How Has This Been Tested

Types of changes

[Docs change] Addition or changes to the documentation
[Refactoring] Large changes to the code that improve its functionality or performance
[Dependency Upgrade] Upgrades one or several dependencies in habitat
[Bug Fix] (non-breaking change which fixes an issue)
[Development] A pull request that add new features to the habitat-lab task and environment codebase. Development Pull Requests must be small (less that 500 lines of code change), have unit testing, very extensive documentation and examples. These are typically new tasks, environments, sensors, etc... The review process for these Pull Request is longer because these changes will be maintained by our core team of developers, so make sure your changes are easy to understand!
[Experiment] A pull request that add new features to the habitat-baselines training codebase. Experiments Pull Requests can be any size, must have smoke/integration tests and be isolated from the rest of the code. Your code additions should not rely on other habitat-baselines code. This is to avoid dependencies between different parts of habitat-baselines. You must also include a README that will document how to use your new feature or trainer. You will be the maintainer of this code, if the code becomes stale or is not supported often enough, we will eventually remove it.

Checklist

My code follows the code style of this project.
I have updated the documentation if required.
I have read the CONTRIBUTING document.
I have completed my CLA (see CONTRIBUTING)
I have added tests to cover my changes if required.

…f/hitl-llm-interfacing

examples/hitl/rearrange_v2/config/lang_rearrange_spot_humanoid_visual.yaml

0mdc

Thank you @zephirefaith ! I did a first pass and raised some concerns.

Let me know if I can help!

0mdc · 2024-05-15T22:01:20Z

examples/hitl/rearrange_v2/config/lang_rearrange_llmspot_guihumanoid.yaml

We should add comments to config headers to explain the use cases covered by each of them.

0mdc · 2024-05-16T00:16:18Z

examples/hitl/rearrange_v2/config/lang_rearrange_llmspot_guihumanoid.yaml

+    max_episode_steps: 0
+    iterator_options:
+      # For the demo, we want to showcase the episodes in the specified order
+      shuffle: False


We'll want to pull those changes here.

0mdc · 2024-05-16T00:18:06Z

examples/hitl/rearrange_v2/config/language_rearrange_multi_agent_llm_gui.yaml

+  # 2. What is the humanoidjoint_action? Do we need it here?
+  # - /habitat/task/actions@habitat.task.actions.agent_1_humanoid_base_velocity: humanoid_base_velocity
+  # - /habitat/task/actions@habitat.task.actions.agent_1_humanoidjoint_action: humanoidjoint_action
+  # - /habitat/task/actions@habitat.task.actions.agent_1_base_velocity: base_velocity


@xavierpuigf Would you know the answers to these?

you don't need agent_1_base_velocity for the human, only agent_1_humanoid_base_velocity. The difference is that humanoid base velocity generates a human realistic motion whereas base velocity only moves the base (used to move the robot for instance).

humanoidjoint_action sets the human joints so that the pose is realistic. You will never be callign it yourself, the agent_1_base_velocity sets that. But you sitll need to define it here

0mdc · 2024-05-16T00:20:38Z

examples/hitl/rearrange_v2/config/language_rearrange_multi_agent_llm_gui.yaml

+      # - agent_1_dynamic_obj_start_sensor
+# --- habitat-llm block
+  environment:
+    max_episode_steps: 750  # this is 20000 in habitat-llm


We'll want to double-check this.

Is this enough to complete the longest tasks?

What's the typical step count for a normal multiplayer run? (I'll get the data)

What happens when this is done? We would need to call the appropriate code in HITL to trigger the next application state.

0mdc · 2024-05-16T00:21:35Z

habitat-hitl/habitat_hitl/environment/controllers/llm_controller.py

+
+        with habitat.config.read_write(self._config):
+            fix_config(self._config)
+        seed = 47668090


Where does this seed come from?

It's the default in all our code. We can move this to config. But same as what other experiments are using.

habitat-hitl/habitat_hitl/environment/controllers/llm_controller.py

0mdc · 2024-05-16T00:33:06Z

habitat-hitl/habitat_hitl/environment/controllers/llm_controller.py

+            verbose=True,
+        )
+        if task_done:
+            print("Task Done")


I suppose that this would be the point where we hook up to HITL so that the agent can signal that its part of the task is done?

Would this also be called after the timeout? (max_episode_steps)

The two conditions this would be called on:

If # of planning steps > max planning steps threshold

LLM reasons that the task is solved/can not be solved further

0mdc · 2024-05-16T00:35:01Z

habitat-lab/habitat/articulated_agent_controllers/humanoid_rearrange_controller.py

+        # distance_covered = max(
+        #     0, new_cum_distance_covered + offset - prev_cum_distance_covered
+        # )
+        # dist_diff = min(distance_to_walk, distance_covered)


We need those changes to set humanoid walk speed. However, this may break other stuff.

Perhaps we could make animation-driven humanoid speed optional via config? If we do that, I'm afraid that it will be hard for other developers to figure out why the speed is limited.

Ideally, animation speed should be driven by locomotion speed.

@xavierpuigf Thoughts?

For now, enabling/disabling this through config and adding comments both at config and code-level sounds like the least effort option to still maintain "other" uses of code.

Wdyt? Also, I do not completely understand this code layer so your help to add comments here would be perfect.

Yep! The fast solution is to disable animation. I have a way to remove the speed limitation but havent had time to get into it yet. If we want to have animation, I can work on integrating it next week.

We should not merge this as is though, it will break the functionalities for any other kind of motion

0mdc · 2024-05-16T00:35:30Z

habitat-lab/habitat/articulated_agent_controllers/humanoid_rearrange_controller.py

@@ -252,7 +255,9 @@ def calculate_walk_pose(
        # The base_transform here is independent of transforms caused by the current
        # motion pose.
        obj_transform_base = look_at_path_T
-        forward_V_dist = forward_V * dist_diff * distance_multiplier
+        # HACK FROM MIKAEL
+        # forward_V_dist = forward_V * dist_diff * distance_multiplier


Sorry for this abomination :)

0mdc · 2024-05-16T00:41:35Z

@zephirefaith You'll probably want to have this double-checked by someone who is familiar with the LLM configs so that we don't misstep here.

Ram81 and others added 3 commits April 29, 2024 16:28

Add task config for multi-agent

2944319

changes to run barebone w/llm

8c22953

basic plumbing into hitl

e16ddbd

zephirefaith requested a review from Ram81 May 1, 2024 18:17

facebook-github-bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label May 1, 2024

zephirefaith requested review from 0mdc, xavierpuigf and akshararai May 1, 2024 18:17

zephirefaith added 9 commits May 1, 2024 15:10

fixing precommit

7849ef6

adding visual sensors

e4631d5

config patching, remove unneeded super() calls

0c0ed67

frankenstein hitl+llm config

c8a9857

bandaids to get to working V0

77880a3

Merge branch 'main' of github.com:facebookresearch/habitat-lab into z…

eed48c1

…f/hitl-llm-interfacing

making visual sensor changes configurable with explicit flags

21ce0c4

delete sensors we do not need

e74a92d

multi-threading + partial obs

8e912e4

zephirefaith self-assigned this May 15, 2024

zephirefaith marked this pull request as ready for review May 15, 2024 20:56

zephirefaith added 3 commits May 15, 2024 17:10

speeding up the humanoid

f62a4e9

logging

c71b55f

Merge branch 'main' of github.com:facebookresearch/habitat-lab into z…

622f582

…f/hitl-llm-interfacing

zephirefaith commented May 15, 2024

View reviewed changes

examples/hitl/rearrange_v2/config/lang_rearrange_spot_humanoid_visual.yaml Outdated Show resolved Hide resolved

0mdc reviewed May 16, 2024

View reviewed changes

zephirefaith changed the title ~~New controller to work with habitat-llm Agents~~ [hitl][hab-llm] New controller to work with habitat-llm Agents May 17, 2024

zephirefaith added 4 commits May 17, 2024 16:26

moving habitat-llm stuff to dedicated file/module

05ac703

+header,+-logging,-llm util

998dd6d

remove unused config

ff0fb40

optional logging, -llb-deps

a87b1ba

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[hitl][hab-llm] New controller to work with habitat-llm Agents #1936

[hitl][hab-llm] New controller to work with habitat-llm Agents #1936

zephirefaith commented May 1, 2024 •

edited

0mdc left a comment

0mdc May 15, 2024

0mdc May 16, 2024

0mdc May 16, 2024

xavierpuigf May 17, 2024

0mdc May 16, 2024

0mdc May 16, 2024

zephirefaith May 17, 2024

0mdc May 16, 2024

zephirefaith May 17, 2024

0mdc May 16, 2024

zephirefaith May 17, 2024

xavierpuigf May 17, 2024

xavierpuigf May 17, 2024

0mdc May 16, 2024

0mdc commented May 16, 2024

[hitl][hab-llm] New controller to work with habitat-llm Agents #1936

Are you sure you want to change the base?

[hitl][hab-llm] New controller to work with habitat-llm Agents #1936

Conversation

zephirefaith commented May 1, 2024 • edited

Motivation and Context

How Has This Been Tested

Types of changes

Checklist

0mdc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0mdc commented May 16, 2024

zephirefaith commented May 1, 2024 •

edited