get_camera_transform from habitat.tasks.rearrange.utils uses inconsistent getting method

[zephirefaith]

Habitat-Lab and Habitat-Sim versions

Habitat-Lab: v0.3.0 or master?

Habitat-Sim: v0.3.0 or master?

Habitat is under active development, and we advise users to restrict themselves to stable releases of Habitat-Lab and Habitat-Sim. The bug you are about to report may already be fixed in the latest version.

Master branch contains 'bleeding edge' code, but we do appreciate bug reports for it!

🐛 Bug

The cameras attached to articulated agents are initialized using ArticulatedAgentCameraParams class here. The position for this sensor is set using the logic here. However when habitat.tasks.rearrange.utils.get_camera_transform function is used to get the absolute transformation of this camera, the getter applies inconsistent logic taking only the translation component of camera's view-matrix. This leads to a camera-pose with right translation but wrong rotation wrt world frame.

Steps to Reproduce

Steps to reproduce the behavior:

1. Initialize environment with Spot in it
2. Access articulated agent camera pose using get_camera_transform
3. Access articulated agent camera pose using sim.agents[0]._sensors[<sensor-name>].render_camera.camera_matrix.inverted()
4. Output of 2 != output of 3

See PCD built with 3 (top) versus with 2 (bottom) below.

[SergioArnaud]

🤯

@zephirefaith I spent several hours debugging this until I found your Issue!

[aclegg3]

Yeah, this is a pain. @SergioArnaud happy to review a fix if you want to tackle this. 😃

[zephirefaith]

Sorry this made you stumble @SergioArnaud! @aclegg3 given this bug has a greater damage than population of 1, I can prioritize this and push a fix today.

[zephirefaith]

Fixed in the PR linked above. Tested with zf/obs-wm-updates branch of habitat-llm. See attached videos of point-cloud from get_camera_transform method and sim.agents[0]._sensors[<sensor-name>].render_camera.camera_matrix.inverted() method. Camera used is articulated_agent_jaw_rgb and articulate_agent_jaw_depth introduced in #1856 .

Screen.Recording.2024-03-13.at.10.57.51.PM.mov

Screen.Recording.2024-03-13.at.10.54.19.PM.mov