Expose node death information in dashboard

[ruisearch42]

Why are these changes needed?

This is one PR of a series to better propagate and expose node death information.

Background: in Ray, a node can be either ALIVE or DEAD, and the death reason of a node could be unexpected (e.g., crash) or expected (e.g., idle termination or spot preemption). Currently, GCS knows the actual death reason, but this information is not shown to the users. As a result, users might think a node crashes when they see it is DEAD, while in reality these could be expected scenarios such as spot preemption. This may give users the wrong impression that Ray is unstable. In the series of changes, we are going to better propagate and expose the node death info.

This PR does the following:

• Expose node death info in dashboard's cluster page where the list of nodes is shown

See more details in design doc:
https://docs.google.com/document/d/1tn6Uj-SoEAaBu5_HWl4dNo3JBzTd_w09RPsIWQpUU_s/edit

Screenshots

Before change:

After change:

Related PRs

#45128
#45357

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[alanwguo]

Can you include a screenshot of the change in this PR?

[ruisearch42]

Can you include a screenshot of the change in this PR?

Sure, PFA.

[jjyao]

Could you have the screenshot in the PR description: with and without the pop-up.

[scottsun94]

Can we hide the "expand.." button when there is no message? We can just show - there?

[alanwguo]

thanks!

[anyscalesam]

Reviewed with @ruisearch42 > after these set of 2 PRs there should be two additional and we should be wrapped up with the Expose Ray failures stability project.

[jjyao]

LGTM