torch.distributed sys.excepthook crashes if distributed backend was deinitialized

[szmigacz]

🐛 Describe the bug

sys.excepthook defined in

pytorch/torch/distributed/distributed_c10d.py

Line 1361 in 0214711

def _distributed_excepthook(*args):

calls get_rank() which is not available if distributed group was already destroyed.

import torch

torch.distributed.init_process_group('gloo')
torch.distributed.destroy_process_group()

raise ZeroDivisionError

Raises:

Error in sys.excepthook:
Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 1339, in _distributed_excepthook
    prefix = f"[rank{get_rank()}]"
                     ^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 1746, in get_rank
    default_pg = _get_default_group()
                 ^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 1008, in _get_default_group
    raise ValueError(
ValueError: Default process group has not been initialized, please make sure to call init_process_group.

Original exception was:
Traceback (most recent call last):
  File "/private/tmp/hook.py", line 8, in <module>
    raise ZeroDivisionError
ZeroDivisionError

Versions

Collecting environment information...
PyTorch version: 2.3.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 14.4.1 (arm64)
GCC version: Could not collect
Clang version: 15.0.0 (clang-1500.3.9.4)
CMake version: version 3.26.4
Libc version: N/A

Python version: 3.10.14 (main, Mar 19 2024, 21:46:16) [Clang 15.0.0 (clang-1500.3.9.4)] (64-bit runtime)
Python platform: macOS-14.4.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Max

Versions of relevant libraries:
[pip3] torch==2.3.0
[conda] Could not collect

cc @ezyang @gchanan @zou3519 @kadeng @msaroufim @mrshenli @pritamdamania87 @zhaojuanmao @satgera @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @penguinwu @fegin @XilunWu @wanchaol @fduwjj @wz337 @tianyu-l @wconstab @yf225 @chauhang @d4l3k

[yf225]

Thanks! We would be open to taking PR for this.

[ezyang]

Thanks @wconstab, I appreciate it