-
Notifications
You must be signed in to change notification settings - Fork 817
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Torchserve support for Intel GPUs #3132
base: master
Are you sure you want to change the base?
Conversation
}catch (IOException | InterruptedException e) { | ||
System.out.println("nvidia-smi not available or failed: " + e.getMessage()); | ||
} | ||
try { | ||
Process process = Runtime.getRuntime().exec("xpu-smi discovery --dump 1"); | ||
int ret = process.waitFor(); | ||
if (ret != 0) { | ||
return 0; | ||
} | ||
List<String> list = | ||
IOUtils.readLines(process.getInputStream(), StandardCharsets.UTF_8); | ||
if (list.isEmpty() || !list.get(0).contains("Device ID")) { | ||
throw new AssertionError("Unexpected xpu-smi response."); | ||
} | ||
for (int i = 1; i < list.size(); i++) { | ||
gpuIds.add(Integer.parseInt(list.get(i))); | ||
} | ||
}catch (IOException | InterruptedException e) { | ||
System.out.println("xpu-smi not available or failed: " + e.getMessage()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is Intel GPU also set CUDA_VISIBLE_DEVICES? If not, this part code will not be executed.
ts/torch_handler/base_handler.py
Outdated
@@ -370,7 +377,10 @@ def inference(self, data, *args, **kwargs): | |||
Torch Tensor : The Predicted Torch Tensor is returned in this function. | |||
""" | |||
with torch.inference_mode(): | |||
marshalled_data = data.to(self.device) | |||
if IPEX_AVAILABLE and IPEX_GPU: # IPEX GPU | |||
marshalled_data = data.to("xpu") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
need add xpu id.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you for the suggestion. We have updated this.
@@ -90,6 +92,36 @@ Add the following lines in `config.properties` to use launcher with its default | |||
cpu_launcher_enable=true | |||
``` | |||
|
|||
## TorchServe with Intel® Extension for PyTorch* and Intel GPUs |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you please provide the information link about intel GPU performance gain?
This reverts commit 9188deb.
Description
This PR will enable torchserve to leverage Intel GPU devices. This adds a config ipex_gpu_enable, to allow gpu enablement in base_handler.py
Type of change
Please delete options that are not relevant.
Checklist: