undefined symbol: iJIT_NotifyEvent #3513
Comments
|
So given you have Intel Extension for Pytorch (IPEX) involved here, albeit using CPU rather than GPU which I am more experienced with, I feel the need to chime in on what can be the fault. First off, IPEX CPU is not supported on ComfyUI and I have not tried it to see if it works. The code with IPEX (contributed in part by me) was written to explicitly detect for GPU and if you don't have an Intel Arc GPU, it is most probably will get confused and not work. |
Huh. Well, that could be the problem. I don't want to use CPU, I've got a GPU. A770. I'll try to reinstall that, thank you |
|
@Whackjob Any updates on this? I do also want to let you know that upon looking closer at your command to run ComfyUI, you are invoking |
|
Thanks for checking back. I've tried removing and redoing it all. I'm sure it's just me, but something always seems to go wrong. I've been googling and troubleshooting as best as I can. I just don't seem to get there. I saw your comment and I figured, oh, there's the problem. So I tried it with just python3. Weirdly, I'm getting the same bloody error. I have to be missing a prerequisite, or have the wrong version of something. But man, I just can't work it out. (venv) whackjob@WhackjobONE:/media/whackjob/16Tons/AI/ComfyUI$ python3 main.py --use-pytorch-cross-attention --highvram It looks like it's still trying to call the CPU to me, but I haven't installed anything CPU instead of GPU. I've been careful about that. I'm sure the error is on my side. I just don't know where. |
|
What does the sanity check in the install guide give you when you try and run the command, which is the following: |
|
whackjob@WhackjobONE:/media/whackjob/16Tons/AI/ComfyUI$ python3 -c "import torch; import intel_extension_for_pytorch as ipex; print(torch.version); print(ipex.version); [print(f'[{i}]: {torch.xpu.get_device_properties(i)}') for i in range(torch.xpu.device_count())];" |
|
Your GPU isn't being detected, you should get a list of the devices on your system after the 2nd line. Here's my output for instance. You need to as I said in my first post, reinstall Intel's oneAPI basekit 2024.1 and unfortunately, there is no easy way of telling what version you have installed so I have no clue what is wrong on that front. If you have the correct version, the sanity test should show up correctly with the output I posted above. |
|
I seem to have that. whackjob@WhackjobONE:/media/whackjob/16Tons/AI/ComfyUI$ sudo apt install intel-basekit Now, I went through all the steps on that page, and I see this: whackjob@WhackjobONE:/media/whackjob/16Tons/AI/ComfyUI$ sudo apt update Looking around, it seems I need to try from other places via VPN? I gave that a shot, no luck. In the end, when I go to install the basekit, it gives me that version, which is right, isn't it? |
|
Well for one, your GPG key isn't good for the Intel graphics APT repository. Can you redo the import? You do have the oneAPI repository properly installed so that shouldn't be the issue. I would think the graphics APT repository shouldn't be at fault but it could very well be given you have to install |
|
whackjob@WhackjobONE:/media/whackjob/16Tons/AI/ComfyUI$ wget --progress=dot:giga -qO - https://repositories.intel.com/graphics/intel-graphics.key | I tell ya, I am cursed. |
|
You might have to run the |
|
I appreciate all the help. No luck yet, it seems. whackjob@WhackjobONE: What's weird is this is almost a fresh install of linux mint. Literally, a week or two. |
|
It shouldn't be forbidden but it could be from Intel's end. Not much you can do there. Do you have |
|
Yessir. All four. whackjob@WhackjobONE: |
|
Sorry, I should've asked earlier. What is the output if you run the following: You should see something like this, especially the ext_oneapi_level_zero line. |
|
Well, I certainly have a variance! |
|
Okay, I think I know what is up. Can you install the latest release of intel-compute-runtime and then run the command again? |
|
Gave it a shot. Seems I already have it. whackjob@WhackjobONE: |
|
No like actually go to the link, download and install it. You have a slightly older version, the latest is 24.13.29138.7 |
|
Followed the link, made the directory, pulled 'em down, installed. It went through clean. Then gave it another shot. Same error, as follows. sycl-ls has different output now though. (venv) whackjob@WhackjobONE:/media/whackjob/16Tons/AI/ComfyUI$ python main.py --use-pytorch-cross-attention --highvram :: WARNING: setvars.sh has already been run. Skipping re-execution. usage: source setvars.sh [--force] [--config=file] [--help] [...] Some POSIX shells do not accept command-line options. In that case, you can pass $ SETVARS_ARGS="ia32 --config=config.txt" ; export SETVARS_ARGS The SETVARS_ARGS environment variable is cleared on exiting setvars.sh. [opencl:acc:0] Intel(R) FPGA Emulation Platform for OpenCL(TM), Intel(R) FPGA Emulation Device OpenCL 1.2 [2024.17.3.0.08_160000] |
|
Yeah so it was the fault of the compute runtime being outdated. I think if you do the sanity test and ComfyUI again, it should work as intended. I do need to sleep now so I can't aid you any further today but I hope everything works. |
|
Nah, I did give it a shot, still get the same iJIT_NotifyEvent. But we can call it a night. I do greatly appreciate the work you've put in and the time you've spent helping. If tomorrow night we can try a bit more, great, if not, it's all good. |
|
The suggestion from pytorch/pytorch#123097 seems to suggest downgrading Intel's MKL but I want to know what versions of those packages you have. Can you execute the following and give me the output? |
|
Hey, sure thing. Thanks for taking another swing at this conundrum. whackjob@WhackjobONE:~$ apt list --installed | grep -i intel-oneapi-mkl WARNING: apt does not have a stable CLI interface. Use with caution in scripts. intel-oneapi-mkl-classic-devel-2024.1/all,now 2024.1.0-691 amd64 [installed,automatic] |
|
All the versions match the setup I have, but I am actually now curious, before we try and downgrade the MKL packages, can you give me the output of all the intel packages you have? The output might be too long so if you need to attach it as a text file, that would be fine. |
|
whackjob@WhackjobONE:~$ apt list --installed | grep -i intel WARNING: apt does not have a stable CLI interface. Use with caution in scripts. intel-basekit-env-2024.1/all,now 2024.1.0-589 all [installed,automatic] |
|
Sorry for the delay but I think I found the issue. You are missing the runtime for MKL SYCL. Can you do the following install command and then try again and see if you are still getting the |
|
I think we've found the issue. Not quite sure how to fix this, though. whackjob@WhackjobONE: whackjob@WhackjobONE: |
|
You need to delete those files, I would presume, according to this Ubuntu forum post detailing the same issue. You can accomplish that with the following: Do that and try repairing your install and installing the runtime. |
|
whackjob@WhackjobONE: |
|
whackjob@WhackjobONE: |
|
All of that installed OK now. I'll give it another shot, now. |
|
whackjob@WhackjobONE:/media/whackjob/16Tons/AI/ComfyUI$ source venv/bin/activate :: WARNING: setvars.sh has already been run. Skipping re-execution. usage: source setvars.sh [--force] [--config=file] [--help] [...] Some POSIX shells do not accept command-line options. In that case, you can pass $ SETVARS_ARGS="ia32 --config=config.txt" ; export SETVARS_ARGS The SETVARS_ARGS environment variable is cleared on exiting setvars.sh. (venv) whackjob@WhackjobONE:/media/whackjob/16Tons/AI/ComfyUI$ source /opt/intel/oneapi/setvars.sh --force :: initializing oneAPI environment ... (venv) whackjob@WhackjobONE:/media/whackjob/16Tons/AI/ComfyUI$ python main.py --use-pytorch-cross-attention --highvram |
|
TL;DR, it seems to have all installed OK, now. But the same bloody error! |
|
Can you rerun the sanity test from #3513 (comment)? |
|
whackjob@WhackjobONE:/media/whackjob/16Tons/AI/ComfyUI$ python3 -c "import torch; import intel_extension_for_pytorch as ipex; print(torch.version); print(ipex.version); [print(f'[{i}]: {torch.xpu.get_device_properties(i)}') for i in range(torch.xpu.device_count())];" |
|
Everything should be then working because the sanity test directly pulls from IPEX. Your IPEX is now using GPU so it's a matter of your version of ComfyUI not working. Are you using the latest? If so, can you try and run IPEX without any arguments? So just run |
|
(venv) whackjob@WhackjobONE:/media/whackjob/16Tons/AI/ComfyUI$ python3 main.py I notice I'm on Python 3.10 and not 3.11. Could that be a factor? Or at least, it's defaulting to that |
|
That only matters if you installed IPEX using the wrong packages for a different Python version but that should all work given the sanity check worked but if you can clarify, were you running the VENV when you ran the sanity check? It doesn't jive that the sanity test gives you the right versions with detecting IPEX but then ComfyUI doesn't run correctly with those versions detected correctly. |
|
Yeah, deactivated the environment, fired it up again, sourced oneapi, then ran the sanity check. This is what I get: whackjob@WhackjobONE:/media/whackjob/16Tons/AI/ComfyUI$ source venv/bin/activate :: WARNING: setvars.sh has already been run. Skipping re-execution. usage: source setvars.sh [--force] [--config=file] [--help] [...] Some POSIX shells do not accept command-line options. In that case, you can pass $ SETVARS_ARGS="ia32 --config=config.txt" ; export SETVARS_ARGS The SETVARS_ARGS environment variable is cleared on exiting setvars.sh. (venv) whackjob@WhackjobONE:/media/whackjob/16Tons/AI/ComfyUI$ python3 -c "import torch; import intel_extension_for_pytorch as ipex; print(torch.version); print(ipex.version); [print(f'[{i}]: {torch.xpu.get_device_properties(i)}') for i in range(torch.xpu.device_count())];" |
|
So you have the correct IPEX install outside of your VENV but not inside your VENV, which is what I am concluding. Can you source inside your VENV and reinstall IPEX and then test if it works? I believe you had other problems given you didn't get the correct output until very recently and you were missing the MKL packages but I think if you reinstall correctly, things should just work then. |
|
I'll give it a shot in the morning, I think. I'm falling asleep at my desk. I should call it before I make some errors. Thanks for the help, again. I feel like we're right on the cusp. I'll post tomorrow. |
|
Who am I kidding, I won't sleep right. So I gave it a shot and just hoped I wouldn't fuck it up: (venv) whackjob@WhackjobONE:/media/whackjob/16Tons/AI/ComfyUI$ python -m pip install torch==2.1.0.post2 torchvision==0.16.0.post2 torchaudio==2.1.0.post2 intel-extension-for-pytorch==2.1.30+xpu oneccl_bind_pt==2.1.300+xpu --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/ :: WARNING: setvars.sh has already been run. Skipping re-execution. usage: source setvars.sh [--force] [--config=file] [--help] [...] Some POSIX shells do not accept command-line options. In that case, you can pass $ SETVARS_ARGS="ia32 --config=config.txt" ; export SETVARS_ARGS The SETVARS_ARGS environment variable is cleared on exiting setvars.sh. (venv) whackjob@WhackjobONE:/media/whackjob/16Tons/AI/ComfyUI$ python3 main.py Import times for custom nodes: Starting server To see the GUI go to: http://127.0.0.1:8188 |
|
Brother, you taught me a lot today. How to troubleshoot this and such. Let me know how I can send you a few bucks for your trouble. You spent a lot of time on this, and I appreciate it. |
|
I'm fine, just hope you can pass it on and help someone else whether regarding this or something else. Please close the issue when you are able to. Some extra small nuggets of info.
|
|
I'm using the A770 16GB. Not the greatest, but it's not bad, either. I've been staring down that ATI Radeon with what was it, 42GB of RAM? I remember zLUDA was continued for AMD stuff. A shame they didn't keep it up for intel's stuff! At least, nobody took that torch up yet. |
It's probably something stupid and easy to fix, but I've had zero success googling and troubleshooting this error myself. I'm just at a loss. I've tried to redo this several times without luck.
(venv) whackjob@WhackjobONE:/media/whackjob/16Tons/AI/ComfyUI$ ipexrun main.py --use-pytorch-cross-attention --highvram
Traceback (most recent call last):
File "/media/whackjob/16Tons/AI/ComfyUI/venv/bin/ipexrun", line 5, in
from intel_extension_for_pytorch.launcher import main
File "/media/whackjob/16Tons/AI/ComfyUI/venv/lib/python3.10/site-packages/intel_extension_for_pytorch/init.py", line 3, in
import torch
File "/media/whackjob/16Tons/AI/ComfyUI/venv/lib/python3.10/site-packages/torch/init.py", line 229, in
from torch._C import * # noqa: F403
ImportError: /media/whackjob/16Tons/AI/ComfyUI/venv/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so: undefined symbol: iJIT_NotifyEvent
The text was updated successfully, but these errors were encountered: