Failed to install on Apple Silicon

[fkostadinov]

Not sure how others are doing, but I just failed to install this on my Apple Silicon Macbook. Giving up now. I am simply recording this for others who might be more lucky or knowledgeable than me and perhaps figure out the missing steps.

First, had to install Python 3.10, because 3.12 led to errors with setuptools no longer being supported, and 3.11 led to some other errors with dependencies in requirements.txt asking to be <3.10. Which was odd, because elsewhere (cannot find that anymore) the authors of the library apparently were using 3.10. No clue how to interpret that.

brew install python@3.10

Which installed Python 3.10.14. Then, pointing to Python 3.10.14 I created a virtual environment in my project:

python3 -m venv venv

Next, install cythong:

pip install cython

No errors so far.

Now trying to install requirements.txt.

pip install requirements.txt

Bad luck. Got the following errors:

INFO: pip is looking at multiple versions of nemo-toolkit to determine which version is compatible with other requirements. This could take a while. ERROR: Ignored the following versions that require a different python version: 0.52.0 Requires-Python >=3.6,<3.9; 0.52.0rc3 Requires-Python >=3.6,<3.9; 0.53.0 Requires-Python >=3.6,<3.10; 0.53.0rc1.post1 Requires-Python >=3.6,<3.10; 0.53.0rc2 Requires-Python >=3.6,<3.10; 0.53.0rc3 Requires-Python >=3.6,<3.10; 0.53.1 Requires-Python >=3.6,<3.10; 0.54.0 Requires-Python >=3.7,<3.10; 0.54.0rc2 Requires-Python >=3.7,<3.10; 0.54.0rc3 Requires-Python >=3.7,<3.10; 0.54.1 Requires-Python >=3.7,<3.10 ERROR: Could not find a version that satisfies the requirement triton (from nemo-toolkit) (from versions: none) ERROR: No matching distribution found for triton

Triton server, eh? After some research I figured I need to install pytorch v2.x.

pip3 install torch torchvision torchaudio

Okay, that went well. Now I went on to try to install Nemo toolkit (cannot exactly recall why I took this step at this point in the process.)

pip install 'nemo_toolkit[asr]'

(Note, depending on the shell you're using the quotes must be placed like so pip install nemo_toolkit['asr'].) That did not work.

Collecting youtokentome>=1.0.5 (from nemo_toolkit[asr]) Downloading youtokentome-1.0.6.tar.gz (86 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 86.7/86.7 kB 12.9 MB/s eta 0:00:00 Installing build dependencies ... done Getting requirements to build wheel ... error error: subprocess-exited-with-error

Ach. What's that now. Let's try to install it.

`pip install “git+https://github.com/LahiLuk/YouTokenToMe”

Nope, does not work.

ERROR: Invalid requirement: '“git+https://github.com/LahiLuk/YouTokenToMe”' Hint: It looks like a path. File '“git+https://github.com/LahiLuk/YouTokenToMe”' does not exist.

Stupid me, it's the quotes. Let's try again.

pip install git+https://github.com/LahiLuk/YouTokenToMe

Aaaaand - fail.

`Collecting fasttext (from nemo_toolkit[all])
Downloading fasttext-0.9.2.tar.gz (68 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 68.8/68.8 kB 3.5 MB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [25 lines of output]
/Users/myself/workspaces/whisper-diarization/venv/bin/python3.10: No module named pip
Traceback (most recent call last):
File "", line 38, in init
ModuleNotFoundError: No module named 'pybind11'`

We need pybind11. Got it.

pip install pybind11

That went smooth. Which is kinda ironic, given that we just tried to install it as a dependency of fasttext. Whatever.

Now I cannot quite remembers why or what I did in between but I think I somehow mistakenly tried to install nemo_toolkit['all'], which then failed, and then I realized I needed probably only the NLP package.

pip install "nemo_toolkit[nlp]"

Which failed again telling me that pybind11 is not available.

`Collecting fasttext (from nemo_toolkit[nlp])
Using cached fasttext-0.9.2.tar.gz (68 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [25 lines of output]
/Users/myself/workspaces/whisper-diarization/venv/bin/python3.10: No module named pip
Traceback (most recent call last):
File "", line 38, in init
ModuleNotFoundError: No module named 'pybind11'`

WTF. After googling I found out that this potentially could be solved by installing wheel first. (Not that I have any clue what it is, nor that I ever want to find out.)

pip install wheel

Success. Let's try fasttext again.

pip install fasttext

Success! This starts looking promising. Let's try pybind11 again.

pip install pybind11

Success again! Can we install the Token-thingy again?

pip install git+https://github.com/LahiLuk/YouTokenToMe

Ah, seems like it's already installed by now. What about nemo_toolkit[asr]?

pip install "nemo_toolkit[asr]"

Already installed again, no errors. Darn, that's good, isn't it? Let's go for the real thing now.

python diarize.py -a my_audio_file.m4a

BIG FAIL.

OSError: dlopen(libopencc.so.1, 0x0006): tried: 'libopencc.so.1' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibopencc.so.1' (no such file), '/opt/homebrew/lib/libopencc.so.1' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/lib/libopencc.so.1' (no such file), '/usr/lib/libopencc.so.1' (no such file, not in dyld cache), 'libopencc.so.1' (no such file)

What the heck does that mean now? Maybe there's an issue with the filename containing spaces? Let's rename without spaces.

Aaaand, nope. No luck still.

Whatever, I'm giving up. I have no freaking idea what's going on here. Below's the entire error message I got in the end.

[fkostadinov]

This is the entire error message I got in the end.

`/Users/myself/workspaces/whisper-diarization/venv/lib/python3.10/site-packages/pyannote/audio/core/io.py:43: UserWarning: torchaudio._backend.set_audio_backend has been deprecated. With dispatcher enabled, this function is no-op. You can remove the function call.
  torchaudio.set_audio_backend("soundfile")
Traceback (most recent call last):

  File "/Users/myself/workspaces/whisper-diarization/diarize.py", line 3, in <module>
    from helpers import *
  File "/Users/myself/workspaces/whisper-diarization/helpers.py", line 7, in <module>
    from whisperx.alignment import DEFAULT_ALIGN_MODELS_HF, DEFAULT_ALIGN_MODELS_TORCH
  File "/Users/myself/workspaces/whisper-diarization/venv/lib/python3.10/site-packages/whisperx/__init__.py", line 1, in <module>
    from .transcribe import load_model
  File "/Users/myself/workspaces/whisper-diarization/venv/lib/python3.10/site-packages/whisperx/transcribe.py", line 10, in <module>
    from .asr import load_model
  File "/Users/myself/workspaces/whisper-diarization/venv/lib/python3.10/site-packages/whisperx/asr.py", line 13, in <module>
    from .vad import load_vad_model, merge_chunks
  File "/Users/myself/workspaces/whisper-diarization/venv/lib/python3.10/site-packages/whisperx/vad.py", line 11, in <module>
    from pyannote.audio.pipelines import VoiceActivityDetection
  File "/Users/myself/workspaces/whisper-diarization/venv/lib/python3.10/site-packages/pyannote/audio/pipelines/__init__.py", line 26, in <module>
    from .speaker_diarization import SpeakerDiarization
  File "/Users/myself/workspaces/whisper-diarization/venv/lib/python3.10/site-packages/pyannote/audio/pipelines/speaker_diarization.py", line 42, in <module>
    from pyannote.audio.pipelines.speaker_verification import PretrainedSpeakerEmbedding
  File "/Users/myself/workspaces/whisper-diarization/venv/lib/python3.10/site-packages/pyannote/audio/pipelines/speaker_verification.py", line 56, in <module>
    from nemo.collections.asr.models import (
  File "/Users/myself/workspaces/whisper-diarization/venv/lib/python3.10/site-packages/nemo/collections/asr/__init__.py", line 15, in <module>
    from nemo.collections.asr import data, losses, models, modules
  File "/Users/myself/workspaces/whisper-diarization/venv/lib/python3.10/site-packages/nemo/collections/asr/models/__init__.py", line 36, in <module>
    from nemo.collections.asr.models.transformer_bpe_models import EncDecTransfModelBPE
  File "/Users/myself/workspaces/whisper-diarization/venv/lib/python3.10/site-packages/nemo/collections/asr/models/transformer_bpe_models.py", line 52, in <module>
    from nemo.collections.nlp.modules.common import TokenClassifier
  File "/Users/myself/workspaces/whisper-diarization/venv/lib/python3.10/site-packages/nemo/collections/nlp/__init__.py", line 15, in <module>
    from nemo.collections.nlp import data, losses, models, modules
  File "/Users/myself/workspaces/whisper-diarization/venv/lib/python3.10/site-packages/nemo/collections/nlp/models/__init__.py", line 31, in <module>
    from nemo.collections.nlp.models.machine_translation import MTEncDecModel
  File "/Users/myself/workspaces/whisper-diarization/venv/lib/python3.10/site-packages/nemo/collections/nlp/models/machine_translation/__init__.py", line 15, in <module>
    from nemo.collections.nlp.models.machine_translation.mt_enc_dec_bottleneck_model import MTBottleneckModel
  File "/Users/myself/workspaces/whisper-diarization/venv/lib/python3.10/site-packages/nemo/collections/nlp/models/machine_translation/mt_enc_dec_bottleneck_model.py", line 23, in <module>
    from nemo.collections.nlp.models.machine_translation.mt_enc_dec_model import MTEncDecModel
  File "/Users/myself/workspaces/whisper-diarization/venv/lib/python3.10/site-packages/nemo/collections/nlp/models/machine_translation/mt_enc_dec_model.py", line 38, in <module>
    from nemo.collections.common.tokenizers.chinese_tokenizers import ChineseProcessor
  File "/Users/myself/workspaces/whisper-diarization/venv/lib/python3.10/site-packages/nemo/collections/common/tokenizers/chinese_tokenizers.py", line 38, in <module>
    import opencc
  File "/Users/myself/workspaces/whisper-diarization/venv/lib/python3.10/site-packages/opencc.py", line 24, in <module>
    libopencc = CDLL('libopencc.so.1', use_errno=True)
  File "/opt/homebrew/Cellar/python@3.10/3.10.14/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)

OSError: dlopen(libopencc.so.1, 0x0006): tried: 'libopencc.so.1' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibopencc.so.1' (no such file), '/opt/homebrew/lib/libopencc.so.1' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/opt/homebrew/lib/libopencc.so.1' (no such file), '/usr/lib/libopencc.so.1' (no such file, not in dyld cache), 'libopencc.so.1' (no such file)`

[rosstex]

Same

[MahmoudAshraf97]

Hi everyone, I'm currently working on a branch that doesn't use whisperX alignment thus skipping the pyannote requirement that we don't even use in our project, I'll let you know when it's done

[keeth]

i fixed this with:

brew install opencc
ln -s /opt/homebrew/lib/libopencc.dylib /opt/homebrew/lib/libopencc.so.1

(you just need to symlink it to one of the paths that it is searching for)

now unfortunately i get a KeyError when i pass --device mps so i am stuck using the CPU for this :(

[MahmoudAshraf97]

@keeth Unfortunately I don't have any device to test MPS support, can you contact me through mail or linkedin and I'll gladly add support for it.

[Sing303]

Will nemo work without an Nvidia graphics card?

[MahmoudAshraf97]

It uses PyTorch underneath so it will work

[pakerfeldt]

Idk, I'm struggling like crazy as well. Tried to get this installed on an Ubuntu server. Gave up and tried to follow your steps here to install on my Macbook with Apple Silicon.

Last thing I did was installing opencc using brew and putting the so in the same folder as I'm running diarize.py. The error I get when running is:

ModuleNotFoundError: No module named 'ctc_forced_aligner'

Makes sense because I did run the requirements.txt manually one by one and skipped ctc_forced_aligner since it .. doesn't work 😄

Running pip install git+https://github.com/MahmoudAshraf97/ctc-forced-aligner.git yields

      /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__algorithm/max.h:31:1: note: candidate function template not viable: requires 3 arguments, but 2 were provided
      max(const _Tp& __a, const _Tp& __b, _Compare __comp)
      ^
      ctc_forced_aligner/forced_align_impl.cpp:131:33: error: no matching function for call to 'max'
              ltrIdx - backPtr_offset[std::max(t - 1, static_cast<long int>(0))];
                                      ^~~~~~~~
      /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__algorithm/max.h:40:1: note: candidate template ignored: deduced conflicting types for parameter '_Tp' ('int64_t' (aka 'long long') vs. 'long')
      max(const _Tp& __a, const _Tp& __b)
      ^
      /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__algorithm/max.h:51:1: note: candidate template ignored: could not match 'initializer_list<_Tp>' against 'int64_t' (aka 'long long')
      max(initializer_list<_Tp> __t, _Compare __comp)
      ^
      /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__algorithm/max.h:60:1: note: candidate function template not viable: requires single argument '__t', but 2 arguments were provided
      max(initializer_list<_Tp> __t)
      ^
      /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__algorithm/max.h:31:1: note: candidate function template not viable: requires 3 arguments, but 2 were provided
      max(const _Tp& __a, const _Tp& __b, _Compare __comp)
      ^
      2 errors generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for ctc-forced-aligner
  Running setup.py clean for ctc-forced-aligner
Failed to build ctc-forced-aligner
ERROR: Could not build wheels for ctc-forced-aligner, which is required to install pyproject.toml-based projects

[MahmoudAshraf97]

@pakerfeldt Hi, this is the new alignment package that was merged recently, can you send me an email or a message on linkedin so we can debug this?

[pakerfeldt]

@MahmoudAshraf97 Sent you a message on LinkedIn!