-
Notifications
You must be signed in to change notification settings - Fork 333
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Loud white noise on short texts with embedding_scale > 1 #46
Comments
Nvm also seems to happen on single words, setting both alpha and beta to zero makes it return to normal though. |
This is not supposed to occur. Setting alpha and beta to 0 means not using the diffusion model at all. What are your packages versions? |
Windows 10 And loading the model to the GPU instead of CPU, but I'll do some further testing to narrow it down |
Could you make a conda environment with Python 3.10 instead? You can run the colab demo and check package versions there and make sure you install these packages instead. |
I tried 3.10 locally, but exactly the same issue. |
Tried it in collab aswell, if I put in just the word "wink" as inference text, it will give me bad white noise. |
I think it could be due to not such a training sample during training. The model has never seen a single word during training (because we removed speech shorter than one second). |
Is there a way around it without retraining it, like dropping the diffusion model on short inputs? |
You can add some filler words before or after the word you want to speak and cut the audio to only get the word you are interested in. |
not a pretty solution either :) |
I got the same problem - it's probably not related to the sentence length but on some first words in the sentance like: "you", "me". For example 'You can do that too." [58] - produces white noise on Colab. So for example if in the longer text there is a sentance starting with "You will need .." this will corrupt audio afterwards. |
@easyrider I have tried this |
@AWAS666 It still works even with a single word |
You are correct, adding the fullstop helps fix it. |
This is a very interesting issue. During training the guidance scale is 1, and for some reason when the input is small it fails to generalize to higher guidance scale. I think probably during training we may have to vary the guidance scale randomly from 1 to 2 then? I will try to do this just for the 2nd stage and see if the problem disappears. |
Great to hear |
I have the same problem specifically for short sentences/phrases (all with puncutation) running on MacOS M2. I noticed that it seems to be more likely when the sentence length is less than about 40 characters. I was already doing TTS on longform audio, so I wrote a script that splits up sentences but if any sentence is less than 40 characters it attaches it to the previous or next sentence. This way every block of text I processed with StyleTTS2 is longer than 40 characters. That fixed the problem entirely for me. I didn't make any changes to punctuation and didn't change any of the words in the text. |
Fix error when resuming training
If I give it a short text like "Hello how are you?" it generates me a 25 second clip of extremly loud white noise.
I tried this on the libritts model only, it also happens on my own model which I finetuned based on it.
The text was updated successfully, but these errors were encountered: