Join the Conversation: Wunjo AI Community Forum

[wladradchenko]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[wladradchenko]

I will be glad to answer your questions

[realyussel]

Hello, your project is great, however I speak Spanish and I would like to know how to create a voice cloning synthesis model for my language. Even if you could give me some direction, I would like to work with you to integrate Spanish into the available languages.

[wladradchenko]

Hello,

Thank you for your interest in Wunjo project and for your willingness to collaborate in expanding its capabilities to include the Spanish language. This is sound cool.

Step 1: Repository and Initial Setup

Firstly, I use the repository available at Real-Time-Voice-Cloning GitHub for cloning voices. Here, you only need to train the synthesizer.pt as the default encoder and vocoder can be utilized as is. These default models will be automatically generated in the saved_models directory during the first run.

Step 2: Dataset Preparation

Finding a high-quality Spanish dataset is pivotal. Although I'm not certain of where to obtain a Spanish dataset, it should have a structure similar to that of LibriTTS:

datasets_root
     └─LibriTTS
         └─train-clean-100
             └─speaker-001
                 └─book-001
                     ├─utterance-001.wav
                     ├─utterance-001.txt
                     ├─utterance-002.wav
                     └─utterance-002.txt

It means what you can have book-001, bool-002, book-003, etc. in speaker-001. Also you can have speaker-001, speaker-002, speaker-003, etc. in train-clean-100. Name of audio equivalent to name of txt file, in txt file will be text from audio.

Step 3: Adapting for the Spanish Language

To tailor the system for the Spanish language, adjust the symbols in the symbols.py file to include Spanish letters.

Step 4: Data Preprocessing

Once you have your dataset, preprocess the audio files using the following commands:

python synthesizer_preprocess_audio.py datasets_root --datasets_name LibriTTS --subfolders train-clean-100 --no_alignments
python synthesizer_preprocess_embeds.py datasets_root/SV2TTS/synthesizer/

Step 5: Training the Model

Start the training with the following command:

python synthesizer_train.py spanish datasets_root/SV2TTS/synthesizer

To expedite the training process, modify the batch size in the hparams.py file:

tts_schedule = [
    (2, 1e-3, 20_000, 32),
    (2, 5e-4, 40_000, 32),
    (2, 2e-4, 80_000, 32),
    (2, 1e-4, 160_000, 32),
    (2, 3e-5, 320_000, 32),
    (2, 1e-5, 640_000, 32)
]

and set tts_cleaner_names as basic_cleaners to not miss specific languages letter without ASCII translate.

tts_cleaner_names = ["basic_cleaners"]

P.S. It is important to note that a larger number of iterations does not necessarily imply a better-performing model. For instance, during my experience training a model to clone a Russian voice, the optimum quality was achieved at the 175,000th iteration. Surprisingly, at 200,000 and even beyond 250,000 iterations, the model began to generate a robotic voice. I speculate that this anomaly might be related to the dataset I used, which is available at open_stt. Upon closer examination of the dataset, I noticed it contains audio files with artificial Russian voices, which could potentially influence the training outcomes. In test.zip you will able to listen how Russian cloning voice sound and difference iteration.

When the training process will finish you would find Spanish synthesizer.pt in synthesizer/saved_models/spanish. You can test the quality of the trained model by running python demo_cli.py -s synthesizer_path_model/synthesizer.pt.

And in order for me to implement voice cloning for the Spanish Language in Wunjo AI, I only need symbols.py file including Spanish letters and the synthesizer.pt model.

[wladradchenko]

Also, I asked ChatGPT about Spanish:

Spanish Linguistic Characteristics

1. Diacritical Marks

Spanish uses several diacritical marks, including:

• Acute accents (´): used to mark stress or to distinguish between homonyms, e.g., "sí" (yes) vs. "si" (if).
• Tilde (~): used in "ñ" to represent a palatal nasal sound. (but in cloning voice I use ~ to split phrase on small phrases, this is not problem to change)
• Dieresis (¨): used in "ü" to indicate that "u" should be pronounced in the combinations "güe" and "güi".

You might want to add modules that correctly handle words with these marks, ensuring that they are correctly pronounced.

P.S. However, unlike Russian, in Spanish it is clear that there is something above the letter, so it seems to me that there is no need to do anything for this. Just save the original letter, and the neural network will understand that it needs to be emphasized, etc. The symbols are enough for this.

2. Homographs

Similar to Russian, Spanish has homographs (words spelled the same but pronounced differently), e.g., "te" (tea) and "te" (you - object pronoun). You might consider adding a module to correctly emphasize these words based on the context.

3. Stress

Word stress is significant in Spanish. Most words follow regular stress rules, but there are exceptions. You might add a module that correctly identifies the stressed syllable in each word, including those that don't follow the standard rules.

Modules for Spanish

Given the linguistic characteristics outlined above, here are some modules you might consider for Spanish:

1. Acute Accent Handler: To correctly handle words with acute accents. (I think enough keep ´ upper character)
2. Tilde Handler: To correctly handle words containing the "ñ" character. (I think enough keep ~ upper character)
3. Dieresis Handler: To manage the correct pronunciation of "ü" in various contexts. (I think enough keep ¨ upper character)
4. Homograph Handler: To differentiate between homographs based on context.
5. Stress Handler: To identify and correctly pronounce the stressed syllable in each word.

You can see witch I use Russian files in folder. In order to Spanish speech will be more clear, need to modify the TPS module in Wunjo AI. I can do this for the Spanish language, however I just need these files for the Spanish words to I will use replace.

If add this in Wunjo AI, I think what will be available training speech model in interface Wunjo AI. I have recorded video about this.

[omikron88]

[wladradchenko]

Hello,

For project launch and bug-related questions, please contact the owner of the original repository at https://github.com/CorentinJ/Real-Time-Voice-Cloning/wiki/Training.

Hovewer, to get started with the Real-Time-Voice-Cloning project, I recommend installing numpy==1.23.5 first for compatibility. You can do this using:

python -m pip install numpy==1.23.5

After that, proceed with the rest of the dependencies as per the project's documentation. Also you can try to install Python3.10 and use ones in that project.

[OneNiNE87]

Hi,

When we run the instance on PC, can it be used locally on phones??

[wladradchenko]

In Wunjo AI a lot neural network models witch can not be adaptive to run on smartphone and requires a VRAM and powerful processor. But u can use backend from Wunjo AI on linux server and mobile frontend witch will work with post and get requests.

For now, I don’t plan to make a mobile application, because I haven’t fully realized my vision in the desktop version yet.

[OneNiNE87]

I see and my best wishes are with you and your app. Wunjo in my opinion is so far on of the best app and it's very under-rated. I hope it gets the hype it deserves soon. I have been trying to build the Windows GPU version but I haven't got success yet. It would be more than great if you can check the steps once and update if there is any changes required. I have been able to fix some of the mismatches but still not able to build a result. I am a newbie :))

[wladradchenko]

Thank you for your kind words and support! It's truly motivating to hear that Wunjo has user like you. If you're enjoying the app, sharing your experience on platforms like Reddit, YouTube, TikTok or any platform can really help others discover it too, and I would be incredibly grateful for that.

Regarding the Windows GPU build, I'm curious to know if there's a specific reason the portable version from an official website isn't meeting your needs. This could help me understand and address any gaps in the current offerings.

When I built the portable version for Windows GPU I have recorded a video of display screen, so that I could later transfer the steps into the documentation, that video without comments and editing, since it is for personal use, but you can see all the steps. If you think this uncut insight into the build process could assist you, I'd be happy to share it.

[OneNiNE87]

Thank you for your kind words and support! It's truly motivating to hear that Wunjo has user like you. If you're enjoying the app, sharing your experience on platforms like Reddit, YouTube, TikTok or any platform can really help others discover it too, and I would be incredibly grateful for that.

Regarding the Windows GPU build, I'm curious to know if there's a specific reason the portable version from an official website isn't meeting your needs. This could help me understand and address any gaps in the current offerings.

When I built the portable version for Windows GPU I have recorded a video of display screen, so that I could later transfer the steps into the documentation, that video without comments and editing, since it is for personal use, but you can see all the steps. If you think this uncut insight into the build process could assist you, I'd be happy to share it.

Hi,

Yes and thank you. I noticed the new download info. I am sharing the app today on the platforms i am using.

Thank you

[OneNiNE87]

Thank you for your kind words and support! It's truly motivating to hear that Wunjo has user like you. If you're enjoying the app, sharing your experience on platforms like Reddit, YouTube, TikTok or any platform can really help others discover it too, and I would be incredibly grateful for that.

Regarding the Windows GPU build, I'm curious to know if there's a specific reason the portable version from an official website isn't meeting your needs. This could help me understand and address any gaps in the current offerings.

When I built the portable version for Windows GPU I have recorded a video of display screen, so that I could later transfer the steps into the documentation, that video without comments and editing, since it is for personal use, but you can see all the steps. If you think this uncut insight into the build process could assist you, I'd be happy to share it.

I am almost new to this, and I am still learning a lot from you. I cannot express my gratitude towards you and your project. It's helping me learn a lot like I recently burned my Ryzen 7cpu. 😂

[omikron88]

Is this useable to implement Czech:
https://github.com/comodoro/deepspeech-cs

[wladradchenko]

Hi, I opened a discussion for this by link

[JunYu1996]

For some reason, wunjo cannot be started using the easy installation method and manual transfer method, and it will automatically open a page with 127.0.0.1 @wladradchenko

[wladradchenko]

Hello. This is flask app, of course this will start in localhost and browser. It is also worth noting that launch problems are not a topic for discussion, but issues. And in order to understand issue beed to compare more information.

Anyway, you can visit documentation page or watch video how install Wunjo from code or installation. There are absolutely all the answers to any problems that users have had.

You may have a launch problem because you launched more than once and did not close previous sessions. To close the session, close your browser, or use a special browser that I made for the application. With this browser there will be no problems with re-enabling.