Can I train the Chinese model?

[Tsangchi-Lam]

I want to train the Chinese model. Do you support mixed input in Chinese and English?

[Kreevoz]

Look at issue #41 to check the current progress.

[yl4579]

You can, but with the current PL-BERT in English the quality won’t be as good it’s originally proposed to be. I’m working on multilingual PL-BERT now and it may take one or two months to finish.

[PleinAcces]

Hi @yl4579
Would it also be necessary to train new WavLM models for languages other than English?

[yl4579]

See yl4579/StyleTTS#10 for more details.

[hermanseu]

@yl4579 I trained styletts2 successfully using Chinese data, it sound very good. As wavlm-base-plus only supporting English, I used a Chinese Hubert model as SLM. When I want to train a model both for Chinese and English, I can not find a pre-trained model sopport Chinese and English at the same time. About SLM，Do you have any suggestions ?

[zhouyong64]

Could you tell me which Hubert model did you use? Also, what's your SLM configuration in StyleTTS's config.yml for this SLM model?

slm:
model: 'microsoft/wavlm-base-plus'
sr: 16000 # sampling rate of SLM
hidden: 768 # hidden size of SLM
nlayers: 13 # number of layers of SLM
initial_channel: 64 # initial channels of SLM discriminator head

[hermanseu]

I use the hubert model of https://github.com/TencentGameMate/chinese_speech_pretrain as SLM for Chinese, and use whisper encoder as SLM for multi-language(Chinese and English currently).

[SoshyHayami]

@hermanseu Can you tell me how did you use the Whisper encoder for this purpose? any code snippets would help, thanks.

[yl4579]

You can try whisper encoder that was trained with multiple languages. You can also try multilingual wav2vec2.0: https://huggingface.co/facebook/wav2vec2-large-xlsr-53

[zhouyong64]

@yl4579 I trained styletts2 successfully using Chinese data, it sound very good.

Did you use the English PL-BERT or did you train PL-BERT with Chinese data?

[hermanseu]

train PL-BERT with Chinese data

[11721206]

is convenient for you to provide your chinese PL-BERT model， thanks very much

[Moonmore]

I trained styletts2 successfully using Chinese data, it sound very good. As wavlm-base-plus only supporting English, I used a Chinese Hubert model as SLM. When I want to train a model both for Chinese and English, I can not find a pre-trained model sopport Chinese and English at the same time. About SLM，Do you have any suggestions ?

What is your modeling unit? IPA or Pinyin?

[hermanseu]

@Moonmore The modeling unit is pinyin.

test.zip is a synth sample.

[zhouyong64]

@Moonmore The modeling unit is pinyin.

test.zip is a synth sample.

Do you use the tone of pinyin when training Chinese PL-BERT? I believe StyleTTS uses F0 for Chinese tones. Can this PL-BERT with tones work with StyleTTS?

[hermanseu]

I trained Chinese PL-BERT without pinyin tones. But maybe PL-BERT with tones will also work normally, so you can try.

[zhouyong64]

I trained Chinese PL-BERT without pinyin tones. But maybe PL-BERT with tones will also work normally, so you can try.

How many samples did you use to train Chinese PL-BERT?

[hermanseu]

@zhouyong64 I used about 84,000,000 text sentences to train the Chinese PL-BERT model.

[Moonmore]

@Moonmore The modeling unit is pinyin.

test.zip is a synth sample.

Sounds really good. I would like to ask if the pinyin unit you mentioned cannot be disassembled into phones? How to align plbert and text input?

[hermanseu]

@Moonmore
I used the same pinyin phonemes(sheng1 mu3 yun4 mu3) to train all the models. But when training asr, I used the phonemes without tones. if the pinyin uint cannot be disassembled, maybe the pinyin can be regard as an phoneme.

@zhouyong64 Sorry for the wrong information of yesterday, I tained PL-BERT with tones, and trained asr without tones.

I trained Chinese PL-BERT without pinyin tones. But maybe PL-BERT with tones will also work normally, so you can try.

[Moonmore]

@Moonmore I used the same pinyin phonemes(sheng1 mu3 yun4 mu3) to train all the models. But when training asr, I used the phonemes without tones. if the pinyin uint cannot be disassembled, maybe the pinyin can be regard as an phoneme.

@zhouyong64 Sorry for the wrong information of yesterday, I tained PL-BERT with tones, and trained asr without tones.

I trained Chinese PL-BERT without pinyin tones. But maybe PL-BERT with tones will also work normally, so you can try.

So can I understand that all text-related models are trained using the same phoneme unit, and the characteristics of each minimum pronunciation modeling unit are obtained. like(ni3 hao3 -> n i3 h ao3), The input length is 4, and the output length of the model is also 4. text encoder and the bert model. and how to construct the plbert label?

[hermanseu]

@Moonmore
Yes, the output lengths of text encoder and bert are same as input lengths.
About plbert label, you can read the logic of dataloader.py in plbert repo. It explained clearly.

[Moonmore]

@Moonmore Yes, the output lengths of text encoder and bert are same as input lengths. About plbert label, you can read the logic of dataloader.py in plbert repo. It explained clearly.

@hermanseu Thank you for your reply.

[georgedei]

How can the above be applied to StyleTTS2? Is there a complete repo already I could look up that is specialized on Mandarin using this G2PW? As a non-expert I am looking at the puzzle pieces but don't see the entire picture. Perhaps its too early in the development.

[yijingshihenxiule]

@hermanseu ,兄弟，请问一下你在训asr模块的时候，意思是分解音素时没有声调吗？ 比如sheng1 mu3 yun4 mu3 --> sheng mu yun mu？
我尝试了加声调，比如sheng1 mu3 yun4 mu3 --> sheng1 mu3 yun4 mu3，训练asr部分的时候出现了负ctc loss，不知道是什么原因。