Rare letters in toki pona [BUG]

[gtbot2007]

You are unable to use uncommon letters in toki pona sentences. Due to this, words such as "yupekosi", "Pingo" and "kalamARR" (among a few others) can not be submitted.

[jessicarose]

Hi @gtbot2007 and thanks so much for raising this issue, could I get a bit more detail in what's preventing uncommon letters from being used? Is this a limitation you've found in the guidelines that could be better clarified or are you getting a bug or error message when trying to input sentences that contain words like "yupekosi", "Pingo" and "kalamARR"?

[HarikalarKutusu]

I believe at least the last one happens because of the default validation rules defined below, which tries to handle abbreviations:

common-voice/server/src/core/sentences/validation/languages/default-locale.ts

Line 41 in 1ba0c95

{

For other cases you need to create language specific validation rules, like the ones here:
https://github.com/common-voice/common-voice/tree/main/server/src/core/sentences/validation/languages

[gtbot2007]

Since they are used so few words and the words are also ”non-standard” they are considered foreign letters. Which is technically true but maybe there should be a case to allow the words “Pingo”, “yupekosi”, “kalamARR”, “yutu” and maybe even “yu” and “y”.

[HarikalarKutusu]

Sorry for the noise, I posted before checking. There IS actually a tok validation file here:
https://github.com/common-voice/common-voice/blob/main/server/src/core/sentences/validation/languages/tok.ts

It seems, each one of the samples you gave is hitting one regex rule there. Either they are not wanted, or the regex rules might need tweaks.