Building and Using a Custom Dictionary

[barrust]

Building and Using a Custom Dictionary

Building a custom dictionary is a great way to utilize the pyspellchecker library. This thread is based on issues that I have run across multiple times and I figured I would put a summary of all the information in a single location. This thread can be updated and added to as needed!

Building a custom dictionary is more preference than anything. Using external data sources can lead to misspellings to unexpectedly occur. This is even the case with the default dictionaries as they are built using the open source subtitles projects. That being said, here are a few tips.

• Find your source to use, usually as a text file
• Decide if the dictionary is to augment the data in a previous dictionary or if it is a stand alone dictionary.
• Once built (see below for examples) check for known errors and remove as needed
• Export dictionary and follow using and loading of custom dictionary tips

There are several methods to load data into a dictionary:

• spell.word_frequency.load_dictionary()
• spell.word_frequency.load_text()
• spell.word_frequency.load_json()
• spell.word_frequency.load_text_file()
• spell.word_frequency.load_words()

Using one of the above function you can now build your custom dictionary.

Stand alone dictionary

This is a new language or person specific dictionary that is not built on a previous language. Make sure to not load a dictionary before building the new dictionary!

from spellchecker import SpellChecker

# if stand alone, make sure to use language=None
spell = SpellChecker(language=None)

spell.word_frequency.load_text_file('my_text_file.txt')

Adding to a default dictionary

In this case, it is recommended to act like it is a stand alone dictionary and set the language parameter to None and load the default dictionary before loading the custom dictionary. This will allow the user to get any future changes to the default dictionary.

from spellchecker import SpellChecker

spell = SpellChecker(language=None)

spell.word_frequency.load_text_file('my_text_file.txt')

Modifying a default dictionary

If you are planning to modify a default dictionary for personal use (not to be sent back to the library) then you will want to load the correct dictionary, modify and export it but then do not load the original dictionary again when using the updated, custom dictionary.

from spellchecker import SpellChecker

spell = SpellChecker()  # load the default English dictionary

# add words that are missing / needed
spell.word_frequency.load_text_file('my_text_file.txt')

# remove some things that are incorrect
spell.remove("the") # I don't like 'the'

Once the data is originally loaded, run some tests to ensure the data is as correct and as expected. Check for commonly misspelled words in your dictionary to make sure they are not present and also check for known words.

# check for some misspelled words
common_misspelled = ['tihs', 'alot']
for word in common_misspelled:
    if word in spell:
        spell.remove(word)

# make sure specific words are present
common_words = ['accept', 'another']
for word in common_misspelled:
    if word not in spell:
        spell[word] = 1  # put it in if missing

Once the data is set, the data can then be exported to be used in the future.

# generally, do not gzip the dictionary until it has been fully confirmed
# to be correct!
spell.export('my_custom_dict.json', gzipped=False)

Using the Custom dictionary

The first major thing that trips up users is loading in the custom dictionary when another dictionary is already loaded. Thus, the user is getting a default dictionary and their custom dictionary.

For example:

from spellchecker import SpellChecker

spell = SpellChecker()  # Note: this loads in the English dictionary!
spell.load_dictionary("my_custom_dict.json")

Unless your custom dictionary is to augment the base dictionary, remember to setup the SpellChecker object without a language. There are two methods to easily accomplish this:

from spellchecker import SpellChecker

# option 1
spell = SpellChecker(language=None)
spell.load_dictionary("my_custom_dict.json")

# option 2
# this only loads the custom dictionary
spell2 = SpellChecker(local_dictionary="my_custom_dict.json")

The two options above are identical! If a filename, local_dictionary, is provided to the original creation, then the default languages are not loaded.

Adding a New Supported Language

It is possible to provide a new language to pyspellchecker. The goal is that the building of dictionaries to be packaged with the library are able to be re-created as needed. To do this, you will need to update the scripts/build_dictionary.py script to support the language. Once that is updated, including all the necessary information, a new dictionary can be created and updated. It is the preferred method instead of submitting a PR with a new dictionary that cannot be re-created.

Issues Creating New Dictionaries

If you are experiencing issues creating a new, custom dictionary, then please submit an issue. A good thing to provide will be the code used to verify the steps included in this discussion. You may be pointed here for "guidance" if the issue is not clear.

If you would like to share your experiences and tips, please add to the discussion!

Thanks!!
@barrust

[sviperm]

Hello! I have couple of questions:

1. Am I right that when using spell.load_dictionary("my_custom_dict.json"), file shoud be in format like this?

{
    "word1": 123,
    "word2": 321,
}

2. In which format should be file if using spell.word_frequency.load_text_file('my_text_file.txt')
3. Correct me if I'm wrong, that if I load custom dictionary this way, default English dictionary will be extended by custom dictionary. Is that right?

spell = SpellChecker(language='ru)
spell.load_dictionary("my_custom_dict.json")

4. I want to improve Russian dictionary, but as you mentioned, that all dictionary have to be reproducable, for this case I need few libs and a lot of preprocessing, where I should create preprocess.py file? scripts/russian/preprocess.py, is this ok?

[barrust]

@sviperm thank you for reaching out! Let me answer your questions in the order you asked:

1. Yes, that is the correct format. Generally, that data is the output of the spell.export() function but if you want to generate it outside, that works.
2. Any text document would work. A bunch of words, sentences, anything. It will attempt to parse it into individual words and load those in. The more times a word is present the more more likely it will be selected when two words could be the correction!
3. As you wrote your code, it will not load the English dictionary, only the Russian dictionary and then it will add your custom dictionary onto the Russian.
4. That is a great question and one that I haven't thought about. What type of pre-processing are you thinking about? We do some processing to parse the open subtitle files but it is pretty minimal so I would be interested in what you are thinking you need to do. So I would say, sure, if you are pulling together a more robust dictionary then something like scripts/preprocessing/russian.py should be ok.

And based on your PR comment, I am happy to accept cleanup on the script that builds the dictionaries. Any help is greatly appreciated!

[amruthasreddy]

hello
spell.word_frequency.load_text_file('my_text_file.txt')
can you provide the sample text file for this so that i can have idea to load words in proper format

[barrust]

The text file is free text and isn't in any particular format. The library will split it into words and load the words into the dictionary.

There is an example of this in the test suite. You can see the file here.

I hope this helps!

[amruthasreddy]

cool, thanks

[amruthasreddy]

how can i create dictionary not with just sequence of words but in the format shown below
pipeline: pipelin, pipline, piplne, piplin
bracket:brckt, brcket, brackt

when i give any of the mispelled words it has to display the correct word.
can i create dictionary like that. if so how can i do it?

[barrust]

Unfortunately, that is not how this spellchecking library works. That can be accomplished using a simple python dictionary as it would have to be all inclusive on how to solve specific misspellings.

If you you wanted, you could use something like this as a preprocessing to this library, etc.

# NOTE: this implementation is not fast but shows the concept
word_correction_list = {
    'pipeline': {'pipelin', 'pipline', 'piplne', 'piplin'}
}

word = 'pipelin' 

for term, common in word_correction_list.items():
    if word in common:
        word = term # reset it here

# otherwise use spellchecker

[vinitamahajan]

This is very helpful. I need to set Custom Dictionary as priority over the default language dictionary.
Is there a way I can do that ?

[barrust]

It should be straight forward, the key is to pass None as the language parameter.

from spellchecker import SpellChecker

spell = SpellChecker(language=None, local_dictionary="./my_dictionary.json")

[WhiteLicorice]

Hello! I got a question. We plan to use this package to build a dictionary for the Filipino language. I want to test the accuracy of the dictionary. How do I go about getting the numbers?

[barrust]

There is no method to directly test the accuracy as it is based on frequency, therefore the more instances of the word in the source material will make it more likely that the spelling would be selected. This is because, if two words are equal distant from the misspelled word, the one that is more likely, i.e., more frequent, will be the one returned.

I hope this helps!

[WhiteLicorice]

Thank you! It's been a great help.