Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Peformance while adding dictionary #54

Open
Eltharis957 opened this issue Dec 8, 2018 · 1 comment
Open

Peformance while adding dictionary #54

Eltharis957 opened this issue Dec 8, 2018 · 1 comment

Comments

@Eltharis957
Copy link

Eltharis957 commented Dec 8, 2018

Thank you for this great code. I'd like to ask you about recommendation/suggestion on how I could make loading a big dictionary faster and not slowing down the application.
I have an idea, please answer as a creator if this is going to work in your opinion:

  1. I thought about adding words in smaller chunks via multiple for loops using CreateDictionaryEntry() function on arrays.
  2. To make this not stall the application, I could run those loops on a background thread. Small enough chunks could not affect performance, while gradually building a dictionary before a user gets to type in a word.

My previous implementation was my own, but it was using SQL database. Queries were too slow for this kind of usage or too many with a bigger dictionary.

An issue - for some reason, CreateDictionaryEntry() is the only way for me that works after using LoadDictionary() first. CreateDictionary() doesn't add more terms to an existing database. Word count stays the same. Maybe it's because it's plain text without a second column for word frequency?

And two short questions:

  1. How to add multi-word phrases to dictionary among normal single word ones? Currently, CreateDictionaryEntry() seems to add some phrases while throwing others out. If I start with a number and a second word has only letters, it shows up in Lookup(). But if the first one starts with an uppercase letter and second is just 3 uppercase letters, Lookup() doesn't work. The same when there's some punctuation.
  2. How to add phrases (single and multi-word) with punctuation and make them show up as suggestions among single word phrase?
@wolfgarbe
Copy link
Owner

wolfgarbe commented Dec 8, 2018

I thought about adding words in smaller chunks via multiple for loops using CreateDictionaryEntry() function on arrays.
To make this not stall the application, I could run those loops on a background thread. Small enough chunks could not affect performance, while gradually building a dictionary before a user gets to type in a word.

Yes, you can certainly do this. But there is no need to use chunks and multiple for loops. You can simply run LoadDictionary() or CreateDictionary() as a whole in a background thread. You should then synchronize CreateDictionaryEntry() and Lookup() via lock().

How to add multi-word phrases to dictionary?

LoadDictionary() and CreateDictionary() do not currently support multi-word phrases. The only way to add multi-word phrases is to do it programmatically via CreateDictionaryEntry()

How to add phrases (single and multi-word) with punctuation

Currently you can add phrases with punctuation only programmatically via CreateDictionaryEntry()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants