Parsing text for individual words

[sparticus1701]

This is more of a question, but I'd like to use this in a project I'm working on. From what I can tell WordList.Check is designed to check single words.

Are there any recommendations on what tool to use that I can break up sentences into words, etc., that should be checked? A naive way would be to just use string.split(), but I'd like to see if there's a tool that can automatically handle numbers, currency, sentence puncuation. I've been looking at some NLP tools but wondering if you've used anything in particular.

[aarondandy]

I have not. The reason I made this port was to make a spell checker for Roslyn but got bored before that could have ever happened. Splitting identifiers into words is much easier than NLP though 😁. I haven't yet done anything with human grammars, so I don't think I can point you in the right direction for that.

[funex]

@sparticus1701 Have a look at this closed issue, it answers your question: #75

"The text boundary positions are found according to the rules described in Unicode Standard Annex 29, Text Boundaries, and Unicode Standard Annex 14, Line Breaking Properties. These are available at http://www.unicode.org/reports/tr14/ and http://www.unicode.org/reports/tr29/."

[funex]

@aarondandy I would close this one.