Skip to content
/ ru_punkt Public

Russian language support for NLTK's PunktSentenceTokenizer

License

Notifications You must be signed in to change notification settings

Mottl/ru_punkt

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

15 Commits
 
 
 
 
 
 

Repository files navigation

ru_punkt

Russian language support for NLTK's PunktSentenceTokenizer

Python 2.7 Python 3x

ru_punkt is a part of nltk_data since 2019-07-04

Instalation

  1. Install NLTK python package:
pip install nltk
  1. Download punkt data:
import nltk
nltk.download('punkt')

Usage

import nltk

text = "Ай да А.С. Пушкин! Ай да сукин сын!"
print("Before:", nltk.sent_tokenize(text))
print("After:", nltk.sent_tokenize(text, language="russian"))

Output:

Before: ['Ай да А.С.', 'Пушкин!', 'Ай да сукин сын!']
After: ['Ай да А.С. Пушкин!', 'Ай да сукин сын!']

Training data

Data for sentence tokenization was taken from 3 sources:
– Articles from Russian Wikipedia (about 1 million sentences);
– Common Russian abbreviations from Russian orthographic dictionary, edited by V. V. Lopatin;
– Generated names initials.

Implementation notes

After some research it was found that the single params.abbrev_types performs better than together with params.collocations and params.ortho_content, so the latter were removed from the trained tokenizer.

About

Russian language support for NLTK's PunktSentenceTokenizer

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published