Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Offset position "longer" than text #83

Open
Kailegh opened this issue Nov 13, 2023 · 1 comment
Open

Offset position "longer" than text #83

Kailegh opened this issue Nov 13, 2023 · 1 comment

Comments

@Kailegh
Copy link

Kailegh commented Nov 13, 2023

I have a match that look like this:

Match({'ruleId': 'MORFOLOGIK_RULE_ES', 'message': 'Se ha encontrado un posible error ortográfico.', 'replacements': ['teléfonos', 'teléfono', 'telefotos'], 'offsetInContext': 43, 'context': '...𝒚 𝒛𝒂𝒑𝒊𝒐𝒍𝒂 podemos compartir tus telefonos con el conductor 𝑺𝒊', 'offset': 307, 'errorLength': 9, 'category': 'TYPOS', 'ruleIssueType': 'misspelling', 'sentence': 'Rider > Lost Items > Standard lost item > Driver found riders itemdescripcion del articulo perdido 𝑴𝒆 𝒐𝒍𝒗𝒊𝒅𝒆 𝒖𝒏𝒂 𝒎𝒐𝒄𝒋𝒊𝒍𝒂 𝒏𝒆𝒈𝒓𝒔 ingresa un numero de telefono alternativo incluye el codigo de tu pais informacion sobre el viaje 𝒂𝒏𝒅𝒓𝒆𝒔𝒊𝒕𝒐 𝒚 𝒛𝒂𝒑𝒊𝒐𝒍𝒂 podemos compartir tus telefonos con el conductor 𝑺𝒊'})

Original sentence look like shown in example:
Rider > Lost Items > Standard lost item > Driver found riders itemdescripcion del articulo perdido 𝑴𝒆 𝒐𝒍𝒗𝒊𝒅𝒆 𝒖𝒏𝒂 𝒎𝒐𝒄𝒋𝒊𝒍𝒂 𝒏𝒆𝒈𝒓𝒔 ingresa un numero de telefono alternativo incluye el codigo de tu pais informacion sobre el viaje 𝒂𝒏𝒅𝒓𝒆𝒔𝒊𝒕𝒐 𝒚 𝒛𝒂𝒑𝒊𝒐𝒍𝒂 podemos compartir tus telefonos con el conductor 𝑺𝒊
Problem is that offset is said to be 307, while sentence length in chars 296.
I think that the problem is that the text has some chars that actually internally take more than one position in unicode encoding (are compose but multiple chars).
The problem is that when I try to reference detection to original text I get an error because that position is wrong and does not reference the true position in the text

@pidefrem
Copy link
Contributor

pidefrem commented Apr 11, 2024

@Kailegh

Your problem is indeed reproducible with the following code:

from language_tool_python import LanguageTool

language = "ES"
tool = LanguageTool(language)
text = 'Rider > Lost Items > Standard lost item > Driver found riders itemdescripcion del articulo perdido 𝑴𝒆 𝒐𝒍𝒗𝒊𝒅𝒆 𝒖𝒏𝒂 𝒎𝒐𝒄𝒋𝒊𝒍𝒂 𝒏𝒆𝒈𝒓𝒔 ingresa un numero de telefono alternativo incluye el codigo de tu pais informacion sobre el viaje 𝒂𝒏𝒅𝒓𝒆𝒔𝒊𝒕𝒐 𝒚 𝒛𝒂𝒑𝒊𝒐𝒍𝒂 podemos compartir tus telefonos con el conductor 𝑺𝒊'
print("len(text)", len(text))
matches = tool.check(text)
for match in matches:
  print(match)
corrected_text = language_tool_python.utils.correct(text, matches)
print(corrected_text)

tool.close()

Note that some words of your text seem to have special formatting.
If you clean your text as follows it seems to work normally:

text_cleaned = 'Rider > Lost Items > Standard lost item > Driver found riders itemdescripcion del articulo perdido Me olvide una mocjila negrs ingresa un numero de telefono alternativo incluye el codigo de tu pais informacion sobre el viaje andresito y zapiola podemos compartir tus telefonos con el conductor Si'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants