-
Notifications
You must be signed in to change notification settings - Fork 736
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Loading tokenizer.model
with Rust API
#1518
Comments
You cannot load a tokenizer.model, you need to write a converter. https://github.com/huggingface/transformers/blob/main/src/transformers/convert_slow_tokenizer.py#L544 is the simplest way to understand the process! |
Ok, I understand. Do you know of a way or a library to do this in Rust without reaching for the Python transformers converter? |
A library no, but we should be able to come up with a small rust code to do this 😉 |
@ArthurZucker are there any specifications or example loaders which I can look at to implement this? |
I also have the same question, for llava reasons😉 |
Hello all,
Thank you for your excellent work here. I am trying to load a
tokenizer.model
file in my Rust application. However, it seems that theTokenizer::from_file
function only support loading from atokenizer.json
file. This causes problems as using a small script to save thetokenizer.json
is error-prone and hard to discover for users. Is there a way to load atokenizer.model
file?The text was updated successfully, but these errors were encountered: