Mono-Electra model type not recognised

[PrithivirajDamodaran]

System Info

4.39.3 and 4.40.2

Who can help?

@younesbelkada @ArthurZucker

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model = AutoModelForSequenceClassification.from_pretrained("webis/monoelectra-base")
tokenizer = AutoTokenizer.from_pretrained("webis/monoelectra-base")

Throws exception

The checkpoint you are trying to load has model type `mono-electra` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

As per the model config, 4.39.3 is the right transformers version. Tried also with 4.40.2

Expected behavior

Model should load fine without exceptions

[amyeroberts]

Hi @PrithivirajDamodaran, thanks for raising an issue!

Inspecting the model's config, one can see the model type mono-electra is specified, as well as the model class FlashMonoElectraModel. Neither of these are defined either within the transformers repo, nor in a modeling file e.g. modeling_monoelectra.py on the repo. That is, as there is no available definition for the model class; and there's no mapping from the model type to the model class within transformers or registered on the remote code, then there's no way to know how or what to load.

If you're interesting in using this model, I'd suggest opening a discussion topic on the repo asking how to use it.

[PrithivirajDamodaran]

Thank you, I examined the files in the repo for any custom code to load the model but there is none. If no custom code and there is no implementation, looks like it's a good idea open a discussion as you suggested to see if the team is actively working on this. Could you please keep this issue open in the mean time?

But IMHO, as for the model name / code goes "mono" is just a prefix for mono-lingual electra and it should work with modeling_electra.py, but I am not sure.

[amyeroberts]

But IMHO, as for the model name / code goes "mono" is just a prefix for mono-lingual electra and it should work with modeling_electra.py

This depends on the checkpoints. If they're compatible with, i.e. have the same architecture as, the transformer's implementation of electra, then changing this value to "electra" in the config would work.