Generate feature-extraction specs from TEI

[Wauplin]

Goes hand-in-hand with huggingface/huggingface_hub#2270.

Any opinion on generating specs for feature-extraction based on TEI auto-generated specs?

For now I duplicated the TGI importer script and adapted it to TEI. Specs are slightly different:

• no parameters open field
• added truncate / normalize at the root of the payload
• we lose some docstring but that can be improved in TEI repo directly
• a badly named export type FeatureExtractionInputInput = string[] | string; since in TEI input can be either string or array of strings.

[julien-c]

what's the % of feature-extraction models on the Hub that are served with TEI? (and it's transformers only, right?)

[Wauplin]

I'd say 0% for serverless Inference API. But for dedicated Inference Endpoints / local TEI we don't have a good client for now.