Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Inference] Support for Messages API OpenAI API specs #574

Open
radames opened this issue Mar 24, 2024 · 0 comments
Open

[Inference] Support for Messages API OpenAI API specs #574

radames opened this issue Mar 24, 2024 · 0 comments
Labels
enhancement New feature or request inference @huggingface/inference related

Comments

@radames
Copy link
Member

radames commented Mar 24, 2024

Considering that TGI now supports the Messages API compatible with OpenAI API specs, it would be great to have native support in the Inference package.

curl localhost:3000/v1/chat/completions \
    -X POST \
    -d '{
  "model": "tgi",
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant."
    },
    {
      "role": "user",
      "content": "What is deep learning?"
    }
  ],
  "stream": true,
  "max_tokens": 20
}' \
    -H 'Content-Type: application/json'

https://huggingface.co/docs/text-generation-inference/messages_api

I tried this , but model is not send and raises a backend error

const mistral = new HfInferenceEndpoint(
  "https://api-inference.huggingface.co/models/mistralai/Mixtral-8x7B-Instruct-v0.1/v1/chat/completions",
  HF_TOKEN,
);
const stream = mistral.streamingRequest({
  model: "tgi",
  messages: [{ role: "user", content: "What's the meaning ..." }],
  parameters: {
    messages: messages,
    temperature: 0.8,
    max_new_tokens: 1000,
    repetition_penalty: 1.2,
    seed: Math.floor(Math.random() * Number.MAX_SAFE_INTEGER),
  },
});
@radames radames added enhancement New feature or request inference @huggingface/inference related labels Mar 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request inference @huggingface/inference related
Projects
None yet
Development

No branches or pull requests

1 participant