-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adding parameter support for Local LLMs and GROQ #808
base: main
Are you sure you want to change the base?
Conversation
Thanks for this @gpapp ! We have an experimental groq branch, which uses their official SDK instead of making manual requests (which is what we'd like to do in the future with the default openai compatible client function). Do you think you could adapt your changes to work on that? This would also make them less risk (in terms of how different people have set up environments wihch may inadvertenly cause problems with extra env flags), as it would only apply to Groq so far. Btw the prompt mismatch is a good catch, I was convinced that was a shortcoming of Mixtral. |
Added more aggressive file content parsing to CodeMonkey.py Added more aggressive response parsing to function_calling.py
Moved return type enforcement to function_calling This should work with OpenAI as well.
merging groq from @senko with own implementation
Updated to use the SDK (which in fact uses the openAI compatible API for now) readded all the modifiers that can be passed as parameters to groq. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Worked for me macOS python 3.11
Some locally served LLMs need more fine-tune in their parameters, than the OpenAI API.
GROQ also opened their API-s, which has a limited parameters, that can be passed over in the API
The prompt example containts a timeout quoted, while it expects numeric value. This is a waste of many tokens to resolve.