Hey can anyone help me how to export the fine-tuned model into GGUF using gradio UI or cli #3539
Replies: 1 comment
-
From what i know, only GPTQ quantization is supported.LLaMA-Factory/src/llmtuner/model/utils/quantization.py Lines 27 to 36 in bd095ee Using cli LLaMA-Factory/examples/merge_lora/quantize.sh Lines 4 to 11 in 845d5ac Or from UI: GGUF quantization is not supported on I already quantization GGUF like
Alternatively, If you still encounter problem, you can try the demo space: gguf-my-repo For more detail guide GGUF (untested), quantization from text-generation-webui/pull/5935``` tutorial on how to convert a model to GGUF and quantize it for Windows and Linux.If you can't find a quantized version of the model you need on HuggingFace, you can quantize the model yourself using this guide. GGUFRequirements:
WindowsPreparation
Convert
After that, the file Quantize
LinuxPreparation
Convert
After that, the file Quantize
|
Beta Was this translation helpful? Give feedback.
-
Hey can anyone help me how to export the fine-tuned model into GGUF using gradio UI or cli
Beta Was this translation helpful? Give feedback.
All reactions