You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I used llm = vllm.LLM( model_name, tensor_parallel_size=4, gpu_memory_utilization=0.85, trust_remote_code=True, dtype="half", enforce_eager=True, enable_lora=True ) and faced the same problem
使用 vllm 启动 openai server 报错。使用官方的 demo 脚本是正常。
启动命令:
python -m vllm.entrypoints.openai.api_server --model /data/huggingface/models--deepseek-ai--DeepSeek-V2-Chat/snapshots/cfa90959d985cd3288fd835519099d9c46fa4842 --tensor-parallel-size 8 --served-model-name deepseek-v2-chat --dtype auto --api-key none --trust-remote-code
error log
The text was updated successfully, but these errors were encountered: