concurrent requests permit limit is broken

[oOraph]

System Info

meta-llama/Meta-Llama-3-70B-Instruct on h100 with latest tgi

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

gpu are idle, and text-generation-server processes are looping over schedule_yield() calls, 100% cpu usage no gpu usage

and tgi simple healthchecks don't work

Expected behavior

available permit leak
no "model is overloaded" when gpu are idle and no batch is processed

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.