Multi-GPU Inference

[daz-williams]

I have TGI setup and running. Is there a monitoring tool to see the performance of each GPU during inferences?

[Narsil]

Yes, TGI comes with prometheus included (on /metrics).

You can usually set it up with something like grafana.

This is the sort of stuff you can get

[ansSanthoshM]

@Narsil ,

I like to create a dash board something like shown above for my TGI server. But i have no idea about prometheus/grafana/metrics.
But i am a quick learner, i am willing to spent time and learn to create a dashboard like this to understand the number of users our hardware can handle. I kindly request you to point me a tutorial / reading materials on how to get these info from TGI /metrics and create a dashboard like this.

[daz-williams]

Use ngrok to expose your TGI instance, then add the URL to Grafana (create a free account)

E.G: https://yourserver.ngrok.dev/metrics

Which is the equivalent of this:

http://localhost:8080/metrics