Enhancement: Load balancing on Gemini API

[msg7086]

What features would you like to see added?

As shown here

LibreChat/api/app/clients/GoogleClient.js

Line 24 in 94eeec3

const loc = 'us-central1';

We only use us-central1 endpoint, which puts query stress all over us-central1 servers, and also put users under the query limit of 1-2 queries per minute per region.

It would be great if you can load balancing this over all regions endpoints, to better spread the stress and also to get around with per region query limits.

More details

Due to quota Generate content requests per minute per project per base model per minute per region per base_model, the amount of requests is limited by per minute per region per base model, and the limit is usually 1. This will be used up very quickly if you are having a conversation with short sentences with Gemini.

Many other regions provide the same capabilities.

['us-west1', 'us-west4', 'us-central1', 'us-south1', 'us-east4', 'northamerica-northeast1', 'europe-central2', 'europe-west1', 'europe-west2', 'europe-west3', 'europe-west4', 'europe-west6', 'asia-east1', 'asia-east2', 'asia-south1', 'asia-northeast1', 'asia-northeast3', 'australia-southeast1'] (may not be a complete list)

We can utilize all of them, and if possible, give the users the ability to override which regions to use from .env file.

We can pick regions randomly, or we can do LRU. The goal is to put query stress evenly on all Google regions, and have a much lower chance to hit quota limit and get an error.

Which components are impacted by your request?

Endpoints

Pictures

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

I'd be interested to see a PR for this, but may not get to this myself

[danny-avila]

Not yet at least, it should be fairly simple to introduce a battle-tested load balancing algo to quickly swap out supported regions. What kind of throughput is hitting the rate limits for you?

[msg7086]

A few back and forth sentences will trigger the rate limits. Google cloud has a 1 req per minute rate limit, so if you reply twice in a minute you hit the rate limit. It's not by tokens but just number of requests, per region. Sounds very weird to me but it is what it is.

[msg7086]

I just sent #2795. Since I'm not a JS developer it'll take some time for me to set up a test environment. Alternatively if you find it simple enough and can drive from there to close, it's also fine to me.