Memory increase while using the client

[Ncoder2]

Description of the problem, including code/CLI snippet

When using the python gitlab client and calling any function say project details for example, there is a constant increase in memory and does not reduce with time. We tested this by running the get projects via http_get method in a loop, the memory kept increasing

Expected Behavior

function calls can increase memory but should be cleared once the call is complete

Actual Behavior

memory keeps increasing every time

Specifications

• python-gitlab version: 3.10.0 and the latest version
• API version you are using (v3/v4): 4
• Gitlab server version (or gitlab.com):

Test Code to reproduce the behaviour, please replace token and project_id

import logging
from memory_profiler import profile
from gitlab_client import GitlabClient
from gitlab import Gitlab


@profile
def func1():
    gl = Gitlab('https://gitlab.com',private_token='{TOKEN}')
    project = gl.http_get("/projects/{project_id}")

if __name__ == '__main__':
    for i in range(10):
        print(i)
        func1()

[nejch]

@Ncoder2 I'm a bit surprised as we use python-gitlab for multiple long-running services that iterate over hundreds of thousands of projects and users on a weekly basis, and haven't seen issues. Do you have some results you could share?

Is there a reason to instantiate the client in every iteration instead of just once?

[Ncoder2]

@nejch , Thanks for the quick response, i just changed the above example code and tried to keep the Gitlab object creation out of the loop and the consumption has reduced. However, our use case is again to get project details of all projects using the gitlab client and take necessary actions/perform code scan etc.

We are using the gitlab client in a fastapi server, when we run a scan on say 1000s of projects, somehow the projects.get increases the memory the fastapi server restarts. In our code, we dont instantiate the gitlab object everytime though.

[Ncoder2]

HI @nejch , what would you suggest for this usecase:
The gitlab client is used as part of a fastapi service, where we retrieve project information based on project id for every request and these requests can be concurrent.

Solution 1:

• Create a gitlab object by providing credentials as a class level variable, but create different project details object for every request, please note, gitlab object remains the same, but gitlab project details object is created for every request

Solution 2:

• For every request, create a new gitlab object and project details object

Asking the two solutions to understand how would gitlab object behave when concurrent requests are made, gitlab project details object is created from the same gitlab object