-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Increasing RAM usage with enable_model_cpu_offload #7970
Comments
I should also mention, that after the first few calls the RAM starts raising very slowly, but still fails after 30-40 calls |
On a shared machine and A100, I get this: (diffusers) sayak@hf-dgx-01:~/diffusers$ CUDA_VISIBLE_DEVICES=2 python test_mco.py
Before loading pipeline - Memory usage: 254.10 GB
Loading pipeline components...: 100%|███████████████████████████████████████████████████████████████████████| 7/7 [00:01<00:00, 6.05it/s]
After loading pipeline - Memory usage: 253.66 GB
After moving model to CPU - Memory usage: 253.66 GB
After generating 1 - Memory usage: 261.04 GB
After generating 2 - Memory usage: 263.83 GB
After generating 3 - Memory usage: 264.49 GB
After generating 4 - Memory usage: 265.72 GB
After generating 5 - Memory usage: 98.67 GB
After generating 6 - Memory usage: 57.82 GB
After generating 7 - Memory usage: 57.52 GB
After generating 8 - Memory usage: 60.57 GB
After generating 9 - Memory usage: 59.72 GB
After generating 10 - Memory usage: 60.33 GB
After generating 11 - Memory usage: 60.98 GB
After generating 12 - Memory usage: 58.98 GB
After generating 13 - Memory usage: 60.24 GB
After generating 14 - Memory usage: 60.52 GB
After generating 15 - Memory usage: 61.35 GB
After generating 16 - Memory usage: 60.92 GB
After generating 17 - Memory usage: 60.86 GB
After generating 18 - Memory usage: 60.40 GB
After generating 19 - Memory usage: 60.92 GB
After generating 20 - Memory usage: 61.55 GB
After generating 21 - Memory usage: 62.41 GB
After generating 22 - Memory usage: 64.44 GB
After generating 23 - Memory usage: 63.87 GB
After generating 24 - Memory usage: 64.40 GB
After generating 25 - Memory usage: 64.65 GB
After generating 26 - Memory usage: 61.38 GB
After generating 27 - Memory usage: 61.91 GB
After generating 28 - Memory usage: 61.99 GB
After generating 29 - Memory usage: 62.49 GB
After generating 30 - Memory usage: 63.00 GB
After generating 31 - Memory usage: 61.95 GB
After generating 32 - Memory usage: 61.87 GB
After generating 33 - Memory usage: 62.37 GB
After generating 34 - Memory usage: 60.64 GB
After generating 35 - Memory usage: 65.77 GB
After generating 36 - Memory usage: 65.86 GB
After generating 37 - Memory usage: 65.34 GB
After generating 38 - Memory usage: 63.55 GB
After generating 39 - Memory usage: 62.52 GB
After generating 40 - Memory usage: 62.46 GB
After generating 41 - Memory usage: 61.85 GB
After generating 42 - Memory usage: 62.65 GB
After generating 43 - Memory usage: 64.70 GB
After generating 44 - Memory usage: 63.54 GB
After generating 45 - Memory usage: 61.94 GB
After generating 46 - Memory usage: 61.52 GB
After generating 47 - Memory usage: 62.03 GB
After generating 48 - Memory usage: 63.48 GB
After generating 49 - Memory usage: 66.18 GB
After generating 50 - Memory usage: 61.92 GB
After deleting pipeline - Memory usage: 61.55 GB
After inference - Memory usage: 61.56 GB I extended the number of runs to 50 to get a more reasonable estimate and also commented the
This is interesting. The increase is small but nonetheless it's there and it has no reason to be there. |
So this is intended behavior? I can see, that you've also experienced an increase, and although it looks small on A100, this is very painful when using a single T4 and 16 GB RAM |
I tried in Colab T4 free. ~1 GB fluctuation during 50 inference 🤔:
Also, I couldn't clear all the RAM either at the end. |
Is there any workaround for that by any chance? |
As a workaround you can try this before
|
Describe the bug
When using enable_model_cpu_offload on StableDiffusionXLPipeline, each consecutive call takes more and more RAM. Also, after deleting pipe not all memory is freed
Reproduction
Logs
System Info
Who can help?
@yiyixuxu @sayakpaul
The text was updated successfully, but these errors were encountered: