Control amount of memory usage

[loleg]

The official hardware specs suggest a 4 GB RAM requirement. However, on my developer machine with 16 GB RAM, I'm experiencing a lot of swapping within minutes of starting up CKAN - without a large database. I suspect my OS as well (current Manjaro, based on Arch Linux) could also be at fault (I have looked into OOM to avoid the system freezing). Nevertheless, I am wondering if we could check the defaults and perhaps suggest a "leaner" installation for VM and dev sandboxes.

This could just be a section of a README with some parameters relevant to memory consumption.

Relevant issues on upstream:

• Memory usage of ckan_default processes increases continuously  ckan#5349
• Digging into python memory issues in CKAN with heapy

[Alexandero89]

Funny that you just created this issue, because i noticed the same problem minutes ago.
Starting up the compose setup on my 16 gb ram machine and with 16 gb of swap i run into OutOfMemory Errors.
This should not happen with a service that claims to use 4gb of ram.

[gacou54]

Same for me for my 16Gb developer machine, a simple docker compose up with the default .env uses all my memory.

Output of docker container stats:

CONTAINER ID   NAME         CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
e145dcaf6cc7   nginx        0.00%     16.31MiB / 15.54GiB   0.10%     13.4kB / 0B       6.95MB / 4.1kB    15
e5c87f9eda01   ckan         0.02%     8.2GiB / 15.54GiB     52.78%    219kB / 193kB     54.1MB / 67.3MB   6
290b2f8e0893   solr         0.22%     105.4MiB / 15.54GiB   0.66%     22.8kB / 39kB     156MB / 658MB     60
06b2d10f2bbe   db           0.02%     26.98MiB / 15.54GiB   0.17%     181kB / 159kB     39.6MB / 21.4MB   15
2b24c443ae96   redis        0.10%     11.25MiB / 15.54GiB   0.07%     10.3kB / 2.14kB   21.4MB / 1.73MB   5
fb9d38bb00cd   datapusher   0.00%     3.045GiB / 15.54GiB   19.60%    21.9kB / 0B       30.4MB / 5.39GB   3

Is there a way to avoid this? We want to evaluate ckan for a data catalog project. Thanks :)

[kowh-ai]

The hardware specs wiki page has been updated to be more realistic with current implementations of CKAN.

Also, Are you @gacou54, @loleg and @Alexandero89 able to provide more detail on how many datasets/resources you have in your environments. Those numbers do look high. I have just installed a fresh 2.10.3 CKAN Docker environment with 1 dataset uploaded. I see the follow stats:

[gacou54]

That is kind of strange. I don't have any dataset, I simply cloned the project, copy-pasted .env.example -> .env and did a docker compose up.

While "upping", the ckan and datapusher seems to consume a lot of memory. The behaviour during "up" is that datapusher and ckan seem to spike at 8Gb each in turn. In this example is a moment when datapusher is spiking during the up.

CONTAINER ID   NAME         CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O         PIDS
38962ece719b   solr         0.16%     696.7MiB / 15.54GiB   4.38%     6.62kB / 0B     92.1MB / 12.3kB   60
da1911bfde41   db           0.03%     18.46MiB / 15.54GiB   0.12%     7.36kB / 781B   19.7MB / 475kB    7
55ed86df5e0d   datapusher   0.00%     8.062GiB / 15.54GiB   51.89%    19.4kB / 0B     22.7MB / 483kB    3
2c52ea786292   redis        0.09%     7.922MiB / 15.54GiB   0.05%     6.62kB / 0B     13.8MB / 0B       5
5865fccad9b9   ckan         0.01%     103.2MiB / 15.54GiB   0.65%     11.4kB / 739B   32.1MB / 20.5kB   3

After the up (which takes several minutes), only the ckan seems to take 8Gb.

Not sure if pertinent, but for the taken storage, this is what I get during the docker compose up, after the up and after the docker compose down, it seems static and quite low. So I am not sure the problem is related to the data.

[gacou54@fedora ~]$ sudo du -sh /var/lib/docker/volumes/ckan-docker_ckan_storage
3.3M	/var/lib/docker/volumes/ckan-docker_ckan_storage
[gacou54@fedora ~]$ sudo du -sh /var/lib/docker/volumes/ckan-docker_pg_data
75M	/var/lib/docker/volumes/ckan-docker_pg_data
[gacou54@fedora ~]$ sudo du -sh /var/lib/docker/volumes/ckan-docker_solr_data
1.2M	/var/lib/docker/volumes/ckan-docker_solr_data

[kowh-ai]

I don't get those spikes at all. My host machine is a fairly elderly Apple Mac Mini with:
Memory: 32GB RAM
CPU: 3.2 GHz 6-Core Intel Core i7

I presume you have run a docker system prune -a --volumes ? (assuming thats possible for you)...then the docker compose build and then docker compose up

[gacou54]

Also, I entered the ckan container and one of the uswgi process seems to use the memory:

 PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
   28    27 ckan     S     189m   1%   1   1% uwsgi --plugins http,python --socket /tmp/uwsgi.sock --wsgi-file /srv/app/wsgi.py --module wsgi:application
   30    27 ckan     S    8208m  48%  10   0% uwsgi --plugins http,python --socket /tmp/uwsgi.sock --wsgi-file /srv/app/wsgi.py --module wsgi:application
   29    27 ckan     S     189m   1%   4   0% uwsgi --plugins http,python --socket /tmp/uwsgi.sock --wsgi-file /srv/app/wsgi.py --module wsgi:application
   26     1 root     S    28608   0%   2   0% {supervisord} /usr/bin/python3 /usr/bin/supervisord --configuration /etc/supervisord.conf
   27     1 ckan     S    16460   0%  11   0% uwsgi --plugins http,python --socket /tmp/uwsgi.sock --wsgi-file /srv/app/wsgi.py --module wsgi:application
    1     0 root     S     2404   0%   5   0% {start_ckan.sh} /bin/bash /srv/app/start_ckan.sh
   49     0 root     S     1696   0%   3   0% sh
   56    49 root     R     1620   0%   8   0% top

I started only the container ckan/ckan-base:2.10.3 with docker run ckan/ckan-base:2.10.3 it takes 6-8Gb of memory. So the issue seems to be more related to this image?

I presume you have run a docker system prune -a --volumes ? (assuming thats possible for you)...then the docker compose build and then docker compose up

I will try that, I can do the prune.

[kowh-ai]

Yes, I use ckan/ckan-base:2.10.3 as the base image too

[gacou54]

I still have the same issues regarding memory usage after the system prune. Maybe the issue should be raised in https://github.com/ckan/ckan-docker-base, since docker run ckan/ckan-base:2.10.3 takes 7-8Gb as is.

[kowh-ai]

Sure, I'll create a new issue in the ckan-docker-base repo...in the meantime are you able to change the base image to 2.10.2 and 2.10.1 to see if that changes anything

[gacou54]

Thanks! Same issue with 2.9.9 and 2.10.1.

In the meantime it is not so important on our side (we finally plan to use a humble s3 object storage (minio) for our project, we don't need most of the ckan features). I was trying to find/document the problem for the others :)