Consider having worker containers for background processes

[pwalsh]

If I understand https://github.com/ckan/ckan-docker/blob/master/docker-compose.yml correct then background processes would be running out of the CKAN container, which is the application server.

Perhaps worth considering having worker containers dedicated to this?

I do see that we are doing the trick of running supervisor in our containers - probably for background processes.

I also see we are running multiple uswgi processes - so we possibly are not using (or expecting users to use) docker for horizontal scaling of the app server.

These could be nice affordances for dev environments using docker compose, but they possibly present performance and scaling bottlenecks for deployments using docker compose. On this, I know there are older perceptions that Compose is not a tool for production, but, that is not the case these days, and also the compose file format is usable for k8s deployments and Docker Swarm deployments.

Many (most) CKAN deployments do not have large scale traffic or data processing requirements, and a deployment story using Docker Compose could be very beneficial.