Supervisor config for worker should set environment variable PYTHONUNBUFFERED

[paulmueller]

CKAN version

2.10.4

Describe the bug

While testing the background worker, I was not able to get any output of the worker processes to be written to the log files.

Steps to reproduce

# clear the logs
echo "" > /var/log/ckan/ckan-worker.stderr.log
echo "" > /var/log/ckan/ckan-worker.stdout.log

# restart the ckan worker via
supervisorctl stop ckan-worker:*`
supervisorctl start ckan-worker:*`

# create a test job
ckan jobs test default

# check whether anything is written to the logs
du /var/log/ckan/ckan-worker.stdout.log
# 4	/var/log/ckan/ckan-worker.stdout.log

Expected behavior

When I submit a test job, I would like to immediately see in the logs whether things work.

Problem and proposed solution

The problem is that Python by default buffers its output. While running ckan jobs worker default outputs the data directly into the terminal without any delay, running it with pipes attached buffers the output (can also be reproduced with ckan jobs worker default > test.log).

A very simple solution would be to set the PYTHONUNBUFFERED environment variable in the supervisor configuration file.

environment=PYTHONUNBUFFERED=1

I could create a PR for this if this is an acceptable solution. 🙏

[wardi]

We've set this in the dockerfile here

ckan/.devcontainer/Dockerfile

Line 5 in a638395

ENV PYTHONUNBUFFERED 1

would that be a better place for this setting? @EricSoroos

[EricSoroos]

I think it's probably better to set in the supervisor config -- there's no real reason that we'd need to be changing that at the docker level, and it would just cause confusion for people not using the docker files.

[kowh-ai]

The Dockerfile above is for GitHub Codespaces only. We don't, AFAIK change the default setting in any of our CKAN Dockerfiles or supporting config. If this change is made here I could make the same change in the ckan-docker-base repo so we are consistent