so-Influxdb missing container on update

[BJOnionMan]

Steps to reproduce:
List the minimal actions needed to reproduce the behaviour.

architecture: Linux 5.15.0-204.147.6.2.el9uek.x86_64 x86_64

During the update from 2.4.50 to 2.4.60 we are getting hung on the following command:

Executing command '/usr/sbin/so-influxdb-manage' in directory '/root'

Currently troubleshooting, making a note for a future solve to this problem.

1st Attempt at solve:

accessed ssl certifcate and key within the manger node /etc/pki/managerssl.xxx
Copied the key and cert in to SOS config under > nginx > ssl > SSL/TSL xxx
Retrying the update...

ts=2024-04-08T00:05:12.208738Z lvl=info msg="http: TLS handshake error from 127.0.0.1:34684: EOF" log_id=0o5f0DvW000 service=http

After trying to run soup again, we have identified another issue where the salt minions are no longer talking to the salt-master (Manager Node).

I am deleting the keys and reaccepting them. To do this:

On the manager execute: salt-key
Run salt-key -d this deletes all the current keys
Restart the minions (Search, Receiver, Forward)
Go back to the manager and run salt-key again
Accept the keys by executing: salt-key -A
Run salt* test.ping to confirm everything is working again

The salt is now working on all the nodes but we still have the following error when trying to run soup

• Specified SLS 'data.*' in environment 'base' is not available on the salt master
• Specified SLS 'global' in environment 'base' is not available on the salt master

Found a GitHub with a similar problem

Trying to run salt * saltutil.refresh_pillar
Have to replace * with the name of each minion
And running salt * saltutil.sync_grains
Again have to replace *

This did not resolve the issue. Same error is occurring.

As I understand now, the file global.sls does not exist where it should. I started a GitHub discussion to try and resolve the missing file(s) and am waiting for a response.

---

Update: I was able to get through the soup command. I now know the issue was not with the script but rather an issue with the influxdb server. When we try to run salt * state.highstate we get hungup on the same command. If you wait long enough it will error out and keep running. The error codes I could find in the influxdb.log are below: *Also included is a screenshot of the error log.

ID: influxdb-setup
Function: cmd.run
Name: /usr/sbin/so-influxdb-manage setup &>> /opt/so/log/influxdb/setup.log
Result: False
Comment: Command "/usr/sbin/so-influxdb-manage setup &>> /opt/so/log/influxdb/setup.log" run
Started: 19:38:12.399766
Duration: 302493.461 ms
Changes:

pid: 83801
retcode: 1
stderr:
stdout:
ID: metrics_link_file
Function: cmd.run
Name: so-influxdb-manage dashboardpath "Security Onion Performance" > /opt/so/saltstack/local/salt/influxdb/metrics_link.txt
Result: False
Comment: Command "so-influxdb-manage dashboardpath "Security Onion Performance" > /opt/so/saltstack/local/salt/influxdb/metrics_link.txt" run
Started: 19:43:15.145821
Duration: 301277.019 ms
Changes:

pid: 103999
retcode: 1
stderr:

Thu Apr 11 19:43:15 UTC 2024 | so-influxdb-manage | Server does not appear to be running or fully initialized - will try again in 10 seconds (1 / 30)

---

We are rolling back to the previous version and accepting the outdated version until there is a fix for the issue above. If we keep the update as is, influxdb will not work.

Expected behaviour:
We expected to run soup and just update our nodes to the latest version.

Actual behaviour:
so-status (see all containers) run soup, so-status, then so-influxdb is missing as a container.
We ran soup, so-rule-update, salt *state.highstate etc

Environment info:

Linux 5.15.0-204.147.6.2.el9uek.x86_64 x86_64

Config:
Copy any non-default config values here or attach the full config as a gist or file.

Logs:
Include snippet of errors in logs or stack traces here.
Sometimes you can get useful information by running the program with the RUST_BACKTRACE=full environment variable.
Finally, the IOx server has a -vv for verbose logging.