Jobs never finish on remote server

[RasmusAntons]

Today and yesterday, any example started with the run_container.sh script is still "being executed" after 10 minutes. This looks similar to the issue reported by Ebrahim on GitLab, but it affects all four examples. Priyabrat was still able to run them about two weeks ago.

rasmus@xeo54:~/Documents/simdome/SimCMCLkinetics$ ./run_container.sh examples eat gpf
2022-10-11 07:48:09,099 ----Running EAT GPF Use Case----
New 'EATEngine' instantiated!
Model flag changed to #3.
INFO 2022-10-11 07:48:09,187 [osp.core.session.wrapper_session]: 16 CUDS objects have been added to Kinetics Wrapper Session
2022-10-11 07:48:09,187 16 CUDS objects have been added to Kinetics Wrapper Session
INFO 2022-10-11 07:48:09,187 [osp.core.session.wrapper_session]: 1 CUDS object has been updated in Kinetics Wrapper Session
2022-10-11 07:48:09,187 1 CUDS object has been updated in Kinetics Wrapper Session
INFO 2022-10-11 07:48:09,187 [osp.core.session.wrapper_session]: 0 CUDS objects have been deleted from Kinetics Wrapper Session
2022-10-11 07:48:09,187 0 CUDS objects have been deleted from Kinetics Wrapper Session

===== Start: KineticsSession =====
CUDS representation of inputs written to: ./input_results.txt
Detected simulation template as EAT_GPF
JSON data successfully generated from CUDS objects.
Job submitted successfully, resulting job ID is simdome-20221011084809
Job still running (attempt 1 of 60)...
Job still running (attempt 2 of 60)...
Job still running (attempt 3 of 60)...
Job still running (attempt 4 of 60)...
Job still running (attempt 5 of 60)...
Job still running (attempt 6 of 60)...
Job still running (attempt 7 of 60)...
Job still running (attempt 8 of 60)...
Job still running (attempt 9 of 60)...
Job still running (attempt 10 of 60)...
Job still running (attempt 11 of 60)...
Job still running (attempt 12 of 60)...
Job still running (attempt 13 of 60)...
Job still running (attempt 14 of 60)...
Job still running (attempt 15 of 60)...
Job still running (attempt 16 of 60)...
Job still running (attempt 17 of 60)...
Job still running (attempt 18 of 60)...
Job still running (attempt 19 of 60)...
Job still running (attempt 20 of 60)...
Job still running (attempt 21 of 60)...
Job still running (attempt 22 of 60)...
Job still running (attempt 23 of 60)...
Job still running (attempt 24 of 60)...
Job still running (attempt 25 of 60)...
Job still running (attempt 26 of 60)...
Job still running (attempt 27 of 60)...
Job still running (attempt 28 of 60)...
Job still running (attempt 29 of 60)...
Job still running (attempt 30 of 60)...
Job still running (attempt 31 of 60)...
Job still running (attempt 32 of 60)...
Job still running (attempt 33 of 60)...
Job still running (attempt 34 of 60)...
Job still running (attempt 35 of 60)...
Job still running (attempt 36 of 60)...
Job still running (attempt 37 of 60)...
Job still running (attempt 38 of 60)...
Job still running (attempt 39 of 60)...
Job still running (attempt 40 of 60)...
Job still running (attempt 41 of 60)...
Job still running (attempt 42 of 60)...
Job still running (attempt 43 of 60)...
Job still running (attempt 44 of 60)...
Job still running (attempt 45 of 60)...
Job still running (attempt 46 of 60)...
Job still running (attempt 47 of 60)...
Job still running (attempt 48 of 60)...
Job still running (attempt 49 of 60)...
Job still running (attempt 50 of 60)...
Job still running (attempt 51 of 60)...
Job still running (attempt 52 of 60)...
Job still running (attempt 53 of 60)...
Job still running (attempt 54 of 60)...
Job still running (attempt 55 of 60)...
Job still running (attempt 56 of 60)...
Job still running (attempt 57 of 60)...
Job still running (attempt 58 of 60)...
Job still running (attempt 59 of 60)...
Maximum number of attempts reached, considering job a failure.
Job was not completed on the remote HPC!
Could not get job outputs (failed job?), returning None
No valid simulation results detected, session has failed.
CUDS representation of results written to: ./output_results.txt
===== End: KineticsSession =====