Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Signal handling - SIGTERM will stop calculation and force file saving #2110

Open
wants to merge 19 commits into
base: develop
Choose a base branch
from

Conversation

bigfooted
Copy link
Contributor

Proposed Changes

Special signal handling for SIGTERM. When SIGTERM is sent to the solver, we catch it and simply exit the for-loop for the number of iterations. Files will then automatically be saved.

Related Work

Related to Github discussion #2094

PR Checklist

Put an X by all that apply. You can fill this out after submitting the PR. If you have any questions, don't hesitate to ask! We want to help. These are a guide for you to know what the reviewers will be looking for in your contribution.

  • I am submitting my contribution to the develop branch.
  • My contribution generates no new compiler warnings (try with --warnlevel=3 when using meson).
  • My contribution is commented and consistent with SU2 style (https://su2code.github.io/docs_v7/Style-Guide/).
  • I have added a test case that demonstrates my contribution, if necessary.
  • I have updated appropriate documentation (Tutorials, Docs Page, config_template.cpp), if necessary.

@bigfooted
Copy link
Contributor Author

bigfooted commented Aug 15, 2023

Here is a simple proof-of-concept for the fluid iteration where we catch SIGTERM. If you run it and hit ctrl-c, we catch the signal and we set stopcalc to true. This will exit the for-loop and also forces a saving of the files.
I think implementing the signal handler like this is the best, because we let the solver handle all the cleanup.

@bigfooted
Copy link
Contributor Author

We probably also have to do something like this at other places where we have external iterations.

@bigfooted bigfooted changed the title [WIP] Signal handling - SIGTERM will save files [WIP] Signal handling - SIGTERM will stop calculation and force file saving Aug 15, 2023
Copy link
Member

@pcarruscag pcarruscag left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be at the driver level, not inside iteration classes otherwise it won't work for multizone.
How are signals handled by MPI? Can this result in a race condition?

@bigfooted
Copy link
Contributor Author

OK, so I am moving the signal handling to COutput::ConvergenceMonitoring as discussed in the dev-meeting.

@bigfooted
Copy link
Contributor Author

@pcarruscag This works for me. Do you think this is OK or we need to do it different/more, maybe a config keyword to activate/deactivate, or other options?

Copy link
Member

@pcarruscag pcarruscag left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have not had time to test it for single/multi zone, steady/unsteady, and primal/adjoint.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants