Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add checkpointing #207

Open
aaronmussig opened this issue Dec 12, 2019 · 0 comments
Open

Add checkpointing #207

aaronmussig opened this issue Dec 12, 2019 · 0 comments
Labels
enhancement Proposed feature or change to GTDB-Tk.

Comments

@aaronmussig
Copy link
Member

If an output file exists and the hash is the same then it should be skipped.

Notably, a few steps will need additional files created i.e. FastANI. In cases like this where multiple threads are processing data then the writer thread should write these results to disk (e.g. every 500 results).

Perhaps each major/minor step could interface through JSON/YAML files which would serve nicely as a method for checkpointing.

@aaronmussig aaronmussig added the enhancement Proposed feature or change to GTDB-Tk. label Dec 18, 2019
aaronmussig added a commit that referenced this issue Apr 30, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Proposed feature or change to GTDB-Tk.
Projects
None yet
Development

No branches or pull requests

1 participant