Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrating the library with an external thread pool #4043

Open
Zeblote opened this issue May 13, 2024 · 2 comments
Open

Integrating the library with an external thread pool #4043

Zeblote opened this issue May 13, 2024 · 2 comments

Comments

@Zeblote
Copy link

Zeblote commented May 13, 2024

Is your feature request related to a problem? Please describe.
The library creates its own thread pool when compressing a large file which then contends with the main application that uses it (especially a problem in i.e. a game engine).

Describe the solution you'd like
An API to set a callback for it creating new tasks so you can execute them on your own thread pool when convenient would be nice. Basically fully replacing the pool implementation that comes with the library.

Describe alternatives you've considered
Suppose the alternative would be to not use the multi-threading feature within a single task, since we can already run individual compression tasks when convenient.

@Cyan4973
Copy link
Contributor

So, you're asking for a feature equivalent to "shared threadPool", exposed at:
https://github.com/facebook/zstd/blob/v1.5.6/lib/zstd.h#L1812 ,
but with the ability to inject an externally defined threadPool via callbacks ?

This would be a new feature indeed.

@Zeblote
Copy link
Author

Zeblote commented May 13, 2024

Yeah, something that would give control over when to run the tasks (and from which threads) to the user. There's an example of another library doing this here: https://github.com/NVIDIA-Omniverse/PhysX/blob/main/physx/include/task/PxCpuDispatcher.h#L49

It's a C++ library, but a similar pattern could easily be made in C by giving it a function pointer where it should submit new tasks to.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants