-
Notifications
You must be signed in to change notification settings - Fork 43
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bikeshed a worker/job queue system #57
Comments
Well, the choice of job system depends on Rustodon's design goals. If the goal is scaling down -- scaling down to small systems and situations where no system administrator is present -- I think in-process, or at least automatically managed by the main process, makes sense. (I say that as to not preclude the possibility of the job runners being e.g. separate threads in a separate process whose lifetime is controlled by the main rustodon process, because that can be done without human intervention, and a separate worker process may be useful for taking advantage of isolation/quota mechanisms provided by the OS.) If the goal is scaling up to multiple machines, then it may make sense to have an explicitly controllable separate process, because existing tools for multi-machine deployments are designed to expect things like separate web and worker processes. I don't know if it's possible, or desirable, to target both spaces. My preference is in-process, but I'm also interested in squeezing an ActivityPub server into nothing more than a single executable and state file (i.e. #12), so that should explain my position there. |
I'd have to agree with the general principle that scaling down is more useful than scaling up. Anyone with tens of thousands of users is likely going to have significant infrastructure they'll want to leverage, in the form of distributed data stores, single sign-on systems, etc. That'll mean they'd have to rewrite a lot of it anyway. On the other hand, being able to easily run an instance on a cheap VPS or even a low-power single board computer maximizes the usefulness of federation and allows wide scaling with lots of instances. |
|
i know @yipdw has Thoughts, it would be good to serialize them here maybe?
see also my post on r/rust, where people basically went 🤷♀️
dumping things here for myself or other people implementing this
existing code for worker pools n stuff
what do we want the interface to look like
what happens if
rustodon(1)
crashes when we still have jobs hanging aroundif they're, eg, deliveries, they need to be retried when it comes back up.
there are methods (for postgres, SKIP LOCKED) to build work queues in database, but we don't really care abt that because we don't have multiple worker processes, and we can dispatch jobs to worker threads entirely in-core. what we really need is a journal table of jobs that have been started but not completed; when the app comes up, we read the journal and reissue those jobs
The text was updated successfully, but these errors were encountered: