Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add jitter on the processor fetch backoff sleep #868

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

pior
Copy link

@pior pior commented Apr 22, 2024

Changes

This PR adds some jitter to the backoff sleep of the processor fetch loop to avoid workers synchronization, and thus improve worker efficiency and job latency.

Context

We run dozens of workers, with an execution rate of up to 5K jobs/sec.

We started to suspect that the job workers were significantly inefficient.

  1. Changing the TaskCheckInterval from 1s (default) to 100ms has reduced the latency from ~500ms to 40ms on a high traffic worker pool, and 10ms to a low traffic worker pool.

  2. Then, adding the jitter has further reduced the latency to 2 ms on both high and low traffic worker pool.

Synchronization on a distributed system is well known, and adding jitter helped avoid it.

Impact of adding a jitter: Queue size:
Screenshot 2024-04-15 at 19 05 32

Latency:
Screenshot 2024-04-15 at 19 05 51

@pior
Copy link
Author

pior commented Apr 25, 2024

Rebased on master

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants