Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Variable limits for runs/dataclips (mem, disk, time) by project #2045

Open
4 tasks
taylordowns2000 opened this issue Apr 30, 2024 · 1 comment
Open
4 tasks
Labels

Comments

@taylordowns2000
Copy link
Member

taylordowns2000 commented Apr 30, 2024

Right now, limits are only set at the instance level, but we need to support setting limits at the project level also. This epic includes a front-end view of the limits for a given project, and several backend changes to ensure that limits are being passed properly to the ws-worker on a per run basis (so that we can define them based on their project—and associated plan, if present).

ToDo:

  • Display limits for the current project in a block on the Project Settings home screen "Setup" (These limits are defined by the instance level limits if this is an unlimited Lightning deployment. They are based on plan limits if this is viewed through a limited/shaped Lightning deployment.)
  • Ensure that max dataclip size can be passed to the worker when a run is started and that dataclips get dropped by Lightning if exceeded.
  • Ensure that maximum run duration is passed to the worker when a run is started and runs are timed out if exceeded.
  • Ensure that maximum run memory is passed to the worker when a run is started and runs are OOMKilled if exceeded.

Note: we know that right now, these limits are respected at least at the level the instance through the use of the following environment variables: https://openfn.github.io/lightning/deployment.html#limits

WORKER_MAX_RUN_MEMORY_MB - how much memory (in MB) can a single run use?
WORKER_MAX_RUN_DURATION_SECONDS - the maximum duration (in seconds) that workflows are allowed to run (keep this below your termination_grace_period if using kubernetes)
WORKER_CAPACITY - the number of runs a ws-worker instance will take on concurrently.
MAX_DATACLIP_SIZE_MB - the maximum size (in MB) of a dataclip created via the webhook trigger URL for a job. This limits the max request size via the JSON plug and may (in future) limit the size of dataclips that can be stored as run_results via the websocket connection from a worker.

@taylordowns2000
Copy link
Member Author

@christad92 , would love to chat with you about this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Icebox
Development

No branches or pull requests

1 participant