Enable configurable queue concurrency at project level

[christad92]

Configurable concurrency is a mechanism that allows us to restrict how many workflows/jobs the worker can run in parallel within a project. When concurrency is set to 1, only a work order/run/job can be processed at a time and other jobs will be enqueued until that is done.

How this worked in v1

openfn often will…
1. get bunch of csv files from sftp --> convert to JSON // Input data 
2. post json payloads to project Inbox (but post the payloads in the order we want to process -e.g., first post Contacts data and then related Donations data bc Donations can’t exist without Contacts)
3. OpenFn would then process all Contacts messages 1-at-a-time first… before then moving onto the Donations message (moving chronologically through the Inbox/Runs queue)

• This is configurable in the project settings as an integer input

Questions:

• What is the current behaviour? Do we allow multiple jobs to run concurrently?

[taylordowns2000]

I'd say we need two things here, almost certainly separate issues. Something around: https://github.com/OpenFn/lightning/blob/main/lib/lightning/runs/queue.ex#L12

1. Artificially limit (or increase?) the number of runs being excecuted at once per project (concurrency dial, just like v1). Something like...?

i'm the worker, give me runs to execute...
WHERE NOT IN runs.state == executing AND runs.project_id IN (
  some sort of list of projects?
)

2. Artificially add a "cool-down" to stall, per project, after each run. Something like...?

i'm the worker, give me runs to execute...
WHERE NOT IN min(age(now(), runs.finished_at)) > '10 seconds' AND runs.project_id IN (
  some sort of list of projects?
)

Nice blog that @stuartc found - https://docs.hatchet.run/blog/multi-tenant-queues#introducing-concurrency-limits

And here's the UI from v1:

[christad92]

Stu is yet to spike the approach to implementing this. cc: @stuartc