v3: Refactor attempt creation to be worker requested

[ericallam]

Changes

Lazy attempt creation:

• The "worker" side now is responsible for requesting a new attempt
• Currently this is only implemented on the "dev" side
• Moves the visibility queue outside of redis and into a graphile job, which we are now using to fail runs that are in progress and hit the visibility timeout (unless the run is pending)
• Better handle cancelling runs when dev reconnects to a different web server instance
• This should allow prod workers to retry without checkpointing and without going back into the queue
• Each attempt gets a fresh process. The process is killed after the attempt completes and there can never be more than two processes up per run.

Other changes:

• Clear paused states before retry
• Detect and handle unrecoverable worker errors
• Remove checkpoints after successful push
• Permanently switch to DO hosted busybox image
• Fix IPC timeout issue, or at least handle it more gracefully
• Handle checkpoint failures
• Basic chaos monkey for checkpoint testing
• Stack traces are back in the dashboard
• Display final errors on root span

Testing checklist

There have been many changes to what happens after attempt completion and before retries. It's not enough to test that tasks complete successfully. In all scenarios, failure needs to be tested as well, 3-4 retries should be enough. This will also ensure we test for memory leaks, particularly when combined with checkpoints.

General guidelines:

• keep an eye on memory usage
• verify checkpoint size of all attempts
• when using waits, always force failure afterwards, not before
• also test with retry delays >30s

All relevant catalog entries start with lazy- and the following payload format can be used with all of them:

{
  "delayInSeconds": 35,
  "forceError": true
}

New SDK

Dev

• immediate return
• immediate return - with forced failure
• 1s wait
• 1s wait - with forced failure
• 35s wait
• 35s wait - with forced failure
• single dependency
• single dependency - with forced failure
• batch dependency
• batch dependency - with forced failure
• consecutive 1s wait
• consecutive 1s wait - with forced failure
• consecutive 35s wait
• consecutive 35s wait - with forced failure
• consecutive single dependency
• consecutive single dependency - with forced failure
• consecutive batch dependency
• consecutive batch dependency - with forced failure
• 35s wait, single dep
• 35s wait, single dep - with forced failure
• 35s wait, batch dep
• 35s wait, batch dep - with forced failure
• single dep, batch dep
• single dep, batch dep - with forced failure
• single dep, 35s wait
• single dep, 35s wait - with forced failure
• batch dep, 35s wait
• batch dep, 35s wait - with forced failure
• batch dep, single dep
• batch dep, single dep - with forced failure
• a random selection of more complex tasks

Prod

• immediate return
• immediate return - with forced failure
• 1s wait
• 1s wait - with forced failure
• 35s wait
• 35s wait - with forced failure
• single dependency
• single dependency - with forced failure
• batch dependency
• batch dependency - with forced failure
• consecutive 1s wait
• consecutive 1s wait - with forced failure
• consecutive 35s wait
• consecutive 35s wait - with forced failure
• consecutive single dependency
• consecutive single dependency - with forced failure
• consecutive batch dependency
• consecutive batch dependency - with forced failure
• 35s wait, single dep
• 35s wait, single dep - with forced failure
• 35s wait, batch dep
• 35s wait, batch dep - with forced failure
• single dep, batch dep
• single dep, batch dep - with forced failure
• single dep, 35s wait
• single dep, 35s wait - with forced failure
• batch dep, 35s wait
• batch dep, 35s wait - with forced failure
• batch dep, single dep
• batch dep, single dep - with forced failure
• a random selection of more complex tasks

Old SDK

Dev

• immediate return
• immediate return - with forced failure
• 1s wait
• 1s wait - with forced failure
• 35s wait
• 35s wait - with forced failure
• single dependency
• single dependency - with forced failure
• batch dependency
• batch dependency - with forced failure
• consecutive 1s wait
• consecutive 1s wait - with forced failure
• consecutive 35s wait
• consecutive 35s wait - with forced failure
• consecutive single dependency
• consecutive single dependency - with forced failure
• consecutive batch dependency
• consecutive batch dependency - with forced failure
• 35s wait, single dep
• 35s wait, single dep - with forced failure
• 35s wait, batch dep
• 35s wait, batch dep - with forced failure
• single dep, batch dep
• single dep, batch dep - with forced failure
• single dep, 35s wait
• single dep, 35s wait - with forced failure
• batch dep, 35s wait
• batch dep, 35s wait - with forced failure
• batch dep, single dep
• batch dep, single dep - with forced failure
• a random selection of more complex tasks

Prod

• immediate return
• immediate return - with forced failure
• 1s wait
• 1s wait - with forced failure
• 35s wait
• 35s wait - with forced failure
• single dependency
• single dependency - with forced failure
• batch dependency
• batch dependency - with forced failure
• consecutive 1s wait
• consecutive 1s wait - with forced failure
• consecutive 35s wait
• consecutive 35s wait - with forced failure
• consecutive single dependency
• consecutive single dependency - with forced failure
• consecutive batch dependency
• consecutive batch dependency - with forced failure
• 35s wait, single dep
• 35s wait, single dep - with forced failure
• 35s wait, batch dep
• 35s wait, batch dep - with forced failure
• single dep, batch dep
• single dep, batch dep - with forced failure
• single dep, 35s wait
• single dep, 35s wait - with forced failure
• batch dep, 35s wait
• batch dep, 35s wait - with forced failure
• batch dep, single dep
• batch dep, single dep - with forced failure
• a random selection of more complex tasks

[changeset-bot]

🦋 Changeset detected

Latest commit: 2099d91

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[nicktrn]

Great work everyone 🤝