New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
api: Create logic for automatically cleaning up dead streams #2013
Commits on May 4, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 33d5601 - Browse repository at this point
Copy the full SHA 33d5601View commit details -
Configuration menu - View commit details
-
Copy full SHA for e1783ad - Browse repository at this point
Copy the full SHA e1783adView commit details -
api: Fix the active cleanup query
- We were missing the parentId check so it was reading way more streams than necessary (all child streams/sessions). - There are A LOT of forgotten streams in our DB (400k parents), so we need to put some limit on this query
Configuration menu - View commit details
-
Copy full SHA for 0822fac - Browse repository at this point
Copy the full SHA 0822facView commit details -
Configuration menu - View commit details
-
Copy full SHA for ef6df93 - Browse repository at this point
Copy the full SHA ef6df93View commit details -
Configuration menu - View commit details
-
Copy full SHA for 672fd27 - Browse repository at this point
Copy the full SHA 672fd27View commit details -
Revert "api: Avoid using all DB resources for cleanup"
This reverts commit aff5120.
Configuration menu - View commit details
-
Copy full SHA for aefeb54 - Browse repository at this point
Copy the full SHA aefeb54View commit details -
api: Create separate DB conn pool for bg jobs
Use it on: - Webhooks queue handler - Tasks queue handler - Usage billing job - (new) Active cleanup job
Configuration menu - View commit details
-
Copy full SHA for 46c6ccd - Browse repository at this point
Copy the full SHA 46c6ccdView commit details -
Configuration menu - View commit details
-
Copy full SHA for 7bc71cc - Browse repository at this point
Copy the full SHA 7bc71ccView commit details -
api: Stop triggering isActive clean-up from GETs
Still monkey-patch responses to keep the same UX, but the actual recordings processing will only be triggered once the active clean-up job is triggered.
Configuration menu - View commit details
-
Copy full SHA for 1d1e433 - Browse repository at this point
Copy the full SHA 1d1e433View commit details -
Configuration menu - View commit details
-
Copy full SHA for 181f9b5 - Browse repository at this point
Copy the full SHA 181f9b5View commit details -
api: Change default limit to 1000
It's a bit too expensive to do 10000 at once
Configuration menu - View commit details
-
Copy full SHA for d60370e - Browse repository at this point
Copy the full SHA d60370eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4af36a4 - Browse repository at this point
Copy the full SHA 4af36a4View commit details -
api/stream: Clean-up lost sessions as well as parents
Turns out there are also children lost from their parents out there, so we have to list them all.
Configuration menu - View commit details
-
Copy full SHA for a2423ea - Browse repository at this point
Copy the full SHA a2423eaView commit details -
Configuration menu - View commit details
-
Copy full SHA for 68770e5 - Browse repository at this point
Copy the full SHA 68770e5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3597feb - Browse repository at this point
Copy the full SHA 3597febView commit details -
Configuration menu - View commit details
-
Copy full SHA for 8034798 - Browse repository at this point
Copy the full SHA 8034798View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7f370ba - Browse repository at this point
Copy the full SHA 7f370baView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0f517b7 - Browse repository at this point
Copy the full SHA 0f517b7View commit details -
Configuration menu - View commit details
-
Copy full SHA for fb3b9ba - Browse repository at this point
Copy the full SHA fb3b9baView commit details -
api/stream: Remove ordering from list query
Just makes things super slow and is not useful
Configuration menu - View commit details
-
Copy full SHA for c82362b - Browse repository at this point
Copy the full SHA c82362bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 02dfce7 - Browse repository at this point
Copy the full SHA 02dfce7View commit details -
api/stream: Make size of clean-up more predictable
Don't filter on the outer level, but rather make sure the pipeline supports an array of child streams.
Configuration menu - View commit details
-
Copy full SHA for 85967f8 - Browse repository at this point
Copy the full SHA 85967f8View commit details -
api/stream: Remove test for deduping by sessionId
Now we clean all of them, but we merge before the cleanup processing only the session once. No easy way to catch that on tests.
Configuration menu - View commit details
-
Copy full SHA for ffe8729 - Browse repository at this point
Copy the full SHA ffe8729View commit details -
Configuration menu - View commit details
-
Copy full SHA for b953e65 - Browse repository at this point
Copy the full SHA b953e65View commit details -
api/stream: Run clean-up synchronously on API
Also parallelize parent streams
Configuration menu - View commit details
-
Copy full SHA for 0c2fb67 - Browse repository at this point
Copy the full SHA 0c2fb67View commit details -
Configuration menu - View commit details
-
Copy full SHA for d2d7fbe - Browse repository at this point
Copy the full SHA d2d7fbeView commit details -
Configuration menu - View commit details
-
Copy full SHA for ceea027 - Browse repository at this point
Copy the full SHA ceea027View commit details
Commits on May 5, 2024
-
api/stream: Only use isActive field from streams
There are some bugged sessions that got isActive set to them
Configuration menu - View commit details
-
Copy full SHA for 8621089 - Browse repository at this point
Copy the full SHA 8621089View commit details
Commits on May 6, 2024
-
Revert "api/stream: Only use isActive field from streams"
This reverts commit 8621089. It was a staging only thing, likely from a development version.
Configuration menu - View commit details
-
Copy full SHA for 5724c14 - Browse repository at this point
Copy the full SHA 5724c14View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1178457 - Browse repository at this point
Copy the full SHA 1178457View commit details -
Configuration menu - View commit details
-
Copy full SHA for 269e235 - Browse repository at this point
Copy the full SHA 269e235View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4ba4937 - Browse repository at this point
Copy the full SHA 4ba4937View commit details -
Configuration menu - View commit details
-
Copy full SHA for ce4d436 - Browse repository at this point
Copy the full SHA ce4d436View commit details -
Configuration menu - View commit details
-
Copy full SHA for 63d39f7 - Browse repository at this point
Copy the full SHA 63d39f7View commit details