Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

api: Create logic for automatically cleaning up dead streams #2013

Merged
merged 34 commits into from May 6, 2024

Commits on May 4, 2024

  1. Configuration menu
    Copy the full SHA
    33d5601 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e1783ad View commit details
    Browse the repository at this point in the history
  3. api: Fix the active cleanup query

    - We were missing the parentId check so it was reading way
    more streams than necessary (all child streams/sessions).
    - There are A LOT of forgotten streams in our DB (400k parents),
    so we need to put some limit on this query
    victorges committed May 4, 2024
    Configuration menu
    Copy the full SHA
    0822fac View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    ef6df93 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    672fd27 View commit details
    Browse the repository at this point in the history
  6. Revert "api: Avoid using all DB resources for cleanup"

    This reverts commit aff5120.
    victorges committed May 4, 2024
    Configuration menu
    Copy the full SHA
    aefeb54 View commit details
    Browse the repository at this point in the history
  7. api: Create separate DB conn pool for bg jobs

    Use it on:
    - Webhooks queue handler
    - Tasks queue handler
    - Usage billing job
    - (new) Active cleanup job
    victorges committed May 4, 2024
    Configuration menu
    Copy the full SHA
    46c6ccd View commit details
    Browse the repository at this point in the history
  8. api: Fix tests

    victorges committed May 4, 2024
    Configuration menu
    Copy the full SHA
    7bc71cc View commit details
    Browse the repository at this point in the history
  9. api: Stop triggering isActive clean-up from GETs

    Still monkey-patch responses to keep the same UX,
    but the actual recordings processing will only be
    triggered once the active clean-up job is triggered.
    victorges committed May 4, 2024
    Configuration menu
    Copy the full SHA
    1d1e433 View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    181f9b5 View commit details
    Browse the repository at this point in the history
  11. api: Change default limit to 1000

    It's a bit too expensive to do 10000 at once
    victorges committed May 4, 2024
    Configuration menu
    Copy the full SHA
    d60370e View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    4af36a4 View commit details
    Browse the repository at this point in the history
  13. api/stream: Clean-up lost sessions as well as parents

    Turns out there are also children lost from their parents out
    there, so we have to list them all.
    victorges committed May 4, 2024
    Configuration menu
    Copy the full SHA
    a2423ea View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    68770e5 View commit details
    Browse the repository at this point in the history
  15. Configuration menu
    Copy the full SHA
    3597feb View commit details
    Browse the repository at this point in the history
  16. Configuration menu
    Copy the full SHA
    8034798 View commit details
    Browse the repository at this point in the history
  17. Configuration menu
    Copy the full SHA
    7f370ba View commit details
    Browse the repository at this point in the history
  18. Configuration menu
    Copy the full SHA
    0f517b7 View commit details
    Browse the repository at this point in the history
  19. Configuration menu
    Copy the full SHA
    fb3b9ba View commit details
    Browse the repository at this point in the history
  20. api/stream: Remove ordering from list query

    Just makes things super slow and is not useful
    victorges committed May 4, 2024
    Configuration menu
    Copy the full SHA
    c82362b View commit details
    Browse the repository at this point in the history
  21. Configuration menu
    Copy the full SHA
    02dfce7 View commit details
    Browse the repository at this point in the history
  22. api/stream: Make size of clean-up more predictable

    Don't filter on the outer level, but rather make sure
    the pipeline supports an array of child streams.
    victorges committed May 4, 2024
    Configuration menu
    Copy the full SHA
    85967f8 View commit details
    Browse the repository at this point in the history
  23. api/stream: Remove test for deduping by sessionId

    Now we clean all of them, but we merge before the cleanup
    processing only the session once. No easy way to catch that on
    tests.
    victorges committed May 4, 2024
    Configuration menu
    Copy the full SHA
    ffe8729 View commit details
    Browse the repository at this point in the history
  24. Configuration menu
    Copy the full SHA
    b953e65 View commit details
    Browse the repository at this point in the history
  25. api/stream: Run clean-up synchronously on API

    Also parallelize parent streams
    victorges committed May 4, 2024
    Configuration menu
    Copy the full SHA
    0c2fb67 View commit details
    Browse the repository at this point in the history
  26. Configuration menu
    Copy the full SHA
    d2d7fbe View commit details
    Browse the repository at this point in the history
  27. Configuration menu
    Copy the full SHA
    ceea027 View commit details
    Browse the repository at this point in the history

Commits on May 5, 2024

  1. api/stream: Only use isActive field from streams

    There are some bugged sessions that got isActive set to them
    victorges committed May 5, 2024
    Configuration menu
    Copy the full SHA
    8621089 View commit details
    Browse the repository at this point in the history

Commits on May 6, 2024

  1. Revert "api/stream: Only use isActive field from streams"

    This reverts commit 8621089.
    
    It was a staging only thing, likely from a development version.
    victorges committed May 6, 2024
    Configuration menu
    Copy the full SHA
    5724c14 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    1178457 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    269e235 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    4ba4937 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    ce4d436 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    63d39f7 View commit details
    Browse the repository at this point in the history