Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(trin-storage): add filter to store #1286

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

njgheorghita
Copy link
Collaborator

@njgheorghita njgheorghita commented May 3, 2024

What was wrong?

Soon, we'll need the ability to remove certain content from our storage according to a specified "validation function". This pr implements the ability to filter such data from the storage.

How was it fixed?

Added filter feature to storage

To-Do

@njgheorghita njgheorghita marked this pull request as ready for review May 3, 2024 15:25
@njgheorghita njgheorghita requested review from morph-dev, ogenev and KolbyML and removed request for morph-dev and ogenev May 3, 2024 15:34
@morph-dev
Copy link
Collaborator

Regardless if we decide to postpone this for later, I would like to highlight an issues with this approach.

Pagination doesn't really guarantee that you will see all entries. Consider following scenarios:

Scenario 1: Entries changing pages

  1. Query first page (entries 1-100) and delete 10 of them because they didn't pass the filter
  2. Query second page (entries 101-200)

Result: Entries that were originally at index 101-110 were never queried (because at the moment of second query they were moved to 91-100).

This can be avoided by either:

  • tracking number of deletes and updating offset
  • saving all entry keys to delete, and delete them once we finished iterating entire db

Scenario 2: Running as a separate task while trin normal operation are running as well

If trin is running normally, we can't avoid issue from Scenario 1 because trin can delete entries (e.g. using pruning) and affect pagination in a way that we can't control.

Note: This can be avoid by simply only running filtering on startup, but that should be documented (if not enforced).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants