-
Notifications
You must be signed in to change notification settings - Fork 500
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: PostgreSQL destination delete stale requires sequential scan #17913
Comments
Just waiting for permission to contribute per Line 20 in 85f0d9b
|
Hi @kd7lxl, thanks for suggesting a contribution. Maybe you can start with a draft PR to show how the implementation would look like? We'll need to also discuss this internally if we want the SDK schema to tell destinations if they should create indexes for columns, or it should be coded in the destination per column |
The scope of this request is to create an index to directly support the query used in cloudquery/plugins/destination/postgresql/client/delete.go Lines 18 to 25 in f649bed
This is not a generic request from the SDK to create indexes, since the SDK or source plugin is not responsible for the design of this Supporting indexes for other columns via the SDK would be an interesting discussion, but beyond the scope here. |
Which problem is this feature request solving?
The postgresql destination plugin deletes stale records like this:
cloudquery/plugins/destination/postgresql/client/delete.go
Lines 18 to 25 in f649bed
The filter on
_cq_source_name
and_cq_sync_time
requires a sequential scan of the table because these fields aren't indexed:On tables with several sources, this could be slow. It seems like it could benefit from an index.
Describe the solution you'd like
Create indexes on
_cq_source_name
and_cq_sync_time
.Pull request (optional)
The text was updated successfully, but these errors were encountered: