Recycle Bin for Packages #7363
Replies: 3 comments
-
Another may be not directly related request is if we can have a 'after_*_purge' function which tell about us what got purged. So probably it can be used to tell some other dependent services and system about the purge. |
Beta Was this translation helpful? Give feedback.
-
Definitely expanding the scope of the original topic, but it's time we talked about Soft deletion in CKANThe biggest issue (in terms of space) with soft-deleted datasets is that it's not clear when to delete external data like uploaded files, datastore tables, views, etc. If we delete them on soft-delete then undeleting will create a dataset with broken links and views. @JatinChimote points out that we don't even have hooks at purge (actual delete) time, so we can't delete then either. So I guess we just never remove files that have been uploaded or datastore data #5446 . Never deleting is a safe choice, but our users paying for every MB of server storage can't be happy about it. At a minimum we should have a way to scan for and remove uploaded files and datastore tables and views that are no longer linked (plus one for each cloud/blob storage extension too?). Another problem with soft deletion is that table space isn't free, and can even create significant performance problems for users doing seemingly normal things: #7119 We also have the issue of soft deleted records preventing new ones from being created: #4267 and ugly hacks to work around the issue in some situations: #2438 These are just a handful of the reported issues, and I'm sure many times the number of users reporting issues instead are quietly frustrated. Entities supporting soft deletionWhere have we implemented soft deletion? Well, these are our database tables with a state flag:
And some numbers from one of our sites with row and soft-deleted row counts:
* tags not used on our site due to performance and multilingual limitations ** resource state=active but parent package is state=deleted Where does soft deletion make sense?We have occasionally restored an accidentally deleted user or package by setting The majority of these soft-deleted records are slowing useful database queries with no benefit at all. I suggest we remove soft deletion and the state column from all these tables except for group, package and user. For group, package and user let's add a CLI command to purge deleted records (with an optional older-than parameter). If resources no longer have soft delete then we can safely remove uploads, datastore tables etc. when the resource is deleted or when their package is purged. Let's cover these cases with tests because leaking storage makes users unhappy. Let's formalize and document the expected soft deletion behavior: which entities it applies to and how to use it to restore accidentally deleted items. New And finally, I really like @jqnatividad 's suggestion. 🗑️ Trash bins 🗑️ for everyone, not just sysadmins! (limited to just those three entities, user-friendly and with restore covered by tests) |
Beta Was this translation helpful? Give feedback.
-
copied my comment to a new issue #7368 for visibility |
Beta Was this translation helpful? Give feedback.
-
Currently, there is a system-wide Trash where the sysadmin can purge "soft-deleted" packages, groups and organizations.
In a current implementation, there is interest in enabling the following functionality:
being able to "recover"/"undelete" a "soft-deleted" package, even if there are constraints on what can be recovered (e.g. only the latest revision can be recovered)
having the ability to have organization-level trash, where org-admins can recover/purge organization packages
having the ability to configure an aging policy on the trash cans (i.e. automatically purge packages after N days), and see the age of items in the trash can
having the ability to Search the trash can or at least have a better interface with pagination/sorting.
as a "stretch" goal - there is also interest in implementing a "personal" trash can, where a user can recover/purge his/her own datasets
Considerations/Questions:
Beta Was this translation helpful? Give feedback.
All reactions