Optimize "search-file" transaction #745
Comments
|
There is nothing PackageKit can do here, it's just a slow operation on most backends.
Then over time we would have a massive, duplicated cache shadowing the one of the native package manager, and causing all kinds of synchronization issues. And if we would expire it often, it would not bring us any gains as we would have to hit the native PM a lot anyway and wait for a result. Having frontends use search-file both async and sparingly is IMHO still far better than papering over the inherent slow-ness of this operation on most backends. |
|
Just compared the worst case execution times of search via Using
|
|
|
Execution times are low after kernel caches FS pages (i.e from second run onwards, but the first Using
|
We already do async + only 3 Issue is the first
No. The idea was to keep the count low (max 10 entries per PK client with LRU cache). GNOME Software does
Not sure what sync issues we would face here as every call to Sample code:cache = {
"/usr/share/metainfo/org.freedesktop.appstream.compose.metainfo.xml": "appstream-compose",
"/usr/share/applications/firefox-esr.desktop": "firefox-esr",
"/usr/share/applications/google-chrome.desktop": "google-chrome-stable",
"/usr/share/applications/empathy.desktop": "empathy"
}
function search_file_cached (filename):
pkg_name = lookup_cache (cache, filename):
if pkg_name:
# get-files(pkg_name) faster than search-file(filename)
pkg_files = pk_client_get_files(pkg_name)
if filename in pkg_files:
return pkg_name
# not-in-cache / cache-entry-invalid , do 'search-file' and update cache.
pkg_name = pk_client_search_files (filename)
# add / replace entry in cache.
update_cache (cache, filename, pkg_name)
return pkg_name |
|
Here is the worst case execution time of top 60 The pattern clearly shows a sudden drop from Execution time of top 60
|
Current
search-filetransaction is very costly causing PK clients to wait for more than a minute. This has been discussed in https://gitlab.gnome.org/GNOME/gnome-software/-/issues/2280.There are 2 cases where this issue is primarily seen:
search-filetransaction after bootsearch-filetransaction while disk I/O is in progressBelow are PK logs from 4 instances of GNOME Software starts after system boot in Debian unstable (
aptbackend). The first instance shows use case[1]explained above, and the fourth instance shows use case[2], when a VM disk write was in progress.Log from
journalctl -xb /usr/libexec/packagekitd | grep search-file:As explained in https://gitlab.gnome.org/GNOME/gnome-software/-/issues/2280#note_2064829, it would be good if we can cache and optimize the
search-filetransaction by looking up cache and usingget-filestransaction to speed up things.As a start, we can use this optimization path when
N=1(whensearch-fileas a single file argument)The text was updated successfully, but these errors were encountered: