Zypper fails to cache repodata when hard links are partially used

[dylanbartos]

Scenario
Zypper is configured to point towards a local repository (file:/opt/Mirror/repos/repo-id). The local repository gets refreshed on a regular basis from one of the public repos (SUSE/Updates/SLE-Product-SLES/15-SP1/x86_64/update for instance). The caching works fine when starting from scratch (an empty /var/cache/zypp), but fails when the mirror is refreshed.

Steps

1. From an empty /var/cache/zypp, the command zypper list-updates is run, the repository is refreshed and a scan is performed. All operations succeed and the /var/cache/zypp/raw/repo-id/repodata and /var/cache/zypp/solv/repo-id directories are created and filled. A cookie file is created at the solv directory which seems aligns to the repomd.xml file in the raw location. From what I could troubleshoot, as long as the cookie file and the repomd file are both present, the cache will be considered present, but the content of the repomd is not validated or hash checked. [zypp] RepoManager.cc(buildCache):1329 repo-id is already cached. However if either the cookie file or the repomd file are not present, the cache is considered not complete. A full refresh occurs.
   [zypp] RepoManager.cc(probeCache):1567 Probed cached type NONE at /var/cache/zypp/raw/repo-id
[zypp] RepoStatus.cc(fromCookieFile):200 No cookie file /var/cache/zypp/raw/repo-id/cookie

2. During the refresh, the repomd.xml file is obtained from the on disk location via a functionhardlinkCopy.
   [zypp-core] PathInfo.cc(hardlinkCopy):916 hardlinkCopy /opt/Mirror/repos/repo-id/repodata/repomd.xml -> /var/cache/zypp/raw/repo-idBsmKLc/repodata/repomd.xml
   In some scenarios, this causes a copy to occur and in other cases, a hard link. In the case of the hard link, the inode of the file at /var/cache/zypp/raw is the same of the file at the mirror directory /opt/Mirror/. I found that the repomd.xml was the only file which received a hard link and the primary.xml and additional repodata received legitimate copies.

3. Per normal operations, the mirror gets refreshed and the /opt/Mirror location gets updated with a fresh repomd.xml and repodata. However, given that the repomd.xml is hard linked, the repomd in the /var/cache/zypp/raw is also updated, since they point to the same bits on the filesystem. The repodata in the /var/cache/zypp/raw are not updated.

4. The next time that zypper list-updates runs, the buildCache fails to complete.

[zypper] repos.cc(do_init_repos):821 checking if to refresh repo-id
[zypp::media++] RepoInfo.cc(probeCache):65 Probed cached type rpm-md at /var/cache/zypp/raw/repo-id
[zypp] RepoManager.cc(buildCache):1329 repo-id is already cached.
[Progress++] progressdata.cc(report):75 {#3|Building repository 'Mirror: SLE-Module-Public-Cloud15-SP1-Updates' cache} START
[Progress++] progressdata.cc(report):75 {#4|} START
[zypp] RepoManager.cc(cleanCache):1626 Removing raw metadata cache for repo-id
[zypp-core] PathInfo.cc(recursive_rmdir):434 recursive_rmdir /var/cache/zypp/solv/repo-id 
[Progress++] progressdata.cc(report):94 {#4|} END
[zypp] RepoManager.cc(buildCache):1363 repo-id building cache...rpm-md
[zypp-core] PathInfo.cc(assert_dir):353 mkdir /var/cache/zypp/solv/repo-id 00755
[zypp] RepoManager.cc(buildCache):1394 repo type is rpm-md
[zypp::exec++] forkspawnengine.cc(start):181 Executing 'repo2solv' '-o' '/var/cache/zypp/solv/repo-id/solv' '-X' '/var/cache/zypp/raw/repo-id'
[zypp::exec++] forkspawnengine.cc(start):333 pid 13669 launched
[zypp] RepoManager.cc(buildCache):1429   /var/cache/zypp/raw/repo-id/repodata/854a8897daea0dff54c91a4c5e8c328cfb4be74454b92df8ad0e8d47cc26ba60-primary.xml.gz: No such file or directory
[zypp::exec] abstractspawnengine.cc(checkStatus):182 Pid 13669 exited with status 1
[zypp-core] Exception.cc(log):186 RepoManager.cc(buildCache):1438 THROW:    [repo-id|file:/opt/Mirror/repos/repo-id] Failed to cache repo (1).
[zypp-core] Exception.cc(log):186 History:
[zypp-core] Exception.cc(log):186  - 'repo2solv' '-o' '/var/cache/zypp/solv/repo-id/solv' '-X' '/var/cache/zypp/raw/repo-id'
[zypp-core] Exception.cc(log):186    /var/cache/zypp/raw/repo-id/repodata/854a8897daea0dff54c91a4c5e8c328cfb4be74454b92df8ad0e8d47cc26ba60-primary.xml.gz: No such file or directory
[zypp-core] Exception.cc(log):186    Command exited with status 1.
[zypp-core] Exception.cc(log):186 
[zypp-core] PathInfo.cc(unlink):704 unlink /var/cache/zypp/solv/repo-id/solv
[zypp-core] PathInfo.cc(doLogResult):294  FAILED: (2)No such file or directory
[zypp-core] Exception.cc(log):186 repos.cc(build_cache):456 CAUGHT:   [repo-id|file:/opt/Mirror/repos/repo-id] Failed to cache repo (1).
[zypp-core] Exception.cc(log):186 History:
[zypp-core] Exception.cc(log):186  - 'repo2solv' '-o' '/var/cache/zypp/solv/repo-id/solv' '-X' '/var/cache/zypp/raw/repo-id'
[zypp-core] Exception.cc(log):186    /var/cache/zypp/raw/repo-id/repodata/854a8897daea0dff54c91a4c5e8c328cfb4be74454b92df8ad0e8d47cc26ba60-primary.xml.gz: No such file or directory
[zypp-core] Exception.cc(log):186    Command exited with status 1.
[zypp-core] Exception.cc(log):186 
[zypper] repos.cc(build_cache):457 Error writing to cache db
[zypper] repos.cc(do_init_repos):903 Skipping repository 'repo-id' because of the above error.

5. I believe that this scenario is occurring because the repomd.xml file is effectively getting updated and the buildCache operation does not refresh the rest of the data because of the hard link. There are a couple of theories I had which might lead to a resolution of the issue:

• The cookie hash checks the repomd.xml to see if it has changed since the last buildCache
• If a hard link is used, all repodata must be hardlinked (I am not confident in this as the primary xml filenames frequently change and should be mapping to the name defined in the repomd.xml.
• Hard links are not used at all, but a full copy operation is expected when refreshing a repository which is on disk (perhaps an update to the logic in the hardlinkCopy function?)

Thank you for your time and consideration.