Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

0-byte files are not restored with RecoveryTool #4884

Closed
2 tasks done
7ERr0r opened this issue Jan 7, 2023 · 6 comments · Fixed by #4991
Closed
2 tasks done

0-byte files are not restored with RecoveryTool #4884

7ERr0r opened this issue Jan 7, 2023 · 6 comments · Fixed by #4991

Comments

@7ERr0r
Copy link

7ERr0r commented Jan 7, 2023 •
edited

The tool Duplicati.CommandLine.RecoveryTool.exe is faulty.
It does not restore empty (0 byte) files properly.
I tried it on:

  • Linux*
  • Wine on Linux (Ubuntu)
  • Windows 11

All these systems did not restore the 0-size files.
These files are critical in database operation.

*Linux is buggy

Backslash path separator is wrong: \. It should be /

Restoring Windows backup of file file.jpg in directory myDirectory is buggy on Linux because:

Windows path is myDirectory\file.jpg

RecoveryTool then restores it as:

$ ls -la
'myDirectory\file.jpg'

❌ Which does not create myDirectory, but a file named with a windows slash path 🤦
✅ It should create a directory: myDirectory and then file.jpg - with linux path myDirectory/file.jpg

Environment info

  • Duplicati version: duplicati-2.0.6.3_beta_2021-06-17.
  • Operating system: Windows 11
  • Backend: Samba/SMB on Linux (Debian)

Description

Zero-byte files are not restored with the Duplicati.CommandLine.RecoveryTool.exe program.

Steps to reproduce

  1. Create files with 0 size. Maybe using Carestream database.
  2. Backup using Duplicati on Windows
  3. Run Duplicati.CommandLine.RecoveryTool.exe restore
  • Actual result:
    ⛔ Exception is thrown and file not created
  • Expected result:
    ✅ File being created

Debug log

   at Duplicati.CommandLine.RecoveryTool.Restore.HashLookupHelper.ReadHash(String hash)
   at Duplicati.CommandLine.RecoveryTool.Restore.Run(List`1 args, Dictionary`2 options, IFilter filter)
49443: D:\pc_backup\D\DatabaseCS\D.RVG\D0000001\.csi_data\.version_4.4\0@R10.rvg\.version_20112016 (0 bytes) error: System.Exception: Unable to locate block with hash: 47DXXXXXHBSa+/TImW+5JCeXXXU=
   at Duplicati.CommandLine.RecoveryTool.Restore.HashLookupHelper.ReadHash(String hash)
   at Duplicati.CommandLine.RecoveryTool.Restore.Run(List`1 args, Dictionary`2 options, IFilter filter)
49444: D:\pc_backup\D\DatabaseCS\D.RVG\D0000001\.csi_data\.version_4.4\0@R10.rvg\meta (2.13 KB) done!
49445: D:\pc_backup\D\DatabaseCS\D.RVG\D0000001\.csi_data\.version_4.4\0@R10.rvg\t.png (45.13 KB) done!
49446: D:\pc_backup\D\DatabaseCS\D.RVG\D0000001\.csi_data\.version_4.4\0@R11.rvg\.nsar_20181121 (0 bytes) error: System.Exception: Unable to locate block with hash: 47DXXXXXHBSa+/TImW+5JCeXXXU=
   at Duplicati.CommandLine.RecoveryTool.Restore.HashLookupHelper.ReadHash(String hash)
   at Duplicati.CommandLine.RecoveryTool.Restore.Run(List`1 args, Dictionary`2 options, IFilter filter)
49447: D:\pc_backup\D\DatabaseCS\D.RVG\D0000001\.csi_data\.version_4.4\0@R11.rvg\.version_20112016 (0 bytes) error: System.Exception: Unable to locate block with hash: 47DXXXXXHBSa+/TImW+5JCeXXXU=
   at Duplicati.CommandLine.RecoveryTool.Restore.HashLookupHelper.ReadHash(String hash)
@yscialom
Copy link

yscialom commented Jan 9, 2023 •
edited

Even though I agree this is an issue, backupping databases from the filesystem is generally a bad idea™. Decent databases provide data backup facilities, either through an SQL dump or through a specific tool.

(I'm not part of the duplicati team)

@7ERr0r
Copy link
Author

7ERr0r commented Jan 12, 2023 •
edited

Edit: Thanks for making the .zip format so simple! It was easy to read all backups in Rust instead of C# 😄

🦀 https://github.com/7ERr0r/duplicati-restore-rs

@7ERr0r
Copy link
Author

7ERr0r commented Jan 13, 2023 •
edited

So RecoveryTool-like program in Rust uses 2x less RAM. And works out of the box without the index subcommand.

RecoveryTool:
image

rust-duplicati-restore:
Taskmgr_ukWCBFEoCH

In both cases source and target are on an SMB server.
Therefore, it's weird the C# version uses my regular drive... why??

@gpatel-fr gpatel-fr mentioned this issue Jan 13, 2023
Closed
@ts678
Copy link
Collaborator

ts678 commented Jan 23, 2023

As a wild guess, I wonder if this fell through the cracks when empty files were made a special case instead of usual case but with actual empty block stored in the destination dblock file. This apparently caused trouble, so was changed, which caused trouble...

Fixed empty block handling.
Fixed empty block handling in concurrent_processing branch #2637
(note RestoreHandler.cs change, however I think RecoveryTool's restore process is quite a bit different -- somebody could verify)
Thanks to @gpatel-fr for being quick on a PR for the current issue (and thanks for noticing). Release production is slow currently because release manager has had no time, but it's nice to have some things in the queue for when needed volunteers are found.

Empty source file can make Recreate download all dblock files fruitlessly with huge delay #3747 was previous regression noticed.
Check block size on recreate #3758 was the one-line cure after a difficult chase...

Therefore, it's weird the C# version uses my regular drive... why??

Regular restore and backup (and more) use lots of temporary files. Temporary folder is configured by tempdir option and more.
RecoveryTool has at least one temporary file usage that I noticed, here. Possibly that's the use? File name likely starts with dup-.
If that doesn't reveal, Process Monitor on Windows or strace on Linux could probably reveal. I don't read C# all that well though.

program in Rust

Thanks for that addition to the tool chest. There's also a Python script, and now I wonder whether that tool restores empty files...
Independent restore program
restore_from_python.py

@duplicatibot
Copy link

This issue has been mentioned on Duplicati. There might be relevant details there:

https://forum.duplicati.com/t/timespan-to-rebuild-a-database/15654/22

@duplicatibot
Copy link

This issue has been mentioned on Duplicati. There might be relevant details there:

https://forum.duplicati.com/t/ftp-delete-not-atomic-verified-transactional/12936/57

Jojo-1000 added a commit to Jojo-1000/duplicati that referenced this issue Jul 11, 2023
@Jojo-1000
Fix recovery tool restore paths on different OS.
6b56236 
On Linux, paths were created with \ instead of / separators if backup was originally from Windows (duplicati#4884).
Also fix bug that created one unnecessary level of directories in the restore path.
@Jojo-1000 Jojo-1000 mentioned this issue Jul 11, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
4 participants
@yscialom @7ERr0r @duplicatibot @ts678