Storj backups are failing - out of storage allowance, but error message didn't say that

[DeflateAwning]

• [ x] I have searched open and closed issues for duplicates.
• [ x] I have searched the forum for related topics.

---

Environment info

• Duplicati version: You are currently running Duplicati - 2.0.7.100_canary_2023-12-27
• Operating system: Windows 10
• Backend: Storj

Description

Backups are currently failing, and have been for a little while now. Seems as though an API changed or something.

The error seems to be that listing the file after uploading it is failing.

[duplicatibot]

This issue has been mentioned on Duplicati. There might be relevant details there:

https://forum.duplicati.com/t/error-while-running-synology-backup/17409/103

[gpatel-fr]

Hello

there is a test build here:

https://github.com/duplicati/duplicati/actions/runs/7996417398#artifacts

You can download it in the next 5 days (after which the builds are deleted) and test it to help validate it for next Canary.

[DeflateAwning]

Even in that build, it still appears to be failing. The error message is still "List verify failed, file was not found after upload: duplicati-xxxxxxxxxxxxxxxxxxxxxx.dindex.zip.aes".

[gpatel-fr]

It may be a specific problem in your setup then.
Can you do the following test: create a new bucket, a new backup (access grant), create a smallish backup (100 Mb) and test it ? That's what I did to test the build.
Also try to use Duplicati.CommandLine.BackendTool.exe to look at the backend like Duplicati does (use 'list' as command and take the Uri from an export as command line of the job):

Duplicati.CommandLine.BackendTool.exe list (your-uri)

and post the result.

[DeflateAwning]

The issue was that I had a max storage limit set in my Storj online account, and I had hit that limit. It then prevented me from uploading any more files past that point.

It would be good if someone could reproduce this error, and see if the Storj API returns a more specific error code that could be parsed and relayed to the user. It's a very simple problem with an easy solution, but not knowing what the problem was cost me quite a bit of time.

[ts678]

quota-size is what you can do now. It should be respected, as Duplicati code doesn't classify Storj as something returning quota.

This value can be used to set a known upper limit on the amount of space a backend has. If the backend reports the size itself, this value is ignored.

@TopperDEL (who is the original author) Any opinion on getting either quota values for a warning, or at least a better message?

@DeflateAwning although new build might have fixed some other things, is the message the same as original vague description:

The error seems to be that listing the file after uploading it is failing.

Now:

"List verify failed, file was not found after upload: duplicati-xxxxxxxxxxxxxxxxxxxxxx.dindex.zip.aes".

means that the list worked but it didn't see the file it was expecting. You might get the complaint with Verify files button, then <job> --> Show log --> Remote will show the list output which you can click on to open, then search to see if the claim about the named missing file is accurate.

If so, then the challenge is to figure out if the missing file is relevant. This might involve a look in database, or posting bug report.

[DeflateAwning]

The error message appeared to be the same before and after the upgrade. I'm pretty sure error message ("List verify failed") was indicating that the error was only caught when Storj refused to allow the upload because I was at my quota. It would be better to catch the error during the upload itself, or better yet, to be aware of the "space remaining"

[ts678]

indicating that the error was only caught when Storj refused to allow the upload because I was at my quota. It would be better to catch the error during the upload itself

APIs vary, and I don't know this one (I did call for an expert), but error is typically returned upon failure, not in the middle of work.

I'm pretty sure error message ("List verify failed") was indicating

Looking at the longer message "List verify failed, file was not found after upload", that appears two places in code, associated with

list-verify-uploads

Verify uploads by listing contents.

and it looks to be done as extra check immediately after the upload has claimed to have been done. Did you turn that option on?

I wonder if Storj failed silently without even returning an error? I don't have Storj, but you have logs and you can look over them.
Check the usual job logs history, and the About --> Show log --> Stored for any clues at around the time of the first Storj failure.

or better yet, to be aware of the "space remaining"

This is getting repetitive. Do you wish to just end this talk with that request (which has been referred), or is there something else? Code changes and new releases are not an instant thing, as there is quite an enormous backlog. Any other support needed now? Information to avoid this problem has already been provided, but it requires your telling Duplicati what quota is, at least for now.

I have already looked through what I can find in the SDK documentation, and am not seeing that Storj can provide what you ask, however I am asking (above) the author of the Duplicati backend and the libraries that it uses, to see if I might be missing things.

[DeflateAwning]

The Storj docs for duplicati say to turn on the "List verify uploads" config, presumably because it's extremely cheap to list files, and also because it doesn't seem like the failure is caught before that verify step.

I wonder if Storj failed silently without even returning an error?

I think this is the case. ^

</request> :) (sorry for the repetition)

[TopperDEL]

I indeed do not have any clues about the quota from the uplink.NET-library point of view, which is the base for the Storj DCS backend. Listing is indeed cheep - so just checking if the file got uploaded or not is the best way to proof that it is there (or not).

I may check if I can get the "quota exceeded"-error from uplink bubble through up to duplicati to give a more reasonable error message. I will verify that and come back with an info.

[ts678]

Their description is incorrect IMO. This is not the final listing. There is also a final listing, but arguably an earlier catch is better...

[TopperDEL]

Ok, I've investigated a bit. So, uplink indeed sends me error codes like "STORAGE_LIMIT_EXCEEDED" and others. They also do get reported via a ERROR_MESSAGE-property and that one also gets written into the result of the upload-operation.

BUT: the result of a file upload is never checked. If it failed or not does not get returned to Duplicati. I think I messed this up or relied on duplicati realizing that problem with list-verify.

I've created PR #5120 for this. It raises a proper exception that should then be taken care of by Duplicati.

[ts678]

Thanks for the investigation and the PR. It's already had its commit. I guess this still doesn't get us an early warning as could be obtained if quota were available on demand, but it's a clearer failure result if the user doesn't manually say what quota is set to.

[DeflateAwning]

This is huge! Have yet to test, but I suspect this fixes several of the issues which were preventing me from actively recommending Duplicati-with-Storj as a great backup option.

Just to confirm, did these make it into the v2.0.7.101-2.0.7.101_canary_2024-03-08 release?

[ts678]

Just to confirm, did these make it into the v2.0.7.101-2.0.7.101_canary_2024-03-08 release?

It looks that way to me. I don't know why it didn't make the release notice though.

[kenkendk]

Yes, sorry, I somehow missed giving credit to @TopperDEL for fixing the quota issue.