Internal consistency check failed, generated index block has wrong hash #3202
Comments
|
Confirmed - same problem here with v2.0.3.6-2.0.3.6_canary_2018-04-23 under CentOS 6.9. |
|
Confirmed running 2.0.3.6_canary_2018-04-23 on Windows 7 x64 |
|
Confirmed running 2.0.3.6_canary_2018-04-23 on Fedora Linux 26 x64 Workaround: Assumption: |
|
Setting --concurrency-block-hashers to 1 and then repairing the database
seems to have fixed it for me.
…On Wed, May 16, 2018, 18:19 spoppi ***@***.***> wrote:
Confirmed running 2.0.3.6_canary_2018-04-23 on Fedora Linux 26 x64
Workaround:
Downgrade to 2.0.3.5 using the steps from
https://forum.duplicati.com/t/downgrading-reverting-to-a-lower-version/3393
Assumption:
According to the changelog 2.0.3.6 "adds concurrent processing for the
backup" which I assume might interfere with hash calculation.
Maybe setting the new options --concurrency-* to lower values (ie 1) might
also help but hasn't been tested.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#3202 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AE5SjCwPM4nUt45NEZEKHG89fL4Rdp3zks5tzEN-gaJpZM4Ttr-D>
.
|
|
I deleted whole backup set and database and clean started with parameter --concurrency-block-hashers=1, however issue remains. Backup runs once and completed. But when you'll try to run next backup, it fails. I tried it twice, same result both times. I'm eagerly waiting for a new (next) version which would fix this. Yet I saw this error message, maybe there's something else going on? |
|
I'm hitting this as well (on a backup I've been trying to repair for a while - I've done several clean recreates, purged broken files...I'm about to the point of just restarting the backup from scratch). I'm also running 2.0.3.6. Edit: This was the specific error I got:
Scratch that, it looks like its essentially rebuilding the blocklist based on the the hashes of the entries, and then validating that that block has the expected hash. |
|
The code in there has actually not been updated, and should not be used concurrently. It simply builds an index file, by reading the local database. My best guess is that something else is using the database at the same time causing it to trip up the reader and return mixed results. With 2.0.3.6 I removed the old code that was building the index file in parallel with building the I will be re-introducing the previous "create-as-you-go" functionality, as many users have reported significant slowdowns while creating the index file from the database. |
…avoids issues with locks and transactions being prematurely released. This might help with #3202
|
I don't know if it's a race condition / transient issue (though I suppose it could be). I repeated the backup run that gave the inconsistent hash error, and the second time found the exact same hash mismatch, which makes it seem like something repeatable is happening, and race conditions usually aren't. I've also been trying to extract the database results and run a mini program that runs the GetBlocklists() method with that explicit input, in the hopes of matching either the reported or actual hash, and so far I haven't been able to, so there's likely something wrong there. I have the list of hashes for the blocklist in question (if you'd like to take a look at them, I can send them in a PM). As for the commit you added to change the IEnumerables to arrays, I suspect in this case that will make things worse (and depending on how the IEnumerables are being consume, might actually point to where the error is coming from). In the case of the GetBlocklistsAsync method, the returned Tuple has a byte[] as the second item, and instead of being a new byte array for each blocklist, it reuses a buffer. With your recent commit, by the time anything starts processing the tuples, the buffer will be the same for every entry in the list. If the bug is being caused by the IEnumerable's results being performed concurrently (e.g., the buffer is being updated before the previous tuple has finished being processed) a better fix might be to change GetBlocklists() to return a scoped copy of the buffer, so that the buffer instance each Tuple has is unique: string curHash = null;
int index = 0;
byte[] buffer = new byte[blocksize];
using(var rd = cmd.ExecuteReader(sql, volumeid))
while (rd.Read())
{
var blockhash = rd.GetValue(0).ToString();
if ((blockhash != curHash && curHash != null) || index + hashsize > buffer.Length)
{
byte[] returnedBuffer = new byte[index];
Array.Copy(buffer, 0, returnedBuffer, 0, index);
yield return new Tuple<string, byte[], int>(curHash, returnedBuffer, index);
curHash = null;
index = 0;
}
var hash = Convert.FromBase64String(rd.GetValue(1).ToString());
Array.Copy(hash, 0, buffer, index, hashsize);
curHash = blockhash;
index += hashsize;
}
if (curHash != null)
{
byte[] returnedBuffer = new byte[index];
Array.Copy(buffer, 0, returnedBuffer, 0, index);
yield return new Tuple<string, byte[], int>(curHash, returnedBuffer, index);
} |
|
The new commit did what you expected, and that appears to be the opposite of fixing it. |
|
With the removal of the shared buffer, all the unittests now pass. I am still unsure if this will fix the actual problem, or just make it appear less frequently. If the problem was not caused by a race condition, but instead some kind of invalid data, we should see the problem pop up again with this fix. |
|
Even before this 2.0.3.6 version I've seen surprisingly lot i-file corruption, but b-files or dlists haven't been ever corrupted. It sounds likely it has been more or less broken all the time. |
|
Are there any updates on this issue? I have a client with this exact error, but only on one of many backups. |
|
Just delete the corrupt file, and next backup will run auto repair to replace it with empty place holder. That's what I've done. Yet, I haven't confirmed how it possibly affects restore. I think I'll need leave full restore tests running during the weekend. |
|
This issue has been mentioned on Duplicati. There might be relevant details there: |
Reproduction stepsThanks to @ts678 for finding this:
The general circumstances for this failure are:
Core problemThe SQL query in SELECT "A"."Hash", "C"."Hash" FROM (SELECT "BlocklistHash"."BlocksetID", "Block"."Hash", * FROM "BlocklistHash","Block" WHERE "BlocklistHash"."Hash" = "Block"."Hash" AND "Block"."VolumeID" = ?) A, "BlocksetEntry" B, "Block" C WHERE "B"."BlocksetID" = "A"."BlocksetID" AND "B"."Index" >= ("A"."Index" * 32) AND "B"."Index" < (("A"."Index" + 1) * 32) AND "C"."ID" = "B"."BlockID" ORDER BY "A"."BlocksetID", "B"."Index";The subquery will fetch the same blocklist multiple times from different file versions (BlocksetID 3 and 8 in this case): SELECT "BlocklistHash"."BlocksetID", "Block"."Hash", * FROM "BlocklistHash","Block" WHERE "BlocklistHash"."Hash" = "Block"."Hash" AND "Block"."VolumeID" = ?;
BlocksetID Hash BlocksetID Index Hash ID Hash Size VolumeID
3 DWypxEXA7TXbM2azclwoTH+Y7rEgA9kbDa4/K3ycNdA= 3 0 DWypxEXA7TXbM2azclwoTH+Y7rEgA9kbDa4/K3ycNdA= 4 DWypxEXA7TXbM2azclwoTH+Y7rEgA9kbDa4/K3ycNdA= 1024 3
3 v106/7c+/S7Gw2rTES3ZM+/tY8Thy//PqI4nWcFE8tg= 3 1 v106/7c+/S7Gw2rTES3ZM+/tY8Thy//PqI4nWcFE8tg= 6 v106/7c+/S7Gw2rTES3ZM+/tY8Thy//PqI4nWcFE8tg= 32 3
8 v106/7c+/S7Gw2rTES3ZM+/tY8Thy//PqI4nWcFE8tg= 8 1 v106/7c+/S7Gw2rTES3ZM+/tY8Thy//PqI4nWcFE8tg= 6 v106/7c+/S7Gw2rTES3ZM+/tY8Thy//PqI4nWcFE8tg= 32 3The external query will then get all block hashes for each of these results and append them. If different hashes are interlaced, this causes no problem, but if the same hash appears back to back, the blocklist is wrong. This subquery needs to be reworked to return each hash only once. |
Closes duplicati#3202 Also add test to ensure that repair works as intended.
Closes duplicati#3202 Also add test to ensure that repair works as intended.
Environment info
Description
Backup sets basically get broken by just running backup. Three totally independent systems, with different backends and clients. As well as one "local / file" backup, where backup data is stored on second local disk. Something is seriously wrong if this happens?
"Internal consistency check failed, generated index block has wrong hash"
Steps to reproduce
Debug log
Checking remote backup ... Listing remote folder ... Fatal error => System.Exception: Internal consistency check failed, generated index block has wrong hash, 6yil2I1aMnlEYfq7luTpJh8RytKe5Gm7eeUuk0UAELU= vs I5/lRw4QR1CWI8t2ojmy/pgUFgAUcQqr8z+zpybt3rk= at Duplicati.Library.Main.Operation.Common.IndexVolumeCreator+<CreateIndexVolume>d__0.MoveNext () <0x409eb6e0 + 0x00b5b> in <filename unknown>:0 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x7f58718016d0 + 0x00029> in <filename unknown>:0 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) <0x7f58717ff6b0 + 0x000a7> in <filename unknown>:0 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) <0x7f58717ff630 + 0x0006b> in <filename unknown>:0 at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) <0x7f58717ff5e0 + 0x0003a> in <filename unknown>:0 at System.Runtime.CompilerServices.TaskAwaiter 1[TResult].GetResult () <0x7f58717ff8d0 + 0x00017> in <filename unknown>:0 at Duplicati.Library.Main.Operation.Backup.RecreateMissingIndexFiles+<>c__DisplayClass1_0+<<Run>b__0>d.MoveNext () <0x409e7740 + 0x006be> in <filename unknown>:0 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x7f58718016d0 + 0x00029> in <filename unknown>:0 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) <0x7f58717ff6b0 + 0x000a7> in <filename unknown>:0 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) <0x7f58717ff630 + 0x0006b> in <filename unknown>:0 at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) <0x7f58717ff5e0 + 0x0003a> in <filename unknown>:0 at System.Runtime.CompilerServices.TaskAwaiter.GetResult () <0x7f58717ff5c0 + 0x00012> in <filename unknown>:0 at CoCoL.AutomationExtensions+<RunTask>d__10 1[T].MoveNext () <0x40899140 + 0x00231> in <filename unknown>:0 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x7f58718016d0 + 0x00029> in <filename unknown>:0 at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) <0x7f58717ff6b0 + 0x000a7> in <filename unknown>:0 at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) <0x7f58717ff630 + 0x0006b> in <filename unknown>:0 at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) <0x7f58717ff5e0 + 0x0003a> in <filename unknown>:0 at System.Runtime.CompilerServices.TaskAwaiter.GetResult () <0x7f58717ff5c0 + 0x00012> in <filename unknown>:0 at Duplicati.Library.Main.Operation.BackupHandler+<RunAsync>d__18.MoveNext () <0x40932000 + 0x01d8e> in <filename unknown>:0The text was updated successfully, but these errors were encountered: