Unexpected trie node error BAD BLOCK

[tompetrovic]

System information

Geth version: 1.14.3
CL client & version: prysm@v5.0.3
OS & Version: Ubuntu 22.04.4

Actual behaviour

My node has suddenly stopped syncing yesterday. I tried to fix it by removing the state (kept the ancient) and resyncing. It was working fine for a few hours but then the same error appeared again.

Could this be a hardware issue? The node has been running without any issues for more than 4 years now. Just in case if that's important... I use separate hard drives for state and ancients. State is on the NVMe, ancients are on a regular SSD.

Backtrace

WARN [05-15|09:39:21.179] NewPayloadV1: inserting block failed     error="invalid merkle root (remote: ec4bbed3d3dc17d824f56df84654ec7478d5a230d721f056d62f10735285c443 local: e7ced498748e1c2dfcf862fa9e84e5a0278d09095a4724566271e8d4d5d5a5af) dberr: updateStateObject (b8abeaafd2b564202598ff38158ecf48450ba109) error: missing trie node 1b585afb7f630480fb2124b717cc3810c8e6d44c83be20a5fbd8dd236878546e (path 05010e08) unexpected node: (0000000000000000000000000000000000000000000000000000000000000000 [5 1 14 8]), 1b585afb7f630480fb2124b717cc3810c8e6d44c83be20a5fbd8dd236878546e!=c5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470, loc: disk, depth: 128, blob: nil"
WARN [05-15|10:24:32.513] Too many bad block import attempt, trying number=19,871,215 hash=980a14..aadeb5
ERROR[05-15|10:24:32.529] Unexpected trie node                     location=disk owner=000000..000000 path="[5 1 14 8]"                             expect=1b585a..78546e got=c5d246..85a470 blob=nil
ERROR[05-15|10:24:32.576] Unexpected trie node                     location=disk owner=000000..000000 path="[5 1 14 8]"                             expect=1b585a..78546e got=c5d246..85a470 blob=nil
INFO [05-15|10:24:32.579] Skip duplicated bad block                number=19,871,215 hash=980a14..aadeb5
ERROR[05-15|10:24:32.581] "\n########## BAD BLOCK #########\nBlock: 19871215 (0x980a149bfc450f6ad8f7fea4cd819ac3ab6f363aa6d365d7a2cbd302d1aadeb5)\nError: invalid merkle root (remote: ec4bbed3d3dc17d824f56df84654ec7478d5a230d721f056d62f10735285c443 local: e7ced498748e1c2dfcf862fa9e84e5a0278d09095a4724566271e8d4d5d5a5af) dberr: updateStateObject (b8abeaafd2b564202598ff38158ecf48450ba109) error: missing trie node 1b585afb7f630480fb2124b717cc3810c8e6d44c83be20a5fbd8dd236878546e (path 05010e08) unexpected node: (0000000000000000000000000000000000000000000000000000000000000000 [5 1 14 8]), 1b585afb7f630480fb2124b717cc3810c8e6d44c83be20a5fbd8dd236878546e!=c5d2460186f7233c927e7db2dcc703c0e500b653ca82273b7bfad8045d85a470, loc: disk, depth: 128, blob: nil\nPlatform: geth (devel) go1.22.3 amd64 linux\nVCS: ab48ba42-\nChain config: &params.ChainConfig{ChainID:1, HomesteadBlock:1150000, DAOForkBlock:1920000, DAOForkSupport:true, EIP150Block:2463000, EIP155Block:2675000, EIP158Block:2675000, ByzantiumBlock:4370000, ConstantinopleBlock:7280000, PetersburgBlock:7280000, IstanbulBlock:9069000, MuirGlacierBlock:9200000, BerlinBlock:12244000, LondonBlock:12965000, ArrowGlacierBlock:13773000, GrayGlacierBlock:15050000, MergeNetsplitBlock:<nil>, ShanghaiTime:(*uint64)(0xc000460da0), CancunTime:(*uint64)(0xc000460da8), PragueTime:(*uint64)(nil), VerkleTime:(*uint64)(nil), TerminalTotalDifficulty:58750000000000000000000, TerminalTotalDifficultyPassed:true, Ethash:(*params.EthashConfig)(0x2e8c740), Clique:(*params.CliqueConfig)(nil)}\nReceipts: \n...

[holiman]

Could this be a hardware issue?

Yes, definitely! (as in: yes, it definitely can be, not that it definitely is).

The NVMe holding the state is the interesting one, which can lead to this type of corruption. I would advise to do a disk health check, check for corrupted bits. Drives do degrade over time, and four years of heavy mainnet-node operation takes its toll.

[rjl493456442]

My node has suddenly stopped syncing yesterday. I tried to fix it by removing the state (kept the ancient) and resyncing. It was working fine for a few hours but then the same error appeared again.

Just to confirm that you use v.1.14.3 to resync the state right?

[rjl493456442]

Could be relevant with #29436

[rjl493456442]

The correct node blob should be
0xf90211a0cedb62eb6fe837c799ed182fb39585c46848cf189414427222ae23fca3b02226a0e22c7cc42c5fc438a01a8872cde5cd39f78e3ad6f9117100772de7454ebabebaa0f8eb3c8bb6bb557782c511353d8b8cd64364d0180a1c7e448726a5a927720f6ca0d965da52d4522432e80a835b3a67ee8b4f12f4f3e430365e57c82b25cf475053a0c09c3f76ae9fad65807b71ef8803ac409cfa9099a712d96db37f85adeb408de5a01d8728e8b9397db70b5043716aeee4bd6deea4d28cd29ff460a8bf12a66d9330a02c5157537434575edf3eadf4a13e127e215f85f9f685e6af9963493b40bca307a0f03d034d832b8e89c33fa5d164b64400edfc6cd16977f3d0d0d487cb664ce90fa0d12cb13e4e3dfdcaef7f261df8d1d668655bacd16296a023b51db3896e3f88aca0bb378bc60d27d36c8239e85f2abcdfa4cd4cead2e05735fa7c48522719ee59e6a070c7c4b49f2f3eac127edaf6ac474c47f17c768e43d359306fd7e6687509e7bba036a2622ccb096155f068b57d4256b9a3ca9688c1ed724ff005f4c81f25897155a00421c22708f34e2ef1d4a6899df90e21a658752310c83c345e4de0488dc2240ba0a926c4cb6ce26e86fc9f5302000c11d92513620cf75b56813fd401ce40934d56a0b49139eee3a05eaa0293da49864f5612d9971c2524da815126a22aab7c2d7b32a0d3fbddf581ab011def20ae226d832939603900c9b0ad6c4cca2f7be138393a1280

[
  cedb62eb6fe837c799ed182fb39585c46848cf189414427222ae23fca3b02226,
  e22c7cc42c5fc438a01a8872cde5cd39f78e3ad6f9117100772de7454ebabeba,
  f8eb3c8bb6bb557782c511353d8b8cd64364d0180a1c7e448726a5a927720f6c,
  d965da52d4522432e80a835b3a67ee8b4f12f4f3e430365e57c82b25cf475053,
  c09c3f76ae9fad65807b71ef8803ac409cfa9099a712d96db37f85adeb408de5,
  1d8728e8b9397db70b5043716aeee4bd6deea4d28cd29ff460a8bf12a66d9330,
  2c5157537434575edf3eadf4a13e127e215f85f9f685e6af9963493b40bca307,
  f03d034d832b8e89c33fa5d164b64400edfc6cd16977f3d0d0d487cb664ce90f,
  d12cb13e4e3dfdcaef7f261df8d1d668655bacd16296a023b51db3896e3f88ac,
  bb378bc60d27d36c8239e85f2abcdfa4cd4cead2e05735fa7c48522719ee59e6,
  70c7c4b49f2f3eac127edaf6ac474c47f17c768e43d359306fd7e6687509e7bb,
  36a2622ccb096155f068b57d4256b9a3ca9688c1ed724ff005f4c81f25897155,
  0421c22708f34e2ef1d4a6899df90e21a658752310c83c345e4de0488dc2240b,
  a926c4cb6ce26e86fc9f5302000c11d92513620cf75b56813fd401ce40934d56,
  b49139eee3a05eaa0293da49864f5612d9971c2524da815126a22aab7c2d7b32,
  d3fbddf581ab011def20ae226d832939603900c9b0ad6c4cca2f7be138393a12,
  "",
]

[rjl493456442]

Could you please try to run these commands?

geth --datadir <your-path>  db get 0x4105010e0800
geth --datadir <your-path>  db get 0x4105010e0801
geth --datadir <your-path>  db get 0x4105010e0802
geth --datadir <your-path>  db get 0x4105010e0803
geth --datadir <your-path>  db get 0x4105010e0804

...

geth --datadir <your-path>  db get 0x4105010e080f

These information might be useful for debugging.

[rjl493456442]

The account mutations between block 19860215 to block 19871213 with address-hash-prefix 51e8 are list as below

Account history:

        blockrange: [#19860215-#19871213]
#19860387
                51e8aa8987122e6ad07988ffd1f49a3bed49a50f01b4622da034f39888bb3511 - (address: a74e545535fe6e7dd8fa9a9af8ff109ba5a3e234): 0xca68861a341948e0608080
#19860400
                51e8aa8987122e6ad07988ffd1f49a3bed49a50f01b4622da034f39888bb3511 - (address: a74e545535fe6e7dd8fa9a9af8ff109ba5a3e234): 0xcb6887118d04a5179b438080
#19860411
                51e8aa8987122e6ad07988ffd1f49a3bed49a50f01b4622da034f39888bb3511 - (address: a74e545535fe6e7dd8fa9a9af8ff109ba5a3e234): 0xcb69870e389decb7133f8080
#19860447
                51e8aa8987122e6ad07988ffd1f49a3bed49a50f01b4622da034f39888bb3511 - (address: a74e545535fe6e7dd8fa9a9af8ff109ba5a3e234): 0xcb6a870cd82c294957438080
#19860804
                51e8b7a7684e5dbc3718035ab186a15fc072520f88a8e1c659523f3899ce750d - (address: 4716859f22b833ce53ad103f36a92e148814db9f): 0xcb808706ce978363c0008080
#19861121
                51e83e34f917dc22cf6f5a5761c2f3551d621bf269d01315dfea1c985de6aa32 - (address: 0aa747752680dd066475724ea55cba7e5c5baec8): 0xce8202ea880560d461819ad3fa8080
#19861320
                51e807bbd88ed7dd3950ea0fa8e009f7a9ff220e01fc7d99b02aafb82930bd54 - (address: 9afbcfe5d935041d2db3f77f6e48ab339ae934e2): 0xcc01880161d58be6a9b5888080
#19861349
                51e807bbd88ed7dd3950ea0fa8e009f7a9ff220e01fc7d99b02aafb82930bd54 - (address: 9afbcfe5d935041d2db3f77f6e48ab339ae934e2): 0xcd018901e71ad01bc955b5888080
#19861356
                51e807bbd88ed7dd3950ea0fa8e009f7a9ff220e01fc7d99b02aafb82930bd54 - (address: 9afbcfe5d935041d2db3f77f6e48ab339ae934e2): 0xcd028901e5b676c9de9a85488080
#19861362
                51e807bbd88ed7dd3950ea0fa8e009f7a9ff220e01fc7d99b02aafb82930bd54 - (address: 9afbcfe5d935041d2db3f77f6e48ab339ae934e2): 0xcd038901b685ecf2c8a080688080
#19861369
                51e807bbd88ed7dd3950ea0fa8e009f7a9ff220e01fc7d99b02aafb82930bd54 - (address: 9afbcfe5d935041d2db3f77f6e48ab339ae934e2): 0xcc04880dd81806bac828f88080
#19861734
                51e8bb1cf582e1aacf1a0a0499a593eeebbe404401f00b8efb8a11cd5b885699 - (address: 877eb9f56260a17ddab7f1304ca32f6ced930a94): 0xcb3f8708a8c33da67ec28080
#19861740
                51e8bb1cf582e1aacf1a0a0499a593eeebbe404401f00b8efb8a11cd5b885699 - (address: 877eb9f56260a17ddab7f1304ca32f6ced930a94): 0xcc3f882a0e48dffb0dd2af8080
#19861830
                51e81ad73e634e79f98ff569a4e22d74a8da362e26d83778e2d8503de48ef921 - (address: 148fe6416a6fde00d4799a7499c2683eb9f5c8ab): 0xcb5d873cfcbfb5b021028080
#19863164
                51e8fee3fec15e1bcb8164dbd58fa9a15674449a0e4b77cf8214e4dbb0fb2520 - (address: b4816804c473b649e68b2bda4a59614c9a4415f2): 0xcc80880429b413a3bbca708080

#19863822
                51e8a1bb4cccd30a7939261e6d719831f5014a7686676be3de4455e18f6825ed - (address: ebf3d9b72ae84ed38dc778860796e7e5c6d37b08):
#19863824
                51e8a1bb4cccd30a7939261e6d719831f5014a7686676be3de4455e18f6825ed - (address: ebf3d9b72ae84ed38dc778860796e7e5c6d37b08): 0xcb8087071afd498d00008080
#19864124
                51e8a1bb4cccd30a7939261e6d719831f5014a7686676be3de4455e18f6825ed - (address: ebf3d9b72ae84ed38dc778860796e7e5c6d37b08): 0xcb808711c37937e080008080
#19864577
                51e87ea85a46cd3e85db269d23210d7106682862927e6ab47e155f1c85b0cdf5 - (address: ae1db9ca12d35af5c07c82d57b915124a54e2f9c):
#19864578
                51e87ea85a46cd3e85db269d23210d7106682862927e6ab47e155f1c85b0cdf5 - (address: ae1db9ca12d35af5c07c82d57b915124a54e2f9c): 0xcb808748518f776cc9528080
#19864883
                51e839f74b08a7d521aff8be1074917569e7c77b1416083d6b56c1fef70b65c2 - (address: b0815f581a8f241287579f728e89f4cc6376c553): 0xc403808080
#19865911
                51e8b7a7684e5dbc3718035ab186a15fc072520f88a8e1c659523f3899ce750d - (address: 4716859f22b833ce53ad103f36a92e148814db9f): 0xcb0187068b4703416c308080
#19866021
                51e87975160609d41f1cbb3acab62f0dc9c6becb5623f5904296c038b87ec669 - (address: 6f944907c937e1f98b9c4836da9e7ae78304ac7b): 0xcb0d8730cfc2b96fb2768080
#19866037
                51e87975160609d41f1cbb3acab62f0dc9c6becb5623f5904296c038b87ec669 - (address: 6f944907c937e1f98b9c4836da9e7ae78304ac7b): 0xcc0d8801885be5f73772768080
#19866725
                51e807bbd88ed7dd3950ea0fa8e009f7a9ff220e01fc7d99b02aafb82930bd54 - (address: 9afbcfe5d935041d2db3f77f6e48ab339ae934e2): 0xca058640573a4300908080
#19866824
                51e807bbd88ed7dd3950ea0fa8e009f7a9ff220e01fc7d99b02aafb82930bd54 - (address: 9afbcfe5d935041d2db3f77f6e48ab339ae934e2): 0xcd058901e5b93ae71cef00908080
#19866834
                51e807bbd88ed7dd3950ea0fa8e009f7a9ff220e01fc7d99b02aafb82930bd54 - (address: 9afbcfe5d935041d2db3f77f6e48ab339ae934e2): 0xcd068901b5d77931f0a1b9b08080
#19867162
                51e8595815c03329e37ef8c4f1a0f8422030494883c9a76acb824b4760fff11a - (address: 6426335b0a04eefb5861061e19cb746b3a5e853b): 0xcb4f879902a8f47b14d08080
#19867215
                51e8595815c03329e37ef8c4f1a0f8422030494883c9a76acb824b4760fff11a - (address: 6426335b0a04eefb5861061e19cb746b3a5e853b): 0xcb508786d74f0be0d3188080
#19869238
                51e8c0be4b4bdb4d5f142c267672d8205902722cb01ee2e24ee0943875a87578 - (address: a4baa7b5dc8a4ef2c8e346f21ae641aee73a722a): 0xce8202598801c8a69a8b6a86398080
#19869508
                51e807bbd88ed7dd3950ea0fa8e009f7a9ff220e01fc7d99b02aafb82930bd54 - (address: 9afbcfe5d935041d2db3f77f6e48ab339ae934e2): 0xcc078806eca22dde38d1d08080
#19869647
                51e807bbd88ed7dd3950ea0fa8e009f7a9ff220e01fc7d99b02aafb82930bd54 - (address: 9afbcfe5d935041d2db3f77f6e48ab339ae934e2): 0xcd078901dec4e60a1980d1d08080
#19869659
                51e807bbd88ed7dd3950ea0fa8e009f7a9ff220e01fc7d99b02aafb82930bd54 - (address: 9afbcfe5d935041d2db3f77f6e48ab339ae934e2): 0xcd088901af94c9ec14e088508080
#19869662
                51e84e92425f28401c12041f04e006cd0940741b36655a6ac9ac5a12943e79ab - (address: 5b53753a703bf2fa0963443fb572caf299895b7a): 0xcb808744364c5bb000008080

[tompetrovic]

Yes I confirm that I've used v.1.14.3 to resync.

Here are the logs from the commands you posted above:

key 0x4105010e0800: 0xf90211a02cf81b0c6bf380b73f61ac54ee4dcef1c0a667b55b9f8255b1dd18c189fb5557a09ef54783c2291c779873aef790aa5aa489d1d2a0e34d5a3c6389d2be92c51f63a020f60ae328e6c14928807fcbf53776f3ad91ce7ea7dcc8ac39df1f47feb86109a0c6d50699a4a588cb6ee893963459d937a9fd4841d8ffabb084388741a3cd1233a0de740067eb683faba025512aabfb425e97fc7b838f67aa19800ee132232bb2bba0782edb3a009cddcb05cacb7a60f6a1bc674fb80419bdf8aeede039498fd82c86a0b8ef4334a8c79fb2f5d3f0cfe25b5c2b7fc812df1844420d6c36702ed2dfabc7a056ff62748a5d5094169c2337f35d288ca937c30c20cc33af54321f8cc813a2e6a0ade2854c6cb4e2019aa5f6800951cdd05589debec7188bfd974761b9eee8995ca096b6cd2cffddda74194d9e74fe565f99600c6dfb46ab22c26ffcb0725577bcd3a02c787332a9f0b0304abc618651aecf05d4f7e5254d4593c2b2c207dcf49e2950a0e2eda843c9cb8c9b7c329ed8d49ebbafdf02e5e82c2294e1a18ef420a4ec1f3aa008bb33795b7a878141dd3c9d221b2760dcac0d9ddaf0294fe4f03a854a935be6a0361722942ce38525455ddaa186e7da4fd65d15a9a29854b1449b0cf537663683a04d705e14abcf8340b92ae2e08d062c9ac044f036bd8ea6c37fc79c33085f7da5a074450d3d38a8c3e625bd4c25044035559fcf56f1e258ef3e25ac4bdeb51ceac480

key 0x4105010e0801: 0xf90211a073eb15bd4410ff60668bbad48675f81c3e4ec9e4bfc1348e5889e6d7bebf96f7a09d1ba82f6e811352226886bb8e8abe0fb0ce590bc32a5d19ee459b4a64c7dfc4a017206af24251509c0c84e10935c844aa2377a22d825c08bc4eb9a9948a1ada19a0baa7e8318e234ce01ab70b28590a070bd4dc6b39a779a78f4e64430f8b11fbd2a0f5cf014b0b5d1202348cc7da81f07382b7310896672b579dce1f19184e08942fa0bfdf45275c8ccceaf0435fda261041ca76734db0bb463180a0e018e95e2c8df3a051833ecd6a23a04005550bbb0db09ea854c5c8061ecc278efdbbe035dce2e979a0f15044a4d93ec2d4929ece4c9fcc272aa06c5a1768daaba9e11cc19385140bd9a0c458d5bc0ebc3ee75f915551ae7b3a5cb0e045133db7f2a1949a5d63ad6def33a04cbaa5afdd46023a76da92d42e76bdc8a20a89589c6b8804d6a21590d9c63e46a06e64d6a1514cd2c6f9c024a5ba19175246be6bd9515f96e0d1efbd5a36a4681ca08fc7b95becb472a1fd5789386d44e1f83cbba40409a036b8085dcca32a6426cfa0adb2eb23299fbf7832e8e7cc967d815baebb35c6c2fb094166a8d1f423f73950a090e2917c4647dbfc784252d2990206c2f917664416e3351755bfc0b28b2505eea0ccb89688f0896348e01e209d393f013d8db889e84205103dd5a0b9457d22a0f4a0ee167bd8d65ccf4f397210ee6d81c11cad9aadeb6a574918d8301ce5a9a35b5f80

key 0x4105010e0802: 0xf90211a0178ce1f1421dd26bfdbd626deacb0c87a3c22d76702a4b257c10b530e91e873ea0d61082925c50e5627ec97da58443efa85131d6c7d80e0860ad69bab48a499e43a04edc266b58bb8ebefeb3517c13e0d4eeae58193f16c07a56b0c39cb202ee61e2a0607dbc6dbc15b0b6a3cfae1029028956bd40d5ef0a9ab9444e71cd7c5baf96c1a0fceba5a3d9350f28b7079541c548d10d3a2a1565326b9f57709a7436bc6d6150a0eaf7044b31eaacec3102be590fd851e4235f664b422ba3dc009e0a94e328ceeea0b0fabf6a747f0ca3f4c52be0520545e8472d45486f79bf270ca04d06eb58b5b9a05ff392a392f92aa9303b45add07ebc5531f9c123224783b131aef46338f9d035a0adde9b723e4285159299d477cf52520798317970c64d56d2b6147e5506a34f99a0c59a7d199bc14f6c135b937dfc9b8a675a229f8e16275156415c3cc3124cff18a07795e27f3e3aef25a8e6dde316f537a1ab0e4bf5dd2c9a0b60c5d4be3646b6b1a068538fcc7999455b35f69bfdfaa113569fac8f4d4f317b445b6b931cbd8b580fa019303b21bf12418f00591a29fb368446f0c508d34282fbfce45b22921afe58c4a044b614e873c7cbf458bbc2bb8a50bfcd0565e2277f52d243d7abff552abe6130a006329da75d0b21fcbd64e3ad985ffad7d61ba8f01e85684a5cede1132dadf1dea093b33518a3914279b9c27514b24ec81f79fe13d482d72289227df41904af82fb80

key 0x4105010e0803: 0xf90211a0566a36edf79de9d497bf1f2a304d0e49dc674db1a67d57924bc0e1a2527986e3a083c2ebc27a058894e1580875db4c0c75f5786130c63dc0956e84ab1422f93631a000c3284a9af1b197c50bf31820955ebb8a91f40d3a4597218da92d4599198119a04b6984ab84e5b4d53cafbbb312e54c2bf4d7dfc470fa144b0c2a2e23b487afb2a06f52882a8e2627ac78f43c0162d2bdc807449652a9793b8538fb0d9782554194a06335efe1b4b676ed69851bf0c896fb9372f88187cb924cbd6ad8e6ab764ad909a0648232f6ed175782c1024b8c32eee0b8418e06d93ec56edc9476d98a770b9e13a03654b4913e361666fd263263b15e082ee8187ab91e02cca0d8ab710fe820aa5ea0380cec32d3f00c2ffbf9d016f36c6a8f50b82c55b81abbb6f8b1fc6dd39fdbc2a02f354ebfdf6cd3862047c3ff503e1cb6909c6742d45c0c5d4d24bcc992dfc224a01e59601f6b60a252c1c1f8396dc11cab5d5767db73b0c8f0153a6af2e85a671ba022a1153fbbe61d4b13d6eed388c04624a3b91ffe19e74584e82ac609caf9c746a063956e5a252176d28f892c839abf1df7faf9bb2dfd349abbc7c6311ea7954b5fa04b8d3889d4719f5e93b80ae741779201c48deff855223f60f5fa09b95687b6baa08dc99b4e5e01121eb2e1910693fadd25f7d3121c2c478b7c81cf631671487b82a0036ed57b2684ea354704d834d3a468bda4ec423dc1a5f699f05001a494525c5080

key 0x4105010e0804: 0xf90211a063dadd4fb6b51911036e47c279f1bdcdeffa219df48f344da3b4cdd52a90a6eca039f12083e1d470beddffab84dd7f38a0cc1cacb0bf43b7ba8375dbfb712f3bcda02c9c3c36f08ef5f978e0823d43505c58459a756268484adbe6f5075bcec22dc1a0180118c8a21eeeaaba64fd4620a1d0a82edcdc67ca430d8ef0e12b040d592094a02e5b4a79a6a32671d3526c544d43fb2823683e9b544e3e9bd743adc9f3f38046a0e710592badead0c95fb84aea7ece98754ea53c21cec1f01816941691fa209d18a0a8b3940075dfc1cab18cf999cae5e010a34560f32ae69c6d88a29183fab5811aa0dce33e34ac664068a7864f62839bee89307563d24640c601cf1b9c65aef51053a06016c93337330ba9a828a129db6051a7fbcaa54e760c6aee1178cd5dedf9db93a0ac24c7469478cd98aafd6b22fb81578a7678136247b040ec155fe443cb0cd585a0423c9f50131d6db49e76246cc563a9bfa61da0cef7fb73bc435267d35e1002e4a0166e0a06133fa6e4579834a7eb6272cfcd36b05392e951ee45942350e4bd414ca077824fed30e58461f002905aef063f92f5a5f3401e3a23d46d3df9096c708f9aa0e9fb5ab2b1592f953e0196acb4e7a2959f8f8ccf46989d6dbf20b66efded2296a029975de6190fc9f2d07427a189a93fadbdeaa143256ccabc876ad314b959894ca053637a12c5252490b9e46f355acb3e17be2b4bbe9f2fd6c06df658b2de9809f880

key 0x4105010e0805: 0xf90211a001d14f12030a40a677dc68a9d2dfbfb3e963d4c89548e95f3a869b9c6057fb82a067a504f166c415bb1ccba792d68204891f07573067b488c00a4e35f9aaca9e56a0782742841bbe36eb8c9e7da2627578ce9a1492b3030327f99fa01c842df46da8a01e51c9c560fedd1c87b72d6598b2ee255bdb85cd1988c4400aaa99e13fd58ffca042ff26929a216b98e042f970a66ecb05ab8804356ce708bf46cd533ba62a0931a0ae99437a98eb9089367cc7a2f3b99fbf8edf0ad73c10aece7666830d3450a605a0a4e70d85cf0865551b105e2542536730a92b11197c363728067e6d0497a2cedea0c1b13c774452eae31a1050eb0f3ad2a37d630637f9da2632fd8c0a2101ac80cca03895571685c6574b2dd38897ae4c0d404f22e4a2ccf94de82aef357e28f98b24a0d514774bcb361e42feb5bdca73e7b789415f3c5a49798fd4e689f57838519ed2a069042c7ae4f9b39cdfeb9d7d6feedfdd68f3a51cee7df9f720b1f0279f3c08dea0e11d9eac97af1d47888e4cc432f96d7a49190397efa0c4b18cb1dafa43a3e734a0dd73769fa724c4bedc72e599d854b2ea476dfa14599a040a5a9bebfdfea04ce7a08d419859de959f95cf14ab274c9aff561254c1d4fff44ff2aace5b19753bf92ca0458b927d440f220d97e5a094fb6a3bff1e96d3145b724ab202cf7acda743a874a0fb698e091e70a947e6d27f45f984650c271334d0468410a22ea8854c203aa7a780

key 0x4105010e0806: 0xf90211a010d2307e07e1385a0dcc2cb9588c2b0c9be1764708c4f23bf352f260d3ceb1d7a07f7f9c008032f1e2974c3bfe32595a448abc3450f80e1b0d39863d0ca7eec8d0a070a5a6de65043cb711eab6aa49acce4ecb968b14a7c5df24a818919df2f3b9e1a02bad3a947a582c7b7d61043dbd30b38dc8dc0c55754ea5d7e6309a5b44ba16e5a0d36b266d8f33a455f6907a8fd3e5674a04c0a92fa9ec0cddba4e1c0fb6231d55a0cb0dd61515da7acac4ab6ca0786eb230f02144c2df15948ea036cd23909170c7a0eb0c32ef90ac7c51ce1484781dd7d31c394e8111a958533447963cb6fde61673a0bd007e3365faa7e2da5bf0b89eadfd2a87a12513a5c719b77947a20c4befe302a05af0915b161a0bb0e8ec968e45a33f2a0919ae10d3e81611cbe389450c4380c3a08f9a9a00d980aa0be251b8eec9e33f245370a5b2faa1d4f45aacfebd4d23de03a008e05c13190b6f7ab21f78702fa1603503158c38f04d4cd14bd7f7ac43122415a0cdee230e5da702fe8c2f65c2fabafc5e53097faef8bc0c24a7e4ca6896b624f1a08226f575e60328fafa68fa7150e5cf25be6a193c3d923b1cebe87cd35318fe5ea0ec461e4f34d46cacb8cffe04836033aa5ff5869f065c999fe801d0cf531b89e9a0cd78ff3df702bd8613cc2d9caee88a9310d50cdc62fe16e1fbae6893ecabfec0a0e328dc1a0bfa98064ed15fdc0b5ab3ad25d82e0dddb05c0f93dfe454c57755b180

key 0x4105010e0807: 0xf90211a0fb9bab86e604b1750c66894abaead0be9f69dec3478060cb9cd629806b96fe79a0a417da3c56e34fe74a0137ea846ce428264c5fafb9f4af55e6ddc1d5a968ed33a01e8d60109ef136dc286591767195c27dc6d39f3a553411ae09a753250f58ceb1a057033473ed97f62c715252d55dce755cbc94634103623fbdeacdfd5bdfbb46caa0275acb5a57d37ae4a4935dc3d9283597c068e737e343db2c58b3e0899376fd99a0ac039f171c95854f03683561831494b9592071154c13450906bc212679c3c0dba0699b5c66adf3c659c41c89685c86dd41a15cb05288ca86367ddf8a8e4ed759c1a0d9e28bb6253bdb7a9bff9aa16ac97026f0c2efe149a1f29b304e3caf99ad067da0362a67ad49b176ec7234465f80593da3ee9a99090dddf3ac4b3d2b581ce4d080a0efaa21b6a09faa2e6d2559e090dabb96999d3beb0ee312026d6141530038112ca004fc826ca36aaef8ebfa7d0de9605e895240af32f760953edbbeb8e90f1045bfa0928a275591d136fe20f00c58c53a8c1ab593ea7bffa97e682ec53fa937bbc0a5a0d6b7953015b1424a1bb4734ac156e426499a637267b0060f8b46044643aa1001a010bbdd5eff7da4550f5c3402dd0858f8e096ad369bf0b60b4c74e26a02fdd3afa0db280972be921e90391ba0f31df8934521414a9ee12a8233d0f2c499785c752aa0e0c67d84ab43a831d83e7fad98888805dd4fa221d14bbed60555ef59680b9cf380

key 0x4105010e0808: 0xf90211a06d0ed3a5894f412c3fd024c2da881f7ef6a0e348b061bba6cadff1fca80d63cba0da856abeec4dd883d4c091a4c0bc9f057d27784bc906386846db69598e8f0423a0e524bcf219273c0418f36511ebd12d668779ff69a7d221314b8e43a664bb3bc2a09e74568f8e2a0ba37c674067fa56465e95d39aebbc48da7854d5da06cb85e199a0ed65950b29b2b786cb7ab048be8b110a513fef52ded4fb460965c4696761ea31a08d1979bad20d1c2f16826d49464e6f371a77c45908c469bfb0eddb308f43071aa027fec2817babbefc33a4e4c6ea3416076880899ad2e4f5c35a4255a54cdc97eaa09c08670b037a83fcd877f1fb5fd4ff3ad23fc13bf78aa610083bd3bb09ce22fba09fef526ad26ec9cc41148445e6f9b17469a76190e2c7bdba5fdbc407baf24774a077d405581bdc215d6ad34bc0262d5da9d484abccfe804247e2de08c0a3d98dd2a03b965ee7537be8a5a8943017d4ad4034588f5df1af7fdc3e42d815b3be05fdf2a0e483e95ea1d6a1494c0a87c9c097ba099505f46aba2788540254f4cf0dd697a6a09fae6206ef526c4ade7032de22764fe6c0ca76251641c172aa9c03564a3e5595a04de94bc5107ca32b90a912d0d4472073b1fdea0d94c1eefc8e9fc918af2edeb6a0f53f3ebbb161c2be1894a62587170656570b4b9f29bb3879d5392698833a4a19a0ea989cb8188542c802c56fcf46209e9f32aef42e6abccec10d15ca704cfaecdd80

key 0x4105010e0809: 0xf90211a0025a6458703864071871e05acab56729b32b5aa4ca9bb043ff1500bb805c967fa02272d18ea70bb4bdce168a9dfc9aa528860d9f64578c3aa9e6804d1ec56d4989a001f22934e9ac2fb9021d076dea068e6e9f007e8f196fffae2097e86fb1f11eb7a098806d6e54be1dcab8c7cb0cc23d1456a524a8edff513e43973d79d263a1932ba01297a07bcc0fa6f1454d6ab2fc3e31acba08cad1d73838b6791d85bf42a3e847a0e4858bcf66953d736e3a197ced26a6ca441964d409486a5b4dbcb64a23897949a00fef3ccbefc816f0eb9406e7de4a1fc543eeb98203c0c1ac6fde153ca96cd08ca03e3b241cddb5bdad61c7e708fac0ea372c3ce646f146d4772b2a840dfb719e9da013670b75307fe82d6a0bce389153f88fecb597fdbd0fefaece63bd3fd02bf214a08acd8acf76cd5e73aff8c3b9a3a20d4867685a3d54adc85b23764af676bceb25a070a8311ec31aa36ab5e6a09468f97ea9adc269f1085bb613057b5c7a3d4be0f9a0defbb0a142bddc26b68399f8053e45478cbae873d1544628b9d49f81acb30873a02d9af25dc6ad52df173e76b3437ff637654a3c1f23cadefdea05690d7c28593aa0938ffc5443d09cf8630479f8d3518bc63892996d0fb8a4950333913edf38cad6a0e69fea159464f4aebf8bad0c10b40eeb1bc3b9a1a38aa784e1fdb79fdeeeb7c3a0ef7b06a5e06aa2785485e70231d7f0b810efe1c9666ca1c07420b5b9035f0c1580

key 0x4105010e080a: 0xf90211a0f6d0415a95ff887ba746cc8b1d89734b8a6ba08e56074924ac04bc7b44690b6ba093dd7f689d4c76f4fd411e024838b9dcbdc801fc7579b2906941da84de729032a089ed8a741457ea9f960fe8ae6e6043241a3328d65f4ce640525159c3c626694ea0a881c465b2b14f8d9bedb70f0bc695440bb4f91a97a4a0144ec3dec7d6e84b11a0a88db8a38f3f86768be8a9be12c591d893640e5822b55d06e8d631499a12f3dea0b8185ee47827ab4dff67e2e1fe3f916869a754f3b1cafb1448243cbcbf4a19f0a05de8464870f45a4f5b150b861f3fe41f98d8083b8169be15c3bbecc9a6b48e74a08c48df490adb9679f35dd9ade25845914bb33879f8f61e6a58324474087be2dda0c0e227c231794355156beb422be6a1cbbce549c8d3547adcc4360212a53bd76ca0186a20f46f943e1509c7421b5f8092d21e3468913d233ff8ee04fd0624fac787a0b4e7331bba8b0236fcde1053ec814c0793ebff7d8692ad68b16d7cc48794e0a3a04899152d9dbc3f5e28995e30f2fc740e59579b622ce8ccdcd8961878cc1cb4f5a0c4d50b5ab0314a49d30e5d9eaf9d28438d0ec93571d373ab63f875ce53ff9737a0c2c7a6d9bce8eb67cef1daff16c05a5e6bcfdb1142a0d9439fa53c8eed7cc428a0de78b4eb887e22b6c3e57ffcf87ce6ecedd6a17efa81cd093864f537181372e4a0dc0098d2c2d8161ccd4f1e7a503cd3c9d5c1f29da77c4f620480afc7badf891c80

key 0x4105010e080b: 0xf90211a044884e6b28b43893d1d41c58c8eca8b61c9cbbb85b31c6bd363489b37260a7a4a084e4cc7bdfea380c8759565cebe0989389480ad6042a3b5c6084c715084f9216a02e04907be3d731ea99d53fdf0f322bb37997a4b0e6d7a73631704ecf9343d269a00d9411c411e4276203f60c9bbde0266b18122db5b30003dd4ebae00d21516bb9a02b50c1f1b780d2aafc837abec5df7c8e5869f76cb8eab51808f95d27f20bde18a0cc55f6e5746d0636365e13c2c111e4c60d8c660ce2576a1fde984f861e3666eba0cefbc77b8ed184e3833386036bd201e2c7701df0dac3485a0cebd206ded091c8a0981409443647755351e9d54e9bcd40dd656b9724a7ce138f3c70099674a2f140a053a656ed13da1d0a89783aed7a8782666b14713a4d9c99e62a1263012dabb4e9a00b0093039ab39b89bf18e43389212aa2545a536cbddf6b573fdb7e44a2c5173fa0e85883d6c4b949e32f5da893e5ed851ef53a527f2cb157a06d7a56f323938d1ea0ade9460e614e14e84a1f7bb6b55900f5f0546822a5fd75f551866932d6b66ebca0ab49d4beb3833b71fe0751aa2d3e05375f50c140879e492701bcdab12a4ce23ea04a5031c4cb345b49d81e4b6be49c632a0c8ab149b0022865b45b532a2f6a2529a0fda02c9c4d4c3fbac30a38a1df6f68fc1e213dcef14421c1db26bdc728dd642ba09e66977b9b5f86c348386f5826cadde1a098695317367aad5aa788d61b31634380

key 0x4105010e080c: 0xf90211a0f6954c8df09a8e868b3052236e2df9077b629212475851cf375c7c6ac45e2f4aa0e2b4056affeec4a66b9f621693c65a349d5748022b675cefa2fa3973617e276ca095c0419282f172ba517c491b8c0b0f8e6fc0af0fbdb3a1b93144ba5d2f9d352fa0770904baccbef2de05d5ef2d8f00e7f0378024c0f08c2e62b5ca5d39136f94f8a02c3a93e58e27be3e1f356f763731ed6bafafdb61190a4cd3bc902e5c514d1ad3a0e6bc66a1d4009ed8dd217b1b9a6bd3a64862db7480dc893405f35321570c1fd7a08a88f8b5a265b44e96bee5527ef34baa79c60a133ff7ea64b9ac4bf78c24b227a062cf37e2a40a24ded2fdd4b8442e8bd625e781e98c68ce879836f7089b73c1bea02cd8e6415e1d1564b9585dd5455695afe4f705f1f39ef809ccae7cde0694ea9da0d77fd4bedc4b5f64f8abcbd08c4cac6ce21dee1c230fd1b8b45fba2edfab2389a01f4efafa724531d9b4559055c1a1474185bc90b18c52f77d3024fb0f506d3d0fa08c61d5505ce2e82954853e6a8720f53f4674b3ac23ad1f7ca0ff472bf1919f49a07262311fd62503b2e33f27a42043243765b0683e645c6d7d8e7aa26461a0ac75a00afc216676ea7567466f7fca2840e5a5bc8a72511f7656e499140082ab4ea881a0241f0cdcaef07ee1c41f9b9363be2d75c16fa35f863dca27a1f094cd7c791957a091d5623c739eecb39e27aa2eb1b85530e5ad63670172ac304344d319e0bd540580

key 0x4105010e080d: 0xf90211a07a41e9e379c5789dffee52f636ae5d552505a50418679f9392c57dda07417fe0a06761559d723d11cbc31d89043af78ed30d106bd75d699dc23b018d654737ee49a0fe015c752808fd7c3079f7a4920eca3f554d5e138a534fb937bf70457197da1ba0c495f3f70214ed20cd2b238f2293e30027e6e71879c1dcae2b0f7890044d84baa06321bb3241a90d616c74f4aa0770e4ab6ea1bbe22060099edf8a31111c251162a0fe2d99352a5265825df3e3e51b94f2668d6abd294801196ace0b0942bce4fcfda0782f7c1f950b0d24cf0c1d9d9688c977ec70d7e0a8ea4078df2ba48ce965c3f1a08cff6847374ca451952b75f94ba2c0414efb5493cc2c4a9bfa5280d7842a7e4ea05ea1c41de623e82761a6a26307fefddb587b02a52e55c59091ec0796ba9ce045a0082d847ba5206d87cd1d8bf2132c5474ac947dfe1d27429fed05a5fad2595bcea0c14c02ef0fbfb71b9b3a50a9d16ca83808850b35f218582588764a22d731aafda098a3ece0376373bab05178c20c1251cdcfcdcc647c967e40d110b6650463561aa01d2e4701305d1777ddc3291fb89a75afc451ce900a9ff22ff334c4653e189e28a0f81a2a5321ae966d54ea4c68aa9c63b928ad3fd7336c92163c003e92b6c63adca0ac9916ffb01b0417c6a90c815de098fd9330cc778f412ddd560058f6760478e3a00a14c62fde916c4dc59de1526b74d724c2995db6cb3a8c30fea3daf24f9f7b3480

key 0x4105010e080e: 0xf90211a0d6988b11b4324eb2a8159a23a5a71d68e1ccf902d3141925174e5be8c7fab048a0ea3e1d46f1054d591db076ab17270c4ab8d55e6240db38f9639a4d09ff58bdbaa048b23f96b3c6138df507137c67a5971786daf1f9a26ff213ad0639947f1efa45a02e8e4e83edc621e9f29d3e250a1e57e58e53111ef8b18e7ca86e2fa25e964a69a0f5e17d96ad5dc2c53e83f95f1a5144322a3f050cde32d5043a14b77caedd2301a056c83f5bd3e77114cf0c071a38984039a426908dad5c075a32c1e189c04857e8a0bd72ca8b0da49366110049819c16fee09a96630b12f852d5fb3c6c09b57a6e77a027fe06f17e57e19ab5d117cf2e6079710697ddde90ca85482bff9e7351f4ade5a0261920a5c41cb60bd7f89d7e94a4a870ad3dbb60f84a2cf2b12ef89de7839233a05d8f1fbdbbd490d4550114694a616c8aecb6cbdf7cad456f0f5cecb17b085e83a0cb21d751329a11f584effbb3da5093210bce05c3fe354b3163b0c636faf1811ba070da411de4b9cd2bbb548f461b33a33e2b8d5d2576cc49885143bd694eb62960a0bba1f8471137009e2c598e4043f8f876e47473a8674fec103c15f7e42b6408aea0bff0057be60693193865765ba483c6abad4f2c5e1fcb6b4f338af74cc2e1267ea0359f08f04adee19e219fea8438d1c34e256055e8d5f4d74070750f9356df2ec2a0f2e7140ab9f8fa672994c4ef2fbdf5e48dbadc34af8d78850110feb6c1e9b2bf80

key 0x4105010e080f: 0xf90211a0b3b243b1816dfc78a0215840ae09b8f44aa9a264cbbbc23064a65f481f2c780da0713f207285ee8a25fecf5122b4b076356d3ade7e0c7719b2fb6061b6dad34cd8a0c72b0415b69d879f7ad00ddaf3506bb374bb90ba643fcac08cc3d9a2dc13ea26a0ef1022370e4ecc0fabde0a67ed6c542c74b5272e7298241d913a9a85414fe6aea0f22918da70f1a34eaa5b6e10991c56985fa9cc54ab7bfed94e40e806672f70a9a016176be65dffc023410c0edb59e478a4aa5e3236219ab79ae2f3632ec05813f1a099fa111bd1e8377b3f10d4ae6101685bbba9b221074a9e582cd554dc1eef09e4a0a2cd9f1fe5c66f721140e6f75f27dc4e6e2448d6f3ca5ecfe15d5e748a959c1ca044067b781a07f3727613d9604ae06ed5ac913738a1f2f7f90295958ad56de134a08473c84b6c1db3fce4966027cf0aae82747ee50157a6edfa7998879ef10c03faa06f0b949a0548303f572f0a5d0896b84b71fe804efcf0d386d650189d8b18c40da0243c7e0453ac61eed40415bfcbae7cea1cd66d1bf6a15c30ea6b827106303abaa04c13de2c2576b6ff7db878706a4be102706f0d1ce33c9feab2bb6b19da4c5abca0bb947d00969f83cd0bf2c8ebc2de1d83b1fb88127f539e8237942ab13d93cc94a0d3aac300d4cfe89d3f9f93034ce942224276805ee5f0a2b052960f10b2f37ddfa08d9e605788542b7ecfd814d2f5c451fd9883ede1ebcaf638b4cb6b273470d33380

[rjl493456442]

Thanks, i have verified all the children are hash-matched. Namely only this particular fullNode is missing.

do you have the logs when you resync your geth by any chance?

[rjl493456442]

geth db put --datadir <your-path> 0x4105010e08 0xf90211a0cedb62eb6fe837c799ed182fb39585c46848cf189414427222ae23fca3b02226a0e22c7cc42c5fc438a01a8872cde5cd39f78e3ad6f9117100772de7454ebabebaa0f8eb3c8bb6bb557782c511353d8b8cd64364d0180a1c7e448726a5a927720f6ca0d965da52d4522432e80a835b3a67ee8b4f12f4f3e430365e57c82b25cf475053a0c09c3f76ae9fad65807b71ef8803ac409cfa9099a712d96db37f85adeb408de5a01d8728e8b9397db70b5043716aeee4bd6deea4d28cd29ff460a8bf12a66d9330a02c5157537434575edf3eadf4a13e127e215f85f9f685e6af9963493b40bca307a0f03d034d832b8e89c33fa5d164b64400edfc6cd16977f3d0d0d487cb664ce90fa0d12cb13e4e3dfdcaef7f261df8d1d668655bacd16296a023b51db3896e3f88aca0bb378bc60d27d36c8239e85f2abcdfa4cd4cead2e05735fa7c48522719ee59e6a070c7c4b49f2f3eac127edaf6ac474c47f17c768e43d359306fd7e6687509e7bba036a2622ccb096155f068b57d4256b9a3ca9688c1ed724ff005f4c81f25897155a00421c22708f34e2ef1d4a6899df90e21a658752310c83c345e4de0488dc2240ba0a926c4cb6ce26e86fc9f5302000c11d92513620cf75b56813fd401ce40934d56a0b49139eee3a05eaa0293da49864f5612d9971c2524da815126a22aab7c2d7b32a0d3fbddf581ab011def20ae226d832939603900c9b0ad6c4cca2f7be138393a1280

Btw you can try this command to fix your corrupted database.