You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
From #1909, it looks like this is a "normal" behavior, as the oioproxy doesn't check at all the integrity of a passed JSON when performing a content/create call.
However when rebuilding we get the following error:
2022-06-08 14:41:53.336 194635 7FD2E20B9370 log ERROR ERROR while rebuilding chunk OPENIO|A5B2970E8FBD895C486A853C2D3848AAF2835548AA1BEF2B251181FB07213EEE|DBB63ACFB7DA0500F2420539E536E998|6E561DA710A5027CA104712707BCEF7398A0C41939744791852B992D0A29AAF5: No spare chunk: found only 0 services matching the criteria (pool=EC573SITE): too many locations already known (12), maximum 12 locations for this storage policy
Which is understandable, as the duplicate chunks count towards valid spare chunk candidates.
STEPS TO REPRODUCE
Somehow create a chunk with a duplicate position. You can forge a chunk list with a duplicate position, and send the JSON to a content/create call to the oioproxy when doing so. Just make sure that the object locate command returns an output as described above.
# Target the chunk to be rebuilt:echo"A5B2970E8FBD895C486A853C2D3848AAF2835548AA1BEF2B251181FB07213EEE|DBB63ACFB7DA0500F2420539E536E998|6E561DA710A5027CA104712707BCEF7398A0C41939744791852B992D0A29AAF5"> /tmp/to_rebuild
oio-blob-rebuilder OPENIO --input-file /tmp/to_rebuild
EXPECTED RESULTS
Successful rebuilding of the chunk
ACTUAL RESULTS
022-06-08 14:41:02.204 194586 7F5FDB2BE4B0 log INFO Failed to find spare chunk (attempt 1/3): found only 0 services matching the criteria (pool=EC573SITE): too many locations already known (12), maximum 12 locations for this storage policy (HTTP 400) (STATUS 400)
2022-06-08 14:41:02.206 194586 7F5FDB2BE4B0 log INFO Failed to find spare chunk (attempt 2/3): found only 0 services matching the criteria (pool=EC573SITE): too many locations already known (12), maximum 12 locations for this storage policy (HTTP 400) (STATUS 400)
2022-06-08 14:41:02.208 194586 7F5FDB2BE4B0 log INFO Failed to find spare chunk (attempt 3/3): found only 0 services matching the criteria (pool=EC573SITE): too many locations already known (12), maximum 12 locations for this storage policy (HTTP 400) (STATUS 400)
2022-06-08 14:41:02.208 194586 7F5FDE38C370 log ERROR ERROR while rebuilding chunk OPENIO|A5B2970E8FBD895C486A853C2D3848AAF2835548AA1BEF2B251181FB07213EEE|DBB63ACFB7DA0500F2420539E536E998|6E561DA710A5027CA104712707BCEF7398A0C41939744791852B992D0A29AAF5: No spare chunk: found only 0 services matching the criteria (pool=EC573SITE): too many locations already known (12), maximum 12 locations for this storage policy
A partial fix would consist in removing duplicate chunk positions before feeding the list to the _get_spare_chunk function (needs to be done for EC and repli):
--- /usr/lib/python2.7/dist-packages/oio/content/ec_old.py 2022-06-08 14:57:10.918012798 +0000
+++ /usr/lib/python2.7/dist-packages/oio/content/ec.py 2022-06-08 14:56:40.222204985 +0000
@@ -58,9 +58,12 @@
# Find a spare chunk address
broken_list = list()
+ used = set()
+ candidates = [c for c in chunks.all() if c.pos not in used and c.pos != current_chunk.pos and (used.add(c.pos) or True)]
+
if not allow_same_rawx and chunk_id is not None:
broken_list.append(current_chunk)
- spare_url, _quals = self._get_spare_chunk(chunks.all(), broken_list)
+ spare_url, _quals = self._get_spare_chunk(candidates, broken_list)
new_chunk = Chunk({'pos': current_chunk.pos, 'url': spare_url[0]})
# Regenerate the lost chunk's data, from existing chunks
Warning however, as this sometimes will generate errors as such:
2022-06-08 14:44:52.500 194742 7FE3C2B4F370 log ERROR ERROR while rebuilding chunk OPENIO|43111ECD2732E20A123114C34E1F8E740ADA5D8256CF27A3AB77213FFFFEB678|B875B33177DA05000CB194A6F73A6CA7|01E9E28CBD0BB4E0DA1A408F3AD246D44CECD1915F6D52D496370BF23874B029: pyeclib_c_reconstruct ERROR: Insufficient number of fragments. Please inspect syslog for liberasurecode error report.
Depending on what chunks have been selected for rebuild. We think that duplicate chunk positions also lead to the same fragment being uploaded in 2 different positions, which is in itself another issue.
The text was updated successfully, but these errors were encountered:
ISSUE TYPE
COMPONENT NAME
oio-blob-rebuilder
SDS VERSION
CONFIGURATION
OS / ENVIRONMENT
SUMMARY
When locating some objects, we can see that some of them have a duplicate position:
From #1909, it looks like this is a "normal" behavior, as the oioproxy doesn't check at all the integrity of a passed JSON when performing a content/create call.
However when rebuilding we get the following error:
Which is understandable, as the duplicate chunks count towards valid spare chunk candidates.
STEPS TO REPRODUCE
Somehow create a chunk with a duplicate position. You can forge a chunk list with a duplicate position, and send the JSON to a content/create call to the oioproxy when doing so. Just make sure that the object locate command returns an output as described above.
EXPECTED RESULTS
Successful rebuilding of the chunk
ACTUAL RESULTS
A partial fix would consist in removing duplicate chunk positions before feeding the list to the _get_spare_chunk function (needs to be done for EC and repli):
Warning however, as this sometimes will generate errors as such:
Depending on what chunks have been selected for rebuild. We think that duplicate chunk positions also lead to the same fragment being uploaded in 2 different positions, which is in itself another issue.
The text was updated successfully, but these errors were encountered: