Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Does riakcs upload part api support multi thread request? [JIRA: RCS-361] #1307

Open
yahuikk opened this issue Mar 25, 2016 · 1 comment
Open

Comments

@yahuikk
Copy link

yahuikk commented Mar 25, 2016

I put a 5.8G file to riakcs with multipart upload api of aws-java-sdk 1.10.37, and riakcs version is 2.0.0. All parts of file were put to riakcs by a thread pool and sent upload-part request separately to riakcs with different thread. All
these part had exclusive part number in request. The issue is I received same part number in some of these responses.Since the part number is not unique, it got failed int completing multi part upload process.

Does riakcs upload part api support muti thread request ? or the way I use these api is wrong?
Respect your reply, thank you.

Some log about these issue:

[ pool-1-thread-4 ] 15:59:46.981 riakcs request partNumber: 4, partSize: 17825792, uploadId: QWwVyCgUS-e-lvmw_r75Xw==, isLastPart: false
[ pool-1-thread-1 ] 15:59:46.981 riakcs request partNumber: 1, partSize: 17825792, uploadId: QWwVyCgUS-e-lvmw_r75Xw==, isLastPart: false
[ pool-1-thread-5 ] 15:59:46.981 riakcs request partNumber: 5, partSize: 17825792, uploadId: QWwVyCgUS-e-lvmw_r75Xw==, isLastPart: false
[ pool-1-thread-3 ] 15:59:46.981 riakcs request partNumber: 3, partSize: 17825792, uploadId: QWwVyCgUS-e-lvmw_r75Xw==, isLastPart: false
[ pool-1-thread-7 ] 15:59:46.982 riakcs request partNumber: 7, partSize: 17825792, uploadId: QWwVyCgUS-e-lvmw_r75Xw==, isLastPart: false
[ pool-1-thread-6 ] 15:59:46.982 riakcs request partNumber: 6, partSize: 17825792, uploadId: QWwVyCgUS-e-lvmw_r75Xw==, isLastPart: false
[ pool-1-thread-8 ] 15:59:46.982 riakcs request partNumber: 8, partSize: 17825792, uploadId: QWwVyCgUS-e-lvmw_r75Xw==, isLastPart: false
[ pool-1-thread-10] 15:59:46.983 riakcs request partNumber: 10, partSize: 17825792, uploadId: QWwVyCgUS-e-lvmw_r75Xw==, isLastPart: false
[ pool-1-thread-1 ] 15:59:51.549 ----- response partNumber: 10, and Etag: 52e9c02f132841aec553e20519d37c84
[ pool-1-thread-7 ] 15:59:52.005 ----- response partNumber: 10, and Etag: 0b7e83ccf9647f0be0fef333f15fd770
[ pool-1-thread-6 ] 15:59:53.438 ----- response partNumber: 10, and Etag: 5ea67032c3081fb88c2f944f1f511573
[ pool-1-thread-5 ] 15:59:53.533 ----- response partNumber: 10, and Etag: b48ca450a526492d940f19ce971d3222
[ pool-1-thread-4 ] 15:59:54.027 ----- response partNumber: 10, and Etag: 03e24720932f4004160f663a38a897ef
[ pool-1-thread-3 ] 15:59:54.093 ----- response partNumber: 10, and Etag: 79fae66a49a0a295c0b78382c842c1da
[ pool-1-thread-8 ] 15:59:54.189 ----- response partNumber: 10, and Etag: 00a51584da889d1cdf6ca8dbd949a60e
[ pool-1-thread-10] 15:59:55.110 ----- response partNumber: 10, and Etag: a3675a8e5264d846e28003d71b015bb0

@Basho-JIRA Basho-JIRA changed the title Does riakcs upload part api support multi thread request? Does riakcs upload part api support multi thread request? [JIRA: RCS-361] Mar 25, 2016
@jadeallenx
Copy link

RiakCS code expects multipart upload part IDs to be globally unique. When part IDs are re-used by some other thread only the very last MD5/Etag value is stored by RiakCS so unless there is a way to force the part IDs to be unique across all threads doing upload operations, this approach won't work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants