Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[PR1] allow parallel lookups #1866

Merged
merged 1 commit into from
May 20, 2024
Merged

[PR1] allow parallel lookups #1866

merged 1 commit into from
May 20, 2024

Conversation

sethiay
Copy link
Collaborator

@sethiay sethiay commented Apr 26, 2024

Description

Allow parallel lookups by
(a) passing FUSE_PARALLEL_DIROPS option which allows parallel dirops (readdir and lookup calls) from FUSE driver. In my experiments i noticed that with this flag, readdir becomes parallel for same directory and lookup becomes parallel for files under same mount. Also, the even without this option, the lookup calls are parallel on repeat lookups of same files. GCSFuse takes locks on dir inode during readdir operations hence with this option. for user only the lookups are parallelized.
(b) take readonly lock on dir inode instead of exclusive lock while doing lookup.

Link to the issue in case of a bug fix.

NA

Testing details

  1. Manual - NA
  2. Unit tests - Added in [PR2] Composite tests for Parallel dirops test. #1906
  3. Integration tests - Run1, Run2, Run3, Run 4, Run 5 (More e2e tests to be added in next PR: [PR3] E2E tests for parallel dirops #1907)
  4. Perf test (Run 4)
    image

@sethiay sethiay added execute-perf-test Execute performance test in PR execute-integration-tests Run only integration tests labels Apr 26, 2024
@sethiay sethiay force-pushed the fix_inode_lock branch 2 times, most recently from 135b862 to cad3614 Compare April 26, 2024 18:50
@sethiay sethiay force-pushed the fix_inode_lock branch 2 times, most recently from 21f3b41 to 91d449f Compare May 13, 2024 10:50
@sethiay sethiay changed the title Changes to allow parallel dirops and lookups [PR1] allow parallel dirops and lookups May 13, 2024
@sethiay sethiay changed the title [PR1] allow parallel dirops and lookups [PR1] allow parallel lookups May 14, 2024
@sethiay sethiay marked this pull request as ready for review May 14, 2024 12:24
@sethiay sethiay requested a review from a team as a code owner May 14, 2024 12:24
raj-prince
raj-prince previously approved these changes May 16, 2024
@sethiay sethiay removed execute-perf-test Execute performance test in PR execute-integration-tests Run only integration tests labels May 16, 2024
@sethiay sethiay force-pushed the fix_inode_lock branch 2 times, most recently from fcfeec1 to 9a208dd Compare May 17, 2024 13:24
Copy link

codecov bot commented May 17, 2024

Codecov Report

Attention: Patch coverage is 18.75000% with 13 lines in your changes are missing coverage. Please review.

Project coverage is 61.40%. Comparing base (2fc919d) to head (9d7ba80).

Files Patch % Lines
internal/fs/fs.go 33.33% 2 Missing and 2 partials ⚠️
internal/fs/inode/base_dir.go 0.00% 4 Missing ⚠️
internal/fs/inode/dir.go 20.00% 4 Missing ⚠️
mount.go 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1866      +/-   ##
==========================================
- Coverage   61.44%   61.40%   -0.04%     
==========================================
  Files         129      129              
  Lines       12298    12311      +13     
==========================================
+ Hits         7556     7560       +4     
- Misses       4403     4410       +7     
- Partials      339      341       +2     
Flag Coverage Δ
unittests 61.40% <18.75%> (-0.04%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@raj-prince raj-prince self-requested a review May 20, 2024 09:26
@sethiay sethiay merged commit 820d793 into master May 20, 2024
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
execute-integration-tests Run only integration tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants