Skip unsupported names in listed prefixes #1876

gargnitingoogle · 2024-04-29T11:39:31Z

Description

Skip unsupported directory names in list prefixes. Warn on unsupported object names returned on listing.

GCS supports objects of path-type <bucket>/<path1>//<path2>
and treats it different from <bucket>/<path1>/<path2>, while they are both the same in linux filesystem.
GCSFuse being a POSIX-compliant file-system only support the latters and throws error on former. This error can disallow the listing of all directories in the parent directory i.e. <path1>.
The current change ignores the listing of prefixes (directory names) which are empty such
as "/" above to ignore the error and logs the above event as a warning.

Similarly, GCS supports <bucket>/<path1>/./<path2> and <bucket>/<path1>/../<path2> but these have special reserved meaning in linux filesystem. So, ignoring these two prefixes as well.

Similarly, linux filesystem does not support '\0' in names of files or directories while it
is supported in GCS object names. So, ignoring those as well in the GCS list call output.

Link to the issue in case of a bug fix.

NA

Testing details

Manual -

GCS bucket structure: 1. <bucket>//hello.txt, 2. <bucket>/a/hello.txt
GCSFuse mount command: gcsfuse --implicit-dirs --log-file=$logfile --debug_fuse --log-format=text $bucket $mountpath

Without the fix,

ls $mountpath
ls: reading directory $mountpath: Input/output error
cat $logfile
- No warning/error log -

With the fix,

    ls $mountpath
    a
    cat $logfile
    ...
    time=... severity=WARNING message="Ignoring unsupported prefix \"/\""
    ...

Unit tests - NA
Integration tests - NA

internal/storage/bucket_handle.go

internal/util/util_test.go

gargnitingoogle · 2024-05-06T11:52:01Z

Presubmit tests passed: logs

ashmeenkaur

Still need to take a look at the tests

ashmeenkaur · 2024-05-08T06:57:16Z

internal/util/util.go

+)
+
+var (
+	UnsupportedObjectNameSubstrings  = []string{"//", "/./", "/../", "\000"}


Nit: I think we can make these vars un-exported. Also are we sure that "\0" always shows up as "\000" in object names?

On my linux machine, I am able to create files/directories with \000 or \0 substring. Is this expected?
Ex: touch \\0

internal/util/util.go

ashmeenkaur · 2024-05-08T07:32:29Z

internal/storage/bucket_handle.go

+				// a/./ as a/, a//b/ as a/b/. 'a/\00/b' is not a valid substring file/directory
+				// in any unix file/directory name. GCSFuse simulates the same behvaiour
+				// by ignoring the GCS objects which have these specially reserved/unsupported unix names/substrings.
+				logger.Warnf("Ignoring unsupported object-prefix (implicit-directory): \"%s\"", attrs.Prefix)


Are we sure that explicit directories cannot be created with these names? I believe \0 is a valid explicit directory on GCS.
If so, we might want to remove implicit-directory keyword from logs

ashmeenkaur · 2024-05-08T09:01:37Z

internal/util/util.go

+
+var (
+	UnsupportedObjectNameSubstrings  = []string{"//", "/./", "/../", "\000"}
+	UnsupportedDirectoryNamePrefixes = []string{"/", "./", "../"}


Shouldn't \0 also be here?

ashmeenkaur · 2024-05-08T09:04:13Z

internal/fs/implicit_dirs_test.go

+	var err error
+
+	// Set up contents.
+	AssertEq(


Let's use stretchr/testify package instead of "github.com/jacobsa/oglematchers" and "github.com/jacobsa/ogletest"

ashmeenkaur · 2024-05-08T09:08:52Z

internal/fs/implicit_dirs_test.go

+
+	// ReadDirPicky on mountdir/foo should fail as the unsupported sub-directories in that can not be read.
+	_, err = fusetesting.ReadDirPicky(path.Join(mntDir, "foo"))
+	AssertNe(nil, err)


Isn't readDir supposed to silently ignore unsupported directories?

Isn't readDir supposed to silently ignore unsupported directories?

ReadDir is, but ReadDirPicky is't.

Can we add another test with WalkDirPath and validate that all the supported directories are present?

GCS supports objects of path-type e.g. <bucket>/<path1>//<path2> and treats it different from <bucket>/<path1>/<path2>. GCSFuse being a POSIX-compliant file-system can only support the latter and throws error on former. This error can disallow the listing of all directories in the parent directory i.e. <path1>. The current change ignores the listing of prefixes (directory names) which are empty such as "/" above to ignore the error and logs the above event as a warning. Similarly, GCS supports <bucket>/<path1>/./<path2> and <bucket>/<path1>/../<path2> but these have special reserved meaning in linux filesystem. So, ignoring these two prefixes as well. Similarly, linux filesystem does not support '\0' in names of files or directories while it is supported in GCS object names. So, ignoring those as well in the GCS list call output.

Unsupported names mean directory names containing '.', '..', '/' or '\0' in them. These tests check that such objects don't show up in the list of prefixes returned by the bucketHandle.ListObjects .

This adds tests for unsupported directories ( directories which are either named '', or '.' or '..', or have '//', '/./', '/../' in their total path in their bucket.

* un-export unnecessarily exported internal constants * update comment * fix a log

gargnitingoogle force-pushed the gargnitin/fix-empty-directory-list-issue branch from 3d1d4bf to dfcbf92 Compare April 29, 2024 11:44

gargnitingoogle added the execute-integration-tests Run only integration tests label Apr 29, 2024

gargnitingoogle force-pushed the gargnitin/fix-empty-directory-list-issue branch 2 times, most recently from ed0d51d to 9024dc6 Compare April 30, 2024 09:11

gargnitingoogle commented Apr 30, 2024

View reviewed changes

internal/storage/bucket_handle.go Outdated Show resolved Hide resolved

gargnitingoogle force-pushed the gargnitin/fix-empty-directory-list-issue branch from 9024dc6 to c1195c4 Compare May 4, 2024 20:45

gargnitingoogle commented May 4, 2024

View reviewed changes

internal/util/util_test.go Outdated Show resolved Hide resolved

internal/util/util_test.go Outdated Show resolved Hide resolved

gargnitingoogle force-pushed the gargnitin/fix-empty-directory-list-issue branch 4 times, most recently from 02488f1 to cca204a Compare May 6, 2024 09:56

gargnitingoogle marked this pull request as ready for review May 6, 2024 10:00

gargnitingoogle requested review from ashmeenkaur and a team as code owners May 6, 2024 10:00

gargnitingoogle requested a review from sethiay May 6, 2024 10:00

gargnitingoogle changed the title ~~Skip empty-directory names in listed prefixes~~ Skip unsupported names in listed prefixes May 6, 2024

gargnitingoogle force-pushed the gargnitin/fix-empty-directory-list-issue branch from 80306c2 to fbbc0b1 Compare May 8, 2024 09:18

ashmeenkaur reviewed May 8, 2024

View reviewed changes

gargnitingoogle force-pushed the gargnitin/fix-empty-directory-list-issue branch from fbbc0b1 to dc55c22 Compare May 9, 2024 07:24

gargnitingoogle added 10 commits May 31, 2024 06:05

Add unit tests for unsupported directory names

dda032f

Unsupported names mean directory names containing '.', '..', '/' or '\0' in them. These tests check that such objects don't show up in the list of prefixes returned by the bucketHandle.ListObjects .

Add integration tests for unsupported directory names

e834ea9

This adds tests for unsupported directories ( directories which are either named '', or '.' or '..', or have '//', '/./', '/../' in their total path in their bucket.

fix lint error

41da696

Update semantics.md documentation

ea2969b

Address some review comments

49ce9fe

* un-export unnecessarily exported internal constants * update comment * fix a log

disable other internal/fs test files

9bda1e6

forward mountconfig in fs_test.go

df5a714

changes in implicit_dirs_test.go

24ee9d1

temp - outstanding changes

a97fe84

gargnitingoogle added 3 commits May 31, 2024 06:34

WIP

a738496

temp

23988ee

remove stray binaries

1d76261

gargnitingoogle force-pushed the gargnitin/fix-empty-directory-list-issue branch from dc55c22 to 1d76261 Compare May 31, 2024 06:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Skip unsupported names in listed prefixes #1876

Skip unsupported names in listed prefixes #1876

gargnitingoogle commented Apr 29, 2024 •

edited

gargnitingoogle commented May 6, 2024

ashmeenkaur left a comment

ashmeenkaur May 8, 2024

ashmeenkaur May 8, 2024 •

edited

ashmeenkaur May 8, 2024 •

edited

ashmeenkaur May 8, 2024

ashmeenkaur May 8, 2024

ashmeenkaur May 8, 2024

gargnitingoogle May 8, 2024

ashmeenkaur May 8, 2024

Skip unsupported names in listed prefixes #1876

Are you sure you want to change the base?

Skip unsupported names in listed prefixes #1876

Conversation

gargnitingoogle commented Apr 29, 2024 • edited

Description

Link to the issue in case of a bug fix.

Testing details

gargnitingoogle commented May 6, 2024

ashmeenkaur left a comment

Choose a reason for hiding this comment

ashmeenkaur May 8, 2024

Choose a reason for hiding this comment

ashmeenkaur May 8, 2024 • edited

Choose a reason for hiding this comment

ashmeenkaur May 8, 2024 • edited

Choose a reason for hiding this comment

ashmeenkaur May 8, 2024

Choose a reason for hiding this comment

ashmeenkaur May 8, 2024

Choose a reason for hiding this comment

ashmeenkaur May 8, 2024

Choose a reason for hiding this comment

gargnitingoogle May 8, 2024

Choose a reason for hiding this comment

ashmeenkaur May 8, 2024

Choose a reason for hiding this comment

gargnitingoogle commented Apr 29, 2024 •

edited

ashmeenkaur May 8, 2024 •

edited

ashmeenkaur May 8, 2024 •

edited