googledrive: Google recently added limit of 500 k files per directory

[joonas-fi]

Google in June 2020 started to enforce a retroactively added limitation of 500 k files per directory. If you go over, it yields error code numChildrenInNonRootLimitExceeded: https://stackoverflow.com/a/62520242

I hit this limitation with other storage system using Drive in similar design that I understand Perkeep uses.

To be clear: I did not reach this limit with Perkeep (and so cannot be 100 % certain Perkeep users will hit this), but by looking at Perkeep code it looks like Perkeep stores blobs in a single, non-root, directory:

perkeep/pkg/blobserver/google/drive/drive.go

Line 65 in c2e3137

parent := config.RequiredString("parent_id")

Also the package description confirms this:

perkeep/pkg/blobserver/google/drive/drive.go

Line 19 in c2e3137

type, storing blobs in a Google Drive folder.

It seems like the Drive root is the only directory that is exempt from this limitation. I totally get Perkeep's design decision to not store blobs in root (to keep them separate from user's possible other files). It looks like you can use a const root as the folder id (to put all bobs in root):

you can use the alias root to refer to the root folder anywhere a file ID is provided.
https://developers.google.com/drive/api/v3/folder

.. so this necessarily doesn't require a code change. But there should be warning in the documentation explaining this.

Other approach would be to make a nested subfolder structure for the blobs, but that doesn't seem perfect either, because if you'd make them on-demand, you'd be suspectible to race conditions if you use multiple Perkeep instances.