Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for downloading from AWS S3 butckets #363

Open
WesleyTheGeolien opened this issue May 24, 2023 · 1 comment
Open

Add support for downloading from AWS S3 butckets #363

WesleyTheGeolien opened this issue May 24, 2023 · 1 comment
Labels
enhancement Idea or request for a new feature

Comments

@WesleyTheGeolien
Copy link

WesleyTheGeolien commented May 24, 2023

Edit by @leouieda on 2024-02-19

Add a AWSDownloader that can fetch the data from AWS S3 storage. It should support an authentication token, ideally with the option to read it from an environment variable. See instructions for adding such a downloader in #382 (comment).


Original issue 👇🏾

Description of the desired feature:
Data can be stored in cloud hosted buckets, s3, google storage, Azure, ...

These can provide either urls (I believe per-signining is possible) or some bucket location + authentification for example see the boto3 s3 python SDK

I am not sure on the data size but here is an example of downloading public data from s3: https://github.com/planet-os/notebooks/blob/master/aws/era5-s3-via-boto.ipynb

Minio can also be used docker image to run s3 locally for testing if better

Are you willing to help implement and maintain this feature?
Not sure I know enough about pooch (first time contribution and usage) to be able to do anything of use but I could possibly help out with guidance / provide further info

@WesleyTheGeolien WesleyTheGeolien added the enhancement Idea or request for a new feature label May 24, 2023
@remrama
Copy link

remrama commented Feb 16, 2024

Based off a similar need, I created a custom GSDownloader that downloads files from Google Cloud Storage. It's focused on files that require authentication. It uses the google-cloud-storage API for the download. Not sure if this request was for a more generalizable BucketDownloader, or something specific for AWS, like S3Downloader, but I wanted to link it here given the high overlap.

@leouieda leouieda changed the title Download data from buckets (S3) Add support for downloading from AWS S3 butckets Feb 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Idea or request for a new feature
Projects
None yet
Development

No branches or pull requests

2 participants