Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow S3 endpoint to be specified for non-commercial use cases #577

Open
petersedivec opened this issue Dec 8, 2021 · 1 comment
Open

Comments

@petersedivec
Copy link

We are using pyrasterframes v0.8.4 because we've been unable to get a more recent version running in databricks. We are attempting to load rasterframes from S3, but haven't been getting the following error when attempting to load from S3

at org.locationtech.rasterframes.expressions.generators.RasterSourceToRasterRefs.eval(RasterSourceToRasterRefs.scala:64) ... 25 more
Caused by: java.net.UnknownHostException: s3.amazonaws.com

We just noticed that in the stack trace and realize that since we're working in a classified environment the s3.amazonaws.com won't resolve. We now are successfully loading the rasterframes with pyrasterframes by manually copying the file from S3 and then loading from local dbfs file storage.

We had started playing around with rasterio and were able to load and use rasterio in databricks. Rasterio allow the S3 endpoint to be specified
with rasterio.Env(AWSSession(session), AWS_S3_ENDPOINT='data.cloudferro.com', AWS_HTTPS='NO') as env

We attempted to follow the different acceptable URI formats, including what was referenced in #38

We've been unable to get our s3 endpoint to work in either s3 or https URI format. Please add option to pass in a s3 endpoint

@pomadchin
Copy link
Member

pomadchin commented Dec 8, 2021

Hey @petersedivec yes indeed this functionality is not exposed through the Py API; but it should not be complicated to add.

The trick is to expose S3ClientProducer API in the acceptable form.

cc @metasim

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants