Replies: 11 comments 2 replies
-
Yes, I am happy to support Azure storage if you would be willing to help with the utilities. I am on vacation until July 20, and I will be able to work through the technical details and PRs with you then. Looking forward to it. |
Beta Was this translation helpful? Give feedback.
-
Thanks Will — enjoy the time away! |
Beta Was this translation helpful? Give feedback.
-
@noamross mentioned http://min.io/ (example: https://github.com/noamross/targets-minio-versioning), which claims to support Azure. @v-a-s-a, is this a reasonable workaround for you? It would be nice to try this before implementing a native solution in |
Beta Was this translation helpful? Give feedback.
-
This is an interesting alternative. The licensing and pricing structure looks like a significant barrier for us. The alternative that is more realistic is simply checking everything into our repository with I've just come back to looking at this today, and I believe the implementation should be fairly lightweight. I've sorted out some decisions around the authentication, and I'll have some utility functions for you to look at soon. From there, you be able to comment as to whether it is worth continuing with the integration. |
Beta Was this translation helpful? Give feedback.
-
I've submitted an intiall pull request with a test and some utilities. It took a bit of time to decide how to to authentication, and I settled on using a service principal (similar to GCP's service account). I've been testing using Azure's free $200 tier on a private account. It's a bare start, but I thought I would get something into the pull request to start. It does look very similar to GCP, and it would have been great if we could use a single API to handle both. Take a look at the initial setup and test and let me know what you think. If it's too much to integrate into targets directly, perhaps we can do a separate package, similar to Otherwise, I'll continue trying to replicate the work done with GCP. Following the changes there has been helpful. The |
Beta Was this translation helpful? Give feedback.
-
Hi @v-a-s-a I am a data scientist at Microsoft and active user of some of the Azure-related R packages like {AzureStor}. I am interested in adopting {targets} in my workflow, backed by Azure Storage, and I came across this Issue and PR. It looks to me like it hasn't had activity for about five months. Are you still planning to work on it, and if so, would you like any help? |
Beta Was this translation helpful? Give feedback.
-
I'm happy to help, too. Azure + targets would make my work life a lot easier. |
Beta Was this translation helpful? Give feedback.
-
Thank you both. As @v-a-s-a said at the top, it would be great to have Azure storage utilities just like the ones for AWS and GCP (using I am naive to Azure Storage, and I am wondering about a few specifics:
|
Beta Was this translation helpful? Give feedback.
-
According to this, Azure Blob Storage "was designed to embrace a strong consistency model which guarantees that when the Storage service commits a data insert or update operation all further accesses to that data will see the latest update." So that's good news.
For Block Blobs the {AzureStor} package uses the PUT blocklist operation: blob_upload_internal.R#L1 and according to this documentation the max size for an upload using PUT blocklist is about 190.7 TiB, so I don't believe we'll need to implement any special handling for large files. |
Beta Was this translation helpful? Give feedback.
-
This issue has been inactive for a few months, and I was planning to crowd-source key parts of the implementation, so I am converting it to a discussion. |
Beta Was this translation helpful? Give feedback.
-
I am very interested in Azure Storage integration with {targets}. I see this discussion was closed. Was there a resolution or was the idea abandoned? If abandoned, were there any workable alternative solutions that others found? |
Beta Was this translation helpful? Give feedback.
-
Prework
Proposal
I've seen hints (e.g. in #769) at this while reading through the saga of GCP integration. For us, the day has come — we have a workflow running on Azure that we are trying to port to
targets
.I'd be happy to start just as @MarkEdmondson1234 did in #722, and produce Azure Storage utility functions based on the existing AWS S3 functions. I can also look at the tests, and try to replicate a similar harness for Azure. Based on discussion in #720, it looks like there is then work to be done integrating these into the
targets
class hierarchy.How does this sound for an initial plan to PR?
Beta Was this translation helpful? Give feedback.
All reactions