Support reading from Azure with adlsv2 managed identity

[jaychia]

Is your feature request related to a problem? Please describe.

In Azure, sometimes users may use Azure managed identity instead of just pure credentials.

Daft should support this. This may require some API changes on the AzureConfig.

Resources:

(What are managed identities for Azure resources?) https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview
(How to use it from databricks) https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/azure-managed-identities

[djouallah]

I did tried daft to read iceberg table using a catalog, it worked well with s3 but got no implemented error when using Azure ADLSGen2 ? is it planned ?

[jaychia]

Hi @djouallah !

This is planned, but requires some work on our end to correctly set up the Azure environment to test it. Let me bump up the priority on this :)

[jaychia]

@djouallah could you provide an example of your setup to help guide development of this feature?

[djouallah]

I am getting this

i get this error
DaftCoreException: DaftError::External Source not yet implemented: abfss

[jaychia]

Source not yet implemented:

Thanks! Actually this error might be because you're using abfss:// instead of abfs://.

Daft currently recognizes only az:// and abfs:// as valid ABFS URLs. Could you try using abfs:// instead and let me know if that works/how it fails?

[samster25]

It might be a simple fix to just map abfss:// to our Azure reader. It would be great if you can try it out @djouallah!

[djouallah]

can you map it please changing from abfss to abfs will break other engines :(

[samster25]

@djouallah Yeah happy to map it on our end, we just want to verify that it indeed fixes your issue.
If you could run this on your end

df = daft.read_parquet("abfs://PATH_TO_PARQUET_FILE_THAT_IS_ABFSS")

and verify that it works

[djouallah]

I understand, I am reading from a catalog, I can't change the URL

[samster25]

Actually just took a look at what azure's fsspec implementation looks like and they just map it.. I'll push up a PR!

[samster25]

PR UP: #2244

[samster25]

@djouallah The latest version of daft should now have the fix for reading abfss!

pip  install getdaft==0.2.24

[djouallah]

@samster25 thanks, now I get this

although the credential are already defined in the catalog

from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.io.fsspec import FsspecFileIO
catalog = SqlCatalog(
    "default",
    **{
        "uri"                : postgresql_db ,
        "adlfs.account-name" : account_name ,
        "adlfs.account-key"  : AZURE_STORAGE_ACCOUNT_KEY,
        "adlfs.tenant-id"    : azure_storage_tenant_id,
        "py-io-impl"         : "pyiceberg.io.fsspec.FsspecFileIO"
    },
)

polars works fine

[samster25]

Ah, we might have missed an option when converting the Iceberg credentials to our IOConfig. I'll make a fix ASAP!

passing currently in storage_options for polars?

[djouallah]

no polars just figure out it by default, did not add anything