Will Gluten support custom file systems?

[rshkv]

I was wondering if Gluten plans to support custom file systems (we're trying Velox but maybe speaking more broadly).

It seems that, for Velox at least, we'd get there through some extension point in which we can call velox::filesystems::registerFileSystem (as Gluten does here for the JoL file system) and somehow pass a custom implementation. (And I wouldn't mind if that implementation needs to be in C++/native or if we can reuse the Java JniFileSystem interface.)

We have a custom Hadoop file system implementation that we use with Spark. When we tried Gluten & Velox on simple queries, we saw it perform worse than vanilla Spark. And I suspect that's because of the serialization overhead introduced by Gluten having to fallback to Java for reads and writes. My hope is that, if we could provide Velox with a native/Velox-friendly implementation for our file system, we'd observe Gluten's speed-ups.