Cannot freeze-to-file and thaw-from-file objects that are larger than 2GB

[RokLenarcic]

Hello. Due to these functions being merely a thin wrapper around freeze and thaw they interact with a file via a byte array. That limits the size of these operations to max size of byte array which is 2GB. I'd love to have the option to freeze and thaw to DataInput/DataOutput objects or ByteBuffer objects.

Incidentally ByteBuffer offers very similar API than DataInput and DataOutput, and the benefit there is MappedByteBuffer use, which is what you get when you memory map a file.

Obviously such use cannot use the current encryptor and compressor system as those need a byte array. But I am sure there's some kind of streaming compression and encryption that could be used.

[ptaoussanis]

@RokLenarcic Hi Rok,

I'd love to have the option to freeze and thaw to DataInput/DataOutput objects or ByteBuffer objects.

Just checking if you noticed freeze-to-out! and thaw-from-in!?

Incidentally ByteBuffer offers very similar API than DataInput and DataOutput, and the benefit there is MappedByteBuffer use, which is what you get when you memory map a file.

I'd be open to a sketch/proof-of-concept PR of some sort if you or someone else felt like proposing a concrete way to support ByteBuffers 👍

[RokLenarcic]

Yeah I saw those functions, but there's a rather large amount of logic (e.g. header and options parsing/use) that is in freeze that I would have to replicate in my own code to have freeze-to-out! work approx the same.

In general the direction that would be needed for large objects is to have a protocol that has functions like a stream: write bytes and close/finalize. Then you could have encryption, compression and the writing implementation itself implement this. Then you could stack these like we do with streams, encryption wrapping the compression wrapping the data/file writing instance of this protocol.

[ptaoussanis]

Sketch/proof-of-concept PR welcome if you or anyone else felt like taking a look at this - otherwise I can look at this myself for a future version of Nippy.

Please 👍 here if there's interest in adding better streaming support to Nippy.