You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The receive_objects function in Group handles the transfer of both in-memory state and persisted logs for replicated objects. As a result of the way our serialization system works, it receives a single "buffer size" for the size of all serialized state (including current in-memory state and prior values in the persistent log), receives all the data into one buffer, and then hands off to Replicated::receive_object to deserialize it into current state and log entries. However, if the replicated object has a lot of persistent log data (i.e. many prior versions on disk), the size of the receive buffer may be too large to fit into memory.
I already encountered this issue when the stack-allocation of char buffer[buffer_size] failed when attempting to receive 10652222 bytes of log data. A temporary fix was to heap-allocate the buffer instead (with char* buffer = new char[buffer_size]), but eventually we may want to transfer logs to a new node that cannot fit even into heap memory.
One way around this problem would be to create a version of mutils::from_bytes that does not require the entire buffer to be in memory at once, but instead can consume it from a socket, much like mutils::post_object is a version of mutils::to_bytes that can serialize data piecemeal into a socket. (This is why the sender of the huge log does not run out of memory, only the receiver).
The text was updated successfully, but these errors were encountered:
The receive_objects function in Group handles the transfer of both in-memory state and persisted logs for replicated objects. As a result of the way our serialization system works, it receives a single "buffer size" for the size of all serialized state (including current in-memory state and prior values in the persistent log), receives all the data into one buffer, and then hands off to Replicated::receive_object to deserialize it into current state and log entries. However, if the replicated object has a lot of persistent log data (i.e. many prior versions on disk), the size of the receive buffer may be too large to fit into memory.
I already encountered this issue when the stack-allocation of
char buffer[buffer_size]
failed when attempting to receive 10652222 bytes of log data. A temporary fix was to heap-allocate the buffer instead (withchar* buffer = new char[buffer_size]
), but eventually we may want to transfer logs to a new node that cannot fit even into heap memory.One way around this problem would be to create a version of
mutils::from_bytes
that does not require the entire buffer to be in memory at once, but instead can consume it from a socket, much likemutils::post_object
is a version ofmutils::to_bytes
that can serialize data piecemeal into a socket. (This is why the sender of the huge log does not run out of memory, only the receiver).The text was updated successfully, but these errors were encountered: