Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

receive_objects can run out of memory when receiving persistent logs #110

Open
etremel opened this issue Mar 8, 2019 · 0 comments
Open

Comments

@etremel
Copy link
Contributor

etremel commented Mar 8, 2019

The receive_objects function in Group handles the transfer of both in-memory state and persisted logs for replicated objects. As a result of the way our serialization system works, it receives a single "buffer size" for the size of all serialized state (including current in-memory state and prior values in the persistent log), receives all the data into one buffer, and then hands off to Replicated::receive_object to deserialize it into current state and log entries. However, if the replicated object has a lot of persistent log data (i.e. many prior versions on disk), the size of the receive buffer may be too large to fit into memory.

I already encountered this issue when the stack-allocation of char buffer[buffer_size] failed when attempting to receive 10652222 bytes of log data. A temporary fix was to heap-allocate the buffer instead (with char* buffer = new char[buffer_size]), but eventually we may want to transfer logs to a new node that cannot fit even into heap memory.

One way around this problem would be to create a version of mutils::from_bytes that does not require the entire buffer to be in memory at once, but instead can consume it from a socket, much like mutils::post_object is a version of mutils::to_bytes that can serialize data piecemeal into a socket. (This is why the sender of the huge log does not run out of memory, only the receiver).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants