Read full message body at once when larger than 4MB

[gomoripeti]

Proposed Changes

When a message (much) larger than 4MB is read from an rdq file in rabbit_msg_store:scan/6) adjust the read size to the full message size instead of appending it together from 4MB chunks, which leaves a lot of garbage and memory fragmentation behind.

This patch is intentionally limited to 3.13.x. More major changes coming in 4.0 will make it unnecessary.

Closes #11072

Types of Changes

What types of changes does your code introduce to this project?
Put an x in the boxes that apply

• Bug fix (non-breaking change which fixes issue #NNNN)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause an observable behavior change in existing systems)
• Documentation improvements (corrections, new content, etc)
• Cosmetic change (whitespace, formatting, etc)
• Build system and/or CI

Checklist

Put an x in the boxes that apply.
You can also fill these out after creating the PR.
If you're unsure about any of them, don't hesitate to ask on the mailing list.
We're here to help!
This is simply a reminder of what we are going to look for before merging your code.

• I have read the CONTRIBUTING.md document
• I have signed the CA (see https://cla.pivotal.io/sign/rabbitmq)
• I have added tests that prove my fix is effective or that my feature works
• All tests pass locally with my changes
• If relevant, I have added necessary documentation to https://github.com/rabbitmq/rabbitmq-website
• If relevant, I have added this change to the first version(s) in release-notes that I expect to introduce it

Further Comments

If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution
you did and what alternatives you considered, etc.

[gomoripeti]

failing khepri tests seem to be unrelated - some timeouts in stream and shovel tests

[michaelklishin]

Thank you. I trust your observations about lower GC pressure but just out of curiosity, what do the before/after numbers look like?

[lhoguin]

It's hard to get numbers for this but this helps avoid OOMs in memory constrained environments. It's a good workaround for 3.13 and a proper fix will be included in 4.0.

[gomoripeti]

yes, to clarify this change is about to reduce peak memory consumption, and not increasing speed or reducing cpu usage.
In a memory constrained env with a 105MiB message size the original function got OOM-killed with ~660MB RSS while for this patched version /usr/bin/time -v reported ~250MB maximum RSS.
I did not experiment with a lot more different sizes or a different server, and I did not fully understand how this looks from inside the Erlang VM (carriers) but tracing garbage_collection did hint at some differences (#11072 (comment))