Jobs slowed down by RPC read loop

[rowlap]

In rpc.transport.Transport.Recv the critical inner loop is

• read up to 4096 bytes from the socket
• scan for LUXI_EOM
• break / return when any complete message is received

Problem 1

4096 bytes is rather small, and has been there since the beginning.

The number of loop iterations would be vastly reduced for a modest increase in read size. On my Linux box, net.core.wmem_default=212992 is the most data which could be sitting in the socket buffer waiting to be read.

Problem 2

When we scan self._buffer for LUXI_EOM, we're checking data which has already been checked.

new_msgs = (self._buffer + data).split(constants.LUXI_EOM)

We can assert that self._buffer does not contain LUXI_EOM, yet the whole string is re-scanned each time.

For reasons, my config.data file has grown to over 4MB, taking 1000 loops just to receive the result of each readConfig call, of which there are several from each INSTANCE_STARTUP job. It appears from profiling that most of the time is spent in Recv().

There is also a good amount of code duplication in FdTransport which might also benefit from any performance improvements.