Restarting crashed queue with function_clause reason after update to RabbitMQ 3.13 #10902
-
Describe the bugWe have an Exchange with type Topic, 18 queues with different routing keys (including # and * masks) are bound to this exchange. Exchange has about 400 messages per second publish rate. We tried:
But nothing helped. Here is an example of the error log: Also we have another error stack trace: It seems like some "race conditions" with message store. Also there is such a hole on the messages read/write chart at the Rabbit Management before queue crashing: Reproduction stepsCannot reproduce at test environment. Expected behaviorNo queue crashing Additional contextNo response |
Beta Was this translation helpful? Give feedback.
Replies: 5 comments 23 replies
-
|
If you don't know how to reproduce this, the only way for us to investigate is to ask for a data directory of the affected node, its configuration and the hostname it is running on. You haven't specified if you are running 3.13.0 or 3.13.1, and what the topology is like besides that it uses a topic exchange. We do not guess in this community. Two known CQ issues in 3.13 were addressed in Changing "signal durability" (I am afraid I don't know what that is) without deleting the queue won't make the queue durable or transient, those settings are immutable. |
Beta Was this translation helpful? Give feedback.
-
Sorry, I will try to explain our problem more detailed.
We are using 3.13.0
We have one topic exchange, 18 queues bounded to this exchange by different routing keys. What else do you need to know about topology?
I mean that we change message delivery_mode from 2 to 1. Sorry, it's a .Net RabbitMQ.Client notation. |
Beta Was this translation helpful? Give feedback.
-
|
Well, CQ compaction has changed (to avoid a different scenario but still) in What we need to know about the topology is everything. Like I said, we do not guess in this community. Queue durability is critically important, for example. Definition export with the users, permissions removed and virtual hosts, queues renamed would tell us everything there is about the topology. |
Beta Was this translation helpful? Give feedback.
-
|
But more important would be to upgrade to Since the file is empty and the queue does not recover with it anyway, you might as well wipe |
Beta Was this translation helpful? Give feedback.
-
|
Rather than those this is about the files in I do not think the queue files would be super useful, just the contents of |
Beta Was this translation helpful? Give feedback.
-
|
We updated RabbitMQ to 3.13.1 and problem is still remains. And we urgently rolled back to RabbitMQ 3.12.13. Now I cannot provide the files you are interested in, and I will try to reproduce the problem in the local environment. |
Beta Was this translation helpful? Give feedback.
-
|
I don't see how this can be "a race condition", let's refrain from speculating. The stack trace tells us that a function that reads a message body gets an Without complete logs and data files we cannot know what happened. The node could have run out of disk space and a message file segment was not written. The metrics gap can be easily explained: if the only active queue or set of queues in the system are affected by this, If all queues are affected at the same time, I strongly suspect that the node has run out of disk space. Such nodes must be decommissioned and replaced. Use a replicated queue type (quorum queues) and/or streams if the data in these queues is important. So, please share full logs with sensitive values such as hostnames and usernames edited out, and the files under |
Beta Was this translation helpful? Give feedback.
-
There is no problems with disk space
I will prepare logs and attach here as fast as possible. Thank you! |
Beta Was this translation helpful? Give feedback.
-
|
Please upgrade first, #10696, #10681 both affect exactly the same code path. |
Beta Was this translation helpful? Give feedback.
-
|
We're experiencing the same issue here after an upgrade to RabbitMQ 3.13.0, on two different servers about 5 times so far: |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the heads up. Unfortunately I can't figure out much without the files requested before or a simple way to reproduce. Please provide information about your environment (OS, hardware, hardware tweaks that may affect disk access...) as well as the kind of work it is doing (payload sizes, frequency, number of queues, etc.) |
Beta Was this translation helpful? Give feedback.
-
|
Having the entire logs would help as well. |
Beta Was this translation helpful? Give feedback.
-
|
We are making progress in identifying this issue. We are trying to find the (very specific) conditions to reproduce. Having a detailed description of the topology and workloads would help. Thank you. |
Beta Was this translation helpful? Give feedback.
-
|
Hello! I believe I now know the precise reasons why this happens. I am working on a fix. Sorry it took so long but I've only managed to reliably reproduce it since yesterday. Both the bug and the needed fix are complex so it will likely take a few days before it is ready. Thank you for your patience. |
Beta Was this translation helpful? Give feedback.
-
|
Hello! The fixes will be released in RabbitMQ 3.13.2. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
I have now found a way to reproduce fairly consistently, with a very large fan-out. |
Beta Was this translation helpful? Give feedback.
-
|
FTR, here is the reproduction method #11288 |
Beta Was this translation helpful? Give feedback.
-
|
Hello, Thanks for identifying & fixing the issue. After going through the PR, I am curious to know if the slow consumers could exhaust cache memory or is there any option to control the cache size, file rolling interval etc to limit any negative impact. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, now that the code was fixed, it shouldn't be possible to exhaust memory. Not that we have ever heard of this happening. It's not really a question of slow consumers either. And no, the cache can't be configured per se because all incoming messages have to go through it. So the normal limits are what matter (number of connections, publisher confirms...) |
Beta Was this translation helpful? Give feedback.
-
|
@ashimrana #11288 has nothing to do with "slow consumers". Quorum queues keep a portion of the data in memory, and there are several configurable settings (some MUST NOT be changed for existing clusters). Streams move data to disk as early as possible and their memory footprint is generally minimal. Both topics are not relevant here. |
Beta Was this translation helpful? Give feedback.
Hello! The fixes will be released in RabbitMQ 3.13.2. Thanks!