RabbitMQ Crashing with bad_environment_value /etc/rabbitmq/erl_inetrc #10777

noelashford · 2024-03-19T00:01:00Z

noelashford
Mar 19, 2024

Describe the bug

(rabbitmq)[root@tunninet-server-noel /]# rabbitmqctl --node rabbit status
Distribution failed: {{:shutdown, {:failed_to_start_child, :net_kernel, {:EXIT, :nodistribution}}}, {:child, :undefined, :net_sup_dynamic, {:erl_distribution, :start_link, [%{name: :"rabbitmqcli-59-rabbit@tunninet-server-noel", supervisor: :net_sup_dynamic, net_tickintensity: 4, net_ticktime: 60, name_domain: :shortnames, clean_halt: false}]}, :permanent, false, 1000, :supervisor, [:erl_distribution]}}
(rabbitmq)[root@tunninet-server-noel /]# epmd -names
epmd: up and running on port 4369 with data:
name rabbit at port 25672
(rabbitmq)[root@tunninet-server-noel /]# exit
exit
root@tunninet-server-noel:~# cat /etc/kolla/rabbitmq/rabbitmq-env.conf
RABBITMQ_NODENAME=rabbit@tunninet-server-noel
RABBITMQ_LOG_BASE=/var/log/kolla/rabbitmq
RABBITMQ_DIST_PORT=25672
RABBITMQ_PID_FILE=/var/lib/rabbitmq/mnesia/rabbitmq.pid
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-kernel inetrc '/etc/rabbitmq/erl_inetrc' +S 2:2 +sbwt none +sbwtdcpu none +sbwtdio none"
RABBITMQ_CTL_ERL_ARGS=""

export ERL_EPMD_ADDRESS=192.168.5.1
export ERL_EPMD_PORT=4369
export ERL_INETRC=/etc/rabbitmq/erl_inetrc
root@tunninet-server-noel:~# sudo ss -tlnp | grep 4369
LISTEN 0 4096 192.168.5.1:4369 0.0.0.0:* users:(("epmd",pid=54136,fd=5))
LISTEN 0 4096 127.0.0.1:4369 0.0.0.0:* users:(("epmd",pid=54136,fd=3))
LISTEN 0 4096 [::1]:4369 [::]:* users:(("epmd",pid=54136,fd=4))

19:58:37.813 [error] rabbit_env: Failed to setup distribution (as rabbit_ctl_44@tunninet-server-noel) to query node rabbit@tunninet-server-noel: {:error,
 {{:shutdown, {:failed_to_start_child, :net_kernel, {:EXIT, :nodistribution}}},
  {:child, :undefined, :net_sup_dynamic,
   {:erl_distribution, :start_link,
    [
      %{
        name: :"rabbit_ctl_44@tunninet-server-noel",
        supervisor: :net_sup_dynamic,
        net_tickintensity: 4,
        net_ticktime: 60,
        name_domain: :shortnames,
        clean_halt: false
      }
    ]}, :permanent, false, 1000, :supervisor, [:erl_distribution]}}}

Reproduction steps

This can be produced by installing this on a dual stack containerized environment.

Expected behavior

rabbitmqctl should be able to query without issue

Additional context

No response

Answered by noelashford

Mar 20, 2024

OK.... WOW! I found the issue and it was quite obscure... why ERLANG and EPMD care about this, IDK. The other day by mistake during testing of something on my FW i left on wide NAT for all interfaces and it thought that it was getting a foreign connection as NAT was messing with it.... I narrowed NAT to my WAN (as it should be) and WOILA! Sorted! Please add this to you FAQ guys ;0 It could save someone else 2 days of insanity ;0

View full answer

lukebakken · 2024-03-19T00:04:33Z

lukebakken
Mar 19, 2024
Maintainer

Hi @noelashford -

You have configured RabbitMQ to use the inetrc file, but not the RabbitMQ command-line tool, rabbitmqctl. Your rabbitmq-env.conf file should look like this:

SERVER_ADDITIONAL_ERL_ARGS='-kernel inetrc /etc/rabbitmq/erl_inetrc +S 2:2 +sbwt none +sbwtdcpu none +sbwtdio none'
CTL_ERL_ARGS="-kernel inetrc /etc/rabbitmq/erl_inetrc"

Note that there is no need to use the RABBITMQ_ prefix with this file.

0 replies

noelashford · 2024-03-19T01:44:59Z

noelashford
Mar 19, 2024
Author

ok, i am running RabbitMQ 3.12.13, i made the changes you noted. Here is my config file:

NODENAME=rabbit@tunninet-server-noel
LOG_BASE=/var/log/kolla/rabbitmq
DIST_PORT=25672
PID_FILE=/var/lib/rabbitmq/mnesia/rabbitmq.pid
SERVER_ADDITIONAL_ERL_ARGS='-kernel inetrc /etc/rabbitmq/erl_inetrc +S 2:2 +sbwt none +sbwtdcpu none +sbwtdio none'
CTL_ERL_ARGS="-kernel inetrc /etc/rabbitmq/erl_inetrc"

export ERL_EPMD_ADDRESS=192.168.5.1
export ERL_EPMD_PORT=4369
export ERL_INETRC=/etc/rabbitmq/erl_inetrc

still wont start....

Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
+ [[ ! -n '' ]]
+ . kolla_extend_start
++ : /var/log/kolla/rabbitmq
++ [[ -n '' ]]
++ [[ ! -d /var/log/kolla/rabbitmq ]]
+++ stat -c %a /var/log/kolla/rabbitmq
++ [[ 2755 != \7\5\5 ]]
++ chmod 755 /var/log/kolla/rabbitmq
Running command: '/usr/sbin/rabbitmq-server'
+ echo 'Running command: '\''/usr/sbin/rabbitmq-server'\'''
+ exec /usr/sbin/rabbitmq-server
2024-03-18 21:43:18.093243 application_controller: ~ts: ~ts~n
        ["syntax error before: ","'/'"]
        "/etc/rabbitmq/erl_inetrc"
Could not start kernel pid (application_controller) ("{bad_environment_value,\"/etc/rabbitmq/erl_inetrc\"}")

btw, is erl_inetrc meant to be an empty file ?

2 replies

lukebakken Mar 19, 2024
Maintainer

Sorry, I forgot that the format of string arguments to the Erlang VM is very picky.

Please see - https://www.erlang.org/doc/apps/erts/inet_cfg

Your environment variables should be set like this:

SERVER_ADDITIONAL_ERL_ARGS='-kernel inetrc "/etc/rabbitmq/erl_inetrc" +S 2:2 +sbwt none +sbwtdcpu none +sbwtdio none'
CTL_ERL_ARGS='-kernel inetrc "/etc/rabbitmq/erl_inetrc"'

But, you do NOT need the erl_inetrc file unless you are using IPv6. Why do you think you needed to add this argument to the above environment variables? I would like to improve our docs if that is where you got the information.

noelashford Mar 19, 2024
Author

Hello, the file was written out by ansible / Jenga in Kolla Ansible in my case, it would seem to have a fault. I tried without it in both of these lines and i still get the same net effect as per below. Not sure what it can be with this server as it worked first go on another server.

noelashford · 2024-03-19T01:55:36Z

noelashford
Mar 19, 2024
Author

I changed it back to:

RABBITMQ_NODENAME=rabbit@tunninet-server-noel
RABBITMQ_LOG_BASE=/var/log/kolla/rabbitmq
RABBITMQ_DIST_PORT=25672
RABBITMQ_PID_FILE=/var/lib/rabbitmq/mnesia/rabbitmq.pid
RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-kernel inetrc '/etc/rabbitmq/erl_inetrc' +S 2:2 +sbwt none +sbwtdcpu none +sbwtdio none"
RABBITMQ_CTL_ERL_ARGS="-kernel inetrc /etc/rabbitmq/erl_inetrc"
export ERL_EPMD_ADDRESS=192.168.5.1
export ERL_EPMD_PORT=4369
export ERL_INETRC=/etc/rabbitmq/erl_inetrc

++ cat /run_command
+ CMD=/usr/sbin/rabbitmq-server
+ ARGS=
+ sudo kolla_copy_cacerts
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
+ [[ ! -n '' ]]
+ . kolla_extend_start
++ : /var/log/kolla/rabbitmq
++ [[ -n '' ]]
++ [[ ! -d /var/log/kolla/rabbitmq ]]
+++ stat -c %a /var/log/kolla/rabbitmq
++ [[ 2755 != \7\5\5 ]]
++ chmod 755 /var/log/kolla/rabbitmq
+ echo 'Running command: '\''/usr/sbin/rabbitmq-server'\'''
+ exec /usr/sbin/rabbitmq-server
Running command: '/usr/sbin/rabbitmq-server'
+ exec /usr/sbin/rabbitmq-server
Running command: '/usr/sbin/rabbitmq-server'
2024-03-18 21:52:53.612109-04:00 [notice] <0.153.0> Protocol 'inet_tcp': register/listen error: epmd_close
2024-03-18 21:52:53.612109-04:00 [notice] <0.153.0>
2024-03-18 21:52:53.617488-04:00 [error] <0.153.0>   crasher:
2024-03-18 21:52:53.617488-04:00 [error] <0.153.0>     initial call: net_kernel:init/1
2024-03-18 21:52:53.617488-04:00 [error] <0.153.0>     pid: <0.153.0>
2024-03-18 21:52:53.617488-04:00 [error] <0.153.0>     registered_name: []
2024-03-18 21:52:53.617488-04:00 [error] <0.153.0>     exception exit: {error,badarg}
2024-03-18 21:52:53.617488-04:00 [error] <0.153.0>       in function  gen_server:init_it/6 (gen_server.erl, line 961)
2024-03-18 21:52:53.617488-04:00 [error] <0.153.0>     ancestors: [net_sup,kernel_sup,<0.47.0>]
2024-03-18 21:52:53.617488-04:00 [error] <0.153.0>     message_queue_len: 0
2024-03-18 21:52:53.617488-04:00 [error] <0.153.0>     messages: []
2024-03-18 21:52:53.617488-04:00 [error] <0.153.0>     links: [<0.150.0>,#Port<0.11>]
2024-03-18 21:52:53.617488-04:00 [error] <0.153.0>     dictionary: [{longnames,false}]
2024-03-18 21:52:53.617488-04:00 [error] <0.153.0>     trap_exit: true
2024-03-18 21:52:53.617488-04:00 [error] <0.153.0>     status: running
2024-03-18 21:52:53.617488-04:00 [error] <0.153.0>     heap_size: 1598
2024-03-18 21:52:53.617488-04:00 [error] <0.153.0>     stack_size: 28
2024-03-18 21:52:53.617488-04:00 [error] <0.153.0>     reductions: 2971
2024-03-18 21:52:53.617488-04:00 [error] <0.153.0>   neighbours:
2024-03-18 21:52:53.617488-04:00 [error] <0.153.0>
2024-03-18 21:52:53.621622-04:00 [error] <0.150.0>     supervisor: {local,net_sup}
2024-03-18 21:52:53.621622-04:00 [error] <0.150.0>     errorContext: start_error
2024-03-18 21:52:53.621622-04:00 [error] <0.150.0>     reason: {'EXIT',nodistribution}
2024-03-18 21:52:53.621622-04:00 [error] <0.150.0>     offender: [{pid,undefined},
2024-03-18 21:52:53.621622-04:00 [error] <0.150.0>                {id,net_kernel},
2024-03-18 21:52:53.621622-04:00 [error] <0.150.0>                {mfargs,{net_kernel,start_link,
2024-03-18 21:52:53.621622-04:00 [error] <0.150.0>                                    [#{name => rabbit_prelaunch_735@localhost,
2024-03-18 21:52:53.621622-04:00 [error] <0.150.0>                                       supervisor => net_sup_dynamic,
2024-03-18 21:52:53.621622-04:00 [error] <0.150.0>                                       net_tickintensity => 4,
2024-03-18 21:52:53.621622-04:00 [error] <0.150.0>                                       net_ticktime => 60,
2024-03-18 21:52:53.621622-04:00 [error] <0.150.0>                                       name_domain => shortnames,
2024-03-18 21:52:53.621622-04:00 [error] <0.150.0>                                       clean_halt => false}]}},
2024-03-18 21:52:53.621622-04:00 [error] <0.150.0>                {restart_type,permanent},
2024-03-18 21:52:53.621622-04:00 [error] <0.150.0>                {significant,false},
2024-03-18 21:52:53.621622-04:00 [error] <0.150.0>                {shutdown,2000},
2024-03-18 21:52:53.621622-04:00 [error] <0.150.0>                {child_type,worker}]
2024-03-18 21:52:53.621622-04:00 [error] <0.150.0>

BOOT FAILED
===========
Exception during startup:

2024-03-18 21:52:53.623401-04:00 [error] <0.136.0>
2024-03-18 21:52:53.623401-04:00 [error] <0.136.0> BOOT FAILED
2024-03-18 21:52:53.623401-04:00 [error] <0.136.0> ===========
2024-03-18 21:52:53.623401-04:00 [error] <0.136.0> Exception during startup:
2024-03-18 21:52:53.623401-04:00 [error] <0.136.0>
2024-03-18 21:52:53.623401-04:00 [error] <0.136.0> error:{badmatch,{error,{{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}},{child,undefined,net_sup_dynamic,{erl_distribution,start_link,[#{name => rabbit_prelaunch_735@localhost,supervisor => net_sup_dynamic,net_tickintensity => 4,net_ticktime => 60,name_domain => shortnames,clean_halt => false}]},permanent,false,1000,supervisor,[erl_distribution]}}}}
2024-03-18 21:52:53.623401-04:00 [error] <0.136.0>
2024-03-18 21:52:53.623401-04:00 [error] <0.136.0>     rabbit_prelaunch_dist:duplicate_node_check/1, line 80
2024-03-18 21:52:53.623401-04:00 [error] <0.136.0>     rabbit_prelaunch_dist:setup/1, line 23
2024-03-18 21:52:53.623401-04:00 [error] <0.136.0>     rabbit_prelaunch:do_run/0, line 115
error:{badmatch,{error,{{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}},{child,undefined,net_sup_dynamic,{erl_distribution,start_link,[#{name => rabbit_prelaunch_735@localhost,supervisor => net_sup_dynamic,net_tickintensity => 4,net_ticktime => 60,name_domain => shortnames,clean_halt => false}]},permanent,false,1000,supervisor,[erl_distribution]}}}}

    rabbit_prelaunch_dist:duplicate_node_check/1, line 80
2024-03-18 21:52:53.623401-04:00 [error] <0.136.0>     rabbit_prelaunch:run_prelaunch_first_phase/0, line 32
2024-03-18 21:52:53.623401-04:00 [error] <0.136.0>     supervisor:do_start_child_i/3, line 420
2024-03-18 21:52:53.623401-04:00 [error] <0.136.0>     supervisor:do_start_child/2, line 406
2024-03-18 21:52:53.623401-04:00 [error] <0.136.0>     supervisor:-start_children/2-fun-0-/3, line 390
2024-03-18 21:52:53.623401-04:00 [error] <0.136.0>     supervisor:children_map/4, line 1258
2024-03-18 21:52:53.623401-04:00 [error] <0.136.0>
    rabbit_prelaunch_dist:setup/1, line 23
    rabbit_prelaunch:do_run/0, line 115
    rabbit_prelaunch:run_prelaunch_first_phase/0, line 32
    supervisor:do_start_child_i/3, line 420
    supervisor:do_start_child/2, line 406
    supervisor:-start_children/2-fun-0-/3, line 390
    supervisor:children_map/4, line 1258

2024-03-18 21:52:54.624780-04:00 [error] <0.136.0>     supervisor: {local,rabbit_prelaunch_sup}
2024-03-18 21:52:54.624780-04:00 [error] <0.136.0>     errorContext: start_error
2024-03-18 21:52:54.624780-04:00 [error] <0.136.0>     reason: {badmatch,
2024-03-18 21:52:54.624780-04:00 [error] <0.136.0>                 {error,
2024-03-18 21:52:54.624780-04:00 [error] <0.136.0>                     {{shutdown,
2024-03-18 21:52:54.624780-04:00 [error] <0.136.0>                          {failed_to_start_child,net_kernel,
2024-03-18 21:52:54.624780-04:00 [error] <0.136.0>                              {'EXIT',nodistribution}}},
2024-03-18 21:52:54.624780-04:00 [error] <0.136.0>                      {child,undefined,net_sup_dynamic,
2024-03-18 21:52:54.624780-04:00 [error] <0.136.0>                          {erl_distribution,start_link,
2024-03-18 21:52:54.624780-04:00 [error] <0.136.0>                              [#{name => rabbit_prelaunch_735@localhost,
2024-03-18 21:52:54.624780-04:00 [error] <0.136.0>                                 supervisor => net_sup_dynamic,
2024-03-18 21:52:54.624780-04:00 [error] <0.136.0>                                 net_tickintensity => 4,net_ticktime => 60,
2024-03-18 21:52:54.624780-04:00 [error] <0.136.0>                                 name_domain => shortnames,
2024-03-18 21:52:54.624780-04:00 [error] <0.136.0>                                 clean_halt => false}]},
2024-03-18 21:52:54.624780-04:00 [error] <0.136.0>                          permanent,false,1000,supervisor,
2024-03-18 21:52:54.624780-04:00 [error] <0.136.0>                          [erl_distribution]}}}}
2024-03-18 21:52:54.624780-04:00 [error] <0.136.0>     offender: [{pid,undefined},
2024-03-18 21:52:54.624780-04:00 [error] <0.136.0>                {id,prelaunch},
2024-03-18 21:52:54.624780-04:00 [error] <0.136.0>                {mfargs,{rabbit_prelaunch,run_prelaunch_first_phase,[]}},
2024-03-18 21:52:54.624780-04:00 [error] <0.136.0>                {restart_type,transient},
2024-03-18 21:52:54.624780-04:00 [error] <0.136.0>                {significant,false},
2024-03-18 21:52:54.624780-04:00 [error] <0.136.0>                {shutdown,5000},
2024-03-18 21:52:54.624780-04:00 [error] <0.136.0>                {child_type,worker}]
2024-03-18 21:52:54.624780-04:00 [error] <0.136.0>
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>   crasher:
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>     initial call: application_master:init/4
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>     pid: <0.134.0>
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>     registered_name: []
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>     exception exit: {{shutdown,
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>                       {failed_to_start_child,prelaunch,
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>                        {badmatch,
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>                         {error,
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>                          {{shutdown,
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>                            {failed_to_start_child,net_kernel,
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>                             {'EXIT',nodistribution}}},
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>                           {child,undefined,net_sup_dynamic,
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>                            {erl_distribution,start_link,
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>                             [#{name => rabbit_prelaunch_735@localhost,
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>                                supervisor => net_sup_dynamic,
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>                                net_tickintensity => 4,net_ticktime => 60,
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>                                name_domain => shortnames,
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>                                clean_halt => false}]},
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>                            permanent,false,1000,supervisor,
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>                            [erl_distribution]}}}}}},
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>                      {rabbit_prelaunch_app,start,[normal,[]]}}
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>       in function  application_master:init/4 (application_master.erl, line 142)
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>     ancestors: [<0.133.0>]
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>     message_queue_len: 1
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>     messages: [{'EXIT',<0.135.0>,normal}]
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>     links: [<0.133.0>,<0.44.0>]
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>     dictionary: []
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>     trap_exit: true
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>     status: running
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>     heap_size: 376
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>     stack_size: 28
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>     reductions: 165
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>   neighbours:
2024-03-18 21:52:54.625188-04:00 [error] <0.134.0>
2024-03-18 21:52:54.625565-04:00 [notice] <0.44.0> Application rabbitmq_prelaunch exited with reason: {{shutdown,{failed_to_start_child,prelaunch,{badmatch,{error,{{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}},{child,undefined,net_sup_dynamic,{erl_distribution,start_link,[#{name => rabbit_prelaunch_735@localhost,supervisor => net_sup_dynamic,net_tickintensity => 4,net_ticktime => 60,name_domain => shortnames,clean_halt => false}]},permanent,false,1000,supervisor,[erl_distribution]}}}}}},{rabbit_prelaunch_app,start,[normal,[]]}}
Kernel pid terminated (application_controller) ("{application_start_failure,rabbitmq_prelaunch,{{shutdown,{failed_to_start_child,prelaunch,{badmatch,{error,{{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}},{child,undefined,net_sup_dynamic,{erl_distribution,start_link,[#{name => rabbit_prelaunch_735@localhost,supervisor => net_sup_dynamic,net_tickintensity => 4,net_ticktime => 60,name_domain => shortnames,clean_halt => false}]},permanent,false,1000,supervisor,[erl_distribution]}}}}}},{rabbit_prelaunch_app,start,[normal,[]]}}}")

Crash dump is being written to: erl_crash.dump...done

0 replies

noelashford · 2024-03-19T02:47:43Z

noelashford
Mar 19, 2024
Author

Seems to be this is the root cause... ?

("{bad_environment_value,\"/etc/rabbitmq/erl_inetrc\"}")

/etc/rabbitmq/erl_inetrc is empty in my case as this is only used if you need ipv6 API.
https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/rabbitmq/templates/erl_inetrc.j2

seems like this is some syntax error in rabbitmq-env.conf ?

root@tunninet-server-noel:~# export ERL_INETRC=/etc/rabbitmq/erl_inetrc
root@tunninet-server-noel:~# echo $ERL_INETRC
/etc/rabbitmq/erl_inetrc

doesnt look like its there unless you need quotes in the above to parse correctly ? Any ideas ? I enabled the crash file but there's nothing much of use in there in this case i reckon?

0 replies

noelashford · 2024-03-19T03:07:54Z

noelashford
Mar 19, 2024
Author

even when i try to remove INETRC altogether, I get this:

(rabbitmq)[root@tunninet-server-noel /]# rabbitmqctl status
Distribution failed: {{:shutdown, {:failed_to_start_child, :net_kernel, {:EXIT, :nodistribution}}}, {:child, :undefined, :net_sup_dynamic, {:erl_distribution, :start_link, [%{name: :"rabbitmqcli-514-rabbit@tunninet-server-noel", supervisor: :net_sup_dynamic, net_tickintensity: 4, net_ticktime: 60, name_domain: :shortnames, clean_halt: false}]}, :permanent, false, 1000, :supervisor, [:erl_distribution]}}

0 replies

noelashford · 2024-03-19T05:27:09Z

noelashford
Mar 19, 2024
Author

Ok, something interesting... i temporarily removed the docker version i was using and tried to install manually - i got the same issue with startup.... thinking there's a config issue maybe causing this ? Any tests you can think of that i can run ?

Mar 19 01:24:44 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:44.495329-04:00 [error] <0.136.0> BOOT FAILED
Mar 19 01:24:44 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:44.495329-04:00 [error] <0.136.0> ===========
Mar 19 01:24:44 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:44.495329-04:00 [error] <0.136.0> Exception during startup:
Mar 19 01:24:44 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:44.495329-04:00 [error] <0.136.0>
Mar 19 01:24:44 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:44.495329-04:00 [error] <0.136.0> error:{badmatch,{error,{{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}},{child,undefined,net_sup_dynamic,{erl_distribution,start_link,[#{name => rabbit_prelaunch_1015971@localhost,>
Mar 19 01:24:44 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:44.495329-04:00 [error] <0.136.0>
Mar 19 01:24:44 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:44.495329-04:00 [error] <0.136.0>     rabbit_prelaunch_dist:duplicate_node_check/1, line 80
Mar 19 01:24:44 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:44.495329-04:00 [error] <0.136.0>     rabbit_prelaunch_dist:setup/1, line 23
Mar 19 01:24:44 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:44.495329-04:00 [error] <0.136.0>     rabbit_prelaunch:do_run/0, line 115
Mar 19 01:24:44 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:44.495329-04:00 [error] <0.136.0>     rabbit_prelaunch:run_prelaunch_first_phase/0, line 32
Mar 19 01:24:44 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:44.495329-04:00 [error] <0.136.0>     supervisor:do_start_child_i/3, line 420
Mar 19 01:24:44 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:44.495329-04:00 [error] <0.136.0>     supervisor:do_start_child/2, line 406
Mar 19 01:24:44 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:44.495329-04:00 [error] <0.136.0>     supervisor:-start_children/2-fun-0-/3, line 390
Mar 19 01:24:44 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:44.495329-04:00 [error] <0.136.0>     supervisor:children_map/4, line 1258
Mar 19 01:24:44 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:44.495329-04:00 [error] <0.136.0>
Mar 19 01:24:44 tunninet-server-noel rabbitmq-server[1015971]: ===========
Mar 19 01:24:44 tunninet-server-noel rabbitmq-server[1015971]: Exception during startup:
Mar 19 01:24:44 tunninet-server-noel rabbitmq-server[1015971]: error:{badmatch,{error,{{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}},{child,undefined,net_sup_dynamic,{erl_distribution,start_link,[#{name => rabbit_prelaunch_1015971@localhost,supervisor => net_sup_dynamic,net_tickintensity => >
Mar 19 01:24:44 tunninet-server-noel rabbitmq-server[1015971]:     rabbit_prelaunch_dist:duplicate_node_check/1, line 80
Mar 19 01:24:44 tunninet-server-noel rabbitmq-server[1015971]:     rabbit_prelaunch_dist:setup/1, line 23
Mar 19 01:24:44 tunninet-server-noel rabbitmq-server[1015971]:     rabbit_prelaunch:do_run/0, line 115
Mar 19 01:24:44 tunninet-server-noel rabbitmq-server[1015971]:     rabbit_prelaunch:run_prelaunch_first_phase/0, line 32
Mar 19 01:24:44 tunninet-server-noel rabbitmq-server[1015971]:     supervisor:do_start_child_i/3, line 420
Mar 19 01:24:44 tunninet-server-noel rabbitmq-server[1015971]:     supervisor:do_start_child/2, line 406
Mar 19 01:24:44 tunninet-server-noel rabbitmq-server[1015971]:     supervisor:-start_children/2-fun-0-/3, line 390
Mar 19 01:24:44 tunninet-server-noel rabbitmq-server[1015971]:     supervisor:children_map/4, line 1258
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.496486-04:00 [error] <0.136.0>     supervisor: {local,rabbit_prelaunch_sup}
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.496486-04:00 [error] <0.136.0>     errorContext: start_error
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.496486-04:00 [error] <0.136.0>     reason: {badmatch,
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.496486-04:00 [error] <0.136.0>                 {error,
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.496486-04:00 [error] <0.136.0>                     {{shutdown,
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.496486-04:00 [error] <0.136.0>                          {failed_to_start_child,net_kernel,
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.496486-04:00 [error] <0.136.0>                              {'EXIT',nodistribution}}},
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.496486-04:00 [error] <0.136.0>                      {child,undefined,net_sup_dynamic,
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.496486-04:00 [error] <0.136.0>                          {erl_distribution,start_link,
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.496486-04:00 [error] <0.136.0>                              [#{name => rabbit_prelaunch_1015971@localhost,
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.496486-04:00 [error] <0.136.0>                                 supervisor => net_sup_dynamic,
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.496486-04:00 [error] <0.136.0>                                 net_tickintensity => 4,net_ticktime => 60,
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.496486-04:00 [error] <0.136.0>                                 name_domain => shortnames,
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.496486-04:00 [error] <0.136.0>                                 clean_halt => false}]},
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.496486-04:00 [error] <0.136.0>                          permanent,false,1000,supervisor,
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.496486-04:00 [error] <0.136.0>                          [erl_distribution]}}}}
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.496486-04:00 [error] <0.136.0>     offender: [{pid,undefined},
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.496486-04:00 [error] <0.136.0>                {id,prelaunch},
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.496486-04:00 [error] <0.136.0>                {mfargs,{rabbit_prelaunch,run_prelaunch_first_phase,[]}},
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.496486-04:00 [error] <0.136.0>                {restart_type,transient},
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.496486-04:00 [error] <0.136.0>                {significant,false},
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.496486-04:00 [error] <0.136.0>                {shutdown,5000},
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.496486-04:00 [error] <0.136.0>                {child_type,worker}]
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.496486-04:00 [error] <0.136.0>
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>   crasher:
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>     initial call: application_master:init/4
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>     pid: <0.134.0>
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>     registered_name: []
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>     exception exit: {{shutdown,
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>                       {failed_to_start_child,prelaunch,
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>                        {badmatch,
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>                         {error,
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>                          {{shutdown,
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>                            {failed_to_start_child,net_kernel,
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>                             {'EXIT',nodistribution}}},
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>                           {child,undefined,net_sup_dynamic,
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>                            {erl_distribution,start_link,
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>                             [#{name => rabbit_prelaunch_1015971@localhost,
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>                                supervisor => net_sup_dynamic,
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>                                net_tickintensity => 4,net_ticktime => 60,
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>                                name_domain => shortnames,
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>                                clean_halt => false}]},
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>                            permanent,false,1000,supervisor,
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>                            [erl_distribution]}}}}}},
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>                      {rabbit_prelaunch_app,start,[normal,[]]}}
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>       in function  application_master:init/4 (application_master.erl, line 142)
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>     ancestors: [<0.133.0>]
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>     message_queue_len: 1
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>     messages: [{'EXIT',<0.135.0>,normal}]
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>     links: [<0.133.0>,<0.44.0>]
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>     dictionary: []
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>     trap_exit: true
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>     status: running
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>     heap_size: 376
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>     stack_size: 28
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>     reductions: 165
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>   neighbours:
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497091-04:00 [error] <0.134.0>
Mar 19 01:24:45 tunninet-server-noel rabbitmq-server[1015971]: 2024-03-19 01:24:45.497704-04:00 [notice] <0.44.0> Application rabbitmq_prelaunch exited with reason: {{shutdown,{failed_to_start_child,prelaunch,{badmatch,{error,{{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}},{child,undefined,ne>
lines 1182-1273/1273 (END)

0 replies

noelashford · 2024-03-19T08:12:00Z

noelashford
Mar 19, 2024
Author

So odd, I installed with 0 issue on another box, on this box, i get the same error no matter what - what gives ;0

I installed this way in Ubuntu, 0 issues, but the host i need this on will not work. I do not understand what can be blocking it, the ports are not in use, the hosts files look pretty similar .... I do not get it ?

#!/bin/sh

sudo apt-get install curl gnupg apt-transport-https -y

## Team RabbitMQ's main signing key
curl -1sLf "https://keys.openpgp.org/vks/v1/by-fingerprint/0A9AF2115F4687BD29803A206B73A36E6026DFCA" | sudo gpg --dearmor | sudo tee /usr/share/keyrings/com.rabbitmq.team.gpg > /dev/null
## Community mirror of Cloudsmith: modern Erlang repository
curl -1sLf https://github.com/rabbitmq/signing-keys/releases/download/3.0/cloudsmith.rabbitmq-erlang.E495BB49CC4BBE5B.key | sudo gpg --dearmor | sudo tee /usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg > /dev/null
## Community mirror of Cloudsmith: RabbitMQ repository
curl -1sLf https://github.com/rabbitmq/signing-keys/releases/download/3.0/cloudsmith.rabbitmq-server.9F4587F226208342.key | sudo gpg --dearmor | sudo tee /usr/share/keyrings/rabbitmq.9F4587F226208342.gpg > /dev/null

## Add apt repositories maintained by Team RabbitMQ
sudo tee /etc/apt/sources.list.d/rabbitmq.list <<EOF
## Provides modern Erlang/OTP releases
##
deb [signed-by=/usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg] https://ppa1.novemberain.com/rabbitmq/rabbitmq-erlang/deb/ubuntu jammy main
deb-src [signed-by=/usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg] https://ppa1.novemberain.com/rabbitmq/rabbitmq-erlang/deb/ubuntu jammy main

# another mirror for redundancy
deb [signed-by=/usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg] https://ppa2.novemberain.com/rabbitmq/rabbitmq-erlang/deb/ubuntu jammy main
deb-src [signed-by=/usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg] https://ppa2.novemberain.com/rabbitmq/rabbitmq-erlang/deb/ubuntu jammy main

## Provides RabbitMQ
##
deb [signed-by=/usr/share/keyrings/rabbitmq.9F4587F226208342.gpg] https://ppa1.novemberain.com/rabbitmq/rabbitmq-server/deb/ubuntu jammy main
deb-src [signed-by=/usr/share/keyrings/rabbitmq.9F4587F226208342.gpg] https://ppa1.novemberain.com/rabbitmq/rabbitmq-server/deb/ubuntu jammy main

# another mirror for redundancy
deb [signed-by=/usr/share/keyrings/rabbitmq.9F4587F226208342.gpg] https://ppa2.novemberain.com/rabbitmq/rabbitmq-server/deb/ubuntu jammy main
deb-src [signed-by=/usr/share/keyrings/rabbitmq.9F4587F226208342.gpg] https://ppa2.novemberain.com/rabbitmq/rabbitmq-server/deb/ubuntu jammy main
EOF

## Update package indices
sudo apt-get update -y

## Install Erlang packages
sudo apt-get install -y erlang-base \
                        erlang-asn1 erlang-crypto erlang-eldap erlang-ftp erlang-inets \
                        erlang-mnesia erlang-os-mon erlang-parsetools erlang-public-key \
                        erlang-runtime-tools erlang-snmp erlang-ssl \
                        erlang-syntax-tools erlang-tftp erlang-tools erlang-xmerl

## Install rabbitmq-server and its dependencies
sudo apt-get install rabbitmq-server -y --fix-missing

0 replies

michaelklishin · 2024-03-19T15:06:00Z

michaelklishin
Mar 19, 2024
Maintainer

@noelashford all we can tell from the logs is that this node terminates very early on boot (during its pre-launch phase, in fact) with a nodistribution for error, that is to say, the node cannot initialize its essential networking and clustering libraries.

Reasons can vary greatly, our team does not have the capacity to investigate this for non-paying and non-contributing users.

Reasons can vary greatly:

epmd does not start due to a security policy (very common with SELinux on certain distributions)
The node cannot connect to epmd (see the Networking guide to find out what port must be open for connections)
The node cannot resolve its hostname

and so on. Security and networking defaults an vary greatly between distributions by default.

0 replies

noelashford · 2024-03-20T02:00:52Z

noelashford
Mar 20, 2024
Author

Thanks for any assistance, it is appreciated & I understand the workload - if you have any info on how to be a contributor please let me know.

It baffles me between two ubuntu 22.04 boxes, works on one, not on the other. Ports are not in use:

root@tunninet-server-noel:~# netstat -nltp | grep -E '(4369|5672|25672)'

Can resolve its name ok without any issue.

root@tunninet-server-noel:~# hostname -s
tunninet-server-noel

root@tunninet-server-noel:~# ping -c 4 $(hostname)
PING tunninet-server-noel (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.031 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.024 ms
64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=64 time=0.023 ms
64 bytes from localhost (127.0.0.1): icmp_seq=4 ttl=64 time=0.021 ms

Seems to be talking on various ports...

root@tunninet-server-noel:/var/log/rabbitmq# tcpdump -i any -n -vv port 4369 or 5672 or 25672 or port 5671
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
22:54:32.768632 lo    In  IP (tos 0x0, ttl 64, id 11196, offset 0, flags [DF], proto TCP (6), length 60)
    74.113.160.58.53656 > 192.168.5.1.5671: Flags [S], cksum 0xb083 (incorrect -> 0x8a28), seq 3687901529, win 65495, options [mss 65495,sackOK,TS val 1368462603 ecr 0,nop,wscale 7], length 0
22:54:32.768645 lo    In  IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.5.1.5671 > 192.168.5.1.53656: Flags [R.], cksum 0x7792 (correct), seq 0, ack 3687901530, win 0, length 0
22:54:32.770454 lo    In  IP (tos 0x0, ttl 64, id 25257, offset 0, flags [DF], proto TCP (6), length 60)
    74.113.160.58.53670 > 192.168.5.1.5671: Flags [S], cksum 0xb083 (incorrect -> 0x3b7e), seq 3588570591, win 65495, options [mss 65495,sackOK,TS val 1368462605 ecr 0,nop,wscale 7], length 0
22:54:32.770464 lo    In  IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.5.1.5671 > 192.168.5.1.53670: Flags [R.], cksum 0x28ea (correct), seq 0, ack 3588570592, win 0, length 0
22:54:32.790917 lo    In  IP (tos 0x0, ttl 64, id 27442, offset 0, flags [DF], proto TCP (6), length 60)
    74.113.160.58.53680 > 192.168.5.1.5671: Flags [S], cksum 0xb083 (incorrect -> 0x0376), seq 1303182849, win 65495, options [mss 65495,sackOK,TS val 1368462626 ecr 0,nop,wscale 7], length 0
22:54:32.790934 lo    In  IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.5.1.5671 > 192.168.5.1.53680: Flags [R.], cksum 0xf0f6 (correct), seq 0, ack 1303182850, win 0, length 0
22:54:32.791098 lo    In  IP (tos 0x0, ttl 64, id 24295, offset 0, flags [DF], proto TCP (6), length 60)
    74.113.160.58.53692 > 192.168.5.1.5671: Flags [S], cksum 0xb083 (incorrect -> 0xb24b), seq 2180127450, win 65495, options [mss 65495,sackOK,TS val 1368462626 ecr 0,nop,wscale 7], length 0
22:54:32.791104 lo    In  IP (tos 0x0, ttl 64, id 46702, offset 0, flags [DF], proto TCP (6), length 60)
    74.113.160.58.53706 > 192.168.5.1.5671: Flags [S], cksum 0xb083 (incorrect -> 0x582d), seq 382656526, win 65495, options [mss 65495,sackOK,TS val 1368462626 ecr 0,nop,wscale 7], length 0
22:54:32.791110 lo    In  IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.5.1.5671 > 192.168.5.1.53692: Flags [R.], cksum 0x9fcc (correct), seq 0, ack 2180127451, win 0, length 0
22:54:32.791113 lo    In  IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.5.1.5671 > 192.168.5.1.53706: Flags [R.], cksum 0x45ae (correct), seq 0, ack 382656527, win 0, length 0
22:54:32.826033 lo    In  IP (tos 0x0, ttl 64, id 15714, offset 0, flags [DF], proto TCP (6), length 60)
    74.113.160.58.53718 > 192.168.5.1.5671: Flags [S], cksum 0xb083 (incorrect -> 0xe8a3), seq 337662487, win 65495, options [mss 65495,sackOK,TS val 1368462661 ecr 0,nop,wscale 7], length 0
22:54:32.826045 lo    In  IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.5.1.5671 > 192.168.5.1.53718: Flags [R.], cksum 0xd647 (correct), seq 0, ack 337662488, win 0, length 0
22:54:32.866128 lo    In  IP (tos 0x0, ttl 64, id 16153, offset 0, flags [DF], proto TCP (6), length 60)
    74.113.160.58.35482 > 192.168.5.1.5671: Flags [S], cksum 0xb083 (incorrect -> 0x0757), seq 83003814, win 65495, options [mss 65495,sackOK,TS val 1368462701 ecr 0,nop,wscale 7], length 0
22:54:32.866140 lo    In  IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.5.1.5671 > 192.168.5.1.35482: Flags [R.], cksum 0xf522 (correct), seq 0, ack 83003815, win 0, length 0
22:54:32.896520 lo    In  IP (tos 0x0, ttl 64, id 45232, offset 0, flags [DF], proto TCP (6), length 60)
    74.113.160.58.35488 > 192.168.5.1.5671: Flags [S], cksum 0xb083 (incorrect -> 0x78cd), seq 198185262, win 65495, options [mss 65495,sackOK,TS val 1368462731 ecr 0,nop,wscale 7], length 0
22:54:32.896531 lo    In  IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.5.1.5671 > 192.168.5.1.35488: Flags [R.], cksum 0x66b7 (correct), seq 0, ack 198185263, win 0, length 0
22:54:32.896697 lo    In  IP (tos 0x0, ttl 64, id 15311, offset 0, flags [DF], proto TCP (6), length 60)
    74.113.160.58.35494 > 192.168.5.1.5671: Flags [S], cksum 0xb083 (incorrect -> 0x9003), seq 2171110489, win 65495, options [mss 65495,sackOK,TS val 1368462731 ecr 0,nop,wscale 7], length 0
22:54:32.896704 lo    In  IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.5.1.5671 > 192.168.5.1.35494: Flags [R.], cksum 0x7ded (correct), seq 0, ack 2171110490, win 0, length 0

I checked app armor to see if it is blocking anything, it is not and the port is open foe epmd. Nothing in complain mode or enforce mode for App Armor.

epmd runs ok...

root@tunninet-server-noel:/var/lib# epmd -debug
epmd: Tue Mar 19 20:15:25 2024: epmd running - daemon = 0
epmd: Tue Mar 19 20:15:25 2024: there is already a epmd running at port 4369 on ipaddr 0.0.0.0

I see in strace,

`

859248 ppoll([{fd=10, events=POLLIN|POLLRDNORM}], 1, NULL, NULL, 8 <unfinished ...>
859210 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 18
859210 fcntl(18, F_GETFL)               = 0x2 (flags O_RDWR)
859210 fcntl(18, F_SETFL, O_RDWR|O_NONBLOCK) = 0
859210 getsockopt(18, SOL_SOCKET, SO_PRIORITY, [0], [4]) = 0
859210 getsockopt(18, SOL_IP, IP_TOS, [0], [4]) = 0
859210 setsockopt(18, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
859210 setsockopt(18, SOL_IP, IP_TOS, [0], 4) = 0
859210 setsockopt(18, SOL_SOCKET, SO_PRIORITY, [0], 4) = 0
859210 getsockopt(18, SOL_SOCKET, SO_PRIORITY, [0], [4]) = 0
859210 getsockopt(18, SOL_IP, IP_TOS, [0], [4]) = 0
859210 setsockopt(18, SOL_TCP, TCP_NODELAY, [1], 4) = 0
859210 setsockopt(18, SOL_SOCKET, SO_PRIORITY, [0], 4) = 0
859210 bind(18, {sa_family=AF_INET, sin_port=htons(25672), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
859210 listen(18, 128)                  = 0
859210 getsockname(18, {sa_family=AF_INET, sin_port=htons(25672), sin_addr=inet_addr("0.0.0.0")}, [112 => 16]) = 0
859210 futex(0x7bb96622c2d0, FUTEX_WAKE_PRIVATE, 1) = 1
859206 <... futex resumed>)             = 0
859210 write(11, "!", 1 <unfinished ...>
859206 futex(0x7bb96622c2d0, FUTEX_WAIT_PRIVATE, 4294967295, {tv_sec=0, tv_nsec=648505632} <unfinished ...>
859248 <... ppoll resumed>)             = 1 ([{fd=10, revents=POLLIN|POLLRDNORM}])
859210 <... write resumed>)             = 1
859248 read(10,  <unfinished ...>
859210 futex(0x7bb96622c2d0, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
859248 <... read resumed>"!", 32)       = 1
859210 <... futex resumed>)             = 1
859206 <... futex resumed>)             = 0
859248 read(10,  <unfinished ...>
859206 futex(0x7bb96622c2d0, FUTEX_WAIT_PRIVATE, 4294967295, {tv_sec=0, tv_nsec=648376058} <unfinished ...>
859248 <... read resumed>0x7bb95d51cc70, 32) = -1 EAGAIN (Resource temporarily unavailable)
859210 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP <unfinished ...>
859248 ppoll([{fd=10, events=POLLIN|POLLRDNORM}], 1, NULL, NULL, 8 <unfinished ...>
859210 <... socket resumed>)            = 19
859210 fcntl(19, F_GETFL)               = 0x2 (flags O_RDWR)
859210 fcntl(19, F_SETFL, O_RDWR|O_NONBLOCK) = 0
859210 getsockopt(19, SOL_SOCKET, SO_PRIORITY, [0], [4]) = 0
859210 getsockopt(19, SOL_IP, IP_TOS, [0], [4]) = 0
859210 setsockopt(19, SOL_TCP, TCP_NODELAY, [1], 4) = 0
859210 setsockopt(19, SOL_SOCKET, SO_PRIORITY, [0], 4) = 0
859210 connect(19, {sa_family=AF_INET, sin_port=htons(4369), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress)
859210 epoll_ctl(4, EPOLL_CTL_ADD, 19, {events=EPOLLOUT|EPOLLONESHOT, data={u32=19, u64=136034499166227}}) = 0
859210 futex(0x7bb96622c2d0, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
859249 <... epoll_wait resumed>[{events=EPOLLOUT, data={u32=19, u64=136034499166227}}], 512, -1) = 1

859244 writev(1, [{iov_base="2024-03-19 21:23:08.050717-04:00 [notice] <0.153.0> Protocol 'inet_tcp': register/listen error: epmd_close\n2024-03-19 21:23:08.050717-04:00 [notice] <0.153.0> \n", iov_len=160}], 1 <unfinished ...>

859246 writev(1, [{iov_base="2024-03-19 21:23:08.075233-04:00 [error] <0.153.0>   crasher:\n2024-03-19 21:23:08.075233-04:00 [error] <0.153.0>     initial call: net_kernel:init/1\n2024-03-19 21:23:08.075233-04:00 [error] <0.153.0>     pid: <0.153.0>\n2024-03-19 21:23:08.075233-04:00 [error] <0.153.0>     registered_name: []\n2024-03-19 21:23:08.075233-04:00 [error] <0.153.0>     exception exit: {error,badarg}\n2024-03-19 21:23:08.075233-04:00 [error] <0.153.0>       **_in function  gen_server:init_it/6 (gen_server.erl, line 961)_**\n2024-03-19 21:23:08.075233-04:00 [error] <0.153.0>     ancestors: [net_sup,kernel_sup,<0.47.0>]\n2024-03-19 21:23:08.075233-04:00 [error] <0.153.0>     message_queue_len: 0\n2024-03-19 21:23:08.075233-04:00 [error] <0.153.0>     messages: []\n2024-03-19 21:23:08.075233-04:00 [error] <0.153.0>     links: [<0.150.0>,#Port<0.9>]\n2024-03-19 21:23:08.075233-04:00 [error] <0.153.0>     dictionary: [{longnames,false}]\n2024-03-19 21:23:08.075233-04:00 [error] <0.153.0>     trap_exit: true\n2024-03-19 21:23:08.075233-04:00 [error] <0"..., iov_len=1382}], 1) = 1382`

Investigating on my side but open to suggestions from the community.

`859247 access("/usr/lib/erlang/lib/kernel-9.2.2/ebin", W_OK <unfinished ...>
859248 ppoll([{fd=10, events=POLLIN|POLLRDNORM}], 1, NULL, NULL, 8 <unfinished ...>
859207 <... futex resumed>)             = 0
859206 <... futex resumed>)             = 1
859247 <... access resumed>)            = -1 EACCES (Permission denied)

859244 access("/usr/lib/erlang/lib/sasl-4.2.1/ebin", R_OK) = 0
859244 access("/usr/lib/erlang/lib/sasl-4.2.1/ebin", W_OK) = -1 EACCES (Permission denied)

lots of them. I ran this as root , root has R/W 644 permissions.

root@tunninet-server-noel:/var/log/rabbitmq# ls -la /usr/lib/erlang/lib/sasl-4.2.1/ebin
total 412
drwxr-xr-x 2 root root  4096 Mar 19 19:51 .
drwxr-xr-x 4 root root  4096 Mar 19 19:51 ..
-rw-r--r-- 1 root root  3192 Mar  7 12:05 alarm_handler.beam

checking is selinux stopped this... (Nope...)
root@tunninet-server-noel:/var/log/rabbitmq# ausearch -m avc -ts recent
<no matches>

Checking if its immutable ... (nope)
root@tunninet-server-noel:/var/log/rabbitmq# lsattr /usr/lib/erlang/lib/sasl-4.2.1/ebin
--------------e------- /usr/lib/erlang/lib/sasl-4.2.1/ebin/sasl_report_tty_h.beam
--------------e------- /usr/lib/erlang/lib/sasl-4.2.1/ebin/systools_make.beam
--------------e------- /usr/lib/erlang/lib/sasl-4.2.1/ebin/systools_rc.beam
--------------e------- /usr/lib/erlang/lib/sasl-4.2.1/ebin/systools_lib.beam
--------------e------- /usr/lib/erlang/lib/sasl-4.2.1/ebin/sasl_report_file_h.beam
--------------e------- /usr/lib/erlang/lib/sasl-4.2.1/ebin/systools_relup.beam
--------------e------- /usr/lib/erlang/lib/sasl-4.2.1/ebin/sasl.appup
--------------e------- /usr/lib/erlang/lib/sasl-4.2.1/ebin/sasl.app
--------------e------- /usr/lib/erlang/lib/sasl-4.2.1/ebin/release_handler.beam
--------------e------- /usr/lib/erlang/lib/sasl-4.2.1/ebin/rb_format_supp.beam
--------------e------- /usr/lib/erlang/lib/sasl-4.2.1/ebin/sasl.beam
--------------e------- /usr/lib/erlang/lib/sasl-4.2.1/ebin/systools.beam
--------------e------- /usr/lib/erlang/lib/sasl-4.2.1/ebin/erlsrv.beam
--------------e------- /usr/lib/erlang/lib/sasl-4.2.1/ebin/misc_supp.beam
--------------e------- /usr/lib/erlang/lib/sasl-4.2.1/ebin/alarm_handler.beam
--------------e------- /usr/lib/erlang/lib/sasl-4.2.1/ebin/rb.beam
--------------e------- /usr/lib/erlang/lib/sasl-4.2.1/ebin/format_lib_supp.beam
--------------e------- /usr/lib/erlang/lib/sasl-4.2.1/ebin/release_handler_1.beam
--------------e------- /usr/lib/erlang/lib/sasl-4.2.1/ebin/sasl_report.beam

app armor seems to be disabled... I dont get why i am seeing this.
`

0 replies

noelashford · 2024-03-20T08:45:46Z

noelashford
Mar 20, 2024
Author

Still stuck on:

Running command: '/usr/sbin/rabbitmq-server'
2024-03-20 04:43:46.043629-04:00 [notice] <0.153.0> Protocol 'inet_tcp': register/listen error: epmd_close
2024-03-20 04:43:46.043629-04:00 [notice] <0.153.0>
2024-03-20 04:43:46.049000-04:00 [error] <0.153.0>   crasher:
2024-03-20 04:43:46.049000-04:00 [error] <0.153.0>     initial call: net_kernel:init/1
2024-03-20 04:43:46.049000-04:00 [error] <0.153.0>     pid: <0.153.0>
2024-03-20 04:43:46.049000-04:00 [error] <0.153.0>     registered_name: []
2024-03-20 04:43:46.049000-04:00 [error] <0.153.0>     exception exit: {error,badarg}
2024-03-20 04:43:46.049000-04:00 [error] <0.153.0>       in function  gen_server:init_it/6 (gen_server.erl, line 961)
2024-03-20 04:43:46.049000-04:00 [error] <0.153.0>     ancestors: [net_sup,kernel_sup,<0.47.0>]
2024-03-20 04:43:46.049000-04:00 [error] <0.153.0>     message_queue_len: 0
2024-03-20 04:43:46.049000-04:00 [error] <0.153.0>     messages: []
2024-03-20 04:43:46.049000-04:00 [error] <0.153.0>     links: [<0.150.0>,#Port<0.11>]
2024-03-20 04:43:46.049000-04:00 [error] <0.153.0>     dictionary: [{longnames,false}]
2024-03-20 04:43:46.049000-04:00 [error] <0.153.0>     trap_exit: true
2024-03-20 04:43:46.049000-04:00 [error] <0.153.0>     status: running
2024-03-20 04:43:46.049000-04:00 [error] <0.153.0>     heap_size: 1598
2024-03-20 04:43:46.049000-04:00 [error] <0.153.0>     stack_size: 28
2024-03-20 04:43:46.049000-04:00 [error] <0.153.0>     reductions: 2975
2024-03-20 04:43:46.049000-04:00 [error] <0.153.0>   neighbours:
2024-03-20 04:43:46.049000-04:00 [error] <0.153.0>
2024-03-20 04:43:46.053793-04:00 [error] <0.150.0>     supervisor: {local,net_sup}
2024-03-20 04:43:46.053793-04:00 [error] <0.150.0>     errorContext: start_error
2024-03-20 04:43:46.053793-04:00 [error] <0.150.0>     reason: {'EXIT',nodistribution}
2024-03-20 04:43:46.053793-04:00 [error] <0.150.0>     offender: [{pid,undefined},
2024-03-20 04:43:46.053793-04:00 [error] <0.150.0>                {id,net_kernel},
2024-03-20 04:43:46.053793-04:00 [error] <0.150.0>                {mfargs,{net_kernel,start_link,
2024-03-20 04:43:46.053793-04:00 [error] <0.150.0>                                    [#{name => rabbit_prelaunch_735@localhost,
2024-03-20 04:43:46.053793-04:00 [error] <0.150.0>                                       supervisor => net_sup_dynamic,
2024-03-20 04:43:46.053793-04:00 [error] <0.150.0>                                       net_tickintensity => 4,
2024-03-20 04:43:46.053793-04:00 [error] <0.150.0>                                       net_ticktime => 60,
2024-03-20 04:43:46.053793-04:00 [error] <0.150.0>                                       name_domain => shortnames,
2024-03-20 04:43:46.053793-04:00 [error] <0.150.0>                                       clean_halt => false}]}},
2024-03-20 04:43:46.053793-04:00 [error] <0.150.0>                {restart_type,permanent},
2024-03-20 04:43:46.053793-04:00 [error] <0.150.0>                {significant,false},
2024-03-20 04:43:46.053793-04:00 [error] <0.150.0>                {shutdown,2000},
2024-03-20 04:43:46.053793-04:00 [error] <0.150.0>                {child_type,worker}]
2024-03-20 04:43:46.053793-04:00 [error] <0.150.0>

0 replies

noelashford · 2024-03-20T10:01:37Z

noelashford
Mar 20, 2024
Author

OK.... WOW! I found the issue and it was quite obscure... why ERLANG and EPMD care about this, IDK. The other day by mistake during testing of something on my FW i left on wide NAT for all interfaces and it thought that it was getting a foreign connection as NAT was messing with it.... I narrowed NAT to my WAN (as it should be) and WOILA! Sorted! Please add this to you FAQ guys ;0 It could save someone else 2 days of insanity ;0

2 replies

noelashford Mar 20, 2024
Author

This is the solution in my case as seen by it now being up

Up 3 minutes (healthy) rabbitmq

dormanze May 15, 2024

I also encountered this problem. It seems that the new version of erlang or rabbitmq is more strict with the node name and network name. You can try the following steps to reproduce the problem:

Start two RabbitMQ instances to form a cluster.
Stop both instances and swap the two nodes in /etc/hosts
Start two instances.
example:
The configuration in /etc/hosts at the first startup is:

172.18.0.115 rabbitmqservice-1
172.18.0.131 rabbitmqservice-0

The new configuration after swapping the two nodes is:

172.18.0.115 rabbitmqservice-0
172.18.0.131 rabbitmqservice-1

I didn't have this problem in rabbitmq 3.11.9+erlang 25.22, when I upgraded to rabbitmq 3.13+erlang 26.22 the problem appeared.
This problem easily occurs in the container.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RabbitMQ Crashing with bad_environment_value /etc/rabbitmq/erl_inetrc #10777

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 11 comments 4 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

RabbitMQ Crashing with bad_environment_value /etc/rabbitmq/erl_inetrc #10777

noelashford Mar 19, 2024

Describe the bug

Reproduction steps

Expected behavior

Additional context

Replies: 11 comments · 4 replies

lukebakken Mar 19, 2024 Maintainer

noelashford Mar 19, 2024 Author

lukebakken Mar 19, 2024 Maintainer

noelashford Mar 19, 2024 Author

noelashford Mar 19, 2024 Author

noelashford Mar 19, 2024 Author

noelashford Mar 19, 2024 Author

noelashford Mar 19, 2024 Author

noelashford Mar 19, 2024 Author

michaelklishin Mar 19, 2024 Maintainer

noelashford Mar 20, 2024 Author

noelashford Mar 20, 2024 Author

noelashford Mar 20, 2024 Author

noelashford Mar 20, 2024 Author

dormanze May 15, 2024

noelashford
Mar 19, 2024

Replies: 11 comments 4 replies

lukebakken
Mar 19, 2024
Maintainer

noelashford
Mar 19, 2024
Author

lukebakken Mar 19, 2024
Maintainer

noelashford Mar 19, 2024
Author

noelashford
Mar 19, 2024
Author

noelashford
Mar 19, 2024
Author

noelashford
Mar 19, 2024
Author

noelashford
Mar 19, 2024
Author

noelashford
Mar 19, 2024
Author

michaelklishin
Mar 19, 2024
Maintainer

noelashford
Mar 20, 2024
Author

noelashford
Mar 20, 2024
Author

noelashford
Mar 20, 2024
Author

noelashford Mar 20, 2024
Author