Docker Swarm new node communication

[gorkaandramari]

Description

Hello everyone,

I have a big issue during 4 weeks. I am trying to attack a node to a swarm with 6 manager nodes, so the new one will be the 7th.

I made a new attacheable overlayer network, so the nodes could communicate between it.

When I am trying to join with manager token, it suceeses but when I type '''docker network ls''', there isn't the the overlay network , I created.

Moreover, when I type "journalctl -xe -u docker" I have this message (between other same messages):

time="2024-05-03T15:43:16.345990049Z" level=error msg="error sending message to peer" error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp x.x.x.x:2377: i/o timeout\""
may 03 15:43:16 bfpe007 dockerd[880]: time="2024-05-03T15:43:16.346095028Z" level=error msg="received message intended for raft_id 9c6be28726ae35c" from=738175eab0a1e608 method="(*Node).ProcessRaftMessage" raft_id=45e6d605b899e60a
may 03 15:43:17 bfpe007 dockerd[880]: time="2024-05-03T15:43:17.345881765Z" level=error msg="error sending message to peer" error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp x.x.x.x:2377: i/o timeout\""
may 03 15:43:17 bfpe007 dockerd[880]: time="2024-05-03T15:43:17.347168013Z" level=error msg="received message intended for raft_id 9c6be28726ae35c" from=738175eab0a1e608 method="(*Node).ProcessRaftMessage" raft_id=45e6d605b899e60a
may 03 15:43:18 bfpe007 dockerd[880]: time="2024-05-03T15:43:18.345104129Z" level=error msg="error sending message to peer" error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp x.x.x.x:2377: i/o timeout\""
may 03 15:43:18 bfpe007 dockerd[880]: time="2024-05-03T15:43:18.346405222Z" level=error msg="received message intended for raft_id 9c6be28726ae35c" from=738175eab0a1e608 method="(*Node).ProcessRaftMessage" raft_id=45e6d605b899e60a
may 03 15:43:19 bfpe007 dockerd[880]: time="2024-05-03T15:43:19.345813928Z" level=error msg="error sending message to peer" error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp x.x.x.x:2377: i/o timeout\""
may 03 15:43:19 bfpe007 dockerd[880]: time="2024-05-03T15:43:19.347186506Z" level=error msg="received message intended for raft_id 9c6be28726ae35c" from=738175eab0a1e608 method="(*Node).ProcessRaftMessage" raft_id=45e6d605b899e60a
may 03 15:43:20 bfpe007 dockerd[880]: time="2024-05-03T15:43:20.345919423Z" level=error msg="error sending message to peer" error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp x.x.x.x:2377: i/o timeout\""
may 03 15:43:20 bfpe007 dockerd[880]: time="2024-05-03T15:43:20.347237879Z" level=error msg="received message intended for raft_id 9c6be28726ae35c" from=738175eab0a1e608 method="(*Node).ProcessRaftMessage" raft_id=45e6d605b899e60a
may 03 15:43:21 bfpe007 dockerd[880]: time="2024-05-03T15:43:21.344940633Z" level=error msg="error sending message to peer" error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp x.x.x.x:2377: i/o timeout\""
may 03 15:43:21 bfpe007 dockerd[880]: time="2024-05-03T15:43:21.346235439Z" level=error msg="received message intended for raft_id 9c6be28726ae35c" from=738175eab0a1e608 method="(*Node).ProcessRaftMessage" raft_id=45e6d605b899e60a
may 03 15:43:22 bfpe007 dockerd[880]: time="2024-05-03T15:43:22.346219674Z" level=error msg="error sending message to peer" error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp x.x.x.x:2377: i/o timeout\""
may 03 15:43:22 bfpe007 dockerd[880]: time="2024-05-03T15:43:22.347220877Z" level=error msg="received message intended for raft_id 9c6be28726ae35c" from=738175eab0a1e608 method="(*Node).ProcessRaftMessage" raft_id=45e6d605b899e60a
may 03 15:43:23 bfpe007 dockerd[880]: time="2024-05-03T15:43:23.344809271Z" level=error msg="error sending message to peer" error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp x.x.x.x:2377: i/o timeout\""
may 03 15:43:23 bfpe007 dockerd[880]: time="2024-05-03T15:43:23.346101598Z" level=error msg="received message intended for raft_id 9c6be28726ae35c" from=738175eab0a1e608 method="(*Node).ProcessRaftMessage" raft_id=45e6d605b899e60a
may 03 15:43:24 bfpe007 dockerd[880]: time="2024-05-03T15:43:24.346000252Z" level=error msg="error sending message to peer" error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp x.x.x.x:2377: i/o timeout\""
may 03 15:43:40 bfpe007 dockerd[880]: time="2024-05-03T15:43:40.351555659Z" level=error msg="error sending message to peer" error="rpc error: code = Canceled desc = context canceled"
may 03 15:43:41 bfpe007 dockerd[880]: time="2024-05-03T15:43:41.347347845Z" level=error msg="received message intended for raft_id 9c6be28726ae35c" from=738175eab0a1e608 method="(*Node).ProcessRaftMessage" raft_id=45e6d605b899e60a
may 03 15:43:41 bfpe007 dockerd[880]: time="2024-05-03T15:43:41.564118284Z" level=warning msg="bulk sync to node c5e33285dfc7 failed: failed to send a TCP message during bulk sync: dial tcp x.x.x.x:7946: i/o timeout"
may 03 15:43:42 bfpe007 dockerd[880]: time="2024-05-03T15:43:42.346325727Z" level=error msg="received message intended for raft_id 9c6be28726ae35c" from=738175eab0a1e608 method="(*Node).ProcessRaftMessage" raft_id=45e6d605b899e60a
may 03 15:43:42 bfpe007 dockerd[880]: time="2024-05-03T15:43:42.352700907Z" level=error msg="error sending message to peer" error="rpc error: code = Canceled desc = context canceled"
may 03 15:43:43 bfpe007 dockerd[880]: time="2024-05-03T15:43:43.346284378Z" level=error msg="received message intended for raft_id 9c6be28726ae35c" from=738175eab0a1e608 method="(*Node).ProcessRaftMessage" raft_id=45e6d605b899e60a

I attach docker info:

Client: Docker Engine - Community
 Version:    24.0.7
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.27.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 24.0.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: j5k49ysr1cu35ii191shg5c2z
  Error: rpc error: code = DeadlineExceeded desc = context deadline exceeded
  Is Manager: true
  Node Address: 150.241.255.66
  Manager Addresses:
   X.X.X.X:2377
   X.X.X.X.:2377
   X.X.X.X:2377
   X.X.X.X.X:2377
   ....

 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e377cd56a71523140ca6ae87e30244719194a521
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-105-generic
 Operating System: Ubuntu 22.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 31.07GiB
 Name: bfpe007
 ID: 13dc6ec9-7f92-4c25-aadb-c8caf1f0ff8e
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

It seems to have problem communicating between nodes, but I have my ports opened in the network and in my nodes. For that reason, It can find the overlay network.

Someone could help me please?? Thank you

P.D.: I dont know if i paste correctly the code of journalctl. I hope...

Edited: What information do you need? I will add whatever you need. Thank you