Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Uncatchable NetworkError 1006 #22493

Open
mayrsascha opened this issue May 14, 2024 · 8 comments
Open

Uncatchable NetworkError 1006 #22493

mayrsascha opened this issue May 14, 2024 · 8 comments
Assignees
@carlosmiei
Labels
bug

Comments

@mayrsascha
Copy link

mayrsascha commented May 14, 2024 •
edited

Operating System

MacOS Sonoma

Programming Languages

JavaScript

CCXT Version

4.3.22

Description

I'm trying to get OHLCs for 100s of coins, please see code below on how I'm achieving that.

Despite wrapping all network calls in a try/catch I'm still getting frequent uncaught NetworkErrors, like this one:

NetworkError connection closed by remote server, closing code 1006 
    file:///opt/render/project/src/node_modules/ccxt/js/src/base/ws/Client.js:215:24 WsClient.onClose
    node_modules/ccxt/node_modules/ws/lib/event-target.js:290:14 callListener
    node_modules/ccxt/node_modules/ws/lib/event-target.js:220:9 WebSocket.onClose
    node:events:519:28 WebSocket.emit
    node:domain:488:12 WebSocket.emit
    node_modules/ccxt/node_modules/ws/lib/websocket.js:265:10 WebSocket.emitClose
    node_modules/ccxt/node_modules/ws/lib/websocket.js:1198:20 Receiver.receiverOnFinish
    node:events:519:28 Receiver.emit
    node:domain:488:12 Receiver.emit
    node:internal/streams/writable:946:10 finish

Is there another way to handle these?

Also candles seem to take a while to get in, am I doing this right or are the many infinite loops blocking data fetching in your experience?

Code

const watchCoin = async ({ exchange, symbol }) => {
   while (true) {
     try {
       const candles = await exchange.watchOHLCV(symbol, '1h', undefined, 1)
       const lastCandle = candles[0]
       // Use candles
     } catch(e) {
       // Send to error tracker
     }
  }
}

const exchanges = [
    ccxt.pro.bybit,
    ccxt.pro.kraken,
    ccxt.pro.binance,
    ccxt.pro.okx,
    ccxt.pro.gate,
    ccxt.pro.kucoin,
    ccxt.pro.bitfinex2,
    ccxt.pro.htx,
    ccxt.pro.bingx,
    ccxt.pro.hitbtc,
    ccxt.pro.bitget
]
for (const exchange of exchanges) {
  const instance = new exchange()
  try {
    exchange.loadMarkets()
  } catch(e) {
     // Send to error tracker
     continue;
  }

  const supportedSymbols = exchange.symbols.filter(symbol => // Check if symbol matches a coin pair I want to track)
  for (const symbol of supportedSymbol) {
     watchCoin({ exchange, symbol })
  }
}
@carlosmiei
Copy link
Collaborator

Hello @mayrsascha, thanks for reporting it, we will take a look.

@carlosmiei carlosmiei self-assigned this May 14, 2024
@carlosmiei carlosmiei added the bug label May 14, 2024
@mayrsascha
Copy link
Author

mayrsascha commented May 14, 2024 •
edited

Thank you @carlosmiei !

Some extra information that just came to mind:

  1. These errors only happen when I keep the app running for some hours, so don't expect this to be seen immediately. These errors happen 2-3 hours on average on my side.
  2. I did research the issue tracker here and found comparable issues, and while they didn't help me to fix this, they might aid a solution here:

#14086
#14557
And especially #15338

Let me know if there is anything else I can provide, thank you!

@andrewmichael042c
Copy link

Hint: You are currently not seeing messages from other users and the system.
Users in groups 'adm', 'systemd-journal' can see all messages.
Pass -q to turn off this notice.
-- Logs begin at Fri 2022-07-15 08:53:10 UTC, end at Mon 2022-08-01 08:45:41 UTC. --
Jul 31 12:28:32.744196 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: Bad Request: http://example.com/
Jul 31 12:28:32.744636 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: Bad Request: http://example.com/
Jul 31 12:28:32.745153 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: INFO: 2.57.122.209:39724 - "GET http%3A//example.com/ HTTP/1.1" 400 Bad Request
Jul 31 15:05:26.571294 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: WARNING: Invalid HTTP request received.
Jul 31 15:05:26.571294 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: Traceback (most recent call last):
Jul 31 15:05:26.571294 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: File "/home/myuser/env/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 136, in handle_events
Jul 31 15:05:26.571294 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: event = self.conn.next_event()
Jul 31 15:05:26.571294 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: File "/home/myuser/env/lib/python3.8/site-packages/h11/_connection.py", line 485, in next_event
Jul 31 15:05:26.571294 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: exc._reraise_as_remote_protocol_error()
Jul 31 15:05:26.571294 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: File "/home/myuser/env/lib/python3.8/site-packages/h11/_util.py", line 77, in _reraise_as_remote_protocol_error
Jul 31 15:05:26.571294 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: raise self
Jul 31 15:05:26.571294 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: File "/home/myuser/env/lib/python3.8/site-packages/h11/_connection.py", line 467, in next_event
Jul 31 15:05:26.571294 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: event = self._extract_next_receive_event()
Jul 31 15:05:26.571294 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: File "/home/myuser/env/lib/python3.8/site-packages/h11/_connection.py", line 409, in _extract_next_receive_event
Jul 31 15:05:26.571294 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: event = self._reader(self._receive_buffer)
Jul 31 15:05:26.571294 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: File "/home/myuser/env/lib/python3.8/site-packages/h11/_readers.py", line 81, in maybe_read_from_IDLE_client
Jul 31 15:05:26.571294 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: raise LocalProtocolError("illegal request line")
Jul 31 15:05:26.571294 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: h11._util.RemoteProtocolError: illegal request line
Jul 31 15:05:26.575159 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: Invalid HTTP request received.
Jul 31 15:05:26.575159 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: Traceback (most recent call last):
Jul 31 15:05:26.575159 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: File "/home/myuser/env/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 136, in handle_events
Jul 31 15:05:26.575159 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: event = self.conn.next_event()
Jul 31 15:05:26.575159 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: File "/home/myuser/env/lib/python3.8/site-packages/h11/_connection.py", line 485, in next_event
Jul 31 15:05:26.575159 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: exc._reraise_as_remote_protocol_error()
Jul 31 15:05:26.575159 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: File "/home/myuser/env/lib/python3.8/site-packages/h11/_util.py", line 77, in _reraise_as_remote_protocol_error
Jul 31 15:05:26.575159 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: raise self
Jul 31 15:05:26.575159 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: File "/home/myuser/env/lib/python3.8/site-packages/h11/_connection.py", line 467, in next_event
Jul 31 15:05:26.575159 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: event = self._extract_next_receive_event()
Jul 31 15:05:26.575159 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: File "/home/myuser/env/lib/python3.8/site-packages/h11/_connection.py", line 409, in _extract_next_receive_event
Jul 31 15:05:26.575159 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: event = self._reader(self._receive_buffer)
Jul 31 15:05:26.575159 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: File "/home/myuser/env/lib/python3.8/site-packages/h11/_readers.py", line 81, in maybe_read_from_IDLE_client
Jul 31 15:05:26.575159 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: raise LocalProtocolError("illegal request line")
Jul 31 15:05:26.575159 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: h11._util.RemoteProtocolError: illegal request line
Jul 31 15:18:29.284663 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: [error ] WebSocket close with status code 1006 [trading.methods] method=watch_ticker task=76c worker=5
Jul 31 15:18:29.312579 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: [error ] Traceback (most recent call last):
Jul 31 15:18:29.312579 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: File "/var/www/bot/trading/methods.py", line 71, in method_loop
Jul 31 15:18:29.312579 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: response = await getattr(client, method)(**args)
Jul 31 15:18:29.312579 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: File "/home/myuser/env/lib/python3.8/site-packages/ccxtpro/ftx.py", line 111, in watch_ticker
Jul 31 15:18:29.312579 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: return await self.watch_public(symbol, 'ticker')
Jul 31 15:18:29.312579 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: File "/home/myuser/env/lib/python3.8/site-packages/ccxtpro/ftx.py", line 64, in watch_public
Jul 31 15:18:29.312579 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: return await self.watch(url, messageHash, request, messageHash)
Jul 31 15:18:29.312579 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: ccxt.base.errors.NetworkError: 1006
Jul 31 15:18:29.312579 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: [trading.methods] task=76c worker=5
Jul 31 15:34:51.904863 ubuntu-1cpu-2gb-pl-waw1 systemd[307421]: Stopped target Main User Target.
Jul 31 15:34:51.904879 ubuntu-1cpu-2gb-pl-waw1 systemd[307421]: Stopped target Basic System.
Jul 31 15:34:51.904888 ubuntu-1cpu-2gb-pl-waw1 systemd[307421]: Stopped target Paths.
Jul 31 15:34:51.904896 ubuntu-1cpu-2gb-pl-waw1 systemd[307421]: Stopped target Sockets.
Jul 31 15:34:51.904906 ubuntu-1cpu-2gb-pl-waw1 systemd[307421]: Stopped target Timers.
Jul 31 15:34:51.904995 ubuntu-1cpu-2gb-pl-waw1 systemd[307421]: dbus.socket: Succeeded.
Jul 31 15:34:51.905560 ubuntu-1cpu-2gb-pl-waw1 systemd[307421]: Closed D-Bus User Message Bus Socket.
Jul 31 15:34:51.905595 ubuntu-1cpu-2gb-pl-waw1 systemd[307421]: dirmngr.socket: Succeeded.
Jul 31 15:34:51.905645 ubuntu-1cpu-2gb-pl-waw1 systemd[307421]: Closed GnuPG network certificate management daemon.
Jul 31 15:34:51.905670 ubuntu-1cpu-2gb-pl-waw1 systemd[307421]: gpg-agent-browser.socket: Succeeded.
Jul 31 15:34:51.905717 ubuntu-1cpu-2gb-pl-waw1 systemd[307421]: Closed GnuPG cryptographic agent and passphrase cache (access for web browsers).
Jul 31 15:34:51.905759 ubuntu-1cpu-2gb-pl-waw1 systemd[307421]: gpg-agent-extra.socket: Succeeded.
Jul 31 15:34:51.905813 ubuntu-1cpu-2gb-pl-waw1 systemd[307421]: Closed GnuPG cryptographic agent and passphrase cache (restricted).
Jul 31 15:34:51.905841 ubuntu-1cpu-2gb-pl-waw1 systemd[307421]: gpg-agent-ssh.socket: Succeeded.
Jul 31 15:34:51.905888 ubuntu-1cpu-2gb-pl-waw1 systemd[307421]: Closed GnuPG cryptographic agent (ssh-agent emulation).
Jul 31 15:34:51.905916 ubuntu-1cpu-2gb-pl-waw1 systemd[307421]: gpg-agent.socket: Succeeded.
Jul 31 15:34:51.905958 ubuntu-1cpu-2gb-pl-waw1 systemd[307421]: Closed GnuPG cryptographic agent and passphrase cache.
Jul 31 15:34:51.905979 ubuntu-1cpu-2gb-pl-waw1 systemd[307421]: pk-debconf-helper.socket: Succeeded.
Jul 31 15:34:51.906016 ubuntu-1cpu-2gb-pl-waw1 systemd[307421]: Closed debconf communication socket.
Jul 31 15:34:51.906036 ubuntu-1cpu-2gb-pl-waw1 systemd[307421]: snapd.session-agent.socket: Succeeded.
Jul 31 15:34:51.906072 ubuntu-1cpu-2gb-pl-waw1 systemd[307421]: Closed REST API socket for snapd user session agent.
Jul 31 15:34:51.906083 ubuntu-1cpu-2gb-pl-waw1 systemd[307421]: Reached target Shutdown.
Jul 31 15:34:51.906133 ubuntu-1cpu-2gb-pl-waw1 systemd[307421]: systemd-exit.service: Succeeded.
Jul 31 15:34:51.906163 ubuntu-1cpu-2gb-pl-waw1 systemd[307421]: Finished Exit the Session.
Jul 31 15:34:51.906174 ubuntu-1cpu-2gb-pl-waw1 systemd[307421]: Reached target Exit the Session.
Jul 31 15:41:08.896061 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: [error ] WebSocket close with status code Connection closed by remote server, closing code 1001 [trading.methods] method=watchMyTrades task=76c worker=5
Jul 31 15:41:08.898394 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: [error ] Traceback (most recent call last):
Jul 31 15:41:08.898394 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: File "/var/www/bot/trading/methods.py", line 56, in method_loop
Jul 31 15:41:08.898394 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: response = await getattr(client, method)()
Jul 31 15:41:08.898394 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: File "/home/myuser/env/lib/python3.8/site-packages/ccxtpro/ftx.py", line 449, in watch_my_trades
Jul 31 15:41:08.898394 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: trades = await self.watch_private('fills', symbol)
Jul 31 15:41:08.898394 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: File "/home/myuser/env/lib/python3.8/site-packages/ccxtpro/ftx.py", line 78, in watch_private
Jul 31 15:41:08.898394 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: return await self.watch(url, messageHash, request, channel)
Jul 31 15:41:08.898394 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: ccxt.base.errors.NetworkError: Connection closed by remote server, closing code 1001
Jul 31 15:41:08.898394 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: [trading.methods] task=76c worker=5
Jul 31 17:11:01.958298 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: WARNING: Invalid HTTP request received.
Jul 31 17:11:01.958298 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: Traceback (most recent call last):
Jul 31 17:11:01.958298 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: File "/home/myuser/env/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 136, in handle_events
Jul 31 17:11:01.958298 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: event = self.conn.next_event()
Jul 31 17:11:01.958298 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: File "/home/myuser/env/lib/python3.8/site-packages/h11/_connection.py", line 485, in next_event
Jul 31 17:11:01.958298 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: exc._reraise_as_remote_protocol_error()
Jul 31 17:11:01.958298 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: File "/home/myuser/env/lib/python3.8/site-packages/h11/_util.py", line 77, in _reraise_as_remote_protocol_error
Jul 31 17:11:01.958298 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: raise self
Jul 31 17:11:01.958298 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: File "/home/myuser/env/lib/python3.8/site-packages/h11/_connection.py", line 467, in next_event
Jul 31 17:11:01.958298 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: event = self._extract_next_receive_event()
Jul 31 17:11:01.958298 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: File "/home/myuser/env/lib/python3.8/site-packages/h11/_connection.py", line 409, in _extract_next_receive_event
Jul 31 17:11:01.958298 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: event = self._reader(self._receive_buffer)
Jul 31 17:11:01.958298 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: File "/home/myuser/env/lib/python3.8/site-packages/h11/_readers.py", line 81, in maybe_read_from_IDLE_client
Jul 31 17:11:01.958298 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: raise LocalProtocolError("illegal request line")
Jul 31 17:11:01.958298 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: h11._util.RemoteProtocolError: illegal request line
Jul 31 17:11:01.962569 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: Invalid HTTP request received.
Jul 31 17:11:01.962569 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: Traceback (most recent call last):
Jul 31 17:11:01.962569 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: File "/home/myuser/env/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 136, in handle_events
Jul 31 17:11:01.962569 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: event = self.conn.next_event()
Jul 31 17:11:01.962569 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: File "/home/myuser/env/lib/python3.8/site-packages/h11/_connection.py", line 485, in next_event
Jul 31 17:11:01.962569 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: exc._reraise_as_remote_protocol_error()
Jul 31 17:11:01.962569 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: File "/home/myuser/env/lib/python3.8/site-packages/h11/_util.py", line 77, in _reraise_as_remote_protocol_error
Jul 31 17:11:01.962569 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: raise self
Jul 31 17:11:01.962569 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: File "/home/myuser/env/lib/python3.8/site-packages/h11/_connection.py", line 467, in next_event
Jul 31 17:11:01.962569 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: event = self._extract_next_receive_event()
Jul 31 17:11:01.962569 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: File "/home/myuser/env/lib/python3.8/site-packages/h11/_connection.py", line 409, in _extract_next_receive_event
Jul 31 17:11:01.962569 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: event = self._reader(self._receive_buffer)
Jul 31 17:11:01.962569 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: File "/home/myuser/env/lib/python3.8/site-packages/h11/_readers.py", line 81, in maybe_read_from_IDLE_client
Jul 31 17:11:01.962569 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: raise LocalProtocolError("illegal request line")
Jul 31 17:11:01.962569 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: h11._util.RemoteProtocolError: illegal request line
Jul 31 17:38:47.506035 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: [error ] WebSocket close with status code 1006 [trading.methods] method=watchOrders task=76c worker=5
Jul 31 17:38:47.511137 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: [error ] Traceback (most recent call last):
Jul 31 17:38:47.511137 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: File "/var/www/bot/trading/methods.py", line 56, in method_loop
Jul 31 17:38:47.511137 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: response = await getattr(client, method)()
Jul 31 17:38:47.511137 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: File "/home/myuser/env/lib/python3.8/site-packages/ccxtpro/ftx.py", line 373, in watch_orders
Jul 31 17:38:47.511137 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: orders = await self.watch_private('orders', symbol)
Jul 31 17:38:47.511137 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: File "/home/myuser/env/lib/python3.8/site-packages/ccxtpro/ftx.py", line 78, in watch_private
Jul 31 17:38:47.511137 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: return await self.watch(url, messageHash, request, channel)
Jul 31 17:38:47.511137 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: ccxt.base.errors.NetworkError: 1006
Jul 31 17:38:47.511137 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: [trading.methods] task=76c worker=5
Jul 31 17:46:24.560874 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: [error ] WebSocket close with status code Connection closed by remote server, closing code 1001 [trading.methods] method=watchOrders task=76c worker=5
Jul 31 17:46:24.563119 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: [error ] Traceback (most recent call last):
Jul 31 17:46:24.563119 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: File "/var/www/bot/trading/methods.py", line 56, in method_loop
Jul 31 17:46:24.563119 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: response = await getattr(client, method)()
Jul 31 17:46:24.563119 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: File "/home/myuser/env/lib/python3.8/site-packages/ccxtpro/ftx.py", line 373, in watch_orders
Jul 31 17:46:24.563119 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: orders = await self.watch_private('orders', symbol)
Jul 31 17:46:24.563119 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: File "/home/myuser/env/lib/python3.8/site-packages/ccxtpro/ftx.py", line 78, in watch_private
Jul 31 17:46:24.563119 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: return await self.watch(url, messageHash, request, channel)
Jul 31 17:46:24.563119 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: ccxt.base.errors.NetworkError: Connection closed by remote server, closing code 1001
Jul 31 17:46:24.563119 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: [trading.methods] task=76c worker=5
Jul 31 18:05:23.054154 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: [error ] WebSocket close with status code 1006 [trading.methods] method=watch_ticker task=76c worker=5
Jul 31 18:05:23.059278 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: [error ] Traceback (most recent call last):
Jul 31 18:05:23.059278 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: File "/var/www/bot/trading/methods.py", line 71, in method_loop
Jul 31 18:05:23.059278 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: response = await getattr(client, method)(**args)
Jul 31 18:05:23.059278 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: File "/home/myuser/env/lib/python3.8/site-packages/ccxtpro/ftx.py", line 111, in watch_ticker
Jul 31 18:05:23.059278 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: return await self.watch_public(symbol, 'ticker')
Jul 31 18:05:23.059278 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: File "/home/myuser/env/lib/python3.8/site-packages/ccxtpro/ftx.py", line 64, in watch_public
Jul 31 18:05:23.059278 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: return await self.watch(url, messageHash, request, messageHash)
Jul 31 18:05:23.059278 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: ccxt.base.errors.NetworkError: 1006
Jul 31 18:05:23.059278 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: [trading.methods] task=76c worker=5
Jul 31 18:15:09.822529 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: [error ] WebSocket close with status code Connection closed by remote server, closing code 1001 [trading.methods] method=watchMyTrades task=76c worker=5
Jul 31 18:15:09.824258 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: [error ] Traceback (most recent call last):
Jul 31 18:15:09.824258 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: File "/var/www/bot/trading/methods.py", line 56, in method_loop
Jul 31 18:15:09.824258 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: response = await getattr(client, method)()
Jul 31 18:15:09.824258 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: File "/home/myuser/env/lib/python3.8/site-packages/ccxtpro/ftx.py", line 449, in watch_my_trades
Jul 31 18:15:09.824258 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: trades = await self.watch_private('fills', symbol)
Jul 31 18:15:09.824258 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: File "/home/myuser/env/lib/python3.8/site-packages/ccxtpro/ftx.py", line 78, in watch_private
Jul 31 18:15:09.824258 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: return await self.watch(url, messageHash, request, channel)
Jul 31 18:15:09.824258 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: ccxt.base.errors.NetworkError: Connection closed by remote server, closing code 1001
Jul 31 18:15:09.824258 ubuntu-1cpu-2gb-pl-waw1 celery[308783]: [trading.methods] task=76c worker=5
Jul 31 18:35:25.718635 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: Invalid HTTP_HOST header: 'example.com'. You may need to add 'example.com' to ALLOWED_HOSTS.
Jul 31 18:35:25.721465 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: Invalid HTTP_HOST header: 'example.com'. You may need to add 'example.com' to ALLOWED_HOSTS.
Jul 31 18:35:25.741688 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: Bad Request: http://example.com/
Jul 31 18:35:25.742039 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: Bad Request: http://example.com/
Jul 31 18:35:25.742469 ubuntu-1cpu-2gb-pl-waw1 uvicorn[308764]: INFO: 45.148.10.81:34626 - "GET http%3A//example.com/ HTTP/1.1" 400 Bad Request

@ragamufin
Copy link

Also having this problem. Everything works for a while and with verbose true I can see 'sending' > {method: 'ping'} and then 'onMessage' > {id: 0, code: 0, msg: 'PONG'} but every now and then there will be an onClose error and I get either:
connection closed by remote server, closing code 1006
or
timed out due to a ping-pong keepalive missing on time

I'm using ccxt 4.2.79 and going against mexc. It happens sporadically

@ragamufin ragamufin mentioned this issue May 19, 2024
Open
@pcriadoperez
Copy link
Contributor

Hi,

I'm trying to reproduce the error but haven't been successfull so far.

  • I've tried simulating client errors by calling client.onError and client.onReset
  • Received errors 1000, 1006, Timeout errors and Rate limit errors from the exchange but all have been catchable.
  • And left the script running for a couple hours
    But no luck so far.

@ragamufin , the error you are seeing are they also uncatchable?

@mayrsascha, are you seeing uncatchable error consistently where it exits the script with the code you provided? How many symbols are you watching per exchange?

@ragamufin
Copy link

@pcriadoperez In my case I was able to narrow down the errors to watchBalance which is the only websocket method I use. The code could run for a day with no error and then at other times within a few hours it errors

Ever since I've surrounded it with a try - catch, I've seen the errors happen about 4 or so times so far and they were caught successfully and the code resumes no problem

I've noticed that if some other part of the code takes up the thread with heavy processing such as writing a lot of data to local storage then errors can happen with the websocket connection. So I wonder if using a worker to run the watch methods on their own thread might be away to avoid the problem more efficiently. For now though, the fact that I can catch it is good enough

@mayrsascha
Copy link
Author

@pcriadoperez Sorry just got to reply to your message now. Here a few key points in relation to your comment:

  • I created a mapping of exchanges and coins that I'm watching as a gist. For each coin I'm trying to watch the symbol pair of coin vs USD, coin vs BTC and coin vs ETH (if that symbol exists on exchange.symbols). So approximately the number of coins per exchange in the Gist times 3.
  • I've double checked my code, I'm only using 4 properties provided by the exchange instance of ccxt: exchange.name, exchange.symbols (after loadMarkets), exchange.loadMarkets() and exchange.watchOHLCV(). Both exchange.loadMarkets() and exchange.watchOHLCV() are wrapped in try/catch blocks. In those try/catch blocks I'm logging to STDOUT (with some extra debug info) and sending errors to Sentry and Bugsnag (with extra metadata). Neither in the logs nor in those error tracking services I'm seeing errors being logged as I would log them in those catch blocks, plus the stack trace doesn't show those lines of code, so everything still points to uncatchable errors ATM.
  • I'm watching for candles with 1h interval, maybe that makes a difference...
  • Yes I'm seeing these errors consistently, every hour or two at least a few of these. Some examples of coins that fail: Bitget USDV/ETH, Bitget LITT/USDT, OKX PHA/USDT, HitBTC BRD/ETH (just to name a very few). It doesn't seem to be limited to any exchange. The only consistent pattern I see that a lot (or maybe all) watched symbols throw an error for the same exchange at the same time (makes sense as the WS connection seems to be broken). The somewhat interesting part is that the connection breaks at any random minute of the hour, even at half past the hour even though I set the interval to 1h and I would naturally expect the connection to potentially break when new candles are incoming just at the new hour...
  • Some example times when connections broke: May 22nd, 13:03:45 UTC for BitGet, May 22nd, 12:27:10 UTC for BitGet, May 22nd, 12:18:18 UTC for HitBTC, May 22nd, 11:18:56 UTC for OKX.
  • If it helps, I'm running my server on render.com in the Frankfurt, Germany region. Standard instance type (1 CPU, 2GB RAM), NODE_ENV=production, NODE_OPTIONS=--max-old-space-size=4096, NODE_VERSION=lts and my memory and CPU usage are way below what's available on a consistent basis.

Please let me know if there is anything else I can provide!

@pcriadoperez
Copy link
Contributor

Hi @mayrsascha,

To try and dial down the error I ran a script for a couple of hours watching all 1100 symbols of bitget OHLCV including the symbols you mentioned. Unfortunately I still was not able to reproduce the error.

Also I would try to add more ram to the machine or reduce the max space size option of node so there is no mismatch there. In use cases like this with a lot of connections to several servers I have seen the ram usage can go up, so I would also double check there is not an issue there.

I've left a script running in the background to see if I can see the error be reproduce

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@carlosmiei carlosmiei

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@ragamufin @pcriadoperez @mayrsascha @carlosmiei @andrewmichael042c