Telnet communication: how to configure server to operate in blocking mode?

[chekhovana]

Concerning communication with jsbsim through telnet, I can observe the following situation: when connection with client isn't established, then method:

jsbsim/src/input_output/FGInputSocket.cpp

Line 112 in d4f85bc

void FGInputSocket::Read(bool Holding)

is invoked in loop, processing incoming empty data packets, which is quite CPU-consuming. The following setting:

<input port="1137" action="blocking_input"/>

doesn't help.

Is there any way to configure server socket to suspend and wait until some non-empty data packet arrives?

Thanks in advance

[seanmcleod]

when connection with client isn't established

Just to be clear, you're saying that you haven't established a telnet connection? As opposed to you've established a connection but the telnet client is idle, i.e. isn't sending any data?

So if you haven't established a connection then given line 115 if (!socket->GetConnectStatus()) return; the function will return immediately and not block waiting for data.

processing incoming empty data packets

So if that is the case, it isn't processing empty data packets, there is no connection to handle so it simply returns.

jsbsim/src/input_output/FGInputSocket.cpp

Lines 112 to 120 in d4f85bc

	void FGInputSocket::Read(bool Holding)
	{
	if (!socket) return;
	if (!socket->GetConnectStatus()) return;
	if (BlockingInput)
	socket->WaitUntilReadable(); // block until a transmission is received
	string raw_data = socket->Receive(); // read data

The blocking_input option only comes into play once there is an established connection, but the telnet client is idle, i.e. connected but not sending data. So maybe in your case you simply need to establish a telnet connection?

Out of interest how and what are you using JSBSim for via the telnet interface?

[chekhovana]

Hello, thanks for the quick response.

I understood that it's necessary to establish telnet connection immediately at start and keep it alive.

Answering your last question: we use JSBSim as plug-in to ArduPilot. In this scenario we do not have access to API. The only way to manage simulation parameters we found is passing property values through telnet.

Is it a proper way of communicating with JSBSim within described architecture? Or any better alternative exists?

[seanmcleod]

Is it a proper way of communicating with JSBSim within described architecture? Or any better alternative exists?

Are you using ArduPilot SITL with JSBSim? https://ardupilot.org/dev/docs/sitl-with-jsbsim.html

[chekhovana]

So if you haven't established a connection then given line 115 if (!socket->GetConnectStatus()) return; the function will return immediately and not block waiting for data.

I've just checked once again, and it seems that socket->GetConnectStatus() returns true even if there is no client connection. So the function doesn't return, if I insert debug print after this line:

  if (!socket->GetConnectStatus()) return;
  cout << "inside read, after get connect status" << endl;

then this debug message outputs to console multiple times. What am I doing wrong?

[seanmcleod]

I see a couple of issues in the JSBSim source code with regards to a telnet socket.

So first off, note that the socket when it is created is set to non-blocking mode. Which means that when accept() is called on line 218 it won't block, and if there is no waiting connection it will return INVALID_SOCKET with a last error of WOULDBLOCK. However on line 219 we set connected = true even if the return value from accept() was INVALID_SOCKET. So this is the first issue, we shouldn't blindly be setting connected = true.

jsbsim/src/input_output/FGfdmSocket.cpp

Lines 209 to 220 in d4f85bc

	if (Protocol == ptTCP) {
	if (listen(sckt, 5) != SOCKET_ERROR) { // successful listen()
	#ifdef _WIN32
	u_long NoBlock = 1;
	ioctlsocket(sckt, FIONBIO, &NoBlock);
	#else
	int flags = fcntl(sckt, F_GETFL, 0);
	fcntl(sckt, F_SETFL, flags | O_NONBLOCK);
	#endif
	sckt_in = accept(sckt, (struct sockaddr*)&scktName, &len);
	connected = true;
	} else

The second issue is on line 114, the check should be if(socket == INVALID_SOCKET).

jsbsim/src/input_output/FGInputSocket.cpp

Lines 112 to 115 in d4f85bc

	void FGInputSocket::Read(bool Holding)
	{
	if (!socket) return;
	if (!socket->GetConnectStatus()) return;

So because of both of these issues you're seeing socket->GetConnectStatus() return true even when there isn't an actual telnet connection yet.

[dpculp]

Have you tried a UDP socket connection? For info see the QtJSBSim project: https://github.com/dpculp/qtjsbsim

[bcoconni]

The second issue is on line 114, the check should be if(socket == INVALID_SOCKET).

jsbsim/src/input_output/FGInputSocket.cpp

Lines 112 to 115 in d4f85bc

	void FGInputSocket::Read(bool Holding)
	{
	if (!socket) return;
	if (!socket->GetConnectStatus()) return;

Note that in the class FGInputSocket, the type of the member socket is a pointer FGfdmSocket* so the test if(socket == INVALID_SOCKET) would not make sense in this context.

jsbsim/src/input_output/FGInputSocket.h

Line 88 in d4f85bc

FGfdmSocket* socket;

[bcoconni]

However on line 219 we set connected = true even if the return value from accept() was INVALID_SOCKET. So this is the first issue, we shouldn't blindly be setting connected = true.

jsbsim/src/input_output/FGfdmSocket.cpp

Lines 209 to 220 in d4f85bc

	if (Protocol == ptTCP) {
	if (listen(sckt, 5) != SOCKET_ERROR) { // successful listen()
	#ifdef _WIN32
	u_long NoBlock = 1;
	ioctlsocket(sckt, FIONBIO, &NoBlock);
	#else
	int flags = fcntl(sckt, F_GETFL, 0);
	fcntl(sckt, F_SETFL, flags | O_NONBLOCK);
	#endif
	sckt_in = accept(sckt, (struct sockaddr*)&scktName, &len);
	connected = true;
	} else

For this code, I'd say that setting connected to true blesses the initialization of sckt, not the success of accept(). Especially because accept() will be called again in FGfdmSocket::Receive() if sckt_in == INVALID_SOCKET:

jsbsim/src/input_output/FGfdmSocket.cpp

Lines 245 to 253 in d4f85bc

	string FGfdmSocket::Receive(void)
	{
	char buf[1024];
	socklen_t len = sizeof(struct sockaddr_in);
	string data; // todo: should allocate this with a standard size as a
	// class attribute and pass as a reference?
	if (sckt_in == INVALID_SOCKET && Protocol == ptTCP) {
	sckt_in = accept(sckt, (struct sockaddr*)&scktName, &len);

In addition if connected is not set to true then FGInputSocket::Read will never call FGfdmSocket::Receive() again and sckt_in will never have the opportunity to be set to a value other than INVALID_SOCKET:

jsbsim/src/input_output/FGInputSocket.cpp

Lines 112 to 120 in d4f85bc

	void FGInputSocket::Read(bool Holding)
	{
	if (!socket) return;
	if (!socket->GetConnectStatus()) return;
	if (BlockingInput)
	socket->WaitUntilReadable(); // block until a transmission is received
	string raw_data = socket->Receive(); // read data

[seanmcleod]

Hmm, looks like I was a bit rushed in my analysis earlier today.

The variable name connected and being set immediately after accept() did lead me to assume it was related to the socket being connected to a client, as opposed to just indicating that listen() succeeded.

In my rush I also confused socket and sckt_in in FGInputSocket::Read().

I did pickup given the non-blocking status of the socket that accept() would be called again in FGfdmSocket::Receive() but didn't check under what conditions Receive() would be called.

In terms of the following:

is invoked in loop, processing incoming empty data packets, which is quite CPU-consuming. The following setting:

<input port="1137" action="blocking_input"/>

doesn't help.

Not clear why that would be the case, since socket->WaitUntilReadable() uses select to wait until the socket is readable, in particular:

In summary, a socket will be identified in a particular set when select returns if:

readfds:

If listen has been called and a connection is pending, accept will succeed.

So if there was no waiting connection then it should've blocked until an incoming connection request came in, i.e. shouldn't have been busy chewing CPU cycles.

[bcoconni]

The variable name connected and being set immediately after accept() did lead me to assume it was related to the socket being connected to a client, as opposed to just indicating that listen() succeeded.

I agree that this code, as it is currently written, is misleading. It is only when I saw that accept() is called again in FGfdmSocket::Receive that I realized that connected = true is not meant to confirm the success of accept().

[seanmcleod]

I'll submit a pull request to fix the socket issues I mentioned above. But as a quick work-around for your build you could just update line 114 in FGInputSocket.cpp for now.

Are you using ArduPilot SITL with JSBSim? https://ardupilot.org/dev/docs/sitl-with-jsbsim.html

[chekhovana]

I'll submit a pull request to fix the socket issues I mentioned above. But as a quick work-around for your build you could just update line 114 in FGInputSocket.cpp for now.

thank you!

Are you using ArduPilot SITL with JSBSim? https://ardupilot.org/dev/docs/sitl-with-jsbsim.html

yes, that's it

[seanmcleod]

So Ardupilot's SITL interface with JSBSim makes use of the telnet interface to send commands to JSBSim. So are you looking to re-implement the SITL interface to JSBSim in Ardupilot? Or are you wanting to keep the current Ardupilot SITL interface in place but you want some additional communication channel?

If you give us some more details on what your current issues are with the SITL interface and what you're trying to achieve it might help us steer you in some direction.

[airspector]

Hi Sean! Thank you for your answers. I'm the member of the same team, and @chekhovana asked me to participate in this discussion as I'm responsible for the system architecture in our project. We are developing fixedwing drone simulator with our own GCS, SITL, JSBSim and Unreal Engine. As far as I understand, there are three integration options for JSBSim:

1. Plug-in for SITL;
2. Plug-in for Unreal Engine;
3. Standalone JSBSim connected to SITL through MAVProxy.

Currently we use the first one because it makes our MVP development easier, but in this case we don't have direct access to JSBSim API. It seems the third option would be the best, despite it'll take some efforts to implement communication between MAVProxy and JSBSim. I would appreciate if you propose better solution.

[seanmcleod]

Okay, so you have your own GCS, but I'm assuming it's compatible with MAVLink and I'm assuming you're also using Arduplane as the autopilot?

And now instead of using FlightGear for visualization you want to use Unreal?

If that is the case why not simply write some code to take the pose (position, attitude) data that JSBSim provides to then drive the pose in Unreal? In other words continue using all the other existing Arduplane SITL links to/from JSBSim.

[airspector]

Okay, so you have your own GCS, but I'm assuming it's compatible with MAVLink and I'm assuming you're also using Arduplane as the autopilot? And now instead of using FlightGear for visualization you want to use Unreal?

Yes, that's correct.

If that is the case why not simply write some code to take the pose (position, attitude) data that JSBSim provides to then drive the pose in Unreal? In other words continue using all the other existing Arduplane SITL links to/from JSBSim.

Yes, we plan to do this in next release. For now we have to implement some high-priority functions from our backlog.

[bcoconni]

On my side, this piece of code is bothering me:

jsbsim/src/input_output/FGfdmSocket.cpp

Lines 273 to 285 in d4f85bc

	#ifdef _WIN32
	// when nothing received and the error isn't "would block"
	// then assume that the client has closed the socket.
	if (num_chars == 0) {
	DWORD err = WSAGetLastError();
	if (err != WSAEWOULDBLOCK) {
	cout << "Socket Closed. Back to listening" << endl;
	closesocket(sckt_in);
	sckt_in = INVALID_SOCKET;
	}
	}
	#endif
	}

This is Windows only code (not sure which platform @chekhovana and @airspector are using ? Has that been mentioned ?). This code basically hangs up as soon as it is no longer receiving data. For a telnet connection I think that would result in the immediate termination of the connection. I guess this piece of code is meant to save CPU cycles but I could not tell for sure. I'm suspecting that this strategy is too aggressive and that it causes the numerous failures of the unit test TestInputSocket.py on Windows for our CI workflow.

It may be related to this discussion topic if JSBSim is being run on Windows but it would not explain why blocking_input is not blocking...

[chekhovana]

Hello, @bcoconni
We don't use Window at all, server is ubuntu 22.04, locally I performed tests on Mac OS.

[seanmcleod]

Not sure why there is that Windows specific code. If I compare the documentation for recv() for Windows and Linux they appear to be pretty much identical.

https://learn.microsoft.com/en-us/windows/win32/api/winsock/nf-winsock-recv
https://www.man7.org/linux/man-pages/man2/recv.2.html

They both return 0 when:

Linux: When a stream socket peer has performed an orderly shutdown, the return value will be 0 (the traditional "end-of-file" return).

Windows: If the socket is connection oriented and the remote side has shut down the connection gracefully, and all data has been received, a recv will complete immediately with zero bytes received.

And if set to non-blocking then both return -1 (SOCKET_ERROR) if a call would block.

Linux: These calls return the number of bytes received, or -1 if an error occurred. In the event of an error, [errno] is set to indicate the error.
EWOULDBLOCK - The socket is marked nonblocking and the receive operation would block

Windows: ...unless the socket is nonblocking. In this case, a value of SOCKET_ERROR is returned with the error code set to WSAEWOULDBLOCK.

Not sure what platform @chekhovana and @airspector are using.

[chekhovana]

hi @seanmcleod!

I can't test how it works under Windows, as I don't have Windows available (

Under Mac OS I can observe another issue, perhaps somehow related to the processing of num_chars == 0 that you mentioned earlier:

established telnet connection, terminated from the client side by escape character ^] (ie without quit command) leads to the situation when any further connections are impossible.

As far as I can see by the code (I am not experienced in c++, unfortunately) this happens because sckt_in variable keeps some valid descriptor != INVALID_SOCKET despite the fact there is no established client connection at the moment.

[seanmcleod]

Yep, that's what I was alluding to, to @bcoconni, i.e. that the Windows specific code inside the #ifdef (although it could be simplified) isn't only needed for Windows, rather that logic is needed for all platforms to handle this case of a client shutting down the connection. In that case, indicated by recv() returning 0 we need to for all platforms:

closesocket(sckt_in); 
sckt_in = INVALID_SOCKET;

So that we can handle a subsequent telnet connection.

Talking of multiple connections, I'm not sure if JSBSim will handle multiple concurrent telnet connections correctly at the moment, I haven't double-checked the logic of the code yet or tested it.

[chekhovana]

In our use case concurrent connections aren't necessary. What we need is the ability to reopen single client connection in both situations:

• previous connection was closed by quit command - it's ok now
• previous connection was terminated by escape sequence ^] - doesn't work at the moment

[chekhovana]

Have you tried a UDP socket connection? For info see the QtJSBSim project: https://github.com/dpculp/qtjsbsim

Hello @dpculp!

No, I haven't tried UDP. Judging by the code, I expect the same issues as in case of using TCP sockets, as both socket types share the same implementation, aren't they?

[bcoconni]

@chekhovana Just letting you know that Pull Request #1081 is currently under review. It's expected to improve the management of socket errors and will log error messages in the console when a socket function encounters an error. This should assist in pinpointing the issue you've reported.

[chekhovana]

@bcoconni Thank you!

[bcoconni]

Hi @chekhovana, we've just merged a pull request that should resolve a few details and provide clearer error messages.

Do you have the opportunity to compile the head of JSBSim master branch ? If so, could you please run some tests with the latest code of JSBSim and let us know if there are some improvements or new error messages issued while running your code ?

Thanks.

[chekhovana]

Hello @bcoconni , thanks for notification.
Sure, I'll do it and report the results.

[chekhovana]

Hi @bcoconni, I tried the latest version, here's my results:

• the problem of reopening connection is fixed - now connection can be reestablished in both cases, after termination by either

  • quit command
  • escape sequence

• the problem of non-blocking mode still exists: the following line suspends execution only if client connection is established:
  

  jsbsim/src/input_output/FGInputSocket.cpp

  Line 118 in 56c6662

  socket->WaitUntilReadable(); // block until a transmission is received

Out of interest: would you mind explaining to me the logic behind accepting incoming connection requests in non-blocking mode? I've never seen this way of using sockets before and can't suggest the reasons for doing so.

[seanmcleod]

JSBSim runs as a single threaded process. So in general with regards to the telnet interface you have JSBSim running at by default 120Hz and in realtime mode with a user flying the aircraft and then every now and then the user may type something into the telnet client to get some state data, or to set some state etc. and so in general you don't want to block and therefore pause the simulation waiting for the user to send a telnet command.

Rather on each simulation timestep you check, in a non-blocking way whether there is any incoming telnet data, and if there is then read it and process it. But if there isn't any incoming telnet data then loop around again and execute the next simulation time step. If run in realtime mode there will generally be some amount of sleep in-between simulation time steps in order to keep to the simulation rate (default 120Hz).

Now much later on there was a requirement for lock stepping with an external process, two examples being Ardupilot wanting to use JSBSim in SITL mode and also @agodemar wanting to interface with OpenFOAM as an external process providing 3D wind velocities.

Which is what lead to the support for the action = BLOCKING_INPUT, see the following issue #74 for more details.

You will see comments/suggestions about making JSBSim configurable to decide whether to only block on reading from a socket or also whether to allow it to block on the listen/accept for an incoming request, which we don't currently support.

Note while reading the issue the mention of Ardupilot SITL and in particular it's use of iterate to allow it to single-step the JSBSim process.

You had mentioned making use of JSBSim via ArduPilot SITL which runs JSBSim and then makes use of the iterate command via a telnet connection, once the telnet connection is established etc.

Does your setup differ to the standad ArduPilot SITL? Are you starting things differently etc.? Give us more details on what you're trying to do and what issue you see with JSBSim not blocking on the listen/accept.

[chekhovana]

Hi @seanmcleod, thank you for the detailed explanation.

You had mentioned making use of JSBSim via ArduPilot SITL which runs JSBSim and then makes use of the iterate command via a telnet connection, once the telnet connection is established etc.

Yes, and in our architecture nothing is changed here - ArduPilot SITL communicates with JSBSim via a telnet connection in a standard way. JSBSim is connected to SITL with the SIM_JSBSim plugin.

Does your setup differ to the standad ArduPilot SITL? Are you starting things differently etc.? Give us more details on what you're trying to do and what issue you see with JSBSim not blocking on the listen/accept.

There's second telnet connection to JSBSim from some external service, which is used to change property values while simulation is running. Number of updates through this channel is quite moderate: from 0 to 30 per hour, depending on the simulation scenario. Mainly we use it to change atmosphere conditions such as turbulence, wind speed and direction.

To summarize, we communicate with JSBSim through 2 channels:

1. from ArduPilot SITL - in a standard way
2. from external service - to make extra updates of property values

Taking into account single-threaded implementation: it's not possible to set blocking read mode in channel 2, as it would also block channel 1.

Is that suggestion right?

[seanmcleod]

@chekhovana thanks for the additional detail about your setup.

I did mention the following previously:

Talking of multiple connections, I'm not sure if JSBSim will handle multiple concurrent telnet connections correctly at the moment, I haven't double-checked the logic of the code yet or tested it.

However, thinking about it a bit more since you posted actually I think you should be able to get these 2 (ArduPilot SITL and atmosphere updates) concurrent telnet connections working with JSBSim today.

So you'll see that ArduPilot SITL creates a telnet control port as part of the script template that it creates.

https://github.com/ArduPilot/ardupilot/blob/40fc2eb91c323433354bbedf94cccb5dacb34902/libraries/SITL/SIM_JSBSim.cpp#L65

https://github.com/ArduPilot/ardupilot/blob/40fc2eb91c323433354bbedf94cccb5dacb34902/libraries/SITL/SIM_JSBSim.cpp#L104-L106

So I'd suggest you simply create another telnet control port on a different port number, either by adding another one to the template script or by adding it to your JSBSim aircraft's FDM xml file.

That way JSBSim will then have two telnet input channels listening for connections, with ArduPlane SITL connecting to the one and time stepping JSBSim while your program with the weather data can connect to the second telnet port and periodically send weather parameter updates.

In this setup there should be no need to modify JSBSim's current telnet and socket handling code, just make sure that you don't set either telnet port to action="blocking_input". And it doesn't matter whether your atmosphere process opens it's telnet socket once and then sends periodic atmosphere updates or whether it connects, sends an update, disconnects each time.

BTW, talking of non-blocking sockets you'll notice that ArduPilot SITL also makes use of non-blocking sockets.

https://github.com/ArduPilot/ardupilot/blob/40fc2eb91c323433354bbedf94cccb5dacb34902/libraries/SITL/SIM_JSBSim.cpp#L296-L315

[chekhovana]

Hi @seanmcleod, thanks a lot for your help and provided information.
Everything is clear now.