Getting FactLoopError on multiple facts.

[karthikdialpad]

Hi,

We are trying to fetch facts from devices using the junos python client library and our call looks something like below

# connection = jnpr.junos.Device(self.ip, **SSH_OPTIONS)
def get_facts(self):
    if not self.connection:
      return {}
    self.connection.facts_refresh()
    return self.connection.facts

After construction the facts when I tried to do json.dump(facts) results in the following FactLoopError.
We are seeing these errors occasionally an it is not consistent enough for us to repro it immediately.
But we are sure that this error happens for different facts like domain, hostname, is_linux, current_re etc.

The Device details:
model : ex4300-48t , firmware version: 21.4R3.15

We are seeing this issue of python2.7 and python3.
In python2.7 we are running
junos-eznc==2.5.4
ncclient==0.6.13

In python3.11 we are running
junos-eznc==2.6.8
ncclient==0.6.13

fst_client.fetch_url for https://XXXXXXXXX/api/networkdevice/heartbeat exception 
A loop was detected while gathering the _is_linux fact. 
The jnpr.junos.facts.is_linux module has already been called. Please report this error.

Traceback (most recent call last): 
File "/usr/local/rackman/lib/python3.11/site-packages/jnpr/junos/device.py", line 834, 
in execute rpc_rsp_e = self._rpc_reply( 
^^^^^^^^^^^^^^^^ 
File "/usr/local/rackman/lib/python3.11/site-packages/jnpr/junos/decorators.py", line 165, 
in wrapper raise ex File "/usr/local/rackman/lib/python3.11/site-packages/jnpr/junos/decorators.py", line 117, 
in wrapper rsp = function(self, *args, **kwargs) 
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
File "/usr/local/rackman/lib/python3.11/site-packages/jnpr/junos/device.py", line 1470, 
in _rpc_reply return self._conn.rpc(rpc_cmd_e, filter_xml)._NCElement__doc 
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
File "/usr/local/rackman/lib/python3.11/site-packages/ncclient/manager.py", line 246, 
in execute return cls(self._session, 
^^^^^^^^^^^^^^^^^^ 
File "/usr/local/rackman/lib/python3.11/site-packages/ncclient/operations/third_party/juniper/rpc.py", line 52, 
in request return self._request(rpc) 
^^^^^^^^^^^^^^^^^^ 
File "/usr/local/rackman/lib/python3.11/site-packages/ncclient/operations/rpc.py", line 375, 
in _request raise self._reply.error ncclient.operations.rpc.RPCError: command is not valid on the ex4300-48t 

During handling of the above exception, another exception occurred: 
Traceback (most recent call last): 
File "/usr/local/rackman/lib/python3.11/site-packages/jnpr/junos/facts/get_software_information.py", line 14, 
in _get_software_information return device.rpc.cli( 
^^^^^^^^^^^^^^^ 
File "/usr/local/rackman/lib/python3.11/site-packages/jnpr/junos/rpcmeta.py", line 297, 
in cli return self._junos.execute(rpc, normalize=normalize) 
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
File "/usr/local/rackman/lib/python3.11/site-packages/jnpr/junos/decorators.py", line 63, 
in wrapper result = function(*args, **kwargs) 
^^^^^^^^^^^^^^^^^^^^^^^^^ 
File "/usr/local/rackman/lib/python3.11/site-packages/jnpr/junos/decorators.py", line 31, 
in wrapper return function(*args, **kwargs) 
^^^^^^^^^^^^^^^^^^^^^^^^^ 
File "/usr/local/rackman/lib/python3.11/site-packages/jnpr/junos/device.py", line 854, 
in execute raise EzErrors.RpcError(cmd=rpc_cmd_e, rsp=rsp, errs=ex) jnpr.junos.exception.RpcError: RpcError(severity: error, bad_element: None, message: command is not valid on the ex4300-48t) 

During handling of the above exception, another exception occurred: 
Traceback (most recent call last): 
File "/usr/local/rackman/app/fst_client.py", line 73,
in fetch_url r = get_response(url, method, params, data, timeout, retries, retry_backoff_factor, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
File "/usr/local/rackman/app/fst_client.py", line 64, 
in get_response r = requests.post(url, params=params, headers=headers, data=json.dumps(data), 
^^^^^^^^^^^^^^^^ 
File "/usr/local/lib/python3.11/json/__init__.py", line 231, 
in dumps return _default_encoder.encode(obj) 
^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
File "/usr/local/lib/python3.11/json/encoder.py", line 200, 
in encode chunks = self.iterencode(o, _one_shot=True) 
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
File "/usr/local/lib/python3.11/json/encoder.py", line 258, 
in iterencode return _iterencode(o, 0) 
^^^^^^^^^^^^^^^^^ 
File "/usr/local/rackman/lib/python3.11/site-packages/jnpr/junos/factory/to_json.py", line 76, 
in default obj = {k: v for k, v in obj.items()} 
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
File "/usr/local/rackman/lib/python3.11/site-packages/jnpr/junos/factory/to_json.py", line 76, 
in <dictcomp> obj = {k: v for k, v in obj.items()} 
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
File "<frozen _collections_abc>", line 861, 
in __iter__ File "/usr/local/rackman/lib/python3.11/site-packages/jnpr/junos/factcache.py", line 95, 
in __getitem__ new_facts = self._callbacks[key](self._device) 
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
File "/usr/local/rackman/lib/python3.11/site-packages/jnpr/junos/facts/get_software_information.py", line 137, 
in get_facts rsp = _get_software_information(device) 
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
File "/usr/local/rackman/lib/python3.11/site-packages/jnpr/junos/facts/get_software_information.py", line 19, 
in _get_software_information if device.facts["_is_linux"]: ~~~~~~~~~~~~
^^^^^^^^^^^^^ 
File "/usr/local/rackman/lib/python3.11/site-packages/jnpr/junos/factcache.py", line 84, 
in __getitem__ raise jnpr.junos.exception.FactLoopError( jnpr.junos.exception.FactLoopError: A loop was detected while gathering the _is_linux fact. The jnpr.junos.facts.is_linux module has already been called. Please report this error.

Can anyone help us find the root cause of it and explain how we can mitigate it?

Thanks