ignore_warnings causing "'str' object has no attribute '_root'" #1245

kpetremann · 2023-04-25T13:21:20Z

Hello team,

We are encountering this issue when loading a configuration with ignore_warning feature enabled:

Traceback (most recent call last):
  File "/opt/salt/venv/lib64/python3.9/site-packages/salt/utils/napalm.py", line 169, in call
    out = getattr(napalm_device.get("DRIVER"), method)(*args, **kwargs)
  File "/opt/salt/venv/lib64/python3.9/site-packages/napalm/junos/junos.py", line 284, in load_merge_candidate
    self._load_candidate(filename, config, False)
  File "/opt/salt/venv/lib64/python3.9/site-packages/napalm/junos/junos.py", line 263, in _load_candidate
    self.device.cu.load(
  File "/opt/salt/venv/lib64/python3.9/site-packages/jnpr/junos/utils/config.py", line 589, in load
    return try_load(rpc_contents, rpc_xattrs, ignore_warning=ignore_warning)
  File "/opt/salt/venv/lib64/python3.9/site-packages/jnpr/junos/utils/config.py", line 500, in try_load
    got = self.rpc.load_config(
  File "/opt/salt/venv/lib64/python3.9/site-packages/jnpr/junos/rpcmeta.py", line 287, in load_config
    return self._junos.execute(rpc, ignore_warning=ignore_warning)
  File "/opt/salt/venv/lib64/python3.9/site-packages/jnpr/junos/decorators.py", line 76, in wrapper
    return function(*args, **kwargs)
  File "/opt/salt/venv/lib64/python3.9/site-packages/jnpr/junos/decorators.py", line 31, in wrapper
    return function(*args, **kwargs)
  File "/opt/salt/venv/lib64/python3.9/site-packages/jnpr/junos/device.py", line 834, in execute
    rpc_rsp_e = self._rpc_reply(
  File "/opt/salt/venv/lib64/python3.9/site-packages/jnpr/junos/decorators.py", line 155, in wrapper
    rsp = NCElement(
  File "/opt/salt/venv/lib64/python3.9/site-packages/ncclient/xml_.py", line 177, in __init__
    self.__doc = self.__transform_reply(result._root)
AttributeError: 'str' object has no attribute '_root'

This is causing an exception resulting, and the commit is discard.

The code in pyeznc
is calling NCElement with an argument with the wrong type:

                rsp = NCElement(
                    etree.tostring(rsp, encoding=encode), self.transform()  <<< first arg is a string
                )._NCElement__doc

class NCElement(object):
    def __init__(self, result, transform_reply, huge_tree=False):
        self.__result = result                                   <<< a string
        self.__transform_reply = transform_reply
        self.__huge_tree = huge_tree
        if isinstance(transform_reply, types.FunctionType):
            self.__doc = self.__transform_reply(result._root)    <<< string does not have a _root argument
        else:
            self.__doc = self.remove_namespaces(self.__result)

Bypassing the error with a try except block confirms this is the root cause.
I don't know what can be passed to NCElement instead of a string. I am not familiar with ncclient.

Can you help please?

Tested versions:
junos-eznc=2.6.5 and 2.6.7

The text was updated successfully, but these errors were encountered:

chidanandpujar · 2023-05-17T08:48:25Z

Hi @kpetremann
Could you please share the RPC details or PyEZ script ,
So that I can reproduce the issue .

Thanks & Regards
Chidanand

kpetremann · 2023-05-17T09:11:48Z

Hi @chidanandpujar,

Thanks for your response.

So far, we had the issue only when using Saltstack scheduler:

the schedule triggers a Salt state to update the configuration
the Salt state is executing load_template which calls napalm load_config method (napalm 4.0.0)
napalm is calling load_merge_candidate of junos-eznc

Our code base is open-source:

the Salt state: https://github.com/criteo/openconfig-saltstack/blob/main/states/afk/init.sls
the State module: https://github.com/criteo/openconfig-saltstack/blob/main/_states/openconfig_routing_policy.py#L490

The Salt proxy-minion is setting ignore_warning to True.

I can try to create a docker compose based lab to reproduce the issue on demand. But I will be on vacation starting tomorrow, so I won't be able to provide you this before June :/

Note: I don't know why it only happens in a schedule context, which is weird. Maybe it happens after multiple execution. We did not run the state manually a lot.

FYI, we deployed this workaround which works. But it is not a proper way to fix the issue.

kpetremann · 2023-05-17T09:30:10Z

The loaded configuration looks like this: https://github.com/criteo/openconfig-saltstack/blob/main/tests/states/openconfig_routing_policy/data/functional_tests/expected_result_junos.txt

It happens even if the configuration is already up to date.

kpetremann · 2023-05-17T09:44:58Z

Additional information, we saw the issue on QFX 5100 & 5110, at least on version 18.4 and 20.2.

chidanandpujar · 2023-05-17T11:26:52Z

Hi @kpetremann ,
Thanks for the details .
Is it possible to collect the netconf trace logs from the QFX device .

set system services netconf ssh
set system services netconf traceoptions file netcof_trace.log
set system services netconf traceoptions file size 1g
set system services netconf traceoptions flag all
commit.

Thanks & Regards
Chidanand

kpetremann · 2023-05-17T14:31:24Z

@chidanandpujar, I reverted my fix to retrigger the issue, but so far no occurrence.

I'll check the result in June after my vacations.
I'll keep you updated then.

kpetremann · 2023-07-06T12:31:44Z

hi @chidanandpujar,
I am able to reproduce the issue.
But when the workflow is triggered, the netconf_trace.log is created but it remains empty.

> show configuration system services netconf      
ssh;
traceoptions {
    file netconf_trace.log size 1g;
    flag all;
}

> show log netconf_trace.log                    

{master:0}
>

kpetremann · 2023-10-29T09:27:31Z

hi @chidanandpujar any update?

chidanandpujar · 2023-11-02T11:47:35Z

Hi @kpetremann
Thanks for the information ,
I am yet to replicate this issue on our local setup.
I will update you .

Thanks & Regards
Chidanand

chidanandpujar · 2023-11-08T06:13:07Z

Hi @kpetremann

     Thanks very much for sharing the details ,
     Netconf traces are empty , so it is not clear what is the RPC and configuration details are loaded .
    from the exception details , it is visible that  config load operation is performed and there might be error/warnings emitted as part of rpc execution .
something similar , cu.load(cnf, ignore_warning='statement not found')
``
Also,  I am not able to access the config file -https://github.com/criteo/openconfig-saltstack/blob/main/tests/states/openconfig_routing_policy/data/functional_tests/expected_result_junos.txt

Thanks

kpetremann · 2023-11-09T09:47:49Z

Hi @chidanandpujar,

For the config file example, the a directory has been renamed, here the new link: https://github.com/criteo/afk-saltstack/blob/main/tests/states/openconfig_routing_policy/data/integration_tests/expected_result_junos.txt

there might be error/warnings emitted as part of rpc execution

As we set the ignore_warning parameter to true, the warning should be ignore, am I right?

chidanandpujar · 2023-11-09T15:46:44Z

Hi @kpetremann
Thanks for sharing the config file ,
yeah correct , if we set the ignore_warning to (true or string or list of strings ) which are emitted as part of rpc response, will be ignored .

https://www.juniper.net/documentation/us/en/software/junos-pyez/junos-pyez-developer/topics/task/junos-pyez-warnings-ignoring.html

I will try to replicate the issue on local test setup .

Thanks

chidanandpujar · 2023-11-21T11:10:07Z

Hi @kpetremann
when I try to load the provided config statements , ignore_warning looks to be not working .
I am able to replicated the issue with ignore_warning .

Traceback (most recent call last):
  File "/root/pyez_release_268_test1/venv/lib/python3.9/site-packages/junos_eznc-2.6.8+0.g0745dd0e.dirty-py3.9.egg/jnpr/junos/device.py", line 834, in execute
    rpc_rsp_e = self._rpc_reply(
  File "/root/pyez_release_268_test1/venv/lib/python3.9/site-packages/junos_eznc-2.6.8+0.g0745dd0e.dirty-py3.9.egg/jnpr/junos/decorators.py", line 145, in wrapper
    raise ex
  File "/root/pyez_release_268_test1/venv/lib/python3.9/site-packages/junos_eznc-2.6.8+0.g0745dd0e.dirty-py3.9.egg/jnpr/junos/decorators.py", line 117, in wrapper
    rsp = function(self, *args, **kwargs)
  File "/root/pyez_release_268_test1/venv/lib/python3.9/site-packages/junos_eznc-2.6.8+0.g0745dd0e.dirty-py3.9.egg/jnpr/junos/device.py", line 1471, in _rpc_reply
    return self._conn.rpc(rpc_cmd_e, filter_xml)._NCElement__doc
  File "/root/pyez_release_268_test1/venv/lib/python3.9/site-packages/ncclient-0.6.13-py3.9.egg/ncclient/manager.py", line 246, in execute
  File "/root/pyez_release_268_test1/venv/lib/python3.9/site-packages/ncclient-0.6.13-py3.9.egg/ncclient/operations/third_party/juniper/rpc.py", line 52, in request
  File "/root/pyez_release_268_test1/venv/lib/python3.9/site-packages/ncclient-0.6.13-py3.9.egg/ncclient/operations/rpc.py", line 373, in _request
ncclient.operations.rpc.RPCError: warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
error: syntax error
warning: statement not found
error: syntax error
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/pyez_release_268_test1/venv/lib/python3.9/site-packages/junos_eznc-2.6.8+0.g0745dd0e.dirty-py3.9.egg/jnpr/junos/utils/config.py", line 500, in try_load
    got = self.rpc.load_config(
  File "/root/pyez_release_268_test1/venv/lib/python3.9/site-packages/junos_eznc-2.6.8+0.g0745dd0e.dirty-py3.9.egg/jnpr/junos/rpcmeta.py", line 287, in load_config
    return self._junos.execute(rpc, ignore_warning=ignore_warning)
  File "/root/pyez_release_268_test1/venv/lib/python3.9/site-packages/junos_eznc-2.6.8+0.g0745dd0e.dirty-py3.9.egg/jnpr/junos/decorators.py", line 76, in wrapper
    return function(*args, **kwargs)
  File "/root/pyez_release_268_test1/venv/lib/python3.9/site-packages/junos_eznc-2.6.8+0.g0745dd0e.dirty-py3.9.egg/jnpr/junos/decorators.py", line 31, in wrapper
    return function(*args, **kwargs)
  File "/root/pyez_release_268_test1/venv/lib/python3.9/site-packages/junos_eznc-2.6.8+0.g0745dd0e.dirty-py3.9.egg/jnpr/junos/device.py", line 854, in execute
    raise EzErrors.RpcError(cmd=rpc_cmd_e, rsp=rsp, errs=ex)
jnpr.junos.exception.RpcError: RpcError(severity: error, bad_element: 65000, message: warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
error: syntax error
warning: statement not found
error: syntax error
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/pyez_release_268_test1/py-junos-eznc/tests/functional/issue_1245.py", line 9, in <module>
    cu.load(path='mx-config.conf', ignore_warning=True, format='set')
  File "/root/pyez_release_268_test1/venv/lib/python3.9/site-packages/junos_eznc-2.6.8+0.g0745dd0e.dirty-py3.9.egg/jnpr/junos/utils/config.py", line 589, in load
    return try_load(rpc_contents, rpc_xattrs, ignore_warning=ignore_warning)
  File "/root/pyez_release_268_test1/venv/lib/python3.9/site-packages/junos_eznc-2.6.8+0.g0745dd0e.dirty-py3.9.egg/jnpr/junos/utils/config.py", line 506, in try_load
    raise ConfigLoadError(cmd=err.cmd, rsp=err.rsp, errs=err.errs)
jnpr.junos.exception.ConfigLoadError: ConfigLoadError(severity: error, bad_element: 65000, message: warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found
error: syntax error
warning: statement not found
error: syntax error
warning: statement not found
warning: statement not found
warning: statement not found
warning: statement not found)

~/pyez_release_268_test1/py-junos-eznc/tests/functional# cat issue_1245.py 
from jnpr.junos import Device
from jnpr.junos.utils.config import Config
load_warnings = ['statement not found']

dev = Device(host='xx.xx.xx.xx', user='xyz', password='xyz')
dev.open()

with Config(dev, mode='exclusive') as cu:  
    cu.load(path='mx-config.conf', ignore_warning=True, format='set')
    
dev.close()

Thanks

chidanandpujar · 2023-11-23T05:27:33Z

Hi @kpetremann
There were some rpc-erros with severity as error reported during loading of the config statements , I have removed them since ,we don't have option to ignore severity errors .

with following config statements , I am able to ignore the warnings .

~/pyez_release_268_test1/py-junos-eznc/tests/functional# cat mx-config.set
delete policy-options community CL-LOCAL
delete policy-options community CL-MAIN
set policy-options community CL-MAIN members 649..:20000
delete policy-options community CL-SERVICE
delete policy-options community CL-DEFAULT
delete policy-options community CL-LOCATION
delete policy-options community CL-CLOS_INFRA
delete policy-options community CL-SERVER
delete policy-options route-filter-list PF-LOOPBACK_IPV4
set policy-options route-filter-list PF-LOOPBACK_IPV4 10.0.0.0/22 exact
set policy-options route-filter-list PF-LOOPBACK_IPV4 10.0.1.0/22 exact
delete policy-options route-filter-list PF-LOOPBACK_IPV6
set policy-options route-filter-list PF-LOOPBACK_IPV6 2001:db8:1::/64 exact
set policy-options route-filter-list PF-LOOPBACK_IPV6 2001:db8:1::/128 exact
delete policy-options policy-statement RM-TEST
delete policy-options policy-statement AUTOGENERATED::RM-TEST::IPV4_UNICAST
delete policy-options policy-statement AUTOGENERATED::RM-TEST::IPV6_UNICAST
delete policy-options policy-statement AUTOGENERATED::RM-TEST::L2VPN_EVPN
set policy-options policy-statement RM-TEST term 10 from route-filter-list PF-LOOPBACK_IPV4
set policy-options policy-statement RM-TEST term 10 from protocol direct
set policy-options policy-statement RM-TEST term 10 from local-preference 1234
set policy-options policy-statement RM-TEST term 10 from community CL-LOCAL
set policy-options policy-statement RM-TEST term 10 then origin egp
set policy-options policy-statement RM-TEST term 10 then local-preference 5678
set policy-options policy-statement RM-TEST term 10 then metric 250
delete policy-options community AUTOGENERATED::RM-TEST:10
set policy-options policy-statement RM-TEST term 10 then community set AUTOGENERATED::RM-TEST:10
set policy-options policy-statement RM-TEST term 10 then reject
set policy-options policy-statement RM-TEST then reject
set policy-options policy-statement AUTOGENERATED::RM-TEST::IPV4_UNICAST term 10 from family inet
set policy-options policy-statement AUTOGENERATED::RM-TEST::IPV4_UNICAST term 10 from route-filter-list PF-LOOPBACK_IPV4
set policy-options policy-statement AUTOGENERATED::RM-TEST::IPV4_UNICAST term 10 from protocol direct
set policy-options policy-statement AUTOGENERATED::RM-TEST::IPV4_UNICAST term 10 from local-preference 1234
set policy-options policy-statement AUTOGENERATED::RM-TEST::IPV4_UNICAST term 10 from community CL-LOCAL
set policy-options policy-statement AUTOGENERATED::RM-TEST::IPV4_UNICAST term 10 then origin egp
set policy-options policy-statement AUTOGENERATED::RM-TEST::IPV4_UNICAST term 10 then local-preference 5678
set policy-options policy-statement AUTOGENERATED::RM-TEST::IPV4_UNICAST term 10 then metric 250
delete policy-options community AUTOGENERATED::RM-TEST:10
set policy-options policy-statement AUTOGENERATED::RM-TEST::IPV4_UNICAST term 10 then community set AUTOGENERATED::RM-TEST:10
set policy-options policy-statement AUTOGENERATED::RM-TEST::IPV4_UNICAST term 10 then reject
set policy-options policy-statement AUTOGENERATED::RM-TEST::IPV4_UNICAST then reject
delete policy-options policy-statement RM-TEST-OUT
delete policy-options policy-statement AUTOGENERATED::RM-TEST-OUT::IPV4_UNICAST
delete policy-options policy-statement AUTOGENERATED::RM-TEST-OUT::IPV6_UNICAST
delete policy-options policy-statement AUTOGENERATED::RM-TEST-OUT::L2VPN_EVPN
set policy-options policy-statement RM-TEST-OUT term 10 then reject
set policy-options policy-statement RM-TEST-OUT then reject
set policy-options policy-statement AUTOGENERATED::RM-TEST-OUT::IPV4_UNICAST term 10 from family inet
set policy-options policy-statement AUTOGENERATED::RM-TEST-OUT::IPV4_UNICAST term 10 then reject
set policy-options policy-statement AUTOGENERATED::RM-TEST-OUT::IPV4_UNICAST then reject


~/pyez_release_268_test1/py-junos-eznc/tests/functional# cat issue_1245.py 
from jnpr.junos import Device
from jnpr.junos.utils.config import Config
load_warnings = ['statement not found']

dev = Device(host='xx.xx.xx.xx', user='xyz', password='xyz')
dev.open()

with Config(dev, mode='exclusive') as cu:  
    cu.load(path="mx-config.set", ignore_warning=True, format='set')
    
dev.close()

:~/pyez_release_268_test1/py-junos-eznc/tests/functional# python issue_1245.py 
:~/pyez_release_268_test1/py-junos-eznc/tests/functional#

Thanks

kpetremann · 2023-11-23T09:53:12Z

Hi @chidanandpujar,

I had a look to your stacktrace when you trigger the exception, I do not think this is the same one.
We do not have errors but only warnings.

The issues we have is that a warning triggers an unhandled exception because the warning is supposed to be an XML message with _root, but according the stacktrace we get, the message it is a string, which of course does not have _root attribute.

See master...kpetremann:py-junos-eznc:root_error_workaround#diff-4bb7532ee354a6233592a5357ecb0d59daa727fccf2afc4544c5d07baf74e052R145

at line 125: it checks if it is a warning or an error. If it is an error it raises an exception at line 145 and so exits the function, which is fine.
The issue is at line 156, which is not run when this is an error.

With the hacky patch, we do have an exception when there is an error and it is ok.
Only a warning is causing an issue, and I do not know which one exactly. Most of the warning are well handled.

I'll try to set up a lab today to reproduce the issue, as I cannot reproduce the issue because of business freeze.

chidanandpujar · 2023-11-24T15:44:45Z

Hi @kpetremann ,
Sure, Thanks ,
Please try to replicate the issue with PyEZ script ,
while debugging the issue , I have found a issue where ignoring_warning was set to False by default and fixed it via #1245

once you are able to replicate the issue, please share config and PyEZ script .

Thanks

apurvaraghu · 2024-03-04T19:56:01Z

Hi @kpetremann,
Were you able to replicate the issue via a PyEz script? Do you still see the issue? Kindly update.

kpetremann · 2024-03-05T18:29:28Z

Hi @apurvaraghu,

I've given it a try but it was not enough.

I'll also check the differences in eznc usage between salt (via napalm) and your script), and I'll try with both the script and salt at the same time.

apurvaraghu · 2024-03-05T19:08:46Z

Thanks, Please update the results.

chidanandpujar · 2024-05-06T10:53:24Z

Hi @kpetremann
Please replicate this issue using PyEZ script and update the config and results .
Till then I will move the issue to need-info state .

Please refer the following PyEZ script .

from jnpr.junos import Device
from jnpr.junos.utils.config import Config
load_warnings = ['statement not found']

dev = Device(host='xx.xx.xx.xx', user='xyz', password='xyz')
dev.open()

with Config(dev, mode='exclusive') as cu:  
    cu.load(path='mx-config.conf', ignore_warning=True, format='set')
    
dev.close()

Thanks
Chidanand

chidanandpujar self-assigned this May 17, 2023

chidanandpujar added the Status: Need Info label May 17, 2023

chidanandpujar added Status: Under Analysis and removed Status: Under Analysis labels Nov 8, 2023

chidanandpujar added Type: Bug and removed Status: Need Info labels Nov 21, 2023

chidanandpujar added a commit to chidanandpujar/py-junos-eznc that referenced this issue Nov 23, 2023

Fix for issue Juniper#1245

a8e26bc

chidanandpujar added Status: Need Info and removed Type: Bug labels May 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ignore_warnings causing "'str' object has no attribute '_root'" #1245

ignore_warnings causing "'str' object has no attribute '_root'" #1245

kpetremann commented Apr 25, 2023

chidanandpujar commented May 17, 2023

kpetremann commented May 17, 2023 •

edited

kpetremann commented May 17, 2023 •

edited

kpetremann commented May 17, 2023

chidanandpujar commented May 17, 2023

kpetremann commented May 17, 2023

kpetremann commented Jul 6, 2023

kpetremann commented Oct 29, 2023

chidanandpujar commented Nov 2, 2023

chidanandpujar commented Nov 8, 2023 •

edited

kpetremann commented Nov 9, 2023

chidanandpujar commented Nov 9, 2023 •

edited

chidanandpujar commented Nov 21, 2023 •

edited

chidanandpujar commented Nov 23, 2023 •

edited

kpetremann commented Nov 23, 2023 •

edited

chidanandpujar commented Nov 24, 2023

apurvaraghu commented Mar 4, 2024

kpetremann commented Mar 5, 2024

apurvaraghu commented Mar 5, 2024 •

edited

chidanandpujar commented May 6, 2024

ignore_warnings causing "'str' object has no attribute '_root'" #1245

ignore_warnings causing "'str' object has no attribute '_root'" #1245

Comments

kpetremann commented Apr 25, 2023

chidanandpujar commented May 17, 2023

kpetremann commented May 17, 2023 • edited

kpetremann commented May 17, 2023 • edited

kpetremann commented May 17, 2023

chidanandpujar commented May 17, 2023

kpetremann commented May 17, 2023

kpetremann commented Jul 6, 2023

kpetremann commented Oct 29, 2023

chidanandpujar commented Nov 2, 2023

chidanandpujar commented Nov 8, 2023 • edited

kpetremann commented Nov 9, 2023

chidanandpujar commented Nov 9, 2023 • edited

chidanandpujar commented Nov 21, 2023 • edited

chidanandpujar commented Nov 23, 2023 • edited

kpetremann commented Nov 23, 2023 • edited

chidanandpujar commented Nov 24, 2023

apurvaraghu commented Mar 4, 2024

kpetremann commented Mar 5, 2024

apurvaraghu commented Mar 5, 2024 • edited

chidanandpujar commented May 6, 2024

kpetremann commented May 17, 2023 •

edited

kpetremann commented May 17, 2023 •

edited

chidanandpujar commented Nov 8, 2023 •

edited

chidanandpujar commented Nov 9, 2023 •

edited

chidanandpujar commented Nov 21, 2023 •

edited

chidanandpujar commented Nov 23, 2023 •

edited

kpetremann commented Nov 23, 2023 •

edited

apurvaraghu commented Mar 5, 2024 •

edited