juniper_junos_command - lxml.etree.XMLSyntaxError: StartTag: invalid element name

[fredtj]

Issue Type

• Bug Report
  Module Name: juniper_junos_command

---

ansible==2.7.7
apache-libcloud==2.4.0
asn1crypto==0.24.0
bcrypt==3.1.6
certifi==2018.8.24
chardet==3.0.4
cryptography==2.6.1
entrypoints==0.3
future==0.18.2
httplib2==0.11.3
idna==2.6
Jinja2==2.10
jmespath==0.9.4
junos-eznc==2.5.3
jxmlease==1.0.3
keyring==17.1.1
keyrings.alt==3.1.1
lockfile==0.12.2
lxml==4.5.2
MarkupSafe==1.1.0
ncclient==0.6.9
netaddr==0.7.19
ntc-templates==1.5.0
ntlm-auth==1.1.0
paramiko==2.4.2
pyasn1==0.4.2
pycrypto==2.6.1
pycurl==7.43.0.2
PyGObject==3.30.4
pykerberos==1.1.14
PyNaCl==1.3.0
pyOpenSSL==19.0.0
pyparsing==2.4.7
pyserial==3.4
PySimpleSOAP==1.16.2
python-apt==1.8.4.1
python-debian==0.1.35
python-debianbts==2.8.2
pywinrm==0.3.0
pyxdg==0.25
PyYAML==5.3.1
reportbug===7.5.3-deb10u1
requests==2.21.0
requests-kerberos==0.11.0
requests-ntlm==1.1.0
scp==0.13.2
SecretStorage==2.3.1
simplejson==3.16.0
six==1.12.0
textfsm==1.1.0
transitions==0.8.2
urllib3==1.24.1
xmltodict==0.11.0
yamlordereddictloader==0.4.0

OS / Environment

SRX110-H2-VA
12.3X48-D105.4

Summary

Using following playbook, an error occurs

- name: Get junos commit diff
  hosts: diff-test
  connection: local
  gather_facts: no
  roles:
    - Juniper.junos
  tasks:
    - name: Diff the config
      juniper_junos_command:
        host: "{{ inventory_hostname }}"
        user: me
        ssh_private_key_file: ../mine
        port: 22
        dest_dir: /home/ansible/juniper-commit-diffs
        command: show system rollback 0 compare 1

Expected results
output of command saved to file

Actual results

FAILED! => {"changed": false, "module_stderr": "/home/ansible/.ansible/tmp/ansible-tmp-1604074072.5774634-243926931920102/AnsiballZ_juniper_junos_command.py:17: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses\n  import imp\n/usr/lib/python3/dist-packages/paramiko/kex_ecdh_nist.py:39: CryptographyDeprecationWarning: encode_point has been deprecated on EllipticCurvePublicNumbers and will be removed in a future version. Please use EllipticCurvePublicKey.public_bytes to obtain both compressed and uncompressed point encoding.\n  m.add_string(self.Q_C.public_numbers().encode_point())\n/usr/lib/python3/dist-packages/paramiko/kex_ecdh_nist.py:96: CryptographyDeprecationWarning: Support for unsafe construction of public numbers from encoded data will be removed in a future version. Please use EllipticCurvePublicKey.from_encoded_point\n  self.curve, Q_S_bytes\n/usr/lib/python3/dist-packages/paramiko/kex_ecdh_nist.py:111: CryptographyDeprecationWarning: encode_point has been deprecated on EllipticCurvePublicNumbers and will be removed in a future version. Please use EllipticCurvePublicKey.public_bytes to obtain both compressed and uncompressed point encoding.\n  hm.add_string(self.Q_C.public_numbers().encode_point())\n/usr/local/lib/python3.7/dist-packages/jnpr/junos/device.py:837: RuntimeWarning: An unknown exception occurred - please report.\n  \"An unknown exception occurred - please report.\", RuntimeWarning\nTraceback (most recent call last):\n  File \"/home/ansible/.ansible/tmp/ansible-tmp-1604074072.5774634-243926931920102/AnsiballZ_juniper_junos_command.py\", line 113, in <module>\n    _ansiballz_main()\n  File \"/home/ansible/.ansible/tmp/ansible-tmp-1604074072.5774634-243926931920102/AnsiballZ_juniper_junos_command.py\", line 105, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/home/ansible/.ansible/tmp/ansible-tmp-1604074072.5774634-243926931920102/AnsiballZ_juniper_junos_command.py\", line 48, in invoke_module\n    imp.load_module('__main__', mod, module, MOD_DESC)\n  File \"/usr/lib/python3.7/imp.py\", line 234, in load_module\n    return load_source(name, filename, file)\n  File \"/usr/lib/python3.7/imp.py\", line 169, in load_source\n    module = _exec(spec, sys.modules[name])\n  File \"<frozen importlib._bootstrap>\", line 630, in _exec\n  File \"<frozen importlib._bootstrap_external>\", line 728, in exec_module\n  File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed\n  File \"/tmp/ansible_juniper_junos_command_payload__rh8996j/__main__.py\", line 520, in <module>\n  File \"/tmp/ansible_juniper_junos_command_payload__rh8996j/__main__.py\", line 434, in main\n  File \"/usr/local/lib/python3.7/dist-packages/jnpr/junos/rpcmeta.py\", line 387, in __call__\n    return self._junos.execute(rpc_cmd, **kvargs)\n  File \"/usr/local/lib/python3.7/dist-packages/jnpr/junos/decorators.py\", line 71, in wrapper\n    return function(*args, **kwargs)\n  File \"/usr/local/lib/python3.7/dist-packages/jnpr/junos/decorators.py\", line 31, in wrapper\n    return function(*args, **kwargs)\n  File \"/usr/local/lib/python3.7/dist-packages/jnpr/junos/device.py\", line 816, in execute\n    filter_xml=kvargs.get(\"filter_xml\"),\n  File \"/usr/local/lib/python3.7/dist-packages/jnpr/junos/decorators.py\", line 117, in wrapper\n    rsp = function(self, *args, **kwargs)\n  File \"/usr/local/lib/python3.7/dist-packages/jnpr/junos/device.py\", line 1419, in _rpc_reply\n    return self._conn.rpc(rpc_cmd_e, filter_xml)._NCElement__doc\n  File \"/usr/local/lib/python3.7/dist-packages/ncclient/manager.py\", line 231, in execute\n    huge_tree=self._huge_tree).request(*args, **kwds)\n  File \"/usr/local/lib/python3.7/dist-packages/ncclient/operations/third_party/juniper/rpc.py\", line 52, in request\n    return self._request(rpc)\n  File \"/usr/local/lib/python3.7/dist-packages/ncclient/operations/rpc.py\", line 350, in _request\n    self._reply.parse()\n  File \"/usr/local/lib/python3.7/dist-packages/ncclient/operations/rpc.py\", line 160, in parse\n    root = self._root = to_ele(self._raw, huge_tree=self._huge_tree) # The <rpc-reply> element\n  File \"/usr/local/lib/python3.7/dist-packages/ncclient/xml_.py\", line 124, in to_ele\n    return x if etree.iselement(x) else etree.fromstring(x.encode('UTF-8'), parser=_get_parser(huge_tree))\n  File \"src/lxml/etree.pyx\", line 3237, in lxml.etree.fromstring\n  File \"src/lxml/parser.pxi\", line 1896, in lxml.etree._parseMemoryDocument\n  File \"src/lxml/parser.pxi\", line 1784, in lxml.etree._parseDoc\n  File \"src/lxml/parser.pxi\", line 1141, in lxml.etree._BaseParser._parseDoc\n  File \"src/lxml/parser.pxi\", line 615, in lxml.etree._ParserContext._handleParseResultDoc\n  File \"src/lxml/parser.pxi\", line 725, in lxml.etree._handleParseResult\n  File \"src/lxml/parser.pxi\", line 654, in lxml.etree._raiseParseError\n  File \"<string>\", line 248\nlxml.etree.XMLSyntaxError: StartTag: invalid element name, line 248, column 40\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

on the device cli the command returns the expected output. out of hundreds of devices, 3 are giving this failure at the moment.

[rahkumar651991]

Hi @fredtj -
Try below playbook.

    - name: Print diff between current and rollback 1. No check. No commit.
      juniper_junos_config:
        rollback: 1
        diff: true
        check: false
        commit: false
      register: response

    - name: Print the msg.
      debug:
        var: response

Sample Output :

 "response": {
        "changed": true,
        "diff": {
            "prepared": "\n[edit]\n+  interfaces {\n+      ge-0/0/1 {\n+          disable;\n+      }\n+  }\n"
        },
        "diff_lines": [
            "",
            "[edit]",
            "+  interfaces {",
            "+      ge-0/0/1 {",
            "+          disable;",
            "+      }",
            "+  }"
        ],
        "failed": false,
        "msg": "Configuration has been: opened, rolled back, diffed, closed.",
    }

Refer - https://junos-ansible-modules.readthedocs.io/en/2.4.0/juniper_junos_config.html for similar examples.

Let us know if this works for you.

[fredtj]

hello, thanks, i did test using that method, but it is not comparing the configs as i would like - it is comparing the configs the wrong way round (ie the + and - etc are reversed)

[fredtj]

@rahkumar651991 i've tracked this problem down, the error occurs when the diff/compare contains an & or < or >

any chance you could take a look please?

[fredtj]

the problem line is in library/juniper_junos_command.py

resp = junos_module.dev.rpc(rpc, ignore_warning=ignore_warning, normalize=bool(format == 'xml'))

[fredtj]

the traceback is very similar to: https://stackoverflow.com/questions/37005830/how-to-read-an-xml-file-with-sign/37006646

is there any workaround you are aware of?

[fredtj]

finally managed to fix this by editing _xml.py in ncclient package and adding recover=True to the XMLParser construct:

  parser = etree.XMLParser(recover=True)
  huge_parser = etree.XMLParser(recover=True, huge_tree=True)

    @property
    def tostring(self):
        """return a pretty-printed string output for rpc reply"""
        parser = etree.XMLParser(recover=True, remove_blank_text=True, huge_tree=self.__huge_tree)
        outputtree = etree.XML(etree.tostring(self.__doc), parser)
        return etree.tostring(outputtree, pretty_print=True)

not sure why it is a problem as when i run "show system rollback 0 compare 1 | display xml" on the device, the ampersand character is returned as &

@rahkumar651991 let me know if i should report this elsewhere

[rahkumar651991]

@fredtj - thanks for the analysis. Could you help me with the exact output you are seeing for the error.
both

1. when run manually on device cli
2. received at the ncclient/ansible

I am not able to replicate the scenario in my local setup. I will try to hardocode the values and try to execute it.

[fredtj]

first, create a commit containing an & character:

me@srx110> show system rollback 0 compare 1 
[edit snmp]
-  location "Blah Blah, Blah Blah Blah, Blah Blah";
+  location "Blah Blah, Blah Blah Blah, Blah Blah &";

me@srx110> show system rollback 0 compare 1 | display xml 
<rpc-reply xmlns:junos="http://xml.juniper.net/junos/12.3X48/junos">
    <rollback-information>
        <configuration-information>
            <configuration-output>
                [edit snmp]
                -  location "Blah Blah, Blah Blah Blah, Blah Blah";
                +  location "Blah Blah, Blah Blah Blah, Blah Blah &amp;";
            </configuration-output>
        </configuration-information>
    </rollback-information>
    <cli>
        <banner></banner>
    </cli>
</rpc-reply>

without the & sign in the commit, ansible runs the command "show system rollback 0 compare 1" OK.

With the & sign, here is the error:

/root/.ansible/tmp/ansible-tmp-1606920050.6425843-134013982255409/AnsiballZ_juniper_junos_command.py:17: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
/usr/lib/python3/dist-packages/paramiko/kex_ecdh_nist.py:39: CryptographyDeprecationWarning: encode_point has been deprecated on EllipticCurvePublicNumbers and will be removed in a future version. Please use EllipticCurvePublicKey.public_bytes to obtain both compressed and uncompressed point encoding.
  m.add_string(self.Q_C.public_numbers().encode_point())
/usr/lib/python3/dist-packages/paramiko/kex_ecdh_nist.py:96: CryptographyDeprecationWarning: Support for unsafe construction of public numbers from encoded data will be removed in a future version. Please use EllipticCurvePublicKey.from_encoded_point
  self.curve, Q_S_bytes
/usr/lib/python3/dist-packages/paramiko/kex_ecdh_nist.py:111: CryptographyDeprecationWarning: encode_point has been deprecated on EllipticCurvePublicNumbers and will be removed in a future version. Please use EllipticCurvePublicKey.public_bytes to obtain both compressed and uncompressed point encoding.
  hm.add_string(self.Q_C.public_numbers().encode_point())
/usr/local/lib/python3.7/dist-packages/jnpr/junos/device.py:837: RuntimeWarning: An unknown exception occurred - please report.
  "An unknown exception occurred - please report.", RuntimeWarning
Traceback (most recent call last):
  File "/root/.ansible/tmp/ansible-tmp-1606920050.6425843-134013982255409/AnsiballZ_juniper_junos_command.py", line 113, in <module>
    _ansiballz_main()
  File "/root/.ansible/tmp/ansible-tmp-1606920050.6425843-134013982255409/AnsiballZ_juniper_junos_command.py", line 105, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/root/.ansible/tmp/ansible-tmp-1606920050.6425843-134013982255409/AnsiballZ_juniper_junos_command.py", line 48, in invoke_module
    imp.load_module('__main__', mod, module, MOD_DESC)
  File "/usr/lib/python3.7/imp.py", line 234, in load_module
    return load_source(name, filename, file)
  File "/usr/lib/python3.7/imp.py", line 169, in load_source
    module = _exec(spec, sys.modules[name])
  File "<frozen importlib._bootstrap>", line 630, in _exec
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/tmp/ansible_juniper_junos_command_payload_gswqsd8b/__main__.py", line 520, in <module>
  File "/tmp/ansible_juniper_junos_command_payload_gswqsd8b/__main__.py", line 434, in main
  File "/usr/local/lib/python3.7/dist-packages/jnpr/junos/rpcmeta.py", line 387, in __call__
    return self._junos.execute(rpc_cmd, **kvargs)
  File "/usr/local/lib/python3.7/dist-packages/jnpr/junos/decorators.py", line 71, in wrapper
    return function(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/jnpr/junos/decorators.py", line 31, in wrapper
    return function(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/jnpr/junos/device.py", line 816, in execute
    filter_xml=kvargs.get("filter_xml"),
  File "/usr/local/lib/python3.7/dist-packages/jnpr/junos/decorators.py", line 117, in wrapper
    rsp = function(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/jnpr/junos/device.py", line 1419, in _rpc_reply
    return self._conn.rpc(rpc_cmd_e, filter_xml)._NCElement__doc
  File "/usr/local/lib/python3.7/dist-packages/ncclient/manager.py", line 231, in execute
    huge_tree=self._huge_tree).request(*args, **kwds)
  File "/usr/local/lib/python3.7/dist-packages/ncclient/operations/third_party/juniper/rpc.py", line 52, in request
    return self._request(rpc)
  File "/usr/local/lib/python3.7/dist-packages/ncclient/operations/rpc.py", line 350, in _request
    self._reply.parse()
  File "/usr/local/lib/python3.7/dist-packages/ncclient/operations/rpc.py", line 160, in parse
    root = self._root = to_ele(self._raw, huge_tree=self._huge_tree) # The <rpc-reply> element
  File "/usr/local/lib/python3.7/dist-packages/ncclient/xml_.py", line 124, in to_ele
    return x if etree.iselement(x) else etree.fromstring(x.encode('UTF-8'), parser=_get_parser(huge_tree))
  File "src/lxml/etree.pyx", line 3237, in lxml.etree.fromstring
  File "src/lxml/parser.pxi", line 1896, in lxml.etree._parseMemoryDocument
  File "src/lxml/parser.pxi", line 1784, in lxml.etree._parseDoc
  File "src/lxml/parser.pxi", line 1141, in lxml.etree._BaseParser._parseDoc
  File "src/lxml/parser.pxi", line 615, in lxml.etree._ParserContext._handleParseResultDoc
  File "src/lxml/parser.pxi", line 725, in lxml.etree._handleParseResult
  File "src/lxml/parser.pxi", line 654, in lxml.etree._raiseParseError
  File "<string>", line 4
lxml.etree.XMLSyntaxError: xmlParseEntityRef: no name, line 4, column 70

[fredtj]

this is also happening with < and > in the commit, so failures occur for things like the following:

set policy-options prefix-list ntp-servers apply-path "system ntp server <*>"

[dineshbaburam91]

As a workaround, you can use the below way. Let me follow up with ncclient bug

  tasks:
    - name: Configure policy
      config:
        rollback: "1"
        diff: true
        check: false
        commit: true 
      register: result

ok: [vm1] => {
    "result": {
        "changed": true,
        "diff": {
            "prepared": "\n[edit]\n-  snmp {\n-      location \"blah blah &\";\n-  }\n"
        },
        "diff_lines": [
            "",
            "[edit]",
            "-  snmp {",
            "-      location \"blah blah &\";",
            "-  }"
        ],
        "failed": false,
        "msg": "Configuration has been: opened, rolled back, diffed, committed, closed."
    }

[einarnn]

Please note that to be valid in an XML document, this text:

set policy-options prefix-list ntp-servers apply-path "system ntp server <*>"

...would ideally be encoded as:

set policy-options prefix-list ntp-servers apply-path "system ntp server <*>"

If it is not encoded thus, the lxml library used by ncclient will correctly generate an error when it tries to parse the payload.

I have commented similarly on an issue raised by @dineshbaburam91 on the ncclient issue tracker, ncclient/ncclient#580.

If you can provide more details on the actual XML payload that is triggering the error in parsing with the error stack shown in #528 (comment), that would be helpful.