vRouer w/DPDK and Mellanox NIC #94
Comments
|
Adding some more finding... Continue our earlier call, from contrail point of view it looks all pointing to the right config, pointing to the right PCI bus/slot and MAC address. Network devices using DPDK-compatible driver0000:05:00.0 'MT27520 Family [ConnectX-3 Pro]' drv=igb_uio unused= Network devices using kernel driver0000:03:00.0 'I350 Gigabit Network Connection' if=em0 drv=igb unused=igb_uio Active Other network devicesroot@Setup-H-C0:~#What is the meaning of the Active ???? root@Setup-H-C0:/var/log/contrail# less contrail-vrouter-dpdk-stdout.log EAL: PCI device 0000:03:00.0 on NUMA socket 0 Nir. |
|
“Active” might mean that it is in use by the kernel driver (I didn’t confirm by looking at the dpdk_nic_bind script, Raja From: Nir Henn notifications@github.com Adding some more finding... Continue our earlier call, from contrail point of view it looks all pointing to the right config, pointing to the right PCI bus/slot and MAC address. Network devices using DPDK-compatible driver 0000:05:00.0 'MT27520 Family [ConnectX-3 Pro]' drv=igb_uio unused= Network devices using kernel driver 0000:03:00.0 'I350 Gigabit Network Connection' if=em0 drv=igb unused=igb_uio Active Other network devices root@Setup-H-C0:~# What is the meaning of the Active ???? root@Setup-H-C0:/var/log/contrail# less contrail-vrouter-dpdk-stdout.log EAL: PCI device 0000:03:00.0 on NUMA socket 0 Nir. — |
|
The two compute nodes show: root@Setup-H-C1:~# contrail-status The controller shows: == Contrail Analytics == == Contrail Config == == Contrail Web UI == == Contrail Database == == Contrail Support Services == root@Setup-H-Master:~# on the vif it shows only one interface 0/2 but no vhost0 Potential problem from the contrail-vrouter-dpdk-stdout.log…. Does the DPDK build is linked with Mellanox OFED drivers correctly?
|
|
We have not tested with Mellanox NICs. We use DPDK 2.1. I am not sure if it has the drivers for Mellanox NICs. Raja From: Nir Henn notifications@github.com The two compute nodes show: contrail-vrouter-nodemgr active root@Setup-H-C1:~# contrail-status contrail-vrouter-nodemgr active The controller shows: contrail-control-nodemgr active contrail-dns active contrail-named active == Contrail Analytics == contrail-analytics-api active contrail-analytics-nodemgr active contrail-collector active contrail-query-engine active contrail-snmp-collector active contrail-topology active == Contrail Config == contrail-config-nodemgr active contrail-device-manager active contrail-discovery:0 active contrail-schema active contrail-svc-monitor active ifmap active == Contrail Web UI == contrail-webui-middleware active == Contrail Database == kafka active == Contrail Support Services == root@Setup-H-Master:~# on the vif it shows only one interface 0/2 but no vhost0 Potential problem from the contrail-vrouter-dpdk-stdout.log…. Does the DPDK build is linked with Mellanox OFED drivers correctly?
— |
|
I have also looked in a Juniper Contrail install instance we have and it looks like it uses DPDK 2.1 which do have reference to mlnx4 driver. There is a procedure from Mellanox how to compile it (DODK) with mlnx but it is not clear what will be the Contrail proceedure to update the node with the new setup. Is there dependency with the DKMS ? What is the proceedure to upadate NIC type regardless the Mellanox support? |
|
You should be able to build the contrail-vrouter-dpdk binary (with any DPDK changes if required) and Raja From: Nir Henn notifications@github.com I have also looked in a Juniper Contrail install instance we have and it looks like it uses DPDK 2.1 which do have reference to mlnx4 driver. There is a procedure from Mellanox how to compile it (DODK) with mlnx but it is not clear what will be the Contrail proceedure to update the node with the new setup. Is there dependency with the DKMS ? What is the proceedure to upadate NIC type regardless the Mellanox support? — |
|
Did anyone get this to work? EAL: PCI device 0000:02:00.0 on NUMA socket -1 Looks like it’s only looking for Intel cards, as Mellanox is being ignored and not added. root@node-110: in an Intel only setup it's working fine: |
|
I also gave it a try in the past and after speaking with Mellanox I learned it is not supported.
They might have it in future releases of Contrail.
Nir.
…--
Nir Henn
Innovation Coach, AT&T Foundry
Desk +97297765298
Mobile +972547606867
AMDOCS | EMBRACE CHALLENGE EXPERIENCE SUCCESS
This email was sent using 100% recycled electrons!
STOP texting while driving. Take the pledge at itcanwait.com<http://www.itcanwait.com/>
From: jjgrinwis <notifications@github.com>
Reply-To: Juniper/contrail-vrouter <reply@reply.github.com>
Date: Saturday, February 18, 2017 at 11:37 AM
To: Juniper/contrail-vrouter <contrail-vrouter@noreply.github.com>
Cc: Nir Henn <nir.henn@amdocs.com>, State change <state_change@noreply.github.com>
Subject: Re: [Juniper/contrail-vrouter] vRouer w/DPDK and Mellanox NIC (#94)
Did anyone get this to work?
We're seeing the same issues and looks like the vRouter is not detecting our Mellanox cards:
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: probe driver: 8086:1521 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: probe driver: 8086:1521 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped
....
Looks like it’s only looking for Intel cards, as Mellanox is being ignored and not added.
root@node-110:# lspci -s 0000:02:00.0
02:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
root@node-110:# lspci -s 0000:01:00.0
01:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
in an Intel only setup it's working fine:
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL: probe driver: 8086:1521 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:01:00.1 on NUMA socket 0
EAL: probe driver: 8086:1521 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:0e:00.0 on NUMA socket 0
EAL: probe driver: 8086:10fb rte_ixgbe_pmd
EAL: PCI memory mapped at 0x7f5540000000
EAL: PCI memory mapped at 0x7f5540080000
PMD: eth_ixgbe_dev_init(): MAC: 2, PHY: 12, SFP+: 3
PMD: eth_ixgbe_dev_init(): port 1 vendorID=0x8086 deviceID=0x10fb
EAL: PCI device 0000:0e:00.1 on NUMA socket 0
EAL: probe driver: 8086:10fb rte_ixgbe_pmd
EAL: PCI memory mapped at 0x7f5540084000
EAL: PCI memory mapped at 0x7f5540104000
PMD: eth_ixgbe_dev_init(): MAC: 2, PHY: 12, SFP+: 4
PMD: eth_ixgbe_dev_init(): port 2 vendorID=0x8086 deviceID=0x10fb
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub<#94 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AKB1nrABRshU3o8sSrBU1sTddycrb_Kbks5rdrvJgaJpZM4J_27s>.
This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement,
you may review at http://www.amdocs.com/email_disclaimer.asp
|
|
We have not tested this with Mellanox NICs yet.
Raja
From: jjgrinwis <notifications@github.com>
Reply-To: Juniper/contrail-vrouter <reply@reply.github.com>
Date: Saturday, February 18, 2017 at 1:37 AM
To: Juniper/contrail-vrouter <contrail-vrouter@noreply.github.com>
Cc: Rajagopalan Sivaramakrishnan <raja@juniper.net>, Comment <comment@noreply.github.com>
Subject: Re: [Juniper/contrail-vrouter] vRouer w/DPDK and Mellanox NIC (#94)
Did anyone get this to work?
We're seeing the same issues and looks like the vRouter is not detecting our Mellanox cards:
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: probe driver: 8086:1521 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: probe driver: 8086:1521 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped
....
Looks like it’s only looking for Intel cards, as Mellanox is being ignored and not added.
root@node-110:# lspci -s 0000:02:00.0
02:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
root@node-110:# lspci -s 0000:01:00.0
01:00.0 Ethernet controller: Mellanox Technologies MT27700 Family [ConnectX-4]
in an Intel only setup it's working fine:
EAL: PCI device 0000:01:00.0 on NUMA socket 0
EAL: probe driver: 8086:1521 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:01:00.1 on NUMA socket 0
EAL: probe driver: 8086:1521 rte_igb_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:0e:00.0 on NUMA socket 0
EAL: probe driver: 8086:10fb rte_ixgbe_pmd
EAL: PCI memory mapped at 0x7f5540000000
EAL: PCI memory mapped at 0x7f5540080000
PMD: eth_ixgbe_dev_init(): MAC: 2, PHY: 12, SFP+: 3
PMD: eth_ixgbe_dev_init(): port 1 vendorID=0x8086 deviceID=0x10fb
EAL: PCI device 0000:0e:00.1 on NUMA socket 0
EAL: probe driver: 8086:10fb rte_ixgbe_pmd
EAL: PCI memory mapped at 0x7f5540084000
EAL: PCI memory mapped at 0x7f5540104000
PMD: eth_ixgbe_dev_init(): MAC: 2, PHY: 12, SFP+: 4
PMD: eth_ixgbe_dev_init(): port 2 vendorID=0x8086 deviceID=0x10fb
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub<#94 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AFni19MiSXn9aYLz59KiXEoiDCxp39fMks5rdrvFgaJpZM4J_27s>.
|
Hi All,
I am trying to setup vRouter w/DPDK and ConnectX-3-Pro.
Is there any documentation how to?
The problem is that I cannot bring vif to recognize the DPPK enabled interface.
The current state is that install script identifies correctly and interface is binded to the DPDK.
root@Setup-H-C0:/opt/contrail/bin# ./dpdk_nic_bind.py -s
Network devices using DPDK-compatible driver
0000:05:00.0 'MT27520 Family [ConnectX-3 Pro]' drv=igb_uio unused=
Network devices using kernel driver
0000:03:00.0 'I350 Gigabit Network Connection' if=em0 drv=igb unused=igb_uio Active
0000:03:00.3 'I350 Gigabit Network Connection' if=eth1 drv=igb unused=igb_uio
Other network devices
Since OFED + PMD drivers are taking over it is not in control of the OS any more and does not show up in ifconfig.
in - /etc/contrail/contrail-vrouter-agent.conf
DPDK or legacy work mode
platform=dpdk
Physical address of PCI used by dpdk
physical_interface_address=0000:05:00.0
MAC address of device used by dpdk
physical_interface_mac=7c:fe:90:b1:ea:50
in - /etc/contrail/agent_param
LOG=/var/log/contrail.log
CONFIG=/etc/contrail/contrail-vrouter-agent.conf
prog=/usr/bin/contrail-vrouter-agent
kmod=vrouter
pname=contrail-vrouter-agent
LIBDIR=/usr/lib64
DEVICE=vhost0
dev=p785p1
vgw_subnet_ip=VGW_SUBNET_IP
vgw_intf=VGW_INTF_LIST
LOGFILE=--log-file=/var/log/contrail/vrouter.log
The text was updated successfully, but these errors were encountered: