SendPacket latency question

[lk-davidegironi]

Hi all.

I'm using Sharppcap Npcap mode to receive and send Ethernet II RAW packet.

I've notice a behaviour I'm not able to explain myself. I suppose it's related to some kind of queue laying on the underline pcap level, but I'm here to ask you.

First of all my hardware:

• PC side: Windows 10 21H2 on i7, NIC is Realtek RTL8111E
• Client side: Xilinx FPGA

Sharppcap function used to read and write packet:

• StartCapture + OnPacketArrival event handle: read the packet
• SendPacket(byte[] p, int size): send the packet back

Then my test logic:

• FPGA use his gigabit PHY to send a RAW packet of 320byte at some samplerate (ex. 1kHz). The RAW packet consists of an header with a counter, some stats bytes and a payload. PC side the packet is read and send back a packet to the FPGA.
• The counter in the FPGA packet is used to estimate stats.
• FPGA sends the packet with counter=N, PC has to reply with packet N. Next packet will be N+1, and so on. Let's for a while skip the overflow problem for the packet counter.
• If a packet takes more than samplerate time to be replied from the PC, this packet is catched as "delayed". stats field: delayed_packets_number
• The time that the packet takes to be replied from the PC is called "flyback time". stats field: flybacktime_microseconds
• PC side there's a stat that counts how many microseconds it takes to send the packet back. stats field: working_time_microseconds

Code testing:

• FPGA side all is tested using two FPGA as master (sender) / slave (replier)
• PC side the C# is in .NET Core / .NET standard. CPU usage is low (5% or less), memory usage also is low (45Mb)

Now the behaviour:

• Running the test at 1kHz I've a mean flyback time of 270us, mean success packet of 900 out of 1000 (90%), mean C# stat working SendPacket time 40us
• Running the test at 10kHz I've a mean flyback time of 50us, mean success packet of 4000 out of 10000 (40%), mean C# stat working SendPacket time 3us

The question:

I don't expect a fully working environment, due to the non-real-time nature of the Windows kernel. But that's something "strange" that I can not explain myself. The flyback time should be always as low as the 10kHz time (or slower), the 1kHz flyback time makes me think that packets are send out only after some kind of queue is fulfilled, and of course with 1kHz time this queue takes more to fulfill. There is something I don't catch here, or something I've made wrong. Here I need your help to achive better performance.

Thank you!

Find below the C# code I'm using. This is not the exact code I'm using cause my code is part of and Interface, but is the extract of that code that has some value for the subject.

private Stopwatch _statsStopwatch1 = new Stopwatch();
private ILiveDevice _device = null;
private int _statsPacketInNum = 0;
private long _statsWorkerTimeMs = 0;

private void Initialize() {
  CaptureDeviceList devices = CaptureDeviceList.Instance;
  _device = devices.Where(r => r.Description != null).Where(r => r.Description == description).FirstOrDefault();
  _device.OnPacketArrival += new PacketArrivalEventHandler(OnPacketArrival);
  _device.Open(new DeviceConfiguration { Mode = DeviceModes.Promiscuous, Immediate = true, ReadTimeout = 1 });
  _device.Filter = "ether src host 48-00-00-00-00-00";
  _device.StartCapture();
}

private void OnPacketArrival(object sender, PacketCapture e) {
  _statsPacketInNum++;
  RawCapture rawPacket = e.GetPacket();
  Packet packet = Packet.ParsePacket(rawPacket.LinkLayerType, rawPacket.Data);
  if (packet != null && packet is EthernetPacket eth) {
    EthernetPacket ethpacket = packet as EthernetPacket;
    if (ethpacket.HasPayloadData) {
      byte[] data = ethpacket.PayloadData;
      // edit data packet
      // ...
      // send packet back
      List<byte> packet = new List<byte>();
      packet.AddRange("48-00-00-00-00-00");
      packet.AddRange("00-57-AA-12-66-22");
      packet.AddRange(new byte[] { 0x55, 0x55 });
      packet.AddRange(data);
      _statsStopwatch1.Reset();
      _statsStopwatch1.Start();
      _device.SendPacket(packet.ToArray(), packet.Count);
      _statsStopwatch1.Stop();
      _statsWorkerTimeMs += _statsStopwatch1.ElapsedMilliseconds; // this value is then divided by _statsPacketInNum to get the mean time
   }
}

[kayoub5]

@lk-davidegironi

• Npcap version? Possibly related to Sometimes pcap_sendpacket takes longer in some situations nmap/npcap#619 Npcap 1.20 have better sending performance as far as I have seen.
• Any antivirus software on the PC?
• Try to change GC mode https://learn.microsoft.com/en-us/dotnet/core/runtime-config/garbage-collector#flavors-of-garbage-collection

[lk-davidegironi]

Thanks @kayoub5 for your interest

• Npcap version: 1.71. I try with other versions 1.60, 1.72, 1.55... same thing
• ESET32 antivirus + firewall (disabling does not makes difference). Try with a laptop i7 based with Realtek board, same behaviour
• No luck changing GC mode

One more thing I've try is using an Intel i210 NIC board. Nothing changed.
I'll keep you updated... in case I found something usefull.

[kayoub5]

• Try Npcap 1.20 or 1.10
• Disabling anti-virus does not remove its performance effects, you have to actually uninstall it (and reboot for changes to take effects)
• Is the cable full duplex or half duplex?

[kayoub5]

other things to consider:

• thread and process priority
• using different device object for sending and receiving
• disable ipv4 and ipv6 from device properties

[lk-davidegironi]

Thank you @kayoub5

• Try Npcap 1.20 and all the version newer than 1.10 = No effects
• Disabling anti-virus does not remove its performance effects, you have to actually uninstall it (and reboot for changes to take effects) = Done on the other laptop PC, no effects
• Is the cable full duplex or half duplex? = Full duplex, also I've try other cables. No effects
• thread and process priority = Although my process is set to RealTime (ProcessPriorityClass.RealTime) it starts in High priority. Changing it by hand on the Task Manager does not affect performances on my i7. I makes some small difference only if i take down to Normal level
• using different device object for sending and receiving = Tried on Realtek RTL8111E and Intel i210, makes no different
• disable ipv4 and ipv6 from device properties = In both the devices I've try, all is disabled except "Npcap Packet Driver" and "Link-Layer Topology Discovery Responder"

I'm wondering if a plain c/c++ software could help me for tests.

[lk-davidegironi]

Something interesting I've noticed:
.NET 5.0, .NET Core 3.0 and 3.1 has almost the same performance
.NET 6.0 or .NET 7.0 is almost half the performance of the above

[kayoub5]

I'm wondering if a plain c/c++ software could help me for tests.

Possible, I would recommend using a different driver (like winpktfilter) or a kernel bypass using dpdk if that does not help.