Poor benchmark performance all .NET versions

[johnnygiter]

URL(s)

https://github.com/dotnet/core/tree/main

Description

Almost last place. Concering all versions
Shame that compilable language is slower than Python Regius

https://programming-language-benchmarks.vercel.app/problem/edigits
few others benchmark not impressive either..
https://programming-language-benchmarks.vercel.app/problem/pidigits

[tannergooding]

Benchmarks like these have to be taken with a grain of salt.

Both e-digits and pi-digits are not representative of real world work and are instead only representative of something that one would do purely for fun. Not only does any non-specialized use of pi not need more than around 39 digits (the number needed to compute the circumference of the observable universe to within 1 atom of precision), but there are better ways to get the first n-digits of these irrational numbers as there are actual supercomputers that have already written highly fine tuned applications that have computed 105 trillion digits and where the code looks nothing like any of the given implementations.

Additionally, the initial code for a platform is often authored in the most naive way possible. It isn't accounting for things like whether the functionality is built-in, whether it is actually executing in the language in question, whether it may be deferring down to another language where the actual core logic is running, etc. The community is then responsible for coming in and giving alternative implementations that may do better and more optimal things, with different languages getting different levels of interest. This means that the actual implementation of for two languages could be very different and make them not exactly comparable.

The way they are measuring the code also has to be accounted for, as they are not always measuring the code in identical scenarios. Some of the languages are being compiled AOT with all possible optimizations enabled, some of these compilers have built-in optimizations to recognize some of these patterns and effectively compile the code down to a near no-op (which is only possible when you have scenarios like this where everything is constant). Others are being compiled JIT and the timings are including that overhead plus not correctly accounting for other features like Tiered Compilation.

---

Now, the main reason the C# implementation for these two benchmarks is much slower is effectively down to the BigInteger implementation and how the code is then using it.

In Python, every number is a "bignum" and so they've spent much more time fine tuning this handling to ensure that the common cases remain performant. Their big integer type is designed to work with any code, to account for people writing the most naive possible algorithms, and has specialized compiler support to do optimizations and other logic to make it as optimized as possible.

In C#, we instead have both fixed-sized types (int, long, Int128, etc) and the arbitrarily sized System.Numerics.BigInteger. We expect that the fixed-sized types are the typical ones used and that BigInteger is only used in specialized scenarios. We likewise have reasonable assumptions in BigInteger around the maximum size we'll see and while we do support larger sizes, they've not received as much care or optimization work to make those scenarios the most efficient possible.

Correspondingly, this means that BigInteger has some design decisions around it like it being immutable and therefore (a + b) / 2 will involve allocating at twice and traversing the memory of inputs 4 times (once for a, once for b, once to create temporary that (a + b) produced, and then once more for that temporary as part of the tmp / 2 operation). The naive implementation is therefore not very efficient and there are often better ways to handle this.

There's also some nuance in that because we have many different integer types, BigInteger doesn't get as much focus since its primarily intended for specialized scenarios and not for "everyday scenarios". This does mean that we have some known optimization opportunities to make it perform better on modern hardware and even to allow users to more easily write naive logic (like done in the benchmark) without seeing terrible performance. There hasn't been an opportunity to do this optimization work as of yet, namely because there is work that is higher priority and more impactful to real world workloads to be completed in other areas first.

The other factor is that the code isn't actually taking into account how C# operates, what functionality is built into the BCL, etc. It's missing opportunities to use constants like double.Tau (while it is using these for Python and Rust), it's missing opportunities to do more efficient string formatting using span/slices, that doesn't force many copies and allocations, and so on.

[johnnygiter]

Simple pow on bigintegers is horribly slow comparing to other languages too (even python is much faster like over 300%)...
Checked on .net6 and .net8 almost same.

using System.Numerics;

class Program
{
    static void Main()
    {

        string baseStr = "1000";
        string exponentStr = "50000";

        BigInteger baseNum = BigInteger.Parse(baseStr);
        BigInteger exponentNum = BigInteger.Parse(exponentStr);
        BigInteger result = BigInteger.Pow(baseNum, (int)exponentNum);
        Console.WriteLine($"Result: {result}");
    }
}

[tannergooding]

That program is:

1. Including the overhead of parsing
2. Not accounting for Tiered Compilation or DPGO (and therefore likely running unoptimized debug code)
3. Doing unnecessary extra work by converting the big integer exponent back to an int

As explained above, there are known cases where the .NET BigInteger code could be improved. It was also explained that there's a lot of nuance towards benchmarking, real world perf considerations, etc. It is generally not sufficient to take a single simple program and use it as some metric of proof showing that it means X is better or faster than Y.