ConcurrentDictionary.TryGetValue crashes with System.IndexOutOfRangeException: Arg_IndexOutOfRangeException

[igor-sidorovich]

Android application type

.NET Android (net7.0-android, net8.0-android, etc.)

Affected platform version

.NET 8.0.200, workload 34.0.52/8.0.100

Description

Since updating from Xamarin to .NET 8 we are getting next crashes in ConcurrentDictionary.

00:49:25:118|FATAL| 6|AndroidRestCrashHandler|-Core-|AndroidEnvironment.UnhandledExceptionRaiser. UtcNow is 2024-04-09T19:19:25. --> System.IndexOutOfRangeException: Arg_IndexOutOfRangeException
at System.Collections.Concurrent.ConcurrentDictionary`2[[System.String, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[BB.Features.Resources.Remote.Models.RemoteResourcePackage, BB.Features, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null]].TryGetValue(String key, RemoteResourcePackage& value)

For me it looks like strange to have such crashes for ConcurrentDictionary.

Steps to Reproduce

Unfortuantle, i do not have repro(I tried but didn't succeed), but on Production we see such crashes.

Did you find any workaround?

No

Relevant log output

No response

[grendello]

@igor-sidorovich even though you cannot reproduce it, can you attach a snippet of code which shows more-or-less what you're doing when the crash happens?

Also, please gather and post info about the crash environment(s) - Android version, locale, device(s).

Lastly, could you attach full crash stack trace, with as much context as possible, that you get in the Google console?

Thanks!

[grendello]

@vitek-karas would you mind looking into this issue?

[igor-sidorovich]

1. We do not have logs in Google console for current crash(looks like because it's managed crash), only SIGSEGV, SIGABRT without stack traces.
2. I think, i can't attach code snippet because it's really huge application and it happens in different places where we use ConcurrentDictionary.
3. List of devices:
   

P.S. As i see, we have only problems with ConcurrentDictionary where key is string. The length of string(key) approximately - 30-80 symbols, keys count in one ConcurrentDictionary approximately - 20000 -34000. I tired different samples, also override GetHashCode to increase collisions, but cannot reproduce anyway.

[simonrozsival]

@igor-sidorovich do you use custom IEqualityComparer<string>? If yes, would it be possible to share the code with us?

If the problem is not with the equality comparer, the problem might be in HashHelper.FastMod which is used to get the index in the hash table. I don't see anything obviously wrong with it, but maybe it doesn't behave correctly on some ARM64 processors?

[igor-sidorovich]

1. We are use ConcurrentDictionary with default constructor -> _packages = new ConcurrentDictionary<string, RemoteResourcePackage>(). As i can see from .net source code default constructor with next parameters:

public ConcurrentDictionary()
            : this(DefaultConcurrencyLevel, DefaultCapacity, growLockArray: true, null) { }

, and looks like we are using default string comparer

private static readonly NonRandomizedStringEqualityComparer WrappedAroundDefaultComparer = new OrdinalComparer(EqualityComparer<string?>.Default);

2. But, as i can see, we always use FastMod on 64bit CPU:

[MethodImpl(MethodImplOptions.AggressiveInlining)]
        private static Node? GetBucket(Tables tables, int hashcode)
        {
            VolatileNode[] buckets = tables._buckets;
            if (IntPtr.Size == 8)
            {
                return buckets[HashHelpers.FastMod((uint)hashcode, (uint)buckets.Length, tables._fastModBucketsMultiplier)]._node;
            }
            else
            {
                return buckets[(uint)hashcode % (uint)buckets.Length]._node;
            }
        }

Do we have option do not use FastMod?

3. And pls clarify, what does it mean "it doesn't behave correctly on some ARM64 processors?"? Thx.

[simonrozsival]

@igor-sidorovich thanks for the additional information.

I tried reproducing the exception today and I wasn't successful. I tried the running the following code on my Samsung S23, and there wasn't any exception thrown by TryGetValue or any other method:

private readonly ConcurrentDictionary<string, string> _dictionary = new();
private Thread[] _threads = new Thread[11];

private void OnCounterClicked(object sender, EventArgs e)
{
	bool gotException = false;

	for (int i = 1; i < _threads.Length; i++)
	{
		_threads[i] = new Thread(() =>
		{
			while (!gotException)
			{
				try
				{
					var key = GenerateRandomString(30, 80);
					var value = "abc";

					_dictionary.TryAdd(key, value);
					_dictionary.TryGetValue(key, out _);

					Interlocked.Increment(ref count);
				}
				catch (Exception ex)
				{
					Console.WriteLine($"Exception: {ex.Message}");
					gotException = true;
				}
			}
		});
	}

	_threads[0] = new Thread(() =>
	{
		while (!gotException)
		{
			Device.BeginInvokeOnMainThread(() => CounterBtn.Text = $"i: {count}, dc: {_dictionary.Count}");
			Thread.Sleep(100);
		}
	});

	foreach (var thread in _threads)
	{
		thread.Start();
	}
}

static string GenerateRandomString(int min, int max)
{
	int length = Random.Shared.Next(min, max);
	var sb = new System.Text.StringBuilder(length	);
	for (int i = 0; i < length; i++)
	{
		sb.Append((char)Random.Shared.Next('A', 'Z' + 1));
	}
	return sb.ToString();
}

The OS eventually kills the app, but nothing suggests it is because of ConcurrentDictionary.

And pls clarify, what does it mean "it doesn't behave correctly on some ARM64 processors?"?

It means that maybe this could some glitch in arm chips? But since it crashes on all those devices in the screenshot running different chips, it's probably not the problem.

I think we need a repro if we want to fix this.

[igor-sidorovich]

I think we need a repro if we want to fix this.

I understand, but i also tried repro on different ARM phones and no any success.
That's why i created ticket, because on Xamarin we do not have such crashes.

Do we have option do not use FastMod?

[simonrozsival]

Do we have option do not use FastMod?

I don't think so. But honestly, I think the chance that this is the problem is quite slim, since it happens on so many different devices and not just on some specific model.

[simonrozsival]

@igor-sidorovich have you had any success reproducing this issue? or do you have some data about the frequency of the crashes and the affected OS versions or phone models you could share with us?

[igor-sidorovich]

i do not have repo, i tried but didn't success. I will send affected OS versions and phone models later, thx.

[igor-sidorovich]

Device	OS
realme 3	Android 10
Redmi 13C	Android 14
A58	Android 13
Redmi 8	Android 10
V2027	Android 12
Galaxy S23 Ultra	Android 14
HUAWEI P30 lite	Android 10
moto g(30)	Android 12
Alcatel 1SE	Android 10
ZERO X Pro	Android 11
Galaxy A22s 5G	Android 13

[simonrozsival]

@igor-sidorovich thanks for the additional info. I was also interested in the frequency - are these rare cases or is this affecting a large number of customers?

[igor-sidorovich]

We opened it only on 0.5% percent of customers. It's - India, Colombia, Thailand, Pakistan. And off course it's rare cases.

[igor-sidorovich]

For 50000 users - we have only 10 crashes.

[simonrozsival]

@jpobst So far, this issue doesn't seem to be Android-specific. I think we should transfer it into dotnet/runtime. Is there some easy way to do it or is it easiest to close this one and open a new one?

[jpobst]

Unfortunately there isn't an easy way to transfer issues to a different organization. We will probably need to open a new one in dotnet/runtime.

[igor-sidorovich]

Should i open new one?

[jpobst]

If you don't mind, please do.

Having the issue in dotnet/runtime will help ensure it stays on the Runtime team's radar and is included in their prioritization.

Please include a link to this issue in the description so they can reference the context here.

Thanks!

[igor-sidorovich]

@simonrozsival, @jpobst Done - dotnet/runtime#102545.

[jpobst]

Thanks!

Tracking this here: dotnet/runtime#102545.