[release/8.0] Removed unused sessions from SSL_CTX internal cache

[github-actions]

Backport of #101684 to release/8.0-staging

/cc @wfurt @rzikm

Customer Impact

Reported by customer via official support. Small repro available.
Customers on Linux sees increased memory usage when establishing parallel connections to the same host (note that parallel requests on HTTP/1.1 will always use parallel connections). Measured overhead can be easily 100M+, which is problem for containers in k8s clusters limited to 300M memory.

It also helped one customer in general "memory problems with .NET 6 -> 8 upgrade" issue - see comment.

Workaround is lowering TLS cache size via:

• System.Net.Security.TlsCacheSize AppCtx switch, or
• DOTNET_SYSTEM_NET_SECURITY_TLSCACHESIZE environment variable

Technical details:

The mechanism of the (bounded) memory leak is as follows:

• On new TLS connection, we receive TLS session ticket so that future connections can use TLS Resume (for faster handshake) (this is new feature in .NET 7).
• There are 2 TLS session caches: internal one in OpenSSL, and managed one in .NET. These caches are supposed to be in sync via callbacks (e.g. OpenSSL may remove an entry internally and notifies managed code via callback that a ticket has been invalidated).
• When many connections are opened in a burst, we receive many TLS session tickets in quick succession, but the implementation keeps only the last one.
• Sessions discarded from managed (.NET) TLS session cache were still being kept in the internal OpenSSL cache, until the ticket expired or the internal cache reached a predefined maximum size.

The fix is to keep the two caches in sync and remove the dropped TLS session tickets from the internal cache as well.

Regression

Yes, the bug is part of TLS Session resumption feature on Linux, introduced in .NET 7. For customers migrating from .NET 6 it manifests as E2E scenario regression.

Testing

Tested on customer provided minimal repro.
Customer was not willing to verify privates in production.
Note: Customer confirmed that the workaround helps them in production, which means we have high confidence, this fix is the real root cause of their production problems and will help them.

Risk

Low, the issue is well understood and the change is localized to the feature. Functional tests verified TLS resumption works.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

[wfurt]

LGTM

[karelz]

Memory usage impact on production systems. E2E scenario regression from .NET 6. We should service it.

[rzikm]

Approved via email on 06/03.