XUnit should not throw NotSupportedException when MemberData references a non-public property

[IanKemp]

Visual Studio 2022 version: 17.1.3
xunit NuGet package version: 2.4.1
xunit.runner.visualstudio NuGet package version: 2.4.3

This is a semi-duplicate of #1538, but that issue was closed by the author because they thought the behaviour was a bug, and then realised that according to the docs it isn't. I'm creating this issue because I believe that it either is a bug, or the error message could stand to be more helpful.

Test case test class:

    public class UnitTest1
    {
        private static IEnumerable<object[]> GenerateTestData()
        {
            yield return new object[] { new List<string>() };
            yield return new object[] { default(List<string>)! };
        }

        // NOTE: not public
        private static IEnumerable<object[]> TestData
            => new[]
            {
                new object[] { new List<string>() },
                new object[] { default(List<string>)! }
            };

        [Theory]
        [MemberData(nameof(GenerateTestData))]
        public void MemberData_Method(List<string>? data)
        {
            Assert.NotNull(data);
        }

        [Theory]
        [MemberData(nameof(TestData))]
        public void MemberData_Property(List<string>? data)
        {
            Assert.NotNull(data);
        }
    }

Since TestData is non-public, attempting to run the MemberData_Property test throws the aforementioned System.NotSupportedException : Specified method is not supported. That's it, nothing more, no stack trace, no indication as to the fact that you've made a mistake and need to fix your code.

First of all, why is this exception thrown? Why does the property need to be public when the method doing the exact same thing can have any visibility? The BCL's reflection API has no problems accessing a member of any type regardless of its visibility, so I don't understand the reason for this seemingly arbitrary restriction. If there is no good reason, this is a bug that should be fixed.

However, if there is a valid technical reason why non-public properties cannot be used as MemberData, then xUnit should not be throwing a raw NotSupportedException, because that's a terrible user experience. It should instead throw a specific exception with helpful text along the lines of "MemberData pointed to by {MemberName} is not accessible - if you're using a property, ensure it is public".

[MisinformedDNA]

Or create a Roslyn analyzer to let us know there is an issue.

[bradwilson]

@MisinformedDNA The Roslyn analyzer exists in xunit.analyzers 1.0.0.

[richardforrestbarker]

On VSCode @ 1.70.0, with xunit @ 2.4.1 and xunit.runner.visualstudio @ 2.4.3

It seems this error is thrown in a few other conditions - such as when the method or property is not static; or the method or property does not return an IEnumerable<object[]>. The message is certainly lacking -- those requirements should probably be expressed.