Run tests in non-linear order (see #9303)

[1bsyl]

Run tests in non-linear order (see #9303)

It mixes the testSuites array, and for each suite, it mixes the sub-tests.

Hence, it changes the public API behavior of SDLTest_RunSuites(). not sure if this is acceptable ?

[1bsyl]

now, some test seems to fails: at least 2:

--seed BM1BUVUADHUWWKPN --filter audio_initOpenCloseQuitAudio
--seed BM1BUVUADHUWWKPN --filter pen_buttonReporting

[madebr]

Hence, it changes the public API behavior of SDLTest_RunSuites(). not sure if this is acceptable ?

I asked about this in #9301.
For SDL_test to be useful for the satellite libraries in SDL3's lifetime, it's important for it to have a stable API (ABI does not matter).
So perhaps it's better for SDLTest_RunSuites to accept a single struct, which must be zero initialized before use.
Then we can add parameters and don't need to worry about backwards compatibility:

SDL_zero(testspec);
#if SDL_VERSION_ATLEAST(3,5,0)
testspec.parallel = SDL_TRUE
#endif 
SDLTest_RunSuites(&testspec);

[1bsyl]

yes, we need to modify the parameters of SDLTest_RunSuites

@slouken
could adding SDL_bool allow_random_order be enough or something more complex ?

int SDLTest_RunSuites(
                  SDLTest_TestSuiteReference *testSuites[], 
                  const char     *userRunSeed,
                  Uint64 userExecKey, 
                  const char *filter,
                  int testIterations)   ;

[slouken]

I wonder if we could always permute them, based on the random seed? Alternatively this could be done in the application just by permitting the test suites before passing them in.

[1bsyl]

A seed should always give the same permutation.

we can permute the suite&case before passing them. (probably we need to create the seed before). but we lose the correct numbering of case/suite in the log.

[slouken]

Good point.

[1bsyl]

I have updated the branch so that we don't lose the suite numbers neither.
and added SDL_bool randomOrder parameter to SDLTest_RunSuites().

Also added a "--no-random-order" parameter to testautomation so that we can still run test in linear order