java.io.InvalidClassException when using testGrouping

[armanbilge]

Reproducer: armanbilge/sandbox@5e98848

sbt:sandbox> test
[error] Uncaught exception when running tests: java.io.InvalidClassException: sbt.testing.TaskDef; local class incompatible: stream classdesc serialVersionUID = -7417691495999416204, local class serialVersionUID = 2663134200025980977
[error] stack trace is suppressed; run last Test / test for the full output
[error] Error: Total 0, Failed 0, Errors 0, Passed 0
[error] Error during tests:
[error]         Forked test harness failed: java.io.EOFException
[error]         at java.base/java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:3214)
[error]         at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1684)
[error]         at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:509)
[error]         at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:467)
[error]         at sbt.React.react(ForkTests.scala:201)
[error]         at sbt.ForkTests$Acceptor$1$.run(ForkTests.scala:132)
[error]         at java.base/java.lang.Thread.run(Thread.java:833)
[error] (Test / test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 1 s, completed Apr 8, 2023, 11:25:36 PM

Linking to scala-native/scala-native#3246, h/t @lafeychine

[armanbilge]

Argh, the correct thing to do is use Tests.InProcess instead of Test.SubProcess. I wonder if a better error is possible here like there is for fork := true if I remember correctly?

Seq(Tests.Group("group", tests, Tests.InProcess))

[sjrd]

It looks like we could indeed add a warning if Test / testGrouping contains a group whose runPolicy is an sbt.Tests.SubProcess.

[gzm0]

I've been trying to do this, and I'm not sure we have a good interception point.

We test fork in loadedTestFrameworks, since we redefine it entirely anyways. However, we cannot test testGrouping in loadedTestFrameworks since there is the following dependency (with defaults):

testGrouping --> definedTests --> loadedTestFrameworks

So it means we'd have to check on the level of test execution tasks. Not sure I want to wrap all of these :-/

That being said, why can we not run tests in a forked JVM? (it doesn't make sense, of course, because all it'll do is start other subprocesses, but why does it not work?). Are we linking / building against the wrong test-interface?

[gzm0]

FWIW: The check for forking originated here (2014):
5e1f3b3

[sjrd]

In the forked JVM, sbt doesn't actually recreate instances of our own Framework. It creates its own framework, which does not have JS support.

[gzm0]

It creates its own framework, which does not have JS support.

Hum, but isn't the point of a framework to abstract over how tests are run? Does this mean other frameworks (that "support" forking) have an additional implicit contract of how tests are to be run?

[gzm0]

Ah, OK. Now I understand. The problem is that sbt's forking assumes a standard implementation of loadedTestFrameworks (which Scala.js completely hijacks).

[sjrd]

Yes, exactly.