binary distribution: package information about jars required for scalac classpath in a separate file

[unkarjedy]

Context:

In IntelliJ IDEA, you can use a plain IntelliJ project without build system SBT/Maven/Gradle/BSP.
In Scala Plugin we have a concept of "Scala SDK" which simply represents the scala compiler (classpath) and scala version.
In case the user already has Scala "installed" (whatever that means) IntelliJ has some logic to detect it.
We have multiple kinds of detectors for Brew, SDKMAN, Coursier, Maven, Ivy.

We also have a "System" detector which can detect Scala SDK from the unzipped binary Scala distribution (see Assets at github)
To construct Scala Compiler classpath, we read some jars from lib directory.

In Scala2 the code of constructing the classpath was quite simple. We just search for a fixed set of compiler jars: scala-compiler-2.X.Y.jar, scala-library-2.X.Y.jar, scala-reflect-2.X.Y.jar:

Issue

In Scala3 things get different. Scala 3 compiler classpath has more dependencies on more libraries.
In general, each minor/major Scala 3 version can have dependencies on different libraries with different versions.
We can't just include all jars from lib folder. We need an explicit list of compiler classpath dependencies because lib folder contains a lot of other jar files, not required for the compiler. AFAIU most of them are needed for scaladoc, not scalac.

With Maven/Coursier/Ivy detectors, we can live with it because we can resolve transitive dependencies for scala3-compiler_3 and get the classpath.
However, with System detector we can't do it.
Three years ago I provided a hardcoded solution for Scala 3.0.0 (see the code)
It worked up to Scala 3.3.1 but shot back in 3.3.2 & 3.4.x (see https://youtrack.jetbrains.com/issue/SCL-22510).

Proposal

I think the only reliable solution is that along with the jar files, scala compiler publishes the classpath required for scalac.
For consistency, it could also publish the classpath for Scaladoc.
For example, it could be just two files:

• lib/scalac-classpath
• lib/scaladoc-classpath

Those files could contain a list of jar names or even jar name patterns on every line.
Then in IntelliJ we could simply read that file content.
If you have better ideas on the file names/format, we can discuss it.

Notes

Note that the information about scalac/scaladoc classpath is already known to the code which does the binary distribution packaging. It's already present in bin/common, bin/common.bat, bin/scaladoc, bin/scaladoc.bat.
However, it would be a too fragile solution to rely on those files as their structure can change and our parsing logic can break.

[unkarjedy]

Note that the information about scalac/scaladoc classpath is already known to the code which does the binary distribution packaging. It's already present in bin/common, bin/common.bat, bin/scaladoc, bin/scaladoc.bat

I also notice that the classpath used in those scripts has one suspicious extra jar scala3-staging_3*.jar
It's not a transitive dependency of scala3-compiler_3, also it's not reported by SBT (show scalaInstance).

What is that jar? Is it indeed required for scalac classpath?

[unkarjedy]

Related links:

• https://youtrack.jetbrains.com/issue/SCL-22510/Cant-configure-locally-installed-Scala-SDK-for-3.3.3-and-3.4.x-versions
• https://youtrack.jetbrains.com/issue/SCL-17273/An-ability-to-create-Scala-3-SDKs-automatically

[bishabosha]

we were proposing to heavily change the lib directory currently in #20351 (e.g. removing it or perhaps converting the contents to alias links) - so would be good to know if its actually used by other tooling - it seems it is - it would be good to work together on this

[unkarjedy]

@bishabosha
Could you please point to the particular place in the discussion mentioning "potential removing of lib folder"?

or perhaps converting the contents to alias links

Alias links to where?

it would be good to work together on this

Is there anything else we can provide from our side?

[bishabosha]

Is there anything else we can provide from our side?

I updated the PR description of #20351. otherwise you can download the zip from 3.5.0-RC1 release

[bishabosha]

I also notice that the classpath used in those scripts has one suspicious extra jar scala3-staging_3*.jar
It's not a transitive dependency of scala3-compiler_3, also it's not reported by SBT (show scalaInstance).

What is that jar? Is it indeed required for scalac classpath?

there is also a scala3-tasty-inspector_3 jar to go alongside scala3-staging_3 - these are not normally on the classpath for scala/scalac, but are added to the classpath if the user passes the -with-compiler argument to the launcher. (Typically the user would need to add explicit libraryDependencies to use them in sbt/mill)