Start work on Julia 2.0, with it off by default, enablable with an ENV var

[PallHaraldsson]

There are a number of breaking Julia 2.0 issues (see the milestone) that I would want implemented, but I would also want all the code that works in 2.0 to also work in 1.x.

I think we're actually going in the wrong direction from 2.0 with e.g. #49336 that may need to be reverted before too late. (that PR was understandable, the title is however misleading).

I'm just thinking as an example, we could throw on mismatched lengths of zip and map when JULIA2 env var is set (and all code where you make sure you do not try that will work in 1.x). When the env isn't set we would keep status quo. Over time we would break more with the JULIA2 env var, and we could PkgEval with it on to see what might break.

What are the most-wanted 2.0 breaking changes?

I would at least want to do something about this: #21600 (by preference, return Float64), #34999, #50128, and maybe #29841. We could e.g. drop again, i.e. revert the revert of this delete... #49915

E.g. I'm not sure about this one #39235, too much code relies on this. I only have changes in mind, for now, that are braking changes though wouldn't really be breaking for most code.

[jakobnissen]

I think there is broad agreement that now is not the time to start working on Julia 2.0. The version change will be hugely disruptive, and the current benefits of allowing breaking changes are not considered worth the hassle. There is still much to do for Julia 1.0.

I'm closing this, because I'm confident that the core devs' response to a suggestion to start work on Julia 2.0 is "no". If there is interest in this from core devs, please do reopen it.

[jishnub]

I think it might be ok to start working on something, with the understanding that it won't be tagged anytime soon. There are quite a few open PRs with the breaking label, so in a sense, the work has already started.

[LilithHafner]

I'd be excited to see concrete work on Julia 2.0 with the expectations that

• The package ecosystem must not be fragmented i.e. Julia 2.x code must be able to call Julia 1.x code and vice versa.
• This would be the beginning of a patient, years-long process.
• It's plausible that much of the work done on 2.0 will never merge

[PallHaraldsson]

The version change will be hugely disruptive

No there will be NO disruption, since all braking changes ("2.0") will off by default. I understand people fear disruption if when 2.0 enabled, e.g. thinking code you see out there in tutorials or books would break, but if you get better answers, correct instead of incorrect (as happens now), or at least throw an overflow exception in those rare cases, then that's also better.

When people talk about 2.0, and understandably fear it, they assume "2.0" will break code and that 2.0 will be released as stable, and 1.x will be dropped. I'm thinking users would need to opt into this 2.0 for the foreseeable future, maybe forever. I can clarify a better way to opt into how you would opt into 2.0, and my plan for 3.0, 4.0 etc. if people are not opposed to start looking into some (I think) important 2.0 issues.

@LilithHafner I consider you a core dev, so how about:

If there is interest in this from core devs, please do reopen it.

After this was closed here I opened a thread on discourse (and I intentionally worded it differently, as a debug flag):

https://discourse.julialang.org/t/would-people-like-an-introduction-of-a-julia-debug-mode-possibly-on-by-default-in-the-repl-pluto-jupyther/102535

I think we need a debug flag, and this issue could also be opened and the title changed to that effect. There's a large overlap (not 100%) for what I would want with 2.0 and a debug flag.

Change comes in three forms:

• additions (we're doing them all the time)
• change, which is semantically deletion and corresponding addition with the change (i.e. 2.0; or non-breaking for semver reasons 1.x change)
• deletion ("2.0")

What we're doing now is mainly additions (often to the sysimage).

I want to emphasize from my first sentence "all the code that works in 2.0 to also work in 1.x." You also want (package) Julia 1.x code to be able to call 2.0 code (packages).

For 2.0 (or phase 1), I'm only thinking of changes that will always give you the same answer, or if not throw an excption, as with the ^overflow PR, and zip is another example. "stop exporting HTML and Text from Base" is maybe not the best example, not a high-priority for me, it will result in a runtime error if people would try to use (sort of comparable to throwing an exception..).

I'm thinking we could make the sysimage radically smaller and thus startup better (without that there are some microbenchmarks we can't win). I have some interest in certain very important changes, and might work on that, but I'm most interest what we can simply drop.

I'm thinking we could have a stdlib and do using julia1 to add everything back that's missing from the sysimage, even if people were to opt into 2.0 (maybe by accident) with the ENV var controlling it.

[LilithHafner]

I personally value shrinking the sysimg size as well as the size of the codebase in this repo as a high priority.

I'm reopening this in the spirit of "go for it, I think we'll learn a lot from your (and other folks') efforts to implement this" not in the spirit of "we're planning to release 2.0 any time soon (or ever)". As for whether or not work like this could be merged under a debug or feature flag or ENV var, that seems like a decision for triage to make in the context of a concrete proposal or pull request.

You don't need permission from anyone to make a PR and prototype this, though seeking community approval as you are doing makes your work more likely to be accepted by the community.

Also tagging triage to get a sense of how folks feel about this.

[KristofferC]

I strongly agree with the initial reaction to close this. There seems to be nothing actionable here. I would suggest closing it again to avoid giving the impression that something along what is described here is something encouraged. Otherwise it seems to open up for wasted work.

[KristofferC]

Reading through this again this type of very speculative stuff is better at discourse or maybe as a discussion so converting to that.

[LilithHafner]

I think that Rust Editions are a good model to follow if we want to implement superficial breaking changes without actually breaking anyone's code. This is a reasonable and quite readable explanation of how they work.

[JeffBezanson]

I think we can do a lot for smaller sysimages via PackageCompiler. Similar to filter_stdlibs, you should be able to opt in to start with "tiny Base" instead of the normal Base. That will eventually allow smaller, fast-starting executables.

OK, but what about julia script.jl? Yes a smaller default sysimage would help "hello, world". However as soon as that script does anything nontrivial it will have to load a bunch of stuff, which is likely to be slower than starting with a big sysimage. If I were doing it all over again, yes Base would be smaller, but in the meantime I don't think this case is a big problem.

[andrewjradcliffe]

OK, but what about julia script.jl? Yes a smaller default sysimage would help "hello, world". However as soon as that script does anything nontrivial it will have to load a bunch of stuff, which is likely to be slower than starting with a big sysimage.

The bold+italicized selection should be emphasized more often. For code which does any serious work, startup time doesn't matter. One shouldn't pay attention to things like this, nor the conclusions casuals might draw from it.

[PallHaraldsson]

However as soon as that script does anything nontrivial it will have to load a bunch of stuff

You have a fixed cost (juliaup 4.428.672 bytes then) julia (21460 bytes) then 104.450.012 byte sysimage along with more, and I've analyzed it, startup is linear in the size of the sysimage (but much slower per byte), and I've been able to half it. It's NOT clear you need the other half, and even if you do, reading it separately shouldn't cost more in two I/O, rather than in one I/O. It's not like there's one anyway with all that happens before and after:

On top of that you have many I/Os for many packages, for serious work.

One shouldn't pay attention to things like this,

But people do [Julia was there AOT compiled, where did it go?], and think Julia 2x to 4x times slower (when looking at default JIT, not AOT), when it's as fast. Yes startup cost doesn't matter for HPC/long-running, but does to shorter scripts and benchmarks.