Investigate linker memory consumption #4906
Assignees
Labels
optimization
Optimization only. Does not affect semantics or correctness.
Comments
gzm0
added
the
optimization
|
Just TBC: we value speed over memory consumption in the linker in general, so in-scope here are only reductions that do not affect speed. |
|
I'm looking at heap Dominators for a Scala.js test suite run. Summary is the following
For |
gzm0
added a commit
to gzm0/scala-js
that referenced
this issue
Oct 8, 2023
Memory profile analysis shows ~120k duplicates of the string "$n" amounting to ~6MB waste (on the test suite). Further digging revealed that this happens through the "$n" call-helper (null check) via: - SJSGen#genCallHelper - VarGen#globalVar - VarGen#globalVarIdent - VarGen#genericIdent A more invasive (but efficient) alternative would be to move the prefix to the call sites. As a result, no runtime interning would have to be performed as the strings would end up being part ot the static constant pool. Discovered while working on scala-js#4906.
|
I have a WIP branch where I attempt to reduce the memory usage of the Emitter by fusing the two last transformation steps (emitting and printing) and caching the result only (remove the intermediate caches). However, I'm running into issues with VisualVM and my YourKit evaluation has expired. YourKit offers free licenses for OSS projects, given they add a link to YourKit (https://www.yourkit.com/java/profiler/purchase/#os_license). @sjrd, would that be acceptable? If yes, I'd reach out to them. |
|
Sure, that is acceptable. |
|
WIP PR for YourKit on scala-js.org: scala-js/scala-js-website#620 I've reached out to sales for licenses for the two of us. |
|
So far, as part of this effort, we have the following changes targeting the linker backend:
After these, the frontend retains 215 MB on the test suite, whereas the backend retains 77 MB. Before going to more extreme measures (like #4963), I'll start focusing on the linker frontend. All in all, the picture for the frontend is unchanged:
ObservationsOptimizerThe optimizer mostly retains IR trees (both original and optimized def). The optimized defs are sometimes significantly larger (I've seen up to 4x). Over 46% of the retained size of the optimizer goes to data structures (shallow sizes). Top 5:
BaseLinker / RefinerThe retained sizes goes almost exclusively to the Inside these, top 5 shallow sizes:
|
|
Optimizer retained size reduction from 166MB to 144MB. BaseLinker / Refiner size reduction by 5MB each: |
|
BaseLinker / Refiner size reduction by ~5MB each: |
|
I'm getting OOME's with 10G of memory for Every entry that I looked at in here has a long hierarchy of dynamic dependencies with a lot of duplication although some differences. It seems like the way that this graph is constructed is not very memory efficient. For example: I was trying to fix our module splits by adding more dynamic imports. We have routes which import entry points and then those entry points import the views. It didn't OOM until I added the imports in the entry points (but the splits weren't right). It still doesn't look like I have the imports right (looks like the route is pulling in the layouts) but it shouldn't OOM. In case it helps: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
IMO there are likely a couple of low-hanging fruits in terms of linker memory consumption. Last time I checked, a large portion of memory was taken by
::(List's cons). It would not surprise me if we could reduce memory consumption significantly by using array backed collections.I'll file this as a follow-up and take a stab.
Originally posted by @gzm0 in #3767 (comment)
The text was updated successfully, but these errors were encountered: