-
-
Notifications
You must be signed in to change notification settings - Fork 64
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Provide a mechanism to allow a process plugin to declare it as single-threaded so dprint can scale it #761
Comments
@bradzacher yes, it should be faster for sure. That said, what do you need a JavaScript-based formatter for? I've found it's way faster to use deno's Rust crates for anything JavaScript based similar to what dprint-plugin-prettier does (https://github.com/dprint/dprint-plugin-prettier). I'm planning to overhaul that plugin to be better soon (going to release a dprint roadmap sometime in the next few days that will include the major changes to that and also will try to prioritize some of this work). Btw, sorry for not responding to your email. I was in Japan the past two weeks and wasn't really checking it (back now). I think most questions in it have been answered now. Also, thanks for the details on worker_threads! That was intresting. |
We want to avoid introducing deno to the repo if we can. Mostly because if we introduce it then we have to maintain it and our team is already stretched kinda thin in regards to ownership and maintenance! A process plugin seemed like a good middle-ground - a smallish piece of JS code that allows us to use the standard NodeJS version for our environment to improve the perf compared to I actually did want to use
No problem at all 😃 You gave me more than enough to work with! With your examples I was able to build a fully-working JS process plugin! I can probably share it if you're interested. |
Ah, ok makes sense! That said, there is a dprint-plugin-deno-base crate that makes it slighly eaiser to use for a scenario like that: https://github.com/dprint/dprint-plugin-prettier/blob/main/base/Cargo.toml (I extracted this out of dprint-plugin-prettier a few months ago... it is very undocumented though) It uses the Deno crates, but it's not really Deno. It uses Deno's helper crate wrappers around v8 and then some of the other crates to get some APIs that are used in Prettier. I had to polyfill a bunch of stuff still though, which was a bit of a pain. Yeah it requires a bundled version atm (I'm planning to support prettier plugins that are configurable soon).
dprint-plugin-prettier used to work this way and use JS for the communication (and when I exprimented with https://github.com/dprint/node-plugin-base/). It was indeed surprisingly extremely annoying to read from stdin. |
Working with nodejs in a process plugin I've found that there's a lot of overhead introduced by the IPC. In a nutshell a format request requires 4 cloned IPC messages to complete - 2 of which contain a file's contents:
null
null
This cloned IPC has a lot of overhead, sadly.
I've been running some tests on Canva's repo and I'm currently averaging ~10ms per file for an unchanged format - which isn't bad, but adds up across ~60k files.
In contrast, I've written a pure-nodejs script for the formatter which spawns 4
child_process
es, sends each 1/4 of the entire list of files, and lets each process do its own read-format-write cycle and that runs at ~1ms per file.I think that if I could remove the nodejs-based
worker_thread
parallelisation layer and instead have dprint just spawnn
instances of the process plugin then things would be much faster and closer to that "reference" implementation perf-wise.What do you think? Do you think that this feature might make sense - giving a plugin the ability to say "hey I'm single threaded - spin up as many of me as you please" and letting dprint do its own thing as it sees fit based on resource utilisation and availability.
The text was updated successfully, but these errors were encountered: