-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cython version of Deferred #12124
Comments
I think this would be a good opportunity to split Deferred into a separate package and treating it as more of a self-contained/top-level tool. There are an increasing number of contexts where a stand-alone Deferred would be handy; sometimes I don't want to pull in all of Twisted, but in the halcyon days of We've already got https://github.com/twisted/deferred , and we could do the work there. The reason to put it in a separate package would be that having a small repo that has a bit of packaging shenanigans to make So, re: distribution; do something like Regarding maintainability: I am given to understand that Cython now has multiple syntaxes, one which uses the old style Cython AST extensions, and another which uses Mypy-style type annotations. Can we get away with a single module using annotations which either is or is not built with Cython, so we have a pure-Python fallback? |
Thanks for looking into this. This looks like a duplicate of #2245 But maybe we can use this ticket to discuss the general implementation plan for mantaining optimized non-Python source in I am +1 for having a separate Rather than having the code a separate repo, maybe it would help just to move the code to I remember there was atempts in the past to have twisted/twisted as a mono repo, with things like I guess that the overal experience for that work was that separate packages are not really wanted. I think that at least for the initial release, it would be easier to have the new deferred code in the same repo. As a long term goal, I think that it would help to add Cython optimization to other parts of Twisted. For example if you transfer a file over HTTPS vs SSH+SFTP the twisted.conch implementation is so much slower. |
There's already https://pypi.org/project/deferred/ , which we own, and does contain a version of Deferred. |
I am -1 on this because it breaks VCS pins in pip. I am open to being convinced, but I think that rather than develop monorepo tooling we should develop config-replication tooling to copy CI stuff between repos. |
It was a different time and different levels of tooling existed. It was exponetnially more painful to add a dependency then than it was now. But also, the axis we chose to split upon was horizontal (i.e.: by protocol) rather than what we are talking about here, which is vertical (split out Deferred, FilePath, etc). I think being somewhat monolithic by protocol still makes sense, as having integrated protocol implementations is a big part of the value proposition of Twisted. But these smaller utility packages that can be self-contained make sense to split out. |
I dislike the idea of psycopg-binary equivalent; libpq interaction is the reason there, and I don't see why it would apply here. I would just have binary wheels by default and a tarball if people don't want to use that. |
OK, to back up and explain my motivation for that suggestion: I don't want people to Is it possible to just ship a pure-python fallback wheel, with the same distribution name? If we do that, how does the end user specify "no, I absolutely want the fast version and I want the build to fail if I don't get it"? |
I am +1 on having the package, not the repo :) But it's ok to have it as a separate repo... we alrady have twisted-iocpsupport I feel that somehow the twisted-iocpsupport is a 2nd class repo. My goal with having the twisted-deferred package in Twisted is to make it more visible. I also expect that it will be easier to test "twisted-defered" agains the twisted trunk branch... basically, we will be able to test the deferred code as part of any twisted/twisted PR I am ok to have a separate repo. The main reason to have a mono repo is to make it easier to run the automated tests.
There is the option of a non-binary wheel fallback. One example is the pyrsistent project. |
I tried |
As far as separate package... this feels like a bunch of duplicated work (packaging, linting, etc) and a bunch of integration work, for a very minor benefit. https://pypistats.org/packages/deferred shows 16 downloads a month, and many of them are probably just mirroring. The fact that @glyph may want Deferred standalone occasionally doesn't mean it's a common use case. And the existence of Future in the stdlib means it's that much less likely to be something someone wants to do. So I'd rather just have it be part of Twisted, make Twisted have binary wheels, and do the thing where if you can't use a binary wheel it still works with pure Python wheel. |
OK, I'm sold. If we're going to spin up cython binary wheel infrastructure, I guess it doesn't much matter where we do this. The fact that we don't need separate packaging infrastructure invalidates my desire to mix the two things together. I still kinda want to split out Deferred for other reasons, but I no longer have any valid reason to conflate it with this issue, so carry on :) |
When we start building binary wheels again, please do make sure to build |
My last foray into this involved the issue that older versions of Python don't do universal2, but I guess... 3.8 has it, excellent. |
As far as sharing code between pure Python and Cython... pretty sure that's doable, will just need to shift part of the type annotations into a |
(Although lack of symlink support on Windows is gonna make it annoying.) |
This seems wrong to me, but then, I guess you already have a prototype so you'd know better. What Cython specifics do you have in mind here, and what would live in a |
Hang on, in what way are symlinks useful here? |
Happy to go with a single package and without a separate repo :) Let me know if you need help with the build system. I see pyrsistent uses cibuildwheel for binary wheels Non-binary/pure-python is done via Linux ARM64 done via docker+qemu macOS is done only via There is also a PR to build for Alpine/Musl Let me know if you need help with the GitHub Actions part. |
Never mind, I can set the extension name separately from the file name. |
One problem here is that Twisted will need to switch from hatch to setuptools, unless I can find some way to bridge the gap (there's hatch-cython but I have so far failed to get it to work; I asked author for help). |
@itamarst Black uses Hatch and manages to build with mypyc, so maybe mypyc would be a better option here than Cython? I don't believe we're doing a ton of optimizable integer math here, and mypyc doesn't carry the runtime dependency baggage |
Hey, thanks for the ping! My personal preference nowadays is to stay very far away from Cython and only use Mypyc or write extension modules in Rust. You can get very good speedups with the former while the latter is better (when I have to) for me personally because I dislike writing C and very much enjoy writing Rust. That being said, I would like to help here. What issue did you run into with that plugin? If it's something on my side I will fix it! |
I couldn't figure out how to use the plugin at all. Basic functionality like "here are the list of files to use for this extension" was completely opaque. It tried to compile every single I can try mypyc, I guess, as an alternative, it's definitely not very interesting cython code, most of the benefit is probably having fast |
Sure let's continue the discussion here: joshua-auchincloss/hatch-cython#45 The docs for the plugin are surprisingly good actually, there are just so many options! |
mypyc is probably blocked on mypyc/mypyc#1051 |
I don't have full context about Deferred but I am always in favor of stripping stuff out into isolated packages; I think that is a great idea. Also, I think there was some confusion in this thread about the example of psycopg that Glyph brought up. One of the responses was talking about psycopg-binary and that it was a train wreck essentially which I agree with. Basically, what happened was at that point they didn't know how to properly build the extension module for distribution so they had an entirely separate package just for compiled wheels because they were broken in subtle ways. What I think Glyph is talking about (which I think is indeed a great example) is the modern/rewritten psycopg package: https://github.com/psycopg/psycopg The main package is pure Python which then depends on specific compiled packages for performance and extra features, which all live in the same repository. |
Another place that could do well with Cython version, probably, is twisted.internet.base.DelayedCall. And there are likely others. I feel like the argument for lots of little compiled packages is not compelling, and personally would argue for either switching Twisted back to setuptools, or having one single compiled-extensions add-on package, not one per feature. |
This is what I was imagining, yes. |
I wonder what's more work, doing that or switching back to setuptools. |
I think the build backend choice is irrelevant here (if you tell me what you want Cython-wise I will provide you an example configuration of how to achieve it) but rather I think the original points are still valid reasons to desire a separate package #12124 (comment) Do what you wish but please understand that what you want is already possible with the existing tooling. I believe your feature request is this which for the time being you actually can very easily do yourself with a custom build hook that inherits from that plugin and simply overrides this property. edit: there is also built-in functionality for conditional execution of build hooks so that you can build a pure Python wheel as an option |
I don't see how can you have multiple extensions, though. Which isn't strictly necessary I guess but the more one adds the more attractive it is. |
An inspection of Twisted application's profiling tends to show lots of Deferred APIs.
As an experiment, I threw together a Deferred version in Cython. Using the HTTP client benchmark I've been using in some previous tickets, I got a 10% speedup. Given Deferreds are only part of the benchmark's work, this suggests a very significant improvement in the Deferred speed!
The two main questions on how to proceed are:
The text was updated successfully, but these errors were encountered: