feat: add Pyodide support #1456

hoodmane · 2023-04-04T22:30:46Z

Resolves #1454. Also close #1002.

TODO:

Automate Python installation in certain CI environments?

Working:

multidict -- (requires this fix to their config)
numpy -- builds wheel but hangs 99% of the way through test suite
scikitlearn
scikit-image -- works on v0.19.3, doesn't work on v0.20 because of Meson
mypy -- NonPlatformWheelError
ultrajson -- one test failure due to fork
ril -- rust compilation errors
pydantic -- one test failure due to asyncio.run

hoodmane · 2023-04-05T20:58:24Z

@henryiii could you approve the github actions workflows?

hoodmane · 2023-04-07T17:08:21Z

@henryiii is the windows_x86 failure a flake?

failed to download pip version 20.0.2, pip download exit code 1
The conflict is caused by:
    The user requested pip==20.0.2
    The user requested (constraint) pip==20.0.2
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
WARNING: Skipping page https://pypi.org/simple/pip/ because the GET request got Content-Type: .The only supported Content-Type is text/html
ERROR: Cannot install pip==20.0.2 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

cibuildwheel/resources/constraints-pyodide-0_22.txt

rth

Thanks a lot for doing this work! It's certainly more work that I anticipated (without knowing much about cibuildwheel) :)

cibuildwheel/resources/pyodide_python.sh

docs/options.md

henryiii · 2023-04-27T23:51:25Z

@henryiii is the windows_x86 failure a flake?

Yes. Some networking issue that is hard to track down.

henryiii · 2023-05-01T04:33:53Z

BTW, would you like me to rebase for you? I don't mind.

hoodmane · 2023-05-01T04:40:11Z

I already merged main locally.

hoodmane · 2023-05-06T15:47:07Z

Okay I think this is working now. Would appreciate a review @henryiii, @joerick

hoodmane · 2023-05-06T15:47:33Z

Presumably we need documentation somewhere for the fact that Python 311 needs to be installed.

docs/options.md

.pre-commit-config.yaml

.github/workflows/test.yml

cibuildwheel/__main__.py

cibuildwheel/pyodide.py

test/test_emscripten.py

joerick · 2023-05-12T10:25:29Z

Splendid work on this so far @hoodmane ! Quality looks really good. Comments above.

henryiii · 2023-05-12T14:02:25Z

We talked about the Python requirement at PyCon a bit - I think it's best to require 3.11 instead of adding a lot of time & bandwidth for a docker image. Since pyodide is a bit behind official CPython, all CI's should have a suitable host Python available. If PyBI's land, then we might be able to make this simpler. :)

I think the name is a bit in the air. Technically, it's emscripten + version + flags. But pyodide selects a specific set, and the hope is it won't update between Python versions. So 3.12 would be a new version of emscripten + flag updates, etc. But I think that's going to be incorrect at least once (I think there's a planned 3.11 update). The hope is by the time wheels are cleared for PyPI, it will be stable. There's also the possibility that CPython will move emscripten to tier 2 (not soon, sadly, looks like only WASI is going to tier 2), in which case CPython could take over the definition of the emscripten version + flags for whatever version it started on.

This will need docs - it should be clear this is experimental and you can't upload the wheels to PyPI yet. Setting it up is a bit more involved than just adding it to a matrix, etc. I can help with docs later (traveling at conference this week).

Personally and quite selfishly, I really want the cmake bug fixed before it ships, I expect scikit-build/scikit-build-core users will want to make pyodide wheels once this ships in cibuildwheel, and can't do that due to a variable being stripped from CMake's environment by emscripten and pyodide isn't auto-fixing out-of-tree wheels. I can help fix this early next week.

hoodmane · 2023-05-12T21:47:25Z

Thanks for the detailed review @joerick!

hoodmane · 2023-05-12T21:51:17Z

pyodide isn't auto-fixing out-of-tree wheels

You're talking about replace_so_abi_tags()? I think we can release a Pyodide 0.23.3 that calls that for out of tree wheels.

@ryanking13 do you see any reason not to call replace_so_abi_tags() when building out of tree? This probably fits into your project of making the out of tree builds more the same as the in-tree ones.

Of course it would be great if we could also fix the issue with Emscripten, but it seems potentially easier to change Pyodide.
emscripten-core/emscripten#19301

ryanking13 · 2023-05-13T00:32:30Z

@ryanking13 do you see any reason not to call replace_so_abi_tags() when building out of tree? This probably fits into your project of making the out of tree builds more the same as the in-tree ones.

Sounds good. We should try to make out of tree and in tree builds identical.

henryiii · 2023-05-15T20:48:24Z

You're talking about replace_so_abi_tags()?

There are three fixes, any one of them would be enough:

Run replace_so_abi_tags() for out-of-tree builds. It's a bit of a hack, but it's a minimum to get things working.
Replace emcmake with pyodide's own implementation. This has to be pretty easy, it's a really simple script. This would allow scikit-build-core to get the extension right to begin with, and would support FindPython properly I think.
Fix emscripten to not strip this variable when using emcmake. That's a harder fix, and would require an upstream change.

I'd really like both 1 and 2 if possible. :)

hoodmane · 2023-05-15T21:58:57Z

Okay I think all the integration tests are passing and I think I addressed the most recent round of review comments. So this is waiting on:

review round 2?
fix the so_abi problems

@henryiii I'll make a PR to run replace_so_abi_tags for out of tree builds, you can work on an emcmake replacement. Then we can release pyodide 0.23.3 and update cibuildwheel to use it. Then we can merge this (assuming that I address any further review comments).

.github/workflows/test.yml

test/conftest.py

test/test_abi_variants.py

test/test_before_test.py

test/test_custom_repair_wheel.py

cibuildwheel/resources/build-platforms.toml

test/test_testing.py

for more information, see https://pre-commit.ci

hoodmane · 2024-05-27T12:18:30Z

Hopefully can fix boost-histogram in Pyodide 0.26.1 =)

mayeut · 2024-05-27T12:29:36Z

pytest needs to work, most people use that vs. python -m pytest. I think this might be on us, as it seems to work if you do it by hand.

@henryiii, did you do it by hand on macOS too or are you referring to the "manual" workflow in boost-histogram ? (I'm seeing a different behavior under cibuildwheel depending on whether we're running on Linux or macOS).

Edit: I guess the manual one was on Linux, I setup a workflow like the one in boost-histogram, it's failing outside of cibuildwheel when running on macOS: https://github.com/mayeut/pybase64/actions/runs/9254992278/job/25457877047

This allows to still test direct invocation of `pytest` on most platforms (including pyodide on Linux) but falls back to `python -m pytest` when running pyodide on macOS.

mayeut

With the pytest invocation stuff being worked-out (mostly).
I don't have any more reservations to this being merged.

The question I have regarding node might be resolved later.

mayeut · 2024-05-27T16:59:56Z

@henryiii, can you review the last commits please ?

cibuildwheel/resources/build-platforms.toml

hoodmane · 2024-05-27T20:00:02Z

I guess it's failing because of the color / formatting in pyodide build -h.

hoodmane · 2024-05-27T20:01:56Z

Hmm test passes locally...

hoodmane · 2024-05-27T21:11:10Z

test/test_build_frontend_args.py

@@ -19,12 +19,11 @@ def test_build_frontend_args(tmp_path, capfd, frontend_name):
    project.generate(project_dir)

    # the build will fail because the frontend is called with '-h' - it prints the help message
+    add_env = {"CIBW_BUILD_FRONTEND": f"{frontend_name}; args: -h"}
+    if utils.platform == "pyodide":
+        add_env["TERM"] = "dumb"  # disable color / style


Hopefully this works? I have no idea why it passes locally though??

Seems that it fixed it indeed =)

it seems too have been style/color as you pointed out.
I don't know why it passes locally either...

Maybe something is checking for the GitHub actions var and assuming it is in a CI run that supports color?

This reverts commit 917646c.

henryiii

IMO, this is ready to merge, we can do followup patches if we find any improvements. It looks like the two outstanding issues might both be on the pyodide side, so we might not want to make a release until 0.26.1.

Unless anyone has any reservations, will merge tomorrow.

henryiii · 2024-05-28T04:47:21Z

Edit: I guess the manual one was on Linux,

Yes, scikit-hep/boost-histogram#938 (linux) makes it through the "pytest" command and dies on the other bug. But we were seeing this issue in cibuildwheel's tests, even on Linux.

henryiii · 2024-05-28T12:31:45Z

Thanks everyone!

hoodmane force-pushed the emscripten branch 2 times, most recently from 38a706b to e6a77ed Compare April 6, 2023 22:30

hoodmane commented Apr 7, 2023

View reviewed changes

cibuildwheel/resources/constraints-pyodide-0_22.txt Outdated Show resolved Hide resolved

hoodmane marked this pull request as ready for review April 7, 2023 21:52

hoodmane mentioned this pull request Apr 7, 2023

Set the default build_frontend to be "default" (which is still "pip") #1462

Merged

hoodmane force-pushed the emscripten branch from ee94c2d to 22c37d7 Compare April 7, 2023 23:30

rth reviewed Apr 27, 2023

View reviewed changes

cibuildwheel/resources/pyodide_python.sh Outdated Show resolved Hide resolved

docs/options.md Outdated Show resolved Hide resolved

rth mentioned this pull request Apr 27, 2023

Extract bindings between Python and JavaScript to a separate repository? pyodide/pyodide#3793

Open

hoodmane force-pushed the emscripten branch from f93b922 to 5cd2b5a Compare May 6, 2023 14:51

henryiii reviewed May 11, 2023

View reviewed changes

docs/options.md Outdated Show resolved Hide resolved

joerick reviewed May 12, 2023

View reviewed changes

lesteve mentioned this pull request May 16, 2023

CI Add Pyodide wheel build scikit-learn/scikit-learn#26374

Merged

joerick reviewed May 19, 2023

View reviewed changes

lesteve mentioned this pull request May 26, 2023

JupyterLite docs link is not tied to docs version of skimage scikit-image/scikit-image#6958

Open

reynoldsnlp mentioned this pull request Jun 1, 2023

Pyodide Wheels #1002

Closed

hoodmane force-pushed the emscripten branch from 323e163 to 4fecdd5 Compare June 3, 2023 00:35

[pre-commit.ci] auto fixes from pre-commit.com hooks

2fac554

for more information, see https://pre-commit.ci

mayeut mentioned this pull request May 27, 2024

chore: better determinism for the test virtual environment #1838

Merged

chore: rework test virtual environment seed packages

43d6042

chore: workaround direct invocation of pytest

4a8bf61

This allows to still test direct invocation of `pytest` on most platforms (including pyodide on Linux) but falls back to `python -m pytest` when running pyodide on macOS.

mayeut approved these changes May 27, 2024

View reviewed changes

mayeut requested a review from henryiii May 27, 2024 16:59

henryiii reviewed May 27, 2024

View reviewed changes

cibuildwheel/resources/build-platforms.toml Outdated Show resolved Hide resolved

hoodmane and others added 3 commits May 27, 2024 12:44

Use release version of pyodide

7aade41

Merge branch 'main' into pr/1456

fa7a7d8

fix: tests for 0.26.0 & parallel initialization of xbuildenv

9b01189

hoodmane and others added 2 commits May 27, 2024 13:04

Debug CI

917646c

fix: test/test_build_frontend_args.py

f13a0b7

hoodmane commented May 27, 2024

View reviewed changes

mayeut added 2 commits May 27, 2024 23:11

Merge branch 'main' into pr/1456

12d90c9

Revert "Debug CI"

6ea11f9

This reverts commit 917646c.

henryiii approved these changes May 28, 2024

View reviewed changes

Merge branch 'main' into emscripten

aac446a

henryiii merged commit 9f2a3cb into pypa:main May 28, 2024
24 checks passed

hoodmane deleted the emscripten branch May 28, 2024 12:43

hoodmane mentioned this pull request May 31, 2024

Better wheel hosting solution pyodide/pyodide#3049

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add Pyodide support #1456

feat: add Pyodide support #1456

hoodmane commented Apr 4, 2023 •

edited

hoodmane commented Apr 5, 2023

hoodmane commented Apr 7, 2023 •

edited

rth left a comment

henryiii commented Apr 27, 2023

henryiii commented May 1, 2023 •

edited

hoodmane commented May 1, 2023

hoodmane commented May 6, 2023

hoodmane commented May 6, 2023

joerick commented May 12, 2023

henryiii commented May 12, 2023

hoodmane commented May 12, 2023

hoodmane commented May 12, 2023

ryanking13 commented May 13, 2023

henryiii commented May 15, 2023

hoodmane commented May 15, 2023 •

edited

hoodmane commented May 27, 2024

mayeut commented May 27, 2024 •

edited

mayeut left a comment

mayeut commented May 27, 2024

hoodmane commented May 27, 2024

hoodmane commented May 27, 2024

hoodmane May 27, 2024

hoodmane May 27, 2024

mayeut May 27, 2024 •

edited

henryiii May 28, 2024

henryiii left a comment

henryiii commented May 28, 2024

henryiii commented May 28, 2024

feat: add Pyodide support #1456

feat: add Pyodide support #1456

Conversation

hoodmane commented Apr 4, 2023 • edited

hoodmane commented Apr 5, 2023

hoodmane commented Apr 7, 2023 • edited

rth left a comment

Choose a reason for hiding this comment

henryiii commented Apr 27, 2023

henryiii commented May 1, 2023 • edited

hoodmane commented May 1, 2023

hoodmane commented May 6, 2023

hoodmane commented May 6, 2023

joerick commented May 12, 2023

henryiii commented May 12, 2023

hoodmane commented May 12, 2023

hoodmane commented May 12, 2023

ryanking13 commented May 13, 2023

henryiii commented May 15, 2023

hoodmane commented May 15, 2023 • edited

hoodmane commented May 27, 2024

mayeut commented May 27, 2024 • edited

mayeut left a comment

Choose a reason for hiding this comment

mayeut commented May 27, 2024

hoodmane commented May 27, 2024

hoodmane commented May 27, 2024

hoodmane May 27, 2024

Choose a reason for hiding this comment

hoodmane May 27, 2024

Choose a reason for hiding this comment

mayeut May 27, 2024 • edited

Choose a reason for hiding this comment

henryiii May 28, 2024

Choose a reason for hiding this comment

henryiii left a comment

Choose a reason for hiding this comment

henryiii commented May 28, 2024

henryiii commented May 28, 2024

hoodmane commented Apr 4, 2023 •

edited

hoodmane commented Apr 7, 2023 •

edited

henryiii commented May 1, 2023 •

edited

hoodmane commented May 15, 2023 •

edited

mayeut commented May 27, 2024 •

edited

mayeut May 27, 2024 •

edited