Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WiP : CPython3 inclusion #1216

Draft
wants to merge 4 commits into
base: master
Choose a base branch
from
Draft

WiP : CPython3 inclusion #1216

wants to merge 4 commits into from

Conversation

tlaurion
Copy link
Collaborator

This is still not working, and aimed at seeking help from the community into resolving packaging problems when crosscompiling python3 even today.

  • Includes upstream fixes for cross compilation recognized problems (see patch in PR.)
  • Builds against python 3.9.2 (debian's 11 current python. Yes, we depend on the host's python to compile destination's)

As of current state creating this draft, I am not able to force python using its own --with-libc="$(musl-cross_libraries)" which should point to musl-cross-make's libc.so. Python still attempts to link against host's libc, resulting in:

Traceback (most recent call last):
File "/home/user/heads/build/x86/python-3.9.2/Lib/decimal.py", line 3, in
from _decimal import *
ImportError: /lib/x86_64-linux-gnu/libc.so: invalid ELF header

Recommendations are sparse, and were documented prior into #233 #689.

When those problems are resolved, this will:
Fixes #233 #689

Lotsa warnings..... Need to troubleshooting before opening PR, IF cross-build worked
- Attempt to remove include from host system with upstream patches against 3.9.2 (debian-11 locally deployed python)
@tlaurion tlaurion marked this pull request as draft September 21, 2022 21:23
@tlaurion
Copy link
Collaborator Author

tlaurion commented Sep 21, 2022

The builds fails at https://app.circleci.com/pipelines/github/tlaurion/heads/1205/workflows/4ad764c2-b1d4-4273-ba58-19cc322c44f0/jobs/10624?invite=true#step-103-3751, trying to link against host's libc instead of musl-cross-make's, even if specified properly in the module https://github.com/osresearch/heads/pull/1216/files#diff-51bef0474d1464907f19301711f3d641a3621d8d86d4f6072350c1bf290078b1R28.

--with-libc="$(musl-cross_libraries)" here is correctly expended to --with-libc=/root/project/crossgcc/x86/x86_64-linux-musl/lib/libc.so from environement variables passed to ./configure script here: https://github.com/osresearch/heads/pull/1216/files#diff-51bef0474d1464907f19301711f3d641a3621d8d86d4f6072350c1bf290078b1R13-R33, resulting in:

PKG_CONFIG=/bin/false READELF=/root/project/crossgcc/x86/bin/x86_64-linux-musl-readelf ac_cv_file__dev_ptmx=no ac_cv_file__dev_ptc=no have_openssl=no CFLAGS=-Os CC=/root/project/crossgcc/x86/bin/x86_64-linux-musl-gcc -D__MUSL__ -isystem /root/project/install/x86/include -L/root/project/install/x86/lib AR=/root/project/crossgcc/x86/bin/x86_64-linux-musl-ar LD=/root/project/crossgcc/x86/bin/x86_64-linux-musl-ld STRIP=/root/project/crossgcc/x86/bin/x86_64-linux-musl-strip NM=/root/project/crossgcc/x86/bin/x86_64-linux-musl-nm OBJCOPY=/root/project/crossgcc/x86/bin/x86_64-linux-musl-objcopy OBJDUMP=/root/project/crossgcc/x86/bin/x86_64-linux-musl-objdump PKG_CONFIG_PATH=/root/project/install/x86/lib/pkgconfig PKG_CONFIG_SYSROOT_DIR=/root/project/install/x86 ./configure --enable-optimizations --with-cxx-main=/root/project/crossgcc/x86/bin/x86_64-linux-musl-gcc --with-libc=/root/project/crossgcc/x86/x86_64-linux-musl/lib/libc.so --host x86_64-linux-musl --build x86_64 --disable-ipv6 --prefix= --exec-prefix=

Still, the following error happens since the wrong libc is picked (host vs musl-cross-make's).
Look for "ImportError: /usr/lib/x86_64-linux-gnu/libc.so: invalid ELF header" below:

if test "xupgrade" != "xno"  ; then \
	case upgrade in \
		upgrade) ensurepip="--upgrade" ;; \
		install|*) ensurepip="" ;; \
	esac; \
	 _PYTHON_PROJECT_BASE=/root/project/build/x86/python-3.9.2 _PYTHON_HOST_PLATFORM=linux-x86_64 PYTHONPATH=/root/project/build/x86/python-3.9.2/build/lib.linux-x86_64-3.9:./Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata__linux_x86_64-linux-gnu python3.9 -m ensurepip \
		$ensurepip --root=/root/project/install/x86/ ; \
fi
Traceback (most recent call last):
  File "/root/project/build/x86/python-3.9.2/Lib/decimal.py", line 3, in <module>
    from _decimal import *
ImportError: /usr/lib/x86_64-linux-gnu/libc.so: invalid ELF header

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 6, in <module>
  File "/root/project/build/x86/python-3.9.2/Lib/runpy.py", line 210, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/root/project/build/x86/python-3.9.2/Lib/runpy.py", line 97, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/root/project/build/x86/python-3.9.2/Lib/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/tmp/tmp6ngaaq2s/pip-20.2.3-py2.py3-none-any.whl/pip/__main__.py", line 26, in <module>
  File "/tmp/tmp6ngaaq2s/pip-20.2.3-py2.py3-none-any.whl/pip/_internal/cli/main.py", line 73, in main
  File "/tmp/tmp6ngaaq2s/pip-20.2.3-py2.py3-none-any.whl/pip/_internal/commands/__init__.py", line 104, in create_command
  File "/root/project/build/x86/python-3.9.2/Lib/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 627, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmp6ngaaq2s/pip-20.2.3-py2.py3-none-any.whl/pip/_internal/commands/install.py", line 17, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmp6ngaaq2s/pip-20.2.3-py2.py3-none-any.whl/pip/_internal/cli/req_command.py", line 16, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmp6ngaaq2s/pip-20.2.3-py2.py3-none-any.whl/pip/_internal/index/collector.py", line 15, in <module>
  File "<frozen zipimport>", line 259, in load_module
  File "/tmp/tmp6ngaaq2s/pip-20.2.3-py2.py3-none-any.whl/pip/_vendor/distlib/compat.py", line 83, in <module>
  File "/root/project/build/x86/python-3.9.2/Lib/xmlrpc/client.py", line 135, in <module>
    from decimal import Decimal
  File "/root/project/build/x86/python-3.9.2/Lib/decimal.py", line 8, in <module>
    from _pydecimal import *
  File "/root/project/build/x86/python-3.9.2/Lib/_pydecimal.py", line 440, in <module>
    import contextvars
  File "/root/project/build/x86/python-3.9.2/Lib/contextvars.py", line 1, in <module>
    from _contextvars import Context, ContextVar, Token, copy_context
ImportError: /usr/lib/x86_64-linux-gnu/libc.so: invalid ELF header
Traceback (most recent call last):
  File "/root/project/build/x86/python-3.9.2/Lib/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/root/project/build/x86/python-3.9.2/Lib/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/root/project/build/x86/python-3.9.2/Lib/ensurepip/__main__.py", line 5, in <module>
    sys.exit(ensurepip._main())
  File "/root/project/build/x86/python-3.9.2/Lib/ensurepip/__init__.py", line 210, in _main
    return _bootstrap(
  File "/root/project/build/x86/python-3.9.2/Lib/ensurepip/__init__.py", line 129, in _bootstrap
    return _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
  File "/root/project/build/x86/python-3.9.2/Lib/ensurepip/__init__.py", line 38, in _run_pip
    return subprocess.run([sys.executable, "-c", code], check=True).returncode
  File "/root/project/build/x86/python-3.9.2/Lib/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/python3.9', '-c', '\nimport runpy\nimport sys\nsys.path = [\'/tmp/tmp6ngaaq2s/setuptools-49.2.1-py3-none-any.whl\', \'/tmp/tmp6ngaaq2s/pip-20.2.3-py2.py3-none-any.whl\'] + sys.path\nsys.argv[1:] = [\'install\', \'--no-cache-dir\', \'--no-index\', \'--find-links\', \'/tmp/tmp6ngaaq2s\', \'--root\', \'/root/project/install/x86/\', \'--upgrade\', \'setuptools\', \'pip\']\nrunpy.run_module("pip", run_name="__main__", alter_sys=True)\n']' returned non-zero exit status 1.
make[1]: *** [Makefile:1255: install] Error 1
make[1]: Leaving directory '/root/project/build/x86/python-3.9.2'
tail /root/project/build/x86/log/python.log
-----
    from _contextvars import Context, ContextVar, Token, copy_context
ImportError: /usr/lib/x86_64-linux-gnu/libc.so: invalid ELF header
Traceback (most recent call last):
  File "/root/project/build/x86/python-3.9.2/Lib/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/root/project/build/x86/python-3.9.2/Lib/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/root/project/build/x86/python-3.9.2/Lib/ensurepip/__main__.py", line 5, in <module>
    sys.exit(ensurepip._main())
  File "/root/project/build/x86/python-3.9.2/Lib/ensurepip/__init__.py", line 210, in _main
    return _bootstrap(
  File "/root/project/build/x86/python-3.9.2/Lib/ensurepip/__init__.py", line 129, in _bootstrap
    return _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
  File "/root/project/build/x86/python-3.9.2/Lib/ensurepip/__init__.py", line 38, in _run_pip
    return subprocess.run([sys.executable, "-c", code], check=True).returncode
  File "/root/project/build/x86/python-3.9.2/Lib/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/bin/python3.9', '-c', '\nimport runpy\nimport sys\nsys.path = [\'/tmp/tmp6ngaaq2s/setuptools-49.2.1-py3-none-any.whl\', \'/tmp/tmp6ngaaq2s/pip-20.2.3-py2.py3-none-any.whl\'] + sys.path\nsys.argv[1:] = [\'install\', \'--no-cache-dir\', \'--no-index\', \'--find-links\', \'/tmp/tmp6ngaaq2s\', \'--root\', \'/root/project/install/x86/\', \'--upgrade\', \'setuptools\', \'pip\']\nrunpy.run_module("pip", run_name="__main__", alter_sys=True)\n']' returned non-zero exit status 1.
make[1]: *** [Makefile:1255: install] Error 1
make[1]: Leaving directory '/root/project/build/x86/python-3.9.2'
make: *** [Makefile:408: /root/project/build/x86/python-3.9.2/.build] Error 1

Issue tracked under #689. This PR will stall until a fix is found, discussions can happen into referred tickets.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Python with musl-libc
1 participant