Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

downloaders/s3: migrate from boto (2.x) to boto3 #5597

Open
yarikoptic opened this issue Apr 22, 2021 · 2 comments · May be fixed by #7575
Open

downloaders/s3: migrate from boto (2.x) to boto3 #5597

yarikoptic opened this issue Apr 22, 2021 · 2 comments · May be fixed by #7575
Assignees
@mslw
Labels
cmd-download-url corpse-in-basement enhancement stale-issue-closed-without-resolution

Comments

@yarikoptic
Copy link
Member

ATM we are still using boto2, whenever boto3 had been out for a while.
We had accumulated some craft to fulfill our cases using boto2, so may be some of it could also go away.
@mih
Copy link
Member

mih commented Nov 7, 2021

Thank you for reporting this issue!

However, at this point I am closing it at a time of ~600 open issues, in an attempt to regain control over the issue tracker. This particular issue was posted more than 6 months ago, has not received a single response, and does not describe an immediate software defect. Closing this issue does not imply that it is not relevant. If any reader objects the closing of this issue, please feel more than free to reopen it with an update on how it is relevant for the current state of the project, and which concrete actions to address it are ongoing or planned.

@mih mih closed this as completed Nov 7, 2021
@renesat renesat mentioned this issue Sep 8, 2022
Closed
@mslw mslw self-assigned this Mar 13, 2023
@mslw mslw reopened this Mar 13, 2023
@mslw mslw mentioned this issue Mar 19, 2023
Closed
@yarikoptic yarikoptic mentioned this issue Mar 21, 2024
Closed
@effigies
Copy link
Contributor

effigies commented Apr 2, 2024 •
edited

I'm now running into an unpleasant interaction of pytest 8.1 and boto in OpenNeuroOrg/openneuro#3020, due to Datalad's dependency.

❯ pipenv run pytest ./tests --pdb
============================================= test session starts =============================================
platform linux -- Python 3.10.10, pytest-8.1.1, pluggy-1.4.0
rootdir: /home/chris/Projects/openneuro/openneuro/services/datalad
configfile: pytest.ini
plugins: xdist-3.5.0, cov-4.1.0
collecting ... 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> captured stderr >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
[WARNING] Requested extension 'next' is not available 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
/home/chris/.local/share/virtualenvs/datalad-CL7RjNid/lib/python3.10/site-packages/_pytest/runner.py:340: in from_call
    result: Optional[TResult] = func()
/home/chris/.local/share/virtualenvs/datalad-CL7RjNid/lib/python3.10/site-packages/_pytest/runner.py:388: in collect
    return list(collector.collect())
/home/chris/.local/share/virtualenvs/datalad-CL7RjNid/lib/python3.10/site-packages/_pytest/python.py:576: in collect
    self._register_setup_module_fixture()
/home/chris/.local/share/virtualenvs/datalad-CL7RjNid/lib/python3.10/site-packages/_pytest/python.py:589: in _register_setup_module_fixture
    self.obj, ("setUpModule", "setup_module")
/home/chris/.local/share/virtualenvs/datalad-CL7RjNid/lib/python3.10/site-packages/_pytest/python.py:315: in obj
    self._obj = obj = self._getobj()
/home/chris/.local/share/virtualenvs/datalad-CL7RjNid/lib/python3.10/site-packages/_pytest/python.py:573: in _getobj
    return importtestmodule(self.path, self.config)
/home/chris/.local/share/virtualenvs/datalad-CL7RjNid/lib/python3.10/site-packages/_pytest/python.py:520: in importtestmodule
    mod = import_path(
/home/chris/.local/share/virtualenvs/datalad-CL7RjNid/lib/python3.10/site-packages/_pytest/pathlib.py:546: in import_path
    mod = _import_module_using_spec(
/home/chris/.local/share/virtualenvs/datalad-CL7RjNid/lib/python3.10/site-packages/_pytest/pathlib.py:630: in _import_module_using_spec
    spec = meta_importer.find_spec(module_name, [str(module_location)])
E   AttributeError: '_SixMetaPathImporter' object has no attribute 'find_spec'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /home/chris/.local/share/virtualenvs/datalad-CL7RjNid/lib/python3.10/site-packages/_pytest/pathlib.py(630)_import_module_using_spec()
-> spec = meta_importer.find_spec(module_name, [str(module_location)])
(Pdb) meta_importer
<boto.vendored.six._SixMetaPathImporter object at 0x76dad9949d20>
(Pdb) l
625  	        "src_tests_test_foo".
626  	    """
627  	    # Checking with sys.meta_path first in case one of its hooks can import this module,
628  	    # such as our own assertion-rewrite hook.
629  	    for meta_importer in sys.meta_path:
630  ->	        spec = meta_importer.find_spec(module_name, [str(module_location)])
631  	        if spec is not None:
632  	            break
633  	    else:
634  	        spec = importlib.util.spec_from_file_location(module_name, str(module_path))
635  	    if spec is not None:

I will see if pytest can be smarter about catching the exception, but if boto is injecting things into the sys module, it's likely to cause more problems as it ages.

boto also imports imp, which should break on Python 3.12.

Edit: Posted upstream to pytest: pytest-dev/pytest#12179

@yarikoptic yarikoptic mentioned this issue Apr 2, 2024
Closed
This was referenced Apr 6, 2024
Open
Open
@yarikoptic yarikoptic mentioned this issue May 6, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mslw mslw

Labels
cmd-download-url corpse-in-basement enhancement stale-issue-closed-without-resolution
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

RF: Drop boto for boto3 effigies/datalad
4 participants
@yarikoptic @effigies @mih @mslw