Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix misleading test collection reporting in terminal output #533

Open
danceratopz opened this issue Apr 29, 2024 · 0 comments
Open

Fix misleading test collection reporting in terminal output #533

danceratopz opened this issue Apr 29, 2024 · 0 comments

Comments

@danceratopz
Copy link
Member

After the pytest header and test collection (but before test execution) pytest incorrectly reports how many tests are collected. This is visible with and without --collect-only.

For example:

fill tests/istanbul/eip1344_chainid/ --fork=Cancun --collect-only

outputs (incorrect output is marked as such):

================================== test session starts ===================================
platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0
Executing tests for: Cancun 
Only executing tests with stable/deployed forks: Specify an upcoming fork via --until=fork to add forks under development.
rootdir: /home/dtopz/code/github/danceratopz/execution-spec-tests
configfile: pytest.ini
plugins: cov-4.1.0, xdist-3.6.1
collecting ... start
collected 3 items / 1 deselected / 2 selected  # <-- INCORRECT, should be "collected 3 items"

<Package eip1344_chainid>
  <Module test_chainid.py>
    <Function test_chainid[fork_Cancun-blockchain_test]>
    <Function test_chainid[fork_Cancun-blockchain_test_hive]>
    <Function test_chainid[fork_Cancun-state_test]>

====================== 2/3 tests collected (1 deselected) in 0.01s =======================   # <-- INCORRECT, should be "3 tests collected in 0.01 s"

For every test module collected, one additional "test" will be incorrectly reported as deselected (above we only collect one module).

The error is due to the fact that we automatically add EIP version tests in the spec_version_checker plugin (src/pytest_plugins/spec_version_checker/spec_version_checker.py) but then also disable them by default in fill's pytest.ini (L9 and L11):

addopts =
-p pytest_plugins.test_filler.test_filler
-p pytest_plugins.forks.forks
-p pytest_plugins.spec_version_checker.spec_version_checker
-p pytest_plugins.test_help.test_help
-m "not eip_version_check"

The most reasonable solution would be to simply not use this plugin when executing fill and have yet another pytest command dedicated to version checking that can be ran regularly in CI. The reason to not enable these tests by default is that each of these version checking tests makes a request to the Github API (to retrieve info from ethereum/EIPs); without authentication setup, fill quickly hits the Github API rate limit.

Note, as of 2024-04-29, eip version checking is currently entirely unused in our CI. To run these tests, it's required to explicitly select tests marked as eip_version_check with:

fill tests/istanbul/eip1344_chainid/ --fork=Cancun -m eip_version_check -v
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant