Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PR: Try to decrease timeouts when running tests and check general tests robustness and speed (CI/Testing) #22077

Merged
merged 31 commits into from
May 31, 2024

Conversation

dalthviz
Copy link
Member

@dalthviz dalthviz commented May 13, 2024

Description of Changes

Some changes to improve CI tests runs robustness:

  • Check if closing the mainwindow instance for tests marked as know_leak helps reducing timeouts/number of tests runs attempts
  • Add missing logic to close editorwindows.
  • Remove usage of connect_to_oi Editor config option
  • Add order mark to a couple of test that were failing consistently: test_add_history and test_save_with_preferred_eol_chars
  • Comment out style application over plots and sidebardialog fixtures to decrease chance of segfault over fast tests
  • Prevent spyder_kernels.comms.commbase.CommError: The comm is not connected. on Windows when running IPython Console test by setting matplotlib backend to inline when running test instead of trying to get it via shellwidget.get_matplotlib_backend()
  • Prevent plots RuntimeError when testing due to scrollbars timer logic
  • Add logic to test_updates_appenv to prevent failing due to possible HTTP 403 errors
  • Change IPython Console tests percentage being run
  • Quit IPdb in some test before finishing them

Affirmation

By submitting this Pull Request or typing my (user)name below,
I affirm the Developer Certificate of Origin
with respect to all commits and content included in this PR,
and understand I am releasing the same under Spyder's MIT (Expat) license.

I certify the above statement is true and correct: dalthviz

@dalthviz dalthviz self-assigned this May 13, 2024
@dalthviz dalthviz changed the title [WIP] PR: Try to decrease timeouts when running tests (CI/Testing) [WIP] PR: Try to decrease timeouts when running tests and general tests robustness (CI/Testing) May 14, 2024
@ccordoba12
Copy link
Member

@dalthviz, while you're checking this, you could also decrease the percentage of IPython console tests run in the slow slots on Windows:

spyder/conftest.py

Lines 70 to 87 in 1fda346

# Break test suite in CIs according to the following criteria:
# * Mark all main window tests, and a percentage of the IPython console
# ones, as slow.
# * All other tests will be considered as fast.
# This provides a more balanced partitioning of our test suite (in terms of
# necessary time to run it) between the slow and fast slots we have on CIs.
slow_items = []
if os.environ.get('CI'):
slow_items = [
item for item in items if 'test_mainwindow' in item.nodeid
]
ipyconsole_items = [
item for item in items if 'test_ipythonconsole' in item.nodeid
]
if os.name == 'nt':
percentage = 0.5

That will make those slots run faster.

@ccordoba12 ccordoba12 added this to the v6.0beta2 milestone May 14, 2024
@dalthviz dalthviz changed the title [WIP] PR: Try to decrease timeouts when running tests and general tests robustness (CI/Testing) [WIP] PR: Try to decrease timeouts when running tests and check general tests robustness (CI/Testing) May 14, 2024
@dalthviz dalthviz changed the title [WIP] PR: Try to decrease timeouts when running tests and check general tests robustness (CI/Testing) [WIP] PR: Try to decrease timeouts when running tests and check general tests robustness and speed (CI/Testing) May 21, 2024
@pep8speaks
Copy link

pep8speaks commented May 21, 2024

Hello @dalthviz! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

Line 67:80: E501 line too long (81 > 79 characters)

Comment last updated at 2024-05-30 21:19:28 UTC

@dalthviz dalthviz changed the title [WIP] PR: Try to decrease timeouts when running tests and check general tests robustness and speed (CI/Testing) PR: Try to decrease timeouts when running tests and check general tests robustness and speed (CI/Testing) May 30, 2024
@dalthviz dalthviz marked this pull request as ready for review May 30, 2024 00:08
Copy link
Member

@ccordoba12 ccordoba12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @dalthviz for your hard work on this!

conftest.py Outdated Show resolved Hide resolved
runtests.py Outdated Show resolved Hide resolved
spyder/app/tests/conftest.py Outdated Show resolved Hide resolved
spyder/plugins/editor/tests/test_plugin.py Outdated Show resolved Hide resolved
spyder/plugins/plots/tests/test_plugin.py Outdated Show resolved Hide resolved
spyder/plugins/plots/widgets/figurebrowser.py Outdated Show resolved Hide resolved
spyder/widgets/tests/test_sidebardialog.py Outdated Show resolved Hide resolved
@ccordoba12
Copy link
Member

There's also a small error when running the teardown of test_outline_in_maximized_editor on the Windows slot tests, but that should be easy to fix.

Copy link
Member

@ccordoba12 ccordoba12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great! Thanks @dalthviz for taking the time to take a look at it.

@ccordoba12 ccordoba12 merged commit b1cbb58 into spyder-ide:master May 31, 2024
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants