Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split integration tests per provider #4185

Open
tokoko opened this issue May 8, 2024 · 0 comments
Open

Split integration tests per provider #4185

tokoko opened this issue May 8, 2024 · 0 comments
Labels
kind/feature New feature or request

Comments

@tokoko
Copy link
Collaborator

tokoko commented May 8, 2024

Is your feature request related to a problem? Please describe.
Integration tests are currently run tests for all implementations (offline, online, registry, materialization engine) in a single pytest call. There are several problems with this approach:

  • It's easy to miss a required dependency in a particular extra as transitive dependencies from the whole project needs to be installed for tests to run.
  • Other implementation dependency might lead to downgrading the dependency for an implementation that's being tested. This means to that we will have library==0.1 tested in ci, while when user installs only the dependencies required for them, they will get library==0.2.
  • We run it local tests twice in ci, once only for local tests and then again for all. With each implementation having a separate test job, we could delay only it tests with ok-to-test label.
  • It's a chore to run tests only for the implementation that you're working on locally.

Cons:

  • we will have to compile python environments for each action run instead of now having only ci requirements file in the repo.
  • Lots of jobs in GH workflows (Is this a bad practce?)

Describe the solution you'd like
I'm thinking of splitting up into runs like this:

  • offline tests - will run tests from /offline_store and /materialization folders. will use default online store (sqlite), registry (file) and all applicable mat. engines whenever necessary.
    • test-python-integration-offline-file
    • test-python-integration-offline-duckdb
    • test-python-integration-offline-spark
    • ...
  • online tests - will run tests from /online_store and /materialization folders. will use default offline store (dask), registry (file) and all applicable mat. engines whenever necessary.
    • test-python-integration-online-sqlite
    • test-python-integration-online-redis
    • ...
  • registry tests - will run tests from /registration
    • test-python-integration-registry-file
    • test-python-integration-registry-sql
    • test-python-integration-registry-s3
    • ...

P.S. There's also e2e folder in integration tests, but at this point I'm not convinced it's actually necessary. One file is testing usage functionality which is set to be deleted anyway. test_universal_e2e.py seems to be a misplaced materialization test.

@tokoko tokoko added the kind/feature New feature or request label May 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant