Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove popularity & matview timeouts from data refresh configurations #4352

Closed
AetherUnbound opened this issue May 17, 2024 · 0 comments · Fixed by #4353
Closed

Remove popularity & matview timeouts from data refresh configurations #4352

AetherUnbound opened this issue May 17, 2024 · 0 comments · Fixed by #4353
Labels
💻 aspect: code Concerns the software code in the repository 🧰 goal: internal improvement Improvement that benefits maintainers, not users good first issue New-contributor friendly help wanted Open to participation from the community 🟩 priority: low Low priority and doesn't need to be rushed 🧱 stack: catalog Related to the catalog and Airflow DAGs 🐍 tech: python Involves Python

Comments

@AetherUnbound
Copy link
Contributor

Description

The DataRefresh dataclass has slots for the following fields that are vestigial and no longer necessary:

  • refresh_metrics_timeout
  • refresh_matview_timeout
  • create_pop_constants_view_timeout
  • create_materialized_view_timeout

These can all be removed from data_refresh_types.py, since they are not used:

openverse/catalog/dags/legacy_data_refresh/data_refresh_types.py

Lines 36 to 44 in bc457c0

refresh_metrics_timeout: timedelta expressing amount of time the
refresh popularity metrics tasks may take.
refresh_matview_timeout: timedelta expressing amount of time the
refresh of the popularity matview may take.
create_pop_constants_view_timeout: timedelta expressing amount of time the
creation of the popularity constants view
may take
create_materialized_view_timeout: timedelta expressing amount of time the
creation of the matview may take

openverse/catalog/dags/legacy_data_refresh/data_refresh_types.py

Lines 63 to 66 in bc457c0

refresh_metrics_timeout: timedelta = timedelta(hours=1)
refresh_matview_timeout: timedelta = timedelta(hours=1)
create_pop_constants_view_timeout: timedelta = timedelta(hours=1)
create_materialized_view_timeout: timedelta = timedelta(hours=1)

openverse/catalog/dags/legacy_data_refresh/data_refresh_types.py

Lines 81 to 84 in bc457c0

refresh_metrics_timeout=timedelta(hours=24),
refresh_matview_timeout=timedelta(hours=72),
create_pop_constants_view_timeout=timedelta(hours=24),
create_materialized_view_timeout=timedelta(hours=72),

Additional context

This was left over from #433
@AetherUnbound AetherUnbound added good first issue New-contributor friendly help wanted Open to participation from the community 🐍 tech: python Involves Python 💻 aspect: code Concerns the software code in the repository 🟩 priority: low Low priority and doesn't need to be rushed 🧰 goal: internal improvement Improvement that benefits maintainers, not users 🧱 stack: catalog Related to the catalog and Airflow DAGs labels May 17, 2024
@johan-naizu johan-naizu mentioned this issue May 17, 2024
Merged
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
💻 aspect: code Concerns the software code in the repository 🧰 goal: internal improvement Improvement that benefits maintainers, not users good first issue New-contributor friendly help wanted Open to participation from the community 🟩 priority: low Low priority and doesn't need to be rushed 🧱 stack: catalog Related to the catalog and Airflow DAGs 🐍 tech: python Involves Python
Projects
Openverse Backlog
Archived in project
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Remove popularity & matview timeouts from data refresh configurations johan-naizu/openverse
1 participant
@AetherUnbound