Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The CI/CD release validation work flows must run the non-deterministic tests multiple times in a loop. #3376

Open
shankarseal opened this issue Mar 22, 2024 · 1 comment · May be fixed by #3458
Open

The CI/CD release validation work flows must run the non-deterministic tests multiple times in a loop. #3376

shankarseal opened this issue Mar 22, 2024 · 1 comment · May be fixed by #3458
Assignees
@matthewige
Labels
ci/cd Issue is specific to CI/CD enhancement New feature or request P2 tests triaged Discussed in a triage meeting
Milestone
2406

Comments

@shankarseal
Copy link
Collaborator

shankarseal commented Mar 22, 2024 •
edited

"... recommending running something three times indicates that there are gaps. Instead going forward, CI/CD could run the same test three times so the release manager doesn't need to count."

_Originally posted by @dthaler in #3357 (comment)
@shankarseal shankarseal mentioned this issue Mar 22, 2024
Merged
@dthaler dthaler added ci/cd Issue is specific to CI/CD enhancement New feature or request tests labels Mar 25, 2024
@dahavey dahavey added the triaged Discussed in a triage meeting label Mar 25, 2024
@dahavey dahavey added this to the 2404 milestone Mar 25, 2024
@matthewige
Copy link
Collaborator

As discussed in the triage meeting: We have a set of non-deterministic tests, that we would like to execute for a longer period of time in the release validation workflow, to help ensure that we are publishing a 'good' build. These will help these tests hit extra conditions (which is essentially what we would like to validate existing flow).

The following was proposed: the stress and fuzz tests should run for 3x the normal PR CICD validation time.

@dahavey dahavey added the P2 label Mar 27, 2024
@matthewige matthewige linked a pull request Apr 16, 2024 that will close this issue
Update all non-deterministic tests to run 3x the normal CI/CD workflow on the release validation pipeline #3458
Draft
@matthewige matthewige linked a pull request Apr 16, 2024 that will close this issue
Draft
@shankarseal shankarseal modified the milestones: 2404, 2405, 2406 May 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@matthewige matthewige

Labels
ci/cd Issue is specific to CI/CD enhancement New feature or request P2 tests triaged Discussed in a triage meeting
Projects
None yet
Milestone
2406
4 participants
@dthaler @matthewige @dahavey @shankarseal