Revives two previously broken MPASSI tests #2323

singhbalwinder · 2023-05-09T00:28:14Z

Revives two MPASSI tests that were previously broken on GPUs in hopes that more recent code changes might have fixed these. I have tested them on a CPU machine (Compy) to ensure that they work on a CPU machine. Due to the missing partition files for the default Compy configurations, I ran the following equivalent tests:

./cs.status.mpassi_tests3 -s
mpassi_tests3: 3 tests
  PASS ERP_Ln22_P64x1.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.compy_intel.atmlndactive-rtm_off
  PASS ERS_D_Ln22_P40x1.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.compy_intel.atmlndactive-rtm_off
  PASS PEM_Ln90_P128x1.ne30pg2_EC30to60E2r2.F2010-SCREAMv1-MPASSI.compy_intel

E3SM-Autotester · 2023-05-09T02:17:01Z

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

E3SM-Autotester · 2023-05-09T02:17:02Z

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection Is Not Necessary for this Pull Request.

E3SM-Autotester · 2023-05-09T02:56:59Z

Status Flag 'Pull Request AutoTester' - Failure: Timed out waiting for job SCREAM_PullRequest_Autotester_Mappy to start: Total Wait = 1803

Other jobs have been previously started - We must stop them...

E3SM-Autotester · 2023-05-09T04:40:51Z

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

E3SM-Autotester · 2023-05-09T05:20:25Z

Status Flag 'Pull Request AutoTester' - Failure: Timed out waiting for job SCREAM_PullRequest_Autotester_Mappy to start: Total Wait = 1803

Other jobs have been previously started - We must stop them...

E3SM-Autotester · 2023-05-09T10:20:26Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 3870
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS
PULLREQUESTNUM	2323
SCREAM_SOURCE_BRANCH	singhbalwinder/mpassi-tests
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`4940e3b`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`06a178c`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 4444
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS
PULLREQUESTNUM	2323
SCREAM_SOURCE_BRANCH	singhbalwinder/mpassi-tests
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`4940e3b`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`06a178c`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Blake

Build Num: 4498
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS
PULLREQUESTNUM	2323
SCREAM_SOURCE_BRANCH	singhbalwinder/mpassi-tests
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`4940e3b`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`06a178c`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: singhbalwinder

E3SM-Autotester · 2023-05-09T11:39:02Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 3870
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS
PULLREQUESTNUM	2323
SCREAM_SOURCE_BRANCH	singhbalwinder/mpassi-tests
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`4940e3b`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`06a178c`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 4444
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS
PULLREQUESTNUM	2323
SCREAM_SOURCE_BRANCH	singhbalwinder/mpassi-tests
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`4940e3b`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`06a178c`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Blake

Build Num: 4498
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS
PULLREQUESTNUM	2323
SCREAM_SOURCE_BRANCH	singhbalwinder/mpassi-tests
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`4940e3b`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`06a178c`
TEST_REPO_ALIAS	SCREAM

E3SM-Autotester · 2023-05-09T11:39:21Z

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
WARNING: NO REVIEWERS HAVE BEEN REQUESTED FOR THIS PULL REQUEST!

E3SM-Autotester · 2023-05-09T11:39:29Z

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

E3SM-Autotester · 2023-05-10T11:40:51Z

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

AaronDonahue · 2023-05-10T16:16:14Z

cime_config/tests.py

-         #  "ERS_Ln22.ne30pg2_EC30to60E2r2.F2010-SCREAMv1-MPASSI.atmlndactive-rtm_off",
+            #  "ERP_D_Ln9.ne4_oQU240.F2010-SCREAMv1-MPASSI.atmlndactive-rtm_off",
+            #  "SMS_D_Ln9.ne4_oQU240.F2010-SCREAMv1-MPASSI-noAero",
+            "ERP_Ln22.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.atmlndactive-rtm_off",


@singhbalwinder have you confirmed that these tests pass on pm-gpu now?

E3SM-Autotester · 2023-05-10T17:02:15Z

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
THE LAST COMMIT TO THIS PULL REQUEST HAS BEEN REVIEWED, BUT NOT ACCEPTED OR REQUIRES CHANGES

E3SM-Autotester · 2023-05-10T17:02:22Z

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

E3SM-Autotester · 2023-05-11T17:10:12Z

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

E3SM-Autotester · 2023-05-17T00:19:38Z

All Jobs Finished; status = PASSED, However PR is now STALE, and must be retested. Set the AT: RETEST Label to force retest....

E3SM-Autotester · 2023-05-18T00:20:49Z

All Jobs Finished; status = PASSED, However PR is now STALE, and must be retested. Set the AT: RETEST Label to force retest....

PeterCaldwell · 2023-05-18T14:58:01Z

@singhbalwinder and @AaronDonahue - my understanding was that Balwinder was going to just submit this PR to the AT and if it passed, we'd integrate it... we would have integrated it a long time ago except that it wasn't passing these tests :-). Though maybe I'm not understanding the situation correctly.

AaronDonahue · 2023-05-18T17:45:53Z

@PeterCaldwell , I'm happy to merge it. I had asked @singhbalwinder to check that this was going to work on pm-gpu which I think was the last step. He was going to report back once he did the check.

E3SM-Autotester · 2023-05-19T02:35:15Z

All Jobs Finished; status = PASSED, However PR is now STALE, and must be retested. Set the AT: RETEST Label to force retest....

E3SM-Autotester · 2023-05-20T02:49:43Z

All Jobs Finished; status = PASSED, However PR is now STALE, and must be retested. Set the AT: RETEST Label to force retest....

E3SM-Autotester · 2023-05-21T03:09:54Z

All Jobs Finished; status = PASSED, However PR is now STALE, and must be retested. Set the AT: RETEST Label to force retest....

E3SM-Autotester · 2023-05-22T03:39:54Z

All Jobs Finished; status = PASSED, However PR is now STALE, and must be retested. Set the AT: RETEST Label to force retest....

E3SM-Autotester · 2023-05-23T04:09:01Z

All Jobs Finished; status = PASSED, However PR is now STALE, and must be retested. Set the AT: RETEST Label to force retest....

E3SM-Autotester · 2023-05-24T04:38:20Z

All Jobs Finished; status = PASSED, However PR is now STALE, and must be retested. Set the AT: RETEST Label to force retest....

E3SM-Autotester · 2023-05-25T15:33:32Z

All Jobs Finished; status = PASSED, However PR is now STALE, and must be retested. Set the AT: RETEST Label to force retest....

PeterCaldwell · 2023-05-25T15:37:23Z

Followed Michael Jackson's advice and WIPped this PR since the AT keeps running but I know that Balwinder is tracking down errors on Perlmutter.

AaronDonahue · 2023-06-08T16:59:59Z

@singhbalwinder , do you have a status update on this PR?

singhbalwinder · 2023-06-08T17:24:07Z

I thought getting the fully coupled compset working is a more urgent task, so I spent time making it work. It works now! I will come to these tests as part of my next steps.

PeterCaldwell · 2023-06-08T17:27:06Z

Great that the coupled compset works! I thought that getting these MPASSI tests working was needed to get the coupled compset working since the coupled compset needs to use MPASSI. But I guess so long as you're not running on the machine that's having the MPASSI fails it isn't...

singhbalwinder · 2023-06-08T17:52:48Z

Yes, you are right. All tests were working on Compy, so I went ahead and started building the fully coupled compset on Compy. I am now looking into the setup to ensure that the simulation is set up correctly before moving to address the failing tests.

E3SM-Autotester · 2023-08-14T18:25:29Z

All Jobs Finished; status = PASSED, However PR is now STALE, and must be retested. Set the AT: RETEST Label to force retest....

Revive two previously broken tests

4940e3b

singhbalwinder added the AT: RETEST Force the autotester (AT) to retest the PR label May 9, 2023

E3SM-Autotester removed the AT: RETEST Force the autotester (AT) to retest the PR label May 9, 2023

singhbalwinder added the AT: RETEST Force the autotester (AT) to retest the PR label May 9, 2023

E3SM-Autotester removed the AT: RETEST Force the autotester (AT) to retest the PR label May 9, 2023

AaronDonahue reviewed May 10, 2023

View reviewed changes

AaronDonahue added testing BFB Bit for bit AT: AUTOMERGE Inform the autotester (AT) that it can merge this PR if reviewers approved, and tests pass AT: Skip Stand-Alone Testing Do not run SCREAM internal unit tests for AT runs labels May 10, 2023

E3SM-Autotester added the AT: STALE Label set by the autotester (AT). PR is stale, and should not be tested. Override with "AT: RETEST" label May 17, 2023

PeterCaldwell added the AT: WIP Inform the autotester (AT) that the PR is a work in progress, and should not be tested label May 25, 2023

Revives two previously broken MPASSI tests #2323

Are you sure you want to change the base?

Revives two previously broken MPASSI tests #2323

Conversation

singhbalwinder commented May 9, 2023

E3SM-Autotester commented May 9, 2023

E3SM-Autotester commented May 9, 2023

E3SM-Autotester commented May 9, 2023

E3SM-Autotester commented May 9, 2023

E3SM-Autotester commented May 9, 2023

E3SM-Autotester commented May 9, 2023

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Blake

Jenkins Parameters

Using Repos:

E3SM-Autotester commented May 9, 2023

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Blake

Jenkins Parameters

E3SM-Autotester commented May 9, 2023

E3SM-Autotester commented May 9, 2023

E3SM-Autotester commented May 10, 2023

AaronDonahue May 10, 2023

Choose a reason for hiding this comment

E3SM-Autotester commented May 10, 2023

E3SM-Autotester commented May 10, 2023

E3SM-Autotester commented May 11, 2023

E3SM-Autotester commented May 17, 2023

E3SM-Autotester commented May 18, 2023

PeterCaldwell commented May 18, 2023

AaronDonahue commented May 18, 2023

E3SM-Autotester commented May 19, 2023

E3SM-Autotester commented May 20, 2023

E3SM-Autotester commented May 21, 2023

E3SM-Autotester commented May 22, 2023

E3SM-Autotester commented May 23, 2023

E3SM-Autotester commented May 24, 2023

E3SM-Autotester commented May 25, 2023

PeterCaldwell commented May 25, 2023

AaronDonahue commented Jun 8, 2023

singhbalwinder commented Jun 8, 2023

PeterCaldwell commented Jun 8, 2023

singhbalwinder commented Jun 8, 2023

E3SM-Autotester commented Aug 14, 2023