cli: TestLossOfQuorumRecovery failed

[github-actions]

cli.TestLossOfQuorumRecovery failed on release-24.1 @ 556680db129ea635342767fdd916628409071ef1:

=== RUN   TestLossOfQuorumRecovery
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestLossOfQuorumRecovery3840205252
    test_log_scope.go:81: use -show-logs to present logs inline
[debug recover collect-info --store=/var/lib/engflow/worker/work/2/exec/_tmp/0ede0149c79dc299cd458e6d0ea4f601/TestLossOfQuorumRecovery1044422253/store-1 /var/lib/engflow/worker/work/2/exec/_tmp/0ede0149c79dc299cd458e6d0ea4f601/TestLossOfQuorumRecovery1044422253/node-1.json]
debug recover collect-info --store=/var/lib/engflow/worker/work/2/exec/_tmp/0ede0149c79dc299cd458e6d0ea4f601/TestLossOfQuorumRecovery1044422253/store-1 /var/lib/engflow/worker/work/2/exec/_tmp/0ede0149c79dc299cd458e6d0ea4f601/TestLossOfQuorumRecovery1044422253/node-1.json
Collected recovery info from:
nodes             1
stores            1
Collected info:
replicas          68
range descriptors 0
[debug recover make-plan --confirm=y --plan=/var/lib/engflow/worker/work/2/exec/_tmp/0ede0149c79dc299cd458e6d0ea4f601/TestLossOfQuorumRecovery1044422253/recovery-plan.json /var/lib/engflow/worker/work/2/exec/_tmp/0ede0149c79dc299cd458e6d0ea4f601/TestLossOfQuorumRecovery1044422253/node-1.json]
    debug_recover_loss_of_quorum_test.go:235: 
        	Error Trace:	github.com/cockroachdb/cockroach/pkg/cli/debug_recover_loss_of_quorum_test.go:235
        	Error:      	"debug recover make-plan --confirm=y --plan=/var/lib/engflow/worker/work/2/exec/_tmp/0ede0149c79dc299cd458e6d0ea4f601/TestLossOfQuorumRecovery1044422253/recovery-plan.json /var/lib/engflow/worker/work/2/exec/_tmp/0ede0149c79dc299cd458e6d0ea4f601/TestLossOfQuorumRecovery1044422253/node-1.json\nTotal replicas analyzed: 68\nRanges without quorum:   68\nDiscarded live replicas: 0\n\nProposed changes:\n  range r1:/Min updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r2:/System/NodeLiveness updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r3:/System/NodeLivenessMax updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r4:/System/tsd updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r5:/System/\"tse\" updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r6:/Table/0 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r7:/Table/3 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r8:/Table/4 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r9:/Table/5 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r10:/Table/6 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r11:/Table/7 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r12:/Table/8 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r13:/Table/9 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r14:/Table/11 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r15:/Table/12 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r16:/Table/13 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r17:/Table/14 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r18:/Table/15 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r19:/Table/16 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r20:/Table/17 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r21:/Table/18 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r22:/Table/19 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r23:/Table/20 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r24:/Table/21 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r25:/Table/22 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r26:/Table/23 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r27:/Table/24 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r28:/Table/25 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r29:/Table/26 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r30:/Table/27 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r31:/Table/28 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r32:/Table/29 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r33:/NamespaceTable/30 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r34:/NamespaceTable/Max updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r35:/Table/32 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r36:/Table/33 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r37:/Table/34 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r38:/Table/35 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r39:/Table/36 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r40:/Table/37 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r41:/Table/38 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r42:/Table/39 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r43:/Table/40 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r44:/Table/41 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r45:/Table/42 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r46:/Table/43 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r47:/Table/44 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r48:/Table/45 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r49:/Table/46 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r50:/Table/47 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r51:/Table/48 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r52:/Table/50 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r53:/Table/51 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r54:/Table/52 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r55:/Table/53 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r56:/Table/54 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r57:/Table/55 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r58:/Table/56 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r59:/Table/57 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r60:/Table/58 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r61:/Table/59 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r62:/Table/60 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r63:/Table/61 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r64:/Table/62 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n3,s3):3,(n2,s2):2].\n  range r65:/Table/63 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r66:/Table/64 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r67:/Table/65 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n  range r68:/Table/66 updating replica (n1,s1):1 to (n1,s1):14. Discarding available replicas: [], discarding dead replicas: [(n2,s2):3,(n3,s3):2].\n\nDiscovered dead nodes, will be marked as decommissioned:\n  n2, n3\n\n\nFound replica inconsistencies:\n\nrange has unapplied split operation\n  r68, /{Table/66-Max} rhs r69, /{Table/Max-Max}\n\nOnly proceed as a last resort!\nERROR: can not create plan because of errors and no --force flag is given\n" does not contain "- node n1"
        	Test:       	TestLossOfQuorumRecovery
        	Messages:   	planner didn't provide correct apply instructions
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestLossOfQuorumRecovery3840205252
--- FAIL: TestLossOfQuorumRecovery (67.78s)

Parameters:

• attempt=1
• run=1
• shard=12

Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv @cockroachdb/server

This test on roachdash | Improve this report!

Jira issue: CRDB-38743

[blathers-crl]

cc @cockroachdb/replication

[nicktrav]

Closed by #121963.