Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

roachtest: prefix all logs with worker tags #114045

Closed
RaduBerinde opened this issue Nov 8, 2023 · 1 comment · Fixed by #124027
Closed

roachtest: prefix all logs with worker tags #114045

RaduBerinde opened this issue Nov 8, 2023 · 1 comment · Fixed by #124027
Labels
C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) E-quick-win Likely to be a quick win for someone experienced. T-testeng TestEng Team
Projects
Test Engineering

Comments

@RaduBerinde
Copy link
Member

RaduBerinde commented Nov 8, 2023 •
edited by cockroach-jira-scripts

Many of the runner logs don't have the worker tag (see below). It would be ideal if all messages had it and you could in principle break up the logs by wX tags to see what each worker is doing.

19:56:24 test_runner.go:743: [w1] Starting test: gopg:1 on cluster=teamcity-12554668-1699382900-25-n1cpu4 (arch="amd64")
19:56:24 cluster_synced.go:1984: teamcity-12554668-1699382900-25-n1cpu4: putting (scp) /go/src/github.com/cockroachdb/cockroach/bin/cockroach.linux-amd64 ./cockroach-default on nodes [1]
19:56:36 cluster_synced.go:2186: 
19:56:36 cluster_synced.go:2191:    1: done
19:56:36 cluster_synced.go:2191:    2: done
19:56:36 cluster_synced.go:2191:    3: done
19:56:36 cluster_synced.go:2191:    4: done
19:56:36 cluster_synced.go:2191:    5: done
19:56:36 cluster_synced.go:2191:    6: done
19:56:36 cluster_synced.go:2191:    7: done
19:56:36 cluster_synced.go:2191:    8: done
19:56:36 cluster_synced.go:2191:    9: done
19:56:41 cluster_synced.go:2186: 
19:56:41 cluster_synced.go:2191:    1: done
19:57:23 roachprod.go:1271: OK
19:57:23 cluster.go:1668: destroying cluster teamcity-12554668-1699382900-08-n3cpu4 [tag:] (3 nodes)... done
19:57:26 test_runner.go:613: [w4] Releasing quota for 12 CPUs

Jira issue: CRDB-33316
@RaduBerinde RaduBerinde added the C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) label Nov 8, 2023
@RaduBerinde RaduBerinde added this to Triage in Test Engineering via automation Nov 8, 2023
@blathers-crl blathers-crl bot added the T-testeng TestEng Team label Nov 8, 2023
@blathers-crl blathers[crl]
Copy link

blathers-crl bot commented Nov 8, 2023

cc @cockroachdb/test-eng

@srosenberg srosenberg added the E-quick-win Likely to be a quick win for someone experienced. label Nov 16, 2023
@exalate-issue-sync exalate-issue-sync bot removed the E-quick-win Likely to be a quick win for someone experienced. label Nov 16, 2023
@srosenberg srosenberg added the E-quick-win Likely to be a quick win for someone experienced. label Nov 20, 2023
vidit-bhat added a commit to vidit-bhat/cockroach that referenced this issue May 13, 2024
@vidit-bhat
roachtest: prefix all logs with worker tags
b627984 
Previously, many of the runner logs didn't have the worker tags.
This was inadequate because we couldn't break up the logs by wX
tags to see what each worker is doing.
This PR prefixes the worker tags to logs that were missing them.

Epic: none
Fixes: cockroachdb#114045
Release note: None
@vidit-bhat vidit-bhat mentioned this issue May 13, 2024
Merged
vidit-bhat added a commit to vidit-bhat/cockroach that referenced this issue May 13, 2024
@vidit-bhat
roachtest: prefix all logs with worker tags
336d8c3 
Previously, many of the runner logs didn't have the worker tags.
This was inadequate because we couldn't break up the logs by wX
tags to see what each worker is doing.
This PR prefixes the worker tags to logs that were missing them.

Epic: none
Fixes: cockroachdb#114045
Release note: None
vidit-bhat added a commit to vidit-bhat/cockroach that referenced this issue May 14, 2024
@vidit-bhat
roachtest: prefix all logs with worker tags
63583c5 
Previously, many of the runner logs didn't have the worker tags.
This was inadequate because we couldn't break up the logs by wX
tags to see what each worker is doing.
This PR prefixes the worker tags to logs that were missing them.

Epic: none
Fixes: cockroachdb#114045
Release note: None
vidit-bhat added a commit to vidit-bhat/cockroach that referenced this issue May 14, 2024
@vidit-bhat
roachtest: prefix all logs with worker tags
cba5427 
Previously, many of the runner logs didn't have the worker tags.
This was inadequate because we couldn't break up the logs by wX
tags to see what each worker is doing.
This PR prefixes the worker tags to logs that were missing them.

Epic: none
Fixes: cockroachdb#114045
Release note: None
craig bot pushed a commit that referenced this issue May 14, 2024
@souravcrl @vidit-bhat
Merge #123697 #124027
40fdf01 
123697: ccl,sql,util: Fix jwt auth and add sensitive error logs r=souravcrl a=souravcrl

ccl,sql,util: Fix jwt auth and add sensitive error logs

We are running into issues with jwt authentication and currently unable to
provide support as we are not logging the error from the http client used in the
authenticator. The PR looks to propagate this obtained error from
`ValidateJWTLogin` http client. The http client now also respects the system
http proxy if set.

Validated the error details when presenting an expired token
```
ERROR: JWT authentication: invalid token
SQLSTATE: 28000
DETAIL: unable to parse token: exp not satisfied Failed running "sql"
```

Validated error on setting wrong proxy params
```
ERROR: JWT authentication: unable to validate token
SQLSTATE: 28000
Failed running "sql"
```
and logged error:
```
I240510 08:31:28.604141 1473 4@util/log/event_log.go:32 ⋮
[T1,Vsystem,n1,client=127.0.0.1:56289,hostssl,user=‹sourav.sarangi›] 3
={"Timestamp":1715329888604122000,"EventType":"client_authentication_failed","InstanceID":1,"Network":"tcp","RemoteAddress":"‹127.0.0.1:56289›","SessionID":"17ce136f2a8ecd480000000000000001","Transport":"hostssl","User":"‹sourav.sarangi›","SystemIdentity":"‹sourav.sarangi›","Reason":"CREDENTIALS_INVALID","Detail":"JWT
authentication: unable to validate token\nunable to fetch jwks: Get
\"https://accounts.google.com/.well-known/openid-configuration\": proxyconnect
tcp: dial tcp [::1]:3129: connect: connection refused","Method":"jwt_token"}
```

Verified access logs after setting up squid proxy and passing env HTTP_PROXY and
HTTPS_PROXY params
```
1715103871.761    144 ::1 TCP_TUNNEL/200 5708 CONNECT accounts.google.com:443 -
HIER_DIRECT/74.125.200.84 -
1715103871.836     73 ::1 TCP_TUNNEL/200 5964 CONNECT www.googleapis.com:443 -
HIER_DIRECT/142.250.182.10 -
```

fixes #123575, CRDB-38386, CRDB-38408
Epic None

Release note: Noneccl,sql,util: Fix jwt auth and add sensitive error logs

We are running into issues with jwt authentication and currently unable to
provide support as we are not logging the error from the http client used in the
authenticator. The PR looks to propagate this obtained error from
`ValidateJWTLogin` http client. The http client now also respects the system
http proxy if set.

Validated the error details when presenting an expired token
```
ERROR: JWT authentication: invalid token
SQLSTATE: 28000
DETAIL: unable to parse token: exp not satisfied Failed running "sql"
```

Validated error on setting wrong proxy params
```
ERROR: JWT authentication: unable to validate token
SQLSTATE: 28000
Failed running "sql"
```
and logged error:
```
I240510 08:31:28.604141 1473 4@util/log/event_log.go:32 ⋮
[T1,Vsystem,n1,client=127.0.0.1:56289,hostssl,user=‹sourav.sarangi›] 3
={"Timestamp":1715329888604122000,"EventType":"client_authentication_failed","InstanceID":1,"Network":"tcp","RemoteAddress":"‹127.0.0.1:56289›","SessionID":"17ce136f2a8ecd480000000000000001","Transport":"hostssl","User":"‹sourav.sarangi›","SystemIdentity":"‹sourav.sarangi›","Reason":"CREDENTIALS_INVALID","Detail":"JWT
authentication: unable to validate token\nunable to fetch jwks: Get
\"https://accounts.google.com/.well-known/openid-configuration\": proxyconnect
tcp: dial tcp [::1]:3129: connect: connection refused","Method":"jwt_token"}
```

Verified access logs after setting up squid proxy and passing env HTTP_PROXY and
HTTPS_PROXY params
```
1715103871.761    144 ::1 TCP_TUNNEL/200 5708 CONNECT accounts.google.com:443 -
HIER_DIRECT/74.125.200.84 -
1715103871.836     73 ::1 TCP_TUNNEL/200 5964 CONNECT www.googleapis.com:443 -
HIER_DIRECT/142.250.182.10 -
```

fixes #123575, CRDB-38386, CRDB-38408
Epic None

Release note: None

124027: roachtest: prefix all logs with worker tags r=vidit-bhat a=vidit-bhat

Previously, many of the runner logs didn't have the worker tags. This was inadequate because we couldn't break up the logs by `wX` tags to see what each worker was doing.
This PR prefixes worker tags to the logs that were missing them.

Epic: none
Fixes: #114045
Release note: None

Co-authored-by: Sourav Sarangi <sourav.sarangi@cockroachlabs.com>
Co-authored-by: Vidit Bhat <vidit.bhat@cockroachlabs.com>
craig bot pushed a commit that referenced this issue May 14, 2024
@vidit-bhat
Merge #124027
1e9eabb 
124027: roachtest: prefix all logs with worker tags r=vidit-bhat a=vidit-bhat

Previously, many of the runner logs didn't have the worker tags. This was inadequate because we couldn't break up the logs by `wX` tags to see what each worker was doing.
This PR prefixes worker tags to the logs that were missing them.

Epic: none
Fixes: #114045
Release note: None

Co-authored-by: Vidit Bhat <vidit.bhat@cockroachlabs.com>
@craig craig bot closed this as completed in 0a2f34e May 14, 2024
Test Engineering automation moved this from Triage to Done May 14, 2024
vidit-bhat added a commit to vidit-bhat/cockroach that referenced this issue May 14, 2024
@vidit-bhat
roachtest: prefix all logs with worker tags
7733e3a 
Previously, many of the runner logs didn't have the worker tags.
This was inadequate because we couldn't break up the logs by wX
tags to see what each worker is doing.
This PR prefixes the worker tags to logs that were missing them.

Epic: none
Fixes: cockroachdb#114045
Release note: None
@vidit-bhat vidit-bhat mentioned this issue May 14, 2024
Merged
vidit-bhat added a commit to vidit-bhat/cockroach that referenced this issue May 14, 2024
@vidit-bhat
roachtest: prefix all logs with worker tags
24798d4 
Previously, many of the runner logs didn't have the worker tags.
This was inadequate because we couldn't break up the logs by wX
tags to see what each worker is doing.
This PR prefixes the worker tags to logs that were missing them.

Epic: none
Fixes: cockroachdb#114045
Release note: None
@vidit-bhat vidit-bhat mentioned this issue May 14, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) E-quick-win Likely to be a quick win for someone experienced. T-testeng TestEng Team
Projects
Test Engineering
Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

roachtest: prefix all logs with worker tags vidit-bhat/cockroach
2 participants
@srosenberg @RaduBerinde