roachtest: prefix all logs with worker tags

[vidit-bhat]

Previously, many of the runner logs didn't have the worker tags. This was inadequate because we couldn't break up the logs by wX tags to see what each worker was doing.
This PR prefixes worker tags to the logs that were missing them.

Epic: none
Fixes: #114045
Release note: None

[cockroach-teamcity]

This change is 

[renatolabs]

This LGTM, and I'm fine as-is, but now that I've taken a closer look, I'm starting to wonder whether we should change our approach.

For instance, looking at the test runner logs from a recent run [1], I see a couple of other entries that don't have the worker tag and would continue to not have that tag even after this PR:

06:06:00 cluster.go:1745: destroying cluster teamcity-15224239-1715492932-14-n9cpu4-geo [tag:] (9 nodes)...
06:06:09 roachprod.go:1421: Destroying cluster teamcity-15224239-1715492932-14-n9cpu4-geo with 9 nodes
[..]
teamcity-15224239-1715492932-17-n1cpu4: stopping and waiting
teamcity-15224239-1715492932-17-n1cpu4: wiping

The main difficulty is that the [w%d] annotation is added via calls to l.PrintfCtx. This means that for it to work properly, two things need to happen: every log call needs to use PrintfCtx instead of Printf; and we need to make sure that the context passed is the context with the worker tag. Both of these are hard to enforce, especially over time.

That said, I'm wondering if we should make this easier for ourselves by not relying on log tags to add the worker annotation. Specifically, we could use the ChildLogger function to create a child logger for each worker. So the final result would be something like

_runner-logs/
    test-runner/
        test_runner.log
        w1.log
        w2.log
        ...
        wN.log

With this approach, test_runner.log would continue to include the combined logs of every worker, while one is able to verify the output of a single worker by checking the corresponding wN.log file. This is similar to what the mixedversion framework does in order for each step to log to its own file [2].

[1] https://teamcity.cockroachdb.com/repository/download/Cockroach_Nightlies_RoachtestNightlyGceBazel/15224239:id/_runner-logs/test_runner-1715492932.log

[2]

cockroach/pkg/cmd/roachtest/roachtestutil/mixedversion/mixedversion.go

Lines 782 to 789 in 2f739e8

	// prefixedLogger returns a logger instance off of the given `l`
	// parameter, and adds a prefix to everything logged by the retured
	// logger.
	func prefixedLogger(l *logger.Logger, prefix string) (*logger.Logger, error) {
	fileName := strings.ReplaceAll(prefix, " ", "-")
	formattedPrefix := fmt.Sprintf("[%s] ", fileName)
	return l.ChildLogger(fileName, logger.LogPrefix(formattedPrefix))
	}

[vidit-bhat]

Tested out the changes. The worker log files are being created parallel to the test_runner.log inside the _runner-logs directory.

Also, the entries you mentioned now do have a tag.

[w1] 16:40:14 cluster.go:1745: destroying cluster viditbhat-1715618211-02-n4cpu4 [tag:] (4 nodes)...
[w1] 16:40:19 roachprod.go:1421: Destroying cluster viditbhat-1715618211-02-n4cpu4 with 4 nodes

[DarrylWong]

Nice! I played around with it locally and everything works for me. One tiny question but otherwise LGTM.

[renatolabs]

Nice! I think we should not return on the worker and fallback to the parent logger as already mentioned, but otherwise LGTM.

[vidit-bhat]

bors r+

[craig]

This PR was included in a batch that was canceled, it will be automatically retried

[craig]

Build succeeded:

• windows_build
• examples_orms
• local_roachtest
• acceptance
• macos_amd64_build
• macos_arm64_build
• check_generated_code
• linux_amd64_fips_build
• linux_arm64_build
• lint
• linux_amd64_build
• local_roachtest_fips
• docker_image_amd64
• unit_tests

[vidit-bhat]

blathers backport 24.1 23.2

[blathers-crl]

Encountered an error creating backports. Some common things that can go wrong:

1. The backport branch might have already existed.
2. There was a merge conflict.
3. The backport branch contained merge commits.

You might need to create your backport manually using the backport tool.

---

error creating merge commit from 0a2f34e to blathers/backport-release-24.1-124027: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 24.1 failed. See errors above.

---

error creating merge commit from 0a2f34e to blathers/backport-release-23.2-124027: POST https://api.github.com/repos/cockroachdb/cockroach/merges: 409 Merge conflict []

you may need to manually resolve merge conflicts with the backport tool.

Backport to branch 23.2 failed. See errors above.

---

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}