Code coverage tooling

[svix-dylan]

No description provided.

[tasn]

Why are you doing this in a subscript like this?

[svix-dylan]

So that these variables are only set in this new session, and don't affect rustc in any subsequent commands a developer might run unrelated to this script.

[jaymell]

Could you do this:

RUSTFLAGS="-Cinstrument-coverage" \
LLVM_PROFILE_FILE="svix-webhooks-%p-%m.profraw" \
RUSTC_BOOTSTRAP=1 \
./run-tests.sh

...

[tasn]

You don't need to do anything, it won't affect anything outside of the script.
Also, I just noticed you didn't run export so actually these won't even work at all.

[jaymell]

Is the -e needed here?

[tasn]

Suggested change

	bash -c "
	# tell Rust to run with coverage instrumentation
	RUSTFLAGS=\"-Cinstrument-coverage\"
	# give grcov a profile name template for output files
	LLVM_PROFILE_FILE=\"svix-webhooks-%p-%m.profraw\"
	# put the compiler in nightly mode
	RUSTC_BOOTSTRAP=1
	# run tests
	./run-tests.sh
	" || true
	# tell Rust to run with coverage instrumentation
	export RUSTFLAGS="-Cinstrument-coverage"
	# give grcov a profile name template for output files
	export LLVM_PROFILE_FILE="svix-webhooks-%p-%m.profraw"
	export TEST_NIGHTLY="+nightly"
	# run tests
	./run-tests.sh

And then in run-tests:

TEST_COMMAND="cargo ${TEST_NIGHTLY} test --all --all-features --all-targets -- --test-threads=1"

[tasn]

Or better yet, just tell people to use nightly (see other comment), and then:

#!/bin/sh -e

# tell Rust to run with coverage instrumentation
export RUSTFLAGS="-Cinstrument-coverage"

# give grcov a profile name template for output files
export DIRNAME="./target/debug/coverage/$(date -u --iso-8601=seconds)"
mkdir -p "$DIRNAME"
export LLVM_PROFILE_FILE="$DIRNAME/svix-webhooks-%p-%m.profraw"

echo "Collecting coverage in $DIRNAME"

# run tests
./run-tests.sh

# generate and open report output
echo "Generating coverage report (may take time)."
~/.cargo/bin/grcov "$DIRNAME" -s svix-server --binary-path ./target/debug/ -t html --branch --ignore-not-existing -o "$DIRNAME"

echo "Coverage HTML file at:\n$(pwd)/$DIRNAME/index.html"

[tasn]

Let's not worry about it for now, just dumped my thoughts.

[tasn]

This won't work if default is not set to nightly. Probably just need to say "use nightly" which will also simplify the rest.

[svix-dylan]

I ran both of these install commands on stable successfully

[tasn]

Yeah, they install fine, but I had issues. Though maybe my issues were unrelated.

[tasn]

Or better yet, just tell people to use nightly (see other comment), and then:

#!/bin/sh -e

# tell Rust to run with coverage instrumentation
export RUSTFLAGS="-Cinstrument-coverage"

# give grcov a profile name template for output files
export DIRNAME="./target/debug/coverage/$(date -u --iso-8601=seconds)"
mkdir -p "$DIRNAME"
export LLVM_PROFILE_FILE="$DIRNAME/svix-webhooks-%p-%m.profraw"

echo "Collecting coverage in $DIRNAME"

# run tests
./run-tests.sh

# generate and open report output
echo "Generating coverage report (may take time)."
~/.cargo/bin/grcov "$DIRNAME" -s svix-server --binary-path ./target/debug/ -t html --branch --ignore-not-existing -o "$DIRNAME"

echo "Coverage HTML file at:\n$(pwd)/$DIRNAME/index.html"

[svix-dylan]

Adding some thoughts here. Apologies about some confusion earlier in this PR, I was playing with multiple strategies and artifacts from both were left in the commits.

grcov uses one of two strategies to generate code coverage: source-based or "gcov"-based coverage (presumably via the gcc utility). Per further research, source-based coverage supports (at least) proc macros, and generally seems to be the preferred approach. This PR has only ever implemented source-based coverage (despite the presence of RUSTC_BOOTSTRAP=1).

Agreed, leaving this for now.