Fix Blackhole implementation for e2e tests #17985
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Hi @henrybear327. Thanks for your PR. I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
henrybear327
force-pushed
the
fix/e2e_blackhole_forward_proxy_per_peer_review
branch
from
0e061d2
to
a714d40
Compare
henrybear327
force-pushed
the
fix/e2e_blackhole_forward_proxy_per_peer_review
branch
2 times, most recently
from
1f1cd51
to
01eab3b
Compare
fuweid
reviewed
May 13, 2024
Signed-off-by: Siyuan Zhang <sizhang@google.com>
Shorten the wait time for the open connection to expire to 5s Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Based on Fu Wei's idea discussed in the [issue](etcd-io#17737), we employ the blocking on L7 but without using external tools. [Background] A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Proposed solution] We introduce an forward proxy for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's forward proxy ----- B's reverse proxy - B ^ newly introduced ^ in the original codebase (renamed) ``` By adding this forward proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. [Implementation] The main subtasks are - set up an environment variable `E2E_TEST_FORWARD_PROXY_IP` - implement forward proxy by extending the existing proxy server code - implement enable/disable of the forward proxy in the e2e test The result is that for every peer, we will have the arch like this ``` A -- A's forward proxy (connections initiated from A will be forwarded from this proxy) | ^ covers case (b) | --- A's (currently existing) reverse proxy (advertised to other peers where the connection should come in from) ^ covers case (a) ``` [Testing] - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` [Issues] - I run into `context deadline exceeded` sometimes ``` etcd_mix_versions_test.go:175: Error Trace: /Users/henrybear327/go/src/etcd/tests/e2e/etcd_mix_versions_test.go:175 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:75 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:31 Error: Received unexpected error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE Test: TestBlackholeByMockingPartitionLeader Messages: failed to put "key-0", error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"v3@v3.6.0-alpha.0/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE ``` [References] [1] issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com> Co-authored-by: Iván Valdés Castillo <iv@nvald.es>
Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
henrybear327
force-pushed
the
fix/e2e_blackhole_forward_proxy_per_peer_review
branch
from
01eab3b
to
51f3475
Compare
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 15, 2024
Very ugly PoC for now Reference: - etcd-io#17985 (comment)
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 15, 2024
Very ugly PoC for now Reference: - etcd-io#17985 (comment) Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 16, 2024
Due to forward proxy's need to parse the CONNECT header, which is a L7 layer feature, thus we are splitting the proxy into 2 types, for better maintainability. Reference: - etcd-io#17985 (comment) Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 16, 2024
Due to forward proxy's need to parse the CONNECT header, which is a L7 layer feature, thus we are splitting the proxy into 2 types, for better maintainability. Reference: - etcd-io#17985 (comment) Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 16, 2024
Due to forward proxy's need to parse the CONNECT header, which is a L7 layer feature, thus we are splitting the proxy into 2 types, for better maintainability. Reference: - etcd-io#17985 (comment) Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 18, 2024
Due to forward proxy's need to parse the CONNECT header, which is a L7 layer feature, thus we are splitting the proxy into 2 types, for better maintainability. Reference: - etcd-io#17985 (comment) Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
henrybear327
added a commit
to henrybear327/etcd
that referenced
this pull request
May 21, 2024
Due to forward proxy's need to parse the CONNECT header, which is a L7 layer feature, thus we are splitting the proxy into 2 types, for better maintainability. Reference: - etcd-io#17985 (comment) Signed-off-by: Chun-Hung Tseng <henrybear327@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.
For each proxy, we remove the reverse proxy and keep only the forward proxy.
PoC-level code quality for now...