executor: add hash join v2 #53208

windtalker · 2024-05-13T02:53:35Z

What problem does this PR solve?

Issue Number: ref #53127

Problem Summary:

What changed and how does it work?

Design doc: #53196
Currently, only inner join and left outer join is supported.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

None

tiprow · 2024-05-13T02:53:50Z

Hi @windtalker. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

hawkingrei · 2024-05-13T02:57:58Z

pkg/executor/join/bench_test.go

+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package join


Nothing? Can we remove it?

Will add a basic test

codecov · 2024-05-13T03:07:29Z

Codecov Report

Attention: Patch coverage is 8.68317% with 2545 lines in your changes are missing coverage. Please review.

Project coverage is 48.8141%. Comparing base (0afe54d) to head (61cc71c).

Additional details and impacted files

@@                Coverage Diff                @@
##             master     #53208         +/-   ##
=================================================
- Coverage   72.5179%   48.8141%   -23.7038%     
=================================================
  Files          1505       1423         -82     
  Lines        429866     409004      -20862     
=================================================
- Hits         311730     199652     -112078     
- Misses        98820     191337      +92517     
+ Partials      19316      18015       -1301

Flag	Coverage Δ
integration	`48.8141% <8.6831%> (?)`
tiprow_ft	`?`
unit	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`∅ <ø> (∅)`
parser	`∅ <ø> (∅)`
br	`24.0471% <ø> (-17.3263%)`	⬇️

hawkingrei · 2024-05-13T07:47:24Z

/ok-to-test

hawkingrei · 2024-05-13T07:48:11Z

pkg/executor/builder.go

+		BuildWorkers:          make([]*join.BuildWorkerV2, v.Concurrency),
+		HashJoinCtxV2: &join.HashJoinCtxV2{
+			OtherCondition:  v.OtherConditions,
+			PartitionNumber: mathutil.Min(int(v.Concurrency), 16),


Suggested change

PartitionNumber: mathutil.Min(int(v.Concurrency), 16),

PartitionNumber: min(int(v.Concurrency), 16),

windtalker · 2024-05-16T08:06:00Z

/test check-dev2

tiprow · 2024-05-16T08:06:23Z

@windtalker: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test fast_test_tiprow
/test tidb_parser_test

Use /test all to run all jobs.

In response to this:

/test check-dev2

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ti-chi-bot · 2024-05-17T02:04:30Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign fixdb for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS
pkg/planner/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

windtalker · 2024-05-17T09:35:14Z

/test unit-test

tiprow · 2024-05-17T09:35:36Z

@windtalker: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test fast_test_tiprow
/test tidb_parser_test

Use /test all to run all jobs.

In response to this:

/test unit-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

Signed-off-by: xufei <xufei@pingcap.com>

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

This reverts commit 4782340.

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

tiprow · 2024-05-22T05:44:16Z

@windtalker: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
fast_test_tiprow	`61cc71c`	link	true	`/test fast_test_tiprow`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

ti-chi-bot · 2024-05-22T06:06:58Z

@windtalker: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
idc-jenkins-ci-tidb/unit-test	`61cc71c`	link	true	`/test unit-test`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

xzhangxian1008 · 2024-05-22T06:28:31Z

pkg/executor/join/hash_join_v2.go

+// step 1. fetch data from build side child and build a hash table;
+// step 2. fetch data from probe child in a background goroutine and probe the hash table in multiple join workers.
+func (e *HashJoinV2Exec) Next(ctx context.Context, req *chunk.Chunk) (err error) {
+	if !e.prepared {


Use atomic type for e.prepared? As Close may be called at the same time.

ti-chi-bot bot added do-not-merge/invalid-title release-note-none size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. sig/planner SIG: Planner labels May 13, 2024

windtalker changed the title ~~Hash join v2 unsafe pointer~~ executor: add hash join v2 May 13, 2024

ti-chi-bot bot removed the do-not-merge/invalid-title label May 13, 2024

windtalker mentioned this pull request May 13, 2024

[WIP] executor: add hash join v2 #53128

Closed

13 tasks

hawkingrei reviewed May 13, 2024

View reviewed changes

windtalker added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 13, 2024

ti-chi-bot bot added the ok-to-test label May 13, 2024

hawkingrei reviewed May 13, 2024

View reviewed changes

windtalker force-pushed the hash_join_v2_unsafe_pointer branch from 811ae7b to b8de444 Compare May 15, 2024 02:21

windtalker changed the title ~~executor: add hash join v2~~ executor: add hash join v2 | tidb-test=pr/2332 May 15, 2024

ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 15, 2024

windtalker force-pushed the hash_join_v2_unsafe_pointer branch from 668ff10 to 6f78742 Compare May 16, 2024 01:47

windtalker force-pushed the hash_join_v2_unsafe_pointer branch from 207cd7a to 43644ee Compare May 17, 2024 02:04

windtalker force-pushed the hash_join_v2_unsafe_pointer branch from 43644ee to 150fdb7 Compare May 17, 2024 06:41

windtalker force-pushed the hash_join_v2_unsafe_pointer branch from 4403be5 to 3d78136 Compare May 20, 2024 07:07

windtalker changed the title ~~executor: add hash join v2 | tidb-test=pr/2332~~ executor: add hash join v2 May 22, 2024

windtalker and others added 3 commits May 22, 2024 13:15

save work

5ef2f1a

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

save work

b00dd3a

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

save work

9983680

Signed-off-by: xufei <xufei@pingcap.com>

windtalker added 26 commits May 22, 2024 13:15

remove nolint:all

23d635f

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

remove useless variable

22273dd

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

refine code to avoid use unsafe.Pointer too much

ab27816

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

run ci test when hash join v2 enable by default

b032776

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

fix bug

911cca0

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

fix bug of enum join int

266d04e

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

fix ci

cb69aec

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

fix corner case that int join enum/bit

e62dfb2

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

fix minor bug

ab47208

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

make test result stable

e6b069c

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

update result under for hash join v2

e24ceee

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

fix ut

e01a877

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

update result under for hash join v2

f06d437

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

use hash join v1 by default

4fb6c37

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

fix goroutine leak

c37557a

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

use hash join v2 by default for test

b73f334

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

refine code

8a36ad2

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

fix test build

fd84507

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

fix ci

80ddbb2

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

fix test build

74073b4

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

address comments

53bb808

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

Revert "use hash join v2 by default for test"

2f22d2e

This reverts commit 4782340.

remove tidb variable and add a temp variable for test

f55604d

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

fix build

01d4d7f

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

avoid data race

4014a25

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

add simple bench tests

61cc71c

Signed-off-by: xufei <xufeixw@mail.ustc.edu.cn>

windtalker force-pushed the hash_join_v2_unsafe_pointer branch from 3d78136 to 61cc71c Compare May 22, 2024 05:41

xzhangxian1008 reviewed May 22, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

executor: add hash join v2 #53208

executor: add hash join v2 #53208

windtalker commented May 13, 2024

tiprow bot commented May 13, 2024

hawkingrei May 13, 2024

windtalker May 13, 2024

codecov bot commented May 13, 2024 •

edited

hawkingrei commented May 13, 2024

hawkingrei May 13, 2024

windtalker May 17, 2024

windtalker commented May 16, 2024

tiprow bot commented May 16, 2024

ti-chi-bot bot commented May 17, 2024

windtalker commented May 17, 2024

tiprow bot commented May 17, 2024

tiprow bot commented May 22, 2024

ti-chi-bot bot commented May 22, 2024

xzhangxian1008 May 22, 2024

	PartitionNumber: mathutil.Min(int(v.Concurrency), 16),
	PartitionNumber: min(int(v.Concurrency), 16),

executor: add hash join v2 #53208

Are you sure you want to change the base?

executor: add hash join v2 #53208

Conversation

windtalker commented May 13, 2024

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

tiprow bot commented May 13, 2024

hawkingrei May 13, 2024

Choose a reason for hiding this comment

windtalker May 13, 2024

Choose a reason for hiding this comment

codecov bot commented May 13, 2024 • edited

Codecov Report

hawkingrei commented May 13, 2024

hawkingrei May 13, 2024

Choose a reason for hiding this comment

windtalker May 17, 2024

Choose a reason for hiding this comment

windtalker commented May 16, 2024

tiprow bot commented May 16, 2024

ti-chi-bot bot commented May 17, 2024

windtalker commented May 17, 2024

tiprow bot commented May 17, 2024

tiprow bot commented May 22, 2024

ti-chi-bot bot commented May 22, 2024

xzhangxian1008 May 22, 2024

Choose a reason for hiding this comment

codecov bot commented May 13, 2024 •

edited