[FLINK-34379][table] Fix adding catalogtable logic #24788
Conversation
| hasAdded = true; | ||
| } | ||
| boolean alreadyExists = false; | ||
| for (ContextResolvedTable table : tables) { |
There was a problem hiding this comment.
Not sure how big the tables could be and how often the method will be called. Does it make sense to use e.g. a Map<K,V> to avoid the loop?
There was a problem hiding this comment.
Good point. But note that tables is of type Set already
There was a problem hiding this comment.
Sorry, I don't get your point. I meant looping the Set might have performance issue.
There was a problem hiding this comment.
I think there are many reasons to use Map instead of Set:
- the logic is point search instead of loop search as I mentioned below.
- O(1) than O(n) for better performance, because the The
DynamicPartitionPruningUtilsclass will be used centrally for batch jobs[1], i.e. for large projects with many tables, it could be a bottleneck. - less code while using e.g. Map.putIfAbsent(K, V)
[1]
There was a problem hiding this comment.
Thanks @jeyhunkarimov, LGTM.
I think we should waiting for @JingGe's +1 also.
jeyhunkarimov
force-pushed
the
FLINK-34379
branch
from
5ceaf0e
to
bdd150d
Compare
jeyhunkarimov
force-pushed
the
FLINK-34379
branch
from
bdd150d
to
01a2aaa
Compare
| hasAdded = true; | ||
| } | ||
| boolean alreadyExists = false; | ||
| for (ContextResolvedTable table : tables) { |
There was a problem hiding this comment.
Sorry, I don't get your point. I meant looping the Set might have performance issue.
| } | ||
| boolean alreadyExists = false; | ||
| for (ContextResolvedTable table : tables) { | ||
| if (table.getIdentifier().equals(catalogTable.getIdentifier())) { |
There was a problem hiding this comment.
This is a typical hash map logic
| hasAdded = true; | ||
| } | ||
| boolean alreadyExists = false; | ||
| for (ContextResolvedTable table : tables) { |
There was a problem hiding this comment.
I think there are many reasons to use Map instead of Set:
- the logic is point search instead of loop search as I mentioned below.
- O(1) than O(n) for better performance, because the The
DynamicPartitionPruningUtilsclass will be used centrally for batch jobs[1], i.e. for large projects with many tables, it could be a bottleneck. - less code while using e.g. Map.putIfAbsent(K, V)
[1]
What is the purpose of the change
(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)
Brief change log
Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:
@Public(Evolving): (no)Documentation