[SPARK-48270][SQL] Improve the OptimizeRepartition with Repartition

[beliefer]

What changes were proposed in this pull request?

This PR propose to improve the OptimizeRepartition with Repartition.

Why are the changes needed?

Repartition is cheaper than RepartitionByExpression.

Does this PR introduce any user-facing change?

'No'.

How was this patch tested?

GA

Was this patch authored or co-authored using generative AI tooling?

'No'.

[beliefer]

ping @cloud-fan

[cloud-fan]

Why is Repartition cheaper than RepartitionByExpression?

[beliefer]

Why is RepartitionByExpression cheaper than Repartition?

Repartition is cheaper than RepartitionByExpression.

[beliefer]

For example:

  private def canEliminateGlobalSort(plan: LogicalPlan): Boolean = plan match {
    case r: RepartitionByExpression => r.partitionExpressions.forall(_.deterministic)
    case r: RebalancePartitions => r.partitionExpressions.forall(_.deterministic)
    case _: Repartition => true
    case _ => false
  }

  protected def userSpecifiedRepartition(p: LogicalPlan): Boolean = p match {
    case _: Repartition => true
    case r: RepartitionByExpression
      if r.optNumPartitions.isDefined || r.partitionExpressions.nonEmpty => true
    case _ => false
  }

      case r: logical.RepartitionByExpression =>
        val shuffleOrigin = if (r.partitionExpressions.isEmpty && r.optNumPartitions.isEmpty) {
          REBALANCE_PARTITIONS_BY_NONE
        } else if (r.optNumPartitions.isEmpty) {
          REPARTITION_BY_COL
        } else {
          REPARTITION_BY_NUM
        }
        exchange.ShuffleExchangeExec(
          r.partitioning, planLater(r.child),
          shuffleOrigin, r.optAdvisoryPartitionSize) :: Nil
      case r @ logical.Repartition(numPartitions, shuffle, child) =>
        if (shuffle) {
          ShuffleExchangeExec(r.partitioning, planLater(child), REPARTITION_BY_NUM) :: Nil
        } else {
          execution.CoalesceExec(numPartitions, planLater(child)) :: Nil
        }

[cloud-fan]

For the places you mentioned, RepartitionByExpression with constant is the same as Repartition?