dyno: Initial resolution of `zip()` expressions and parallel iterators #24915

dlongnecke-cray · 2024-04-23T22:12:47Z

This PR introduces resolution for zip() expressions as well as resolution of parallel iterators for forall and [] loops.

For zip() expressions that appear as the iterand of a serial loop, the strategy is to resolve only serial iterators for each actual in the zip().

For zip() expressions that appear as the iterand of a forall loop, the strategy is to:

For the first actual (known as the leader), attempt to resolve a suitable leader iterator or error
For the leader and all remaining actuals (known as followers), attempt to resolve a suitable follower iterator or error

For [] loops, the strategy is similar, except serial iterators may be used as a substitute for the leader and followers IFF the leader iterator could not be resolved for the first actual. If the leader iterator could be resolved, but e.g., its return type could not, then serial iterators will not be considered as fallbacks.

For iterands that are not zip() expressions, the "standalone" parallel iterator is preferred for parallel loops before attempting to resolve a leader/follower combo. As with zip(), forall loops will emit an error if no form of parallel iterator could be resolved. All other loops will fall back to serial iterators.

Thanks to @vasslitvinov for walking me through the semantics of zip() and parallel iterator resolution.

FUTURE WORK

Iterator forwarding for iterators in the internal modules (e.g., tag=tag, followThis=followThis)
Expand tests, handle edge cases, attempt to resolve leader/follower/standalone iterators for standard/internal types

Reviewed by @DanilaFe. Thanks!

Signed-off-by: David Longnecker <dlongnecke-cray@users.noreply.github.com>

dlongnecke-cray · 2024-04-23T22:50:21Z

I'll fix the CI check failures tomorrow, have to head out for the night.

DanilaFe

I'm halfway through, a few comments, mostly minor changes within individual lines.

DanilaFe · 2024-04-23T22:17:42Z

frontend/include/chpl/types/EnumType.h

+  /** Given an enum type 'et', get a map from the name of each constant
+      in 'et' to each constant represented as a param value.
+      If there are multiple enum constants with the same name (which
+      means the AST is semantically incorrect), then only the first
+      constant is added to the map. */
+  static const std::map<UniqueString, QualifiedType>*
+  getParamConstantsMapOrNull(Context* context, const EnumType* et);


Before I read the implementation, my question is: why would it be nullable?

Answering my own question, probably if we built without the standard modules or something.

I just like the property of returning nullptr if the AST for the thing could not be found.

DanilaFe · 2024-04-23T22:20:01Z

frontend/lib/resolution/Resolver.cpp

+  if (!p || !p->isIndexableLoop()) {
+    context->error(zip, "zip expressions may only appear as a loop's iterand");
+  }


I think you can make this an assertion because we simply can't parse zip any other way.

DanilaFe · 2024-04-23T22:24:38Z

frontend/lib/resolution/Resolver.cpp

+    auto actual = zip->actual(i);
+
+    if (actual->isZip()) {
+      context->error(actual, "nested 'zip' expressions are not supported");


Ditto here. According to the grammar, 'zip' can only occur after foreach or in (see zippered_iterator in chpl.ypp). So you can remove this check.

DanilaFe · 2024-04-23T22:40:46Z

frontend/lib/resolution/Resolver.cpp

+    // Try to resolve the parallel leader using the first actual.
+    if (isFirstActual && prefersParallel) {
+      const bool emitErrors = requiresParallel;
+      leaderType = resolveParallelLeaderIterType(*this, actual, actual,


I think I'd rather have resolveParallelLeaderIterType return a boolean for whether or not the call resolution works, and separately check leaderType for errors. The reason for that is the following: if the user has a leader iterator that matches, but we can't infer its return type. The user also has follower iterators that work, and a serial iterator that's specifically disabled via compilerError or something. Since we know there exists a leader iterator, we should skip resolving the serial loops (via the "RFINOS" principle, or "resolution error is not overload selection"). On the other hand, depending on whether we could find a valid leader iterator, we should try or give up on resolving followers.

I'm going to change the structure of the underlying implementation so that the resolved iterator can be gotten through an optional out formal. Then you can see inspect it to see the details you're describing.

DanilaFe · 2024-04-23T23:03:48Z

frontend/lib/resolution/Resolver.cpp

-                MSC.numBest() == 1 &&
-                MSC.only().fn()->untyped()->kind() == uast::Function::Kind::ITER;
+  auto iterKindFormal = getIterKindConstantOrUnknown(rv, iterKindStr);
+  bool needFollower = iterKindStr ? !strcmp(iterKindStr, "follower") : false;


This tells me you should use std::string or UniqueString. Probably the latter.

What's wrong with a strcmp?! Lol, back in my day...

DanilaFe · 2024-04-23T23:07:06Z

frontend/lib/resolution/Resolver.cpp

+  if (iterand->isCall()) {
+    auto runResult = context->runAndTrackErrors([&](Context* context) {
+      iterand->traverse(rv);
+      return true;
+    });
+    std::swap(runResult.errors(), traversalErrors);
+  } else {
+    iterand->traverse(rv);
+  }


Not sure why "is call" is the line between speculative and non-speculative resolution -- would you mind giving your rationale?

My rationale is, what "call-like" things will actually resolve to an iterator call? I am not even sure if we allow parenless methods to resolve to iterator calls anymore (if it does then sure we can always introspect).

Hmm, what if it's a call-like thing that returns, say, an array? like getMyArray()? I think it being a call expression is not a particularly useful metric. I suspect we never want to silence errors except when we're trying to invoke the 'leader' iterator on a value, which i believe is handled below.

Ok, I can go ahead and just always run the speculation for now.

DanilaFe · 2024-04-23T23:08:20Z

frontend/lib/resolution/Resolver.cpp


-  QualifiedType idxType;
+  // Inspect the resolution result to determine what should be done next.
+  ResolvedExpression& iterandRE = rv.byPostorder.byAst(iterand);


consider auto& here, too.

DanilaFe · 2024-04-23T23:09:18Z

frontend/lib/resolution/Resolver.cpp

+  // Inspect the resolution result to determine what should be done next.
+  ResolvedExpression& iterandRE = rv.byPostorder.byAst(iterand);
+  auto& MSC = iterandRE.mostSpecific();
+  auto fn = !MSC.isEmpty() && MSC.numBest() == 1 ? MSC.only().fn() : nullptr;


I believe you can just use MostSpecificCandidate's operator bool here and write this as

auto fn = MSC.only() ? MSC.only().fn() : nullptr;

That's because .only() returns an empty candidate (without runtime overhead if that worries you) if there are more or less than 1 candidates.

DanilaFe · 2024-04-23T23:11:29Z

frontend/lib/resolution/Resolver.cpp

+    if (!isIter && e->type() == NoMatchingCandidates) {
+      auto nmc = static_cast<ErrorNoMatchingCandidates*>(e.get());
+      if (std::get<0>(nmc->info()) == iterand) {
+        std::swap(noCandidatesError, e);
+        continue;
+      }
+    }


Woah, a use of error introspection. I love it.

dlongnecke-cray · 2024-04-24T21:18:59Z

Adding isSerialIterator isLeaderIterator isFollowerIterator and isStandaloneIterator as per your suggestion.

Signed-off-by: David Longnecker <dlongnecke-cray@users.noreply.github.com>

dlongnecke-cray added 5 commits April 18, 2024 16:49

Begin effort of resolving 'zip()' and parallel iterators

5cd1f05

Signed-off-by: David Longnecker <dlongnecke-cray@users.noreply.github.com>

Stabilize zip() and parallel iterator resolution, all tests passing

01c40b4

Signed-off-by: David Longnecker <dlongnecke-cray@users.noreply.github.com>

Add tests for parallel standlone iterators

1cd240e

Signed-off-by: David Longnecker <dlongnecke-cray@users.noreply.github.com>

Avoid speculation if the iterand does not look like a call

c883bcb

Signed-off-by: David Longnecker <dlongnecke-cray@users.noreply.github.com>

Remove comments, adjust formatting, add factory for tests

c84f6c9

Signed-off-by: David Longnecker <dlongnecke-cray@users.noreply.github.com>

DanilaFe reviewed Apr 24, 2024

View reviewed changes

dlongnecke-cray added 3 commits April 24, 2024 14:39

Attempt to fix failing CI checks (1)

b8f0ba3

Signed-off-by: David Longnecker <dlongnecke-cray@users.noreply.github.com>

Respond to reviewer feedback (1)

1e9cfc0

Signed-off-by: David Longnecker <dlongnecke-cray@users.noreply.github.com>

Always speculatively resolve, add TODO about removing speculation

92a3803

Signed-off-by: David Longnecker <dlongnecke-cray@users.noreply.github.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dyno: Initial resolution of `zip()` expressions and parallel iterators #24915

dyno: Initial resolution of `zip()` expressions and parallel iterators #24915

dlongnecke-cray commented Apr 23, 2024 •

edited

dlongnecke-cray commented Apr 23, 2024

DanilaFe left a comment

DanilaFe Apr 23, 2024

DanilaFe Apr 23, 2024

dlongnecke-cray Apr 24, 2024

DanilaFe Apr 23, 2024

DanilaFe Apr 23, 2024

DanilaFe Apr 23, 2024

dlongnecke-cray Apr 24, 2024

DanilaFe Apr 23, 2024

dlongnecke-cray Apr 24, 2024

DanilaFe Apr 23, 2024

dlongnecke-cray Apr 24, 2024

DanilaFe Apr 26, 2024

dlongnecke-cray Apr 26, 2024

DanilaFe Apr 23, 2024

DanilaFe Apr 23, 2024

DanilaFe Apr 23, 2024

dlongnecke-cray commented Apr 24, 2024

dyno: Initial resolution of zip() expressions and parallel iterators #24915

Are you sure you want to change the base?

dyno: Initial resolution of zip() expressions and parallel iterators #24915

Conversation

dlongnecke-cray commented Apr 23, 2024 • edited

dlongnecke-cray commented Apr 23, 2024

DanilaFe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dlongnecke-cray commented Apr 24, 2024

dyno: Initial resolution of `zip()` expressions and parallel iterators #24915

dyno: Initial resolution of `zip()` expressions and parallel iterators #24915

dlongnecke-cray commented Apr 23, 2024 •

edited