Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spec changes related to specific types. #3302

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Spec changes related to specific types. #3302

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
316 changes: 316 additions & 0 deletions accepted/future-releases/extension-types/feature-specification.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1461,3 +1461,319 @@ wrapper class is so huge that it is likely to be a major use case for
extension types that they can allow us to use a built-in class as the
representation, and still have a specialized interface—that is, an
extension type.

## Specification changes related to specific types
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably be located before the 'Discussion' section, because it contains normative elements.

We might also separate this into normative material (probably rather short) and a background part.

For instance, the paragraph in lines 1467-1468 serves as an introduction to the overall topic ("this is about classes mentioned in the language specification"), but it starts by noting a fact which is already specified (in terms of rules about implements on extension types).


Extension types can implement class types, even class types and their members,
which are mentioned in the language specification.
Comment on lines +1467 to +1468
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is somewhat easy to misunderstand or get confused about. Perhaps:

Suggested change
Extension types can implement class types, even class types and their members,
which are mentioned in the language specification.
Extension types can implement class types, even class types which are
mentioned in the language specification, and even class types that cannot
be implemented by other classes. They can also declare or redeclare
some members with names from the interfaces of such class types.


By disallowing extension members with the same names as the members of `Object`,
we have reduced the problem to where the specification refers to members of
specific interfaces.
The specification needs to be updated to account for the existence of extension
types that are subtypes of those types, and which possibly shadow members from
the interface with extension members.
Comment on lines +1470 to +1475
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
By disallowing extension members with the same names as the members of `Object`,
we have reduced the problem to where the specification refers to members of
specific interfaces.
The specification needs to be updated to account for the existence of extension
types that are subtypes of those types, and which possibly shadow members from
the interface with extension members.
*An extension type instance member cannot have a name whose basename is also the basename of an instance member of `Object`. For other names, the specification needs to take potential extension type members into account in some situations. For instance, an extension type could redeclare `iterator` on a subtype of `Iterable`.*

Still commentary.


The specification, and feature specifications not yet included in the
specification, may uses phrases like (taken from the original collection
elements feature, so no longer direttly relevant):

> 1. Evaluate the iterator expression to a value `sequence`.
> 2. If `sequence` is not an instance of a class that implements `Iterable`,
> throw a dynamic exception.
> 3. Evaluate `sequence.iterator` to a value `iterator`.

Here the “Evaluate `sequence.iterator`” is imprecise, and really means “Invoke
the instance member `iterator` on the object `sequence` (which must exist
because of the second step). However, as *written*, it’s now ambiguous *which*
`iterator` member to invoke, in case the static type of `sequence` is an
extension type which implements `Iterable` and shadows `iterator`.
Comment on lines +1477 to +1490
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All commentary.


**General Rule:** Where the existing language specification can invoke or access
an extension member, it can also invoke an extension type member, and where it
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Presumably, 'access' is needed in order to include tear-offs?:

Suggested change
an extension member, it can also invoke an extension type member, and where it
an extension member, it can also invoke or access an extension type member, and where it

cannot invoke an extension member, it also will not invoke an extension type
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'or access' again, twice?

member.

An example of the former is implicit `call` invocation: If `e1` in `e1(e2)` a
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
An example of the former is implicit `call` invocation: If `e1` in `e1(e2)` a
An example of the former is implicit `call` invocation: If `e1` in `e1(e2)` has a

static type which is not a function type, but which exposes a `call` method, the
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
static type which is not a function type, but which exposes a `call` method, the
static type which is not a function type, but whose interface has a method named `call`, the

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here I actually wanted to include an extension call method. Do we have a word for "static type's interface has a call method, or an call extension methis is avilable and applicable to the type"?

semantics is equivalent to that of `e1.call(e2)`. We apply this rule even the
static type of `e1` itself has no `call` method, but an extension `call` method
applies. If `e1` has an extension-type type, we check whether that has a `call`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In many locations we have already used 'has an extension type' to mean 'has a type which is introduced by an extension type declaration. So:

Suggested change
applies. If `e1` has an extension-type type, we check whether that has a `call`
applies. If `e1` has an extension type, we check whether that type has a `call`

method, and not its representation type. There is no *type* check, only a
*member* check.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commentary, whole paragraph (lines 1497-1503).


The latter case is usually signaled by the specification *requiring* that
something implements a specific interface, and then invoking members of that
interface on the value. We can generally treat those cases as if they first
*up-cast* to the type they tested for, before invoking members, which ensure
that they invoke instance members.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For constant expressions where an int could have static type Object and an extension could provide an operator +, we just say that the invocation must invoke a class instance member (which implies: not an extension instance member). Perhaps that would suffice?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I should probably spell out what I mean by that: We shouldn't try to "magically" find a suitable member when the standard static analysis and standard dynamic semantics of a given expression will invoke something that we don't want. Just like the constant expression, it's simply a compile-time error if the program uses a + 1 in a constant context where a has static type Object and there is an extension + which is applicable, and it isn't enough that we can also know that a evaluates to an int.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

However, I'm not 100% convinced that we always want the class instance member: If the whole point around a given extension type is that it provides a better way to iterate over some elements (say, we can use @JS magic to get a raw JavaScript iteration where we'd otherwise have something more expensive), then perhaps we should allow an extension type member to be used by a for loop?

Copy link
Member Author

@lrhn lrhn Aug 31, 2023 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We generally need to distinguish between places where you must invoke an instance member, and where you may invoke an extension/extension type member.

In the former cases, we can distinguish between always invoking the instance member (which a type check has ensured is there), or refusing to run if the instance member is shadowed by an extension member.

In the latter cases, we may have rules that apply only if the invocation is an instance member invocation.

  • The "must invoke instance member, must not be shadowed" is things like const MyInt(1) + 1, where + denotes an extension type member. That's a compile-time error. This couldn't happen with extension methods, since they wouldn't apply if there was an instance member.

  • The "must, and always does invoke instance member, whether shadowed or not" is, as currently written, something like for (var x in MyIterable()). Here we check that MyIterable implements (or is assignable to) Iterable<T>. Then we invoke the instance get iterator, as if by first casting to Iterable<T> . We don't need to cast, we just make no assumptions about types other than it being an Iterable<T>, for example we don't check the actual type of the Iterator<S> that iterator provides, we just use it as an Iterator<T>. That's how code works today, since extension members wouldn't apply anyway, but extension type members might.

  • The "may invoke extension members, but some rules only apply to instance methods" are rules like the special-ase number type inference, where MyInt(1) + 1, MyInt implements int, has static type int if + denotes an instance method, and whatever the return type of MyInt.+ is if it denotes an extension type method. This couldn't happen with extension methods.

  • And the "may invoke extension members, nothing special at all" includes the "implicit call invocation/tear-off". This is where we'd previously accept extension methods.

If we want to move for (var x in e) into the fourth category (from the second), we can. We need to be very careful about how we do it, so that we don't introduce unsoundness, and also don't introduce changes to how a normal Iterable is treated.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A possible attempt to specify a more structural for/in:

Static semantics

(.. try to infer an element type from var x or whatever that clause is, a type schemea C...)

  • Let S be the static type of e inferred with context type Iterable<C>.
  • If S is dynamic, let S be Iterable<dynamic>. (And we'll do an implicit downcast later)
  • If BOTTOM(S), let E be Never.
  • Otherwise if S implements Iterable<T> for some type T, let E be T.
  • Otherwise if S exposes an extension or extension-type getter named iterator:
    • let R be the return type of the S.iterator getter.
    • If R implements Iterator<V> for some V, let E be V.
    • Otherwise if R has an instance, extension or extensions type method named moveNext
      which can be invoked with zero arguments, and whose return type is assignable to bool,
      and R has an instance, extension or extension type getter named current with
      return type W, then let E be W.
    • otherwise a compile-time error occurs.
  • (... use E as the actual element type that we push back into var x. ...)
    and runtime semantics:
  • Evaluate e to a value v with runtime type O.
  • If the static type of e is dynamic.
    • If the runtime type of v is not a subtype of Iterable<dynamic>, a runtime type error occurs.
    • let i be the result of invoking the iterator getter on v.
    • let I be Iterator<dynamic> (aka. Iterator<E>).
  • Otherwise, if the static type of e implemented Iterable<T> ,
    • let i be the result of invoking the iterator getter on `v
    • let I be Iterator<T> (aka. Iterator<E>).
  • Otherwise the static type of e has an extesion or extension type getter named iterator.
    • let i be the result of invoking the denoted extension or extension type iterator getter
      with v as receiver.
    • if the denoted iterator getter's return type, R, implemented Iterator<V>,
      let I be Iteartor<V> (aka. Iterator<E>)
    • otherwise let I be R:
      (Then the iteration commences)
  • While iterating, do:
    • invoke the I.moveNext method with i as receiver, let n be the return value.
    • if the runtime type of n is not bool a runtime error occurs (possible downcast from dynamic).
    • if n is false, we stop iterating, and the execution of the for/in loop completes.
    • Otherwise n is true.
    • Invoke the I.current getter with i as receiver, let c be the return value.
    • If the initial part of the for/in is a declaration,
      • perform initialization of the loop declaration/pattern with c as matched value.
    • otherwise assign c to the pre-existing loop variable.
    • execute the loop body in a scope where the new variables of the loop declaration are in scope.
    • Continue iterating from the top.

This is not the only, or necessarily the best, design. (And it still hides the complexity of choosing whether to invoke an extension/extension type member or an instance member by just saying "invoke I.member".

We could, possibly should, say that we give precedence to extension type overrides of iterator, moveNext and current, even when the type implements Iterable, but if we do that, we step out of the Iterable framework, and have to look at the actual return type of current to find the element type.
Even if the type implements Iterable<num>, it might have a current returning MyNum, which perhaps implements double.
Today, we don't look that far down, we just say "implements Iterable<E>, OK, element type is E".

Also, the above does not allow iterating something that has an instance member named get iterator, but which doesn't implement Iterable. Why not?
Because we actually wanted to push people towards using Iterable.
We could go back on that and go fully dynamic, say:

  • If S is dynamic, let E be dynamic.
  • else if BOTTOM(S), let E be Never
  • else if S has a getter named Iterator with return type R, and
    • R has a method named moveNext which can be called with zero arguments,
      and has a return type assignable to bool, and
    • R has a getter named current with return type T,
    • then let E be T.
  • else a compile-time error occurs.
  • It's a compile-time error if E is not assignable to the loop variable.

and runtime behavior would be just like desugaring labels: for (decl in e) body to:

{
  S $tmp1 = e;
  R $tmp2 = $tmp.iterator;
  labels: while ($tmp2.moveNext()) {
    decl = $tmp2.current;
    { body }
  }
}

Here we don't try to do any kind of downcast from dynamic, we do dynamic invocations of iterator, moveNext and current (with predictably bad performance).

But you can iterate anything, whether it's an Iterable or not. The Iterable is there to provide all the other methods.


An example of that is something like `yield*`, which checks statically that the
operand’s type is assignable to `Iterable<T>` for some `T`, at runtime checks
that the value’s runtime type implements `Iterable<T>` (only necessary if the
static type is `dynamic`), and then invokes `iterator` and `moveNext`/`current`
on the result. Those invocations *must* be on the instance members.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

100% sure? ;-)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is intended as prescriptive, not predictive. It will invoke the instance members.


We still need to check everywhere that the phrasing doesn’t make assumptions
about *the static type of the instance member*, which might not be visible when
shadowed by and extension type. See “Iterable and `for`/`in` below” for an
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
shadowed by and extension type. See “Iterable and `for`/`in` below” for an
shadowed by an extension type. See “Iterable and `for`/`in` below” for an

example where the current specification no longer works, mainly because it’s
defined by a rewrite, without being explicit about the static types of the
rewritten code, and assuming that invoking `.iterator` on a subtype of
`Iterable` has a predictable result.

The types and places where specific types and members are mentions in the
specification include the following.

#### Object methods

Any reference to invoking `==`, `hashCode`, `runtimeType`, `toString` or
`noSuchMethod` are safe, since they cannot be shadowed by extension types. No
changes needed.

#### The types `dynamic`, `void`, `FutureOr`, `Function`, `Record`, `Null` and `Never`

An extension type cannot implement any of these types.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do not have any rules about Never. Should we have that?

It is not obvious that this will create any issues (other than being useless):

extension type NotEver(Never _) implements StatelessWidget {}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should not be allowed to implement it. Using it as representation type is fine. Implementing it is introducing a new nominative bottom type, which is too weird.

An extension type can only implement interface types and extension types, and among the types that could potentially be counted as interfaces types (depending on perspective), an extension type cannot implement any of the types mentioned here.

Which are precisely the types that we sometimes argue back and forth over whether they are normal class types with special rules, or special types where some have a fake declaration that looks like a class declaration.

It doesn't matter what they really are, as long as the spec explicitly puts them in the correct categories for everything where we make a distinction.


The specification can still safely assume that a subtype of any of `Function`,
`Record`, `Null` and `Never` does not have any extension type methods to worry
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again, we might constrain Never, but it is not obvious to me that it creates any issues to allow it.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm also not sure it creates any issues to allow implementing Null, but I'm not sure.
Out of an abundance of caution, I prefer to not test which assumptions our tools migth make about which bottom types exist, just to allow a declaration which can't actually be used to create any instances.

about.

The `dynamic` and `void` types are just top-types with special semantics, but
that only applies to that type, it’s not inherited by subtypes, so the
specification only checks whether a type *is* `dynamic` or `void`, not whether
it’s a subtype.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, just a comment on this: Extension types are subtypes of these top types just like all other types, and there is no need to add new rules about the treatment of expressions of type void or dynamic because of this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Precisely, this is an "nothing new to worry about here" comment.


It’s not possible to implement `FutureOr<T>` (or the other union type, `T?`),
only “interface types”. This ensures that an extension type doesn’t have
union-type like subtyping behavior.

No changes are needed related to the treatment of these types in the
specification.

#### Conditions and `bool`

Expressions in condition position (`if (condition) …`, (`do {…}`)`while
(condition)`, `condition ? … : …`, `condition || condition`, `condition &&
condition`, `!condition`, `… when condition` ) *must* have a type assignable to
`bool`.

There is no change to that. An extension type occurring in such a position is a
compile-time error if it does not implement `bool`., and not if it does
implement `bool`. All runtime behavior on `bool` values amount to checking
whether it’s the `true` or the `false` value, which is still sound. _I do not
believe we make any assumptions about members of `bool`, or invoke any
explicitly._

#### Numbers and arithmetic operators

We have special rules in type inference for number operations on `int`, `num`
and `double`, to ensure that the static type of `1 + 1` is `int`. We effectively
pretend to have overloaded operators for typing only.

The [newest
version](https://github.com/dart-lang/language/blob/main/accepted/2.12/nnbd/number-operation-typing.md)
of those rules came with null safety. The way the rules are written, they apply,
e.g., to `e1 + e2` when the static type of `e1` is a non-`Never` subtype of
`int` (the receiver is an `int`, maybe typed as a type variable `X` with bound
`int`), and the static type of `e2` is a subtype of `int`. That needs to also
say that the `+` must denote an instance member, to avoid applying to an
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've used the phrase 'extension instance member' for a few years in order to be able to say 'extension static member' as well, and I expect that we'll have to say 'extension type instance member' and 'extension type static member', too. So it's better to say 'class instance member' here:

Suggested change
say that the `+` must denote an instance member, to avoid applying to an
say that the `+` must denote a class instance member, to avoid applying to an

extension type implementing `int` and declaring an extension type `+` operator.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(whitespace)

Suggested change
extension type implementing `int` and declaring an extension type `+` operator.
extension type implementing `int` and declaring an extension type `+` operator.


That amounts to two sections being changed by inserting something to the effect
of “where it’s an instance member”, additions italicized here:

>Let `e` be an expression of one of the forms `e1 + e2`, `e1 - e2`, `e1 * e2`,
>`e1 % e2` or `e1.remainder(e2)`, where the static type of `e1` is a
>non-~~`Never`~~_bottom_ type *T* and *T* <: `num`, ~~and~~ where the static
>type of `e2` is *S* and *S* is assignable to `num`_, and where *T* has an
>implementation of the operator, or `remainder` method, that is an instance
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
>implementation of the operator, or `remainder` method, that is an instance
>implementation of the operator, or `remainder` method, that is a class instance

>member_. Then:

and

>Let `e` be a normal invocation of the form `e1.clamp(e2, e3)`, where the static
>types of `e1`, `e2` and `e3` are *T*<sub>1</sub>, *T*<sub>2</sub> and
>*T*<sub>3</sub> respectively, ~~and~~ where *T*<sub>1</sub>, *T*<sub>2</sub>,
>and *T*<sub>3</sub> are all non-~~`Never`~~_bottom_ subtypes of `num`_, and
>where *T*<sub>1</sub> has an instance member named `clamp`_ . Then:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
>where *T*<sub>1</sub> has an instance member named `clamp`_ . Then:
>where *T*<sub>1</sub> has a class instance member named `clamp`_ . Then:


This will prevent the rules from applying to invocations of extension type
operators or methods, which shadow the platform methods that we know and trust.

No other changes should be needed. The remainder of the rules are about context
types, and require the types involved to be *supertypes* of a number interface
type, which extension types never are.

#### Patterns

All map pattern entry patterns and list pattern element patterns are define in
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checking the pattern feature specification, we have no occurrences of 'entry pattern' or 'element pattern' denoting the entire construct (such as <expression> ':' <pattern>). It could also be confusing if 'entry pattern' means <expression> ':' <pattern> and 'entry subpattern' means just the pattern which is part of the entry. So maybe:

Suggested change
All map pattern entry patterns and list pattern element patterns are define in
Map pattern entries and list pattern elements have a semantics which is defined in

terms of calling `length,`, `operator[]` and/or `containsKey` on the object. All
these invocations *must* be instance member invocations, whether the static type
of the matched value type is an extension type implementing `Map` or `List` or
not, and whether it defines its own `length`, `operator[]` or `containsKey`
extension type members. For all practical purposes, the value is cast to `Map<K,
V>`/`List<E>` for some `K` and `V`, or `E`, *before* accessing elements, and it
uses the caching behavior of instance elements.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer if we avoid the "magic" and simply make it a compile-time error for a collection pattern to invoke length etc. if it is an extension type instance member rather than a class instance member.

Next, I'm not 100% convinced that it should be an error. It could be a useful hook.

Just talked about this IRL: We might consider allowing extension type instance members and/or extension instance members to provide the required members to support iteration:

So the requirement for being usable in a for loop could be (1) must be an Iterable<T> for some T (the static type could be dynamic or some S <: Iterable<T>), or (2) must have a static type which is an extension type that has the required members (iterator, yielding an Iterator<T> for some T, or returning an object typed as E which is an extension type that has moveNext() and current).

We might want to treat for loops differently than other built-in constructs, because they are so performance critical. (For instance, it might be very important that we could perhaps use a native kind of iteration on a JSArray.)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer if we avoid the "magic" and simply make it a compile-time error for a collection pattern to invoke length etc. if it is an extension type instance member rather than a class instance member.

That could apply to collection patterns in declaration patterns, but not in refutable patterns.

That is, it's a compile-time error if the matched value type of an irrefutable list pattern (in a declaration/assignment pattern) has an extension type instance member named length or operator[], and if the matched value type of an irrefutable map pattern has an extension type instance member named contains or operator[].

Or we can say that the list/map patterns always perform class instance member invocations for those methods, which is possibly what it says today. (But most likely it says nothing conclusive, because there used to be nothing to say.)


Object patterns, on the other hand, base their member accesses on the *static
type* of the object pattern type. A `case ExtensionType(foo: p1, bar: p2)` use
the `foo` and `bar` getters of `ExtensionType`, whether they are extension type
members, instance members, or even extension members. _It follows the general
rule in being a place where extension members are allowed today._

Relational patterns can use extension and extension type implementations of `<`,
`<=`, `>` and `>=`, and the `==` operator *cannot* be shadowed by extension
methods.

#### Iterable and `for`/`in`

A `for (D x in e) body` loop, whether statement or element, with value
declaration `D x` and `e` having static type, `T`, we require that `T` is
assignable to `Iterable<dynamic>`, which we today knows is equivalent to
implementing `Iterable<S>` for some type `S`, or being `dynamic` or a bottom
type.

The behavior of the for-in statement is currently specified by a *rewrite* to
the following code:

```dart
T _$id1 = e;
var _$id2 = _$id1.iterator;
while (_$id2.moveNext()) {
D x = _$id2.current;
{
body
}
}
```

That rewrite is no longer valid with extension types. The type `T` may be an
extension type which *does* implement `Iterable<S>` for some `S`, but which then
*shadows* the default `iterator`, and returns something entirely unrelated to
`Iterator<S>`. _We do not intend to invoke that extension type member._
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned, we could do it anyway.


The behavior of a `for`/`in` element is defined in terms of a `for`/`in`
statement.

##### New specification:

It’s (still) a compile-time error if `T` is not assignable to
`Iterable<dynamic>`. _This, also still, means that `T` is a bottom type,
`dynamic` or a type which implements `Iterable<S>` for some `S`. That type may
just also be an extension type._

If `T` implements `Iterable<S>`, let `E` be `S`, otherwise let `E` be `Never` if
`T` is a bottom type, or `dynamic` if `T` is `dynamic`.

Now rewrite the `for`-`in` loop to:

```dart
Iterable<E> _$id1 = e;
Iterator<E> _$id2 = _$id1.iterator;
while (_$id2.moveNext()) {
D x = _$id2.current;
{
body
}
}
```

Like before, any errors that would occur in that program are reported as errors
in the original `for`-`in` statement.

_**TODO**: Specify this without using a “desugaring” rewrite. Problems like
these are exactly why rewrites are problematic, they tend to introduce *more*
information, through the extra syntax or inferred types, than what the original
program warranted. Also, the rewrite supposedly happens after type inference, so
we need to either perform type inference on the desugared code, or assign a
context type and static type to each new expression, because those are
referenced by the dynamic semantics._

##### Consequences to existing code

The only real change is that the type of `_$id1` is not the static type of `e`.
That means that if there is a class which overrides its iterator to be more
specific than required, that information is lost.

Example:

```dart
class VeryInts extends Iterable<num> {
Iterator<int> get iterator => <int>[1].iterator;
}
void main() {
for (int i in VeryInts()) print(i);
}
```

With the previous *specification*, the type that `.iterator` should be accessed
on was `VeryInts`, which means that `_$id2` will get static type
`Iterator<int>`, and its `.current` is then assignable to `int i`. With the new
specification, we don’t look at the members of the actual type, instead we
immediately up-cast to `Iterable<E>` and work with that.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd definitely prefer that the developer must write that upcast and the treatment after that point is completely standard, rather than getting into a situation where some members are called based on the type Iterable<E> for some E while the receiver has an extension type V which has an iterator member with a different behavior.


*Luckily*, our implementations already do that: CFE (Dart2js on DartPad):

> ```
> lib/main.dart:5:12:
> Error: A value of type 'num' can't be assigned to a variable of type 'int'.
> for (int i in VeryInts()) print(i);
> ^
> ```

and Analyzer:

> ```
> error: line 5 • The type 'VeryInts' used in the 'for' loop must implement 'Iterable' with a type argument that can be assigned to 'int'.
> ```

So, this looks like it might be a spec-only change for the static semantics, to
make it match the actual implementation, and it’s *likely* that the
implementations will also do the right thing when faced with extension types.
They *must* only invoke instance `iterator`, `moveNext` and `current` members.

#### Iterables and synchronous `yield*`

The synchronous `yield*` operator takes an expression which, like `for`/`in`,
must be assignable to `Iterable<T>`, and it too must use only instance members to
iterate the operand where it state that it invokes `iterator`, `moveNext` and
`current`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably have the same level of support for a structural extension-or-extension-type based iteration in a for loop and in a yield* statement.


#### Stream and `await for`, asynchronous `yield*`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should use rules that are as similar as possible with for and await for.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If so, then the rule should be to not invoke extension type members.

I really do not want to get into defining which part of the Stream interface that await for and yield* uses.

  • Call listen with three functions arguments, and cancelOnError: true, get object back. (This is the simple step.)
  • Occasionally call pause() or pause(Future), maybe resume and cancel on the returned object.
  • Maybe read the isPaused getter. (Not necessary, but can be used as optimization, to never pause more than once.)
  • Possibly call onData, onError, onDone methods with function arguments (or maybe even with null). That's a valid approach to changing the continuation that the value comes back to, instead of keeping state and a trampoline on the side.
  • Maybe even call asFuture (but I doubt that's useful).
  • Await the result of calling cancel.

Basically, you have to return something which implements the entire StreamSubscription interface.
For Iterator, that's easy - it's two members and we always call both of them.
For StreamSubscription, you can possibly get away with only pause, resume and cancel. And if we let you get away with that, then it becomes a breaking change for us to change the implementation in the future.

So I want to enforce that whatever listen returns, it implements StreamSubscription.

Also, the compilation of async operations is much more complicated than a simple for/in loop. The way lowering and compilation of async functions works, we are likely reusing some functionality that does class instance member invocations. Having to create new compiler paths to remember an extension type, just to call an extension type pause method on something which is a StreamSubscription, is just not worth it.

The story of "An await for doesn't see your static type, only the Stream<T> that it implements, because that's all it needs" is simple and consistent.
That's why that's what I went with for non-async for/in too.


The `await for` specification is deliberately vague in how it’s implemented.
Like synchronous `for`/`in`, it checks that the stream expression’s static type
is assignable to `Stream<Object?>`, which again means implementing `Stream<S>`
for some `S` or being `dynamic` or `Never`. Then it “listens on the stream”, and
“when an event is emitted”, it executes the loop body.

It does talk about a “stream subscription”, and calling `pause`, `resume` and
`cancel` on that. Every call, including the alluded `listen` call on the
original stream, must be an instance method call on a `Stream` or
`StreamSubscription`-typed object. Any assumptions about the static types
of such a call is based on the element type, which is `S` if the stream
expression's static type implements `Stream<S>`, and `Never` or `dynamic`
if it is a bottom type or `dynamic`.

The behavior of an `async for` element is defined in terms of an `async for`
statement, anhd should just work if the other one does.

The same vagueness applies to `yield*`, and it too must use only
instance methods to process the stream elements.

#### Futures and `await`, asynchronous `return`

An `await`, and the implicit optional await built into an `async` function’s
`return`, checks whether its operand is a `Future<T>`, where `T` is
*flatten*(`S`) and `S` is the static type of the operand.

When it comes to *extension types*, the *flatten* function works without
changes. If an extension type `E` implements `Future<S>`, then *flatten*(`E`) is
`S`, just as for an instance type, and if not, *flatten*(`E`) is `E`. _This is
safe and sound. (We can prove that, but won’t do so here. The rule “`T` \<:
<code>FutureOr\<*flatten*(T)\></code> for all `T`" still applies, both before
and after extension type erasure.)_

Implementations must ensure that they don’t call any extension type `then`
methods or similar, but should not otherwise be affected.