Late parameters, late-init-query operator, parameter element

[lrhn]

(Because I'm sure I've written this before, but it's probably in a comment of another issue, here is a stand-alone version.)

The copyWith problem is that it's impossible to distinguish an omitted argument from an explicit passing of the default value.
That's generally considered a good thing, but it makes forwarding arguments to another function awkward.

Dart once had a "was argument passed" operator, ?x which was true if x had an argument passed, and false if it got its default value implicitly. It was intended to make it easier to forward arguments to other functions, but since those other functions could now also behave differently depending on whether an argument was passed or not, it actually made it harder. It was quickly removed again.

I propose that we add two things, both a way to see if an argument was passed, and a way to easily pass or not pass an argument in constant code size.

Proposal

Allow optional parameters to be declared late. A late parameter need not be initialized, so it can be optional and not have a default value. (It's not allowed to have a default value, because if it has one, it doesn't need to be late.)

Add a prefix operator ?? (strawman syntax) which applies to a late local variable, and which evaluates to a bool value that is true if the variable is initialized, and false if it's not. (This value affects "definite assignment" analysis, so it's possible to write if (??x) { return x; } else { x = 42; } no matter what the assignment analysis said about the late variable x before.)
(See also #1028. Maybe ?? should only apply to late parameters, but it can be extended to any other late variable.)

Finally, allow argument elements, which are arguments which can evaluate to no value.

<argumentList> ::= '(' <arguments> ')'
<arguments> ::= (<argument>? ',')* <argument>?
<argument> ::= (<identifier> ':')? <argumentElement>
<argumentElement> ::= 
      'if' '(' <expression> '>') <argumentElement> ('else' <argumentElement>)?
      'switch' '(' <expression> ')' '{' <elementSwitchCases> '}'
    | '?' <expression>
    |<argumentExpression>
<argumentExpression> ::= <expression> 
<elementSwitchCases> ::= (<elementSwitchCase> ',')* (<elementSwitchDefault> | <elementSwitchCase>) ','?
<elementSwitchCase> ::= 'case' <pattern> ('when' <expression>) ':' <argumentElement>
<elementSwitchDefault> ::= 'default' ':' <argumentElement>

Anywhere we currently allow an <expression> as an argument, we now use <argument>, which allows omitting a value for that argument.

• The if can have no else branch.
• The switch needs not be exhaustive
• The ? <expression> is a non-value if the expression is null.
• The <argument> itself can be omitted (otherwise ?null would be the shortest "no value" syntax).

(The element-switch and null-aware element ?e should both be added as collection elements as well.)

That allows a call of foo(, if (false) 0, switch (0) { case 1: 0 }, ?null, 42) to call foo with the first four arguments being omitted.
It's the same as foo(,,,, 42). It's an error if a parameter is not optional, and its argument element can evaluate to no value (more precisely: It's empty, any if has no else, any switch is not exhaustive, or ?x occurs at all).

It's new that non-trailing optional positional arguments can be omitted, and later parameters can be passed.

There is no good way to avoid that property, while allowing conditionally passing an argument, not without completely disallowing that for positional arguments. Then we'd disallow late positional parameters too. That's an annoying asymmetry with named parameters, so it's better to just allow non-trailing positional parameters to be omitted. The majority of functions can't tell the difference (yet, without the ?? feature), they'll just get their default values.
(A few functions may deliberately use private sentinel default values to check whether an argument was passed. That's not generally possible, which is why it doesn't solve the callWith problem, but if the parameter type is Object? anyway, any sentinel value can be used. Calling those functions incorrectly may give you a different result than you expected (well, what did you expect?). So don't do that. The fuynctions will probably be rewritten to use ??parameter instead of ~identical(parameter, _sentinel) soon enough.)

Semantics

The semantics are straight-forward.

• A late parameter with no argument value passed is uninitialized after "binding actuals to formals", behaving just like any other uninitialized local late variable. If an argument was passed, it's initialized to that value. The variable starts as potentially assigned, which means it's not a static error to read or write it, but may be a runtime error. Reading it will throw, if unassigned, writing will succeed if unassigned or non-final. Static type is the declared type. (For a covariant parameter, the parameter's local variable's type is the declared type, even if the function parameter's runtime type is Object?. Same as always.)
• The ??parameterVariable has static type bool, the operator checks whether the late parameter is initialized or not. If used in a conditional position, the operator promotes the variable to definitle assigned on the true branch and definitely unassigned on the false branch. It's a compile-time error to use it on anything other than a late variable name.
• Argument elements are evaluated as would be expected. if they end up with no value (if with false value and no else branch, non-exhaustive switch with no matching case, ?e with a null value, or just an omitted <argument>), then the argument list has no value at that position (or at that name, if named, an omitted <argument> is always counted as a positional argument). This is new, argument lists now need a way to represent "no value" at any position or name, in a way that can be acted on when binding actuals to formals. See first item.

Consequences

It's now possible to write a copyWith:

class Point 
  final int x, int y;
  final Color? color;
  Point(this.x, this.y, {this.color});
  Point copyWith({late int x, late int y, late Color? color}) =>
    Point(??x ? x : this.x, ??y ? y : this.y, color: ??color ? color : this.color);
}

(I'm quite certain that the first request we'll get is to allow ?? as infix, treating an unassigned late variable the same as a null-valued nullable non-late variable. Then copyWith would be:

  Point copyWith({late int x, late int y, late Color? color}) =>
      Point(x ?? this.x, y ?? this.y, color: color ?? this.color);

That's ambiguous when a variable is both nullable and late, since reading the variable is allowed (it only might throw).
More likely the ?? will be saved for special cases, and you can still use null where it's not a valid value:

  Point copyWith({int? x, int? y, late Color? color}) =>
      Point(x ?? this.x, y ?? this.y, color: ??color ? color : this.color);

Still, expect this request.)

It's also now possible to forward any known argument list, without knowing the default values:

Result foo(int x, {late Banana banana, late SecretSauce sauce}) {
  log("foo(${["$x", if (??b) "banana: $banana", if (??sauce) "sauce: $sauce"].join(", ")})");
  return super.foo(x, if (??banana) banana: banana, if (??sauce) sauce: sauce);
}

or from noSuchMethod:

  noSuchMethod(i) {
    if (i.memberName ==  #foo) {
      return target.foo(i.positionalArguments[0] as int,
           if (i.namedArguments case {#banana: Banana banana}) banana: banana,
           if (i.namedArguments case {#sauce: SecretSauce sauce}) sauce: sauce,
      );
   } else {
     return super.noSuchMethod(i);
   }
 }

The forwarding is more signficant than just copyWith. If copyWith was the only problem to solve, allowing non-constant default values is enough:

  Point copyWith({int x = this.x, int y = this.y, Color? color = this.color}) =>
      Point(x, y, color: color);

(Which we should just do. #140! But that does allow seeing whether an argument was passed using side-effects through external variables, so we might still want argument elements to counteract that ability.)

[jakemac53]

I propose that we add two things, both a way to see if an argument was passed, and a way to easily pass or not pass an argument in constant code size.

Definitely agreed that we need both, and I really like that the latter one also helps with default values.

I don't think it fully fixes the default value issue because often it is actually an override, and you aren't just calling super or do need access to the default value even if you are calling super. But, it still helps with some cases.

[jakemac53]

This proposal to me is a nice solution that avoid adding something like undefined, but essentially gives you that power in a limited way using an existing concept (late variables), and without introducing an actual instance of some weird undefined object.

[abitofevrything]

Why not allow late parameters to have an initializer (same syntax as default values) that is evaluated on every invocation if the parameter is not explicitly passed?

The copyWith could then further be reduced to this (omitting some fields as I'm typing this on my phone):

class Point {
  final int x;

  Point(this.x);

  Point copyWith({late int x = this.x}) => Point(x);
}

[tatumizer]

Late variables are evaluated lazily, so the presence of default value won't prohibit the use of ?? a - unless the variable was accessed before the test:

f({late x=1, late y=2}) {
  print(x); // x gets initialized here
  print(?? x); // true - variable is already initialized
  print(?? y); // false - variable is not initialized yet: asking the question doesn't count as an "access"
}

The example of "print(x)" shows that the confusion between "passed" and "initialized" might be common. For that reason, maybe it's better to adopt a narrow definition: '?? a` means "parameter was passed" regardless of whether it is initialized or not at the moment. Not sure.

[abitofevrything]

I think the ?? prefix operator could be generalised to all late variables - not just parameters - so having it use a parameter-specific term like "passed" might not work great in the future.

I still wouldn't want it to become common practise to have late final variables that have their initialisation checked with ?? though as that goes against the spirit of late final. Better have a nullable field.

[abitofevrything]

I'm less enthusiastic about the "adding more places a variable can be uninitialised" part of this proposal though. My opinion is that uninitialised variables are dangerous, with better alternatives available.

Not having to pass a value for a parameter is fine, but it should only be allowed for parameters that can already be omitted (optional positional and non-required named parameters). For the named parameters, it's completely useless as you can just not pass the named parameter.

There should probably also be a lint encouraging people to use named parameters instead of optional positional ones if they do find themselves having to omit a value to "skip" a positional parameter. The foo(,,) syntax likely looks like a syntax error to most people.

[mateusfccp]

I'm less enthusiastic about the "adding more places a variable can be uninitialised" part of this proposal though.

Maybe you just want #140, then?

[jakemac53]

I think the ?? prefix operator could be generalised to all late variables - not just parameters - so having it use a parameter-specific term like "passed" might not work great in the future.

+1 it is annoying today that you can't check the initialized status of late variables, and it means you can't use them in places you might like to. I think extending this proposal to all late variables helps solve multiple problems, which is always a good sign for any proposal.

[mmcdon20]

If we have both late parameters and non-constant default values, then I think you should allow late parameters to have default values also.

This could allow you to for example assign a late parameter to the result of an expensive function call that may or may not get executed due to lazy evaluation.

---

EDIT: Correct me if I am wrong, but I assume a late parameter would allow you to short circuit an argument, for example:

bool and(bool a, late bool b) => a && b;
bool or(bool a, late bool b) => a || b;

bool truePrint(String s) {
  print(s);
  return true;
}

void main() {
  and(true, truePrint('A')); // side effect, prints A
  and(false, truePrint('B')); // no side effect, second argument never evaluated
  or(true, truePrint('C')); // no side effect, second argument never evaluated
  or(false, truePrint('D')); // side effect, prints D
}

[Mike278]

A workaround I use for "the copyWith problem" is just to generate separate with$NAME methods:

void main() {
  final x = Foo('some', 1);
  final y = x.withBar('y');
  final z = x.withBaz(null).withBar('another');
  print('$x\n$y\n$z');
}

class Foo {
  final String bar;
  final int? baz;
  Foo(this.bar, this.baz);

  Foo withBar(String newBar) => Foo(newBar, baz);
  Foo withBaz(int? newBaz) => Foo(bar, newBaz);

  @override
  String toString() => 'Foo(baz=$bar, baz=$baz)';
}

The intermediate objects obviously aren't ideal performance-wise but IMO in practice it's pretty uncommon that you're updating a large number of fields at once, and IIUC Dart's garbage collector is optimized for large numbers of short-lived objects. If it started to become a problem I can imagine a few optimizations for chaining multiple withs together.

[mateusfccp]

As an alternative to the ??argument syntax, we could have something like in Common Lisp, where a new name is introduced:

(defun foo (a b &optional (c 3 c-supplied-p))
  (list a b c c-supplied-p))

In Dart, we could reuse some token to indicate this (strawman using +):

class Point 
  Point(this.x, this.y, {this.color});

  final int x, int y;
  final Color? color;

  Point copyWith({
    late int x +hasX,
    late int y +hasY,
    late Color? color +hasColor,
  }) {
    return Point(
      hasX ? x : this.x,
      hasY ? y : this.y,
      color: hasColor ? color : this.color,
    );
}

[Cat-sushi]

I see some relation with #877.

[Cat-sushi]

@abitofevrything

Why not allow late parameters to have an initializer (same syntax as default values) that is evaluated on every invocation if the parameter is not explicitly passed?

I think late doesn't necessarily mean lazy like top level variables.
So, the syntax you proposed is confusing for me.

[tatumizer]

Suppose we introduced a method isInitialized for the late variable (like kotlin does).

late int foo;
print(foo.isInitialized)

The problem is that methods are always defined for types. Does late int have a type different from int, such that this type provides the method isInitialized ?. Probably not (the type is still int). So, isInitialized is not a method on type. Then on what? On declaration?
Here we enter the territory of metaprogramming.

What will change if we replace isInitialized with an operator like ?? ? Will it solve the problem?
Operators are defined for types, too, they just use symbols rather than letters for method names. Renaming isInitialized to ?? doesn't make the problem disappear - it just makes it less visible (but not by much). We still don't know what is so special about the type of late int that makes the operator ?? applicable to it. We still can't answer a simple question: what is the type of if (cond) value in foo(if (cond) value)? - so to support conditional passing of parameters, we would either need to invent yet another mechanism, or somehow shoehorn this expression into an artificial "late" variable, or something...

Given the rigor around types maintained by dart, it would be more consistent to create an honest type representing the omitted parameters. This can be done without introducing "another null". Please give some consideration to the idea of NoneOr, as outlined in this comment

[lrhn]

We still don't know what is so special about the type of late int that makes the operator ?? applicable to it.

Nothing would make ?? applicable to the type int. It's not a bout types, being late is an implementation detail, not part of the API. A late variable is still just a getter and a setter at the API level.

What we could do is make ?? applicable to a late variable, independently of the type. You can doo foo ?? something, but it can't be used on something which isn't a late variable's getter.
(It should still work for something nullable too. The big question is what happens if something is nullable and a late variable. I guess it would check both, and skip both if the variable is uninitialized or if it's initialized to null. So don't do that.)

Since being late is not part of a class interface, or member signature at all, such an operator can probably only work for local variables, maybe static/top-level variables inside the same library, and maybe even instance variables when read through super in a subclass in the same library.

[tatumizer]

If it's not about types, then I don't understand your earlier objections to nameof(foo). I can't see much difference between nameof(foo) and isInitialized(foo) (the verbal equivalent for ??)
:-)

[lrhn]

The difference is not typing, it's whether the operation looks like a function, but behaves like a special operator.

If it's the latter, I want a special operator for it, so I don't have to look at nameof(foo) and try to guess whether nameof is the special operator, or some user-defined function in scope.
And if it cannot be a user defined function, then we've made nameof a reserved word, which is a hard breaking change, so that's a hard sell.

Dart originally defined assert as a name exposed by dart:core, and special-cased it if the assert in assert(...) denoted that name. It was possible, but the edge cases were annoying. (Like, can you tear-off the assert function? Is it a function at all. If not, what is it then?)
It's a reserved word now.

We did the opposite with Function in function types, where the word is contextually reserved. It means that the parser decides whether to treat the word as a reserved word, or as an identifier. There is no way to use that word in a way where you can access the reserved word by name.

That is why I'd prefer nameof identifier to nameof(identifier). The former is currently not valid syntax, so there is no question where the name should be looked up.

And same for isInitialized, I'd prefer if (defined name) to if (isInitialized(name)), because I can immediately see that his is not just a function call. And if (defined formatHarddriveOnRead) cry(); is definitely safe (or too late). I'd be very, very worried about if (isInitialized(formatHarrdriveOnRead)) cry();!

Similar things should look similar, distinct things should look different. That's the cornerstone of experience-based readability. We can teach people to recognize patterns better than lots of individual rules.

[tatumizer]

Probably off-topic in this thread, but anyway...
Would it be too late for dart to introduce the whole class of reserved "annotations" like @nameof(foo)?
They can be distinguished by the lowercase letter. This syntax is quite common for built-in "functions" in other languages.
What's good about it is that it looks like a call to meta-function, which it really is.
(If this won't work, there should be some other syntax guaranteed to not conflict with user-defined identifiers).

[lrhn]

It's never too late to invent entirely new features!

Using @ is probably too close to existing syntax, but %nameof(...) and \nameof(...) are free.

Not sure it's worth it, we don't plan to add a lot of "really language features, not functions". But maybe it's just what we need for inline assembly operations for FFI

[tatumizer]

Similar things should look similar, distinct things should look different

This is an argument against the proposed syntax ??foo. The "operator" ?? is unlike anything we see in dart: it applies only to variable names and is much closer to nameof(foo) than to, say, -foo. I'm not sure even whether ?? this.x is correct (probably not), but if x is a late instance variable, then it's hard to explain why ?? x is correct and ?? this.x is not.
But if ??x is "special", it might require a distinct syntax.

Another (unrelated) argument could be that var x = ?? a ? a ?? 42 : a is a bit cryptic
(I attempted to say "if (a is defined and not null) set x= a else set x=42", but I could be mistaken :-)

Despite that, I admit I kinda like the proposed operator ?? . Maybe the philosophical principle "Similar things should look similar, distinct things should look different" should be amended by "... unless the exception can be justified by convenience or something". :-)