Macro application annotation identification

[lrhn]

We need the compilers to recognize annotations as macro applications, at a point in compilation where the program isn't yet complete (because macros haven't run), which means expression evaluation isn't necessarily possible.
Since macro applications are annotations, and annotations are expressions, we need some way to recognize an expression as representing a macro, and preferably without needing to evaluate another annotation to mark the first annotation class as a macro application.

Here's an attempt to flesh out one idea, based on in-person discussions from last week.

Pubspec/package-config-defined macro annotations

We want to recognize specific annotation expressions as macro application triggers.

We do that by singeling out a macro application annotation class, and then considering every constant instance of that class as a macro application trigger. We will recognize instances of that class in a number of ways, without requiring expression evaluation.

Further, we associate a macro implementation with the macro application. There are a number of ways we can do that, I'll mention some and leave that open for now.

The metadata specifying which class is a macro application trigger, and which macro implementation it executes, will be available in the .dart-tool/package_config.json file, as a non-breaking addition to the format. (The format for a program that doesn't depend on any macro-defining packages will be the same as today, and having a macro available will add extra information.)

The information in the package configuration is generated from similar information in the declaring package's pubspec.yaml.

Pubspec format

A package which introduces a macro application class can add (something like) the following to its pubspec.yaml:

macros:
  - 
    application:
       library: macros.dart
       class: MyMacro 
    implementation:
       library: src/macros/my_macro_impl.dart
       class: MyMacroImpl
  - 
     application:
       local_library: tools/test_macro.dart
       class: MyTest
     implementation:
       local_library: tools/test_macro_impl.dart
       script: true

The library: is followed by a path inside the current package's lib/ directory, so a package path without the leading package:mypackagename/, and local_library is a file path relative to the pubspec.yaml file itself, which must not be inside the lib/ directory. This allows a package to declare a local macro for itself, fx something only used by its own tests.

Files inside lib/ must not depend on a local macro. (They can't without importing using a file: URI, so that should just come naturally.)

The implementation: (somehow) specifies how to run the implementation code.

Package-config format

This information is written into the package-config.json file for any package which depends on the macro-declaring package.

Local macros are only included for the current package, public macros are included for all packages in the program.

The format for a package: entry is changed to include a macros: [] field:

   {
      "name": "my_package",
      "rootUri": ".../pub_cache/hosted/my_package/1.2.3/",
      "packageUri": "lib/",
      "languageVersion": "3.7",
      "macros": [
        { 
          "application": {
            "library": "macros.dart",
            "class": "MyMacro"
          },
          "implementation": {
            "library": "my_macro.dart",
            "class": "MyMacroImpl"
          }
        }
      ]
    },

The library URI is relative to the surrounding package's package: URI root: package:my_package/.
Which means that both the macro application annotation and the macro implementation entry point must be in the same package. (They can always call code from other packages afterwards.)
It's not possible to designate an existing annotation from another package as a macro trigger.
(No making @override a macro trigger!)

A local_library entry can be included for the current package, which has a path relative to the package_config.json file itself. (It can use non-relative file paths, but generally shouldn't. I guess it can technically use a package: URI as the implementation, but it should use library for that.)

  {
    "name": "this_package",
    "rootUri": "../",
    "packageUri": "lib/"
    "macros": [
      {
        "application": {
           "local_library": "../tools/test_macro.dart",
           "class": "MyTest"
        },
        "implementation": {
           "local_library": "../tools/test_macro_impl.dart",
           "script": true
        }
      }
    ]
  }

No entry may have a library path for the application and local_library for the implementation.
The other direction is allowed, but is the considered a package local macro.

Tool usage

With this information available, a source processing tool (compiler, analyzer) can recognize a macro application class syntactically. It's the class with the given name in the given library.

The next step is then recognizing source annotations which are instances of that class.

To do that, the compiler (I'll use that to cover any program which may run macros) can directly recognize a constant expression as denoting an instance of a (macro trigger) class if:

• The expression is a constant object creation expression whose type clause (the clause before the .name and (args)) denotes the class in the preliminary scope (the resolution we find before running macros, knowning that macro-added code may change the binding of some names).
  • A type claus denotes the class if it has the form T or T<Types> and the (possibly qualified) identifier T resolves to the class, or T resolves to a type alias declaration, whose RHS type clause denotes the class.
• The expression is a (possibly qualified) identifier which resolves to constant variable declaration, and the constant variable's RHS expression denotes an instance of the class.
• The expression is (expr) and expr denotes an instance of the class. (Just to not be daft.)

This is farily limited, and deliberately so.
There is no conditional expression. You can't do const macro = isWeb ? MyMacro.web() : MyMacro.native();.
You'll have to do const macro = MyMacro(isWeb); and handle the condition inside the constructor.
(Which can be a factory constructor which creates an instance of a subclass. The macro application detection doesn't care, it's creating a constant of the macro application trigger type.)

It is possible to have a conditional import include one of two constants with the same name, and use that as macro trigger annotation.

Author support

This requires a macro author to update the pubspec.yaml file every time they make a change to the naming or location of a macro application annotation class or a macro implementation.
That gets old fast.

I propose that the analysis/language server helps with this, by us adding a @Macro annotation to dart:core.
Then an author can annotate the macro application class as:

@macro("src/macro/impl.dart", "MyMacro")
class MyMacro {
  // ...
}

and get a diagnostic, with a quick-fix, if that doesn't match the macro listing in pubspec.yaml.

That is, the way to write a macro is to write:

@macro("src/macro/impl.dart", "MyMacroImpl")
class MyMacro {
  const MyMacro();
}

and then have a quick-fix insert the corresponding lines into pubspec.yaml and update package_config.json,
and have another quick-fix create src/macro/impl.dart if it doesn't exist and add a class MyMacroImpl implements ProperMacroInterfaces {} to it.

Maybe allow the implementation to have:

@macro.impl("/macro.dart", "MyMacro")

so that if just one of the @macro annotations get out of sync with declarations, they can be auto-fixed to the new name and location, based on finding the back-link.

Macro implementation execution

The macro implementation is run as a separate program, in a context where it has access to the "macro execution context" that triggers the macro execution, and which allows introspection and code generation.

Currently the macro implementation is a class declaration, which means that it's implicitly constructed by the macro execution context (it must have an unnamed constructor that can be called with zero arguments).
That can still work. There is some set-up that happens before macro execution starts, so it's not unreasonable to allow that set-up to be generated specifically for the macro class, so that it can be instantiated. (Or use dart:mirrors. Or not!)

Other options are:

• Point to a constant declaration which must hold the macro implementation object. Rather than relying on calling a default constructor to create the object, have the author create the instance themselves. Having to be constant also ensures the object isn't stateful, and doesn't keep information between stages. (Unless it cheats and uses global variables, but we can say that we don't promise that later stages are run in the same isolate.)
• Point to a not-necessarily-constant getter (can be a final variable, can be a real getter) which produces the instance. That allows an implementation object with state, but we can still say that each phase asks for a new instance, and may not do so in the same isolate, if we want to keep phases separate. (Do we? Seems potentially inefficient.)
• Just treat the library as a script, and call its main function with ([], macroContext), and then the main function has to register its macro implementation in the context. Introduces no new language features, and basically treats running macro implementation as an Isolate.spawnUri. The macro implementation doesn't even have to implement specific interfaces, it can just register callbacks for all the phases it wants them for, and can do so later, say it can decide in phase 1 that it needs to run in phase 2 too, but doesn't have to commit early by implementing the "phase 2 interface".

All of these approaches have the restriction that a macro implementation cannot do asynchronous initialization.
The solution to that is to delay that initialization until the start of phase 1. Probably not a problem in practice.