Generator State Machine?

[sneakers-the-rat]

In trying to work with the generators, I think I would find it really helpful to have a state machine diagram to know how and in what order different parts of the build process happen. The spec is complex enough, and it seems that it's hard to keep the generators in sync, it might be worth spending some time on standardizing some desired build state machine for generators and drafting up what it might take to work towards that. Might help with code consolidation and testing, as well as general reasoning about what one can and can't expect from the generators.

i know it's a pretty huge (and vague) idea, but just putting it out there as a discussion because i'm also curious about what y'all are thinking about the longterm direction for the architecture of the generators :)

[cmungall]

Which generators have you been working with? A bit of inside baseball here is that there are two lineages of generators in the stack. One follows the visitor pattern, and makes use of schemaloader. The other users schemaview. This is irrelevant to users of the framework but it's a major gotcha for contributors. We are planning to rewrite all visitor based ones as simpler schemaview based ones. I suspect your comments are based on looking at visitor based ones (which requires a bit of mental reasoning about what gets called when, and has more state machine like logic).

For schemaview based ones, it's really up to each individual generator how it implements things and documents its inner workings. Some make heavy use of jinja2 templates (which have definite advantages, but sometimes disadvatages). Some are fairly simple and just involve iterating through all elements and doing a simple transform. Some, such as owlgen and jsonschemagen involve more complex logic and rewrites in order to fully exploit the semantics of the underlying layer.

Longer term, I think that it might be useful to abstract some logic into the logicalmodel_transformer, which does symbolic reasoning over the schema, allowing simplifications and common rewrites (for example, when mapping to a framework that doesn't support inheritance, or only partial support, there are standard rewrites that preserve semantics as much as possible).

One possibility is also to reuse some of the linkml-transformer logic (as schemas can be data in LinkML), although that as yet is not intended for more complex schema mappings. In some ways this is the opposite of state machines as we conceive of the mapping declaratively rather than succession of states....

[sneakers-the-rat]

Maybe my question is more about unifying the behavior of the different generators - in working with the pydantic generator, I found that I had to override many parts of the class to make minor changes, since the logic of the generation process is distributed across many coupled method calls. Since each generator has so much independent logic, the outputs of each of the generators is inconsistent in which features of the language they support. Unifying that behavior would go a long way to making it possible to know what one will get out the other side of a generator, as well as how to patch into the generator to modify that process.

of course i'm not suggesting the generators should have sprung fully formed from the team's brains, iterating over what works is the only way to build something as complex as this. It is good to hear about the longer term plans for unifying the generators, and i'd like to pitch in on that how i can. One thing i have been working on in my fork of the pydantic generator has been modeling the target domain as well, so I can make a cleaner mapping from source LinkML properties to target pydantic properties. That might help make some of the transformations needed by different formats more apparent - eg. a given target format requires attributes to be unrolled vs. inherited could be a property of a target format that corresponds to a given transformation. Just spitballing.

Is this roadmap for the generators written down somewhere? if not it might be nice to have a bit of a roadmap to know where y'all are headed to be able to see how i could contribute to that effort :)