The ultimate test and compliance suite

[cmungall]

The linkml repo includes many tests for different aspects of LinkML - generators, validation of example data files using various schema constructs, ...

• the test suite has grown organically and it's not easy to find if a particular schema construct is explicitly tested in any given context
• it it heavily tied to python and would need to be rewritten when developing linkml frameworks for other languages

Ideally it would be very easy to navigate from any given schema construct (e.g. any_of; or use of combinations of is_a and slot_usage), and see very clearly:

• informative examples of data instances that are expected to pass or fail validation using those constructs
• whether a particular generator (in a particular language - java, python, etc) intends to supports this in part or in full
• whether that generator actually supports this, i.e. does the accompany unit test pass or fail

This would help with one of the main barriers people face - it's not always obvious that for example json-schema generation doesn't and can't support referential integrity, or that all aspects of the language can be supported in python.

This compliance suite would be complementary to the existing linkml unit tests, but there may be some redundancy. It may make sense for this to live in the linkml-model repo, since it is part of the language specification.

I can see a few ways of doing this:

1. manually authoring an example schema for each construct plus example pass/fail instances, and expected outputs of generators, and regularly running against this
2. doing this using pytest, but rather than simply testing, also generate all intermediate products (schemas, example files), plus markdown, and pushing this to the website, so everything is fully transparent and accessible to non-core developers
3. some kind of hybrid, writing these as Jupyter notebooks, executed as part of CI using papermill

The first approach is the most straightforward, but may involve a lot of repetitive boilerplate. It could also be hard to refactor if we want to do something like change conventions or layout.

The second approach has the advantage that it's easy to programmatically generate different combinations (as a trivial example, to test validation of simple literal types like int/string/float, this could be done with a single parameterized pytest function). It fits will into our existing structure. And it should not be hard to come up with some routines and conventions to generate publishable output.

The jupyter approach has certain advantages, it's easy to extend, and would also work well as a general educational tool on how to use the linkml framework programmatically. From a software engineering perspective it seems odd to use a notebook as part of testing.

[cmungall]

This is now supported. https://github.com/linkml/linkml/tree/main/tests/test_compliance

We would still like to generate a markdown page summarizing the results of running the suite