Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

💡 maven schema-registry:register all #1914

Open
mac2000 opened this issue Jun 15, 2021 · 10 comments · May be fixed by #1915
Open

💡 maven schema-registry:register all #1914

mac2000 opened this issue Jun 15, 2021 · 10 comments · May be fixed by #1915

Comments

@mac2000
Copy link

mac2000 commented Jun 15, 2021

In case if we do not want to have all 100+ schemas listed in the subjects section of the plugin configuration

Will be so nice if the plugin will just use everything it can find under outputDirectory if the subjects section missing

image

mac2000 added a commit to mac2000/schema-registry that referenced this issue Jun 15, 2021
mac2000 added a commit to mac2000/schema-registry that referenced this issue Jun 15, 2021
@OneCricketeer
Copy link
Contributor

Similar to #690

@OneCricketeer
Copy link
Contributor

One problem with this is that AVDL record schemas cannot have hyphens, so using TopicNameStrategy registration isn't possible

@mac2000
Copy link
Author

mac2000 commented Jun 16, 2021

@OneCricketeer not sure if I got you right, but after downloading I got a bunch of TopicA-key.avcs, TopicA-value.avcs files, so not sure about AVDL files but, if starting from scratch everything should work fine.

@OneCricketeer
Copy link
Contributor

IDL files generate AVSC and can more easily be used for cross-referenced schemas with import support and composition

https://avro.apache.org/docs/current/idl.html

I'm talking about compilation / resource processing via the Avro Maven plugin, not related to the Registry

@ena1106
Copy link

ena1106 commented Nov 9, 2021

Using TopicRecordNameStrategy the list of subject-schema mapping is not just long but also unreadable.

<schemaTypes>
     <my-topic-com.mycompany.avro.schemas.test.Event1>AVRO</my-topic-com.mycompany.avro.schemas.test.Event1>
     <my-topic-com.mycompany.avro.schemas.test.Event2>AVRO</my-topic-com.mycompany.avro.schemas.test.Event2>
</schemaTypes>

It would also be nice to have a schemaType like AVRO-IDL to check schema compatibility from idl files.
There is no official maven plugin to convert from avdl files to avsc.
The avro-maven-plugin with goal idl-protocol lets you generate classes directly from avdl files. With this setup you don't have avsc files in the project.

Given these two limitations (long mapping list and lack of avdl support) I think this plugin is unusable in a production environment.

Please tell me if I am missing something.

@OneCricketeer
Copy link
Contributor

The idl-protocol goal does create AVSC files.

In the past, we used it with the exec plugin to define a class and maven goal that iterates over AVDL or AVSC and registers them to the registry

@ena1106
Copy link

ena1106 commented Nov 9, 2021

Can you please tell me where are the AVSC files generated by the idl-protocol goal? I can only see classes under target/generated-sources/avro/classes

Maybe the exec plugin was getting the schemas from the classes? The json string is present in a static variable of generated classes.

@OneCricketeer
Copy link
Contributor

Well within each class, there's a static SCHEMA$ field that has that information.

But this is what I was referring to http://www.virtualroadside.com/blog/index.php/2014/06/08/automatically-generating-avro-schemata-avsc-files-using-maven/

@ena1106
Copy link

ena1106 commented Nov 10, 2021

I already read that blog post. They generate avsc files with maven-exec and Avro tools library, not using the idl-protocol goal of the avro-maven-plugin.

@OneCricketeer
Copy link
Contributor

OneCricketeer commented Nov 10, 2021

Does that matter? I just couldn't remember which plugins we used. Based on what I have posted, I think we used the exec plugin followed by the avro plugin's schema goal to generate classes, then you can also use Schema Registry maven plugin to register those AVSC as well

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants