Announcing AI Gateway

[subnetmarco]

Hello everyone 👋 , today we announced a set of new AI plugins in the broader Kong Gateway 3.6 release. You can read the full AI announcement here.

Essentially we can now use Kong Gateway to proxy AI traffic to cloud and self-hosted LLMs with a whole set of L7 AI Capabilities when it comes to prompts, observability and request/response transformation, and more. All other plugins can now be also used on top of upstream AI services to establish more advanced governance, security and traffic control patterns.

You can learn more about the new AI capabilities and even see demonstration videos at the official product page.

I am opening this discussion thread to collect feedback and ideas around this new AI use-case, and looking for feedback and suggestions from the community.

[kandakji]

Hi Marco @subnetmarco , this is great. We'd be happy to contribute to this AI proxy to add support for Amazon Bedrock invoke model API in order to add it to the list of providers you support

[subnetmarco]

That would be awesome to have.

[tysoekong]

Hi @kandakji , I built the initial range of provider integrations to support this AI category.

Would really like to meet, so we can plan out adding support for the Bedrock Inference format! We want to enable as many providers and model variety as possible.