Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
-
Updated
Jun 1, 2024 - Python
Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
A universal scalable machine learning model deployment solution
Serve, optimize and scale PyTorch models in production
AI + Data, online. https://vespa.ai
A flexible, high-performance serving system for machine learning models
An MLOps framework to package, deploy, monitor and manage thousands of production machine learning models
A multi-modal vector database that supports upserts and vector queries using unified SQL (MySQL-Compatible) on structured and unstructured data, while meeting the requirements of high concurrency and ultra-low latency.
A scalable inference server for models optimized with OpenVINO™
A high-performance inference system for large language models, designed for production environments.
RayLLM - LLMs on Ray
Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon
Friendli: the fastest serving engine for generative AI
An Alternative for Triton Inference Server. Boosting DL Service Throughput 1.5-4x by Ensemble Pipeline Serving with Concurrent CUDA Streams for PyTorch/LibTorch Frontend and TensorRT/CVCUDA, etc., Backends
Docs for torchpipe: https://github.com/torchpipe/torchpipe
In this repository, I will share some useful notes and references about deploying deep learning-based models in production.
Database system for AI-powered apps
A REST API for vLLM, production ready
A flexible, high-performance carrier for machine learning models(『飞桨』服务化部署框架)
ClearML - Model-Serving Orchestration and Repository Solution
Add a description, image, and links to the serving topic page so that developers can more easily learn about it.
To associate your repository with the serving topic, visit your repo's landing page and select "manage topics."