GitHub stars · StarDay

1ollama / ollamaRun and manage language models locally.GoMIT · Permissive186,067

2openai / whisperProvide speech-recognition models and audio-transcription code.PythonMIT · Permissive111,787

3QwenLM / Qwen3Collect usage information and model entry points for the Qwen3 language-model family.PythonLicense not declared27,715

4liguodongiot / llm-actionThis project aims to share the technical principles and practical experience related to large models (large model engineering, large model application implementation)HTMLApache-2.0 · Permissive25,259

5jundot / omlxLLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the macOS menu barPythonApache-2.0 · Permissive23,405

6NVIDIA / TensorRT-LLMTensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.PythonLicense not declared15,025

7bentoml / OpenLLMRun any open-source LLMs, such as DeepSeek and Llama, as OpenAI compatible API endpoint in the cloud.PythonApache-2.0 · Permissive12,583

8skypilot-org / skypilotThe AI Compute Platform for frontier teams. SkyPilot turns fragmented AI compute into one AI supercomputer, so frontier AI teams build custom intelligence faster.PythonApache-2.0 · Permissive10,802

9bentoml / BentoMLThe easiest way to serve AI apps and models - Build Model Inference APIs, Job queues, LLM apps, Multi-model pipelines, and more!PythonApache-2.0 · Permissive8,935

10flyteorg / flyteType-safe, distributed orchestration of agents, ML pipelines, and real-time inference on your k8s — in pure Python with async/await, also other languages (rust, go and ts)GoApache-2.0 · Permissive7,842

11vllm-project / vllm-omniA framework for efficient model inference with omni-modality modelsPythonApache-2.0 · Permissive7,572

12bojieli / ai-infra-book"In-depth Understanding of AI Infra: Quantitative Analysis and System Design" (authored by Li Bojie) open source manuscript: Starting from hardware constraints and model architecture, quantitatively derive the design of LLM inference and training system. Includes the full text of the book, PDF, supporting calculation tools and experimentsPythonApache-2.0 · Permissive7,348

13kserve / kserveStandardized Distributed Generative and Predictive AI Inference Platform for Scalable, Multi-Framework Deployment on KubernetesGoApache-2.0 · Permissive6,195

14Michael-A-Kuykendall / shimmy⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary.RustApache-2.0 · Permissive6,008

15gpustack / gpustackA GPU cluster manager for high-performance AI model serving (vLLM, SGLang) and on-demand SSH-accessible GPU instances.PythonApache-2.0 · Permissive5,984

16beclab / OlaresOpen-Source Personal Cloud OS for Always-On AgentsGoAGPL-3.0 · Share derivative source5,348

17superduper-io / superduperSuperduper: End-to-end framework for building custom AI applications and agents.PythonApache-2.0 · Permissive5,330

18HuaizhengZhang / AI-Infra-from-Zero-to-Hero🚀 Awesome System for Machine Learning ⚡️ AI System Papers and Industry Practice. ⚡️ System for Machine Learning, LLM (Large Language Model), GenAI (Generative AI). 🍻 OSDI, NSDI, SIGCOMM, SoCC, MLSys, etc. 🗃️ Llama3, Mistral, etc. 🧑‍💻 Video Tutorials.MIT · Permissive4,460

19ahkarami / Deep-Learning-in-ProductionIn this repository, I will share some useful notes and references about deploying deep learning-based models in production.License not declared4,378

20ModelTC / LightLLMLightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its lightweight design, easy scalability, and high-speed performance.PythonApache-2.0 · Permissive4,357