microsoft / MInference
[NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.
- Repository stars
- 1,232All-time #35,563
- Fork
- 85
- Last push
- updated 28 days ago
- Repository created
- since 2024
About
[NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.Summarised by AI from the README; the original prevails
Author description: [NeurIPS'24 Spotlight, ICLR'25, ICML'25] To speed up Long-context LLMs' inference, approximate and dynamic sparse calculate the attention, which reduces inference latency by up to 10x for pre-filling on an A100 while maintaining accuracy.
Basic listing based on official public information; functionality has not been independently evaluated.
- Pricing
- Pricing not verifiedCheck current official software, hosting, model and infrastructure pricing.