gitaskhub

TurboQuant: Near-optimal KV cache quantization for LLM inference (3-bit keys, 2-bit values) with Triton kernels + vLLM integration

Stars · 1,716
Language · Python
License · GPL-3.0
Ask anything about this repo to start.

By chatting or signing in you agree to the Terms and chat-message logging (revocable in History).