A high-throughput and memory-efficient inference and serving engine for LLMs (Windows build & kernels)

Language · Python

License · Apache-2.0

Ask anything about this repo to start.