git
ask
hub
Privacy
Terms
Sign in with GitHub
haiquanlu/mix-quant
↗
Stars ·
36
Language ·
Python
Simplify and Visualize This Repo
Scan the safety of this repo
Self Host this repo
Check out more work of Developer: haiquanlu
Ask anything about this repo to start.
Install Mix-Quant in a conda env with the forked vLLM submodule and run the example proxy
Launch the three-process server with NVFP4 prefilling on GPU 0 and BF16 decoding on GPU 1
Benchmark Mix-Quant on LongBench-v2 against plain BF16 vLLM and compare throughput
Swap the proxy with a custom router that picks decoding precision per request length
Adapt the disaggregated pattern to use INT8 prefilling instead of NVFP4
Full explanation on explaingit →
📎
Send
By chatting or signing in you agree to the
Terms
and chat-message logging (revocable in
History
).