[Feature Request] How can we conduct tests by integrating software like sglang or vllm ?

[BigCousin-z]

Feature Request

How can we conduct tests by integrating software like sglang or vllm ?

Motivation

How can we conduct tests by integrating software like sglang or vllm ?

Your Contribution

How can we conduct tests by integrating software like sglang or vllm ?

[Hanyuezhuohua]

Thank you for your interest in our project. Our primary contribution is a Triton kernel for the NSA layer, while sglang and vllm focus on accelerating LLMs. You can integrate our NSA layer by replacing the attention component in your LLMs, enabling efficient serving.

Please note that we are still developing and optimizing our kernel, so official integration with efficient serving engines will be released at a later date.

[BigCousin-z]

Thank you very much for your reply. May I ask which large models support the NSA technology? Does a model like DeepSeek R1 support it or others ?

[Hanyuezhuohua]

Thank you very much for your reply. May I ask which large models support the NSA technology? Does a model like DeepSeek R1 support it or others?

Actually, no open-source model now supports NSA technology, as it is a new model architecture, where the hardware implementation requires a group size of no less than 16 for the Grouped Query Attention. But I guess there will be more model training in a similar manner later.