[Kernels] LoRA - Retire SGMV and BGMV Kernels

[varun-sundar-rabindranath]

Retire SGMV and BGMV LoRA kernels in favor of V1 kernels. This cleans up the tests and kernel dispatch logic.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[varun-sundar-rabindranath]

@jeejeelee This PR is good to go ! Can you take a pass at it when you get a chance please. Thanks ! 🙌

[jeejeelee]

Have you tested the impact of replacing these kernels on the performance of V0 lora, especially in cudagraph and eager modes?

[varun-sundar-rabindranath]

I did some minimal benchmarking for LoRA Rank ablations - https://docs.google.com/spreadsheets/d/1RMOnNc8sm5swWC-ZyI0x4LdmAfLJAoX30xWSi7p3Hng/edit?usp=sharing
I am running more benchmarks that covers more problem sizes. I'll share them when they are ready 👍

[chenhongyu2048]

@varun-sundar-rabindranath Hi bro, I'm confused about some items in the performance data table, such as single-lora roofline using torch.mm (f16xf16=>f16) and SGMV_EXPAND(add_inputs=True) (f32xf16=>f16).
What is single-lora roofline using torch.mm? and SGMV_EXPAND I guess it's the implementation of the previous V0?

[varun-sundar-rabindranath]

Hi @chenhongyu2048 ,

single-lora roofline using torch.mm

This is just the numbers for a single matmul. This is a very loose bound on good the LoRA kernel performance could be. Please take a look at

vllm/benchmarks/kernels/benchmark_lora.py

Line 604 in d5615af

def bench_torch_mm(ctx: BenchmarkContext,

SGMV_EXPAND I guess it's the implementation of the previous V0?

Yes.

[chenhongyu2048]

Thanks. I've tried benchmark_lora.py and confirmed the performance improvement relative to V0 LoRA in my use case (H20).

[jeejeelee]

QQ: What do these comments mean?

[varun-sundar-rabindranath]

For the cpu case, in punica_cpu.py, I see that we still dispatch to different kernels/operations based on if the tokens were all prefill or decode.

For the V1 kernels (now lora_kernels), we dont have this distinction. So the is_prefill variable in LoRAMapping is ignored.

[jeejeelee]

It seems that deleting these is due to CUDAgraph, is that right?

      #No LoRA request, so return directly
      if self.no_lora:
          return

[varun-sundar-rabindranath]

CUDAGraphs is not the main issue.
The issue is torch.compile. In V1, all model_execute runs are done via torch.compile and, IIRC it doesn't support dynamic control flow.
But, let me try re-introducing it and see if I can make it work, so I can record my findings here.

[varun-sundar-rabindranath]

I have PR here #15152 that re-introduces this flag in a way that would work for both V0 and V1.

[jeejeelee]

Could you plz merge with the main branch, so we can test the LoRA using torch==2.6.0