[Quantization] Fp8 Channelwise Dynamic Per Token GroupedGEMM #15587

robertgshaw2-redhat · 2025-03-27T01:45:29Z

SUMMARY:

follow on to @ElizaWszola's CUTLASS FP8 Grouped GEMM
enables support for channelwise scales and per token activations

LM EVAL:

channelwise, dynamic per token

lm-eval --model vllm --model_args "pretrained=nm-testing/Mixtral-8x7B-Instruct-v0.1-FP8" --tasks gsm8k --batch_size "auto"

> |Tasks|Version|     Filter     |n-shot|  Metric   |   |Value|   |Stderr|
> |-----|------:|----------------|-----:|-----------|---|----:|---|-----:|
> |gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.655|±  |0.0131|
> |     |       |strict-match    |     5|exact_match|↑  |0.652|±  |0.0131|

static per tensor

lm-eval --model vllm --model_args "pretrained=nm-testing/Mixtral-8x7B-Instruct-v0.1-FP8" --tasks gsm8k --batch_size "auto"

> |Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
> |-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
> |gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.6406|±  |0.0132|
> |     |       |strict-match    |     5|exact_match|↑  |0.6376|±  |0.0132|

Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

Co-authored-by: Lucas Wilkinson <wilkinson.lucas@gmail.com> Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

…of tensors Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

Signed-off-by: ElizaWszola <ewszola@redhat.com>

Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

Signed-off-by: ElizaWszola <ewszola@redhat.com>

Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>

github-actions · 2025-03-27T01:45:38Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>

robertgshaw2-redhat · 2025-03-27T01:51:26Z

vllm/model_executor/layers/fused_moe/layer.py

@@ -884,32 +884,6 @@ def make_expert_params_mapping(
            ]
        ]

-    def _load_fp8_scale(self, param: torch.nn.Parameter,


this is dead code (its not used in the codebase)

tlrmchlsmth · 2025-03-27T03:01:23Z

vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe.py

+        per_tensor = (self.weight_quant.strategy == QuantizationStrategy.TENSOR
+                      and self.input_quant.strategy
+                      == QuantizationStrategy.TENSOR)
+        per_channel = (
+            self.weight_quant.strategy == QuantizationStrategy.CHANNEL
+            and self.input_quant.strategy == QuantizationStrategy.TOKEN)


we can mix and match

ok, will change this in a FUP

…oject#15587) Signed-off-by: ElizaWszola <eliza@neuralmagic.com> Signed-off-by: ElizaWszola <ewszola@redhat.com> Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Co-authored-by: ElizaWszola <eliza@neuralmagic.com> Co-authored-by: Lucas Wilkinson <wilkinson.lucas@gmail.com> Co-authored-by: ElizaWszola <ewszola@redhat.com>

…oject#15587) Signed-off-by: ElizaWszola <eliza@neuralmagic.com> Signed-off-by: ElizaWszola <ewszola@redhat.com> Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Co-authored-by: ElizaWszola <eliza@neuralmagic.com> Co-authored-by: Lucas Wilkinson <wilkinson.lucas@gmail.com> Co-authored-by: ElizaWszola <ewszola@redhat.com> Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>

…oject#15587) Signed-off-by: ElizaWszola <eliza@neuralmagic.com> Signed-off-by: ElizaWszola <ewszola@redhat.com> Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Co-authored-by: ElizaWszola <eliza@neuralmagic.com> Co-authored-by: Lucas Wilkinson <wilkinson.lucas@gmail.com> Co-authored-by: ElizaWszola <ewszola@redhat.com> Signed-off-by: xinyuxiao <xinyuxiao2024@gmail.com>

…oject#15587) Signed-off-by: ElizaWszola <eliza@neuralmagic.com> Signed-off-by: ElizaWszola <ewszola@redhat.com> Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Co-authored-by: ElizaWszola <eliza@neuralmagic.com> Co-authored-by: Lucas Wilkinson <wilkinson.lucas@gmail.com> Co-authored-by: ElizaWszola <ewszola@redhat.com> Signed-off-by: Louis Ulmer <ulmerlouis@gmail.com>

…oject#15587) Signed-off-by: ElizaWszola <eliza@neuralmagic.com> Signed-off-by: ElizaWszola <ewszola@redhat.com> Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com> Co-authored-by: ElizaWszola <eliza@neuralmagic.com> Co-authored-by: Lucas Wilkinson <wilkinson.lucas@gmail.com> Co-authored-by: ElizaWszola <ewszola@redhat.com>

ElizaWszola and others added 30 commits December 6, 2024 14:36

Cutlass grouped gemm files

1825ef8

Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

runs, bad result

5fd48e5

Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

A little closer to working

d5942cf

Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

Working for identical sizes

c570c69

Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

Grouped gemm working

6ed63f2

Co-authored-by: Lucas Wilkinson <wilkinson.lucas@gmail.com> Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

Small cleanup

e2b1fc0

Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

Merge branch 'main' into grouped-gemm-with-group-id

dd163f5

Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

Benchmark grouped cutlass against bfloat16 torch.mm

acfd3ef

Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

Merge branch 'main' into grouped-gemm-with-group-id

c6231b6

Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

Start working on fused moe cutlass implementation

f1a5666

Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

Working halfway

6414e31

Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

working mul test but the topk_weights are not yet included in kernel

67e2dd4

Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

cleaned up cutlass moe test, fixes

6523529

Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

benchmark fused

b302d98

Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

pass input as one tensor with an array of offsets rather than a list …

342d1a4

…of tensors Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

Using tensors rather than tensor lists works with test_cutlass test

7549e3d

Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

Merge branch 'main' into grouped-gemm-with-group-id

64c2a68

Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

cleanup, add import

1ea7874

Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

working fused op

d608164

Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

benchmark, create strides directly on device, small name refactor

286f6c8

Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

works with cuda graphs

b6867bb

Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

move stride tensor creation outside c++ code, cleanup

df04bc0

Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

cleanup benchmark

88c7134

Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

profile

02e1d4e

Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

tuned shapes, fix

1d9c429

Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

Merge branch 'main' into grouped-gemm-with-group-id

b824ad2

Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

Performance, add channelwise scales everywhere

ae90eee

Signed-off-by: ElizaWszola <ewszola@redhat.com>

name fix

f191b35

Signed-off-by: ElizaWszola <ewszola@redhat.com>

Merge branch 'main' into grouped-gemm-with-group-id

22d4f7b

perf improvements in data preparation

51941ff

Signed-off-by: ElizaWszola <ewszola@redhat.com>

ElizaWszola and others added 13 commits March 19, 2025 14:45

format

f0c2f06

Signed-off-by: ElizaWszola <ewszola@redhat.com>

format 3

8d0e700

Signed-off-by: ElizaWszola <ewszola@redhat.com>

Add hack for accepting int input in weak_ref_tensors

84dbc2a

Signed-off-by: ElizaWszola <eliza@neuralmagic.com>

Fixes

5ad4b0b

Signed-off-by: ElizaWszola <ewszola@redhat.com>

format utils.py

41eb522

Signed-off-by: ElizaWszola <ewszola@redhat.com>

Merge branch 'main' into grouped-gemm-with-group-id

f5b5c7d

Signed-off-by: ElizaWszola <ewszola@redhat.com>

Make handling of both input scales consistent in the code

c6076b3

Signed-off-by: ElizaWszola <ewszola@redhat.com>

Fix handling optional vals

c8f1567

Signed-off-by: ElizaWszola <ewszola@redhat.com>

feedback: version checks, file structure

96296cb

Signed-off-by: ElizaWszola <ewszola@redhat.com>

Change cmake flag, remove unused code

3977d67

Signed-off-by: ElizaWszola <ewszola@redhat.com>

update kernel run conditions in scaled_mm_entry.cu

83ee170

Signed-off-by: ElizaWszola <ewszola@redhat.com>

added channelwise, dynamic per token

fbe2b80

Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>

updated

e0af782

Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>

robertgshaw2-redhat requested review from mgoin and tlrmchlsmth as code owners March 27, 2025 01:45

mergify bot added the documentation Improvements or additions to documentation label Mar 27, 2025

updated

e0bae3c

Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>

robertgshaw2-redhat commented Mar 27, 2025

View reviewed changes

tlrmchlsmth reviewed Mar 27, 2025

View reviewed changes

tlrmchlsmth approved these changes Mar 27, 2025

View reviewed changes

robertgshaw2-redhat enabled auto-merge (squash) March 27, 2025 03:34

github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Mar 27, 2025

robertgshaw2-redhat merged commit 43ed414 into vllm-project:main Mar 27, 2025
44 of 46 checks passed

ckhordiasma mentioned this pull request Apr 17, 2025

[do not merge] pr test for nm changes into 2.20 red-hat-data-services/vllm#107

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Quantization] Fp8 Channelwise Dynamic Per Token GroupedGEMM #15587

[Quantization] Fp8 Channelwise Dynamic Per Token GroupedGEMM #15587

robertgshaw2-redhat commented Mar 27, 2025 •

edited by github-actions bot

Loading

github-actions bot commented Mar 27, 2025

robertgshaw2-redhat Mar 27, 2025 •

edited

Loading

tlrmchlsmth Mar 27, 2025

robertgshaw2-redhat Mar 27, 2025

[Quantization] Fp8 Channelwise Dynamic Per Token GroupedGEMM #15587

[Quantization] Fp8 Channelwise Dynamic Per Token GroupedGEMM #15587

Conversation

robertgshaw2-redhat commented Mar 27, 2025 • edited by github-actions bot Loading

github-actions bot commented Mar 27, 2025

robertgshaw2-redhat Mar 27, 2025 • edited Loading

Choose a reason for hiding this comment

tlrmchlsmth Mar 27, 2025

Choose a reason for hiding this comment

robertgshaw2-redhat Mar 27, 2025

Choose a reason for hiding this comment

robertgshaw2-redhat commented Mar 27, 2025 •

edited by github-actions bot

Loading

robertgshaw2-redhat Mar 27, 2025 •

edited

Loading