Fix broken tests #14713

jovsa · 2025-03-13T00:16:07Z

Fixed some of the broken tests when running pytest tests/

fix broken test_pass_manager tests
added imports for test_flashinfer so you can run test_fash_attn.py

github-actions · 2025-03-13T00:16:16Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

youkaichao · 2025-03-13T03:27:57Z

requirements/test.in

+runai-model-streamer-s3==0.11.0
+
+flashinfer-python==0.2.3 # required for flashinfer tests


this is installed in the docker. we need to build it. the pypi version is just source code. we cannot install it directly from pypi.

youkaichao · 2025-03-13T03:28:46Z

tests/compile/test_pass_manager.py

@ProExpertProg can you help take a look at this change to see if it is necessary?

houseroad

@zou3519 may give a pass as well.

ProExpertProg

Thanks for fixing these tests! Been on my TODO list for a while. Could you also add the test to CI?

ProExpertProg · 2025-03-14T21:51:57Z

tests/compile/test_pass_manager.py

+    if isinstance(callable, InductorPass):
+        pass_manager.add(callable)
+        # should succeed for proper InductorPass instances
+        if works:
            pickle.dumps(pass_manager)
+        else:
+            with pytest.raises(BypassFxGraphCache):
+                pickle.dumps(pass_manager)
+    else:


I think this is more complex than it needs to be. The outer if is not necessary, just update the works boolean if necessary

Thanks for the feedback!
Made the changes in this commit: 06b6d03

jovsa · 2025-03-15T06:23:51Z

Thanks for fixing these tests! Been on my TODO list for a while. Could you also add the test to CI?

Done here: b8d0be0

zou3519

seems reasonable to me but I'll defer to @ProExpertProg and @youkaichao

ProExpertProg

Upon closer inspection, I realized the test is very outdated. After I added it in #10273, we ended up drastically changing the caching logic. There is no more bypassing the code cache and throwing inside pickle.dumps. But there is an assertion in pass_manager.add. I added some suggestions to make it easier to make the change, and thanks again for doing this.

ProExpertProg · 2025-03-17T18:59:48Z

tests/compile/test_pass_manager.py

+callable_decorated = CallableInductorPass(simple_callable,
+                                          InductorPass.hash_source(__file__))


Nit: I would call this callable_uuid as the uuid is provided manually

Done here: c325dfa

tests/compile/test_pass_manager.py

* fix test_pass_manager tests * added imports for test_flashinfer Signed-off-by: JovanSardinha <jovan.sardinha@gmail.com>

Signed-off-by: JovanSardinha <jovan.sardinha@gmail.com>

Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com> Signed-off-by: JovanSardinha <jovan.sardinha@gmail.com>

Signed-off-by: JovanSardinha <jovan.sardinha@gmail.com>

ProExpertProg · 2025-03-19T19:38:17Z

@youkaichao I think the CI is failing because it's using torch 2.5 but the precompiled kernels it downloads use 2.6. Why is torch==2.5.1 being used?

We could also just force-merge and address later

ProExpertProg · 2025-03-19T20:10:21Z

@jovsa culd you rebase/merge main for the CI tests?

Signed-off-by: JovanSardinha <jovan.sardinha@gmail.com> Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>

Signed-off-by: JovanSardinha <jovan.sardinha@gmail.com> Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com> Signed-off-by: Louis Ulmer <ulmerlouis@gmail.com>

Signed-off-by: JovanSardinha <jovan.sardinha@gmail.com> Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>

mergify bot added the ci/build label Mar 13, 2025

jovsa marked this pull request as ready for review March 13, 2025 00:26

DarkLight1337 requested a review from youkaichao March 13, 2025 03:18

youkaichao reviewed Mar 13, 2025

View reviewed changes

houseroad reviewed Mar 14, 2025

View reviewed changes

ProExpertProg reviewed Mar 14, 2025

View reviewed changes

jovsa requested a review from ProExpertProg March 15, 2025 06:23

zou3519 reviewed Mar 17, 2025

View reviewed changes

ProExpertProg reviewed Mar 17, 2025

View reviewed changes

jovsa and others added 9 commits March 18, 2025 01:06

fix broken tests

da60c0e

* fix test_pass_manager tests * added imports for test_flashinfer Signed-off-by: JovanSardinha <jovan.sardinha@gmail.com>

updated requiremnts for flashinfer tests

9bcf599

Signed-off-by: JovanSardinha <jovan.sardinha@gmail.com>

removing flashinfer-python pypy

dc68d10

Signed-off-by: JovanSardinha <jovan.sardinha@gmail.com>

removing whitespace

541a4c6

Signed-off-by: JovanSardinha <jovan.sardinha@gmail.com>

simplified test_pass_manager logic

206939e

Signed-off-by: JovanSardinha <jovan.sardinha@gmail.com>

added compile/test_pass_manager.py to CI

412847d

Signed-off-by: JovanSardinha <jovan.sardinha@gmail.com>

Update tests/compile/test_pass_manager.py

48da152

Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com> Signed-off-by: JovanSardinha <jovan.sardinha@gmail.com>

Update tests/compile/test_pass_manager.py

6d053f0

Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com> Signed-off-by: JovanSardinha <jovan.sardinha@gmail.com>

changed callable_decorated to callable_uuid

684175d

Signed-off-by: JovanSardinha <jovan.sardinha@gmail.com>

jovsa force-pushed the jovsa/fixing-tests branch from c325dfa to 684175d Compare March 18, 2025 01:07

jovsa requested a review from ProExpertProg March 18, 2025 01:23

remove comments

6f840f1

Signed-off-by: JovanSardinha <jovan.sardinha@gmail.com>

ProExpertProg approved these changes Mar 18, 2025

View reviewed changes

youkaichao approved these changes Mar 19, 2025

View reviewed changes

youkaichao enabled auto-merge (squash) March 19, 2025 17:31

github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Mar 19, 2025

Merge branch 'vllm-project:main' into jovsa/fixing-tests

9f577cb

mgoin approved these changes Mar 19, 2025

View reviewed changes

Merge branch 'vllm-project:main' into jovsa/fixing-tests

95be8c3

youkaichao merged commit 70e500c into vllm-project:main Mar 20, 2025
59 checks passed

erictang000 pushed a commit to erictang000/vllm that referenced this pull request Mar 25, 2025

Fix broken tests (vllm-project#14713)

639d471

Signed-off-by: JovanSardinha <jovan.sardinha@gmail.com> Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>

gmarinho2 pushed a commit to gmarinho2/vllm that referenced this pull request Apr 1, 2025

Fix broken tests (vllm-project#14713)

1db234f

Signed-off-by: JovanSardinha <jovan.sardinha@gmail.com> Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>

nishith-fujitsu pushed a commit to nishith-fujitsu/vllm that referenced this pull request Apr 9, 2025

Fix broken tests (vllm-project#14713)

1387670

Signed-off-by: JovanSardinha <jovan.sardinha@gmail.com> Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>

ckhordiasma mentioned this pull request Apr 17, 2025

[do not merge] pr test for nm changes into 2.20 red-hat-data-services/vllm#107

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix broken tests #14713

Fix broken tests #14713

jovsa commented Mar 13, 2025 •

edited by github-actions bot

Loading

github-actions bot commented Mar 13, 2025

youkaichao Mar 13, 2025

youkaichao Mar 13, 2025

houseroad left a comment

ProExpertProg left a comment

ProExpertProg Mar 14, 2025

jovsa Mar 15, 2025

jovsa commented Mar 15, 2025

zou3519 left a comment

ProExpertProg left a comment

ProExpertProg Mar 17, 2025

jovsa Mar 18, 2025

ProExpertProg commented Mar 19, 2025

ProExpertProg commented Mar 19, 2025

		runai-model-streamer-s3==0.11.0

		flashinfer-python==0.2.3 # required for flashinfer tests

		callable_decorated = CallableInductorPass(simple_callable,
		InductorPass.hash_source(__file__))

Fix broken tests #14713

Fix broken tests #14713

Conversation

jovsa commented Mar 13, 2025 • edited by github-actions bot Loading

github-actions bot commented Mar 13, 2025

youkaichao Mar 13, 2025

Choose a reason for hiding this comment

youkaichao Mar 13, 2025

Choose a reason for hiding this comment

houseroad left a comment

Choose a reason for hiding this comment

ProExpertProg left a comment

Choose a reason for hiding this comment

ProExpertProg Mar 14, 2025

Choose a reason for hiding this comment

jovsa Mar 15, 2025

Choose a reason for hiding this comment

jovsa commented Mar 15, 2025

zou3519 left a comment

Choose a reason for hiding this comment

ProExpertProg left a comment

Choose a reason for hiding this comment

ProExpertProg Mar 17, 2025

Choose a reason for hiding this comment

jovsa Mar 18, 2025

Choose a reason for hiding this comment

ProExpertProg commented Mar 19, 2025

ProExpertProg commented Mar 19, 2025

jovsa commented Mar 13, 2025 •

edited by github-actions bot

Loading