Update to torch==2.6.0

[mgoin]

Only updates for CUDA. Successfully built locally on H100 CUDA 12.5 system and tested with vllm serve meta-llama/Llama-3.1-8B-Instruct

We should upgrade other hardware backends separately. For instance, CPU is blocked by IPEX in the Dockerfile.cpu

FIX #12719

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[tlrmchlsmth]

Nice, CI looks green

[houseroad]

Shall we merge #12393 first? cc: @youkaichao

[fialhocoelho]

LGTM. I built vLLM by merging this PR, and it worked perfectly 🚀

[mgoin]

Confirmed that this update will break V1 at the current state, we should wait for #12393 at least

VLLM_USE_V1=1 vllm serve meta-llama/Llama-3.1-8B-Instruct
...
ERROR 02-04 15:27:21 core.py:210]   File "/home/mgoin/code/vllm/vllm/compilation/backends.py", line 616, in __call__
ERROR 02-04 15:27:21 core.py:210]     PiecewiseCompileInterpreter(self.split_gm, submod_names_to_compile,
ERROR 02-04 15:27:21 core.py:210]   File "/home/mgoin/code/vllm/vllm/compilation/backends.py", line 424, in run
ERROR 02-04 15:27:21 core.py:210]     return super().run(*fake_args)
ERROR 02-04 15:27:21 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-04 15:27:21 core.py:210]   File "/home/mgoin/venvs/vllm/lib/python3.12/site-packages/torch/fx/interpreter.py", line 167, in run
ERROR 02-04 15:27:21 core.py:210]     self.env[node] = self.run_node(node)
ERROR 02-04 15:27:21 core.py:210]                      ^^^^^^^^^^^^^^^^^^^
ERROR 02-04 15:27:21 core.py:210]   File "/home/mgoin/venvs/vllm/lib/python3.12/site-packages/torch/fx/interpreter.py", line 230, in run_node
ERROR 02-04 15:27:21 core.py:210]     return getattr(self, n.op)(n.target, args, kwargs)
ERROR 02-04 15:27:21 core.py:210]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 02-04 15:27:21 core.py:210]   File "/home/mgoin/code/vllm/vllm/compilation/backends.py", line 439, in call_module
ERROR 02-04 15:27:21 core.py:210]     compiled_graph_for_general_shape = wrap_inductor(
ERROR 02-04 15:27:21 core.py:210]                                        ^^^^^^^^^^^^^^
ERROR 02-04 15:27:21 core.py:210]   File "/home/mgoin/code/vllm/vllm/compilation/backends.py", line 254, in wrap_inductor
ERROR 02-04 15:27:21 core.py:210]     original_load = FxGraphCache.load
ERROR 02-04 15:27:21 core.py:210]                     ^^^^^^^^^^^^^^^^^
ERROR 02-04 15:27:21 core.py:210] torch._dynamo.exc.BackendCompilerFailed: backend='<vllm.compilation.backends.VllmBackend object at 0x71985bc685c0>' raised:
ERROR 02-04 15:27:21 core.py:210] AttributeError: type object 'FxGraphCache' has no attribute 'load'
ERROR 02-04 15:27:21 core.py:210] 
ERROR 02-04 15:27:21 core.py:210] While executing %submod_0 : [num_users=5] = call_module[target=submod_0](args = (%l_input_ids_, %s0, %l_self_modules_embed_tokens_parameters_weight_, %l_self_modules_layers_modules_0_modules_input_layernorm_parameters_weight_, %l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_parameters_weight_, %l_positions_, %l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache_), kwargs = {})

[youkaichao]

@mgoin can you help review and stamp that PR?

[zhouyuan]

@mgoin Thanks a lot for the update. IPEX CPU w/ PT 2.6 will be released next week. Will update on this as soon as the binary is out.

Cc: @Guobing-Chen @bigPYJ1151

Thanks, -yuan

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @mgoin.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[jiangshaoping]

I wanna when this PR will be merged？

[mgoin]

I added the enable_auto_functionalized_v2 guard and merged with main, so let's see where the CI ends up

[mgoin]

I kicked off a manual build to get past the timeout for docker build. https://buildkite.com/vllm/ci/builds/15327/table

Unfortunately there are a few errors left that seem related to the upgrade. I can't look into this right now so happy for others to contribute

[DarkLight1337]

LoRA and multi-modal tests should be fixed on main, let's see what errors are left.

[DarkLight1337]

There seems to be an import error in bitsandbytes: https://buildkite.com/vllm/ci/builds/15327#01958c9b-93f5-4350-aa81-c3dcf079bfe7

[ProExpertProg]

Yeah looking into it, it seems that triton==3.2 does not have triton.ops whereas 3.1 does.

[tlrmchlsmth]

Possibly good to go now?? 🤞 🤞

edit: of course not -- I'll fix the pre-commit

[xihuai18]

hi, How can I build vllm using torch 2.5.1 after this PR? Was there anyone succeeded?

[ProExpertProg]

hi, How can I build vllm using torch 2.5.1 after this PR? Was there anyone succeeded?

Can you try pip install -e . --no-build-isolation in an environment with torch==2.5.1 already installed?

[xihuai18]

hi, How can I build vllm using torch 2.5.1 after this PR? Was there anyone succeeded?

Can you try pip install -e . --no-build-isolation in an environment with torch==2.5.1 already installed?

I am trying:

git clone https://github.com/vllm-project/vllm.git
cd vllm
python use_existing_torch.py
pip install -r requirements/build.txt
pip install -e . --no-build-isolation
``

[ProExpertProg]

Are you getting an error? You might need to downgrade other dependencies as well, that would be my only other guess.

[xihuai18]

Are you getting an error? You might need to downgrade other dependencies as well, that would be my only other guess.

I am building wheels for torch 2.5.1, but I meet many errors. I hope vllm could officially provide wheels for torch2.5.1 since torch2.6.0 would lead to many dependence problems when using vllm with some integrations such as verl or ms-swift.

[ProExpertProg]

Could you create a new issue and post the errors? I don't think providing official 2.5.1 wheels is on the roadmap for v0.8.0+. But you're welcome to use an earlier version or cherry pick the commits you need.