[Frontend] support image embeds

[chaunceyjiang]

Fix #13540

prompt = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": [
        {
            "type": "image_embeds",
            "image_embeds": {
                    "image_embeds": "{base64_image_embedding}" ,
                    "image_sizes": ...        # Required by openbmb/MiniCPM-V-2_6
                    "image_grid_thw":  {base64}   # Required by Qwen/Qwen2-VL-2B-Instruct

                },
        }],
    },
]


or

prompt = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": [
        {
            "type": "image_embeds",
            "image_embeds": "{base64_image_embedding}" 
        }],
    },
]

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[DarkLight1337]

After merging #14017, can you update the Multimodal Inputs documentation page with an example on how to pass embedding inputs in online inference? Thanks

[DarkLight1337]

I think we should not use data URL to pass the image embeddings. You can directly pass the binary array data to be decoded on server side (similar to the format of input_audio rather than audio_url).

[gyin94]

@chaunceyjiang thanks for adding this! may I ask whether this change is also compatible with MultiModalHasher?

vllm/vllm/multimodal/hasher.py

Line 24 in 084bbac

class MultiModalHasher:

when we enable prefix caching, the image_embeds shall be hashable as well?

[DarkLight1337]

@chaunceyjiang thanks for adding this! may I ask whether this change is also compatible with MultiModalHasher?

vllm/vllm/multimodal/hasher.py

Line 24 in 084bbac

class MultiModalHasher:

when we enable prefix caching, the image_embeds shall be hashable as well?

The point of prefix caching is to avoid recomputing the embeddings in the visual encoder. There is no point in caching when those embeddings are already provided by the input.

[DarkLight1337]

Btw there is no need to force-push. We squash-merge PRs anyway.

[tienanh28122000]

Hi @chaunceyjiang ,

Thank you for your work on enabling image embeds in the Online Serving setup. Building on your commit, I have extended the implementation to support audio embeds in vLLM.

However, I’ve encountered an issue in multi-turn conversations. Currently, audio embeddings do not persist across turns. For example, in the following scenario:

chat_completion = client.chat.completions.create(
    messages=[
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": [
        {
            "type": "text",
            "text": "Answer this audio question",
        },
        embeds_1,
        ],
    },
    {"role": "assistant", "content": "My name is Qwen"},
    {"role": "user", "content": [
        {
            "type": "text",
            "text": "Answer this audio question",
        },
        embeds_2,
        ],
    }
],
    model=model,
)

The model does not properly handle multiple audio_embeds in one message. Given that the model requires specific parameters and does not allow extra parameters (e.g., it only accepts audio_embeds param), do you have any suggestions on how to properly support audio embeddings in multi-turn interactions?

Looking forward to your insights!

[DarkLight1337]

To support this, I guess we need to remove the restriction of only allowing one message to have embeddings...

[chaunceyjiang]

The model does not properly handle multiple audio_embeds in one message.

@tienanh28122000 Yes, there is such a restriction currently; the specific reason can be referred to at #13955 (comment).

[chaunceyjiang]

@DarkLight1337 @ywang96 I think this PR is almost done, could you please help add a ready label?

[DarkLight1337]

I'm fine with this PR as an initial step. Do you plan to remove the restriction in a separate PR?

[chaunceyjiang]

Do you plan to remove the restriction in a separate PR?

Yes, I may need some time to figure out how to remove this restriction.

[DarkLight1337]

Sounds good, let's merge this first then.

[gyin94]

@chaunceyjiang @DarkLight1337 thanks for adding this change. may I ask when we can allow multi image embeddings in one message?