[Feature]: Implement Concurrent Partial Prefills In V1 Engine

[robertgshaw2-redhat]

🚀 The feature, motivation and pitch

In V0, we support concurrent partial prefills to avoid TTFT latency with long requests. Implement it in V1

cc @WoosukKwon

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[plops655]

I can take this up.

[ccw1996]

I'm interested in contributing this.

[houseroad]

Shall we mark this as done? Since I feel in the context of chunked prefill, probably doesn't make sense to add support for max_num_partial_prefills and max_long_partial_prefills. cc: @comaniac

[comaniac]

Shall we mark this as done? Since I feel in the context of chunked prefill, probably doesn't make sense to add support for max_num_partial_prefills and max_long_partial_prefills. cc: @comaniac

IIRC in v0 we still apply max_num_partial_prefills for example even when chunked prefill is enabled, but I'm not fully sure which one (max_num_partial_prefills or long_prefill_token_threshold has a higher priority). This is a topic worth to discuss. cc @joerunde @njhill