feat: Add enable-local-kvindexer into MDC #4777

karen-sy · 2025-12-05T21:14:44Z

Overview:

To register enable_local_kvindexer (#4519) as a worker-side argument, we need to put it into the MDC.

Details:

This PR:

adds the arg to frontend-side configs for vllm and mock engine workers
refactors KvRouter's WorkerQueryClient (worker_query.rs) to watch the MDC (via the same discovery channel used by router) for the arg, such that workers are queried only when local kvindexers are known to be enabled
misc: refactors WorkerQueryClient to be component-scoped, not namespace-scoped
misc: refactor main Rust integration test (test_distributed_kvindexer_e2e) to sanity-check correctness for mockerengine

Where should the reviewer start?

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

closes GitHub issue: #xxx

Signed-off-by: c-fteixeira <[email protected]>

Signed-off-by: Dan Gil <[email protected]>

Signed-off-by: Neal Vaidya <[email protected]>

… / ModelReady (#4708) Signed-off-by: Guan Luo <[email protected]> Signed-off-by: GuanLuo <[email protected]> Co-authored-by: Ryan McCormick <[email protected]>

Signed-off-by: Olga Andreeva <[email protected]>

Signed-off-by: PeaBrane <[email protected]>

Signed-off-by: hongkuanz <[email protected]>

Signed-off-by: jthomson04 <[email protected]> Co-authored-by: ishandhanani <[email protected]>

) Signed-off-by: Anant Sharma <[email protected]>

Signed-off-by: Indrajit Bhosale <[email protected]> Co-authored-by: Ryan McCormick <[email protected]>

Signed-off-by: PeaBrane <[email protected]>

Signed-off-by: [email protected] <[email protected]>

Signed-off-by: alec-flowers <[email protected]> Signed-off-by: Karen Chung <[email protected]> Signed-off-by: jthomson04 <[email protected]> Co-authored-by: alec-flowers <[email protected]> Co-authored-by: jthomson04 <[email protected]>

Signed-off-by: [email protected] <[email protected]>

Signed-off-by: PeaBrane <[email protected]>

Signed-off-by: Vladislav Nosivskoy <[email protected]>

Signed-off-by: alec-flowers <[email protected]>

Signed-off-by: Keiven Chang <[email protected]>

Signed-off-by: Neal Vaidya <[email protected]>

Signed-off-by: Jacky <[email protected]>

…#4618) Signed-off-by: Krishnan Prashanth <[email protected]>

Signed-off-by: William Arnold <[email protected]>

… task (#4758)

Signed-off-by: Dillon Cullinan <[email protected]>

Signed-off-by: jthomson04 <[email protected]>

…4838) Signed-off-by: hongkuanz <[email protected]> Co-authored-by: Biswa Panda <[email protected]>

… transport (#4875) Signed-off-by: Jacky <[email protected]>

Signed-off-by: Dmitry Tokarev <[email protected]>

…4880) Signed-off-by: PeaBrane <[email protected]> Signed-off-by: Yan Ru Pei <[email protected]> Co-authored-by: Keiven C <[email protected]>

Signed-off-by: Vladislav Nosivskoy <[email protected]> Co-authored-by: Ryan McCormick <[email protected]>

Signed-off-by: Keiven Chang <[email protected]> Co-authored-by: Keiven Chang <[email protected]>

…4835) Signed-off-by: Keiven Chang <[email protected]> Co-authored-by: Keiven Chang <[email protected]>

Signed-off-by: Dillon Cullinan <[email protected]> Signed-off-by: Anant Sharma <[email protected]> Co-authored-by: Anant Sharma <[email protected]>

copy-pr-bot · 2025-12-11T19:19:22Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

…pgrade (#4899) Signed-off-by: Ziqi Fan <[email protected]>

Signed-off-by: Dmitry Tokarev <[email protected]>

c-fteixeira and others added 30 commits December 3, 2025 09:39

chore(helm): disable etcd PDB by default (#4602)

b4e7829

Signed-off-by: c-fteixeira <[email protected]>

docs: update TensorRT-LLM commit for multimodal EPD to v1.2.0rc3 (#4713)

d5a75dd

Signed-off-by: Dan Gil <[email protected]>

ci: publish docs automatically (#3263)

66f319a

Signed-off-by: Neal Vaidya <[email protected]>

feat: add Kserve readiness endpoint support: ServerLive / ServerReady…

d90dc39

… / ModelReady (#4708) Signed-off-by: Guan Luo <[email protected]> Signed-off-by: GuanLuo <[email protected]> Co-authored-by: Ryan McCormick <[email protected]>

fix: Updates CUDA fatbin for KVBM (#4700)

991d8c7

Signed-off-by: Olga Andreeva <[email protected]>

chore: Arc wrap prometheus_registry in MetricsRegistry (#4677)

c5e8c4c

Signed-off-by: PeaBrane <[email protected]>

chore: bump vLLM to 0.11.2 (#4476)

60feb95

fix: remove printing metadata json (#4737)

96f8008

feat: add more metrics in planner (#4710)

732c1ed

Signed-off-by: hongkuanz <[email protected]>

chore: update deploy code owner for backend k8s examples (#4743)

ea93cb5

chore: Update sgl version (#4716)

1f3e97a

Signed-off-by: jthomson04 <[email protected]> Co-authored-by: ishandhanani <[email protected]>

fix: copy wheel meta files in vllm dev stage for editable install (#4738

845a06b

) Signed-off-by: Anant Sharma <[email protected]>

feat: add lora k8s deployment example (#4714)

be67f67

test: Multimodal Tool Calling vLLM Test (#4663)

0651a4f

Signed-off-by: Indrajit Bhosale <[email protected]> Co-authored-by: Ryan McCormick <[email protected]>

chore: no need to arc wrap client (#4741)

5b24b42

Signed-off-by: PeaBrane <[email protected]>

feat: Add GPU discovery utilities (#4695)

c099252

Signed-off-by: [email protected] <[email protected]>

chore: bump vLLM to 0.12.0 (#4736)

d7c11b6

Signed-off-by: alec-flowers <[email protected]> Signed-off-by: Karen Chung <[email protected]> Signed-off-by: jthomson04 <[email protected]> Co-authored-by: alec-flowers <[email protected]> Co-authored-by: jthomson04 <[email protected]>

fix: Use /proc/driver/nvidia/gpus to get PCI addresses (#4717)

37d0fc8

Signed-off-by: [email protected] <[email protected]>

feat: Turn on canary health check by default (#4368)

d580031

Signed-off-by: [email protected] <[email protected]>

test: reenable router + vllm tests (#4746)

67273ab

Signed-off-by: PeaBrane <[email protected]>

fix: parallel tool call indices in streaming (#4723)

8601ccd

Signed-off-by: Vladislav Nosivskoy <[email protected]>

fix: vLLM xfer bug (#4745)

173201f

Signed-off-by: alec-flowers <[email protected]>

refactor: remove legacy NATS metrics and stats_handler (#4680)

00b64ae

Signed-off-by: Keiven Chang <[email protected]>

fix: skip HuggingFace download for non-llms (#4686)

118323f

Signed-off-by: Neal Vaidya <[email protected]>

ci: auto-publish docs with correct version number (#4740)

5942ef8

Signed-off-by: Neal Vaidya <[email protected]>

test: Extend Migration Tests to TRT-LLM (#4709)

a689c4b

Signed-off-by: Jacky <[email protected]>

test: Extend Migration Tests to SGLang (#4711)

b8c4a5f

Signed-off-by: Jacky <[email protected]>

build: Enable frontend decoding (media-nixl) in regular dynamo builds (…

76f4bc8

…#4618) Signed-off-by: Krishnan Prashanth <[email protected]>

feat: Add python-configurable engine routes for sglang profiling (#4617)

3bc129d

Signed-off-by: William Arnold <[email protected]>

fix: use channel to avoid race condition from async nats registration…

501ef02

… task (#4758)

dillon-cullinan and others added 16 commits December 10, 2025 22:35

ci: Disable sccache until its successfully resolved (#4869)

a409b29

Signed-off-by: Dillon Cullinan <[email protected]>

fix: Don't set min_tokens when ignore_eos is true (#4872)

664b458

Signed-off-by: jthomson04 <[email protected]>

test: router + trtllm CI tests (#4805)

ac8d36c

chore: add gb200 fp4 8k1k (#4874)

69817c2

fix: worker to graceful shutdown after finishing in-flight requests (#…

c8845b4

…4838) Signed-off-by: hongkuanz <[email protected]> Co-authored-by: Biswa Panda <[email protected]>

test: Ensure Request Cancellation, Migration, Rejection Work with TCP…

00f8615

… transport (#4875) Signed-off-by: Jacky <[email protected]>

chore: TRTLLM 1.2.0rc4 (#4836)

5250303

Signed-off-by: Dmitry Tokarev <[email protected]>

feat: include ttft and total request time in nvext (engine-agnostic) (#…

2845aa1

…4880) Signed-off-by: PeaBrane <[email protected]> Signed-off-by: Yan Ru Pei <[email protected]> Co-authored-by: Keiven C <[email protected]>

feat: add cached tokens prometheus metric (#4534)

f8bb53c

Signed-off-by: Vladislav Nosivskoy <[email protected]> Co-authored-by: Ryan McCormick <[email protected]>

refactor: simplify, remove --dev-image option from build.sh (#4867)

111e08c

Signed-off-by: Keiven Chang <[email protected]> Co-authored-by: Keiven Chang <[email protected]>

test: add dynamic port allocation for fault_tolerant test execution (#…

c6b440e

…4835) Signed-off-by: Keiven Chang <[email protected]> Co-authored-by: Keiven Chang <[email protected]>

fix: Fix CUDA_VISIBLE_DEVICES for agg_multimodal_epd.sh (#4888)

9b7152d

ci: Fixing sccache behavior (#4866)

242a4d5

Signed-off-by: Dillon Cullinan <[email protected]> Signed-off-by: Anant Sharma <[email protected]> Co-authored-by: Anant Sharma <[email protected]>

fix: [lora] refactor test and clean up examples (#4884)

3c2b72b

recovery for startup and gaps

c76c74a

test: graceful shutdown for tcp req plane (#4883)

5e96c9a

karen-sy requested a review from a team as a code owner December 11, 2025 19:19

ziqifan617 and others added 2 commits December 11, 2025 19:25

fix: remove small --gpu-memory-utilization to avoid OOM due to vllm u…

31f31e8

…pgrade (#4899) Signed-off-by: Ziqi Fan <[email protected]>

docs: update docs to remove KVBM cuda graph limitation (#4902)

5c2415d

karen-sy force-pushed the karenc/mdc_from_frontend branch from f89ffd1 to cea1ade Compare December 11, 2025 19:32

fix: relax pydantic requirements (#4865)

2ef0f5f

Signed-off-by: Dmitry Tokarev <[email protected]>

karen-sy force-pushed the karenc/mdc_from_frontend branch from cea1ade to c76c74a Compare December 11, 2025 19:33

pull-request-size bot added size/XXL and removed size/L labels Dec 11, 2025

copy-pr-bot bot had a problem deploying to GITLAB December 11, 2025 19:33 Failure

karen-sy added 3 commits December 11, 2025 11:34

Merge remote-tracking branch 'origin/main' into karenc/mdc_from_frontend

6a5b2dd

add back integration guard

69135ad

clippy

f2be88e

karen-sy closed this Dec 11, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Add enable-local-kvindexer into MDC #4777

feat: Add enable-local-kvindexer into MDC #4777

Uh oh!

karen-sy commented Dec 5, 2025

Uh oh!

copy-pr-bot bot commented Dec 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

feat: Add enable-local-kvindexer into MDC #4777

feat: Add enable-local-kvindexer into MDC #4777

Uh oh!

Conversation

karen-sy commented Dec 5, 2025

Overview:

Details:

Where should the reviewer start?

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

Uh oh!

copy-pr-bot bot commented Dec 11, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants