merge: merge main into ryan/kvbm-nova-251204

Merged 238 commits from main branch to bring the feature branch up to date.

Key conflicts resolved:
- Removed lib/kvbm-kernels references (deleted in main)
- Kept nova/nova-backend/kvbm workspace members from feature branch
- Maintained v2 module API refactoring from feature branch
- Updated Cargo.lock files to reflect new dependencies

Major updates from main include:
- LoRA support for vLLM (#4810)
- Multimodal documentation (#4510)
- Scaling adapter features (#4699, #4825)
- Tool calling support (#4822, #4722)
- NIXL connect improvements (#4433)

Signed-off-by: Ryan Olson <[email protected]>

Author: ryanolson

.devcontainer/README.md

@@ -1,24 +1,26 @@
 <!--
 SPDX-FileCopyrightText: Copyright (c) 2024-2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 SPDX-License-Identifier: Apache-2.0
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
 -->
 
 # NVIDIA Dynamo Development Environment
 
 > Warning: Dev Containers (aka `devcontainers`) is an evolving feature and we are not testing in CI. Please submit any problem/feedback using the issues on GitHub.
 
+## Known Issues
+
+### Docker Version Compatibility
+
+**Docker 29.x has compatibility issues with Dev Containers (by Anysphere):**
+- It is known that Docker Engine version **29.0.1** (released November 14, 2025) may cause Dev Containers to hang all the time, rendering it unusable
+- The Dev Containers extension (v1.0.26) and devcontainer CLI (v0.75.0) were not tested against Docker 29.x
+- Symptoms: Container builds successfully but connection hangs, requiring a manual reload to connect
+- This may be fixed in a later version of the Anysphere Dev Containers extension and/or Docker Engine patch
+
+**Recommended Docker Version:**
+- Use Docker Engine **28.5.2** or earlier for stable Dev Container operation
+- Docker 27.x and 28.x series are confirmed working with current Dev Containers tooling
+
 ## Framework-Specific Devcontainers
 
 This directory contains framework-specific devcontainer configurations generated from a Jinja2 template:

.devcontainer/post-create.sh

@@ -48,7 +48,9 @@ show_and_run() {
     # Run commands with debug output shown
     set -x
     "$@"
+    local exit_code=$?
     { set +x; } 2>/dev/null
+    return $exit_code
 }
 
 set -x
@@ -62,28 +64,7 @@ pre-commit run --all-files || true # don't fail the build if pre-commit hooks fa
 export CARGO_TARGET_DIR=${CARGO_TARGET_DIR:-$WORKSPACE_DIR/target}
 mkdir -p $CARGO_TARGET_DIR
 
-uv pip uninstall --yes ai-dynamo ai-dynamo-runtime 2>/dev/null || true
-
-# Build project, with `dev` profile it will be saved at $CARGO_TARGET_DIR/debug
-cargo build --locked --profile dev --features dynamo-llm/block-manager
-
-# install the python bindings
-
-# Install maturin if not already installed.
-# TODO: Uncomment for SGLANG. Right now, the CI team is making a refactor
-#       to the Dockerfile, so this is a temporary fix.
-# if ! command -v maturin &> /dev/null; then
-#     echo "Installing maturin..."
-#     retry uv pip install maturin[patchelf]
-# else
-#     echo "maturin is already installed"
-# fi
-
-# install ai-dynamo-runtime
-(cd $WORKSPACE_DIR/lib/bindings/python && retry maturin develop)
-
-# install ai-dynamo
-cd $WORKSPACE_DIR && retry env DYNAMO_BIN_PATH=$CARGO_TARGET_DIR/debug uv pip install -e .
+# Note: Build steps moved to after sanity check - see instructions at the end
 
 { set +x; } 2>/dev/null
 
@@ -114,16 +95,26 @@ else
     echo "⚠️ SSH agent forwarding not configured - SSH_AUTH_SOCK is not set"
 fi
 
-show_and_run $WORKSPACE_DIR/deploy/sanity_check.py
+echo -e "\n🔍 Running sanity check..."
+if show_and_run $WORKSPACE_DIR/deploy/sanity_check.py; then
+    SANITY_STATUS="✅ Sanity check passed"
+else
+    SANITY_STATUS="⚠️ Sanity check failed - review output above"
+fi
 
 cat <<EOF
 
-✅ SUCCESS: Built cargo project, installed Python bindings, configured pre-commit hooks
+========================================
+$SANITY_STATUS
+✅ Pre-commit hooks configured
+
+Now build the project:
+  cargo build --locked --profile dev --features dynamo-llm/block-manager
+  cd lib/bindings/python && maturin develop --uv
+  DYNAMO_BIN_PATH=$CARGO_TARGET_DIR/debug uv pip install -e .
+
+Optional: cd lib/bindings/kvbm && maturin develop --uv  # For KVBM support
 
-Example commands:
-  cargo build --locked --profile dev              # Build Rust project in $CARGO_TARGET_DIR
-  cd lib/bindings/python && maturin develop --uv  # Update Python bindings (if you changed them)
-  cargo fmt && cargo clippy                       # Format and lint code before committing
-  cargo doc --no-deps                             # Generate documentation
-  uv pip install -e .                             # Install various Python packages Dynamo depends on
+If cargo build fails with a Cargo.lock error, try to update it with 'cargo update'
+========================================
 EOF

.github/actions/docker-build/action.yml

@@ -16,9 +16,6 @@ inputs:
   image_tag:
     description: 'Custom image tag (optional, defaults to framework:latest)'
     required: false
-  ngc_ci_access_token:
-    description: 'NGC CI Access Token'
-    required: false
   ci_token:
     description: 'CI Token'
     required: false
@@ -49,6 +46,12 @@ inputs:
   torch_backend:
     description: 'Optional override for TORCH_BACKEND build-arg (e.g., cu129)'
     required: false
+  enable_kvbm:
+    description: 'Enable KVBM support (optional)'
+    required: false
+  dynamo_base_image:
+    description: 'Pre-built Dynamo base image to use instead of building from scratch'
+    required: false
 
 outputs:
   image_tag:
@@ -61,18 +64,9 @@ runs:
     - name: Set up Docker Buildx
       uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 #v3.11.1
       with:
-        driver: docker
-    - name: Login to ECR
-      shell: bash
-      env:
-        ECR_HOSTNAME: ${{ inputs.aws_account_id }}.dkr.ecr.${{ inputs.aws_default_region }}.amazonaws.com
-      run: |
-        aws ecr get-login-password --region ${{ inputs.aws_default_region }} | docker login --username AWS --password-stdin ${ECR_HOSTNAME}
-    - name: Login to NGC
-      if: github.event.pull_request.head.repo.full_name == github.repository || github.event_name == 'push'
-      shell: bash
-      run: |
-        echo "${{ inputs.ngc_ci_access_token }}" | docker login nvcr.io -u '$oauthtoken' --password-stdin
+        driver: docker-container
+        # Enable BuildKit for enhanced metadata
+        buildkitd-flags: --debug
     - name: Cleanup
       if: always()
       shell: bash
@@ -88,7 +82,12 @@ runs:
         AWS_ACCESS_KEY_ID: ${{ inputs.aws_access_key_id }}
         AWS_SECRET_ACCESS_KEY: ${{ inputs.aws_secret_access_key }}
         PLATFORM: ${{ inputs.platform }}
+        ECR_HOSTNAME: ${{ inputs.aws_account_id }}.dkr.ecr.${{ inputs.aws_default_region }}.amazonaws.com
+        GITHUB_RUN_ID: ${{ github.run_id }}
+        GITHUB_JOB: ${{ github.job }}
+        GITHUB_REF_NAME: ${{ github.ref_name }}
       run: |
+        set -x
         # Determine image tag
         if [ -n "${{ inputs.image_tag }}" ]; then
           IMAGE_TAG="${{ inputs.image_tag }}"
@@ -97,43 +96,89 @@ runs:
         fi
 
         BUILD_START_TIME=$(date -u +%Y-%m-%dT%H:%M:%SZ)
-        echo "🕐 Build started at: ${BUILD_START_TIME}"
         echo "BUILD_START_TIME=${BUILD_START_TIME}" >> $GITHUB_ENV
 
         echo "image_tag=$IMAGE_TAG" >> $GITHUB_OUTPUT
+
+        # Create build logs directory
+        mkdir -p build-logs
+        BUILD_LOG_FILE="build-logs/build-${{ inputs.framework }}-$(echo '${{ inputs.platform }}' | sed 's/linux\///').log"
+        echo "BUILD_LOG_FILE=${BUILD_LOG_FILE}" >> $GITHUB_ENV
+        echo "📝 Build log will be saved to: ${BUILD_LOG_FILE}"
+
         # Collect optional overrides provided by the workflow
+        # Set base cache args and set --cache-to if this is a main commit
         EXTRA_ARGS=""
+        EXTRA_ARGS="--cache-to type=inline "
+        EXTRA_ARGS+="--cache-from type=registry,ref=${ECR_HOSTNAME}/ai-dynamo/dynamo:${{ inputs.framework }}-${PLATFORM##*/}-cache "
+        EXTRA_ARGS+="--cache-from type=registry,ref=${ECR_HOSTNAME}/ai-dynamo/dynamo:main-${{ inputs.framework }}-${PLATFORM##*/} "
+        if [[ "$GITHUB_REF_NAME" == "main" ]]; then
+          EXTRA_ARGS+="--cache-to type=registry,ref=${ECR_HOSTNAME}/ai-dynamo/dynamo:${{ inputs.framework }}-${PLATFORM##*/}-cache,mode=max "
+        fi
+
+        echo "$EXTRA_ARGS"
+        # Collect optional overrides provided by the workflow
         if [ -n "${{ inputs.base_image_tag }}" ]; then
-          EXTRA_ARGS+=" --base-image-tag ${{ inputs.base_image_tag }}"
+          EXTRA_ARGS+="--base-image-tag ${{ inputs.base_image_tag }} "
         fi
         if [ -n "${{ inputs.runtime_image_tag }}" ]; then
-          EXTRA_ARGS+=" --build-arg RUNTIME_IMAGE_TAG=${{ inputs.runtime_image_tag }}"
+          EXTRA_ARGS+="--build-arg RUNTIME_IMAGE_TAG=${{ inputs.runtime_image_tag }} "
         fi
         if [ -n "${{ inputs.cuda_version }}" ]; then
-          EXTRA_ARGS+=" --build-arg CUDA_VERSION=${{ inputs.cuda_version }}"
+          EXTRA_ARGS+="--build-arg CUDA_VERSION=${{ inputs.cuda_version }} "
         fi
         if [ -n "${{ inputs.torch_backend }}" ]; then
-          EXTRA_ARGS+=" --build-arg TORCH_BACKEND=${{ inputs.torch_backend }}"
+          EXTRA_ARGS+="--build-arg TORCH_BACKEND=${{ inputs.torch_backend }} "
+        fi
+        if [ -n "${{ inputs.dynamo_base_image }}" ]; then
+          EXTRA_ARGS+=" --dynamo-base-image ${{ inputs.dynamo_base_image }}"
+        fi
+        if [ -n "${{ inputs.enable_kvbm }}" ]; then
+          EXTRA_ARGS+=" --build-arg ENABLE_KVBM=${{ inputs.enable_kvbm }}"
         fi
 
+        # Execute build and capture output (show on console AND save to file)
         ./container/build.sh --tag "$IMAGE_TAG" \
           --target ${{ inputs.target }} \
           --vllm-max-jobs 10 \
           --framework ${{ inputs.framework }} \
           --platform ${{ inputs.platform }} \
           --use-sccache \
           --sccache-bucket "$SCCACHE_S3_BUCKET" \
-          --sccache-region "$AWS_DEFAULT_REGION" $EXTRA_ARGS
+          --sccache-region "$AWS_DEFAULT_REGION" $EXTRA_ARGS 2>&1 | tee "${BUILD_LOG_FILE}"
+
+        BUILD_EXIT_CODE=${PIPESTATUS[0]}
 
         BUILD_END_TIME=$(date -u +%Y-%m-%dT%H:%M:%SZ)
-        echo "🕐 Build ended at: ${BUILD_END_TIME}"
         echo "BUILD_END_TIME=${BUILD_END_TIME}" >> $GITHUB_ENV
 
+        # Exit with the build's exit code
+        exit ${BUILD_EXIT_CODE}
+
+    - name: Run Sanity Check on Runtime Image
+      if: inputs.target == 'runtime'
+      shell: bash
+      run: |
+        IMAGE_TAG="${{ steps.build.outputs.image_tag }}"
+        echo "Running sanity check on image: $IMAGE_TAG"
+
+        # Run the sanity check script inside the container
+        # The script is located in /workspace/deploy/sanity_check.py in runtime containers
+        set +e
+        docker run --rm "$IMAGE_TAG" python /workspace/deploy/sanity_check.py --runtime-check --no-gpu-check
+        SANITY_CHECK_EXIT_CODE=$?
+        set -e
+        if [ ${SANITY_CHECK_EXIT_CODE} -ne 0 ]; then
+          echo "ERROR: Sanity check failed - ai-dynamo packages not properly installed"
+          exit ${SANITY_CHECK_EXIT_CODE}
+        else
+          echo "✅ Sanity check passed"
+        fi
+
     - name: Capture Build Metrics
       id: metrics
       shell: bash
       run: |
-        echo "📊 Capturing build metrics for ${{ inputs.framework }}..."
 
         # Create metrics directory
         mkdir -p build-metrics
@@ -162,21 +207,17 @@ runs:
           echo "⚠️  No image tag available"
         fi
 
-        echo "📊 Final metrics captured"
-
-        # Create consolidated metrics JSON file
-        echo "🔍 Debug: inputs.platform = '${{ inputs.platform }}'"
         PLATFORM_ARCH=$(echo "${{ inputs.platform }}" | sed 's/linux\///')
-        echo "🔍 Debug: PLATFORM_ARCH = '${PLATFORM_ARCH}'"
+        echo "  Architecture: ${PLATFORM_ARCH}"
         echo "PLATFORM_ARCH=${PLATFORM_ARCH}" >> $GITHUB_ENV
         JOB_KEY="${{ inputs.framework }}-${PLATFORM_ARCH}"
-        echo "🔍 Debug: JOB_KEY = '${JOB_KEY}'"
+        echo "  Job Key: ${JOB_KEY}"
 
         # Create job-specific metrics file
         mkdir -p build-metrics
-        METRICS_FILE="build-metrics/metrics-${{ inputs.framework }}-${PLATFORM_ARCH}.json"
+        METRICS_FILE="build-metrics/metrics-${{ inputs.framework }}-${PLATFORM_ARCH}-${{ github.run_id }}-${{ job.check_run_id }}.json"
 
-        # Create the job metrics file directly
+        # Create the job metrics file
         cat > "$METRICS_FILE" << EOF
         {
           "framework": "${{ inputs.framework }}",
@@ -190,15 +231,88 @@ runs:
         }
         EOF
 
-        echo "📁 Created build metrics file for ${JOB_KEY}:"
         cat "$METRICS_FILE"
 
-        # Metrics captured and saved to JSON file
+    - name: Generate Comprehensive Build Metrics
+      id: comprehensive-metrics
+      if: always()
+      shell: bash
+      run: |
+        echo "=========================================="
+        echo "📊 GENERATING COMPREHENSIVE BUILD METRICS"
+        echo "=========================================="
+
+        # Create metrics directory
+        mkdir -p build-metrics
+
+        PLATFORM_ARCH="${{ env.PLATFORM_ARCH }}"
+        WORKFLOW_ID="${{ github.run_id }}"
+        JOB_ID="${{ job.check_run_id }}"
+        FRAMEWORK_LOWER=$(echo "${{ inputs.framework }}" | tr '[:upper:]' '[:lower:]')
+
+        # Make parser executable
+        chmod +x .github/scripts/parse_buildkit_output.py
+
+        # Check for build logs and build stage arguments dynamically
+        BUILD_LOG="build-logs/single-stage-build.log"
+
+        # Path to container metadata created in previous step
+        CONTAINER_METADATA="build-metrics/metrics-${{ inputs.framework }}-${PLATFORM_ARCH}-${WORKFLOW_ID}-${JOB_ID}.json"
+
+        # Output single comprehensive JSON with all build stages
+        COMPREHENSIVE_JSON="build-metrics/build-${{ inputs.framework }}-${PLATFORM_ARCH}-${WORKFLOW_ID}-${JOB_ID}.json"
+
+        echo "🚀 Parsing BuildKit outputs and merging with container metrics..."
 
-    # Upload job-specific build metrics as artifact
-    - name: Upload Build Metrics
+        # Build stage arguments dynamically based on which logs exist
+        STAGE_ARGS=()
+
+        if [ -f "$BUILD_LOG" ]; then
+          echo "  ✓ Found base image log: ${BUILD_LOG}"
+          STAGE_ARGS+=("runtime:${BUILD_LOG}")
+        else
+          echo "  ℹ️  No image log found"
+        fi
+
+        # Check for any additional stage logs (e.g., build-logs/stage3-*.log)
+        for extra_log in build-logs/stage*.log; do
+          if [ -f "$extra_log" ]; then
+            stage_name=$(basename "$extra_log" .log)
+            echo "  ✓ Found additional stage log: ${extra_log} (${stage_name})"
+            STAGE_ARGS+=("${stage_name}:${extra_log}")
+          fi
+        done
+
+        echo "Container Metadata: ${CONTAINER_METADATA}"
+        echo "Output: ${COMPREHENSIVE_JSON}"
+        echo ""
+
+        # Run parser with all discovered stages
+        # Usage: parse_buildkit_output.py <output_json> <stage1_name:log_file> [stage2_name:log_file] ... [--metadata=<file>]
+        set +e
+        python3 .github/scripts/parse_buildkit_output.py \
+          "$COMPREHENSIVE_JSON" \
+          "${STAGE_ARGS[@]}" \
+          "--metadata=${CONTAINER_METADATA}"
+        PARSER_EXIT_CODE=$?
+        set -e
+
+        echo ""
+        echo "📊 Parser exit code: ${PARSER_EXIT_CODE}"
+
+        if [ ${PARSER_EXIT_CODE} -eq 0 ] && [ -f "$COMPREHENSIVE_JSON" ]; then
+          echo "✅ Comprehensive build metrics generated successfully"
+          echo "📄 Output file: ${COMPREHENSIVE_JSON}"
+        else
+          echo "⚠️  Metrics generation had issues but continuing..."
+        fi
+
+    # Upload comprehensive build metrics as artifact
+    - name: Upload Comprehensive Build Metrics
       uses: actions/upload-artifact@v4
+      if: always()
       with:
-        name: build-metrics-${{ inputs.framework }}-${{ env.PLATFORM_ARCH }}
-        path: build-metrics/metrics-${{ inputs.framework }}-${{ env.PLATFORM_ARCH }}.json
+        name: build-metrics-${{ inputs.framework }}-${{ inputs.target }}-${{ env.PLATFORM_ARCH }}-${{ github.run_id }}-${{ job.check_run_id }}
+        path: build-metrics/build-${{ inputs.framework }}-${{ env.PLATFORM_ARCH }}-${{ github.run_id }}-${{ job.check_run_id }}.json
         retention-days: 7
+