Skip to content

Commit 4302a03

Browse files
author
Dan Gil
committed
feat: Add automated dependency version tracking and extraction
DYN-1235 This PR implements comprehensive dependency tracking across all Dynamo components: Features: - Extracts 246 dependencies from 10 sources (Dockerfiles, requirements.txt, pyproject.toml, go.mod, Helm charts, etc.) - Sorts dependencies with critical ones first in each component section - Tracks version changes with dual diff columns (latest nightly + last release) - Identifies 32 NVIDIA products automatically - Generates package source URLs (PyPI, NGC Catalog, Docker Hub, Artifact Hub) - Runs nightly via CI with automated PRs on dependency changes - Creates permanent snapshots when release branches are cut Components covered: - trtllm (15 deps) - vllm (15 deps) - sglang (11 deps) - operator (90 deps) - shared (160 deps) CSV columns include: Component, Category, Dependency Name, Version, Source File, GitHub URL, Package Source URL, Status, Diff from Latest, Diff from Release, Critical, NVIDIA Product, Notes
1 parent bfbcae7 commit 4302a03

File tree

7 files changed

+2379
-0
lines changed

7 files changed

+2379
-0
lines changed

‎.github/reports/README.md‎

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
# Dependency Reports
2+
3+
This directory contains the latest dependency extraction reports for the Dynamo repository.
4+
5+
## Files
6+
7+
### `dependency_versions_latest.csv`
8+
The most recent dependency extraction results. Updated nightly by the automated CI workflow.
9+
10+
### `unversioned_dependencies_latest.csv`
11+
List of dependencies without explicit version constraints. These should be reviewed and pinned for reproducible builds.
12+
13+
### `releases/dependency_versions_vX.X.X.csv`
14+
Permanent snapshots of dependencies for each release version. Created automatically when release branches are cut.
15+
16+
**Examples:**
17+
- `releases/dependency_versions_v1.2.3.csv` - Release 1.2.3 snapshot
18+
- `releases/dependency_versions_v2.0.0.csv` - Release 2.0.0 snapshot
19+
20+
**CSV Columns:**
21+
- **Component** - Component category (trtllm, vllm, sglang, operator, shared)
22+
- **Category** - Dependency type (Base Image, Framework, Go Module, Python Package, Docker Compose Service, Helm Chart, etc.)
23+
- **Dependency Name** - Human-readable name
24+
- **Version** - Version number or constraint
25+
- **Source File** - Relative path to file defining the dependency
26+
- **GitHub URL** - Clickable link to source line
27+
- **Package Source URL** - Direct link to package documentation:
28+
- PyPI for Python packages
29+
- Docker Hub or NGC Catalog for containers
30+
- Artifact Hub for Helm charts
31+
- pkg.go.dev for Go modules
32+
- Official download pages for languages/tools
33+
- **Status** - Legacy status field (New, Changed, Unchanged)
34+
- **Diff from Latest** - Comparison to latest nightly:
35+
- `New` - New dependency not in latest nightly
36+
- `Unchanged` - Same version as latest nightly
37+
- `X → Y` - Version changed from X to Y
38+
- `N/A` - No latest nightly to compare against
39+
- **Diff from Release** - Comparison to latest release:
40+
- `New` - New dependency not in latest release
41+
- `Unchanged` - Same version as latest release
42+
- `X → Y` - Version changed from X to Y
43+
- `N/A` - No release snapshot to compare against
44+
- **Critical** - Yes/No flag for critical dependencies
45+
- **NVIDIA Product** - Yes/No flag indicating if dependency is an NVIDIA product
46+
- **Notes** - Additional context
47+
48+
**CSV Sorting:**
49+
The CSV is sorted to make critical dependencies easy to identify:
50+
1. By Component (trtllm → vllm → sglang → operator → shared)
51+
2. By Critical status (Yes before No) within each component
52+
3. Alphabetically by dependency name
53+
54+
**Extraction Sources:**
55+
The script extracts dependencies from multiple sources:
56+
- **Dockerfiles** - Base images and ARG/ENV versions
57+
- **requirements.txt** - Python packages (main, test, docs, standard)
58+
- **pyproject.toml** - Project metadata and dependencies
59+
- **go.mod** - Go module dependencies
60+
- **shell scripts** - Version variables from install scripts
61+
- **docker-compose.yml** - Service container versions
62+
- **Chart.yaml** - Helm chart and dependency versions
63+
- **rust-toolchain.toml** - Rust compiler version
64+
- **Cargo.toml** - Rust Git dependencies
65+
- **K8s recipe YAML** - Git-based pip installs from recipe files
66+
67+
### Critical Dependencies
68+
69+
Critical dependencies are flagged in the CSV to highlight components that require special attention for:
70+
- Security updates
71+
- Version compatibility
72+
- Production stability
73+
- Compliance requirements
74+
75+
The list of critical dependencies is maintained in `../workflows/extract_dependency_versions_config.yaml` under the `critical_dependencies` section. Examples include:
76+
- CUDA (compute platform)
77+
- PyTorch (ML framework)
78+
- Python (runtime)
79+
- Kubernetes (orchestration)
80+
- NATS (message broker)
81+
- etcd (key-value store)
82+
83+
## Timestamped Versions
84+
85+
Timestamped CSV files (e.g., `dependency_versions_20251009_1924.csv`) are:
86+
- **Generated** by the nightly workflow
87+
- **Stored** in GitHub Artifacts (90-day retention)
88+
- **Not committed** to the repo to avoid clutter
89+
- **Available** for download from the workflow run page
90+
91+
## Workflows
92+
93+
### Nightly Tracking (`.github/workflows/dependency-extraction-nightly.yml`)
94+
- **Schedule:** Daily at 2 AM UTC
95+
- **Trigger:** Can be manually triggered via Actions UI
96+
- **Output:** Updates `*_latest.csv` files, creates PR when changes detected
97+
- **Artifacts:** Uploads timestamped CSVs for 90-day retention
98+
99+
### Release Snapshots (`.github/workflows/dependency-extraction-release.yml`)
100+
- **Trigger:** Automatically when `release/*.*.*` branches are pushed
101+
- **Output:** Creates permanent `releases/dependency_versions_vX.X.X.csv`
102+
- **Purpose:** Permanent record of dependencies for each release
103+
- **Artifacts:** Stored for 365 days (1 year)
104+
105+
## Manual Extraction
106+
107+
To run manually from repository root:
108+
109+
```bash
110+
# Basic extraction
111+
python3 .github/workflows/extract_dependency_versions.py
112+
113+
# With options
114+
python3 .github/workflows/extract_dependency_versions.py \
115+
--output .github/reports/dependency_versions_latest.csv \
116+
--report-unversioned
117+
118+
# Validate configuration
119+
python3 .github/workflows/extract_dependency_versions.py --validate
120+
121+
# See all options
122+
python3 .github/workflows/extract_dependency_versions.py --help
123+
```
124+
125+
## Files
126+
127+
- 🤖 [Extraction Script](../workflows/extract_dependency_versions.py)
128+
- ⚙️ [Configuration](../workflows/extract_dependency_versions_config.yaml)
129+
- 📋 [Nightly Workflow](../workflows/dependency-extraction-nightly.yml)
130+
- 📸 [Release Workflow](../workflows/dependency-extraction-release.yml)
131+

‎.github/reports/releases/.gitkeep‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
# Release Dependency Snapshots

‎.github/workflows/dependency-extraction-nightly.yml‎

Lines changed: 155 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,155 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2025 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
16+
name: Nightly Dependency Extraction
17+
18+
on:
19+
schedule:
20+
# Run at 2 AM UTC every day
21+
- cron: '0 2 * * *'
22+
workflow_dispatch: # Allow manual trigger
23+
24+
permissions:
25+
contents: write
26+
pull-requests: write
27+
28+
jobs:
29+
extract-dependencies:
30+
runs-on: ubuntu-latest
31+
32+
steps:
33+
- name: Checkout repository
34+
uses: actions/checkout@v4
35+
with:
36+
fetch-depth: 0 # Need history for comparison
37+
38+
- name: Set up Python
39+
uses: actions/setup-python@v5
40+
with:
41+
python-version: '3.12'
42+
43+
- name: Install dependencies
44+
run: pip install pyyaml
45+
46+
- name: Run dependency extraction
47+
run: |
48+
TIMESTAMP=$(date +%Y%m%d_%H%M)
49+
50+
# Generate timestamped version (for artifacts)
51+
python3 .github/workflows/extract_dependency_versions.py \
52+
--output .github/reports/dependency_versions_${TIMESTAMP}.csv \
53+
--report-unversioned
54+
55+
# Copy to latest version (for repo tracking)
56+
mkdir -p .github/reports
57+
cp .github/reports/dependency_versions_${TIMESTAMP}.csv .github/reports/dependency_versions_latest.csv
58+
59+
# Copy unversioned report if it exists
60+
if [ -f "unversioned_dependencies_${TIMESTAMP}.csv" ]; then
61+
cp unversioned_dependencies_${TIMESTAMP}.csv .github/reports/unversioned_dependencies_latest.csv
62+
fi
63+
64+
echo "TIMESTAMP=${TIMESTAMP}" >> $GITHUB_ENV
65+
66+
- name: Check for changes
67+
id: check_changes
68+
run: |
69+
if [[ -n $(git status --porcelain .github/reports/*_latest.csv) ]]; then
70+
echo "has_changes=true" >> $GITHUB_OUTPUT
71+
72+
# Count dependencies by status from latest
73+
new_count=$(grep -c ",New," .github/reports/dependency_versions_latest.csv 2>/dev/null || echo "0")
74+
changed_count=$(grep -c ",Changed," .github/reports/dependency_versions_latest.csv 2>/dev/null || echo "0")
75+
unchanged_count=$(grep -c ",Unchanged," .github/reports/dependency_versions_latest.csv 2>/dev/null || echo "0")
76+
77+
echo "new_deps=$new_count" >> $GITHUB_OUTPUT
78+
echo "changed_deps=$changed_count" >> $GITHUB_OUTPUT
79+
echo "unchanged_deps=$unchanged_count" >> $GITHUB_OUTPUT
80+
else
81+
echo "has_changes=false" >> $GITHUB_OUTPUT
82+
fi
83+
84+
- name: Create Pull Request
85+
if: steps.check_changes.outputs.has_changes == 'true'
86+
uses: peter-evans/create-pull-request@v6
87+
with:
88+
token: ${{ secrets.GITHUB_TOKEN }}
89+
commit-message: 'chore: Update dependency versions [automated]'
90+
title: '[Automated] Nightly Dependency Version Update - $(date +%Y-%m-%d)'
91+
body: |
92+
## 🤖 Automated Dependency Version Update
93+
94+
This PR contains the nightly dependency extraction results.
95+
96+
### 📊 Summary
97+
- **New Dependencies:** ${{ steps.check_changes.outputs.new_deps }}
98+
- **Changed Versions:** ${{ steps.check_changes.outputs.changed_deps }}
99+
- **Unchanged:** ${{ steps.check_changes.outputs.unchanged_deps }}
100+
101+
### 📋 Files Updated
102+
- ✅ `.github/reports/dependency_versions_latest.csv` - Latest dependency snapshot
103+
- ✅ `.github/reports/unversioned_dependencies_latest.csv` - Unversioned deps report (if applicable)
104+
105+
> **Note:** Timestamped versions are stored in GitHub Artifacts (90-day retention) to avoid repo clutter.
106+
107+
### ✔️ Review Checklist
108+
- [ ] Review new dependencies for security/licensing concerns
109+
- [ ] Check version changes for breaking updates
110+
- [ ] Verify unversioned dependencies report
111+
- [ ] Update baseline count if increase is expected
112+
113+
---
114+
115+
🔗 **Documentation:** [Dependency Extraction Guide](../docs/dependency_extraction.md)
116+
📦 **Artifacts:** Download timestamped CSVs from workflow run
117+
118+
_Generated by nightly dependency extraction workflow_
119+
_Timestamp: ${{ env.TIMESTAMP }}_
120+
branch: automated/dependency-extraction-${{ github.run_number }}
121+
delete-branch: true
122+
labels: |
123+
automated
124+
dependencies
125+
documentation
126+
127+
- name: Upload artifacts
128+
if: always()
129+
uses: actions/upload-artifact@v4
130+
with:
131+
name: dependency-extraction-${{ github.run_number }}
132+
path: |
133+
.github/reports/dependency_versions_*.csv
134+
.github/reports/unversioned_dependencies_*.csv
135+
retention-days: 90
136+
137+
- name: Summary
138+
if: always()
139+
run: |
140+
echo "## Dependency Extraction Results" >> $GITHUB_STEP_SUMMARY
141+
echo "" >> $GITHUB_STEP_SUMMARY
142+
if [[ "${{ steps.check_changes.outputs.has_changes }}" == "true" ]]; then
143+
echo "✅ **Changes Detected**" >> $GITHUB_STEP_SUMMARY
144+
echo "" >> $GITHUB_STEP_SUMMARY
145+
echo "- New Dependencies: ${{ steps.check_changes.outputs.new_deps }}" >> $GITHUB_STEP_SUMMARY
146+
echo "- Changed Versions: ${{ steps.check_changes.outputs.changed_deps }}" >> $GITHUB_STEP_SUMMARY
147+
echo "- Unchanged: ${{ steps.check_changes.outputs.unchanged_deps }}" >> $GITHUB_STEP_SUMMARY
148+
echo "" >> $GITHUB_STEP_SUMMARY
149+
echo "📝 A pull request has been created for review." >> $GITHUB_STEP_SUMMARY
150+
else
151+
echo "ℹ️ **No Changes Detected**" >> $GITHUB_STEP_SUMMARY
152+
echo "" >> $GITHUB_STEP_SUMMARY
153+
echo "All dependencies remain unchanged since the last extraction." >> $GITHUB_STEP_SUMMARY
154+
fi
155+

0 commit comments

Comments
 (0)