feat: implement catalog workflow automation with standalone PR creation #25

ChristopherJHart · 2025-10-30T19:37:39Z

Summary

This PR implements the catalog workflow automation feature that enables automatic contribution of generated test automation back to the catalog repository. The implementation supports a dual-workflow approach where catalog-destined test cases create standalone PRs while non-catalog test cases follow the traditional issue+PR workflow.

Key Features

1. Dual Workflow Support

Project test cases (no catalog_destined or false): Create issues+PRs in project repository
Catalog test cases (catalog_destined: true): Create standalone PRs in catalog repository (no issues)

2. Smart OS Detection

Primary: Extract OS from Test Tags using regex pattern (?:^|\s)os:(\S+)
Fallback: Parse OS from filename pattern (e.g., verify_ios_xe_*.robot)

3. Path Transformation

Automatically transforms paths for catalog structure
Example: iosxe/verify_*.robot → catalog/IOS-XE/verify_*.robot

4. PR Metadata Writeback

Writes catalog PR information back to test_cases.yaml files
Fields: catalog_pr_git_url, catalog_pr_number, catalog_pr_url, catalog_pr_branch

5. Conventional Branch Naming

Pattern: feat/{os_name}/add-{script_name}
Example: feat/ios-xe/add-verify-iosxe-error-disable-detection-reason-presence

Architecture Changes

New Functions

load_catalog_destined_test_cases() (test_cases_processor.py)

Reads test_cases.yaml files directly
Filters for catalog_destined=true test cases
Returns list ready for catalog PR creation

create_catalog_pull_requests() (pull_requests.py)

Standalone catalog PR creation (no issues)
144-line function handling complete catalog workflow
Extracts OS, transforms paths, creates PRs, writes metadata

Modified Functions

run_process_issues_workflow() (driver.py)

Added call to create_catalog_pull_requests() after issues workflow
Fixed base_directory path resolution using test_cases_dir.parent
Fixed double slash in catalog repo URL construction

sync_github_pull_requests() (pull_requests.py)

Filters out catalog-destined issues at the start
Simplified to always use project adapter
Removed conditional catalog adapter logic

Bug Fixes

✅ Fixed console script entry point in pyproject.toml
✅ Changed to non-recursive glob to avoid .backups/ directory
✅ Fixed double slash in GitHub Enterprise URLs
✅ Updated default catalog repo to Testing-as-Code/tac-catalog
✅ Fixed base_directory path resolution for robot files

Usage

github-ops-manager repo US-PS-SVS/project process-issues \
    --github-api-url="https://wwwin-github.cisco.com/api/v3/" \
    --testing-as-code-workflow \
    --test-cases-dir="workspace/jobfiles/test_cases" \
    issues.yaml

Single command execution handles:

Creating issues+PRs for project test cases
Creating standalone catalog PRs for catalog-destined test cases

Files Changed

github_ops_manager/processing/test_cases_processor.py: +45 lines
github_ops_manager/synchronize/driver.py: +44 lines
github_ops_manager/synchronize/pull_requests.py: +181 lines, -71 lines
github_ops_manager/configuration/cli.py: Updated help text
pyproject.toml: Added console script entry point

Test Cases

Example test case with catalog_destined field:

test_cases:
  - title: '[IOS-XE] Verify Error Disable Detection Reason Presence'
    generated_script_path: iosxe/verify_iosxe_error_disable_detection_reason_presence.robot
    catalog_destined: true
    labels:
      - GenAI
      - script-already-created

Closes

Fixes #22
Fixes #23
Fixes #24

🤖 Generated with Claude Code

Implements automated catalog contribution workflow that creates PRs against catalog repository with proper directory structure and writes metadata back to test case files. **Issue #22: Add catalog workflow CLI flags** - Added --catalog-workflow flag to enable catalog workflow mode - Added --catalog-repo option (default: US-PS-SVS/catalog) - Added --test-cases-dir option (default: workspace/test_cases/) - Threaded flags through driver.py to override target repo when catalog mode enabled - Built catalog repository URL from github_api_url for metadata writeback **Issue #23: Implement PR metadata writeback** - Created github_ops_manager/processing/test_cases_processor.py module - Implemented find_test_cases_files() to locate test_cases.yaml files - Implemented load_test_cases_yaml() and save_test_cases_yaml() with format preservation - Implemented find_test_case_by_filename() to match test cases by generated_script_path - Implemented update_test_case_with_pr_metadata() to add PR fields - Added write_pr_metadata_to_test_cases() async function in pull_requests.py - Writes catalog_pr_git_url, catalog_pr_number, catalog_pr_url, catalog_pr_branch back to YAML **Issue #24: Handle robot file copy to catalog structure** - Added OS_TO_CATALOG_DIR_MAP with 15+ OS mappings (ios-xe, nxos, iosxr, etc.) - Implemented normalize_os_to_catalog_dir() for OS name translation - Implemented extract_os_from_robot_filename() to parse filenames - Updated get_desired_pull_request_file_content() to transform paths for catalog - Robot files now placed in catalog/<OS_NAME>/ directory structure - Updated commit_files_to_branch() to accept and pass catalog_workflow flag - Updated sync_github_pull_request() to accept catalog parameters and call writeback - Updated sync_github_pull_requests() to thread catalog parameters **Files Changed:** - github_ops_manager/processing/test_cases_processor.py (NEW) - github_ops_manager/configuration/cli.py (CLI flags) - github_ops_manager/synchronize/driver.py (parameter threading, repo override) - github_ops_manager/synchronize/pull_requests.py (path transformation, writeback) **Benefits:** - Eliminates manual catalog contribution friction - Automatic PR creation with proper catalog directory structure - Full metadata trail from generation → PR → catalog integration - Test automation available via PR branch while under review 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Refactors catalog workflow to be data-driven by reading catalog_destined attribute from individual test cases rather than using a global --catalog-workflow flag. This allows mixing catalog and non-catalog PRs in the same run. **Key Changes:** **CLI (configuration/cli.py:200-254)** - Removed --catalog-workflow flag - Kept --catalog-repo and --test-cases-dir as configuration options - Updated help text to indicate use with catalog_destined=true test cases - Added docstring explaining automatic detection behavior **Driver (synchronize/driver.py:19-159)** - Removed catalog_workflow parameter from run_process_issues_workflow() - Removed conditional repo override logic (now handled per-PR) - Always passes catalog config and auth parameters to sync_github_pull_requests() - Builds catalog_repo_url unconditionally for potential use **Pull Requests (synchronize/pull_requests.py:401-534)** - sync_github_pull_requests() now accepts auth parameters - Detects catalog-destined issues via getattr(issue, 'catalog_destined', False) - Creates catalog adapter only if catalog-destined issues exist - Fetches catalog repository state (issues, PRs, default branch) - Routes each PR to appropriate adapter based on catalog_destined attribute - Catalog workflow features (path transformation, metadata writeback) applied per-issue **Benefits:** - Granular control: Mix catalog and non-catalog in same test_cases.yaml - Data-driven design: Test cases declare their own intent - Single workflow run handles both types - Cleaner separation: tac-quicksilver sets flag, github-ops-manager respects it - More flexible and maintainable **Breaking Change:** - --catalog-workflow CLI flag removed (replaced by per-issue catalog_destined field) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Changes default catalog repository from US-PS-SVS/catalog to Testing-as-Code/tac-catalog across all configuration points. Updated in: - github_ops_manager/configuration/cli.py (CLI default) - github_ops_manager/synchronize/driver.py (function parameter) - github_ops_manager/synchronize/pull_requests.py (function parameter) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Improves reliability of OS detection by parsing the os:<os> tag from the Test Tags section in robot files rather than inferring from filenames. **Why This Is Better:** - Test Tags are structured metadata intentionally placed by tac-quicksilver - More reliable than filename parsing (filenames can vary) - Uses regex pattern to find os:<os> tag in robot file content - Falls back to filename parsing if Test Tags parsing fails **Implementation:** - Added extract_os_from_robot_content() with regex pattern `os:([a-zA-Z0-9_-]+)` - Updated extract_os_from_robot_filename() docstring to note it's a fallback - Modified get_desired_pull_request_file_content() to: 1. First try Test Tags extraction (preferred) 2. Fall back to filename parsing if needed 3. Log which extraction method succeeded **Example:** ```robot Test Tags ... os:ios-xe ... category:foundations ... feature:interfaces ``` Extracts "ios-xe" from Test Tags → maps to "catalog/IOS-XE/" directory **Files Changed:** - github_ops_manager/processing/test_cases_processor.py (new function) - github_ops_manager/synchronize/pull_requests.py (updated extraction logic) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Changed pattern from [a-zA-Z0-9_-]+ to \S+ for more flexible matching. \S+ matches any non-whitespace character, which is simpler and handles more edge cases than the explicit character class. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Added [project.scripts] section to pyproject.toml to expose the CLI as a console script. This allows the package to be invoked via: - uv run github-ops-manager - github-ops-manager (after installation) Without this entry point, the CLI was not accessible as a command after package installation. Entry point: github-ops-manager -> github_ops_manager.configuration.cli:typer_app 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Changed find_test_cases_files() to use non-recursive globbing (*.yaml instead of **/*.yaml) to avoid picking up backup files in subdirectories like .backups/ **Problem:** Recursive glob pattern would find files like: - workspace/test_cases/test_cases.yaml ✓ - workspace/test_cases/.backups/test_cases_old.yaml ✗ (unwanted) **Solution:** Only search immediate directory, not subdirectories: - Before: test_cases_dir.glob('**/*.yaml') # Recursive - After: test_cases_dir.glob('*.yaml') # Non-recursive This prevents processing stale/backup test case files that could cause weird behavior or duplicate PR creation attempts. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

This refactoring addresses the architectural issue where catalog-destined test cases were incorrectly trying to create issues+PRs in the catalog repo. The new architecture creates standalone PRs for catalog without issues. Changes: - Add load_catalog_destined_test_cases() to read test_cases.yaml directly - Add create_catalog_pull_requests() for standalone catalog PR creation - Update driver.py to call catalog PR function after issues workflow - Simplify sync_github_pull_requests() to filter out catalog-destined - Fix base_directory path resolution (use test_cases_dir.parent) Workflow now supports: 1. Non-catalog test cases: Create issues+PRs in project repo 2. Catalog-destined test cases: Create standalone PRs in catalog repo Fixes #22 #23 #24 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

When github_api_url ends with a trailing slash (e.g., 'https://wwwin-github.cisco.com/api/v3/'), the URL construction was creating double slashes like: https://wwwin-github.cisco.com//Testing-as-Code/tac-catalog Fixed by using .rstrip('/') to remove trailing slashes from base_url before constructing the catalog_repo_url. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Changed catalog branch naming from: catalog/{os_name}/{script_name} To: feat/add-{script_name} This follows Git best practices with conventional commit/branch naming patterns. Example: Before: catalog/ios-xe/verify-iosxe-error-disable-detection-reason-presence After: feat/add-verify-iosxe-error-disable-detection-reason-presence 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Changed catalog branch naming to include OS for better organization: feat/{os_name}/add-{script_name} This groups branches by operating system, making it easier to manage catalog contributions in the repository. Example: feat/ios-xe/add-verify-iosxe-error-disable-detection-reason-presence feat/nxos/add-verify-nxos-vlan-configuration 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

- Add new CLI parameters --create-tracking-issues and --tracking-issue-labels - Create synchronize/tracking_issues.py module with issue creation logic - Add Jinja2 template for tracking issue body (templates/tracking_issue.j2) - Update create_catalog_pull_requests() to return PR metadata - Add writeback of project_issue_number and project_issue_url to test_cases.yaml - Update driver.py to conditionally create tracking issues after catalog PRs - Add update_test_case_with_issue_metadata() helper function This enables automatic creation of project repository issues that track catalog PR review and parameter learning tasks, with full traceability via metadata writeback to test_cases.yaml files.

Test case titles in test_cases.yaml may include OS tag prefixes like [IOS-XE] or [NX-OS], but these tags are stripped when creating test case groups in cxtm.yaml. The tracking issue template now uses the clean title (without OS tags) in CLI commands so that scripts learn and scripts run commands target the correct test case group names. - Add strip_os_tag_from_title() helper function - Pass both original and clean titles to template - Update template to use clean title in CLI commands - Keep original title for display purposes

Compute and display a suggested project repository branch name in tracking issues by replacing 'feat/' or 'feature/' prefix with 'learn/' from the catalog branch name. This provides users with a consistent naming convention for parameter learning branches. Examples: feat/nx-os/add-verify-nxos-module-port-number -> learn/nx-os/add-verify-nxos-module-port-number - Add compute_project_branch_name() helper function - Pass suggested_project_branch to template - Display suggestion in first task item

Replace inline code blocks with fenced code blocks (triple backticks) for all CLI commands in the tracking issue template. This enables GitHub's copy button feature, making it easier for users to copy and paste: - git checkout command for suggested branch name - tac-tools scripts learn command - tac-tools scripts run command Each command is now in its own fenced block with bash syntax highlighting.

Fixes a critical bug where test_cases.yaml files could be wiped out when writing PR or issue metadata back to files. The bug occurred when a test case's _source_file metadata pointed to a file that no longer contained that test case (e.g., after the test case was moved to another file). The code would load the file, fail to find the matching test case, but still save the file anyway - overwriting it with whatever was loaded (often just `test_cases: []`). Changes: - Add test_case_found flag to track if matching test case was located - Only save file if a matching test case was actually found and updated - Add warning logs when test case not found to aid debugging This prevents accidental data loss in criteria_needs_review.yaml and other test case files. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Adds an additional safety check in save_test_cases_yaml() that refuses to save a file if: 1. The file exists and has test cases 2. The new data would replace it with test_cases: [] This provides defense-in-depth against data loss, complementing the test_case_found checks added in commit 2d333eb. If this check triggers, it logs an error with the existing and new test case counts to aid debugging. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

CRITICAL FIX: The previous implementation had a fatal flaw where open(filepath, "w") immediately truncates the file to 0 bytes BEFORE yaml.dump() is called. If yaml.dump() failed or data was invalid, the file would be left completely empty. This commit implements atomic file writing: 1. Write to a temporary file in the same directory 2. Only if yaml.dump() succeeds, atomically rename temp to target 3. os.replace() is atomic on POSIX - either complete or not at all 4. If anything fails, original file remains untouched This prevents the exact scenario where files are being blanked out (not even containing test_cases: []) as reported in CI runs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

CRITICAL FIX: The fetch-files command was creating 0-byte files when fetching files larger than 1MB because GitHub API's get_content endpoint returns content=None for large files and provides a download_url instead. The criteria_needs_review.yaml file is 2.7MB, which exceeds the 1MB limit, causing it to be fetched as empty and then overwriting the good file during mv operations in CI. Changes: - Check if response.parsed_data.content is None or empty - If so, use the download_url to fetch raw file content via httpx - Raise error if no download_url is available - Add logging for large file downloads This fixes the root cause of files being blanked out in CI. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

…est_cases" This reverts commit fb37bf4.

Enhanced tracking issues created for catalog PRs to include a "Test Requirement" section with key metadata extracted from the test case definition: - purpose - commands (list only, no outputs) - pass_criteria - sample_parameters - parameters_to_parsed_data_mapping This provides reviewers with immediate visibility into what the test requirement is designed to do without needing to navigate to the catalog PR or test case files. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

- Use literal block scalars (|) for purpose, pass_criteria, and parameters_to_parsed_data_mapping fields to properly handle multi-line strings and special characters - Quote command strings to handle pipes and other special characters - Ensures generated YAML in tracking issues is always valid 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

…kflow-automation

ChristopherJHart and others added 23 commits October 28, 2025 14:55

Revert "fix: add defensive check to prevent wiping files with empty t…

2f2f022

…est_cases" This reverts commit fb37bf4.

Merge remote-tracking branch 'origin/master' into feature/catalog-wor…

9d5fd87

…kflow-automation

ChristopherJHart merged commit 65e9965 into master Dec 20, 2025
2 checks passed

ChristopherJHart deleted the feature/catalog-workflow-automation branch December 20, 2025 14:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: implement catalog workflow automation with standalone PR creation #25

feat: implement catalog workflow automation with standalone PR creation #25

Uh oh!

ChristopherJHart commented Oct 30, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

feat: implement catalog workflow automation with standalone PR creation #25

feat: implement catalog workflow automation with standalone PR creation #25

Uh oh!

Conversation

ChristopherJHart commented Oct 30, 2025

Summary

Key Features

1. Dual Workflow Support

2. Smart OS Detection

3. Path Transformation

4. PR Metadata Writeback

5. Conventional Branch Naming

Architecture Changes

New Functions

Modified Functions

Bug Fixes

Usage

Files Changed

Test Cases

Closes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants