feat: add unified test requirements processing command #30
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Implements automated catalog contribution workflow that creates PRs against catalog repository with proper directory structure and writes metadata back to test case files. **Issue #22: Add catalog workflow CLI flags** - Added --catalog-workflow flag to enable catalog workflow mode - Added --catalog-repo option (default: US-PS-SVS/catalog) - Added --test-cases-dir option (default: workspace/test_cases/) - Threaded flags through driver.py to override target repo when catalog mode enabled - Built catalog repository URL from github_api_url for metadata writeback **Issue #23: Implement PR metadata writeback** - Created github_ops_manager/processing/test_cases_processor.py module - Implemented find_test_cases_files() to locate test_cases.yaml files - Implemented load_test_cases_yaml() and save_test_cases_yaml() with format preservation - Implemented find_test_case_by_filename() to match test cases by generated_script_path - Implemented update_test_case_with_pr_metadata() to add PR fields - Added write_pr_metadata_to_test_cases() async function in pull_requests.py - Writes catalog_pr_git_url, catalog_pr_number, catalog_pr_url, catalog_pr_branch back to YAML **Issue #24: Handle robot file copy to catalog structure** - Added OS_TO_CATALOG_DIR_MAP with 15+ OS mappings (ios-xe, nxos, iosxr, etc.) - Implemented normalize_os_to_catalog_dir() for OS name translation - Implemented extract_os_from_robot_filename() to parse filenames - Updated get_desired_pull_request_file_content() to transform paths for catalog - Robot files now placed in catalog/<OS_NAME>/ directory structure - Updated commit_files_to_branch() to accept and pass catalog_workflow flag - Updated sync_github_pull_request() to accept catalog parameters and call writeback - Updated sync_github_pull_requests() to thread catalog parameters **Files Changed:** - github_ops_manager/processing/test_cases_processor.py (NEW) - github_ops_manager/configuration/cli.py (CLI flags) - github_ops_manager/synchronize/driver.py (parameter threading, repo override) - github_ops_manager/synchronize/pull_requests.py (path transformation, writeback) **Benefits:** - Eliminates manual catalog contribution friction - Automatic PR creation with proper catalog directory structure - Full metadata trail from generation → PR → catalog integration - Test automation available via PR branch while under review 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Refactors catalog workflow to be data-driven by reading catalog_destined attribute from individual test cases rather than using a global --catalog-workflow flag. This allows mixing catalog and non-catalog PRs in the same run. **Key Changes:** **CLI (configuration/cli.py:200-254)** - Removed --catalog-workflow flag - Kept --catalog-repo and --test-cases-dir as configuration options - Updated help text to indicate use with catalog_destined=true test cases - Added docstring explaining automatic detection behavior **Driver (synchronize/driver.py:19-159)** - Removed catalog_workflow parameter from run_process_issues_workflow() - Removed conditional repo override logic (now handled per-PR) - Always passes catalog config and auth parameters to sync_github_pull_requests() - Builds catalog_repo_url unconditionally for potential use **Pull Requests (synchronize/pull_requests.py:401-534)** - sync_github_pull_requests() now accepts auth parameters - Detects catalog-destined issues via getattr(issue, 'catalog_destined', False) - Creates catalog adapter only if catalog-destined issues exist - Fetches catalog repository state (issues, PRs, default branch) - Routes each PR to appropriate adapter based on catalog_destined attribute - Catalog workflow features (path transformation, metadata writeback) applied per-issue **Benefits:** - Granular control: Mix catalog and non-catalog in same test_cases.yaml - Data-driven design: Test cases declare their own intent - Single workflow run handles both types - Cleaner separation: tac-quicksilver sets flag, github-ops-manager respects it - More flexible and maintainable **Breaking Change:** - --catalog-workflow CLI flag removed (replaced by per-issue catalog_destined field) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Changes default catalog repository from US-PS-SVS/catalog to Testing-as-Code/tac-catalog across all configuration points. Updated in: - github_ops_manager/configuration/cli.py (CLI default) - github_ops_manager/synchronize/driver.py (function parameter) - github_ops_manager/synchronize/pull_requests.py (function parameter) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Improves reliability of OS detection by parsing the os:<os> tag from the Test Tags section in robot files rather than inferring from filenames. **Why This Is Better:** - Test Tags are structured metadata intentionally placed by tac-quicksilver - More reliable than filename parsing (filenames can vary) - Uses regex pattern to find os:<os> tag in robot file content - Falls back to filename parsing if Test Tags parsing fails **Implementation:** - Added extract_os_from_robot_content() with regex pattern `os:([a-zA-Z0-9_-]+)` - Updated extract_os_from_robot_filename() docstring to note it's a fallback - Modified get_desired_pull_request_file_content() to: 1. First try Test Tags extraction (preferred) 2. Fall back to filename parsing if needed 3. Log which extraction method succeeded **Example:** ```robot Test Tags ... os:ios-xe ... category:foundations ... feature:interfaces ``` Extracts "ios-xe" from Test Tags → maps to "catalog/IOS-XE/" directory **Files Changed:** - github_ops_manager/processing/test_cases_processor.py (new function) - github_ops_manager/synchronize/pull_requests.py (updated extraction logic) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Changed pattern from [a-zA-Z0-9_-]+ to \S+ for more flexible matching. \S+ matches any non-whitespace character, which is simpler and handles more edge cases than the explicit character class. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Added [project.scripts] section to pyproject.toml to expose the CLI as a console script. This allows the package to be invoked via: - uv run github-ops-manager - github-ops-manager (after installation) Without this entry point, the CLI was not accessible as a command after package installation. Entry point: github-ops-manager -> github_ops_manager.configuration.cli:typer_app 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Changed find_test_cases_files() to use non-recursive globbing (*.yaml
instead of **/*.yaml) to avoid picking up backup files in subdirectories
like .backups/
**Problem:**
Recursive glob pattern would find files like:
- workspace/test_cases/test_cases.yaml ✓
- workspace/test_cases/.backups/test_cases_old.yaml ✗ (unwanted)
**Solution:**
Only search immediate directory, not subdirectories:
- Before: test_cases_dir.glob('**/*.yaml') # Recursive
- After: test_cases_dir.glob('*.yaml') # Non-recursive
This prevents processing stale/backup test case files that could cause
weird behavior or duplicate PR creation attempts.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
This refactoring addresses the architectural issue where catalog-destined test cases were incorrectly trying to create issues+PRs in the catalog repo. The new architecture creates standalone PRs for catalog without issues. Changes: - Add load_catalog_destined_test_cases() to read test_cases.yaml directly - Add create_catalog_pull_requests() for standalone catalog PR creation - Update driver.py to call catalog PR function after issues workflow - Simplify sync_github_pull_requests() to filter out catalog-destined - Fix base_directory path resolution (use test_cases_dir.parent) Workflow now supports: 1. Non-catalog test cases: Create issues+PRs in project repo 2. Catalog-destined test cases: Create standalone PRs in catalog repo Fixes #22 #23 #24 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
When github_api_url ends with a trailing slash (e.g., 'https://wwwin-github.cisco.com/api/v3/'), the URL construction was creating double slashes like: https://wwwin-github.cisco.com//Testing-as-Code/tac-catalog Fixed by using .rstrip('/') to remove trailing slashes from base_url before constructing the catalog_repo_url. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Changed catalog branch naming from:
catalog/{os_name}/{script_name}
To:
feat/add-{script_name}
This follows Git best practices with conventional commit/branch naming patterns.
Example:
Before: catalog/ios-xe/verify-iosxe-error-disable-detection-reason-presence
After: feat/add-verify-iosxe-error-disable-detection-reason-presence
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Changed catalog branch naming to include OS for better organization:
feat/{os_name}/add-{script_name}
This groups branches by operating system, making it easier to manage
catalog contributions in the repository.
Example:
feat/ios-xe/add-verify-iosxe-error-disable-detection-reason-presence
feat/nxos/add-verify-nxos-vlan-configuration
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Add new CLI parameters --create-tracking-issues and --tracking-issue-labels - Create synchronize/tracking_issues.py module with issue creation logic - Add Jinja2 template for tracking issue body (templates/tracking_issue.j2) - Update create_catalog_pull_requests() to return PR metadata - Add writeback of project_issue_number and project_issue_url to test_cases.yaml - Update driver.py to conditionally create tracking issues after catalog PRs - Add update_test_case_with_issue_metadata() helper function This enables automatic creation of project repository issues that track catalog PR review and parameter learning tasks, with full traceability via metadata writeback to test_cases.yaml files.
Test case titles in test_cases.yaml may include OS tag prefixes like [IOS-XE] or [NX-OS], but these tags are stripped when creating test case groups in cxtm.yaml. The tracking issue template now uses the clean title (without OS tags) in CLI commands so that scripts learn and scripts run commands target the correct test case group names. - Add strip_os_tag_from_title() helper function - Pass both original and clean titles to template - Update template to use clean title in CLI commands - Keep original title for display purposes
Compute and display a suggested project repository branch name in tracking issues by replacing 'feat/' or 'feature/' prefix with 'learn/' from the catalog branch name. This provides users with a consistent naming convention for parameter learning branches. Examples: feat/nx-os/add-verify-nxos-module-port-number -> learn/nx-os/add-verify-nxos-module-port-number - Add compute_project_branch_name() helper function - Pass suggested_project_branch to template - Display suggestion in first task item
Replace inline code blocks with fenced code blocks (triple backticks) for all CLI commands in the tracking issue template. This enables GitHub's copy button feature, making it easier for users to copy and paste: - git checkout command for suggested branch name - tac-tools scripts learn command - tac-tools scripts run command Each command is now in its own fenced block with bash syntax highlighting.
Fixes a critical bug where test_cases.yaml files could be wiped out when writing PR or issue metadata back to files. The bug occurred when a test case's _source_file metadata pointed to a file that no longer contained that test case (e.g., after the test case was moved to another file). The code would load the file, fail to find the matching test case, but still save the file anyway - overwriting it with whatever was loaded (often just `test_cases: []`). Changes: - Add test_case_found flag to track if matching test case was located - Only save file if a matching test case was actually found and updated - Add warning logs when test case not found to aid debugging This prevents accidental data loss in criteria_needs_review.yaml and other test case files. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Adds an additional safety check in save_test_cases_yaml() that refuses to save a file if: 1. The file exists and has test cases 2. The new data would replace it with test_cases: [] This provides defense-in-depth against data loss, complementing the test_case_found checks added in commit 2d333eb. If this check triggers, it logs an error with the existing and new test case counts to aid debugging. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
CRITICAL FIX: The previous implementation had a fatal flaw where open(filepath, "w") immediately truncates the file to 0 bytes BEFORE yaml.dump() is called. If yaml.dump() failed or data was invalid, the file would be left completely empty. This commit implements atomic file writing: 1. Write to a temporary file in the same directory 2. Only if yaml.dump() succeeds, atomically rename temp to target 3. os.replace() is atomic on POSIX - either complete or not at all 4. If anything fails, original file remains untouched This prevents the exact scenario where files are being blanked out (not even containing test_cases: []) as reported in CI runs. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
CRITICAL FIX: The fetch-files command was creating 0-byte files when fetching files larger than 1MB because GitHub API's get_content endpoint returns content=None for large files and provides a download_url instead. The criteria_needs_review.yaml file is 2.7MB, which exceeds the 1MB limit, causing it to be fetched as empty and then overwriting the good file during mv operations in CI. Changes: - Check if response.parsed_data.content is None or empty - If so, use the download_url to fetch raw file content via httpx - Raise error if no download_url is available - Add logging for large file downloads This fixes the root cause of files being blanked out in CI. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
…est_cases" This reverts commit fb37bf4.
Enhanced tracking issues created for catalog PRs to include a "Test Requirement" section with key metadata extracted from the test case definition: - purpose - commands (list only, no outputs) - pass_criteria - sample_parameters - parameters_to_parsed_data_mapping This provides reviewers with immediate visibility into what the test requirement is designed to do without needing to navigate to the catalog PR or test case files. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
- Use literal block scalars (|) for purpose, pass_criteria, and parameters_to_parsed_data_mapping fields to properly handle multi-line strings and special characters - Quote command strings to handle pipes and other special characters - Ensures generated YAML in tracking issues is always valid 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Add new `repo process-test-requirements` CLI command that eliminates the need for issues.yaml by processing test requirements directly from test_cases.yaml files. Features: - Create GitHub issues for test cases missing issue metadata - Create project PRs for non-catalog test cases with generated scripts - Create catalog PRs for catalog-destined test cases - Write all metadata back to source test_cases.yaml files - Support for issue templates, labels, and catalog repository configuration This enables a simplified workflow where test requirements are managed directly in test_cases.yaml without maintaining a separate issues.yaml. Also includes: - 71 new unit tests for the new functionality - Renamed `test_case_needs_*` functions to `requires_*_creation` to avoid pytest collection conflicts - Updated pytest configuration to prevent source file collection 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add migration support for transitioning from legacy issues.yaml to test_cases.yaml workflow: - Add issues_yaml_migration module with clear deprecation markers - Add --issues-yaml option to process-test-requirements command - Migration matches issues to test cases by title - Extracts issue/PR metadata from issues.yaml - Writes metadata to corresponding test_cases.yaml files - Marks migrated issues with `migrated: true` field - Skips already-migrated issues on subsequent runs Migration module is clearly marked for removal post-migration with: - Prominent deprecation notice in module docstring - TODO comments over main entry points - Visual separators in CLI integration 32 unit tests added for migration functionality. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Base automatically changed from
feature/catalog-workflow-automation
to
master
December 20, 2025 14:49
The migration module was incorrectly reading metadata from issues.yaml,
but the legacy workflow never stored metadata there. Fixed to:
- Query GitHub API for issues/PRs matching titles
- Match issues by exact title
- Match PRs by legacy "GenAI, Review: {title}" format
- Write found metadata to test_cases.yaml files
- Mark issues in issues.yaml with `migrated: true`
Made migration async to support GitHub API calls and moved CLI
integration inside async function to access the GitHub adapter.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Configure ruamel.yaml with proper indent settings to match the original file format: - width=4096 to prevent line wrapping - indent(mapping=2, sequence=4, offset=2) to preserve indentation This prevents spurious diffs from formatting changes when only metadata fields are updated. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Jinja2 templates use attribute access (obj.attr) which doesn't work with ruamel.yaml CommentedMap objects. Added _convert_to_dict() helper to recursively convert CommentedMap/CommentedSeq to regular dict/list before template rendering. Fixes error: "'ruamel.yaml.comments.CommentedMap object' has no attribute 'genai_regex_pattern'" 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
The tac_issues_body.j2 template was failing when command data was missing optional keys like genai_regex_pattern or parser_used. Added |default filters to gracefully handle missing keys instead of raising UndefinedError. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Added truncation of command outputs to prevent GitHub 422 errors when issue bodies exceed 65,536 characters. The truncation is applied proportionally across all command_output and parsed_output fields before template rendering. - Added max_body_length parameter to process_test_requirements() - Modified render_issue_body_for_test_case() to accept and apply truncation - Default limit set to 60,000 chars (5,000 char safety margin) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Previously only files with "test_case" in the filename were processed, which excluded files like criteria_needs_review.yaml. Now all .yaml and .yml files in the test_cases directory are processed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
When creating project PRs via process-test-requirements, the PR body now includes a reference to the associated project issue (if available) with proper closing keywords (Closes #<issue_number>). This ensures GitHub automatically links the PR to the issue it implements. Changes: - PR body now includes "Closes #<issue_number>" when project_issue_number exists - PR body includes a link to the tracking issue URL - Added tests to verify PR body includes/excludes issue reference as expected 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
Update project PR title from "Test Automation: <title>" to "GenAI, Review: <title>" for better clarity that these PRs require human review of AI-generated content. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Resolved conflicts in test_cases_processor.py by keeping HEAD: - YAML formatting settings (width, indent) for format preservation - Process all YAML files behavior (intentional change from bebae75) - New functions for test requirements processing feature Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds a new
repo process-test-requirementsCLI command that eliminates the need forissues.yamlby processing test requirements directly fromtest_cases.yamlfiles.Features
project_issue_numbermetadataCloses #<issue_number>for automatic linkingtest_cases.yamlfilesBenefits
test_cases.yamlissues.yamlfileTechnical Changes
github_ops_manager/synchronize/test_requirements.pyreposubcommand:process-test-requirementstest_cases_processor.pytest_case_needs_*functions torequires_*_creationto avoid pytest collection conflictsTest plan
uv run pytest tests/unit/ -v(292 tests pass)uv run ruff check github_ops_manager/(clean)Usage
Example Project PR
When a project PR is created, it will have:
GenAI, Review: [IOS-XE] Verify Interface StatusCloses #<issue_number>to auto-close the tracking issue🤖 AI Generation Metadata
🤖 Generated with Claude Code