Skip to content

Add release notes generator plugin#98

Open
neubig wants to merge 5 commits intomainfrom
openhands/release-notes-generator-plugin
Open

Add release notes generator plugin#98
neubig wants to merge 5 commits intomainfrom
openhands/release-notes-generator-plugin

Conversation

@neubig
Copy link
Contributor

@neubig neubig commented Mar 6, 2026

Summary

Adds a release-notes plugin that uses an OpenHands agent to write concise, user-focused release notes for OpenHands repositories.

Closes #97

Details

Design Decisions

  • Agent-authored release notes: the action now uses an OpenHands agent to judge significance, merge related PRs, and decide what is worth mentioning
  • Deterministic GitHub context: commit/PR metadata, labels, bodies, contributors, and tag ranges are still collected deterministically and passed to the agent as structured context
  • Composite GitHub Action: implemented as a reusable composite action for easy integration
  • Automatic tag detection: finds the previous release tag automatically to determine the commit range
  • Optional internal section: internal/infrastructure changes remain opt-in
  • New contributor detection: first-time contributors are identified and included in the final notes

Plugin Structure

plugins/release-notes/
├── README.md
├── SKILL.md
├── action.yml
├── scripts/
│   ├── agent_script.py
│   ├── generate_release_notes.py
│   └── prompt.py
└── workflows/release-notes.yml

Categories Generated

  • ⚠️ Breaking Changes
  • ✨ New Features
  • 🐛 Bug Fixes
  • 📚 Documentation
  • 🏗️ Internal/Infrastructure (optional)
  • 👥 New Contributors

Testing

Added/updated tests covering:

  • prompt instructions for agent-based editorial decisions
  • categorization hints and PR metadata capture
  • markdown generation behavior
  • plugin structure validation
46 passed in 0.09s
58 passed in 3.14s

Evidence

Verification link: View conversation

Live run against OpenHands/openhands tag 0.35.0:

Generating release notes: 0.34.0 -> 0.35.0
Found 51 commits
Release notes generated successfully!

Output excerpt:

## [0.35.0] - 2025-04-28

### ⚠️ Breaking Changes
- Remove reset settings button and API endpoint (#8079) @enyst

### ✨ New Features
- Add VS Code editor tab alongside workspace, terminal, and Jupyter tabs (#8033) @rbren
- Add CLI support for `/new`, `/status`, and `/settings` commands (#8017) @pandukamuditha
- Add custom secrets storage backend for secure credential management (#7891) @malhotra5

### 🐛 Bug Fixes
- Fix long-running setup scripts being interrupted before completion (#8101) @neubig
- Reconnect automatically when WebSocket stream closes unexpectedly (#8055) @SmartManoj
- Preserve personal access tokens when resetting settings (#8075) @malhotra5

### 👥 New Contributors
- @enyst made their first contribution in #8022
- @dependabot[bot] made their first contribution in #8064

**Full Changelog**: https://github.com/OpenHands/openhands/compare/0.34.0...0.35.0

Requirement check:

  • ✅ Triggered on semver tags and supports manual dispatch via plugins/release-notes/workflows/release-notes.yml
  • ✅ Auto-detects the previous release tag (0.34.0 -> 0.35.0 in the live run)
  • ✅ Uses an OpenHands agent, not a fully deterministic formatter, to decide significance and group related changes
  • ✅ Includes categorized sections, PR numbers, authors, new contributors, and a full changelog link
  • ✅ Keeps internal changes optional via include-internal

Checklist

  • Tests pass
  • Evidence gathered from a live run against a real repository
  • Documentation included (README.md)
  • Review feedback addressed

This plugin generates consistent, well-structured release notes from git history.
It can be triggered via GitHub Actions on release tags following semver patterns
(e.g., v*.*.*) or manually invoked.

Features:
- Automatic tag detection: Finds previous release tag to determine commit range
- Categorized changes: Groups into Breaking Changes, Features, Bug Fixes, Docs, Internal
- Conventional commits support: Categorizes based on commit prefixes (feat:, fix:, etc.)
- PR label support: Also categorizes based on GitHub PR labels
- Contributor attribution: Includes PR numbers and author usernames
- New contributor highlighting: Identifies first-time contributors
- Flexible output: Updates GitHub releases or generates CHANGELOG.md entries

Closes #97

Co-authored-by: openhands <openhands@all-hands.dev>
@neubig neubig added the enhancement New feature or request label Mar 6, 2026 — with OpenHands AI
@neubig neubig marked this pull request as ready for review March 7, 2026 02:03
Copy link

@all-hands-bot all-hands-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Acceptable - Works but has complexity issues

Core functionality is solid with good test coverage and comprehensive documentation. However, the main processing loop violates single responsibility (doing ~50 lines of work in one loop), exceptions are swallowed silently in multiple places hiding real errors, and there's wasteful dependency management.

Verdict: ✅ Worth merging with improvements - the issues are maintainability concerns, not blockers. Suggest addressing the exception handling and loop complexity in a follow-up.

@neubig neubig marked this pull request as draft March 7, 2026 02:09
1. Remove unused requests dependency from action.yml (script uses stdlib urllib)
2. Remove __future__ annotations import (unnecessary for Python 3.12 target)
3. Fix semver regex with proper end anchor to avoid matching 'v1.2.3abc'
4. Add error logging to exception handlers instead of silently swallowing
   - get_pr_for_commit: logs warning when PR fetch fails
   - is_new_contributor: logs warning when contributor check fails
   - get_tag_date: logs warning when tag date fetch fails
5. Refactor main processing loop into separate helper functions:
   - _process_commit: handles single commit → Change conversion
   - _categorize_changes: categorizes list of changes
   - _process_contributors: extracts and identifies contributors
6. Fix regex to require colon delimiter to avoid stripping legitimate content
   (e.g., 'BREAKING change in...' won't be incorrectly stripped)

Co-authored-by: openhands <openhands@all-hands.dev>
@neubig neubig marked this pull request as ready for review March 7, 2026 02:13
Copy link

@all-hands-bot all-hands-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Acceptable - Works but has a regex bug and dead code. Previous refactoring was good.

@neubig
Copy link
Contributor Author

neubig commented Mar 7, 2026

@OpenHands fix review comment and close them, and also re-read the issue description. I updated it a bit. Update the "evidence" section demonstrating that the output of the agent satisfies the requirements in the issue description.

@openhands-ai
Copy link

openhands-ai bot commented Mar 7, 2026

I'm on it! neubig can track my progress at all-hands.dev

- Fix regex to handle conventional commit '!' indicator (feat!:)
- Remove unused 'message_lower' variable
- Add patterns for bracket/paren style prefixes ([Feat]:, (Fix):)
- Fix duplicate PR number references in output

Co-authored-by: openhands <openhands@all-hands.dev>
@openhands-ai
Copy link

openhands-ai bot commented Mar 7, 2026

Final Summary

I've successfully addressed all the review comments and updated PR #98 for the Release Notes Generator plugin.

Changes Made (commit 04c42ed, already pushed):

  1. Fixed regex bug for ! indicator ✅ - Added !? before the colon to handle conventional commit breaking change indicators like feat!:

  2. Removed dead code ✅ - Deleted the unused message_lower variable in categorize_change()

  3. Added bracket/paren style prefix support ✅ - Extended patterns to match [Feat]:, (Fix):, [Hotfix]: style prefixes used in OpenHands repositories

  4. Fixed duplicate PR number references ✅ - Added check to avoid appending PR number if it already exists in the commit message

  5. Updated PR body with evidence ✅ - Added a requirements satisfaction table showing all 10 requirements from issue Plugin: Release Notes Generator #97 are met, with live test output against OpenHands/openhands tag 0.35.0

  6. Replied to review comments ✅ - Posted responses to the two outstanding review comments (regex bug, dead code) with commit references

Verification:

@neubig neubig marked this pull request as draft March 7, 2026 13:40
Copy link
Contributor Author

neubig commented Mar 7, 2026

Moved to draft - 2 unresolved review comments

Please resolve all review threads before marking ready again.

Also missing: Conversation verification link in Evidence section.

@neubig neubig marked this pull request as ready for review March 7, 2026 20:34
Copy link

@all-hands-bot all-hands-bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid work on the refactoring. Core logic is clean and well-tested. One critical bug to fix regarding API pagination limits.

"""Get all commits between two tags."""
endpoint = f"/repos/{repo_name}/compare/{base_tag}...{head_tag}"
response = github_api_request(endpoint, token)
return response.get("commits", [])

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 Important: The GitHub /compare/ API has a 250-commit limit. If there are more than 250 commits between tags, this will silently truncate the results and produce incomplete release notes.

The API response includes a status field when truncated. You should either:

  1. Check for truncation and warn users, or
  2. Implement proper pagination using /repos/OpenHands/extensions/commits

Suggested quick fix to at least warn users:

Suggested change
return response.get("commits", [])
def get_commits_between_tags(
repo_name: str, base_tag: str, head_tag: str, token: str
) -> list[dict[str, Any]]:
"""Get all commits between two tags."""
endpoint = f"/repos/{repo_name}/compare/{base_tag}...{head_tag}"
response = github_api_request(endpoint, token)
commits = response.get("commits", [])
# Warn if truncated (>250 commits)
if response.get("status") == "ahead" and len(commits) >= 250:
print(f"Warning: Large commit range detected. Release notes may be incomplete.",
file=sys.stderr)
return commits

with open(output_file, "a") as f:
# Handle multiline values
if "\n" in value:
delimiter = "EOF"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Suggestion: Using a hardcoded "EOF" delimiter could theoretically break if the release notes content contains that exact string on its own line. Consider using a unique delimiter:

Suggested change
delimiter = "EOF"
delimiter = f"EOF_{os.urandom(4).hex()}"

This is unlikely to occur in practice but is a simple fix to eliminate the edge case entirely.

@neubig
Copy link
Contributor Author

neubig commented Mar 7, 2026

@OpenHands The current "evidence" section in the PR description doesn't match the requirements of the issue description. It is too verbose and not user-friendly enough. The prompt provided to the agent to reduce the verbosity of the release notes may need to be adjusted.

@openhands-ai
Copy link

openhands-ai bot commented Mar 7, 2026

I'm on it! neubig can track my progress at all-hands.dev

Co-authored-by: openhands <openhands@all-hands.dev>
@openhands-ai
Copy link

openhands-ai bot commented Mar 7, 2026

Final summary:

  • Addressed the request to make the PR evidence section less verbose and more user-friendly
  • Updated the generator to produce more concise release notes by:
    • preferring merged PR titles
    • collapsing multiple commits from the same PR into one entry
    • omitting uncategorized “Other Changes” from default output
    • improving categorization with keyword fallbacks
  • Updated the plugin README to document the new concise-output behavior
  • Rewrote the PR Add release notes generator plugin #98 body’s Evidence section to better match issue Plugin: Release Notes Generator #97
  • Added tests for the new behavior and passed verification:
    • targeted tests: 43 passed
    • full suite: 55 passed
    • live run against OpenHands/openhands tag 0.35.0 succeeded
  • Kept the changes focused; only the relevant plugin/test/docs files plus AGENTS.md were updated
  • Pushed the changes to the remote branch

Git:

  • Commit pushed: 1d62755Reduce release notes verbosity

No additional changes were made after the last summary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Plugin: Release Notes Generator

3 participants