Add confidence parameter to issue mutation MCP tools#2605
Conversation
| - `pullNumber`: The pull request number (number, required) | ||
| - `repo`: Repository name (string, required) | ||
| - `reviewers`: GitHub usernames to request reviews from (string[], required) | ||
| - `reviewers`: GitHub usernames or ORG/team-slug team reviewers to request reviews from (string[], required) |
There was a problem hiding this comment.
This seems to come from an unrelated change that probably didn't update the docs
Add an optional confidence integer parameter (0–100) to update_issue_type, update_issue_labels, and set_issue_fields MCP tools. The confidence score is passed through to the REST/GraphQL API on mutation calls. - Rename structs to WithIntent (labelWithIntent, issueTypeWithIntent) - Add confidence schema property (integer, min 0, max 100) with prompt guidance describing what different confidence levels represent - Update tool descriptions to encourage including confidence scores - Pass confidence in the API request body alongside rationale/suggest Closes github/plan-track-agentic-toolkit#219 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
alondahari
force-pushed
the
alondahari/add-confidence-parameter
branch
from
311380b to
7b01177
Compare
| SingleSelectOptionID *githubv4.ID `json:"singleSelectOptionId,omitempty"` | ||
| Delete *githubv4.Boolean `json:"delete,omitempty"` | ||
| Rationale *githubv4.String `json:"rationale,omitempty"` | ||
| Confidence *int `json:"confidence,omitempty"` |
| var confidence *int | ||
| if _, exists := fieldMap["confidence"]; exists { | ||
| c, err := OptionalIntParam(fieldMap, "confidence") | ||
| if err != nil { | ||
| return utils.NewToolResultError(err.Error()), nil, nil | ||
| } | ||
| if c < 0 || c > 100 { | ||
| return utils.NewToolResultError("confidence must be between 0 and 100"), nil, nil | ||
| } | ||
| confidence = &c | ||
| } | ||
| if confidence != nil { | ||
| input.Confidence = confidence | ||
| } |
|
|
||
| - **update_issue_type** - Update Issue Type | ||
| - **Required OAuth Scopes**: `repo` | ||
| - `confidence`: How confident you are in this choice (0–100). 90–100: very high — clear signal or explicit user request. 70–89: high — strong signals, likely correct. 50–69: moderate — reasonable inference but ambiguous. 30–49: low — best guess, user review recommended. 0–29: very low — speculative. (number, optional) |
|
|
||
| - **update_issue_type** - Update Issue Type | ||
| - **Required OAuth Scopes**: `repo` | ||
| - `confidence`: How confident you are in this choice (0–100). 90–100: very high — clear signal or explicit user request. 70–89: high — strong signals, likely correct. 50–69: moderate — reasonable inference but ambiguous. 30–49: low — best guess, user review recommended. 0–29: very low — speculative. (number, optional) |
There was a problem hiding this comment.
What's the value of adding these range categories?
They seem somewhat arbitrary and don't actually exist anywhere in the code.
We could think about the confidence be something like an enum (with values mapped to named categories), but since we're using a numeric range, simply overlaying arbitrary ranges (which aren't enforced/referenced anywhere in the code) doesn't seem like it makes a lot of sense
There was a problem hiding this comment.
The idea is that we will have the same buckets as a guide for maintainers when setting the thresholds. I'm afraid that without any guidance it would be totally unpredictable. Do you think using an enum instead would be better?
There was a problem hiding this comment.
i think so. i don't really see a reason to care about a difference between a confidence value of 74 vs 76...
I think a few (named) categories would make more sense.
we could keep the 0-100 range on the backend, and use categories in all of the API/UI endpoints (which we would translate to a 0-100 number). That way we can add more fine-grained categories later if we want.
There was a problem hiding this comment.
We discussed this on Slack and decided to go ahead with this configuration for experimentation for now. We can always change it in the future.
Summary
Adds an optional
confidenceinteger parameter (0–100) to the three issue mutation MCP tools:update_issue_type,update_issue_labels, andset_issue_fields. The confidence score is passed through to the REST/GraphQL API on every mutation call.Changes
WithRationaletoWithIntent(labelWithIntent,issueTypeWithIntent) to better reflect they carry multiple intent signalsconfidenceschema property (integer, min 0, max 100) with prompt guidance in the description explaining what different confidence levels represent:Testing
Closes github/plan-track-agentic-toolkit#219