Clarify FAOS optimization guidance and improve evaluator contracts leveraging azd#2383
Merged
Conversation
Address PR microsoft#2174 review comments by expanding FAOS as Foundry Agent Optimization Service on first use and making the Step 8 Python config snippet copy-paste safe. The minimum contract example now imports os and preserves the app's existing model-selection fallback instead of hard-coding MODEL_DEPLOYMENT_NAME unless that is already what the app uses. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
# Conflicts: # plugin/skills/microsoft-foundry/SKILL.md
When creating custom evaluator prompts, treat the runtime-enforced JSON schema as authoritative: result plus reason. Preserve user-provided rubric text, but remove or normalize conflicting output schemas such as score/reasoning or duplicate OUTPUT FORMAT blocks before calling evaluator_catalog_create. Add observe skill test coverage for the promptText guardrail so future edits keep the result/reason contract visible. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove the 0-to-1 scoringType/minScore/maxScore details from the sample custom evaluator prompt. The important guardrail is the output contract: preserve the rubric, but avoid conflicting output schemas because the runtime enforces result/reason. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Restore plugin/skills/microsoft-foundry/SKILL.md from upstream/main so this PR no longer carries a line-ending-only change for the top-level skill file. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
XOEEst
requested review from
RickWinter,
XiaofuHuang,
anchenyi,
ankitbko,
jugonzales,
tendau and
vebudumu
as code owners
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the microsoft-foundry skill documentation to reflect the newer “Agent Optimizer” scaffolding workflow (replacing the previous FAOS optimize sub-skill), and expands the project context resolution guidance to better integrate with azd environments while treating .foundry/agent-metadata*.yaml as overlay/cache in azd projects.
Changes:
- Replace
faos-optimizewith a newagent-optimizersub-skill, including new scaffold + Python patterns reference docs. - Overhaul “context resolution” and metadata layering guidance to prefer
azd env get-valuesfor deployment context, and update observe/trace/eval docs accordingly (includingeval.yamlas local intent). - Update trigger snapshot baselines to match the updated skill description keywords.
Reviewed changes
Copilot reviewed 29 out of 29 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| plugin/skills/microsoft-foundry/SKILL.md | Updates skill description keywords and replaces faos-optimize with agent-optimizer; expands .foundry/azd context-resolution guidance. |
| plugin/skills/microsoft-foundry/references/agent-metadata-contract.md | Reframes metadata as overlay/cache in azd projects and updates the effective-context layering + schema guidance. |
| plugin/skills/microsoft-foundry/foundry-agent/trace/trace.md | Updates trace workflow to prefer azd-derived App Insights context and treat metadata as overlay/cache. |
| plugin/skills/microsoft-foundry/foundry-agent/trace/references/search-traces.md | Aligns prerequisites with the new “effective context”/overlay wording. |
| plugin/skills/microsoft-foundry/foundry-agent/observe/observe.md | Updates observe workflow to use azd-first context resolution, include eval.yaml intent, and add .foundry/suites/. |
| plugin/skills/microsoft-foundry/foundry-agent/observe/references/evaluation-suite-generation.md | Adds eval.yaml as an explicit suite source option and documents verification/registration steps. |
| plugin/skills/microsoft-foundry/foundry-agent/observe/references/deploy-and-setup.md | Updates auto-setup to use azd-first context and optionally leverage verified eval.yaml before generating suites. |
| plugin/skills/microsoft-foundry/foundry-agent/eval-datasets/eval-datasets.md | Updates dataset workflows to rely on azd or metadata overlay for context; mentions eval.yaml in state detection. |
| plugin/skills/microsoft-foundry/foundry-agent/eval-datasets/references/eval-trending.md | Updates prerequisites to allow azd-derived project endpoint/environment resolution. |
| plugin/skills/microsoft-foundry/foundry-agent/deploy/deploy.md | Adjusts DoD/gates and post-deploy suite setup to treat azd as source-of-truth and incorporate eval.yaml setup option. |
| plugin/skills/microsoft-foundry/foundry-agent/agent-optimizer/agent-optimizer.md | Introduces the new Agent Optimizer scaffold sub-skill entrypoint and lifecycle/guardrails. |
| plugin/skills/microsoft-foundry/foundry-agent/agent-optimizer/references/scaffold.md | Adds a step-by-step scaffold workflow for optimization readiness (baseline files + SDK wiring). |
| plugin/skills/microsoft-foundry/foundry-agent/agent-optimizer/references/python-patterns.md | Adds Python scaffolding patterns (baseline folder, tools.json format, wiring patterns, env vars). |
| plugin/skills/microsoft-foundry/foundry-agent/faos-optimize/faos-optimize.md | Removes the legacy FAOS optimize sub-skill doc. |
| plugin/skills/microsoft-foundry/foundry-agent/faos-optimize/references/python-patterns.md | Removes the legacy FAOS Python patterns doc. |
| tests/microsoft-foundry/snapshots/triggers.test.ts.snap | Updates trigger snapshots to reflect revised skill description keywords. |
| tests/microsoft-foundry/resource/create/snapshots/triggers.test.ts.snap | Updates trigger snapshots to reflect revised skill description keywords. |
| tests/microsoft-foundry/models/deploy/deploy-model/snapshots/triggers.test.ts.snap | Updates trigger snapshots to reflect revised skill description keywords. |
| tests/microsoft-foundry/models/deploy/deploy-model-optimal-region/snapshots/triggers.test.ts.snap | Updates trigger snapshots to reflect revised skill description keywords. |
| tests/microsoft-foundry/models/deploy/customize-deployment/snapshots/triggers.test.ts.snap | Updates trigger snapshots to reflect revised skill description keywords. |
| tests/microsoft-foundry/models/deploy/capacity/snapshots/triggers.test.ts.snap | Updates trigger snapshots to reflect revised skill description keywords. |
| tests/microsoft-foundry/foundry-agent/troubleshoot/snapshots/triggers.test.ts.snap | Updates trigger snapshots to reflect revised skill description keywords. |
| tests/microsoft-foundry/foundry-agent/trace/snapshots/triggers.test.ts.snap | Updates trigger snapshots to reflect revised skill description keywords. |
| tests/microsoft-foundry/foundry-agent/observe/snapshots/triggers.test.ts.snap | Updates trigger snapshots to reflect revised skill description keywords. |
| tests/microsoft-foundry/foundry-agent/invoke/snapshots/triggers.test.ts.snap | Updates trigger snapshots to reflect revised skill description keywords. |
| tests/microsoft-foundry/foundry-agent/eval-datasets/snapshots/triggers.test.ts.snap | Updates trigger snapshots to reflect revised skill description keywords. |
| tests/microsoft-foundry/foundry-agent/deploy/snapshots/triggers.test.ts.snap | Updates trigger snapshots to reflect revised skill description keywords. |
| tests/microsoft-foundry/foundry-agent/create/snapshots/triggers.test.ts.snap | Updates trigger snapshots to reflect revised skill description keywords. |
| tests/microsoft-foundry/finetuning/snapshots/triggers.test.ts.snap | Updates trigger snapshots to reflect revised skill description keywords. |
jugonzales
previously approved these changes
May 26, 2026
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jugonzales
approved these changes
May 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request updates the documentation for the Microsoft Foundry skill, primarily to add and clarify support for making Python agents optimization-ready with the new Agent Optimizer workflow. It introduces a new sub-skill, renames and updates references to the optimization process, and significantly expands the agent setup and configuration instructions to better integrate with Azure Developer CLI (
azd) environments. Additionally, it adds detailed documentation for scaffolding Python agents for optimization, including code patterns and configuration best practices.The most important changes are:
Agent Optimizer Sub-skill and Workflow Updates
agent-optimizer, replacing the previousfaos-optimizesub-skill, to guide users in making Python agents optimization-ready for the Agent Optimizer in Foundry. This includes updating all relevant references and workflow tables to use the new sub-skill and terminology. [1] [2] [3]Expanded and Clarified Agent Setup Instructions
azdproject context, clarify overlay/cache layering, and detail how to resolve deployment context, environment, and configuration from bothazdand.foundrymetadata. The setup flow is now more robust and explicit about source-of-truth and user prompts. [1] [2]Python Agent Optimizer Patterns Documentation
python-patterns.md, detailing how to scaffold Python agents for optimization: installing the required SDK, structuring the.agent_configs/baseline/folder, wiring runtime code, and following best practices for configuration and environment variables.Skill Description and Usage Clarifications
These changes provide clearer guidance for preparing agents for optimization, improve integration with Azure Dev tooling, and add actionable Python patterns for developers.
References:
Checklist
cd tests && npm test)npm run test:skills:integration -- <skill>)USE FOR/DO NOT USE FOR/PREFER OVERclauses: confirmed no routing regressions for competing skillsRelated Issues