Skip to content

Add multi-turn evaluators Python sample#721

Closed
JoseCSantos wants to merge 1 commit into
microsoft-foundry:mainfrom
JoseCSantos:feat/multi-turn-evaluators
Closed

Add multi-turn evaluators Python sample#721
JoseCSantos wants to merge 1 commit into
microsoft-foundry:mainfrom
JoseCSantos:feat/multi-turn-evaluators

Conversation

@JoseCSantos
Copy link
Copy Markdown

Adds a new Python sample under samples/python/multi-turn-evaluators showing how to evaluate pre-recorded multi-turn conversations using the four Azure AI Foundry multi-turn evaluators (Coherence, Task Completion, Customer Satisfaction, Groundedness).

What's included

  • multi_turn_evaluators.ipynb: end-to-end notebook (load traces, convert to Foundry schema, create per-evaluator evals, submit runs, retrieve and analyze results).
  • data/: 15 small labeled conversation traces covering all four evaluators.
  • multi_turn_traces_foundry.jsonl: pre-built JSONL in Foundry format that can also be uploaded manually via the Foundry portal UI.
  • README.md, requirements.txt, .env.example (with documented value formats).

Notes

  • A GPT-5.x judge model is recommended/expected.
  • The notebook groups traces by their target evaluator and creates one eval per evaluator to avoid noisy cross-evaluation.

@github-actions
Copy link
Copy Markdown
Contributor

👋 Thanks for your interest in contributing, @JoseCSantos!

This repository does not accept pull requests directly. If you'd like to report a bug, suggest an improvement, or propose a new sample, please open an issue instead.

See CONTRIBUTING.md for more details.

@github-actions github-actions Bot closed this May 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant