Add external agent sample

.gitignore

@@ -6,6 +6,9 @@
 # macOS
 .DS_Store
 
+# LiveShare cache files
+*.lscache
+
 # User-specific files
 *.rsuser
 *.suo

samples/python/external-agents/observability/.env.example

@@ -0,0 +1,11 @@
+AZURE_SUBSCRIPTION_ID=<subscription-id>
+FOUNDRY_PROJECT_ENDPOINT=https://<resource>.services.ai.azure.com/api/projects/<project>
+RESOURCE_GROUP=<resource-group>
+LOCATION=eastus2
+ACA_ENV=<container-apps-environment>
+ACR_NAME=<acr-name>
+APPLICATIONINSIGHTS_CONNECTION_STRING=InstrumentationKey=<key>;IngestionEndpoint=<endpoint>
+AZURE_OPENAI_ENDPOINT=https://<aoai-resource>.openai.azure.com
+AZURE_OPENAI_DEPLOYMENT=gpt-4o-mini
+AZURE_OPENAI_API_VERSION=2024-10-21
+AZURE_OPENAI_API_KEY=<optional-api-key>

samples/python/external-agents/observability/Dockerfile

@@ -0,0 +1,16 @@
+FROM python:3.11-slim
+
+ENV PYTHONUNBUFFERED=1 \
+    PYTHONDONTWRITEBYTECODE=1 \
+    PORT=8000
+
+WORKDIR /app
+
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+COPY weather_agent.py .
+
+EXPOSE 8000
+
+CMD ["python", "weather_agent.py"]

samples/python/external-agents/observability/README.md

@@ -0,0 +1,169 @@
+# External Agent Observability — Weather Agent on ACA
+
+This sample shows the **end-to-end story for a Foundry "external" agent**:
+a third-party agent runtime that lives **outside** Foundry, registered
+into Foundry purely so its OpenTelemetry traces and Foundry-side
+evaluations light up in the portal.
+
+The runtime here is a tiny [LangChain](https://python.langchain.com/)
+weather agent (LangGraph ReAct), instrumented with the
+[Microsoft OpenTelemetry distro](https://github.com/microsoft/opentelemetry-distro-python)
+so its spans flow into the Application Insights connected to your
+Foundry project. We deploy it to **Azure Container Apps** to play the
+role of "agent hosted somewhere outside Foundry" (it could just as
+easily be GCP Cloud Run, AWS, or on-prem).
+
+> **Preview note.** External agents are gated behind
+> `Foundry-Features: ExternalAgents=V1Preview` while in public preview.
+> The SDK calls below opt in via `allow_preview=True`.
+
+> **Distro note.** The Microsoft OTel distro does not yet accept an
+> explicit `otel_agent_id` input
+> ([microsoft/opentelemetry-distro-python#148](https://github.com/microsoft/opentelemetry-distro-python/issues/148)).
+> Until that ships, this sample relies on the default `gen_ai.agent.id`
+> emitted by the LangChain instrumentation. The Foundry registration
+> uses the matching agent name so the Foundry trace view will resolve
+> once the distro fix is available.
+
+## Microsoft OpenTelemetry distro — references
+
+To learn more about the distro or to find samples in another language,
+start here:
+
+- **Docs:** [Microsoft OpenTelemetry overview](https://learn.microsoft.com/en-us/azure/microsoft-opentelemetry/overview)
+- **Samples by language:**
+  - .NET — [microsoft/opentelemetry-distro-dotnet](https://github.com/microsoft/opentelemetry-distro-dotnet)
+  - Python — [microsoft/opentelemetry-distro-python](https://github.com/microsoft/opentelemetry-distro-python)
+  - JavaScript — [microsoft/opentelemetry-distro-javascript](https://github.com/microsoft/opentelemetry-distro-javascript)
+
+## What's in this folder
+
+| File | Purpose |
+| --- | --- |
+| [weather_agent.py](weather_agent.py) | LangChain weather agent + Microsoft OTel distro, exposed as a FastAPI HTTP service. This is the "external runtime". |
+| [.env.example](.env.example) | Placeholder environment template for local configuration. |
+| [Dockerfile](Dockerfile) | Container image for the weather agent. |
+| [deploy.sh](deploy.sh) / [deploy.ps1](deploy.ps1) | Build, push to ACR, deploy to ACA, and generate traffic. |
+| [generate_traffic.py](generate_traffic.py) | Hits the deployed agent with a handful of weather questions. |
+| [register_external_agent.py](register_external_agent.py) | Registers the runtime in Foundry as `kind=external` via the `azure-ai-projects` SDK. |
+| [run_trace_eval.py](run_trace_eval.py) | Runs a one-off trace-based eval over the registered agent and prints scores. |
+| [requirements.txt](requirements.txt) | Python deps for both the runtime and the helper scripts. |
+
+## Architecture
+
+```
+   ┌────────────────────────┐         OTel spans          ┌─────────────────────┐
+   │  Weather agent (ACA)   │ ─────────────────────────▶  │ Application Insights │
+   │  LangChain + MS distro │   gen_ai.agent.id =         │ (linked to project) │
+   └────────────────────────┘   "weather-agent"           └─────────┬───────────┘
+                                                                    │
+                              register_external_agent.py            │ trace view
+                                       │                            ▼
+                                       ▼                     ┌─────────────────────┐
+                              ┌─────────────────────┐        │   Foundry Portal    │
+                              │  Foundry Project    │ ◀────  │  Agents → traces    │
+                              │  agent kind=external│        │  Evaluations        │
+                              └─────────────────────┘        └─────────────────────┘
+```
+
+## Prerequisites
+
+1. **Azure resources**
+   - A Foundry project with an Application Insights connection.
+   - An Azure Container Apps environment + ACR in the same subscription.
+   - An Azure OpenAI deployment (e.g. `gpt-4o-mini`) for both the agent
+     LLM and the eval judge. (You can split them with the
+     `EVAL_MODEL_DEPLOYMENT` env var.)
+2. **Permissions** — `Contributor` (or equivalent) on the RG, plus
+   permission to create agents in the Foundry project (e.g.
+   `Azure AI User`).
+3. **Tooling** — Azure CLI, Docker is *not* required (we use
+   `az acr build`), Python 3.11+.
+
+## Step 1 — Configure environment
+
+Start from [.env.example](.env.example), create a local `.env`, and
+load those values into your shell before running the scripts. The local
+`.env` file is ignored by git.
+
+```bash
+export AZURE_SUBSCRIPTION_ID="<subscription-id>"
+export FOUNDRY_PROJECT_ENDPOINT="https://<resource>.services.ai.azure.com/api/projects/<project>"
+export RESOURCE_GROUP="..."
+export LOCATION="eastus2"
+export ACA_ENV="..."
+export ACR_NAME="..."
+export APPLICATIONINSIGHTS_CONNECTION_STRING="InstrumentationKey=...;IngestionEndpoint=..."
+export AZURE_OPENAI_ENDPOINT="https://<aoai>.openai.azure.com"
+export AZURE_OPENAI_DEPLOYMENT="gpt-4o-mini"
+export AZURE_OPENAI_API_VERSION="2024-10-21"
+export AZURE_OPENAI_API_KEY="..."
+```
+
+PowerShell users: set the same names with `$env:NAME = "..."`.
+
+## Step 2 — Deploy the external runtime to ACA and generate traffic
+
+```bash
+cd samples/python/external-agents/observability
+./deploy.sh           # or .\deploy.ps1 on Windows
+```
+
+The script will:
+
+1. `az acr build` the image.
+2. Create or update the Container App.
+3. Wait for `/healthz`.
+4. Run [generate_traffic.py](generate_traffic.py) to ask several weather
+   questions.
+5. Wait for OTel export and ingestion. Override the default 90 seconds
+   with `TRACE_INGEST_WAIT_SECS` if needed.
+
+## Step 3 — Register the external agent in Foundry
+
+```bash
+python register_external_agent.py
+```
+
+This calls `project_client.agents.create_version(...)` with an
+`ExternalAgentDefinition`, which atomically creates the Foundry agent
+record on first call (per the spec, external agents are versionless
+from the user's perspective). After registration succeeds, open the
+Foundry portal:
+
+> **Project → Agents → `weather-agent` → Traces**
+
+You should see the spans you just generated, attributed to the new
+`external` agent.
+
+## Step 4 — Run a one-off trace evaluation
+
+```bash
+python run_trace_eval.py
+```
+
+This:
+
+1. Resolves the registered agent's `otel_agent_id`.
+2. Creates an OpenAI-compatible eval group with two built-in trace
+   evaluators (`intent_resolution`, `task_adherence`).
+3. Creates an `azure_ai_traces_preview` run scoped to that
+   `agent_id` over the last 24 hours.
+4. Polls until completion and prints per-criterion pass/fail counts.
+
+> The trace-eval surface is currently only on the OpenAI-compatible
+> `evals` API (`project_client.get_openai_client().evals`). When the
+> native `project_client.evaluations` surface adds trace-eval support,
+> this sample will move there.
+
+## Cleanup
+
+```bash
+az containerapp delete -n "$AGENT_NAME" -g "$RESOURCE_GROUP" --yes
+# Optional: delete the Foundry registration (does not affect the runtime)
+python -c "import os; from azure.identity import DefaultAzureCredential; \
+from azure.ai.projects import AIProjectClient; \
+AIProjectClient(endpoint=os.environ['FOUNDRY_PROJECT_ENDPOINT'], \
+credential=DefaultAzureCredential(), allow_preview=True) \
+.agents.delete(agent_name=os.environ.get('AGENT_NAME','weather-agent'))"
+```

samples/python/external-agents/observability/deploy.ps1

@@ -0,0 +1,105 @@
+<#
+.SYNOPSIS
+  Deploy the weather agent to Azure Container Apps and generate traffic.
+
+.DESCRIPTION
+  Mirrors deploy.sh for Windows / PowerShell users. Requires the
+  Azure CLI (`az`) and Python on PATH.
+
+.NOTES
+  Required env vars:
+    AZURE_SUBSCRIPTION_ID,
+    RESOURCE_GROUP, LOCATION, ACA_ENV, ACR_NAME,
+    APPLICATIONINSIGHTS_CONNECTION_STRING,
+    AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_DEPLOYMENT
+  Optional:
+    AZURE_OPENAI_API_KEY, AGENT_NAME (default weather-agent), IMAGE_TAG (default current timestamp),
+    TRACE_INGEST_WAIT_SECS (default 90)
+#>
+
+$ErrorActionPreference = "Stop"
+
+function Require-Env($name) {
+  if (-not (Get-Item "Env:$name" -ErrorAction SilentlyContinue)) {
+    throw "Missing required env var: $name"
+  }
+}
+
+foreach ($v in @(
+  "AZURE_SUBSCRIPTION_ID",
+  "RESOURCE_GROUP","LOCATION","ACA_ENV","ACR_NAME",
+  "APPLICATIONINSIGHTS_CONNECTION_STRING",
+  "AZURE_OPENAI_ENDPOINT","AZURE_OPENAI_DEPLOYMENT"
+)) { Require-Env $v }
+
+Write-Host "==> Selecting Azure subscription"
+az account set --subscription $env:AZURE_SUBSCRIPTION_ID --only-show-errors
+
+$AgentName  = if ($env:AGENT_NAME)    { $env:AGENT_NAME }    else { "weather-agent" }
+$OtelAgentId = if ($env:OTEL_AGENT_ID) { $env:OTEL_AGENT_ID } else { $AgentName }
+$ImageTag   = if ($env:IMAGE_TAG)     { $env:IMAGE_TAG }     else { Get-Date -Format "yyyyMMddHHmmss" }
+$TraceIngestWaitSecs = if ($env:TRACE_INGEST_WAIT_SECS) { [int]$env:TRACE_INGEST_WAIT_SECS } else { 90 }
+$Image     = "$($env:ACR_NAME).azurecr.io/${AgentName}:${ImageTag}"
+
+Write-Host "==> Building and pushing image to ACR"
+az acr build --registry $env:ACR_NAME --image "${AgentName}:${ImageTag}" --no-logs --only-show-errors . | Out-Null
+
+Write-Host "==> Deploying to Azure Container Apps"
+$envVars = @(
+  "AGENT_NAME=$AgentName",
+  "OTEL_AGENT_ID=$OtelAgentId",
+  "APPLICATIONINSIGHTS_CONNECTION_STRING=$($env:APPLICATIONINSIGHTS_CONNECTION_STRING)",
+  "AZURE_OPENAI_ENDPOINT=$($env:AZURE_OPENAI_ENDPOINT)",
+  "AZURE_OPENAI_DEPLOYMENT=$($env:AZURE_OPENAI_DEPLOYMENT)",
+  "AZURE_OPENAI_API_VERSION=$(if ($env:AZURE_OPENAI_API_VERSION) { $env:AZURE_OPENAI_API_VERSION } else { '2024-10-21' })",
+  "AZURE_OPENAI_API_KEY=$($env:AZURE_OPENAI_API_KEY)"
+)
+
+$exists = az containerapp show -n $AgentName -g $env:RESOURCE_GROUP --only-show-errors 2>$null
+if ($LASTEXITCODE -eq 0 -and $exists) {
+  az containerapp update `
+    -n $AgentName `
+    -g $env:RESOURCE_GROUP `
+    --image $Image `
+    --set-env-vars $envVars `
+    --only-show-errors | Out-Null
+} else {
+  az containerapp create `
+    --name $AgentName `
+    --resource-group $env:RESOURCE_GROUP `
+    --environment $env:ACA_ENV `
+    --image $Image `
+    --target-port 8000 `
+    --ingress external `
+    --min-replicas 1 --max-replicas 2 `
+    --registry-server "$($env:ACR_NAME).azurecr.io" `
+    --system-assigned `
+    --registry-identity system `
+    --env-vars $envVars `
+    --only-show-errors | Out-Null
+}
+
+$Fqdn = az containerapp show -n $AgentName -g $env:RESOURCE_GROUP `
+        --query properties.configuration.ingress.fqdn -o tsv --only-show-errors
+$AgentUrl = "https://$Fqdn"
+Write-Host "==> Agent URL: $AgentUrl"
+
+Write-Host "==> Waiting for /healthz"
+for ($i = 0; $i -lt 30; $i++) {
+  try { Invoke-WebRequest -UseBasicParsing -Uri "$AgentUrl/healthz" -TimeoutSec 10 | Out-Null; break }
+  catch { Start-Sleep -Seconds 5 }
+}
+
+Write-Host "==> Generating traffic"
+$env:AGENT_URL = $AgentUrl
+$env:AGENT_NAME = $AgentName
+$env:OTEL_AGENT_ID = $OtelAgentId
+python generate_traffic.py
+
+if ($TraceIngestWaitSecs -gt 0) {
+  Write-Host "==> Waiting ${TraceIngestWaitSecs}s for OTel export and ingestion"
+  Start-Sleep -Seconds $TraceIngestWaitSecs
+}
+
+Write-Host "==> Done. Agent URL: $AgentUrl"
+Write-Host "==> Then run: python register_external_agent.py; python run_trace_eval.py"