Integrate ae-agent as long-running agent for ArtEvalBench

Author: bastoica

benchmarks/arteval_bench/data/benchmark/README_ae_agent.md

@@ -0,0 +1,38 @@
+# Run ArtEval Benchmark with AE Agent
+
+This directory contains `arteval_tasks.jsonl` and other benchmark task definitions. To run the benchmark with **ae_agent**, start from the **benchmark root** (`benchmarks/arteval_bench/`).
+
+## Run from benchmark root
+
+```bash
+cd benchmarks/arteval_bench
+
+# Use ae_agent with data/benchmark/arteval_tasks.jsonl as input
+python src/main.py \
+  -i ./data/benchmark/arteval_tasks.jsonl \
+  -a ae_agent \
+  -m claude-sonnet-4-5-20250929 \
+  -o ./outputs/ae_agent_$(date +%Y-%m-%d_%H-%M-%S)
+```
+
+Or, if `run.sh` supports passing an agent argument:
+
+```bash
+cd benchmarks/arteval_bench
+./run.sh claude-sonnet-4-5-20250929 ae_agent
+```
+
+## Environment
+
+- Set `ANTHROPIC_API_KEY` or `ANTHROPIC_FOUNDRY_API_KEY`.
+- Optional: `ANTHROPIC_FOUNDRY_BASE_URL`, `CLAUDE_CODE_USE_FOUNDRY=1`.
+- The ae_agent implementation lives under `src/agents/ae_agent/`, synced with the standalone ae-agent repo (runner, install, utils, interactive_runner).
+
+## Task format
+
+Each line of `arteval_tasks.jsonl` is one JSON object, including at least:
+
+- `artifact_id`, `artifact_dir`, `artifact_readme`, `artifact_url`
+- `evaluator`: evaluation command (e.g. `cd /repo && python3 _agent_eval/main.py`)
+- `docker_env`: Docker image
+- `run_on_host`: when `true`, run on the host instead of Docker

benchmarks/arteval_bench/data/benchmark/run_ae_agent.sh

@@ -0,0 +1,16 @@
+#!/bin/bash
+# Run ArtEval benchmark with ae_agent. Execute this script from the benchmark root.
+# Usage: ./run_ae_agent.sh [optional: model name, default claude-sonnet-4-5-20250929]
+
+set -e
+BENCH_ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
+MODEL_NAME="${1:-claude-sonnet-4-5-20250929}"
+cd "$BENCH_ROOT"
+echo "==> ArtEval benchmark root: $BENCH_ROOT"
+echo "==> Model: $MODEL_NAME"
+echo "==> Agent: ae_agent"
+python src/main.py \
+  -i ./data/benchmark/arteval_tasks.jsonl \
+  -a ae_agent \
+  -m "$MODEL_NAME" \
+  -o "./outputs/ae_agent_${MODEL_NAME//\//_}_$(date +%Y-%m-%d_%H-%M-%S)"

benchmarks/arteval_bench/src/agents/ae_agent/README.md

@@ -0,0 +1,42 @@
+# AE Agent (ArtEval sub-agent)
+
+This agent is the **ae-agent** logic integrated as a sub-agent of the system-intelligence-benchmark ArtEval benchmark. It uses the Claude Agent SDK to run artifact evaluation tasks inside the benchmark container. Code is synced from the standalone [ae-agent](https://github.com/Couen/ae-agent) repo.
+
+## Files (synced from ae-agent)
+
+- **install.sh**: Installs `claude-agent-sdk==0.1.24` and configures `~/.claude/settings.json` (48h Bash timeout).
+- **runner.sh**: Entry point invoked as `runner.sh <model> <task_or_path>`. Forwards to `runner.py`. Uses `/agent/current_task.txt` when the benchmark passes task via file.
+- **runner.py**: Runs the task with Claude Agent SDK; supports rate-limit retry (429), message_formatter; second argument can be task text or path to file.
+- **utils.py**: `DEFAULT_TIMEOUT_MS` for the runner.
+- **interactive_runner.py**: Interactive multi-turn session inside container (e.g. `docker exec -it <cid> python3 /agent/interactive_runner.py <model>`).
+- **__init__.py**: Package marker.
+
+## Usage from the benchmark
+
+From the benchmark root (`benchmarks/arteval_bench/`):
+
+```bash
+python src/main.py -i ./data/benchmark/arteval_tasks.jsonl -a ae_agent -m claude-sonnet-4-5-20250929 -o ./outputs/ae_agent_run
+```
+
+Or use the helper script from `data/benchmark/`:
+
+```bash
+./data/benchmark/run_ae_agent.sh [model_name]
+```
+
+The benchmark will:
+
+1. Upload the agent to `/agent` in the container.
+2. For ae_agent: upload task to `/agent/current_task.txt`, then run `runner.sh "$model" /agent/current_task.txt` (avoids shell quoting with large tasks).
+3. Use long-running and live-log behavior (48h timeout, live log streaming, `_agent_eval` removal before run and re-upload before evaluation, container kept for debugging).
+4. Pass through `ANTHROPIC_API_KEY`, `ANTHROPIC_FOUNDRY_API_KEY`, `ANTHROPIC_FOUNDRY_BASE_URL`, `CLAUDE_CODE_USE_FOUNDRY` when set.
+
+## Dependencies
+
+- Python 3 with `claude-agent-sdk` (installed by `install.sh`).
+- Optional: `message_formatter` for prettier output (if present in the environment).
+
+## Relation to standalone ae-agent repo
+
+The standalone ae-agent repo provides a full CLI (`main.py`, `run_eval.py`, `utils.py`) and host/Docker orchestration. This sub-agent is the in-container runner only; the benchmark’s `run_eval_in_env.py` handles orchestration, task file upload, and Foundry env vars.

benchmarks/arteval_bench/src/agents/ae_agent/__init__.py

@@ -0,0 +1,4 @@
+"""AE Agent for ArtEvalBench - Claude Agent SDK runner for artifact evaluation tasks.
+
+Contract: artifact at /repo, this agent at /agent; task passed as CLI arg or path to file (/agent/current_task.txt).
+"""

benchmarks/arteval_bench/src/agents/ae_agent/install.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+# Setup AE Agent environment inside benchmark container.
+# Ensures claude-agent-sdk is available so runner.py can run.
+set -e
+if ! python3 -c "import claude_agent_sdk" 2>/dev/null; then
+  echo "Installing claude-agent-sdk..."
+  pip3 install claude-agent-sdk==0.1.24 || pip3 install --break-system-packages claude-agent-sdk==0.1.24 || true
+  if ! python3 -c "import claude_agent_sdk"; then
+    echo "WARNING: claude_agent_sdk still not importable; runner may fail."
+  fi
+fi
+# 48h Bash timeout for long-running artifact tasks
+mkdir -p ~/.claude
+cat > ~/.claude/settings.json << 'EOF'
+{
+  "env": {
+    "BASH_MAX_TIMEOUT_MS": "172800000",
+    "BASH_DEFAULT_TIMEOUT_MS": "172800000"
+  }
+}
+EOF
+echo "AE Agent environment ready (~/.claude/settings.json configured)."

benchmarks/arteval_bench/src/agents/ae_agent/interactive_runner.py

@@ -0,0 +1,105 @@
+#!/usr/bin/env python3
+"""Interactive runner for AE Agent - runs inside container after main task.
+
+Used when interactive=True: docker exec -it <container_id> python3 /agent/interactive_runner.py <model>
+Artifact at /repo; API keys from container env.
+"""
+
+import asyncio
+import os
+import sys
+
+sys.path.insert(0, '/agent')
+
+try:
+    from utils import DEFAULT_TIMEOUT_MS
+except ImportError:
+    DEFAULT_TIMEOUT_MS = 172_800_000
+
+try:
+    from claude_agent_sdk import ClaudeAgentOptions, ClaudeSDKClient
+except ImportError as e:
+    print(f"ERROR: claude_agent_sdk not available: {e}", file=sys.stderr)
+    sys.exit(1)
+
+
+def _build_system_prompt() -> str:
+    try:
+        timeout_ms_env = os.environ.get("BASH_MAX_TIMEOUT_MS")
+        timeout_ms = int(timeout_ms_env) if timeout_ms_env else DEFAULT_TIMEOUT_MS
+    except ValueError:
+        timeout_ms = DEFAULT_TIMEOUT_MS
+
+    return """You are an experienced software engineer in an interactive session.
+
+ENVIRONMENT:
+- You are inside a Docker container with root permissions.
+- The artifact repository is at /repo. Change to it: cd /repo
+- You have access to Read, Write, and Bash tools.
+
+TIMEOUT: Long-running commands can take hours; do not set short timeouts.
+
+You will receive follow-up instructions from the user. Complete each one and respond.
+If the user asks to stop or says 'quit'/'exit', acknowledge and they will end the session."""
+
+
+def _display_message(msg) -> None:
+    if hasattr(msg, 'content'):
+        for block in msg.content:
+            if hasattr(block, 'text'):
+                print(block.text, end='', flush=True)
+    print(flush=True)
+
+
+async def _interactive_loop(model_name: str) -> int:
+    options = ClaudeAgentOptions(
+        system_prompt=_build_system_prompt(),
+        allowed_tools=["Read", "Write", "Bash"],
+        setting_sources=["user"],
+    )
+
+    print("\n" + "=" * 60, flush=True)
+    print("Interactive mode - Agent ready. Type your instructions (or 'quit'/'exit' to end).", flush=True)
+    print("=" * 60 + "\n", flush=True)
+
+    async with ClaudeSDKClient(options=options) as client:
+        await client.query(
+            "Please confirm you are in /repo and ready for the user's follow-up instructions. Reply briefly that you are ready."
+        )
+        async for msg in client.receive_response():
+            _display_message(msg)
+
+        while True:
+            try:
+                user_input = input("\n>>> ").strip()
+            except (EOFError, KeyboardInterrupt):
+                print("\nExiting interactive mode.", flush=True)
+                return 0
+
+            if not user_input:
+                continue
+            if user_input.lower() in ('quit', 'exit', 'q'):
+                print("Exiting interactive mode.", flush=True)
+                return 0
+
+            await client.query(user_input)
+            async for msg in client.receive_response():
+                _display_message(msg)
+
+    return 0
+
+
+def main() -> int:
+    model_name = os.environ.get("AE_AGENT_MODEL", "claude-sonnet-4-5-20250929")
+    if len(sys.argv) >= 2:
+        model_name = sys.argv[1]
+
+    if not os.environ.get('ANTHROPIC_API_KEY') and not os.environ.get('ANTHROPIC_FOUNDRY_API_KEY'):
+        print("ERROR: ANTHROPIC_API_KEY or ANTHROPIC_FOUNDRY_API_KEY must be set.", file=sys.stderr)
+        return 1
+
+    return asyncio.run(_interactive_loop(model_name))
+
+
+if __name__ == "__main__":
+    sys.exit(main())