feat(ai): PR 5 — test extractor + core runner [5/7]

[ianwhitedeveloper]

Summary

PR 5 of 7 in the consolidation of draft PR #394 — decomposing the 80-commit monolith into small, focused, dependency-ordered PRs. Targets ai-testing-framework-implementation-consolidation (staging area before master).

Depends on: PR 1 (ai-errors, constants), PR 2 (debug-logger, tap-yaml), PR 3 (agent-parser, execute-agent), PR 4 (agent-config, validation) — all merged.

Note: This PR supersedes the original pre-PR 5 prep work that was on this branch. The outputFormat string strategy and related tasks were deferred to post-consolidation (see Architectural Notes below). The branch was reset to consolidation HEAD and this implementation applied cleanly.

---

What's in This PR

New modules

• source/test-extractor.js — Two-agent prompt pipeline for extracting and evaluating test assertions:

  • buildExtractionPrompt — instructs an LLM to parse a test file into structured { userPrompt, importPaths, assertions[] } data
  • buildResultPrompt — instructs the result agent to execute the user prompt and return plain text
  • buildJudgePrompt — instructs the judge agent to evaluate one result against one requirement and return TAP YAML
  • extractTests — full extraction pipeline: calls extraction agent → validates result → resolves imported prompt files → returns { userPrompt, promptUnderTest, assertions }

• source/ai-runner.js — Orchestration layer for the two-agent test execution flow:

  • readTestFile — reads a test file from disk
  • runAITests — full pipeline: read file → extract tests → run N runs (result agent once per run, judge agents per assertion in parallel) → aggregate results
  • verifyAgentAuthentication — delegates to validation.js with injected executeAgent

Modified modules

• source/ai-errors.js — Three new error types (AgentConfigReadError, AgentConfigParseError, AgentConfigValidationError) + formatZodError moved here from agent-config.js (correct dependency direction for PR 5)
• source/execute-agent.js — truncateOutput helper extracted to eliminate the duplicated truncation expression
• source/agent-config.js — Updated to use specific AgentConfig* error types; imports formatZodError from ai-errors.js
• source/extraction-parser.js — resolveImportPaths signature updated from (importPaths, projectRoot, debug) to (importPaths, projectRoot, logger) for proper dependency injection

Dependency graph (this PR)

execute-agent.js  (leaf — spawns subprocess)
      ▲                        ▲
      │                        │
test-extractor.js          ai-runner.js
      ▲
      │
ai-runner.js

No cycles. test-extractor.js imports executeAgent directly from execute-agent.js — not from ai-runner.js. This was the circular dependency in the original feature branch.

---

WIP Fixes Applied

#	Issue	Resolution
12	Error handling/Zod placement in pipeline	formatZodError moved to ai-errors.js; agent-config.js now imports from there
13	Re-exports in test-extractor.js	Removed — parseTAPYAML and parseExtractionResult imported directly by callers
—	resolveImportPaths raw console.error(debug) calls	Replaced with injected logger parameter
—	AgentConfig* error types (Eric's PR #410 review)	Added to ai-errors.js; agent-config.js uses specific types; tests use handleAIErrors routing
—	truncateOutput duplicated expression	Extracted as named helper in execute-agent.js

---

Architectural Notes (Deferred)

outputFormat string strategy + parseOutput

The parseOutput: fn pattern stays as-is for PRs 5–7. The outputFormat declarative string strategy (making configs fully serializable for riteway ai init) and the IO/mapping separation in executeAgent are deferred to the post-consolidation epic. See the consolidation plan's Post-Consolidation section for details.

runAITests default agentConfig

The runAITests default ({ command: 'claude', args: [...] }) duplicates getAgentConfig('claude'). Deferred to post-consolidation alongside the outputFormat refactor — tracked in the plan.

---

Test Results

150 tests across 13 test files, all passing. 18 new tests: 12 in test-extractor.test.js + 6 in ai-runner.test.js.

• test-extractor tests cover buildExtractionPrompt, buildResultPrompt, buildJudgePrompt (full string comparison), and extractTests (integration tests using real node subprocesses for the agent mock, temp files for imports)
• ai-runner tests cover readTestFile and runAITests end-to-end using a two-agent node mock (extraction → result → judge) with hermetic temp dirs — no reliance on project files

npm test       → 150/150 passing (13 test files)
npm run lint   → Lint complete.
npm run ts     → TypeScript check complete.

---

Checklist

• All WIP fixes from the consolidation plan applied (Expose tape's fail function #12, drop should param in callback #13, logger injection, AgentConfig* errors, truncateOutput)
• No circular dependency — test-extractor.js → execute-agent.js (not ai-runner.js)
• No re-exports in test-extractor.js
• formatZodError moved to ai-errors.js (correct dependency direction)
• handleAIErrors routing pattern used throughout — no .cause.code inspection in tests
• test.each for any table-driven cases; no for...of loops in test files
• Integration tests use real node subprocesses; unit tests mock only at boundaries
• All error tests use allNoop spread pattern
• npm test — 150/150 passing
• npm run lint — clean
• npm run ts — clean