feat(ai): PR 5 — test extractor + core runner [5/7]

[ianwhitedeveloper]

Summary

PR 5 of 7 in the consolidation of draft PR #394 — decomposing the 80-commit monolith into small, focused, dependency-ordered PRs.

Depends on: PR 1 (ai-errors, constants), PR 2 (tap-yaml), PR 3 (agent-parser, execute-agent), PR 4 (agent-config, validation) — all merged. Plus #417 (debug-logger removal) which is the direct base branch for this PR.

Note: This PR supersedes the pre-PR 5 prep work that was on this branch (#415, now closed). The outputFormat string strategy and related tasks were deferred to post-consolidation (see Architectural Notes below). The branch was reset to consolidation HEAD and this implementation applied cleanly.

---

What's in This PR

New modules

• source/test-extractor.js — Two-agent prompt pipeline for extracting and evaluating test assertions:

  • buildExtractionPrompt — instructs an LLM to parse a test file into structured { userPrompt, importPaths, assertions[] } data
  • buildResultPrompt — instructs the result agent to execute the user prompt and return plain text
  • buildJudgePrompt — instructs the judge agent to evaluate one result against one requirement and return TAP YAML
  • extractTests — full extraction pipeline: calls extraction agent → validates result → resolves imported prompt files → returns { userPrompt, promptUnderTest, assertions }

• source/ai-runner.js — Orchestration layer for the two-agent test execution flow:

  • runAITests — full pipeline: read file → extract tests → run N runs (result agent once per run, judge agents per assertion in parallel) → aggregate results
  • verifyAgentAuthentication — delegates to validation.js with injected executeAgent

Modified modules

• source/ai-errors.js — Three new error types (AgentConfigReadError, AgentConfigParseError, AgentConfigValidationError)
• source/agent-config.js — Updated to use specific AgentConfig* error types; z.prettifyError() used directly for Zod error formatting (custom formatZodError abstraction removed as unnecessary)

Dependency graph (this PR)

execute-agent.js  (leaf — spawns subprocess)
      ▲                        ▲
      │                        │
test-extractor.js          ai-runner.js

No cycles. test-extractor.js imports executeAgent directly from execute-agent.js — not from ai-runner.js. This was the circular dependency in the original feature branch.

---

Fixes and remediations details

WIP Fixes Applied

#	Issue	Resolution
12	Zod error formatting in pipeline	Custom formatZodError removed; z.prettifyError() used directly in agent-config.js
13	Re-exports in test-extractor.js	Removed — parseTAPYAML and parseExtractionResult imported directly by callers
—	AgentConfig* error types (Eric's PR #410 review)	Added to ai-errors.js; agent-config.js uses specific types; tests use handleAIErrors routing

---

Review Remediations Applied

#	Finding	Resolution
1	Fragile error tests used package.json at process.cwd()	Replaced with hermetic temp dirs + explicit projectRoot in all affected tests
2	No error path tests in ai-runner.test.js	Added: file-not-found (ENOENT) and extraction agent non-zero exit (AgentProcessError routing)
3	loadAgentConfig command injection trust boundary undocumented	Added trust boundary note to JSDoc
4	Misleading testFilePath && guard in resolveImportPaths condition	Removed — resolution depends only on importPaths.length, not testFilePath
7	No test for testFilePath omitted path (import resolution still works)	Added resolves imports using projectRoot when testFilePath is not provided
8	Enhancement comments referenced style guide filenames	Converted to self-contained // TODO: markers
11	runAITests JSDoc missing agentConfig default	Documented [options.agentConfig=getAgentConfig()] with default note

Deferred with TODO(post-consolidation to mitigate derailing overall recomposition plan - can be decided upon once all functionality + e2e tests are in place): markers (tracked in consolidation plan):

• buildJudgePrompt missing promptUnderTest guard (inconsistency with buildResultPrompt)
• Validation ordering before IO in extractTests (structural errors should surface before resolveImportPaths)
• judgeAssertion parameter bloat from logging (assertionIndex/totalAssertions for console.log only)
• createTwoAgentMockArgs TAP YAML backtick fragility (use JSON.stringify)
• Path traversal in resolveImportPaths — intentional, requires GitHub issue for remediation epic

---

Architectural Notes (Deferred)

outputFormat string strategy + parseOutput

The parseOutput: fn pattern stays as-is for PRs 5–7. The outputFormat declarative string strategy (making configs fully serializable for riteway ai init) and the IO/mapping separation in executeAgent are deferred to the post-consolidation epic. See the consolidation plan's Post-Consolidation section for details.

runAITests default agentConfig

runAITests uses getAgentConfig('claude') as its default — eliminates the previously duplicated inline claude config literal. Deferred cleanup tracked in the plan.

---

Test Results

136 tests across 12 test files, all passing. 19 new tests: 13 in test-extractor.test.js + 6 in ai-runner.test.js.

• test-extractor tests cover buildExtractionPrompt, buildResultPrompt, buildJudgePrompt (full string comparison), and extractTests (integration tests using real node subprocesses for the agent mock, hermetic temp dirs for all fixtures — no reliance on project files)
• ai-runner tests cover runAITests end-to-end using a two-agent node mock (extraction → result → judge) with hermetic temp dirs, including error paths: file-not-found and extraction agent failure

npm test       → 136/136 passing (12 test files)
npm run lint   → Lint complete.
npm run ts     → TypeScript check complete.

---

Checklist

• No circular dependency — test-extractor.js → execute-agent.js (not ai-runner.js)
• No re-exports in test-extractor.js
• z.prettifyError() used directly — custom formatZodError abstraction removed
• handleAIErrors routing pattern used throughout — no .cause.code inspection in tests
• test.each for any table-driven cases; no for...of loops in test files
• Integration tests use real node subprocesses; all fixtures use hermetic temp dirs
• All error tests use allNoop spread pattern
• Error path tests added: file-not-found + agent process failure
• testFilePath guard removed — import resolution correct without it
• loadAgentConfig trust boundary documented
• npm test — 136/136 passing
• npm run lint — clean
• npm run ts — clean

Made with Cursor

[ianwhitedeveloper]

I determined this was a completely unnecessary abstraction