Stabilize runtime orchestration and fix flaky CI tests (pingdotgg#488)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: codex <codex@users.noreply.github.com>
Co-authored-by: Julius Marminge <julius0216@outlook.com>

Author: 4 people

.docs/architecture.md

@@ -5,17 +5,139 @@ T3 Code runs as a **Node.js WebSocket server** that wraps `codex app-server` (JS
 ```
 ┌─────────────────────────────────┐
 │  Browser (React + Vite)         │
-│  Connected via WebSocket        │
+│  wsTransport (state machine)    │
+│  Typed push decode at boundary  │
 └──────────┬──────────────────────┘
            │ ws://localhost:3773
 ┌──────────▼──────────────────────┐
 │  apps/server (Node.js)          │
 │  WebSocket + HTTP static server │
-│  ProviderManager                │
-│  CodexAppServerManager          │
+│  ServerPushBus (ordered pushes) │
+│  ServerReadiness (startup gate) │
+│  OrchestrationEngine            │
+│  ProviderService                │
+│  CheckpointReactor              │
+│  RuntimeReceiptBus              │
 └──────────┬──────────────────────┘
            │ JSON-RPC over stdio
 ┌──────────▼──────────────────────┐
 │  codex app-server               │
 └─────────────────────────────────┘
 ```
+
+## Components
+
+- **Browser app**: The React app renders session state, owns the client-side WebSocket transport, and treats typed push events as the boundary between server runtime details and UI state.
+
+- **Server**: `apps/server` is the main coordinator. It serves the web app, accepts WebSocket requests, waits for startup readiness before welcoming clients, and sends all outbound pushes through a single ordered push path.
+
+- **Provider runtime**: `codex app-server` does the actual provider/session work. The server talks to it over JSON-RPC on stdio and translates those runtime events into the app's orchestration model.
+
+- **Background workers**: Long-running async flows such as runtime ingestion, command reaction, and checkpoint processing run as queue-backed workers. This keeps work ordered, reduces timing races, and gives tests a deterministic way to wait for the system to go idle.
+
+- **Runtime signals**: The server emits lightweight typed receipts when important async milestones finish, such as checkpoint capture, diff finalization, or a turn becoming fully quiescent. Tests and orchestration code wait on these signals instead of polling internal state.
+
+## Event Lifecycle
+
+### Startup and client connect
+
+```mermaid
+sequenceDiagram
+    participant Browser
+    participant Transport as WsTransport
+    participant Server as wsServer
+    participant Layers as serverLayers
+    participant Ready as ServerReadiness
+    participant Push as ServerPushBus
+
+    Browser->>Transport: Load app and open WebSocket
+    Transport->>Server: Connect
+    Server->>Layers: Start runtime services
+    Server->>Ready: Wait for startup barriers
+    Ready-->>Server: Ready
+    Server->>Push: Publish server.welcome
+    Push-->>Transport: Ordered welcome push
+    Transport-->>Browser: Hydrate initial state
+```
+
+1. The browser boots [`WsTransport`][1] and registers typed listeners in [`wsNativeApi`][2].
+2. The server accepts the connection in [`wsServer`][3] and brings up the runtime graph defined in [`serverLayers`][7].
+3. [`ServerReadiness`][4] waits until the key startup barriers are complete.
+4. Once the server is ready, [`wsServer`][3] sends `server.welcome` from the contracts in [`ws.ts`][6] through [`ServerPushBus`][5].
+5. The browser receives that ordered push through [`WsTransport`][1], and [`wsNativeApi`][2] uses it to seed local client state.
+
+### User turn flow
+
+```mermaid
+sequenceDiagram
+    participant Browser
+    participant Transport as WsTransport
+    participant Server as wsServer
+    participant Provider as ProviderService
+    participant Codex as codex app-server
+    participant Ingest as ProviderRuntimeIngestion
+    participant Engine as OrchestrationEngine
+    participant Push as ServerPushBus
+
+    Browser->>Transport: Send user action
+    Transport->>Server: Typed WebSocket request
+    Server->>Provider: Route request
+    Provider->>Codex: JSON-RPC over stdio
+    Codex-->>Ingest: Provider runtime events
+    Ingest->>Engine: Normalize into orchestration events
+    Engine-->>Server: Domain events
+    Server->>Push: Publish orchestration.domainEvent
+    Push-->>Browser: Typed push
+```
+
+1. A user action in the browser becomes a typed request through [`WsTransport`][1] and the browser API layer in [`nativeApi`][12].
+2. [`wsServer`][3] decodes that request using the shared WebSocket contracts in [`ws.ts`][6] and routes it to the right service.
+3. [`ProviderService`][8] starts or resumes a session and talks to `codex app-server` over JSON-RPC on stdio.
+4. Provider-native events are pulled back into the server by [`ProviderRuntimeIngestion`][9], which converts them into orchestration events.
+5. [`OrchestrationEngine`][10] persists those events, updates the read model, and exposes them as domain events.
+6. [`wsServer`][3] pushes those updates to the browser through [`ServerPushBus`][5] on channels defined in [`orchestration.ts`][11].
+
+### Async completion flow
+
+```mermaid
+sequenceDiagram
+    participant Server as wsServer
+    participant Worker as Queue-backed workers
+    participant Cmd as ProviderCommandReactor
+    participant Checkpoint as CheckpointReactor
+    participant Receipt as RuntimeReceiptBus
+    participant Push as ServerPushBus
+    participant Browser
+
+    Server->>Worker: Enqueue follow-up work
+    Worker->>Cmd: Process provider commands
+    Worker->>Checkpoint: Process checkpoint tasks
+    Checkpoint->>Receipt: Publish completion receipt
+    Cmd-->>Server: Produce orchestration changes
+    Checkpoint-->>Server: Produce orchestration changes
+    Server->>Push: Publish resulting state updates
+    Push-->>Browser: User-visible push
+```
+
+1. Some work continues after the initial request returns, especially in [`ProviderRuntimeIngestion`][9], [`ProviderCommandReactor`][13], and [`CheckpointReactor`][14].
+2. These flows run as queue-backed workers using [`DrainableWorker`][16], which helps keep side effects ordered and test synchronization deterministic.
+3. When a milestone completes, the server emits a typed receipt on [`RuntimeReceiptBus`][15], such as checkpoint completion or turn quiescence.
+4. Tests and orchestration code wait on those receipts instead of polling git state, projections, or timers.
+5. Any user-visible state changes produced by that async work still go back through [`wsServer`][3] and [`ServerPushBus`][5].
+
+[1]: ../apps/web/src/wsTransport.ts
+[2]: ../apps/web/src/wsNativeApi.ts
+[3]: ../apps/server/src/wsServer.ts
+[4]: ../apps/server/src/wsServer/readiness.ts
+[5]: ../apps/server/src/wsServer/pushBus.ts
+[6]: ../packages/contracts/src/ws.ts
+[7]: ../apps/server/src/serverLayers.ts
+[8]: ../apps/server/src/provider/Layers/ProviderService.ts
+[9]: ../apps/server/src/orchestration/Layers/ProviderRuntimeIngestion.ts
+[10]: ../apps/server/src/orchestration/Layers/OrchestrationEngine.ts
+[11]: ../packages/contracts/src/orchestration.ts
+[12]: ../apps/web/src/nativeApi.ts
+[13]: ../apps/server/src/orchestration/Layers/ProviderCommandReactor.ts
+[14]: ../apps/server/src/orchestration/Layers/CheckpointReactor.ts
+[15]: ../apps/server/src/orchestration/Layers/RuntimeReceiptBus.ts
+[16]: ../packages/shared/src/DrainableWorker.ts

.docs/provider-architecture.md

@@ -3,7 +3,9 @@
 The web app communicates with the server via WebSocket using a simple JSON-RPC-style protocol:
 
 - **Request/Response**: `{ id, method, params }` → `{ id, result }` or `{ id, error }`
-- **Push events**: `{ type: "push", channel, data }` for orchestration read-model updates
+- **Push events**: typed envelopes with `channel`, `sequence` (monotonic per connection), and channel-specific `data`
+
+Push channels: `server.welcome`, `server.configUpdated`, `terminal.event`, `orchestration.domainEvent`. Payloads are schema-validated at the transport boundary (`wsTransport.ts`). Decode failures produce structured `WsDecodeDiagnostic` with `code`, `reason`, and path info.
 
 Methods mirror the `NativeApi` interface defined in `@t3tools/contracts`:
 
@@ -12,3 +14,17 @@ Methods mirror the `NativeApi` interface defined in `@t3tools/contracts`:
 - `shell.openInEditor`, `server.getConfig`
 
 Codex is the only implemented provider. `claudeCode` is reserved in contracts/UI.
+
+## Client transport
+
+`wsTransport.ts` manages connection state: `connecting` → `open` → `reconnecting` → `closed` → `disposed`. Outbound requests are queued while disconnected and flushed on reconnect. Inbound pushes are decoded and validated at the boundary, then cached per channel. Subscribers can opt into `replayLatest` to receive the last push on subscribe.
+
+## Server-side orchestration layers
+
+Provider runtime events flow through queue-based workers:
+
+1. **ProviderRuntimeIngestion** — consumes provider runtime streams, emits orchestration commands
+2. **ProviderCommandReactor** — reacts to orchestration intent events, dispatches provider calls
+3. **CheckpointReactor** — captures git checkpoints on turn start/complete, publishes runtime receipts
+
+All three use `DrainableWorker` internally and expose `drain()` for deterministic test synchronization.

.docs/workspace-layout.md

@@ -3,4 +3,5 @@
 - `/apps/server`: Node.js WebSocket server. Wraps Codex app-server, serves the built web app, and opens the browser on start.
 - `/apps/web`: React + Vite UI. Session control, conversation, and provider event rendering. Connects to the server via WebSocket.
 - `/apps/desktop`: Electron shell. Spawns a desktop-scoped `t3` backend process and loads the shared web app.
-- `/packages/contracts`: Shared Zod schemas and TypeScript contracts for provider events, WebSocket protocol, and model/session types.
+- `/packages/contracts`: Shared effect/Schema schemas and TypeScript contracts for provider events, WebSocket protocol, and model/session types.
+- `/packages/shared`: Shared runtime utilities consumed by both server and web. Uses explicit subpath exports (e.g. `@t3tools/shared/git`, `@t3tools/shared/DrainableWorker`) — no barrel index.

.mise.toml

@@ -0,0 +1,3 @@
+[tools]
+node = "24.13.1"
+bun = "1.3.9"

.plans/17-provider-neutral-runtime-determinism.md

@@ -0,0 +1,109 @@
+# Plan: Provider-Neutral Runtime Determinism and Flake Elimination
+
+## Summary
+Replace timing-sensitive websocket and orchestration behavior with explicit typed runtime boundaries, ordered push delivery, and server-owned completion receipts. The cutover is broad and single-shot: no compatibility shim, no mixed old/new transport. The design must reduce flakes without baking Codex-specific lifecycle semantics into generic runtime code.
+
+## Implementation Status
+
+All 7 sections are implemented. CI passes (format, lint, typecheck, test, browser test, build). One deferred item remains: the shared `WsTestClient` helper from section 7 — tests use direct transport subscription and receipt-based waits instead.
+
+### New files
+
+| File | Purpose |
+|------|---------|
+| `packages/shared/src/DrainableWorker.ts` | Queue-based Effect worker with deterministic `drain` signal |
+| `packages/shared/src/schemaJson.ts` | Two-phase JSON→Schema decode helpers (`decodeJsonResult`, `formatSchemaError`) |
+| `apps/server/src/wsServer/pushBus.ts` | `ServerPushBus` — ordered typed push pipeline with auto-incrementing sequence |
+| `apps/server/src/wsServer/readiness.ts` | `ServerReadiness` — Deferred-based barriers for startup sequencing |
+| `apps/server/src/orchestration/Services/RuntimeReceiptBus.ts` | Receipt schema union: checkpoint captured, diff finalized, turn quiesced |
+| `apps/server/src/orchestration/Layers/RuntimeReceiptBus.ts` | PubSub-backed receipt bus implementation |
+| `apps/server/src/watchFileWithStatPolling.ts` | Stat-polling file watcher for containers where `fs.watch` is unreliable |
+| `apps/server/vitest.config.ts` | Server-specific test config (timeout bumps) |
+| `apps/server/src/wsServer/pushBus.test.ts` | Push bus serialization and welcome-gating tests |
+| `packages/shared/src/DrainableWorker.test.ts` | Drainable worker enqueue/drain lifecycle tests |
+
+### Key modifications
+
+| File | Change |
+|------|--------|
+| `packages/contracts/src/ws.ts` | Channel-indexed `WsPushPayloadByChannel` map, `WsPush` union schema, `WsPushSequence` |
+| `apps/server/src/wsServer.ts` | Integrated `ServerPushBus` and `ServerReadiness`; welcome gated on readiness |
+| `apps/server/src/keybindings.ts` | Explicit runtime with `start`/`ready`/`snapshot`; dual `fs.watch` + stat-polling watcher |
+| `apps/web/src/wsTransport.ts` | Connection state machine (`connecting`→`open`→`reconnecting`→`closed`→`disposed`); two-phase decode at boundary; cached latest push by channel |
+| `apps/web/src/wsNativeApi.ts` | Removed decode logic; delegates to pre-validated transport messages |
+| `apps/server/src/orchestration/Layers/CheckpointReactor.ts` | Uses `DrainableWorker`; publishes completion receipts |
+| `apps/server/src/orchestration/Layers/ProviderCommandReactor.ts` | Uses `DrainableWorker` for command processing |
+| `apps/server/src/orchestration/Layers/ProviderRuntimeIngestion.ts` | Uses `DrainableWorker` for event ingestion |
+| `apps/server/integration/OrchestrationEngineHarness.integration.ts` | Receipt-based waits replace polling loops |
+
+## Key Changes
+### 1. Strengthen the generic boundaries, not the Codex boundary — DONE
+- `ProviderRuntimeEvent` remains the canonical provider event contract; `ProviderService` remains the only cross-provider facade.
+- Raw Codex payloads and event ordering stay isolated in `CodexAdapter.ts` and `codexAppServerManager.ts`.
+- `ProviderKind` was not expanded. The runtime stays provider-neutral by contract.
+
+### 2. Replace loose websocket envelopes with channel-indexed typed pushes — DONE
+- `packages/contracts/src/ws.ts` now derives push messages from a `WsPushPayloadByChannel` channel-to-schema map. `WsPush` is a union schema replacing `channel: string` + `data: unknown`.
+- Every server push carries `sequence: number`, auto-incremented in `ServerPushBus`.
+- `packages/shared/src/schemaJson.ts` provides structured decode diagnostics via `formatSchemaError`.
+- `packages/contracts/src/ws.test.ts` covers typed push envelope validation and channel/payload mismatch rejection.
+
+### 3. Introduce explicit server readiness and a single push pipeline — DONE
+- `apps/server/src/wsServer/pushBus.ts`: `ServerPushBus` with `publishAll` (broadcast) and `publishClient` (targeted) methods, backed by one ordered path. All pushes flow through it.
+- `apps/server/src/wsServer/readiness.ts`: `ServerReadiness` with Deferred-based barriers for HTTP listening, push bus, keybindings, terminal subscriptions, and orchestration subscriptions.
+- `server.welcome` is emitted only after connection-scoped and server-scoped readiness is complete.
+- `wsServer.ts` no longer publishes directly from ad hoc background streams.
+
+### 4. Turn background watchers into explicit runtimes — DONE
+- `apps/server/src/keybindings.ts` refactored as explicit `KeybindingsShape` service with `start`, `ready`, `snapshot` semantics.
+- Initial config load, cache warmup, and dual watcher attachment (`fs.watch` + `watchFileWithStatPolling`) complete before `ready` resolves.
+- `watchFileWithStatPolling.ts` is the thin adapter for environments where `fs.watch` is unreliable.
+
+### 5. Replace polling-based orchestration waiting with receipts — DONE
+- `RuntimeReceiptBus` service defines three receipt types: `CheckpointBaselineCapturedReceipt`, `CheckpointDiffFinalizedReceipt` (with `status: "ready"|"missing"|"error"`), and `TurnProcessingQuiescedReceipt`.
+- `CheckpointReactor`, `ProviderCommandReactor`, and `ProviderRuntimeIngestion` use `DrainableWorker` and publish receipts on completion.
+- Integration harness and checkpoint tests await receipts instead of polling snapshots and git refs.
+
+### 6. Centralize client transport state and decoding — DONE
+- `apps/web/src/wsTransport.ts` implements an explicit connection state machine: `connecting`, `open`, `reconnecting`, `closed`, `disposed`.
+- Two-phase decode (JSON parse → Schema validate) happens at the transport boundary. `wsNativeApi.ts` receives pre-validated messages.
+- Cached latest welcome/config modeled as explicit `latestPushByChannel` state.
+
+### 7. Replace ad hoc test helpers with semantic test clients — MOSTLY DONE
+- `DrainableWorker` replaces timing-sensitive `Effect.sleep` with deterministic `drain()` across reactor tests.
+- Orchestration harness waits on receipts/barriers instead of `waitForThread`, `waitForGitRef`, and retry loops.
+- Behavioral assertions moved to deterministic unit-style harnesses; narrow integration tests kept for real filesystem/socket behavior.
+- **Deferred:** Shared `WsTestClient` helper (connect, awaitSemanticWelcome, awaitTypedPush, trackSequence, matchRpcResponseById). Tests use direct transport subscription instead.
+
+## Provider-Coupling Guardrails
+- No generic runtime API may depend on Codex-native event names, thread IDs, or request payload shapes.
+- No readiness barrier may be defined as "Codex has emitted X." Readiness is owned by the server runtime, not by provider event order.
+- No websocket channel payload may contain raw provider-native payloads unless the channel is explicitly debug/internal.
+- Any provider-specific divergence must be exposed through provider capabilities from `ProviderService.getCapabilities()`, not `if provider === "codex"` branches in shared runtime code.
+- Generic tests must use canonical `ProviderRuntimeEvent` fixtures. Codex-specific ordering and translation tests stay in adapter/app-server suites only.
+- Keep UI/provider-specific knobs such as Codex-only options scoped to provider UX code. Do not pull them into generic transport or orchestration state.
+
+## Test Plan
+- Contracts:
+  - schema tests for typed push envelopes and structured decode diagnostics
+  - ordering tests for `sequence`
+- Server:
+  - readiness tests proving `server.welcome` cannot precede runtime readiness
+  - push bus tests proving terminal/config/orchestration pushes are serialized and typed
+  - keybindings runtime tests with fake watch source plus one real watcher integration test
+- Orchestration:
+  - receipt tests proving checkpoint refs and projections are complete before completion signals resolve
+  - replacement of polling-based checkpoint/integration waits with receipt-based waits
+- Web:
+  - transport tests for invalid JSON, invalid envelope, invalid payload, reconnect queue flushing, cached semantic state
+- Validation gate:
+  - `bun run lint`
+  - `bun run typecheck`
+  - `mise exec -- bun run test`
+  - repeated full-suite run after cutover to confirm flake removal
+
+## Assumptions and Defaults
+- This remains a single-provider product during the cutover, but the runtime contracts must stay provider-neutral.
+- No backward-compatibility layer is required for old websocket push envelopes.
+- The goal is deterministic runtime behavior first; reducing retries and sleeps in tests is a consequence, not the primary mechanism.
+- If a completion signal cannot be expressed provider-neutrally, it does not belong in the shared runtime layer and must stay adapter-local.