Agent-first architecture: agent_id as primary scoping key for pipelines, flows, jobs

[chubes4]

Vision

Turn Data Machine into agent infrastructure for WordPress. The agent — not the user — is the primary entity. Pipelines, flows, jobs, and memory files belong to agents. Users have access to agents (many-to-many). External runtimes (Kimaki, REST API clients, cron) execute as agents.

This enables:

• Site-wide shared agents that multiple users can access
• Personal agents per user for private workflows
• External runtimes (Kimaki, CLI) that authenticate as a specific agent
• Multi-site agent management — WordPress becomes agentic infrastructure

Current State

• datamachine_agents table exists with id, agent_slug, owner_id
• pipelines, flows, jobs have user_id columns (added in multi-agent Phase 1)
• PR feat(#573): multi-agent UI scoping — REST API user_id filtering and ownership guards #734 adds user-level REST API scoping — but scoping stops at the WordPress user
• Agent files (SOUL.md, MEMORY.md) are already filesystem-scoped by agent slug
• Kimaki sessions already authenticate as a specific agent via AGENTS.md injection

Proposed Model

Agent (agent_id) — primary entity
 ├── pipelines (agent_id column)
 ├── flows (agent_id column, inherited from pipeline)
 ├── jobs (agent_id column, inherited from flow)
 ├── memory files (SOUL.md, MEMORY.md — already scoped by slug)
 ├── runtime config (endpoint, API keys, model preferences)
 └── access grants (many-to-many with users)
      ├── User A → role: admin (full control)
      ├── User B → role: operator (run flows, view jobs)
      └── User C → role: viewer (read-only)

Key Principles

1. Agent is the citizen — not a sub-object of a user. Agents have identity, memory, and workflows independent of any particular user.
2. Users access agents — a join table (datamachine_agent_access) maps users to agents with role-based permissions.
3. Runtimes execute as agents — CLI, REST API, Kimaki, Action Scheduler all authenticate as a specific agent. The agent_id is the scoping key, not user_id.
4. Backward compatible — agent_id = NULL or 0 means "unscoped / single-agent mode" (existing installs work without migration).

Schema Changes

Modify existing tables

-- Add agent_id to pipelines, flows, jobs (alongside existing user_id)
ALTER TABLE datamachine_pipelines ADD COLUMN agent_id bigint(20) unsigned DEFAULT NULL;
ALTER TABLE datamachine_flows ADD COLUMN agent_id bigint(20) unsigned DEFAULT NULL;
ALTER TABLE datamachine_jobs ADD COLUMN agent_id bigint(20) unsigned DEFAULT NULL;

New table: agent access grants

CREATE TABLE datamachine_agent_access (
    id bigint(20) unsigned NOT NULL AUTO_INCREMENT,
    agent_id bigint(20) unsigned NOT NULL,
    user_id bigint(20) unsigned NOT NULL,
    role varchar(50) NOT NULL DEFAULT 'operator',
    granted_at datetime NOT NULL DEFAULT CURRENT_TIMESTAMP,
    PRIMARY KEY (id),
    UNIQUE KEY agent_user (agent_id, user_id),
    KEY user_id (user_id)
);

Roles

• admin — full control (create/edit/delete pipelines, manage agent settings, grant access)
• operator — run flows, view jobs, add to queues
• viewer — read-only access to pipelines, flows, jobs

Implementation Phases

Phase 1: Schema + DB layer

• Add agent_id columns with migration
• Create datamachine_agent_access table
• Update DB query methods to accept agent_id filter
• Backfill: existing data gets agent_id from the agent matching owner_id = user_id

Phase 2: Permission layer

• PermissionHelper::resolve_scoped_agent_id($request) — resolve which agent the request is scoped to
• PermissionHelper::can_access_agent($agent_id, $role) — check if acting user has required role on this agent
• REST endpoints scope by agent_id instead of (or in addition to) user_id

Phase 3: REST API + Admin UI

• Agent switcher in admin UI header (which agent am I working as?)
• Pipeline/flow creation assigns to active agent
• Agent management page: create agents, grant access, configure runtime
• ?agent_id=X query param on all list endpoints

Phase 4: Runtime authentication

• Kimaki/REST sessions authenticate as a specific agent (not just a user)
• PermissionHelper::acting_agent_id() alongside acting_user_id()
• Agent API keys for headless/external runtime authentication

Relationship to Existing PRs

• PR feat(#573): multi-agent UI scoping — REST API user_id filtering and ownership guards #734 (user-level scoping) — still valuable as the foundation. Agent-level scoping layers on top. user_id remains for "who created this" audit trail; agent_id becomes the primary scoping key.
• PR fix(#730): queue-add uses unified check-duplicate instead of QueueValidator #733 (Queue validator should run during queue add, not just publish #730 queue-add dedup) — independent, no conflict.

Questions

1. Should user_id columns be kept alongside agent_id for audit trail ("who created this pipeline"), or replaced entirely?
2. Should the owner_id on datamachine_agents become the default admin in datamachine_agent_access, or stay as a separate field?
3. For single-agent installs (no explicit agent setup), should DM auto-create a default agent on activation?

Depends on: #573 (user-level scoping — foundation)