[OneWorkflow][PoC] registerStep

[shahargl]

Custom Workflow Step Registration - PoC

Summary

This PR introduces a step registry mechanism that allows Kibana plugins to register custom workflow step types without modifying the core workflow engine. This is a proof of concept to validate the extensibility approach.

Motivation

The workflow engine currently has a fixed set of built-in step types (HTTP, Elasticsearch, Kibana, etc.). Teams need the ability to create domain-specific step types (e.g., enrichment steps, custom actions, ML inference) without requiring changes to the core engine.

Changes

Core Engine (workflows_execution_engine)

New Components:

• StepTypeRegistry: Registry class for managing custom step types
• CustomStepImpl: Execution handler for registered steps
• registerStepType(): Public API in plugin setup contract

Step Type Definition:

interface StepTypeDefinition {
  id: string;
  title: string;
  description?: string;
  inputSchema: ZodSchema;     // Validates inputs
  outputSchema: ZodSchema;    // Validates outputs
  handler: StepHandler;       // Execution logic
  documentation?: {           // Optional UI docs
    summary, details, url, examples
  }
}

Handler Context:
Custom step handlers receive:

• Validated input based on inputSchema
• Context manager for accessing workflow state
• Step state persistence API: setStepState() / getStepState()
• Logger scoped to the step execution
• Abort signal for cancellation support

UI/Validation (workflows_management)

Schema Updates:

• Added .passthrough() to step schemas to allow dynamic state properties
• Updated getSchemaAtPath() to recognize passthrough objects
• Registered steps included in workflow YAML schema generation

Hover Documentation:

• New RegisteredStepMonacoHandler for custom step tooltips
• Shows step title, description, examples, and documentation links
• Removes generic "Connector" terminology for registered steps

Example Plugin (workflows_step_example)

Created a proof-of-concept plugin demonstrating registration of a setvar step type:

Registration (plugin setup):

plugins.workflowsExecutionEngine.registerStepType(setvarStepDefinition);

Usage in workflows:

steps:
  - name: setVarStep
    type: setvar
    with:
      variables:
        x: 10
        userName: "Alice"
        
  - name: useVariable
    type: console
    with:
      message: "User {{ steps.setVarStep.userName }} has count {{ steps.setVarStep.x }}"

Technical Approach

Encapsulated State Management

Custom steps persist data using context.contextManager.setStepState(). The state is automatically spread into the step context, making it accessible to subsequent steps via {{ steps.stepName.property }}. This approach requires no changes to the core context management system.

Priority-Based Handler Resolution

The hover provider uses a priority system:

1. Elasticsearch/Kibana handlers (priority 20/15)
2. Registered step handler (priority 15)
3. Generic connector handler (priority 0, fallback)

This ensures registered steps get appropriate documentation instead of generic connector hints.

Schema Validation Flow

YAML → Zod schema validation → Handler execution → Output validation
         ↓                                              ↓
    inputSchema                                   outputSchema

Files Changed

Core:

• src/platform/plugins/shared/workflows_execution_engine/server/step_type_registry.ts (new)
• src/platform/plugins/shared/workflows_execution_engine/server/step/custom_step_impl.ts (new)
• src/platform/plugins/shared/workflows_execution_engine/server/plugin.ts (modified)
• src/platform/plugins/shared/workflows_execution_engine/server/types.ts (modified)

Types:

• src/platform/packages/shared/kbn-workflows/common/step_registry_types.ts (new)

UI/Validation:

• src/platform/plugins/shared/workflows_management/public/widgets/workflow_yaml_editor/lib/monaco_connectors/registered_step_monaco_handler.ts (new)
• src/platform/plugins/shared/workflows_management/public/features/workflow_context/lib/get_steps_collection_schema.ts (modified)
• src/platform/plugins/shared/workflows_management/common/lib/zod/zod_utils.ts (modified)

Example:

• examples/workflows_step_example/ (new)

Testing

Manual testing performed:

• Step registration during plugin setup
• Step execution with input validation
• State persistence across steps
• UI validation of dynamic properties
• Hover documentation display

TODO for production:

• Unit tests for StepTypeRegistry
• Integration tests for custom step execution
• UI tests for hover provider
• Error handling test cases

Breaking Changes

None. This is an additive change that introduces new APIs without modifying existing behavior.

Next Steps

If this approach is validated, next steps would include:

1. Comprehensive test coverage
2. Developer documentation
3. API stability review
4. Consider autocomplete improvements for registered step types

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: b4863ac

Failed CI Steps

• Linting
• Linting (with types)
• Check Types

History

• 💔 Build #359623 failed 0454aec
• 💔 Build #359621 failed ecd0cef
• 💔 Build #359620 failed b316124