diff --git a/docs/architecture.mdx b/docs/architecture.mdx index 8d91dfb6..ef146219 100644 --- a/docs/architecture.mdx +++ b/docs/architecture.mdx @@ -1,146 +1,123 @@ --- title: "Architecture" -description: "System design and module responsibilities for ShipSec Studio" +description: "How ShipSec Studio is designed under the hood — from the visual builder to infrastructure." --- ## What is ShipSec Studio? -ShipSec Studio is an open-source, no-code security workflow orchestration platform. Build, execute, and monitor security automation workflows through a visual interface — focus on security, not infrastructure. +ShipSec Studio is an **open-source, no-code security workflow orchestration platform**. Build, execute, and monitor security automation workflows through a visual canvas — focus on security logic, not infrastructure plumbing. --- ## System Overview -``` -┌────────────────────────────────────────────────────────────────────────┐ -│ FRONTEND │ -│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ -│ │ Visual │ │ Terminal │ │ Timeline │ │ Config │ │ -│ │ Builder │ │ Viewer │ │ Replay │ │ Panel │ │ -│ │ (ReactFlow) │ │ (xterm.js) │ │ (Zustand) │ │ (Forms) │ │ -│ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │ -└───────────────────────────────────┬────────────────────────────────────┘ - │ REST API + Unified SSE -┌───────────────────────────────────▼─────────────────────────────────────┐ -│ BACKEND │ -│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ -│ │Workflows │ │ Secrets │ │ Storage │ │ Trace │ │ Auth │ │ -│ │ + DSL │ │(AES-256) │ │ (MinIO) │ │ Events │ │ (Clerk) │ │ -│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ -│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ -│ │ Webhooks │ │Schedules │ │ Agents │ │Human │ │Integr- │ │ -│ │ │ │ (CRON) │ │ │ │Inputs │ │ations │ │ -│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ -└───────────────────────────────────┬─────────────────────────────────────┘ - │ Temporal Client -┌───────────────────────────────────▼─────────────────────────────────────┐ -│ TEMPORAL │ -│ Workflow Orchestration • Retry Logic • Durability │ -└───────────────────────────────────┬─────────────────────────────────────┘ - │ Activity Execution -┌───────────────────────────────────▼─────────────────────────────────────┐ -│ WORKER │ -│ ┌─────────────────────────────────────────────────────────────────┐ │ -│ │ COMPONENT REGISTRY │ │ -│ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ -│ │ │ Tools │ │ AI │ │ Human │ │ Core │ │ │ -│ │ │(Security)│ │ Agents │ │ in Loop │ │ Utils │ │ │ -│ │ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ │ -│ └─────────────────────────────────────────────────────────────────┘ │ -│ ┌─────────────────────────────────────────────────────────────────┐ │ -│ │ SERVICE ADAPTERS │ │ -│ │ Secrets │ Storage │ Artifacts │ Trace │ Terminal │ Logs │ │ -│ └─────────────────────────────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────────────────────┘ - │ -┌───────────────────────────────────▼─────────────────────────────────────┐ -│ INFRASTRUCTURE │ -│ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ -│ │PostgreSQL│ │ MinIO │ │ Redis │ │Redpanda │ │ Loki │ │ -│ │ (Data) │ │ (Files) │ │(Terminal)│ │ (Kafka) │ │ (Logs) │ │ -│ └──────────┘ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ -└─────────────────────────────────────────────────────────────────────────┘ -``` + + Svgviewer Output + --- ## Technology Stack -| Layer | Stack | -|-------|-------| -| **Frontend** | React 19, TypeScript, Vite, TailwindCSS, Radix UI, ReactFlow, xterm.js, Zustand | -| **Backend** | NestJS, TypeScript, Bun, Drizzle ORM, Clerk Auth | -| **Worker** | Node.js, TypeScript, Temporal SDK, Docker | -| **Infrastructure** | PostgreSQL 16, Temporal, MinIO, Redis, Redpanda (Kafka), Loki | +| Layer | Technologies | +| ------------------ | ------------------------------------------------------------------------------- | +| **Frontend** | React 19, TypeScript, Vite, TailwindCSS, Radix UI, ReactFlow, xterm.js, Zustand | +| **Backend** | NestJS, TypeScript, Bun, Drizzle ORM, Clerk Auth | +| **Worker** | Node.js, TypeScript, Temporal SDK, Docker | +| **Infrastructure** | PostgreSQL 16, Temporal, MinIO, Redis, Redpanda (Kafka), Loki | --- -## Core Deep-Dives - -To keep this guide concise, complicated subsystems are documented in their own dedicated files: - -- **[Workflow Compilation (DSL)](/architecture/workflow-compilation)**: How visual graphs are transformed into executable instructions. -- **[Temporal Orchestration](/architecture/temporal-orchestration)**: How we use Temporal for durability and worker scaling. -- **[Streaming Pipelines](/architecture/streaming-pipelines)**: How terminal, logs, and events are delivered in real-time. -- **[Human-in-the-Loop](/architecture/human-in-the-loop)**: How we pause workflows for manual approvals and forms. +## Core Subsystems + +These are the most complex parts of the system — each has its own dedicated deep-dive: + + + + How visual node graphs are compiled into executable workflow instructions. + + + How we use Temporal for durability, retries, and worker scaling. + + + How terminal output, logs, and events are delivered in real-time via SSE. + + + How workflows pause mid-execution for manual approvals and form inputs. + + --- ## Component Categories -Components are the building blocks of workflows: +Components are the drag-and-drop building blocks of every workflow. -| Category | Description | Examples | -|----------|-------------|----------| -| **security** | Security scanning and enumeration tools | Subfinder, DNSX, Nuclei, Naabu, HTTPx, TruffleHog | -| **ai** | AI/ML and agent components | LLM Generate, AI Agent, MCP Providers | -| **core** | Utility and data processing | HTTP Request, File Loader, Logic Script, JSON Transform | -| **notification** | Alerts and messaging | Slack, Email | -| **manual-action** | Human-in-the-loop | Approvals, Forms, Selection | -| **github** | GitHub integrations | Remove Org Membership | +| Category | What it does | Examples | +| ----------------- | --------------------------------- | ------------------------------------------------------- | +| **security** | Security scanning and enumeration | Subfinder, DNSX, Nuclei, Naabu, HTTPx, TruffleHog | +| **ai** | AI/LLM and agent components | LLM Generate, AI Agent, MCP Providers | +| **core** | Utility and data processing | HTTP Request, File Loader, Logic Script, JSON Transform | +| **notification** | Alerts and messaging | Slack, Email | +| **manual-action** | Human-in-the-loop controls | Approvals, Forms, Selection | +| **github** | GitHub integrations | Remove Org Membership | --- ## Security Architecture ### Authentication & Multi-tenancy -- **Clerk Integration** — Production-ready authentication for hosted environments. -- **Local Auth** — Default for local setup using `ADMIN_USERNAME` / `ADMIN_PASSWORD`. -- **Organization Isolation** — All data scoped by `organization_id`. + +- **Clerk Integration** — Production-ready auth for hosted deployments +- **Local Auth** — Default for local dev via `ADMIN_USERNAME` / `ADMIN_PASSWORD` env vars +- **Organization Isolation** — Every database record is scoped by `organization_id` — no data leaks between tenants ### Secrets Management -- **AES-256-GCM** encryption at rest. -- **Versioned secrets** with active/inactive tracking. -- **Master key** via `SECRET_STORE_MASTER_KEY` environment variable. + +- **AES-256-GCM** encryption for all secrets at rest +- **Versioned secrets** with active/inactive state tracking +- **Master key** provided via the `SECRET_STORE_MASTER_KEY` environment variable ### Container Isolation -- **IsolatedContainerVolume** — Per-tenant, per-run Docker volumes. See **[Isolated Volumes](/development/isolated-volumes)**. -- **Network isolation** — Components run with `network: none` or `bridge`. -- **Automatic cleanup** — Volumes destroyed after execution. + +- **IsolatedContainerVolume** — Each workflow run gets a dedicated, per-tenant Docker volume +- **Network isolation** — Components execute with `network: none` or `bridge` depending on requirements +- **Automatic cleanup** — Volumes are destroyed immediately after execution completes + + + For a full breakdown of isolated volume behavior, see the [Isolated Volumes](/development/isolated-volumes) guide. + --- ## Development URLs -All application services are accessible through nginx on port 80: +All services are accessible through **nginx on port 80** in development: -| Service | URL | -|---------|-----| -| Frontend | http://localhost/ | -| Backend API | http://localhost/api/ | -| Analytics | http://localhost/analytics/ | -| Temporal UI | http://localhost:8081 | -| MinIO Console | http://localhost:9001 | -| Redpanda Console | http://localhost:8082 | -| Loki | http://localhost:3100 | +| Service | URL | +| ---------------- | --------------------------- | +| Frontend | http://localhost/ | +| Backend API | http://localhost/api/ | +| Analytics | http://localhost/analytics/ | +| Temporal UI | http://localhost:8081 | +| MinIO Console | http://localhost:9001 | +| Redpanda Console | http://localhost:8082 | +| Loki | http://localhost:3100 | - Individual service ports (5173, 3211, 5601) are available for debugging but should not be used in normal development. All traffic flows through nginx on port 80. + Direct service ports (`5173`, `3211`, `5601`) are available for low-level debugging only. All normal development traffic should flow through nginx on port 80. --- -## Learn More - -- **Component Development**: `/development/component-development` -- **Getting Started**: `/getting-started` + + + ← Previous + + + Next → + + \ No newline at end of file diff --git a/docs/components/ai.mdx b/docs/components/ai.mdx index f8afb31b..7c0301bf 100644 --- a/docs/components/ai.mdx +++ b/docs/components/ai.mdx @@ -1,183 +1,209 @@ --- title: "AI Components" -description: "LLM integrations for intelligent workflow automation" +description: "Connect LLMs to your security workflows — triage alerts, investigate findings, and extract structured data automatically." --- -AI components in ShipSec Studio follow a **Provider-Consumer** architecture. +AI components let you plug **large language models directly into your workflows**. Summarize scan results, triage alerts, run autonomous investigations, and extract structured data — all without writing a single line of code. -1. **Providers**: Handle credentials, model selection, and API configuration (OpenAI, Gemini, OpenRouter). -2. **Consumers**: Execute specific tasks (Text Generation or Autonomous Agents) using the configuration emitted by a Provider. +They follow a simple **Provider → Consumer** pattern: + +- **Providers** handle credentials and model selection — set it once, reuse everywhere. +- **Consumers** do the actual AI work — text generation or autonomous agent reasoning. --- ## Providers -Provider nodes normalize credentials and model settings into a reusable **LLM Provider Config**. +A Provider node is always the **first step** in any AI chain. It holds your API key and model choice, and outputs a reusable config that Consumer nodes plug into. + + + By keeping credentials and model selection in one Provider node, you can swap models for your entire workflow by changing just one thing. + ### OpenAI Provider -Configures access to OpenAI or OpenAI-compatible endpoints. +Connects to OpenAI's API (or any OpenAI-compatible endpoint). | Input | Type | Description | -|-------|------|-------------| -| `apiKey` | Secret | OpenAI API key (typically from a Secret Loader) | +|---|---|---| +| `apiKey` | Secret | Your OpenAI API key — use **Secret Loader** to supply this safely | | Parameter | Type | Description | -|-----------|------|-------------| -| `model` | Select | `gpt-5.2`, `gpt-5.1`, `gpt-5`, `gpt-5-mini` | -| `apiBaseUrl` | Text | Optional override for the API base URL | +|---|---|---| +| `model` | Select | `gpt-5.2` · `gpt-5.1` · `gpt-5` · `gpt-5-mini` | +| `apiBaseUrl` | Text | Override the API base URL — useful for self-hosted or proxy endpoints | + +--- ### Gemini Provider -Configures access to Google's Gemini models. +Connects to Google's Gemini model family. | Input | Type | Description | -|-------|------|-------------| -| `apiKey` | Secret | Google AI API key | +|---|---|---| +| `apiKey` | Secret | Your Google AI API key | | Parameter | Type | Description | -|-----------|------|-------------| -| `model` | Select | `gemini-3-pro-preview`, `gemini-3-flash-preview`, `gemini-2.5-pro` | -| `apiBaseUrl` | Text | Optional override for the API base URL | +|---|---|---| +| `model` | Select | `gemini-3-pro-preview` · `gemini-3-flash-preview` · `gemini-2.5-pro` | +| `apiBaseUrl` | Text | Optional API base URL override | | `projectId` | Text | Optional Google Cloud project identifier | +--- + ### OpenRouter Provider -Configures access to multiple LLM providers through OpenRouter's unified API. +Connects to [OpenRouter](https://openrouter.ai) — a unified API that gives you access to models from Anthropic, Google, Meta, Mistral, and more through a single key. | Input | Type | Description | -|-------|------|-------------| -| `apiKey` | Secret | OpenRouter API key | +|---|---|---| +| `apiKey` | Secret | Your OpenRouter API key | | Parameter | Type | Description | -|-----------|------|-------------| -| `model` | Text | Model slug (e.g., `openrouter/auto`, `anthropic/claude-3.5-sonnet`) | -| `apiBaseUrl` | Text | Optional override for the API base URL | -| `httpReferer` | Text | Application URL for OpenRouter ranking | -| `appTitle` | Text | Application title for OpenRouter ranking | +|---|---|---| +| `model` | Text | Model slug — e.g. `anthropic/claude-sonnet-4-5`, `openrouter/auto` | +| `apiBaseUrl` | Text | Optional API base URL override | +| `httpReferer` | Text | Your app URL — used for OpenRouter analytics | +| `appTitle` | Text | Your app name — used for OpenRouter analytics | --- ## Consumers -Consumer nodes perform the actual AI work. They require a **Provider Config** output from one of the providers above. +Consumer nodes do the actual AI work. Every consumer has a `chatModel` input — **connect your Provider output here.** ### AI Generate Text -Performs a one-shot chat completion. +A single-shot prompt-response. Give it a prompt, get back text. Simple and fast. | Input | Type | Description | -|-------|------|-------------| -| `userPrompt` | Text | The primary request or data to process | -| `chatModel` | Credential | **Required.** Connect a Provider output here | -| `modelApiKey` | Secret | Optional. supersedes the API key in the Provider Config | +|---|---|---| +| `chatModel` | Credential | **Required** — connect a Provider output here | +| `userPrompt` | Text | The question, instruction, or data to process | +| `modelApiKey` | Secret | Optional — overrides the API key from the Provider | | Parameter | Type | Description | -|-----------|------|-------------| -| `systemPrompt`| Textarea | Instructions that guide the model's behavior | -| `temperature` | Number | Creativity vs. determinism (0.0 to 2.0) | -| `maxTokens` | Number | Maximum tokens to generate | +|---|---|---| +| `systemPrompt` | Textarea | High-level instructions that shape the model's behavior (e.g. "You are a senior security analyst") | +| `temperature` | Number | Controls creativity vs. consistency — `0.0` is deterministic, `2.0` is creative | +| `maxTokens` | Number | Cap on how long the response can be | | Output | Type | Description | -|--------|------|-------------| -| `responseText`| Text | The assistant's response | -| `usage` | JSON | Token consumption metadata | -| `rawResponse` | JSON | Full API response for debugging | +|---|---|---| +| `responseText` | Text | The model's response | +| `usage` | JSON | Token usage metadata | +| `rawResponse` | JSON | Full API response — useful for debugging | --- ### AI SDK Agent -An autonomous agent that uses reasoning steps and tool-calling to solve complex tasks. +An **autonomous agent** that reasons step-by-step, calls tools, and iterates until it completes a task. Think of it as giving the AI a goal rather than a single question. | Input | Type | Description | -|-------|------|-------------| -| `userInput` | Text | The task or question for the agent | -| `chatModel` | Credential | **Required.** Connect a Provider output here | -| `conversationState` | JSON | Optional. Connect from a previous turn for memory | -| `mcpTools` | List | Optional. Connect tools from MCP Providers | +|---|---|---| +| `chatModel` | Credential | **Required** — connect a Provider output here | +| `userInput` | Text | The task or goal for the agent | +| `conversationState` | JSON | Optional — pass in a previous turn's state to give the agent memory | +| `mcpTools` | List | Optional — connect tools from MCP Provider nodes | | Parameter | Type | Description | -|-----------|------|-------------| -| `systemPrompt`| Textarea | Core identity and constraints for the agent | -| `temperature` | Number | Reasoning creativity (default 0.7) | -| `stepLimit` | Number | Max "Think -> Act -> Observe" loops (1-12) | -| `memorySize` | Number | Number of previous turns to retain in context | -| `structuredOutputEnabled` | Toggle | Enable to enforce a specific JSON output structure | -| `schemaType` | Select | How to define the schema: `json-example` or `json-schema` | -| `jsonExample` | JSON | Example JSON object for schema inference (all properties become required) | -| `jsonSchema` | JSON | Full JSON Schema definition for precise validation | -| `autoFixFormat` | Toggle | Attempt to extract valid JSON from malformed responses | +|---|---|---| +| `systemPrompt` | Textarea | Core identity and constraints — defines who the agent is and what it's allowed to do | +| `temperature` | Number | Reasoning creativity (default: `0.7`) | +| `stepLimit` | Number | Max Think → Act → Observe loops before stopping (range: `1–12`) | +| `memorySize` | Number | How many previous turns to keep in context | +| `structuredOutputEnabled` | Toggle | Force the agent to always return a specific JSON structure | +| `schemaType` | Select | How to define the output schema: `json-example` or `json-schema` | +| `jsonExample` | JSON | Provide an example JSON object — all fields become required | +| `jsonSchema` | JSON | Provide a full JSON Schema for precise validation | +| `autoFixFormat` | Toggle | Try to salvage valid JSON from a malformed response | | Output | Type | Description | -|--------|------|-------------| -| `responseText`| Text | Final answer after reasoning is complete | -| `structuredOutput` | JSON | Parsed structured output (when enabled) | -| `conversationState` | JSON | Updated state to pass to the next agent node | -| `reasoningTrace` | JSON | Detailed step-by-step logs of the agent's thoughts | +|---|---|---| +| `responseText` | Text | The agent's final answer after all reasoning steps | +| `structuredOutput` | JSON | Parsed, validated JSON output (when Structured Output is enabled) | +| `conversationState` | JSON | Updated state — loop this back into the next agent node for multi-turn memory | +| `reasoningTrace` | JSON | Step-by-step log of every thought and action the agent took | | `agentRunId` | Text | Unique session ID for tracking and streaming | + + Use **Structured Output** whenever you need the agent's response to feed into another component downstream. It guarantees the JSON schema is correct every time — no fragile prompt-parsing required. + + --- ## MCP Tools (Model Context Protocol) -ShipSec Studio supports the **Model Context Protocol (MCP)**, allowing AI agents to interact with external tools over HTTP. +MCP lets your AI Agent **call external tools** — things like searching logs, querying APIs, or running lookups — as part of its reasoning loop. You define what tools are available; the agent decides when and how to use them. ### MCP HTTP Tools -Exposes a set of tools from a remote HTTP server that implements the MCP contract. +Connects to a remote server that exposes tools over HTTP using the MCP protocol. -| Input | Type | Description | -|-------|------|-------------| -| `endpoint` | Text | The HTTP URL where the MCP server is hosted | -| `headersJson` | Text | Optional JSON of headers (e.g., Auth tokens) | -| `tools` | JSON | List of tool definitions available on that endpoint | - -| Parameter | Type | Description | -|-----------|------|-------------| -| `endpoint` | Text | Destination URL for tool execution | -| `tools` | JSON | Array of tools with `id`, `title`, and `arguments` | +| Input / Parameter | Type | Description | +|---|---|---| +| `endpoint` | Text | The URL of the MCP server | +| `headersJson` | Text | Optional JSON headers (e.g. `{"Authorization": "Bearer ..."}`) | +| `tools` | JSON | Tool definitions — each needs an `id`, `title`, and `arguments` schema | | Output | Type | Description | -|--------|------|-------------| -| `tools` | List | Normalized MCP tool definitions for the AI Agent | +|---|---|---| +| `tools` | List | Normalized tool list ready to plug into an AI Agent | + +--- ### MCP Tool Merge -Combines multiple MCP tool lists into a single consolidated list. +Combines tool lists from multiple MCP servers into one unified list for the agent. | Input | Type | Description | -|-------|------|-------------| -| `toolsA`, `toolsB`| List | Multiple upstream MCP tool outputs | - -| Parameter | Type | Description | -|-----------|------|-------------| -| `slots` | JSON | Configure additional input ports for merging | +|---|---|---| +| `toolsA`, `toolsB` | List | Outputs from two or more MCP HTTP Tools nodes | +| `slots` | JSON | Add more input ports if you have more than two sources | | Output | Type | Description | -|--------|------|-------------| -| `tools` | List | De-duplicated list of tools ready for an agent | +|---|---|---| +| `tools` | List | De-duplicated, merged tool list — connect directly to the Agent's `mcpTools` input | --- -## Use Cases +## Real-World Use Cases + +### Alert Triage +> *"Is this alert real or noise?"* +``` +OpenAI Provider → AI Generate Text +``` + +Pass raw alert data into the prompt and let the model filter signal from noise before it ever reaches a human. +``` +System Prompt: "You are a security analyst. Classify alerts as TRUE_POSITIVE or FALSE_POSITIVE with a one-sentence reason." +User Prompt: "{{alertPayload}}" +``` -### Automated Alert Triage -**Flow:** `Provider` → `AI Generate Text` +--- -Analyze incoming security alerts to filter out false positives. -**Prompt:** "Given this alert payload: {{alert}}, determine if it's a real threat or noise." +### Autonomous Investigation +> *"Investigate this IP using every tool available."* +``` +OpenAI Provider + MCP HTTP Tools → AI SDK Agent +``` -### Investigative Agent -**Flow:** `Provider` + `MCP Tool` → `AI Agent` +Give the agent a task and a set of tools (Splunk search, VirusTotal lookup, etc.) and let it figure out the investigation steps itself. +``` +Task: "Investigate IP {{ip}}. Use the available tools to determine if it is malicious." +``` -An agent that searches through logs and performs lookups to investigate a specific IP address. -**Task:** "Investigate the IP {{ip}} using the available Splunk and VirusTotal tools." +--- -### Structured Output for Data Extraction -**Flow:** `Provider` → `AI Agent` (with Structured Output enabled) +### Structured Data Extraction +> *"Turn this messy report into clean JSON."* +``` +OpenAI Provider → AI SDK Agent (Structured Output enabled) +``` -Extract structured data from unstructured security reports. Enable **Structured Output** and provide a JSON example: +Enable **Structured Output** and provide a JSON example. The agent will always return data in exactly this shape — no prompt-wrangling, no parsing errors. ```json { "severity": "high", @@ -185,20 +211,26 @@ Extract structured data from unstructured security reports. Enable **Structured "remediation_steps": ["Patch CVE-2024-1234", "Restart service"] } ``` -The agent will always return validated JSON matching this schema, ready for downstream processing. --- ## Best Practices - - **The Provider Concept**: Always place a Provider node (OpenAI/Gemini/OpenRouter) at the start of your AI chain. This allows you to swap models or providers for the entire workflow by changing just one node. - +**Use System Prompts for behavior, User Prompts for data.** System prompts define *who the model is* and *how it should respond*. User prompts carry the actual data to process. Keep them separate. -### Prompt Engineering -1. **Use Structured Output**: When you need consistent JSON for downstream nodes, enable **Structured Output** instead of relying on prompt instructions. This guarantees schema compliance and eliminates parsing errors. -2. **Use System Prompts**: Set high-level rules (e.g., "You are a senior security researcher") in the System Prompt parameter instead of the User Input. -3. **Variable Injection**: Use `{{variableName}}` syntax to inject data from upstream nodes into your prompts. +**Inject upstream data with `{{variableName}}` syntax.** Any output from a previous component can be dropped into a prompt — just reference it by name. + +**Loop `conversationState` for multi-turn memory.** Connect the `conversationState` output of one Agent node into the `conversationState` input of the next to give your agent persistent memory across steps. + +**Always use Secret Loader for API keys.** Never paste API keys directly into Provider parameters — always connect them via the Secret Loader component. + +--- -### Memory & State -For multi-turn conversations, always loop the `conversationState` output of the AI Agent back into the `conversationState` input of the next agent invocation (or store it in a persistent variable). + + + ← Previous + + + Next → + + \ No newline at end of file diff --git a/docs/components/core.mdx b/docs/components/core.mdx index 3ad0c765..3303467f 100644 --- a/docs/components/core.mdx +++ b/docs/components/core.mdx @@ -1,46 +1,48 @@ --- title: "Core Components" -description: "Essential building blocks for workflow inputs, outputs, and data transformation" +description: "The essential building blocks for starting workflows, moving data, transforming it, and storing results." --- -Core components handle workflow triggers, file operations, data transformation, and output destinations. +Core components are the **glue of every workflow**. They handle how a workflow starts, how data moves between steps, how secrets are accessed, and where results end up. Every workflow you build will use at least a few of these. --- ## Triggers +Triggers are always the **first component** in a workflow. Nothing runs until a trigger fires. + ### Manual Trigger -Starts a workflow manually. Configure runtime inputs to collect data (files, text, etc.) when triggered. +The simplest way to start a workflow — you click **Run**, fill in any required inputs, and it kicks off. -| Parameter | Type | Description | -|-----------|------|-------------| -| `runtimeInputs` | JSON | Define inputs to collect at runtime | +| Parameter | Type | What it does | +|---|---|---| +| `runtimeInputs` | JSON | Define what fields to show the user at runtime (files, text, numbers, etc.) | -**Supported input types:** `file`, `text`, `number`, `json`, `array` +**Supported input types:** `file` · `text` · `number` · `json` · `array` -**Example use cases:** -- Collect uploaded scope files before running security scans -- Prompt operators for target domains or API keys +**Good for:** +- Asking an operator to upload a scope file before a scan starts +- Prompting for a target domain or API key right before execution --- ### Webhook -Sends JSON payloads to external HTTP endpoints with retries and timeouts. +Fires a workflow automatically when an external system sends an HTTP request to a generated URL. -| Parameter | Type | Description | -|-----------|------|-------------| -| `url` | URL | Destination endpoint | -| `method` | Select | POST, PUT, or PATCH | -| `payload` | JSON | Request body | -| `headers` | JSON | HTTP headers | -| `timeoutMs` | Number | Request timeout (default: 30000) | -| `retries` | Number | Retry attempts (default: 3) | +| Parameter | Type | What it does | +|---|---|---| +| `url` | URL | Where to send the request | +| `method` | Select | `POST`, `PUT`, or `PATCH` | +| `payload` | JSON | The request body | +| `headers` | JSON | Any custom HTTP headers | +| `timeoutMs` | Number | How long to wait before giving up (default: `30000` ms) | +| `retries` | Number | How many times to retry on failure (default: `3`) | -**Example use cases:** -- Send scan results to Slack or Teams -- POST assets to a custom API +**Good for:** +- Automatically triggering a scan when a new asset is added to your system +- Posting scan results to a custom internal API --- @@ -48,49 +50,51 @@ Sends JSON payloads to external HTTP endpoints with retries and timeouts. ### File Loader -Loads file content from storage for use in workflows. +Loads a previously uploaded file from storage so its contents can flow into the rest of your workflow. | Input | Type | Description | -|-------|------|-------------| -| `fileId` | UUID | File ID from uploaded file | +|---|---|---| +| `fileId` | UUID | The ID of the file you want to load | | Output | Type | Description | -|--------|------|-------------| -| `file` | Object | File metadata + base64 content | -| `textContent` | String | Decoded UTF-8 text | +|---|---|---| +| `file` | Object | File metadata plus base64-encoded content | +| `textContent` | String | The file's contents as plain readable text | --- ### Text Splitter -Splits text into an array of strings by separator. +Takes a block of text and breaks it into a list of items — like splitting a list of domains by line. -| Parameter | Type | Description | -|-----------|------|-------------| -| `text` | String/File | Text content to split | -| `separator` | String | Split character (default: `\n`) | +| Parameter | Type | What it does | +|---|---|---| +| `text` | String / File | The text you want to split | +| `separator` | String | What character to split on (default: `\n` — new line) | | Output | Type | Description | -|--------|------|-------------| -| `items` | Array | Split strings | -| `count` | Number | Number of items | +|---|---|---| +| `items` | Array | The list of split strings | +| `count` | Number | How many items were produced | -**Example:** Split newline-delimited subdomains before passing to scanners. + + Use this after **File Loader** to turn a newline-separated list of domains into an array you can pass to a scanner. + --- ### Text Joiner -Joins array elements into a single string. +The opposite of Text Splitter — takes a list of items and merges them into one string. -| Parameter | Type | Description | -|-----------|------|-------------| -| `items` | Array | Array of strings to join | -| `separator` | String | Join character (default: `\n`) | +| Parameter | Type | What it does | +|---|---|---| +| `items` | Array | The list of strings to join | +| `separator` | String | What to put between each item (default: `\n`) | | Output | Type | Description | -|--------|------|-------------| -| `text` | String | Joined string | +|---|---|---| +| `text` | String | The combined result | --- @@ -98,113 +102,135 @@ Joins array elements into a single string. ### Secret Loader -Fetches secrets from the ShipSec-managed secret store. +Safely fetches a stored secret (API key, password, token) from ShipSec's encrypted secret store and makes it available to downstream components — **without ever exposing the value in logs.** -| Parameter | Type | Description | -|-----------|------|-------------| -| `secretName` | Secret | Secret name or UUID | -| `version` | Number | Optional version pin | -| `outputFormat` | Select | `raw` or `json` | +| Parameter | Type | What it does | +|---|---|---| +| `secretName` | Secret | The name or UUID of the secret to fetch | +| `version` | Number | Pin to a specific version (optional) | +| `outputFormat` | Select | `raw` for plain text, `json` for parsed object | | Output | Type | Description | -|--------|------|-------------| -| `secret` | Any | Resolved secret value (masked in logs) | -| `metadata` | Object | Secret version info | +|---|---|---| +| `secret` | Any | The secret value — automatically masked in all logs | +| `metadata` | Object | Version info and metadata about the secret | - Secret values are automatically masked in all logs and terminal output. + Secret values are **automatically redacted** from all logs, terminal output, and trace events. They are never stored in plain text. --- ## Data Transformation +These components let you reshape, filter, and debug data as it flows through your workflow. + ### Array Pick -Extracts specific items from an array by index. +Pulls specific items out of an array by their position (index). -| Parameter | Type | Description | -|-----------|------|-------------| -| `array` | Array | Source array | -| `indices` | Array | Indices to pick | +| Parameter | Type | What it does | +|---|---|---| +| `array` | Array | The source list | +| `indices` | Array | Which positions to extract (e.g. `[0, 2, 4]`) | | Output | Type | Description | -|--------|------|-------------| -| `picked` | Array | Selected items | +|---|---|---| +| `picked` | Array | Just the items you selected | + +--- ### Array Pack -Combines multiple values into a single array. +Bundles multiple separate values into a single array — useful when you need to combine outputs from different components. -| Parameter | Type | Description | -|-----------|------|-------------| -| `values` | Any[] | Values to pack | +| Parameter | Type | What it does | +|---|---|---| +| `values` | Any[] | The values to bundle together | | Output | Type | Description | -|--------|------|-------------| -| `array` | Array | Packed array | +|---|---|---| +| `array` | Array | The combined array | + +--- ### Console Log -Outputs data to workflow logs for debugging. +Prints data to the workflow's log panel. Use this when you're building or debugging a workflow and want to inspect what's flowing between steps. + +| Parameter | Type | What it does | +|---|---|---| +| `data` | Any | The value you want to inspect | +| `label` | String | An optional label to identify the log entry | -| Parameter | Type | Description | -|-----------|------|-------------| -| `data` | Any | Data to log | -| `label` | String | Optional label | + + Console Log doesn't affect your workflow's execution — it's purely for visibility. Remove it before running workflows in production. + --- -## Storage Destinations +## Storage & Destinations + +Where your workflow results end up. ### Artifact Writer -Writes workflow artifacts to ShipSec storage. +Saves a file or data blob to ShipSec's built-in storage, making it downloadable from the workflow run page. -| Parameter | Type | Description | -|-----------|------|-------------| -| `content` | Any | Content to store | -| `filename` | String | Artifact filename | -| `mimeType` | String | Content type | +| Parameter | Type | What it does | +|---|---|---| +| `content` | Any | The data to store | +| `filename` | String | What to name the file | +| `mimeType` | String | The file type (e.g. `text/plain`, `application/json`) | | Output | Type | Description | -|--------|------|-------------| -| `artifactId` | UUID | Stored artifact ID | -| `url` | String | Download URL | +|---|---|---| +| `artifactId` | UUID | A unique ID for the stored artifact | +| `url` | String | A direct download URL | + +--- ### File Writer -Writes content to a file in workflow storage. +Writes raw text content to a file path in workflow storage. + +| Parameter | Type | What it does | +|---|---|---| +| `content` | String | The text to write | +| `path` | String | Where to save it (e.g. `results/output.txt`) | -| Parameter | Type | Description | -|-----------|------|-------------| -| `content` | String | File content | -| `path` | String | File path | +--- ### Destination S3 -Uploads files to an S3-compatible bucket. +Uploads a file directly to any S3-compatible bucket (AWS S3, MinIO, Cloudflare R2, etc.). -| Parameter | Type | Description | -|-----------|------|-------------| -| `bucket` | String | S3 bucket name | -| `key` | String | Object key | -| `content` | Buffer | File content | -| `credentials` | Object | AWS credentials | +| Parameter | Type | What it does | +|---|---|---| +| `bucket` | String | The bucket name | +| `key` | String | The object path/key inside the bucket | +| `content` | Buffer | The file content to upload | +| `credentials` | Object | AWS credentials (use **AWS Credentials** component) | + +--- ### AWS Credentials -Provides AWS credentials for S3 operations. +Provides AWS credentials to other components that need them (like **Destination S3**). Connect its output to the `credentials` input of any AWS-powered component. -| Parameter | Type | Description | -|-----------|------|-------------| -| `accessKeyId` | Secret | AWS Access Key ID | -| `secretAccessKey` | Secret | AWS Secret Access Key | -| `region` | String | AWS region | +| Parameter | Type | What it does | +|---|---|---| +| `accessKeyId` | Secret | Your AWS Access Key ID | +| `secretAccessKey` | Secret | Your AWS Secret Access Key | +| `region` | String | AWS region (e.g. `us-east-1`) | | Output | Type | Description | -|--------|------|-------------| -| `credentials` | Object | Credential object for S3 components | +|---|---|---| +| `credentials` | Object | A credentials object ready to plug into S3 components | + + + Always use the **Secret Loader** to supply `accessKeyId` and `secretAccessKey` — never paste keys directly into parameters. + --- @@ -212,37 +238,48 @@ Provides AWS credentials for S3 operations. ### Analytics Sink -Indexes workflow output data into OpenSearch for analytics dashboards, queries, and alerts. Connect the `results` port from upstream security scanners. +Sends workflow output data into **OpenSearch** so you can query it, visualize it in dashboards, and track findings over time. Connect it to the `results` port of any scanner. | Input | Type | Description | -|-------|------|-------------| -| `data` | Any | Data to index. Works best with `list` from scanner `results` ports. | +|---|---|---| +| `data` | Any | The data to index — works best with `list` from scanner `results` ports | | Output | Type | Description | -|--------|------|-------------| -| `indexed` | Boolean | Whether data was successfully indexed | -| `documentCount` | Number | Number of documents indexed | -| `indexName` | String | Name of the OpenSearch index used | - -| Parameter | Type | Description | -|-----------|------|-------------| -| `indexSuffix` | String | Custom suffix for the index name. Defaults to slugified workflow name. | -| `assetKeyField` | Select | Field to use as asset identifier. Options: auto, asset_key, host, domain, subdomain, url, ip, asset, target, custom | -| `customAssetKeyField` | String | Custom field name when assetKeyField is "custom" | -| `failOnError` | Boolean | When enabled, workflow stops if indexing fails. Default: false (fire-and-forget) | - -**How it works:** - -1. Each item in the input array becomes a separate document -2. Workflow context is added under `shipsec.*` namespace -3. Nested objects are serialized to JSON strings (prevents field explosion) -4. All documents get the same `@timestamp` - -**Example use cases:** -- Index Nuclei scan results for trend analysis -- Store TruffleHog secrets for tracking over time -- Aggregate vulnerability data across workflows +|---|---|---| +| `indexed` | Boolean | Whether indexing succeeded | +| `documentCount` | Number | How many records were indexed | +| `indexName` | String | The OpenSearch index that was used | + +| Parameter | Type | What it does | +|---|---|---| +| `indexSuffix` | String | Custom suffix appended to the index name. Defaults to the workflow name. | +| `assetKeyField` | Select | Which field to use as the asset identifier. Options: `auto`, `host`, `domain`, `subdomain`, `url`, `ip`, `asset`, `target`, `custom` | +| `customAssetKeyField` | String | Your own field name when `assetKeyField` is set to `custom` | +| `failOnError` | Boolean | If `true`, the workflow stops when indexing fails. Default: `false` (fire-and-forget) | + +**How it works behind the scenes:** + +1. Each item in the input array becomes its own searchable document +2. Workflow metadata is added automatically under the `shipsec.*` namespace +3. Nested objects are serialized to prevent index field explosion +4. All documents share the same `@timestamp` for time-series querying + +**Good for:** +- Tracking Nuclei vulnerability findings over time +- Storing TruffleHog secrets for audit trails +- Aggregating results across multiple workflows into one dashboard - See [Workflow Analytics](/development/workflow-analytics) for detailed setup and querying guide. + Analytics Sink requires OpenSearch to be configured. See the [Workflow Analytics](/development/workflow-analytics) guide for full setup instructions. + +--- + + + + ← Previous + + + Next → + + \ No newline at end of file diff --git a/docs/components/overview.mdx b/docs/components/overview.mdx index ba42b9d7..0344ca7d 100644 --- a/docs/components/overview.mdx +++ b/docs/components/overview.mdx @@ -1,87 +1,146 @@ --- title: "Components Overview" -description: "Drag-and-drop building blocks for security automation workflows" +description: "The drag-and-drop building blocks you connect together to automate security workflows — no code required." --- -Components are the building blocks of ShipSec Studio workflows. Each component performs a specific task and can be connected together to create powerful automation pipelines. +## What Are Components? + +Think of components like **LEGO bricks for security automation**. Each one does one specific job — scan subdomains, probe for live servers, detect leaked secrets, send a Slack message. You drag them onto a canvas, connect them together, and ShipSec Studio runs the whole chain automatically. + +No scripting. No glue code. Just connect and run. + +--- ## Component Categories - + - Triggers, file handling, data transformation, and outputs + Triggers, file handling, data transformation, logic, and outputs. The backbone of every workflow. - Subdomain discovery, port scanning, DNS resolution, secret detection + Industry-standard security tools — subdomain discovery, port scanning, DNS resolution, secret detection, and more. - Provider configurations and autonomous agents + Connect LLM providers and autonomous agents to analyze, triage, and summarize findings automatically. -## How Components Work +--- -Each component has: +## How a Component Works -- **Inputs** – Data ports that accept connections from other components -- **Outputs** – Data ports that can be connected to downstream components -- **Parameters** – Configurable settings in the sidebar +Every component has three parts you interact with: -Components run inside Docker containers for isolation and reliability. Workflows are orchestrated by [Temporal](https://temporal.io/) with automatic retries and resumability. +- **Inputs** — the left-side connection points. This is where data flows *in* from a previous component. +- **Outputs** — the right-side connection points. This is where processed data flows *out* to the next component. +- **Parameters** — the settings panel on the right sidebar. Configure behavior per-component without touching any code. -## Component Structure + + Every component runs inside its own **isolated Docker container**. This means a broken or misbehaving component can't affect the rest of your workflow. Each run is sandboxed, reproducible, and clean. + -```typescript -interface ComponentDefinition { - id: string; // Unique identifier - label: string; // Display name - category: string; // Category for grouping - runner: RunnerConfig; // Execution configuration - inputSchema: ZodSchema; - outputSchema: ZodSchema; - execute: (input: Input, context: ExecutionContext) => Promise; -} -``` +Workflows are orchestrated by [Temporal](https://temporal.io/) under the hood — which means if something fails mid-run, it **automatically retries** and picks up exactly where it left off. You never lose progress. + +--- ## Example Pipelines +Here are two real-world workflows you can build in minutes by connecting components on the canvas. + ### Attack Surface Discovery +> *"What's exposed on the internet for this domain?"* ``` -Manual Trigger → Subfinder → DNSx → httpx → Notify +Manual Trigger → Subfinder → DNSx → HTTPx → Notify ``` -1. **Manual Trigger** – User provides target domains -2. **Subfinder** – Discovers subdomains -3. **DNSx** – Resolves DNS records -4. **httpx** – Probes for live HTTP services -5. **Notify** – Sends results to Slack, Discord, or Teams +| Step | Component | What it does | +|---|---|---| +| 1 | **Manual Trigger** | You provide one or more target domains | +| 2 | **Subfinder** | Discovers all known subdomains | +| 3 | **DNSx** | Resolves each subdomain to an IP address | +| 4 | **HTTPx** | Checks which ones have live, responding web services | +| 5 | **Notify** | Sends the final list to Slack, Discord, or Teams | + +--- -### Secret Detection +### Secret Detection + AI Triage +> *"Are there any leaked credentials in this repo, and how bad are they?"* ``` Manual Trigger → TruffleHog → OpenAI Provider → AI Generate Text → Notify ``` -1. **Manual Trigger** – User provides repository URL -2. **TruffleHog** – Scans for leaked credentials -3. **OpenAI Provider** – Set up model and API key -4. **AI Generate Text** – Analyzes and prioritizes findings -5. **Notify** – Alerts team via Slack, Discord, or email +| Step | Component | What it does | +|---|---|---| +| 1 | **Manual Trigger** | You provide a Git repository URL | +| 2 | **TruffleHog** | Scans every commit for leaked API keys, tokens, and passwords | +| 3 | **OpenAI Provider** | Configures your LLM model and API key | +| 4 | **AI Generate Text** | Analyzes the findings and writes a prioritized summary | +| 5 | **Notify** | Alerts your team with the AI-generated report | + + + You can save any workflow as a **template** and reuse it across projects. Build once, run everywhere. + + +--- + +## What Components Can Access + +When a component runs, it has access to a set of shared services behind the scenes. You don't configure these manually — they're available automatically. + +| Service | What it gives the component | +|---|---| +| `storage` | Upload and download files to the workflow's storage bucket | +| `secrets` | Securely read encrypted API keys and credentials | +| `artifacts` | Save output files that can be downloaded after the run | +| `trace` | Record events for the real-time timeline view | +| `logger` | Write structured logs visible in the log panel | +| `terminal` | Stream live terminal output to the Studio UI | + +--- + +## Under the Hood (For the Curious) + +Every component is defined with a strict input/output contract using TypeScript and [Zod](https://zod.dev) schemas. This is what guarantees that components connect cleanly — the output of one component always matches what the next one expects. +```typescript +interface ComponentDefinition { + id: string; // Unique identifier + label: string; // Display name in the canvas + category: string; // Which category group it appears in + runner: RunnerConfig; // How and where it executes (Docker image, etc.) + inputSchema: ZodSchema; // Validates incoming data + outputSchema: ZodSchema; // Validates outgoing data + execute: (input: Input, context: ExecutionContext) => Promise; +} +``` + + + You don't need to understand this to use components. This is only relevant if you're **building your own custom component** from scratch. + -## Execution Context +--- -Components receive an `ExecutionContext` with access to: +## Want to Build Your Own? -| Service | Description | -|---------|-------------| -| `storage` | File upload/download to MinIO | -| `secrets` | Encrypted secrets access | -| `artifacts` | Workflow artifact storage | -| `trace` | Event recording | -| `logger` | Structured logging | -| `terminal` | Terminal output streaming | +If the built-in components don't cover your use case, you can build a custom one. It's a TypeScript file, a Docker image, and a schema — and it'll show up in the canvas just like any other component. -## Building Custom Components + + + Step-by-step guide to building, testing, and registering a custom component. + + + Browse all available security tools — Subfinder, Nuclei, DNSX, HTTPx, and more. + + -See the [Component Development Guide](/development/component-development) for detailed instructions on building your own components. +--- + + + + ← Previous + + + Next → + + \ No newline at end of file diff --git a/docs/components/security.mdx b/docs/components/security.mdx index 38c39655..d6af5a00 100644 --- a/docs/components/security.mdx +++ b/docs/components/security.mdx @@ -1,99 +1,107 @@ --- title: "Security Components" -description: "Open source security tools for security automation workflows" +description: "Production-ready wrappers around the most trusted open-source security tools — drag, connect, and run." --- -Security components wrap popular open-source tools for subdomain discovery, DNS resolution, port scanning, and secret detection. +Security components wrap the **industry's most trusted open-source tools** into drag-and-drop workflow blocks. No CLI knowledge required — configure everything from the sidebar and connect tools together in seconds. --- ## Subdomain Discovery +Find every subdomain attached to a target domain before you start scanning. More subdomains = more attack surface coverage. + ### Subfinder -[GitHub](https://github.com/projectdiscovery/subfinder) · Docker: `ghcr.io/shipsecai/subfinder` + Passive subdomain enumeration using 40+ data sources — no direct interaction with the target. + [GitHub](https://github.com/projectdiscovery/subfinder) · Docker: `ghcr.io/shipsecai/subfinder` -Discovers subdomains using passive sources. - | Input | Type | Description | -|-------|------|-------------| -| `domains` | Array | Target domains to enumerate | -| `providerConfig` | Secret | Optional provider-config.yaml for authenticated sources | +|---|---|---| +| `domains` | Array | The target domains you want to enumerate | +| `providerConfig` | Secret | Optional `provider-config.yaml` to unlock authenticated sources (Shodan, Censys, etc.) | | Output | Type | Description | -|--------|------|-------------| -| `subdomains` | Array | Discovered subdomain hostnames | -| `rawOutput` | String | Raw tool output | -| `subdomainCount` | Number | Total subdomains found | - -**Example command:** `subfinder -d example.com -silent` +|---|---|---| +| `subdomains` | Array | All discovered subdomain hostnames | +| `subdomainCount` | Number | Total count | +| `rawOutput` | String | Raw tool output for debugging | +```bash +# What runs under the hood +subfinder -d example.com -silent +``` --- ### Amass -[GitHub](https://github.com/owasp-amass/amass) · Docker: `ghcr.io/shipsecai/amass` + Deep subdomain enumeration using both passive sources and active DNS techniques. + [GitHub](https://github.com/owasp-amass/amass) · Docker: `ghcr.io/shipsecai/amass` -Active and passive subdomain enumeration. - | Input | Type | Description | -|-------|------|-------------| +|---|---|---| | `domains` | Array | Target domains | -| `mode` | Select | `passive` or `active` | +| `mode` | Select | `passive` — no direct contact · `active` — DNS brute-force and zone transfers | | Parameter | Type | Description | -|-----------|------|-------------| -| `timeout` | Number | Timeout in minutes | -| `maxDns` | Number | Max DNS queries per second | +|---|---|---| +| `timeout` | Number | Max run time in minutes | +| `maxDns` | Number | DNS queries per second (throttle to avoid rate limits) | + + + Use `passive` mode for stealth. Use `active` mode when you have explicit authorization and want maximum coverage. + --- ### ShuffleDNS + MassDNS -[GitHub (ShuffleDNS)](https://github.com/projectdiscovery/shuffledns) · [GitHub (MassDNS)](https://github.com/blechschmidt/massdns) · Docker: `ghcr.io/shipsecai/shuffledns-massdns` + High-speed DNS bruteforcing using a wordlist — resolves millions of potential subdomains in seconds. + [ShuffleDNS](https://github.com/projectdiscovery/shuffledns) · [MassDNS](https://github.com/blechschmidt/massdns) · Docker: `ghcr.io/shipsecai/shuffledns-massdns` -High-performance DNS bruteforcing and resolution. This is a combined image that has both ShuffleDNS with MassDNS pre-installed. +This is a combined image with both tools pre-installed. ShuffleDNS manages the wordlist bruteforce; MassDNS handles the high-throughput resolution. | Input | Type | Description | -|-------|------|-------------| -| `domains` | Array | Target domains | -| `wordlist` | File | Wordlist for bruteforcing | -| `resolvers` | Array | Custom DNS resolvers | +|---|---|---| +| `domains` | Array | Target domains to bruteforce against | +| `wordlist` | File | Subdomain wordlist (e.g. `subdomains-top1million.txt`) | +| `resolvers` | Array | Custom DNS resolver IPs to use | --- ## DNS Resolution +Once you have a list of subdomains, resolve them — find out which ones are real, what IPs they point to, and who's hosting them. + ### DNSX -[GitHub](https://github.com/projectdiscovery/dnsx) · Docker: `ghcr.io/shipsecai/dnsx` + Fast, multi-threaded DNS resolution with support for every record type. + [GitHub](https://github.com/projectdiscovery/dnsx) · Docker: `ghcr.io/shipsecai/dnsx` -Resolves DNS records with support for multiple record types and custom resolvers. - | Input | Type | Description | -|-------|------|-------------| -| `domains` | Array | Domains to resolve | -| `recordTypes` | Array | DNS types: A, AAAA, CNAME, MX, NS, TXT, etc. | -| `resolvers` | Array | Custom resolver IPs (e.g., 1.1.1.1:53) | +|---|---|---| +| `domains` | Array | Subdomains or hostnames to resolve | +| `recordTypes` | Array | Which DNS records to fetch: `A`, `AAAA`, `CNAME`, `MX`, `NS`, `TXT`, etc. | +| `resolvers` | Array | Custom resolver IPs (e.g. `1.1.1.1:53`) | | Parameter | Type | Description | -|-----------|------|-------------| -| `threads` | Number | Concurrent workers (default: 100) | -| `retryCount` | Number | Retry attempts (default: 2) | -| `rateLimit` | Number | Requests per second | -| `showCdn` | Boolean | Annotate CDN providers | -| `showAsn` | Boolean | Include ASN info | +|---|---|---| +| `threads` | Number | How many lookups to run simultaneously (default: `100`) | +| `retryCount` | Number | Retries per failed lookup (default: `2`) | +| `rateLimit` | Number | Max requests per second | +| `showCdn` | Boolean | Tag results with their CDN provider (Cloudflare, Akamai, etc.) | +| `showAsn` | Boolean | Include ASN / ISP information in results | | Output | Type | Description | -|--------|------|-------------| +|---|---|---| | `results` | Array | DNS responses grouped by record type | | `resolvedHosts` | Array | Unique resolved hostnames | | `rawOutput` | String | Raw JSONL output | @@ -102,64 +110,70 @@ Resolves DNS records with support for multiple record types and custom resolvers ## HTTP Probing -### httpx +Figure out which discovered hosts actually have a web server running — and capture key metadata about each one. + +### HTTPx -[GitHub](https://github.com/projectdiscovery/httpx) · Docker: `ghcr.io/shipsecai/httpx` + Bulk HTTP probing — status codes, titles, tech stack, TLS info, and more. + [GitHub](https://github.com/projectdiscovery/httpx) · Docker: `ghcr.io/shipsecai/httpx` -Probes hosts for live HTTP services and captures response metadata. - | Input | Type | Description | -|-------|------|-------------| +|---|---|---| | `targets` | Array | Hostnames or URLs to probe | | Parameter | Type | Description | -|-----------|------|-------------| -| `ports` | String | Comma-separated ports (e.g., "80,443,8080") | -| `statusCodes` | String | Filter by HTTP status codes | -| `threads` | Number | Concurrency level | -| `followRedirects` | Boolean | Follow HTTP redirects | -| `tlsProbe` | Boolean | Probe TLS endpoints | -| `path` | String | Specific path to probe | +|---|---|---| +| `ports` | String | Ports to check (e.g. `"80,443,8080,8443"`) | +| `statusCodes` | String | Only return results with these status codes | +| `threads` | Number | Concurrent probe workers | +| `followRedirects` | Boolean | Follow HTTP redirects automatically | +| `tlsProbe` | Boolean | Include TLS certificate details | +| `path` | String | Append a specific path to every probe (e.g. `/admin`) | | Output | Type | Description | -|--------|------|-------------| -| `results` | Array | HTTP response metadata | +|---|---|---| +| `results` | Array | Per-host HTTP metadata (status, title, tech, headers) | | `rawOutput` | String | Raw httpx JSON lines | - -**Example command:** `httpx -l targets.txt -json -status-code 200,301` +```bash +# What runs under the hood +httpx -l targets.txt -json -status-code 200,301 +``` --- ## Port Scanning +Discover which ports are open on a target — the first step to understanding what services are exposed. + ### Naabu -[GitHub](https://github.com/projectdiscovery/naabu) · Docker: `ghcr.io/shipsecai/naabu` + Fast, reliable port scanner using SYN and CONNECT probes. + [GitHub](https://github.com/projectdiscovery/naabu) · Docker: `ghcr.io/shipsecai/naabu` -Fast active port scanning using SYN/CONNECT probes. - | Input | Type | Description | -|-------|------|-------------| -| `targets` | Array | Hostnames or IPs to scan | +|---|---|---| +| `targets` | Array | Hostnames or IP addresses to scan | | Parameter | Type | Description | -|-----------|------|-------------| -| `ports` | String | Custom ports (e.g., "80,443,1000-2000") | -| `topPorts` | Number | Scan top N common ports | -| `rate` | Number | Packets per second | +|---|---|---| +| `ports` | String | Specific ports or ranges (e.g. `"22,80,443,1000-2000"`) | +| `topPorts` | Number | Scan the top N most common ports | +| `rate` | Number | Packets per second — lower this on rate-limited networks | | `retries` | Number | Retry attempts per port | -| `enablePing` | Boolean | Ping probe before scanning | +| `enablePing` | Boolean | Ping the host first to check if it's up | | Output | Type | Description | -|--------|------|-------------| +|---|---|---| | `findings` | Array | Open ports per target | -| `openPortCount` | Number | Total open ports found | - -**Example command:** `naabu -host scanme.sh -top-ports 100` +| `openPortCount` | Number | Total open ports found across all targets | +```bash +# What runs under the hood +naabu -host scanme.sh -top-ports 100 +``` --- @@ -168,26 +182,31 @@ Fast active port scanning using SYN/CONNECT probes. ### Nuclei -[GitHub](https://github.com/shipsecai/tools/pkgs/container/nuclei) · Docker: `ghcr.io/shipsecai/nuclei` + Template-based vulnerability scanning — thousands of community-maintained detection rules built in. + [GitHub](https://github.com/projectdiscovery/nuclei) · Docker: `ghcr.io/shipsecai/nuclei` -Template-based vulnerability scanning. This is nuclei custom image with nuclei-templates baked in. +This is a custom Nuclei image with the full **nuclei-templates** library baked in — no separate download needed. | Input | Type | Description | -|-------|------|-------------| +|---|---|---| | `targets` | Array | URLs or hosts to scan | -| `templates` | Array | Template IDs or paths | +| `templates` | Array | Specific template IDs or paths to run (leave empty to run all) | | Parameter | Type | Description | -|-----------|------|-------------| -| `severity` | Array | Filter by severity (critical, high, medium, low) | -| `rate` | Number | Requests per second | -| `concurrency` | Number | Parallel template executions | +|---|---|---| +| `severity` | Array | Only run templates matching these severities: `critical`, `high`, `medium`, `low` | +| `rate` | Number | HTTP requests per second | +| `concurrency` | Number | Number of templates running in parallel | | Output | Type | Description | -|--------|------|-------------| -| `findings` | Array | Detected vulnerabilities | -| `criticalCount` | Number | Critical findings count | +|---|---|---| +| `findings` | Array | All detected vulnerabilities with full detail | +| `criticalCount` | Number | Number of critical-severity findings | + + + Only run Nuclei against systems you own or have explicit written authorization to test. Active scanning without permission is illegal in most jurisdictions. + --- @@ -196,64 +215,65 @@ Template-based vulnerability scanning. This is nuclei custom image with nuclei-t ### TruffleHog -[GitHub](https://github.com/trufflesecurity/trufflehog) · Docker: `ghcr.io/shipsecai/trufflehog` + Scans Git history, cloud storage, filesystems, and container images for leaked credentials. + [GitHub](https://github.com/trufflesecurity/trufflehog) · Docker: `ghcr.io/shipsecai/trufflehog` -Scans for leaked credentials across repositories, filesystems, and cloud storage. - | Input | Type | Description | -|-------|------|-------------| -| `scanTarget` | String | Repository URL, path, bucket, or image | -| `scanType` | Select | git, github, gitlab, s3, filesystem, docker | +|---|---|---| +| `scanTarget` | String | A repo URL, file path, S3 bucket, or Docker image to scan | +| `scanType` | Select | `git` · `github` · `gitlab` · `s3` · `filesystem` · `docker` | | Parameter | Type | Description | -|-----------|------|-------------| -| `onlyVerified` | Boolean | Show only verified secrets (default: true) | -| `branch` | String | Specific branch to scan | -| `sinceCommit` | String | Scan commits since reference (for PR scanning) | +|---|---|---| +| `onlyVerified` | Boolean | Only surface secrets that TruffleHog can actively verify as live (default: `true`) | +| `branch` | String | Limit scan to a specific Git branch | +| `sinceCommit` | String | Only scan commits after this reference — great for PR-level scanning | | Output | Type | Description | -|--------|------|-------------| -| `secrets` | Array | Detected secrets with verification status | -| `verifiedCount` | Number | Number of verified secrets | -| `hasVerifiedSecrets` | Boolean | Alert flag | - -**Example command:** `trufflehog git https://github.com/org/repo --results=verified --json` +|---|---|---| +| `secrets` | Array | All detected secrets with type, location, and verification status | +| `verifiedCount` | Number | How many secrets are confirmed active | +| `hasVerifiedSecrets` | Boolean | `true` if any live secrets were found — use this to trigger alerts | +```bash +# What runs under the hood +trufflehog git https://github.com/org/repo --results=verified --json +``` --- ## Cloud Security -### Prowler Scan +### Prowler -[GitHub](https://github.com/prowler-cloud/prowler) · Docker: `ghcr.io/shipsecai/prowler` + AWS, Azure, and GCP security posture auditing against CIS benchmarks and best practices. + [GitHub](https://github.com/prowler-cloud/prowler) · Docker: `ghcr.io/shipsecai/prowler` -Cloud (AWS, Azure, GCP) security posture management. Best practices auditing. - | Input | Type | Description | -|-------|------|-------------| -| `credentials` | Object | AWS credentials | -| `checks` | Array | Specific checks to run | +|---|---|---| +| `credentials` | Object | AWS credentials (use the **AWS Credentials** core component) | +| `checks` | Array | Specific check IDs to run (leave empty to run all) | | Parameter | Type | Description | -|-----------|------|-------------| -| `severity` | Array | Filter by severity | -| `services` | Array | AWS services to audit | +|---|---|---| +| `severity` | Array | Filter findings by severity | +| `services` | Array | Limit the audit to specific AWS services (e.g. `s3`, `iam`, `ec2`) | + +--- ### Supabase Scanner -[GitHub](https://ghcr.io/shipsecai/supabase-scanner) · Docker: `ghcr.io/shipsecai/supabase-scanner` + Checks Supabase projects for common misconfigurations and exposed data. + Docker: `ghcr.io/shipsecai/supabase-scanner` -Scans Supabase instances for misconfigurations. - | Input | Type | Description | -|-------|------|-------------| -| `projectUrl` | String | Supabase project URL | -| `apiKey` | Secret | Supabase API key | +|---|---|---| +| `projectUrl` | String | Your Supabase project URL | +| `apiKey` | Secret | Supabase API key (use **Secret Loader**) | --- @@ -262,33 +282,46 @@ Scans Supabase instances for misconfigurations. ### Notify -[GitHub](https://github.com/projectdiscovery/notify) · Docker: `ghcr.io/shipsecai/notify` + Send alerts to Slack, Discord, Telegram, or email at any point in your workflow. + [GitHub](https://github.com/projectdiscovery/notify) · Docker: `ghcr.io/shipsecai/notify` -Sends alerts to Slack, Discord, Telegram, or email. - | Input | Type | Description | -|-------|------|-------------| -| `message` | String | Alert message | -| `provider` | Select | slack, discord, telegram, email | -| `config` | Secret | Provider configuration | +|---|---|---| +| `message` | String | The alert message to send | +| `provider` | Select | `slack` · `discord` · `telegram` · `email` | +| `config` | Secret | Provider configuration (webhook URL, API key, etc.) | ---- + + Put **Notify** at the end of every workflow so your team gets a summary the moment a run completes — whether it found something or not. + -## Example Workflow +--- -A complete attack surface discovery pipeline: +## Full Example Pipeline +Here's a complete **attack surface discovery** workflow using security components end-to-end: ``` -Manual Trigger (domains input) - ↓ -Subfinder (subdomain discovery) - ↓ -DNSx (DNS resolution) - ↓ -httpx (HTTP probing) - ↓ -Nuclei (vulnerability scanning) - ↓ -Notify (notify team) +Manual Trigger ← operator provides target domains + ↓ + Subfinder ← discovers all subdomains + ↓ + DNSx ← resolves subdomains to IPs + ↓ + HTTPx ← finds live web services + ↓ + Nuclei ← scans for vulnerabilities + ↓ + Notify ← alerts the team with findings ``` + +--- + + + + ← Previous + + + Next → + + \ No newline at end of file diff --git a/docs/development/isolated-volumes.mdx b/docs/development/isolated-volumes.mdx index 40000599..c94e25af 100644 --- a/docs/development/isolated-volumes.mdx +++ b/docs/development/isolated-volumes.mdx @@ -1,83 +1,94 @@ --- title: "Isolated Volumes" -description: "Docker-in-Docker file mounting solution for multi-tenant security" +description: "How ShipSec Studio keeps every workflow run's files completely separate — even when running Docker inside Docker." --- -## Problem Summary +## The Problem This Solves -In a Docker-in-Docker (DinD) setup where the worker container runs inside Docker and creates containers: +ShipSec Studio runs security tools inside Docker containers. But there's a catch — the worker itself also runs inside Docker. This "Docker-in-Docker" (DinD) setup breaks the normal way of sharing files with containers, and creates a serious security risk in multi-tenant environments. -1. **Volume Path Mismatch**: Volume mount paths are relative to the Docker daemon's filesystem, not the worker container -2. **Security Risk**: Shared volumes in multi-tenant SaaS allow cross-tenant data access -3. **Limited stdin Approach**: Using stdin for data transfer doesn't support file-based tools +Three problems needed solving: ---- +- **Volume path mismatch** — file paths are relative to the host Docker daemon, not the worker container. Normal file mounts simply don't work. +- **Cross-tenant data leakage** — shared volumes mean one tenant's files could be visible to another's. +- **No stdin fallback** — many security tools only accept file-based input, so piping data in via stdin isn't an option. -## Solution: Isolated Named Volumes +--- -Use unique Docker named volumes created per `tenantId + runId + timestamp`: +## The Solution: One Volume Per Run +Every workflow execution gets its own **unique, isolated Docker named volume** — tied to the tenant ID, run ID, and a timestamp: ``` -tenant-${tenantId}-run-${runId}-${timestamp} +tenant-{tenantId}-run-{runId}-{timestamp} + +# Example +tenant-acme-run-wf-abc123-1732150000 ``` -### Architecture +It's created before the tool runs, mounted into the container, read after execution, then **immediately destroyed**. No shared state. No leftovers. +--- + +## How It Works — Step by Step ``` -┌─────────────────────────────────────────────────────────┐ -│ Docker Host │ -│ │ -│ ┌─────────────────────────────────────────────┐ │ -│ │ Worker Container (DinD) │ │ -│ │ │ │ -│ │ 1. Creates volume via Docker CLI │ │ -│ │ docker volume create tenant-A-run-1-... │ │ -│ │ │ │ -│ │ 2. Populates files using temp container │ │ -│ │ docker run -v vol:/data alpine sh -c .. │ │ -│ │ │ │ -│ │ 3. Runs actual tool with volume mounted │ │ -│ │ docker run -v vol:/inputs dnsx ... │ │ -│ │ │ │ -│ │ 4. Reads output files using temp container │ │ -│ │ docker run -v vol:/data alpine cat ... │ │ -│ │ │ │ -│ │ 5. Cleans up volume │ │ -│ │ docker volume rm tenant-A-run-1-... │ │ -│ └─────────────────────────────────────────────┘ │ -│ │ -│ ┌──────────────────────────────────────────┐ │ -│ │ Docker Volumes (on Docker Host) │ │ -│ │ │ │ -│ │ • tenant-A-run-123-1732090000 │ │ -│ │ • tenant-B-run-456-1732090001 │ │ -│ │ • tenant-A-run-789-1732090002 │ │ -│ │ │ │ -│ │ Each volume isolated per tenant + run │ │ -│ └──────────────────────────────────────────┘ │ -└─────────────────────────────────────────────────────────┘ +Docker Host +└── Worker Container (DinD) + │ + ├── 1. CREATE volume + │ docker volume create tenant-A-run-1-... + │ + ├── 2. POPULATE files via temp Alpine container + │ docker run -v vol:/data alpine sh -c "echo ... > /data/domains.txt" + │ + ├── 3. RUN the security tool with volume mounted + │ docker run -v vol:/inputs dnsx ... + │ + ├── 4. READ output files via temp Alpine container + │ docker run -v vol:/data alpine cat /data/results.json + │ + └── 5. DESTROY the volume + docker volume rm tenant-A-run-1-... + +Docker Host — Named Volumes (completely separate per tenant + run) +├── tenant-A-run-123-1732090000 +├── tenant-B-run-456-1732090001 +└── tenant-A-run-789-1732090002 ``` --- -## Security Benefits +## Security Improvements | Aspect | Old Approach | Isolated Volumes | -|--------|-------------|------------------| -| **Tenant Isolation** | ❌ Shared volume or stdin | ✅ Unique volume per tenant+run | -| **Path Traversal** | ⚠️ Possible with file mounts | ✅ Validated filenames | -| **Data Leakage** | ❌ Files persist in shared space | ✅ Immediate cleanup | -| **Audit Trail** | ❌ None | ✅ Volume labels track tenant/run | -| **DinD Compatible** | ❌ File mounts don't work | ✅ Named volumes work perfectly | +|---|---|---| +| **Tenant Isolation** | ❌ Shared volume or stdin | ✅ Unique volume per tenant + run | +| **Path Traversal** | ⚠️ Possible with file mounts | ✅ Filenames validated — no `..` or `/` | +| **Data Leakage** | ❌ Files persist in shared space | ✅ Destroyed immediately after use | +| **Audit Trail** | ❌ None | ✅ Every volume labeled `studio.managed=true` | +| **DinD Compatible** | ❌ File mounts break | ✅ Named volumes work perfectly | --- -## Implementation +## Approach Comparison + +Not sure which approach fits your use case? Here's the full breakdown: -### Before: File Mounting (Broken in DinD) +| Feature | File Mounts | stdin | Isolated Volumes | +|---|---|---|---| +| **DinD Compatible** | ❌ | ✅ | ✅ | +| **File-based tools** | ✅ | ❌ | ✅ | +| **Config files** | ✅ | ❌ | ✅ | +| **Output files** | ❌ Hard | ❌ | ✅ | +| **Binary / large files** | ✅ | ⚠️ Memory limits | ✅ | +| **Tenant isolation** | ❌ | ⚠️ Process-level only | ✅ Volume-level | +--- + +## Code Examples + +### Before — File Mounts (Broken in DinD) ```typescript -// ❌ WRONG - Breaks in DinD, no tenant isolation +// ❌ WRONG — breaks in DinD, no tenant isolation const hostInputDir = await mkdtemp(path.join(tmpdir(), 'dnsx-input-')); await writeFile(path.join(hostInputDir, 'file.txt'), data); @@ -88,10 +99,9 @@ const runnerConfig: DockerRunnerConfig = { }; ``` -### After: Isolated Volumes (DinD Compatible) - +### After — Isolated Volumes (DinD Compatible) ```typescript -// ✅ CORRECT - DinD compatible, tenant isolated +// ✅ CORRECT — DinD compatible, fully tenant isolated const tenantId = context.tenantId ?? 'default-tenant'; const volume = new IsolatedContainerVolume(tenantId, context.runId); @@ -102,7 +112,7 @@ try { }); const runnerConfig: DockerRunnerConfig = { - volumes: [volume.getVolumeConfig('/inputs', true)] + volumes: [volume.getVolumeConfig('/inputs', true)] // read-only }; await runComponentWithRunner(runnerConfig, ...); @@ -110,50 +120,15 @@ try { const outputs = await volume.readFiles(['results.json']); } finally { - await volume.cleanup(); + await volume.cleanup(); // Always runs — even if an error is thrown } ``` --- -## Comparison: All Approaches - -| Feature | File Mounts | stdin Approach | Isolated Volumes | -|---------|-------------|----------------|------------------| -| **DinD Compatible** | ❌ No | ✅ Yes | ✅ Yes | -| **File-based tools** | ✅ Yes | ❌ No | ✅ Yes | -| **Config files** | ✅ Yes | ❌ No | ✅ Yes | -| **Output files** | ❌ Hard to read | ❌ No | ✅ Yes | -| **Binary files** | ✅ Yes | ❌ No | ✅ Yes | -| **Large files** | ✅ Yes | ⚠️ Memory limits | ✅ Yes | -| **Tenant isolation** | ❌ No | ⚠️ Process-level | ✅ Volume-level | - ---- - -## Usage Examples - -### Simple Input Files - -```typescript -const volume = new IsolatedContainerVolume(tenantId, runId); - -try { - await volume.initialize({ - 'targets.txt': targets.join('\n') - }); - - const config = { - volumes: [volume.getVolumeConfig('/inputs', true)] - }; - - await runTool(config); -} finally { - await volume.cleanup(); -} -``` - ### Input + Output Files +When the tool needs to both read inputs *and* write outputs to the same volume: ```typescript const volume = new IsolatedContainerVolume(tenantId, runId); @@ -164,7 +139,7 @@ try { const config = { command: ['--input', '/data/config.yaml', '--output', '/data/results.json'], - volumes: [volume.getVolumeConfig('/data', false)] // Read-write + volumes: [volume.getVolumeConfig('/data', false)] // read-write }; await runTool(config); @@ -177,20 +152,23 @@ try { } ``` -### Multiple Volumes +--- + +### Separate Input and Output Volumes +For maximum isolation, use two volumes — one read-only for inputs, one write-only for outputs: ```typescript -const inputVol = new IsolatedContainerVolume(tenantId, `${runId}-in`); +const inputVol = new IsolatedContainerVolume(tenantId, `${runId}-in`); const outputVol = new IsolatedContainerVolume(tenantId, `${runId}-out`); try { await inputVol.initialize({ 'data.csv': csvData }); - await outputVol.initialize({}); // Empty volume for outputs + await outputVol.initialize({}); // empty — just here to receive output const config = { volumes: [ - inputVol.getVolumeConfig('/inputs', true), - outputVol.getVolumeConfig('/outputs', false) + inputVol.getVolumeConfig('/inputs', true), // read-only + outputVol.getVolumeConfig('/outputs', false) // read-write ] }; @@ -199,176 +177,135 @@ try { const results = await outputVol.readFiles(['output.json']); } finally { - await Promise.all([ - inputVol.cleanup(), - outputVol.cleanup() - ]); + await Promise.all([inputVol.cleanup(), outputVol.cleanup()]); } ``` --- -## Volume Lifecycle - -1. **Create**: `docker volume create tenant-A-run-123-...` -2. **Populate**: Use temporary Alpine container to write files -3. **Mount**: Container uses the volume via `-v volumeName:/path` -4. **Read**: Use temporary Alpine container to read files -5. **Cleanup**: `docker volume rm tenant-A-run-123-...` +## Security Details -### Automatic Cleanup - -Volumes are always cleaned up via `finally` blocks: +### Path Validation +Filenames passed to `initialize()` are automatically validated. Anything that looks like a path traversal attempt is rejected before it ever reaches Docker. ```typescript -try { - await volume.initialize(...); - await runTool(...); -} finally { - await volume.cleanup(); // Always runs, even on error -} -``` - -### Orphan Cleanup - -For volumes that weren't cleaned up (e.g., worker crash): - -```bash -# List studio-managed volumes -docker volume ls --filter "label=studio.managed=true" - -# Remove old volumes -docker volume prune --filter "label=studio.managed=true" -``` - ---- - -## Security Requirements - -### Tenant Isolation +// ✅ These are fine +await volume.initialize({ + 'file.txt': data, + 'subdir/file.txt': data +}); -Every execution gets a unique volume: -``` -tenant-{tenantId}-run-{runId}-{timestamp} +// ❌ These are blocked automatically +await volume.initialize({ + '../file.txt': data, // path traversal — rejected + '/etc/passwd': data // absolute path — rejected +}); ``` -Example: `tenant-acme-run-wf-abc123-1732150000` - -### Read-Only Mounts - +### Read-Only vs Read-Write Mounts ```typescript -// Input files should be read-only -volume.getVolumeConfig('/inputs', true) // ✅ read-only - -// Only make writable if tool needs to write -volume.getVolumeConfig('/outputs', false) // ⚠️ read-write +volume.getVolumeConfig('/inputs', true) // ✅ read-only — use for input files +volume.getVolumeConfig('/outputs', false) // ⚠️ read-write — only when tool must write output ``` -### Nonroot Container Support - -Volumes automatically support containers running as nonroot users (e.g., distroless images with uid 65532). + + Default to read-only. Only grant write access when the tool explicitly needs to produce output files. + -After files are written to the volume, permissions are set to `777` to allow any container user to read/write: +### Nonroot Container Support -```typescript -// This happens automatically in volume.initialize() -// chmod -R 777 /data -``` +Some tools (like distroless images) run as a nonroot user (`uid 65532`). Files written to volumes via Alpine (which runs as root) would be unreadable by these containers. -**Why this is needed:** -- Files are written to volumes using Alpine containers (running as root) -- Distroless nonroot images run as uid 65532 -- Without permission changes, nonroot containers can't write output files +`volume.initialize()` automatically runs `chmod -R 777` on the volume after writing files. This is safe because: +- Each volume is scoped to a single tenant + run +- Volumes are destroyed immediately after execution +- No cross-tenant access is ever possible -**This is safe because:** -- Each volume is isolated per tenant + run -- Volumes are cleaned up after execution -- No cross-tenant access is possible +--- -### Path Validation +## Cleanup -Filenames are automatically validated: +### Automatic Cleanup (Normal Flow) +Volumes are always destroyed via `finally` blocks — even if the tool crashes or throws an error: ```typescript -// ✅ OK -await volume.initialize({ - 'file.txt': data, - 'subdir/file.txt': data // Subdirs OK -}); - -// ❌ Rejected (security) -await volume.initialize({ - '../file.txt': data, // Path traversal blocked - '/etc/passwd': data // Absolute paths blocked -}); +try { + await volume.initialize(...); + await runTool(...); +} finally { + await volume.cleanup(); // guaranteed to run +} ``` ---- +### Orphan Cleanup (After a Worker Crash) -## Security Guarantees +If the worker process itself crashes mid-execution, volumes may be left behind. Clean them up with: +```bash +# List all volumes managed by ShipSec Studio +docker volume ls --filter "label=studio.managed=true" -| Security Feature | How It Works | -|-----------------|--------------| -| **Tenant Isolation** | Volume name includes tenant ID | -| **No Collisions** | Timestamp prevents conflicts | -| **Path Safety** | Filenames validated (no `..` or `/`) | -| **Automatic Cleanup** | Finally blocks guarantee removal | -| **Audit Trail** | Volumes labeled `studio.managed=true` | -| **DinD Compatible** | Named volumes work in nested Docker | +# Remove all orphaned managed volumes +docker volume prune --filter "label=studio.managed=true" +``` --- ## Performance -### Volume Creation Overhead - -- **Creation**: ~50-100ms per volume -- **File writes**: ~10-50ms per file (depends on size) -- **Cleanup**: ~50-100ms per volume +Volume creation adds a small, predictable overhead: -**Total overhead**: ~100-250ms per execution +| Operation | Typical Duration | +|---|---| +| Volume creation | ~50–100ms | +| File writes | ~10–50ms per file | +| Volume cleanup | ~50–100ms | +| **Total overhead** | **~100–250ms** | -This is acceptable for security tools that typically run for seconds/minutes. +This is negligible for security tools that typically run for seconds or minutes. -### Optimization Tips - -1. **Batch file writes**: Write all files in one `initialize()` call -2. **Reuse volumes**: For sequential operations in same run, reuse the volume -3. **Lazy cleanup**: Clean up volumes in background job if latency-sensitive +**Tips to minimize overhead:** +- Write all files in a single `initialize()` call — don't call it multiple times +- Reuse the same volume for sequential steps within the same run +- Run `cleanup()` in the background if you're latency-sensitive and don't need to await it --- ## When to Use Each Approach -### Use Isolated Volumes When: -- ✅ Running in DinD environment -- ✅ Need multi-tenant isolation -- ✅ Tool requires file-based config -- ✅ Tool writes output files -- ✅ Handling binary/large files - -### Use stdin/stdout When: -- ✅ Tool supports stdin input -- ✅ Single-tenant or dev environment -- ✅ Small text-only inputs -- ✅ Don't need output files - -### Use File Mounts When: -- ✅ NOT running in DinD (direct Docker) -- ✅ Development/testing only -- ✅ Quick prototyping + + + Running in DinD · Multi-tenant environment · Tool needs config files · Tool writes output files · Binary or large file handling + + + Tool supports stdin input · Single-tenant or local dev · Small text-only data · No output files needed + + + NOT running in DinD · Local development only · Quick prototyping · Not going to production + + --- ## Migration Checklist -To migrate a component to use isolated volumes: +Migrating an existing component to use isolated volumes: - [ ] Import `IsolatedContainerVolume` -- [ ] Get `tenantId` from context (use fallback for now) -- [ ] Create volume instance: `new IsolatedContainerVolume(tenantId, runId)` -- [ ] Replace file writes with `volume.initialize({ files })` -- [ ] Replace volume mount with `volume.getVolumeConfig()` -- [ ] Add `finally` block with `volume.cleanup()` -- [ ] If tool writes outputs, use `volume.readFiles()` to retrieve them -- [ ] Test in DinD environment +- [ ] Get `tenantId` from `context.tenantId` (with a fallback for local dev) +- [ ] Create an instance: `new IsolatedContainerVolume(tenantId, runId)` +- [ ] Replace manual file writes with `volume.initialize({ files })` +- [ ] Replace volume mount config with `volume.getVolumeConfig()` +- [ ] Wrap execution in a `try/finally` block with `volume.cleanup()` in `finally` +- [ ] If the tool writes outputs, use `volume.readFiles([...])` to retrieve them +- [ ] Test end-to-end in a DinD environment before shipping + +--- + + + + ← Previous + + + Next → + + \ No newline at end of file diff --git a/docs/docs.json b/docs/docs.json index 046e4014..3eb3fcf5 100644 --- a/docs/docs.json +++ b/docs/docs.json @@ -9,6 +9,20 @@ }, "favicon": "/favicon.ico", "navigation": { + "global": { + "anchors": [ + { + "anchor": "Website", + "icon": "globe", + "href": "https://studio.shipsec.ai" + }, + { + "anchor": "GitHub", + "icon": "github", + "href": "https://github.com/shipsecai/studio" + } + ] + }, "tabs": [ { "tab": "Documentation", @@ -16,7 +30,7 @@ { "group": "Getting Started", "pages": [ - "index", + "Introduction", "quickstart", "installation", "command-reference" @@ -49,21 +63,7 @@ } ] } - ], - "global": { - "anchors": [ - { - "anchor": "Website", - "href": "https://studio.shipsec.ai", - "icon": "globe" - }, - { - "anchor": "GitHub", - "href": "https://github.com/shipsecai/studio", - "icon": "github" - } - ] - } + ] }, "logo": { "light": "/logo/logo-dark.svg", @@ -88,5 +88,8 @@ "github": "https://github.com/shipsecai", "linkedin": "https://linkedin.com/company/shipsecai" } + }, + "seo": { + "indexing": "all" } } diff --git a/docs/images/svgviewer-output(1).svg b/docs/images/svgviewer-output(1).svg new file mode 100644 index 00000000..665f9fbe --- /dev/null +++ b/docs/images/svgviewer-output(1).svg @@ -0,0 +1,215 @@ + + + + + + + + + + + Frontend / Visual Workspace + + + + Visual Builder + ReactFlow • Canvas + + + + Terminal Viewer + xterm.js • Real-time + + + + Timeline Replay + Zustand • State + + + + Config Panel + Radix UI • Forms + + + + + REST API + UNIFIED SSE + + + + Service Core (NestJS) + + + + Workflows + DSL Engine + + + + Secrets + AES-256-GCM + + + + Storage + + + + Trace Events + + + + Auth (Clerk) + + + + + Webhooks + + + + Schedules + + + + AI Agents + + + + Human Input + + + + Integrations + + + + + TEMPORAL CLIENT gRPC + + + + Temporal Orchestration + Durability • Managed Retries • State Persistence + + + + ACTIVITY DISPATCH + + + + Execution Worker (Node/Bun) + + COMPONENT REGISTRY + + + Security Tools + Nuclei • Subfinder + + + + AI Agents + + + + Human-in-Loop + + + + Core Utilities + + + + + SERVICE ADAPTERS + SECRETS • STORAGE • ARTIFACTS • TRACE • TERMINAL • LOGS + + + + + + + Infrastructure Layer + + + + PostgreSQL + Primary Data + + + + MinIO + Object Store + + + + Redis + Pub/Sub & Cache + + + + Redpanda + Event Stream + + + + Loki + Log Aggregation + + + \ No newline at end of file diff --git a/docs/index.mdx b/docs/index.mdx index 4ab6cabe..eeafcd36 100644 --- a/docs/index.mdx +++ b/docs/index.mdx @@ -1,78 +1,107 @@ --- title: "Introduction" -description: "Open source, no-code security automation platform for security teams" +description: "The open-source, no-code security automation platform for modern security teams." --- -**ShipSec Studio** is the open source, no-code security automation platform for security teams. Design reusable security workflows, see every run unfold live, and keep humans in the loop with opinionated guardrails. +## Welcome to ShipSec Studio - - Workflow Builder +**ShipSec Studio** is the open-source, no-code security automation platform. Design reusable security workflows, watch every run unfold live, and maintain full control with self-hosted, encrypted, isolated execution. + + + ShipSec Workflow Builder -## Why ShipSec Studio? +--- + +## Why ShipSec? -- **No code required** – Build powerful security workflows by connecting components visually. No scripting, no glue code, no maintenance headaches. -- **Built for security teams** – Pre-built components for subdomain discovery, vulnerability scanning, secret detection, and more. Focus on security, not infrastructure. -- **Workflows that don't fail silently** – Automatic retries, resumable executions, and real-time log streaming, giving visibility into every step. Know exactly what's happening and why. -- **Self-hosted and open source** – Your data stays on your infrastructure. Run it on a laptop or scale to production with the same codebase. -- **Secure by default** – Encrypted secrets, isolated container execution, and team-based access controls built in from day one. + + + Connect security tools visually — no scripting or glue code required. + + + Every step unfolds in real-time with streaming logs and execution status. + + + Self-hosted, AES-256 encrypted secrets, and isolated container execution per run. + + + +--- ## Product Tour - Visual drag-and-drop editor for building security workflows - - Workflow Builder - + Visual drag-and-drop editor for building security pipelines. + Workflow Builder - Watch workflow execution unfold in real-time with streaming logs - - Execution Timeline - + Watch runs unfold in real-time with streaming logs and a step-by-step timeline. + Execution Timeline - - Secure storage for API keys and credentials - - Secret Manager - + + Enterprise-grade encrypted storage for API keys and credentials. + Secret Manager - - Integrate with external services and APIs - - Connections Catalog - + + Native connectors for the most popular security tools and APIs. + Connections Catalog +--- + ## Get Started - Get up and running in 5 minutes + Build your first security workflow in under 5 minutes. - - Self-host on your own infrastructure + + Self-host on your own infrastructure with Docker. - Understand the system design + Understand how ShipSec uses Temporal.io for durable execution. - Explore available components + Explore the library of pre-built security automation components. +--- + ## Technology Stack | Layer | Technologies | -|-------|-------------| -| **Frontend** | React 19, TypeScript, Vite, Tailwind CSS, Radix UI, ReactFlow, xterm.js | -| **Backend** | NestJS, TypeScript, Bun runtime, PostgreSQL, Drizzle ORM, Clerk Auth | -| **Worker** | Node.js, TypeScript, Temporal.io, Docker containers | -| **Infrastructure** | PostgreSQL, Temporal, MinIO, Redis, Loki, Redpanda (Kafka) | +|---|---| +| **Frontend** | React 19, TypeScript, Tailwind CSS, ReactFlow, xterm.js | +| **Backend** | NestJS, Bun, PostgreSQL, Drizzle ORM, Clerk Auth | +| **Orchestration** | Temporal.io, Docker containers | +| **Infrastructure** | Redis, MinIO, Loki, Redpanda (Kafka) | -## Need Help? +--- + +## Join the Community + +ShipSec Studio is open-source and built in public. Come build with us. + + + + Star the repo, open issues, or submit a PR. + + + Get help, share workflows, and hang out with the community. + + + Enterprise inquiries or direct help — email us anytime. + + + +--- -- **Support** – [support@shipsec.ai](mailto:support@shipsec.ai) -- **GitHub** – [github.com/shipsecai/studio](https://github.com/shipsecai/studio) + + + Next → + + \ No newline at end of file diff --git a/docs/installation.mdx b/docs/installation.mdx index 7a2d45f6..791dc456 100644 --- a/docs/installation.mdx +++ b/docs/installation.mdx @@ -1,306 +1,343 @@ --- title: "Installation" -description: "Self-host ShipSec Studio on your infrastructure" +description: "Self-host ShipSec Studio on your own infrastructure — from zero to running in minutes." --- -## Prerequisites +## What You'll Need -- Docker Desktop with ≥8 GB RAM -- [Bun](https://bun.sh) runtime -- [just](https://github.com/casey/just) command runner -- Open ports: `5433`, `7233`, `8081`, `9000`, `9001`, `3100` +Before installing, make sure you have these tools ready on your machine: ---- - -## Installation Steps +- **Docker Desktop** — with at least **8 GB RAM** allocated _(Settings → Resources → Memory)_ +- **[Bun](https://bun.sh)** — a fast JavaScript runtime (replaces Node.js) +- **[just](https://github.com/casey/just)** — a simple command runner (like `make`, but friendlier) +- **Open ports** — the following ports must be free on your machine: -### 1. Clone & initialize + `5433` · `7233` · `8081` · `9000` · `9001` · `3100` + + On macOS, install Bun and just instantly with Homebrew: ```bash -git clone https://github.com/shipsecai/studio.git -cd studio -just init + brew install bun just ``` + - - **macOS users:** If you see native module errors, run: - ```bash - export SDKROOT=$(xcrun --show-sdk-path) - bun install - ``` - +--- -The `just init` command automatically: -- Installs project dependencies with Bun -- Creates environment files from examples (`backend/.env`, `worker/.env`, `frontend/.env`) +## Installation Steps -### 2. Configure environment variables + + + This downloads the ShipSec Studio source code to your machine and runs a one-time setup script. +```bash + git clone https://github.com/shipsecai/studio.git + cd studio + just init +``` -Edit the generated `.env` files as needed: + `just init` does three things automatically: + - Installs all project dependencies using Bun + - Creates your environment config files (`backend/.env`, `worker/.env`, `frontend/.env`) + + **macOS users:** If you see native module errors during `just init`, run this first, then retry: ```bash -# Edit environment files -nano backend/.env -nano worker/.env -nano frontend/.env + export SDKROOT=$(xcrun --show-sdk-path) + bun install ``` + + -### 3. Start development environment + + Open and edit the generated config files to match your setup. For most users, **the defaults work out of the box** — you only need to change these if you have custom infrastructure. +```bash + nano backend/.env + nano worker/.env + nano frontend/.env +``` + + + One command starts everything: ```bash -just dev + just dev ``` -The `just dev` command automatically: -- Checks that environment files exist -- Starts Docker infrastructure (Postgres, Temporal, MinIO, Loki) -- Waits for services to be ready -- Runs database migrations -- Starts backend, worker, and frontend with hot-reload + Behind the scenes, this automatically: + - Verifies your environment files exist + - Launches all Docker services (Postgres, Temporal, MinIO, Loki) + - Waits for each service to become healthy + - Runs database migrations + - Starts the backend, worker, and frontend — all with **hot-reload** (changes reflect instantly) + + + + Once everything is running, open your browser and go to: + + **[http://localhost](http://localhost)** + + Sign in or create a new account to get started. + + --- ## Service Endpoints -In development mode, all application services are accessible through nginx on port 80: +> Think of these as the "addresses" for each piece of the system. In development, almost everything runs through a single gateway (nginx) on port 80 — so you rarely need to memorize these. -| Service | URL | Notes | -|---------|-----|-------| -| Frontend | http://localhost/ | Workflow builder (via nginx) | -| Backend API | http://localhost/api/ | REST API (via nginx) | -| Analytics | http://localhost/analytics/ | OpenSearch Dashboards (via nginx) | -| Temporal UI | http://localhost:8081 | Workflow management | -| PostgreSQL | http://localhost:5433 | Database | -| MinIO | http://localhost:9000 | Object storage | -| MinIO Console | http://localhost:9001 | Credentials: `minioadmin/minioadmin` | -| Loki | http://localhost:3100 | Logs | +| Service | URL | What it does | +|---|---|---| +| Frontend | [http://localhost/](http://localhost/) | The main workflow builder UI | +| Backend API | [http://localhost/api/](http://localhost/api/) | The REST API powering the app | +| Analytics | [http://localhost/analytics/](http://localhost/analytics/) | OpenSearch Dashboards (visualizations) | +| Temporal UI | [http://localhost:8081](http://localhost:8081) | Inspect running/completed workflows | +| PostgreSQL | `localhost:5433` | The main database | +| MinIO | [http://localhost:9000](http://localhost:9000) | File/object storage | +| MinIO Console | [http://localhost:9001](http://localhost:9001) | Storage UI — login: `minioadmin / minioadmin` | +| Loki | `localhost:3100` | Log aggregation | - Individual service ports (5173, 3211, 5601) are available for debugging but should not be used in normal development. All traffic flows through nginx on port 80. + Direct ports like `5173` (frontend) and `3211` (backend) exist for deep debugging only. For all normal development, use **port 80 via nginx** — it handles routing automatically. --- ## Analytics Stack (Optional) -ShipSec Studio includes an optional analytics stack powered by OpenSearch for indexing and visualizing workflow execution data. - -### Starting the Analytics Stack +ShipSec Studio ships with a built-in analytics layer powered by **OpenSearch** — useful for indexing, searching, and visualizing your workflow execution data over time. -The analytics services are included in the infrastructure docker-compose file: + + This is completely optional. If you skip this section, everything else still works perfectly. The Analytics Sink component will simply log a warning and skip indexing if OpenSearch isn't configured. + +### Starting the Analytics Services ```bash -# Start infrastructure including OpenSearch just infra up ``` -This will start: -- **OpenSearch** on port `9200` - Search and analytics engine -- **OpenSearch Dashboards** on port `5601` - Visualization and query UI +This starts: +- **OpenSearch** on port `9200` — the search and analytics engine +- **OpenSearch Dashboards** on port `5601` — the visual query and dashboard UI ### Configuring Analytics -Add these environment variables to your backend and worker `.env` files: - +Add these variables to your `backend/.env` and `worker/.env` files: ```bash -# Backend (.env) +# backend/.env and worker/.env OPENSEARCH_URL=http://localhost:9200 OPENSEARCH_USERNAME=admin OPENSEARCH_PASSWORD=admin OPENSEARCH_DASHBOARDS_URL=http://localhost/analytics - -# Frontend (.env) +``` +```bash +# frontend/.env VITE_OPENSEARCH_DASHBOARDS_URL=http://localhost/analytics ``` ### Setting Up the Index Template -After starting OpenSearch, create the security findings index template: - +Run this once after OpenSearch is up. It creates the data structure for storing security findings: ```bash cd backend bun run setup:opensearch ``` -This creates the `security-findings-*` index template with proper mappings for workflow execution data. - -### Using Analytics +This creates a `security-findings-*` index template with the correct field mappings. -1. **Analytics Sink Component**: Add the "Analytics Sink" component to your workflows to index output data -2. **Dashboards Link**: Access OpenSearch Dashboards from the Studio sidebar -3. **Query API**: Use the `/api/analytics/query` endpoint to query indexed data programmatically +### Using Analytics in Workflows -### Analytics Service Endpoints +Once configured, you can use analytics in three ways: -| Service | URL | Notes | -|---------|-----|-------| -| OpenSearch | http://localhost:9200 | Search engine API | -| OpenSearch Dashboards | http://localhost/analytics/ | Visualization UI (via nginx) | - - - The analytics stack is optional. If OpenSearch is not configured, the Analytics Sink component will gracefully skip indexing and log a warning. - +- **Analytics Sink Component** — drag it into any workflow to automatically index output data +- **OpenSearch Dashboards** — accessible from the Studio sidebar for building visualizations +- **Query API** — call `/api/analytics/query` to query indexed data from your own scripts or tools --- ## Production Deployment -For production, use the Docker-based deployment: - +When you're ready to move beyond development and run ShipSec Studio in a live environment: ```bash -# Start production environment with cached images +# Start using pre-built (cached) images — fastest option just prod -# Or build and start +# Rebuild images from scratch, then start just prod build ``` -Production endpoints (all via nginx on port 80): -- Frontend: http://localhost/ -- Backend API: http://localhost/api/ -- Analytics: http://localhost/analytics/ -- Temporal UI: http://localhost:8081 +Production exposes the same URLs as development — all routed through nginx on port 80: + +| Service | URL | +|---|---| +| Frontend | http://localhost/ | +| Backend API | http://localhost/api/ | +| Analytics | http://localhost/analytics/ | +| Temporal UI | http://localhost:8081 | --- -## Common Commands +## Command Reference -### Development +A quick cheat sheet for the most common operations. Don't worry about memorizing all of these — `just dev` and `just dev stop` cover 90% of daily use. +### Development ```bash -just dev # Start development environment -just dev stop # Stop everything -just dev logs # View application logs -just dev status # Check service status -just dev clean # Stop and remove all data +just dev # Start the full development environment +just dev stop # Stop everything cleanly +just dev logs # Stream live application logs +just dev status # Check which services are running +just dev clean # Stop everything and wipe all local data ``` ### Production - ```bash -just prod # Start with cached images -just prod build # Rebuild and start -just prod stop # Stop production -just prod logs # View production logs -just prod status # Check production status -just prod clean # Remove all data +just prod # Start production (uses cached images) +just prod build # Rebuild images and start +just prod stop # Stop production environment +just prod logs # Stream production logs +just prod status # Check production service health +just prod clean # Wipe all production data ``` -### Infrastructure - +### Infrastructure Only ```bash -just infra up # Start infrastructure only -just infra down # Stop infrastructure -just infra logs # View infrastructure logs -just infra clean # Remove infrastructure data +just infra up # Start only the Docker services (DB, storage, etc.) +just infra down # Stop infrastructure services +just infra logs # View infrastructure-level logs +just infra clean # Remove all infrastructure data volumes ``` ### Utilities - ```bash -just status # Show status of all services -just db-reset # Reset database (drops all data) -just build # Build images only +just status # Overview of all running services +just db-reset # ⚠️ Wipe and reset the database completely +just build # Build Docker images without starting them ``` --- ## Troubleshooting -### macOS: Native module build errors - + + + This happens when Bun can't find the macOS SDK. Fix it by pointing Bun to the right SDK path: ```bash -export SDKROOT=$(xcrun --show-sdk-path) -bun install + export SDKROOT=$(xcrun --show-sdk-path) + bun install ``` -### Services not starting + Then re-run `just init`. + + + Check what's running and restart cleanly: ```bash -# Check status -just dev status + just dev status -# Restart everything -just dev stop -just dev + # If something looks stuck: + just dev stop + just dev ``` + -### Database connection failed - + + Run migrations manually and inspect the database logs: ```bash -# Run migrations manually -bun run migrate - -# Check database logs -just infra logs + bun run migrate + just infra logs ``` + -### PM2 processes failing - + + Usually caused by missing `.env` files. Check they exist, then restart: ```bash -# Check environment files -ls -la */.env + ls -la */.env # Verify all three .env files exist -# Restart everything -just dev stop -just dev + just dev stop + just dev -# Check logs -just dev logs + just dev logs # Watch for errors on startup ``` + -### Frontend not loading - + + Check if the dev server is running, and start it manually if needed: ```bash -# Check if dev server is running -just dev status + just dev status -# Start manually if needed -bun --cwd frontend dev + # Start frontend manually if it's missing: + bun --cwd frontend dev ``` + -### Temporal namespace missing - + + Run the Temporal setup container to create the required namespace: ```bash -docker compose -p shipsec --profile setup up -d + docker compose -p shipsec --profile setup up -d ``` + -### Clean rebuild - + + The nuclear option. This wipes everything and starts from scratch: ```bash -just dev stop -just infra clean -just infra up -bun run migrate -just dev + just dev stop + just infra clean + just infra up + bun run migrate + just dev ``` + + `just infra clean` deletes all local data including your database. Only use this if you're okay losing your local data. + + + + --- -## Manual Setup (Optional) +## Manual Setup (Advanced) -If you need more control over the setup process: +If you need granular control and want to start services individually rather than using `just dev`: ### Infrastructure only - ```bash -just infra up -just status -just infra logs +just infra up # Start Postgres, Temporal, MinIO, Loki +just status # Confirm everything is healthy +just infra logs # Watch infrastructure logs ``` -### Applications only - +### Applications only (after infrastructure is running) ```bash -pm2 start pm2.config.cjs -pm2 status -pm2 logs +pm2 start pm2.config.cjs # Start backend, worker, and frontend via PM2 +pm2 status # Check process health +pm2 logs # Stream logs from all processes ``` --- ## Getting Help -- Check [GitHub Issues](https://github.com/shipsecai/studio/issues) for similar problems -- Review logs with `just dev logs` for error details -- Email support at [support@shipsec.ai](mailto:support@shipsec.ai) +Stuck? Here's where to go: + + + + Search existing issues or open a new one with your error logs. + + + Run `just dev logs` — most errors explain themselves clearly in the output. + + + Reach the team directly at support@shipsec.ai + + + +--- + + + + ← Previous + + + Next → + + \ No newline at end of file diff --git a/docs/quickstart.mdx b/docs/quickstart.mdx index ee978a10..ac865923 100644 --- a/docs/quickstart.mdx +++ b/docs/quickstart.mdx @@ -1,101 +1,129 @@ --- title: "Quickstart" -description: "Get ShipSec Studio running in 5 minutes" +description: "Get ShipSec Studio running in 5 minutes." --- ## Prerequisites -- Docker Desktop (≥8 GB RAM) -- [Bun](https://bun.sh) runtime -- [just](https://github.com/casey/just) command runner +Before you begin, ensure you have the following installed: + +- **Docker Desktop** — with at least 8 GB RAM allocated +- **[Bun](https://bun.sh)** — JavaScript runtime & package manager +- **[just](https://github.com/casey/just)** — command runner + + + On macOS, you can install both Bun and just via Homebrew: `brew install bun just` + + +--- ## Quick Setup - ```bash +```bash git clone https://github.com/shipsecai/studio.git cd studio just init - ``` - - This automatically installs dependencies and creates environment files. +``` + + `just init` installs all dependencies and scaffolds your environment files automatically — no manual `.env` configuration required. - - - ```bash + + +```bash just dev - ``` - - This starts all infrastructure (Postgres, Temporal, MinIO, Loki), runs migrations, and launches the full stack with hot-reload. +``` + + This single command spins up the **full stack** — including Postgres, Temporal, MinIO, and Loki — runs database migrations, and launches all services with hot-reload enabled. - + - Navigate to [http://localhost](http://localhost) and sign in or create an account. + Navigate to [http://localhost](http://localhost) in your browser. Sign in with an existing account or create a new one to get started. +--- + ## Access Points -All services are accessible through nginx on port 80: +All services are proxied through **nginx on port 80**. Use these URLs during development: | Service | URL | -|---------|-----| -| Frontend | http://localhost/ | -| Backend API | http://localhost/api/ | -| Temporal UI | http://localhost:8081 | +|---|---| +| Frontend | [http://localhost/](http://localhost/) | +| Backend API | [http://localhost/api/](http://localhost/api/) | +| Temporal UI | [http://localhost:8081](http://localhost:8081) | - Individual service ports are available for debugging (frontend: 5173, backend: 3211) but should not be used in normal development. + Direct service ports (`5173` for the frontend, `3211` for the backend) are available for low-level debugging but should not be used for normal development workflows. +--- + ## Your First Workflow +Build and run a **Subdomain Discovery** workflow in under a minute. + - Click **New Workflow** in the dashboard. Name it "Subdomain Discovery". + From the dashboard, click **New Workflow** and name it `Subdomain Discovery`. - - - Drag **Manual Trigger** to the canvas - - Drag **Subfinder** component to the canvas - - Connect the trigger output to the Subfinder input + + - Drag a **Manual Trigger** onto the canvas + - Drag a **Subfinder** component onto the canvas + - Connect the trigger's output node to the Subfinder's input node - - Click **Save**, then **Run**. Watch the execution unfold in real-time with streaming logs and terminal output. + + Click **Save**, then **Run**. Execution unfolds in real-time — watch streaming logs and live terminal output as your workflow runs. -## Common Commands +--- +## Common Commands ```bash -# View logs +# Stream live logs just dev logs -# Check status +# Check service health just dev status -# Stop everything +# Stop all services just dev stop -# Restart +# Full restart just dev stop && just dev ``` +--- + ## Next Steps - Detailed setup and configuration options + Detailed setup, environment configuration, and advanced options. - All available just commands + Complete list of available `just` commands and their usage. - Explore available components + Explore the built-in component library for building workflows. - Understand the system design + Understand how the system is designed and how services interact. + +--- + + + + ← Previous + + + Next → + + \ No newline at end of file