Skip to content

Propagate PipelineRequestConfigParameters through pull-mode (NATS) tasks #1275

@mihow

Description

@mihow

Surfaced during PR #1272 occurrence-tracking E2E testing against ADC PR #77.

Problem

PipelineRequestConfigParameters (the config dict in PipelineRequest) only reaches sync HTTP /process workers. The NATS pull-mode task schema does not carry it.

ami/ml/schemas.py::PipelineProcessingTask:

class PipelineProcessingTask(pydantic.BaseModel):
    id: str
    image_id: str
    image_url: str
    reply_subject: str | None = None
    # TODO: Do we need these?
    # detections: list[DetectionRequest] | None = None
    # config: PipelineRequestConfigParameters | dict | None = None

config is commented out. ami/ml/orchestration/jobs.py::queue_all_images_for_job publishes only the per-image fields. Pull-mode workers cannot see include_features, include_logits, or any future per-project / per-pipeline flag.

Why now

PR #1272 needs include_features=true to populate Classification.features_2048 for tracking.

  • Sync HTTP: works via Pipeline.default_config = {"include_features": True}.
  • Pull-mode (ADC): worker must start with AMI_INCLUDE_FEATURES=true in its own env. That makes features a worker-level on/off, not per-project, and silently ignores antenna's ProjectPipelineConfig.

Scope

  • Add config: PipelineRequestConfigParameters | dict | None = None to PipelineProcessingTask.
  • Populate task.config from pipeline.get_config(project_id=...) in ami/ml/orchestration/jobs.py.
  • Update ADC worker (and any other pull-mode worker) to read config.include_features from the incoming task; keep env var as fallback.
  • Document new contract in processing_services/README.md + schema docstring.
  • Test: ProjectPipelineConfig.config={'include_features': true} reaches the ADC worker via NATS task.

Risk / migration

Additive + optional. Existing pull-mode workers that ignore config keep working; defaults stay safe.

Linked

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions