Skip to content

Conversation

@jswxstw
Copy link
Member

@jswxstw jswxstw commented Nov 3, 2025

Fixes #14990

Motivation

#14940 (comment)

Modifications

  • Add archive location if artifact is needed in data source.
  • Remove unused outputs.artifacts in example data-transformations.

Verification

Disable archiveLogs and use the workflow below for testing and validation.

# See doc docs/data-sourcing-and-transformation.md
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: data-transformations-
  annotations:
    workflows.argoproj.io/description: |
      This workflow demonstrates using a data template to list in an S3 bucket
      and then process those log files.
    workflows.argoproj.io/version: ">= 3.1.0"
spec:
  entrypoint: data-transformations
  templates:
    - name: data-transformations
      steps:
        - - name: list-log-files
            template: list-log-files
        - - name: process-logs
            template: process-logs
            withParam: "{{steps.list-log-files.outputs.result}}"
            arguments:
              artifacts:
                - name: file
                  s3:
                    key: "{{item}}"

              parameters:
                - name: file-name
                  value: "{{item}}"

    - name: list-log-files
      data:
        source:
          artifactPaths:
            name: test-bucket
            s3:
              bucket: my-bucket

        transformation:
          - expression: "filter(data, {# endsWith \"main.log\"})"

    - name: process-logs
      inputs:
        parameters:
          - name: file-name
        artifacts:
          - name: file
            path: /file
      container:
        image:  argoproj/argosay:v2
        command: [ sh, -c ]
        args:
          - |
            echo {{inputs.parameters.file-name}}
            head /file
# argo get data-transformations-2lxxp
Name:                data-transformations-2lxxp
Namespace:           argo
ServiceAccount:      unset (will run with the default ServiceAccount)
Status:              Succeeded
Conditions:          
 PodRunning          False
 Completed           True
Created:             Mon Nov 03 19:39:04 +0800 (31 minutes ago)
Started:             Mon Nov 03 19:39:04 +0800 (31 minutes ago)
Finished:            Mon Nov 03 19:39:26 +0800 (30 minutes ago)
Duration:            22 seconds
Progress:            3/3
ResourcesDuration:   0s*(1 cpu),4s*(100Mi memory)

STEP                                                                                              TEMPLATE              PODNAME                                DURATION  MESSAGE
 ✔ data-transformations-2lxxp                                                                     data-transformations                                                     
 ├───✔ list-log-files                                                                             list-log-files        data-transformations-2lxxp-1577854924  6s          
 └─┬─✔ process-logs(0:data-transformations-s787k/data-transformations-s787k-1684829053/main.log)  process-logs          data-transformations-2lxxp-1296497094  8s          
   └─✔ process-logs(1:fantastic-dragon/fantastic-dragon/main.log)                                 process-logs          data-transformations-2lxxp-3200542704  6s

Documentation

@jswxstw
Copy link
Member Author

jswxstw commented Nov 4, 2025

/retest

@Joibel Joibel self-assigned this Nov 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Running the sample workflow for data transformation results in a workflow stuck in "Running" status

2 participants