[Fix-18166] Restore file parameter transfer feature

[loftiest]

Was this PR generated or assisted by AI?

NO

Purpose of the pull request

Fix #18166 – Restore the file parameter transfer feature

Brief change log

• Add TaskFilesTransferUtils to transfer file paramter.
• Integrate upstream file download in PhysicalTaskExecutor#initializeTaskContext.
• Add success hook in PhysicalTaskExecutor#trackTaskExecutorState to trigger output file upload only on SUCCEEDED.
• Update CuringParamsServiceImpl to map input file parameter values as keys, so they can be matched against the workflow varPool output file parameters and resolved from prepareParamsMap.
• Add unit tests for TaskFilesTransferUtils and PhysicalTaskExecutor.

Verify this pull request

This change added tests and can be verified as follows:

• Added unit tests covering download/upload with FILE parameters, including edge cases.
• Added integration test verifying that file transfer and varPool can coexist and work correctly together.
• Manually tested with Shell tasks on a local cluster, confirming downstream task receives the file.

Pull Request Notice

Pull Request Notice

If your pull request contains incompatible change, you should also add it to docs/docs/en/guide/upgrade/incompatible.md

[boring-cyborg]

Thanks for opening this pull request! Please check out our contributing guidelines. (https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/en/contribute/join/pull-request.md)

[sonarqubecloud]

Quality Gate failed

Failed conditions
0.0% Coverage on New Code (required ≥ 60%)

See analysis details on SonarQube Cloud

[ruanwenjun]

We do not recommend using localparams to store file paths, as this can easily lead to issues such as file renaming and may be affected by varpool. Instead, we should add a new switch at the task level to indicate whether the current task’s output should be uploaded, and include a parameter to specify whether to use the output from a particular upstream task.

[loftiest]

We do not recommend using localparams to store file paths, as this can easily lead to issues such as file renaming and may be affected by varpool. Instead, we should add a new switch at the task level to indicate whether the current task’s output should be uploaded, and include a parameter to specify whether to use the output from a particular upstream task.

Thanks for the suggestion! I agree that storing file paths in localParams and coupling with varPool can be fragile, especially with renaming. A dedicated task-level switch would indeed be a more robust long-term solution.

However, the goal of this PR is to restore the file parameter transfer feature, rather than redesign it. The existing FILE parameter mechanism has been used and documented for several versions. Changing it now would be a breaking change and should go through a separate improvement proposal.

I think your idea is great as a new feature/improvement. Could we track that in a new issue? I'd be happy to help design and implement it.

Thanks!

[SbloodyS]

Since this function has not been implemented at present, I suggest using wenjun's suggestion to rewrite it. @loftiest

[loftiest]

Hi @SbloodyS , Thanks for the follow-up. I understand the concern about localParams and varPool coupling, but I respectfully hold a different view on the approach.

The file parameter transfer feature has been documented in the official documentation for multiple versions, including the current 3.4.1: https://dolphinscheduler.apache.org/zh-cn/docs/3.4.1/guide/parameter/file-parameter. Users who follow the official guide have built their workflows based on this documented feature. When they upgrade to 3.4.1, their workflows silently break — and there is no warning, no deprecation notice, and no migration guide anywhere. Deleting a documented feature without any indication is incorrect and harmful to users who cannot upgrade safely.

I agree a redesign would be valuable, but whether we restore or redesign, users need clear notice and a transition period. To reconcile both needs, I propose a phased approach:

Phase 1 (this PR or a follow-up): Restore file transfer using the existing localParams with FILE type (consistent with current docs), but decouple it from varPool now — store remote paths in TaskExecutionContext instead of varPool, so the upload/download logic is already clean and independent.

Phase 2 (separate PR/version): Introduce a new explicit file transfer configuration. Mark the old FILE-based localParams approach as @deprecated with a prominent documentation notice, and remove it only in a subsequent version, giving users at least one full version cycle to migrate.

The key principle: don't silently break documented features. Users need sufficient notice and a proper transition path.

What do you think? Thanks!