Can I create a workflow similar to a DAG (Directed Acyclic Graph) using flows?

[jinkookchoi]

Hello. I would like to implement a machine learning workflow using bullmq. I want to design steps for data loading, preprocessing, and modeling using flows. I'm wondering if it's possible to have a structure where a specific step can look at the same children.

For example:
Data Loading -> Data Splitting -> Data Processing 1 -> Data Merging -> Other processes...
..................................................-> Data Processing 2 ->

The issue is, after the Data Splitting step, I'm unsure if there is a way to branch into Data Processing 1 and Data Processing 2 steps separately. If you have any insights on how to achieve this, I would appreciate your help.

[manast]

If what you need to do is split data and then merge it, this tutorial shows how to do it for video transcoding https://blog.taskforce.sh/splitting-heavy-jobs-using-bullmq-flows/
Not sure if this is what you need though, if not, please let me know in what it differs and I will try to give you a better answer.

[jinkookchoi]

I would like to run it with the following DAG.

I would like to implement a condition that branches from Step A to Step B and Step C respectively.
If I define Flows in Step B and Step C and specify children as Step A, Step A is executed twice. However, I want to branch from one step instance. Is it possible to have a workflow that continues from Step A to Step B and Step C? Please refer to the picture below.

[manast]

I am afraid that is not currently possible as that would require two parents (B and C) to share the same child (child A). Maybe a solution could be to move step B between C and D instead.

[tolgap]

@manast are there any plans to support a flow like this? Maybe the FlowProducer class can have modes in which it creates parent/child structures. The default would be the current behavior, and maybe there can be a flowType parameter that can be set to DAG to have job depend on each other (not only their execution moment, but also their data). That would also mean that only the first job contains a data key. The other job in the DAG would get their data passed in automatically based on what the previous job provided.

Is there any want/need in BullMQ for DAG-style jobs? Would it be too much of a maintenance burden?

[ianwoodfill]

Has anyone come up with a creative solution to this?