[WIP] Add prefect manager support

[ptigas]

Summary

Include a summary of major changes in bullet points:

• Added Prefect workflow manager integration for jobflow
• Implemented PrefectManager class with flow conversion and deployment capabilities
• Added support for converting jobflow Jobs and Flows to Prefect workflows
• Created example demonstrating Prefect integration patterns
• Added reference resolution system for handling job dependencies in Prefect
• Implemented error handling and parallel execution support

Additional dependencies introduced (if any)

• prefect: Core Prefect workflow orchestration library (≥3.0.0) - Required for the new Prefect manager functionality to convert and execute
  jobflow workflows as Prefect flows. This enables users to leverage Prefect's distributed execution, monitoring, and deployment capabilities.
• pytest: Testing framework - Added as a main dependency (was previously only in test dependencies) to support the new Prefect manager tests
  and examples.

Note: Updated minimum Python version requirement from 3.10 to 3.11 to align with Prefect 3.x requirements.

TODO (if any)

• Documentation updates needed for the new Prefect integration
• Consider adding more advanced Prefect features like retries and timeouts configuration
• May need performance optimization for large workflows with many dependencies

Checklist

Work-in-progress pull requests are encouraged, but please put [WIP] in the pull request title.

Before a pull request can be merged, the following items must be checked:

• Code is in the https://www.python.org/dev/peps/pep-0008/. The easiest way to handle this is to run the following in the correct sequence on
  your local machine. Start with running https://black.readthedocs.io/en/stable/index.html on your new code. This will automatically reformat
  your code to PEP8 conventions and removes most issues. Then run https://pycodestyle.readthedocs.io/en/latest/, followed by
  http://flake8.pycqa.org/en/latest/.
• Docstrings have been added in the https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_numpy.html. Run
  http://www.pydocstyle.org/en/2.1.1/index.html on your code.
• Type annotations are highly encouraged. Run http://mypy-lang.org/ to type check your code.
• Tests have been added for any new functionality or bug fixes.
• All linting and tests pass.

Note that the CI system will run all the above checks. But it will be much more efficient if you already fix most errors prior to submitting
the PR. It is highly recommended that you use the pre-commit hook provided in the repository. Simply pip install pre-commit and then
pre-commit install and a check will be run prior to allowing commits.

[utf]

Wow!

Thanks for this contribution @ptigas, this is a major addition and something I know many users will be interested in. I'm familiar with Prefect but haven't used it before so I'd appreciate feedback from others. Potentially @Andrew-S-Rosen might have comments.

At first glance the code looks very clean and thank you for adding rigorous tests. This has reminded me I'll need to update the CI to drop Python 3.10 support and add 3.13.

One major comment up-front - one feature of jobflow is support for dynamic flows through the Response object. Do you think it will be possible to add support for this with Prefect? To be clear, this isn't a roadblock to getting this merged but I'd like to understand more of the scope of what you want to achieve. For example, a number of the atomate2 workflows use dynamic flows (e.g. the elastic constant workflow).

If it's helpful, I'm happy to arrange a call to discuss.

[Andrew-S-Rosen]

Wow, this is looking to be very nicely done! I was really impressed to see how you handled the .submit() vs. serial logic. That is slick.

I would absolutely be interested in reviewing this further, but I probably won't have a chance to take a look until the middle of next week. One comment in the meantime while you continue working on the PR.

I think this PR saves me a lot of work I was planning on doing, so I may owe you... 😅

[Andrew-S-Rosen]

It would be helpful if this could take a str | TaskRunner as the input. For instance, if I want to use the DaskTaskRunner, which is the main way to interact with Slurm, then it would not be possible as-is.

[ptigas]

Hi @Andrew-S-Rosen, I've updated to take TaskRunner as an argument. I've never used DaskTaskRunner before, so I might need some help testing that part.

[ptigas]

@utf @Andrew-S-Rosen thank you for the supportive comments :)

One major comment up-front - one feature of jobflow is support for dynamic flows through the Response object. Do you think it will be possible to add support for this with Prefect? To be clear, this isn't a roadblock to getting this merged, but I'd like to understand more of the scope of what you want to achieve. For example, a number of the atomate2 workflows use dynamic flows (e.g. the elastic constant workflow).

I've updated the code to support dynamic flows. I've also added an example and tested it in a prefect cluster, and it seems to be working alright.

Also, I'd be happy to arrange a call to discuss the workflow I'm after in a bit more detail (interested in running atomate2 workflows in prefect, currently testing this and I'll report back or in a call about results).

[Andrew-S-Rosen]

Please remove this file from the PR. Thanks!

[Andrew-S-Rosen]

We should avoid adding these to the core dependency stack. Having them in a dedicated prefect optional dependency section like you have done is okay.

If you want them to be tested in CI, you can include them in strict.

[Andrew-S-Rosen]

What is the needed to have prefect-docker here? That should not be needed by all users, such as those using Prefect Cloud. I think we can just have prefect here.

[Andrew-S-Rosen]

I think this is better suited for documentation rather than as a module within the codebase.

[Andrew-S-Rosen]

Hi @ptigas: Thank you for your patience. Unfortunately, I remain extremely busy to review this in the depth it deserves. I will have a software engineer working with my group in September who is going to focus on Prefect + Jobflow, so we can almost certainly test things out in detail then. Until then, I have left a few minor comments in the meantime.

I am happy to chat if you think it would be helpful. Just let me know.