Skip to content

add a new stream property 'output_done_marker'#255

Merged
dustinswales merged 4 commits into
ufs-community:noaa/developfrom
guoqing-noaa:output_done_marker4gsl
Jun 2, 2026
Merged

add a new stream property 'output_done_marker'#255
dustinswales merged 4 commits into
ufs-community:noaa/developfrom
guoqing-noaa:output_done_marker4gsl

Conversation

@guoqing-noaa
Copy link
Copy Markdown
Collaborator

@guoqing-noaa guoqing-noaa commented Jun 2, 2026

This PR adds a new stream property output_done_marker.

By default, output_done_marker takes a value of no. When set to yes (i.e., output_done_marker="yes"), MPAS-Model writes a corresponding ${filename_template}.done file (for example, history.2024-08-16_00.00.00.nc.done) after the associated forecast-hour output file has been completely written.

This greatly simplifies the rrfs-workflow (v2) logic to trigger new tasks only after a forecast file has been fully written. It helps prevent hard-to-debug workflow failures due to incomplete forecast files (sometimes only a few bytes away from a completion).

Resolve #237

Mandatory Questions

  • Does this PR include any additions or changes to external inputs (e.g., microphysics lookup tables, static data for gravity-wave drag -- things like that)?
    • no
  • Does this PR require updating one or more baselines for the CI tests? If so, what?
    • no

Reviews

  • Is this PR currently draft, still being worked on, or ready for review?
    • ready for review, tested and worked as expected
  • For when this PR begins review, please list the developers/collaborators you'd like to prioritize for review; e.g.,:

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a new stream XML attribute, output_done_marker, to allow MPAS-Model to create a ${filename}.done marker file after a stream write completes, enabling external workflows to reliably detect when an output file write has finished.

Changes:

  • Parse output_done_marker="yes" from stream XML and propagate it into the stream manager as a new stream property.
  • Add a new stream property/field (output_done_marker) and trigger .done marker creation after successful stream writes.
  • Introduce a write_done_marker(...) helper to create the .done file with a timestamp.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
src/framework/xml_stream_parser.c Parses the new output_done_marker attribute from XML and sets a corresponding stream property.
src/framework/mpas_stream_manager.F Adds the property handling, writes .done markers after successful writes, and implements write_done_marker.
src/framework/mpas_stream_manager_types.inc Adds a new stream property constant for the done-marker feature.
src/framework/mpas_stream_list_types.inc Adds output_done_marker to the per-stream type with a default of disabled.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/framework/mpas_stream_manager.F
Comment thread src/framework/mpas_stream_manager.F
Comment thread src/framework/mpas_stream_manager.F
@clark-evans
Copy link
Copy Markdown
Collaborator

@guoqing-noaa I view Copilot's feedback as suggestions. Now that the tests are passing - one had to be manually re-run because it failed early in its workflow - I'm OK approving as is.

Copy link
Copy Markdown
Collaborator

@dustinswales dustinswales left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@guoqing-noaa Looks good to me.
Do we want to extend one of the tests to create ".done" files for one of the output streams?

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
@guoqing-noaa
Copy link
Copy Markdown
Collaborator Author

@guoqing-noaa Looks good to me. Do we want to extend one of the tests to create ".done" files for one of the output streams?

Sure, do you want me add it in this PR? That will leave us one failed CI test. Thanks!

@dustinswales
Copy link
Copy Markdown
Collaborator

@guoqing-noaa Looks good to me. Do we want to extend one of the tests to create ".done" files for one of the output streams?

Sure, do you want me add it in this PR? That will leave us one failed CI test. Thanks!

That would be excellent. Thanks!

@guoqing-noaa
Copy link
Copy Markdown
Collaborator Author

@dustinswales added output_done_marker="yes" to the hrrrv5.rap.summer CI case. Thanks!

@dustinswales dustinswales moved this from Needs review to Ready to merge in MPAS-A PRs to process Jun 2, 2026
@dustinswales dustinswales merged commit 13aef82 into ufs-community:noaa/develop Jun 2, 2026
18 checks passed
@github-project-automation github-project-automation Bot moved this from Ready to merge to Done in MPAS-A PRs to process Jun 2, 2026
@guoqing-noaa guoqing-noaa deleted the output_done_marker4gsl branch June 2, 2026 20:18
@guoqing-noaa
Copy link
Copy Markdown
Collaborator Author

@dustinswales @clark-evans Thanks a lot for a quick review and merge!!

@clark-evans clark-evans mentioned this pull request Jun 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

add a stream attribute output_done_marker to allow generating a 'done' file after stream writing is completed

4 participants