add a new stream property 'output_done_marker'#255
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces a new stream XML attribute, output_done_marker, to allow MPAS-Model to create a ${filename}.done marker file after a stream write completes, enabling external workflows to reliably detect when an output file write has finished.
Changes:
- Parse
output_done_marker="yes"from stream XML and propagate it into the stream manager as a new stream property. - Add a new stream property/field (
output_done_marker) and trigger.donemarker creation after successful stream writes. - Introduce a
write_done_marker(...)helper to create the.donefile with a timestamp.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
src/framework/xml_stream_parser.c |
Parses the new output_done_marker attribute from XML and sets a corresponding stream property. |
src/framework/mpas_stream_manager.F |
Adds the property handling, writes .done markers after successful writes, and implements write_done_marker. |
src/framework/mpas_stream_manager_types.inc |
Adds a new stream property constant for the done-marker feature. |
src/framework/mpas_stream_list_types.inc |
Adds output_done_marker to the per-stream type with a default of disabled. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
@guoqing-noaa I view Copilot's feedback as suggestions. Now that the tests are passing - one had to be manually re-run because it failed early in its workflow - I'm OK approving as is. |
dustinswales
left a comment
There was a problem hiding this comment.
@guoqing-noaa Looks good to me.
Do we want to extend one of the tests to create ".done" files for one of the output streams?
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Sure, do you want me add it in this PR? That will leave us one failed CI test. Thanks! |
That would be excellent. Thanks! |
|
@dustinswales added |
|
@dustinswales @clark-evans Thanks a lot for a quick review and merge!! |
This PR adds a new stream property
output_done_marker.By default,
output_done_markertakes a value ofno. When set toyes(i.e.,output_done_marker="yes"), MPAS-Model writes a corresponding${filename_template}.donefile (for example,history.2024-08-16_00.00.00.nc.done) after the associated forecast-hour output file has been completely written.This greatly simplifies the rrfs-workflow (v2) logic to trigger new tasks only after a forecast file has been fully written. It helps prevent hard-to-debug workflow failures due to incomplete forecast files (sometimes only a few bytes away from a completion).
Resolve #237
Mandatory Questions
Reviews