legacy monitor for obsforge#212
Conversation
|
@givelberg Has it been running for a while? If yes, can you post some examples to look at? |
| system: ocean | ||
| model: gdas | ||
|
|
||
| data_root: /lfs/h2/emc/da/noscrub/emc.da/obsForge/COMROOT/realtime |
There was a problem hiding this comment.
For testing, lines 4-5 are ok, but we need to point at the operational paths before we implement this. As you know, obsForge is on hold for now, i.e. GFSv17 will generate the ocean obs.
| # do not create __pucache__ dir: | ||
| export PYTHONDONTWRITEBYTECODE=1 | ||
|
|
||
| OBSFORGE_HOME="/lfs/h2/emc/obsproc/noscrub/edward.givelberg/newmonitor/obsForge" |
There was a problem hiding this comment.
Reminder that we need to update this path before code hand-off to NCO.
| OBSFORGE_HOME="/lfs/h2/emc/obsproc/noscrub/edward.givelberg/newmonitor/obsForge" | ||
| ASCII_MONITOR_HOME="$OBSFORGE_HOME/utils/ascii-monitor" | ||
|
|
||
| RUNDIR="/lfs/h2/emc/obsproc/noscrub/edward.givelberg/newmonitor" |
There was a problem hiding this comment.
@CoryMartin-NOAA , since GFSv17 will generate the ocean obs, shall we add the monitoring reports generation to GFSv17 (i.e. to the prep obs jobs) or to the legacy obsproc (we are preparing obsproc v1.3.0)?
Another dumb question - will GSFv17 write the marine obs IODA files under $COMROOT/gfs ?
There was a problem hiding this comment.
@ilianagenkova what do you prefer for the monitoring? and yes it will be writing them in the GFS COM directory.
There was a problem hiding this comment.
Happy with ocean obs to be written in GFS COM.
If NCO doesn't mind that obsproc would be reading GFS output, I'd rather have the monitor in obsproc, so that we can write the reports along with the myriad of other monitoring reports. Will check with Steven E.
There was a problem hiding this comment.
Works for me, it already depends on GFS output for prepBUFR right? So it shouldn't be an added dependency (I hope!)
|
@HyundeokChoi-NOAA and @apchoiCMD , could one of you review this? Thanks |
|
@givelberg , are you rewriting this? If so, close/delete this PR please. |
|
@ilianagenkova Happy to look at it again- |
|
@givelberg None of the insitu data appears to be listed in the config.yaml. 
|
|
@HyundeokChoi-NOAA , thanks for checking the counts! The in-situ data is already monitored as an /atmos data set , b/c the same data (but processed differently) is used by the atmos DA and marine DA. |
|
@ilianagenkova , IMO, in-situ obs are needed to monitor since bufr2ioda converter rejects obs during the processing and they generate |
Sure, but that could be done in the GFS/GDAS monitoring Ed Safford is working on, or for marine obs - what Andy and Ed G. are working on. This monitoring "watches" the obs after they are read by GSI ( or soon by SOCA). The monitoring that obsproc/obsforme does is one step earlier - we watch the obs as they arrive from the data providers, and alarm them if any data drops in volume. Perhaps it's good to self-check, i.e. obsproc/obsforge can report how many files are generated, are any blank/too small/too big, but such functionality does not exist in the code today. I think Hyundeok has scripts that do that for his run, but it's not part of obsForge. @CoryMartin-NOAA , shall we add something to obsForge (now that we have more time to work on it) ? |
|
I agree @ilianagenkova that this is monitoring for incoming obs, not for obs that go into GFS/GDAS. That will be done by the DA monitoring tools. I think we should think about scoping out requirements for observation ingest monitoring first before diving in and adding any tools to ObsForge. |
@ilianagenkova @CoryMartin-NOAA Thanks for clarifying it and now more understandable than before. |
5 participants
A monitoring code to count the number of obs in obs spaces.
It writes reports in ascii format and uses past reports to generate current report.
The files are written in the legacy format used by NCO.