7.14.0.1 GEMPAK causes MRMS data not to plot
sebenste opened this issue · comments
A buddy of mine sent me this:
It appears my decoders for the MRMS data are not creating output files. So I checked permissions and stuff and that was fine, then I realized I probably need to update the grid tables. So I was adding in a few symbolic links to match what I have on server1 (running manually updated tables) and I tried the decoder again. Still no output. So then I noticed that on server 1 when the decoder runs I see 3 tables listed. So for the composite reflectivity I have wmocenter.tbl; g2varsnoaa1.tbl; and g2vcrdwmo255.tbl. On server 2, where I did the upgrade last night, the g2vcrdwmo255.tbl was not showing. I also noticed there was a difference in the files sizes of several files and these were the non-symbolic link files. So I don't know what has happened but for the MRMS data its not decoding.
Also, on the updated nawips, the file g2vars.tbl is 10 bytes larger.
Any suggestions?
Here's screen snaps of the exact same script running on server one, and server two, server two has the new version of NAWIPS on it. Look at the last paragraph and you can see what is missing, the vertical coordinate table.
Models are plotting correctly.
I'm going to ask what might be a silly question, what's the difference between the tables found here (https://github.com/Unidata/gempak/tree/main/gempak/tables/grid) vs. LDM (https://github.com/Unidata/LDM/tree/master/gempak/tables)?
I didn't dig too deep, but one difference I saw was in wmocenter.tbl. Gempak currently has this:
161 National Oceanic and Atmospheric Administration NOAA
!161 NSSL NSSL
while LDM's has this:
!161 National Oceanic and Atmospheric Administration NOAA
161 NSSL NSSL
What would the implications be for differences between these two sets of tables? Additionally, could/should these tables be standardized across the two packages?
I spent over 4 hours going over with Kevin Polston every table file, and ones that did not match, I checked for anything that was related or could be related to MRMS. Again, we tried to use DCGRIB to try to decode the data as you can see from the screen captures. On the 7.14 version of GEMPAK, it works; 7.14.1 does not.
Copying over missing files and missing soft links did not
fix the issue. I am unsure of how to proceed. There is a section of soft links that were there previously, but not in the new version. Restoring those did not help, either. He really wants to have MRMS plotted, and the other version made MRMS work just fine.
Thoughts?
Still looking...I can't figure this one out. It's over my head. We can't even get any grids out of the script above. Any ideas are greatly appreciated!
@sebenste I'll try to recreate this issue tomorrow. To help me out, what's the source of the MRMS data you're trying to use (FNEXRAD, NGRID or someplace else)? The pqact statement you're using to save the data might help too.
FNEXRAD. Here is an example from one product. They are commented out until they work again.
FNEXRAD (Radar)/([A-Z]{4,6})/(MRMS_(.*)_([0-9]{8})-([0-9]{2})[0-9]{4}.grib2)
FILE -close -overwrite
/home/ldm/data/images/sat/MRMS/\1/\2/\4/\5/\3
/data/ldm/pub/native/radar/MRMS/Radar/CONUS/MRMS_CREF_1HR_MAX_00.50_20210518-170000.grib2
FNEXRAD (Radar)/([A-Z]{4,6})/(MRMS_(.*)_([0-9]{8})-([0-9]{2})[0-9]{4}.grib2)
EXEC /home/ldm/hail_decode.csh
#FNEXRAD (Radar)/([A-Z]{4,6})/(MRMS_(SHI_00.50)_([0-9]{8})-([0-9]{2})[0-9]{4}.grib2)
EXEC /home/ldm/mrms_shi.csh
#FNEXRAD (Radar)/([A-Z]{4,6})/(MRMS_(VII_00.50)_([0-9]{8})-([0-9]{2})[0-9]{4}.grib2)
EXEC /home/ldm/mrms_vii_regions2.csh
#FNEXRAD (Radar)/([A-Z]{4,6})/(MRMS_(VII_00.50)_([0-9]{8})-([0-9]{2})[0-9]{4}.grib2)
EXEC /home/ldm/mrms_vii_regions3.csh
#FNEXRAD (Radar)/([A-Z]{4,6})/(MRMS_(VIL_00.50)_([0-9]{8})-([0-9]{2})[0-9]{4}.grib2)
EXEC /home/ldm/mrms_vil_regions.csh
#FNEXRAD (Radar)/([A-Z]{4,6})/(MRMS_(VIL_Density_00.50)_([0-9]{8})-([0-9]{2})[0-9]{4}.grib2)
EXEC /home/ldm/mrms_vil_density.csh
#FNEXRAD (Radar)/([A-Z]{4,6})/(MRMS_(VIL_Max_120min_00.50)_([0-9]{8})-([0-9]{2})[0-9]{4}.grib2)
EXEC /home/ldm/mrms_vil_max120.csh
#FNEXRAD (Radar)/([A-Z]{4,6})/(MRMS_(Reflectivity_-20C_00.50)_([0-9]{8})-([0-9]{2})[0-9]{4}.grib2)
EXEC /home/ldm/mrms_refm20c_all.csh
#FNEXRAD (Radar)/([A-Z]{4,6})/(MRMS_(VIL_Max_1440min_00.50)_([0-9]{8})-([0-9]{2})[0-9]{4}.grib2)
EXEC /home/ldm/mrms_vil_max1440.csh
#FNEXRAD (Radar)/([A-Z]{4,6})/(MRMS_(MESH_00.50)_([0-9]{8})-([0-9]{2})[0-9]{4}.grib2)
EXEC /home/ldm/mrms_mesh_all.csh
#FNEXRAD (Radar)/([A-Z]{4,6})/(MRMS_(MESH_Max_30min_00.50)_([0-9]{8})-([0-9]{2})[0-9]{4}.grib2)
EXEC /home/ldm/mrms_mesh_4panel_all.csh
#FNEXRAD (Radar)/([A-Z]{4,6})/(MRMS_(MESH_Max_30min_00.50)_([0-9]{8})-([0-9]{2})[0-9]{4}.grib2)
EXEC /home/ldm/mrms_mesh30_regions.csh
#FNEXRAD (Radar)/([A-Z]{4,6})/(MRMS_(MESH_Max_60min_00.50)_([0-9]{8})-([0-9]{2})[0-9]{4}.grib2)
EXEC /home/ldm/mrms_mesh60_regions.csh
#FNEXRAD (Radar)/([A-Z]{4,6})/(MRMS_(MESH_Max_120min_00.50)_([0-9]{8})-([0-9]{2})[0-9]{4}.grib2)
EXEC /home/ldm/mrms_mesh120.csh
#FNEXRAD (Radar)/([A-Z]{4,6})/(MRMS_(POSH_00.50)_([0-9]{8})-([0-9]{2})[0-9]{4}.grib2)
EXEC /home/ldm/mrms_posh_regions.csh
#FNEXRAD (Radar)/([A-Z]{4,6})/(MRMS_(ReflectivityAtLowestAltitude_00.50)_([0-9]{8})-([0-9]{2})[0-9]{4}.grib2)
EXEC /home/ldm/mrms_rala_regions2.csh
#FNEXRAD (Radar)/([A-Z]{4,6})/(MRMS_(ReflectivityAtLowestAltitude_00.50)_([0-9]{8})-([0-9]{2})[0-9]{4}.grib2)
EXEC /home/ldm/mrms_rala_regions3.csh
#FNEXRAD (Radar)/([A-Z]{4,6})/(MRMS_(Reflectivity_-20C_00.50)_([0-9]{8})-([0-9]{2})[0-9]{4}.grib2)
EXEC /home/ldm/mrms_refm20c_regions1.csh
#FNEXRAD (Radar)/([A-Z]{4,6})/(MRMS_(Reflectivity_-20C_00.50)_([0-9]{8})-([0-9]{2})[0-9]{4}.grib2)
EXEC /home/ldm/mrms_refm20c_regions3.csh
@sebenste I was able to get it to save a .gem file that looked good in GDINFO; I did not try to visualize. This was on a fresh 7.14.0.1 install, and your original post said something about custom grib tables so ymmv, but here's what got it working for me...
wget https://raw.githubusercontent.com/Unidata/LDM/master/gempak/tables/g2varsnssl.tbl
cp g2varsnssl.tbl /home/gempak/NAWIPS/gempak/tables/grid/g2varsnoaa1.tbl
ln -s g2vcrdwmo2.tbl g2vcrdwmo255.tbl
The MRMS products are defined in g2varsnssl.tbl and I didn't see them defined elsewhere in this release, so that appears to be the missing content. Note the name change to g2varsnoaa1.tbl above.
He tried that, and it still didn't work.
@sebenste Can you send me a zip of the ~/NAWIPS/gempak/tables/grid directory (preserving symlinks) from the problematic server?
Thanks @mzuranski for the help here. If I am reading this diff correctly, the only present difference between GEMPAK 7.5.1 and 7.14.0.1 in the gempak/tables/grid/ directory is a new symlink from g2varsncep1.tbl to g2varsncep201.tbl ... ooo, but this commit came in after 7.5.1 3ca2365
@akrherz I don't think MRMS support was ever added to Unidata Gempak, so this might be more of an "enhancement" that we're just way late to the game on.
@mzuranski I think what is being said above is that this was working in 7.14.0 and is now failing with 7.14.0.1, maybe my merged PR #82 now fixes this? I have not allocated time to test this.
Mike,
We've been doing some experimenting trying to get this to work, including doing an ldmadmin updategempaktables, copying those new tables into the GEMPAK tables directory, and then linking them. So I don't know if that would help you or not.
@mzuranski I think what is being said above is that this was working in 7.14.0 and is now failing with 7.14.0.1, maybe my merged PR #82 now fixes this? I have not allocated time to test this.
Daryl,
What can my buddy do to test this?
@sebenste this tarball file should contain the most updated code for testing.
He will be able to install it after 3 PM today, and then he will let you know.
He also has a question for both of you that is unrelated, and I don't know where to put this:
"Can you ask him how to run the nex2gini program and how to save the output? If I can figure that out then I can probably create the n0b mosaic. Which would be very sweet!"
Kevin asked:
"The question I have is....in the old setup which worked, I had those tables as symbolic links (the g2vcrdwmo255, 0, 1, 3 and 6 tables). In the new version I believe most of those were not symbolic links. Would I need to adjust that to the way it was....or go with the newer update? That's my biggest question. And I'll be surprised if it doesn't work. Daryl gets his stuff to work, which is good."
Also, MRMS worked partially under 7.5.1. We got them to work fully after we incorporated the UNIDATA table update which added the rest of the products.
I am not finding a g2vcrdwmo255 file? Looking at the other g2vcrdwmo files, indeed, there are some differences, but I am unsure yet which is the right answer.
I'm not sure if the 255 file was a softlink, or an additional UNIDATA one. I don't remember. If you do an ldmadmin updategempaktables, you might get that file.
g2vcrdwmo255 was very likely a soft link. I know this is for LDM but it helped me find my above solution: https://www.unidata.ucar.edu/support/help/MailArchives/ldm/msg07599.html
The git history here seems to have no recollection of that file?!?
It was softlinked before.
OK, he says the patch didn't work. Still can't find the missing grid. I wonder if it cannot find the WMO GRIB2 Vertical Coordinate Table, as you see on his working machine. I don't know what that is.
One thing for sure, this is absolutely a table issue.
Kevin reverted his server back to 7.14.0...and almost everything started working fine again. Shrug...
I'm following up with him to figure this out.
Sounds good! Thanks, Daryl.