Update to NCEP NAWIPS 7.12.1

Question

Update to NCEP NAWIPS 7.12.1

akrherz opened this issue 3 years ago · comments

Throwing myself into the fire here to see what all is involved with merging NCEP's NAWIPS into Unidata's GEMPAK. So we first simply extract the release overtop the GEMPAK tree. Oh boy, what did I get myself into :)

Some notes.

There's a new versions folder, which has a bunch of legacy subversion logs from release 6.2.0 and older. culling.
There's a new site folder, which appears to just contain development resources for various shell environments, culling.
scripts/python/.gitignore is new and ignores .pyc files, culling.
scripts/decoder/ is new and contains some perl decoders, which seem unused, culling.
sample_files folder exists and appears to contain unused things, culling.
binaries gempak/utilities/pdfbld/pdfbld and gempak/utilities/hlpbld/hlpbld exist, which seems like an accident, culling.
a bunch of new jar files exist in gempak/utilities/a2conv/distVC/lib/, punting on this attm
appears to contain the same table changes found in pr #52, taking
provides a suite of its own GNUmakefile and test*.mk files, my understanding is that we can cull these.
two .old files exist in gempak/txt/gemplt, culling
new files in the bin folder I think are unused, culling.
new folders in doc/ I think are unused, culling.
environ/ folder is un-needed, culling.
extlibs.tgz was extracted verbatim and another series of differences to resolve, will do below
polarssl is a requirement for the encrypted lightning decoder, this was likely jettisoned previously, culling.
unsure what to make of gempak/include/MCHPRM.Linux.gfortran, punting
Wow, gempak/maps/ascii/ is 244MB uncompressed, is it even used? Gonna hope no, culling.
gempak/maps/hipowo.nws is new and appears to be used.
gempak/maps/programs/nam{check,plt,ssf,upd} appear to have been bundled in error, culling.
gempak/maps/programs/reducebnd* appears to have been bundled in error, culling.
gempak/maps/programs/ssfnam ditto.
gempak/source/cgemlib/cvg/output cruft.
gempak/source/contrib/awc took all new stuff there for now. Getting impatient :)
gempak/source/contrib/opc has more lightning stuff, culling.
gempak/source/device/device/ keeping for now.
gempak/source/programs/dc/dcigdr/makefile seems redundant, culling.
gempak/source/programs/dc/dump_lw_ltg/ more lightning stuff, culling.
gempak/source/programs/gd/gdp/ seems unused, culling
gempak/source/programs/gp/gpmap/utf/ seems unused, culling.
taking gempak/source/programs/util/{mpdwpc,sfcsplt,tag2vgf} for now.

Now to sort out what is going on with extlibs.

Library	Description	What to do?
DOM	libxml2 xml access stuff	culling
GAMET	unsure, but difference with file path exists, same code	culling
GNU	has an older `libiconv`, which may be needed here, oye	punting
GPC	again, different folder structure	culling
HDF5	older versions used that Unidata	culling
JasPer	appears to be a unclean tree and older version	culling
PNG	older version than Unidata	culling
bzip2	older version than Unidata	culling
glib	Woof. Lets avoid this one for now.	culling
ncepBUFR	Updated to 11.3.0	taking
netCDF	older versions than Unidata	culling
pkg-config	avoiding taking for now	culling
zlib	older versions than Unidata	culling

Will update as progress is made.

daryl herzmann · Answer 1 · Sat Feb 06 2021 05:41:16 GMT+0800 (China Standard Time)

Initial swing at having it build on RHEL8 fails with

gcc -DUNDERSCORE -DLinux -I/home/gempak/GEMPAK7/gempak/include -I/home/gempak/GEMPAK7/os/linux64/include -I/usr/X11R6/include   -DG_64BIT -g -O2 -fPIC  -c cigetq.c
cigetq.c: In function ‘cigetq’:
cigetq.c:47:31: warning: passing argument 3 of ‘cgetmq_’ makes pointer from integer without a cast [-Wint-conversion]
       cgetmq ( iproc, idebug, initmq, &jsatty, key, mbchan, smchan, &ier );
                               ^~~~~~
In file included from cigetq.c:1:
gemsys.h:45:18: note: expected ‘int *’ but argument is of type ‘int’
          int    *jsatty,
          ~~~~~~~~^~~~~~
In file included from /home/gempak/GEMPAK7/gempak/include/gemprm.h:169,
                 from gemsys.h:8,
                 from cigetq.c:1:
/home/gempak/GEMPAK7/gempak/include/uscore.h:212:17: error: too many arguments to function ‘cgetmq_’
 #define cgetmq  cgetmq_
                 ^~~~~~~
cigetq.c:47:7: note: in expansion of macro ‘cgetmq’
       cgetmq ( iproc, idebug, initmq, &jsatty, key, mbchan, smchan, &ier );
       ^~~~~~
/home/gempak/GEMPAK7/gempak/include/uscore.h:212:17: note: declared here
 #define cgetmq  cgetmq_
                 ^~~~~~~
gemsys.h:43:6: note: in expansion of macro ‘cgetmq’
 void cgetmq ( int    mproc,
      ^~~~~~
cigetq.c:84:14: warning: implicit declaration of function ‘semget’; did you mean ‘memset’? [-Wimplicit-function-declaration]
     semid  = semget ( *key, 1, 0 );
              ^~~~~~
              memset
make[3]: *** [Makefile:50: /home/gempak/GEMPAK7/os/linux64/lib/syslib.a(cigetq.o)] Error 1

Oye, overlaying the new release left some cruft around :(. This file does not exist in NCEP nawips.

daryl herzmann · Answer 2 · Sat Feb 06 2021 06:02:33 GMT+0800 (China Standard Time)

lolz, I stumble into pain associated with 16158ea, which was something Michael did for me many moons ago. I am now of the mindset that I took the wrong overlay approach and need to get a patch diff of the NCEP NAWIPS versions to apply here.

daryl herzmann · Answer 3 · Mon Feb 08 2021 22:24:28 GMT+0800 (China Standard Time)

I spammed gembud seeking NCEP NAWIPS 7.5.1 to atttempt to produce the needed diff. Fingers crossed. Otherwise, I may need to write some fancy pants code that does directory diffs and inspects the code comment headers for changes.

daryl herzmann · Answer 4 · Tue Feb 09 2021 10:56:28 GMT+0800 (China Standard Time)

I have 7.5.1 NCEP NAWIPS now, so here's the massive diff, which I am manually reviewing the diff for now:

gempak/include/proto_gemlib.h

-/* void im_rgin ( char filnam[], int *lenf, int *offset, int *lendat, int *iret ); */
-/* S. Chiswell - modified im_rgin for compressed gini in NOAAPORT */
-void im_rgin ( char filnam[], int *lenf, int *imcflg, int *offset, int *lendat, int *iret );
+void im_rgin ( char filnam[], int *lenf, int *offset, int *lendat, int *iret );

gempak/source/programs/gui/nwx/nwxp_dir.c

- * S. Jacobs/NCEP       4/13   Added checks for TAFS_DEC               *
- * M. James/Unidata    10/09   changed _exten for "_" file exts        *       
+ * S. Jacobs/NCEP       4/13   Added checks for TAFS_DEC               *
-        if  ( ( srchInfo.smethod == OBS ) &&
-            ( ( strcmp ( dtyp_info->datatyp,"SFC_HRLY" ) == 0 ) ||
-              ( strcmp ( dtyp_info->datatyp, "SND_DATA" ) == 0 ) ||
-              ( strcmp ( dtyp_info->datatyp, "SYN_DATA" ) == 0 ) ) ) {
-               sprintf( _exten, "%s", dtyp_info[idtyp].filext );
-       }
-       else {
-               sprintf( _exten, ".%s", dtyp_info[idtyp].filext );
-       }
+       sprintf( _exten, ".%s", dtyp_info[idtyp].filext );

gempak/source/programs/gui/nwx/nwxp_srcho.c

- * M. James/Unidata    10/09   changed cst_ilst to allow for _ sep     *
+ * B. Hebbard/NCEP     01/21   Changed century breakpoint from         *
+ *                             2020/2021 to 2040/2041 (iyr)            *
-               
-                if  ( ( srchInfo.smethod == OBS )  &&
+               cst_ilst ( &tmpstr[1], '.', IMISSD, 2, iarr, &num, &ier );
+               if  ( ( srchInfo.smethod == OBS )  &&
                    ( ( strcmp ( datatype, "SFC_HRLY" ) == 0 ) ||
                      ( strcmp ( datatype, "SND_DATA" ) == 0 ) ||
                      ( strcmp ( datatype, "SYN_DATA" ) == 0 ) ) ) {
 
-                   cst_ilst ( &tmpstr[1], '_', IMISSD, 2, iarr, &num, &ier );

gempak/source/programs/gui/nwx/nwxp_txtw.c

- * M. James/Unidata     11/09   changed cst_ilst to allow for _ sep     *
+ * B. Hebbard/NCEP     01/21   Changed century breakpoint from         *
+ *                             2020/2021 to 2040/2041 (iyr)            *
+       cst_ilst ( filnme, '.', IMISSD, 2, iarr, &num, &ier );
         datatype = nwxTable->dtyp_info[srchInfo.idtyp].datatyp;
 
        if  ( ( srchInfo.smethod == OBS )  &&
@@ -130,12 +132,10 @@ int  iarr[2], num, ier, iyr, imn, idy, ihr;
              ( strcmp ( datatype, "SND_DATA" ) == 0 ) ||
              ( strcmp ( datatype, "SYN_DATA" ) == 0 ) ) ) {
 
-           cst_ilst ( filnme, '_', IMISSD, 2, iarr, &num, &ier );

daryl herzmann · Answer 5 · Tue Feb 09 2021 11:25:14 GMT+0800 (China Standard Time)

NAWIPS 7.12.1 provided ncepBUFR v11.3.0, but also a YK21 fix within v10.2.3. Just took the v10.2.3 fix for the moment and need to investigate this update some more. Would require a Makefile update too.

daryl herzmann · Answer 6 · Sun Feb 21 2021 19:39:04 GMT+0800 (China Standard Time)

The NCEP RFC memo this weekend had this

RFC 7785 - Implement NAWIPS v7.13.0, which includes updated configuration
tables for Unisys to MRMS NIDS (NEXRAD Data Dissemination Service) radar
image transition. Unisys NIDS radar data will be decommissioned by March 9, 2021. NSSL has developed software which converts the MRMS (Multi-Radar Multi-Sensor) data to NIDS-alike radar imagery data.

I have inquired upstream as to if/when this will be released on the website.

daryl herzmann · Answer 7 · Tue Feb 23 2021 06:06:24 GMT+0800 (China Standard Time)

The 7.12.1 to 7.13.0 diff is very small. Will bump my PR to bring it in as well.

daryl herzmann · Answer 8 · Tue Jan 04 2022 01:51:52 GMT+0800 (China Standard Time)

PR merged, closing. Can followup with specific issues found.