Default to pqinsert -i

For deduplication purposes, pqinsert -i is preferred when working with GRB products, since equivalent files are produced that are non-identical. JerryR noted that Unidata is using "pqinsert -i" to make the MD5 signature based on product name and not content. Ref https://www.unidata.ucar.edu/software/ldm/man/pqinsert

• Add PQINSERT_FLAGS which defaults to -i to nc_fits_pqinsert.sh
• Add same to hcast2scmi.sh
• test on VM ingestors to ensure no impact
• confirm that deduplication happens when multiple GRB systems feed data to a single downstream LDM

created merge request !6 (merged) to address this issue

mentioned in merge request !6 (merged)

mentioned in commit 76dbaf2d

SCMI note

If this is also needed for SCMI, the "creation" time in the filename is currently based off the tile-set creation time and not the upstream product creation time - which implies that deduplication of SCMI is likely to require an update to AXI-tools to force creation time to be uniformly assigned from the product time and not the actual SCMI tileset creation time.

marked the checklist item Add PQINSERT_FLAGS which defaults to -i to nc_fits_pqinsert.sh as completed

marked the checklist item Add same to hcast2scmi.sh as completed

marked the checklist item test on VM ingestors to ensure no impact as completed

In testing on kettle129 (g17)

cd /opt/nws_glue/abi
mv nc_fits_pqinsert.sh{,.orig}
wget https://gitlab.ssec.wisc.edu/cspp_geo/nws_glue/raw/2-default-to-pqinsert-i/abi/nc_fits_pqinsert.sh
chmod a+x nc_fits_pqinsert.sh
pstree -u ldm -p
kill <PIDs of nc_fits_pqinsert.sh tree>
pstree -u ldm -p
tail -f ~ldm/var/log/nc_fits_pqinsert.log
# wait for 5-min cron-based restart and confirm "-i" added to pqinsert commands

Also in testing on g16 kettle135.

[ nc-fits-pqinsert-stdout 20200527T224506 ] + /home/ldm/bin/pqinsert -i -v -q /home/ldm/var/queues/ldm.pq -f EXP OR_SUVI-L1b-Fe195_G16_s20201482244328_e20201482244338_c20201482244518.nc

Tested by forcing a redundant G17 product insertion on kettle129 (G17 VM), using an equivalent product file from another test ingestor. Note the refusal to pqinsert the product, even though content MD5 is different.

cd /data/output-csppV1.0/product
source ../c7-env.sh
unset CSPP_LOCK_DIR
unset CSPP_LOG_DIR
fn=$(ls -1rt OR_ABI*RadF*C16*nc |tail -2 |head -1); echo $fn
md5sum $fn
mv $fn /tmp
scp rayg@grbmon.ssec.wisc.edu:/scratch/nickb/cspp-geo-grb-1.0/output/product/$fn .
md5sum $fn
/opt/nws_glue/abi/nc_fits_pqinsert.sh pqinsert_deliver $PWD/$fn
mv -f /tmp/$fn .

As expected, files with different content signatures are still considered a redundant insert

[root@c7-ldm-grbd-axid-auto-g17 product]# fn=$(ls -1rt OR_ABI*RadF*C16*nc |tail -2 |head -1); echo $fn
OR_ABI-L1b-RadF-M6C16_G17_s20201490240319_e20201490249397_c20201490249447.nc
[root@c7-ldm-grbd-axid-auto-g17 product]# md5sum $fn
5cf9b94ba6df36a1e9d3948169d97106  OR_ABI-L1b-RadF-M6C16_G17_s20201490240319_e20201490249397_c20201490249447.nc
[root@c7-ldm-grbd-axid-auto-g17 product]# mv $fn /tmp
[root@c7-ldm-grbd-axid-auto-g17 product]# scp rayg@grbmon.ssec.wisc.edu:/scratch/nickb/cspp-geo-grb-1.0/output/product/$fn .
OR_ABI-L1b-RadF-M6C16_G17_s20201490240319_e20201490249397_c20201490249447.nc                                                                                                                                                        100%   21MB 108.7MB/s   00:00    
[root@c7-ldm-grbd-axid-auto-g17 product]# md5sum $fn
5351ebf72896e9de5ea2b528fa6296f4  OR_ABI-L1b-RadF-M6C16_G17_s20201490240319_e20201490249397_c20201490249447.nc
[root@c7-ldm-grbd-axid-auto-g17 product]# /opt/nws_glue/abi/nc_fits_pqinsert.sh pqinsert_deliver $PWD/$fn
+ /home/ldm/bin/pqinsert -i -v -q /home/ldm/var/queues/ldm.pq -f EXP /data/output-csppV1.0/product/OR_ABI-L1b-RadF-M6C16_G17_s20201490240319_e20201490249397_c20201490249447.nc
20200528T030636.281313Z pqinsert[119280]            pqinsert.c:main:443                 ERROR Product already in queue: 9b93a529ac46ea3b917f661e8a04e7b5   22124151 20200528030636.279196 EXP 000  /data/output-csppV1.0/product/OR_ABI-L1b-RadF-M6C16_G17_s20201490240319_e20201490249397_c20201490249447.nc
WARNING: /home/ldm/bin/pqinsert: already in queue
[root@c7-ldm-grbd-axid-auto-g17 product]# mv -f /tmp/$fn .

# confirm the product is in the queue, just in case any of the above fails
[root@c7-ldm-grbd-axid-auto-g17 product]# /home/ldm/bin/pqcat -f EXP -vl - >/dev/null 2>/tmp/pqcat.log
[root@c7-ldm-grbd-axid-auto-g17 product]# grep $fn /tmp/pqcat.log
20200528T030955.359188Z pqcat[126352]               pqcat.c:writeprod:80                INFO    22124151 20200528030630.093462 EXP 000  /data/output-csppV1.0/product/OR_ABI-L1b-RadF-M6C16_G17_s20201490240319_e20201490249397_c20201490249447.nc

marked the checklist item confirm that deduplication happens when multiple GRB systems feed data to a single downstream LDM as completed

closed