AIT_FW_Quicklooks issueshttps://gitlab.ssec.wisc.edu/cspp_geo/AIT_FW_Quicklooks/-/issues2021-07-16T16:37:59Zhttps://gitlab.ssec.wisc.edu/cspp_geo/AIT_FW_Quicklooks/-/issues/1Cannot pass multiple files on the command line2021-07-16T16:37:59ZAlan De SmetCannot pass multiple files on the command lineYou can pass a single directory or a single filename. If I have a bunch of output files, the directory holding them isn't a great option as AIT_FW_Quicklooks will generate a lot of warnings about files it doesn't understand (log files, ...You can pass a single directory or a single filename. If I have a bunch of output files, the directory holding them isn't a great option as AIT_FW_Quicklooks will generate a lot of warnings about files it doesn't understand (log files, CMIP). Because I can't pass multiple files, `FW_ABI*.nc` doesn't work; it doesn't pass argument parsing.CSPP Geo GRB 1.1Tommy JasminTommy Jasminhttps://gitlab.ssec.wisc.edu/cspp_geo/AIT_FW_Quicklooks/-/issues/2Warnings from matplotlib in output2018-09-19T17:37:57ZAlan De SmetWarnings from matplotlib in outputWhen using the Shellb3 from ftp://ftp/pub/shellb3/ShellB3-Linux-x86_64-CentOS6-20170801-rGIT8dfd19cb-py3-portable-core-cspp.tar.xz , AIT_FW_Quicklooks repeatedly generates this warning:
```
/home/adesmet/data/tmp/cspp-geo-aitf-1.0.3beta...When using the Shellb3 from ftp://ftp/pub/shellb3/ShellB3-Linux-x86_64-CentOS6-20170801-rGIT8dfd19cb-py3-portable-core-cspp.tar.xz , AIT_FW_Quicklooks repeatedly generates this warning:
```
/home/adesmet/data/tmp/cspp-geo-aitf-1.0.3beta/lib/ShellB3/lib/python3.6/site-packages/matplotlib-2.0.2-py3.6-linux-x86_64.egg/matplotlib/ticker.py:1685: UserWarning: Steps argument should be a sequence of numbers
increasing from 1 to 10, inclusive. Behavior with
values outside this range is undefined, and will
raise a ValueError in future versions of mpl.
warnings.warn('Steps argument should be a sequence of numbers\n'
```
It seems like either there is a bug that should be fixed or a warning that should be suppressed.https://gitlab.ssec.wisc.edu/cspp_geo/AIT_FW_Quicklooks/-/issues/3Error using --rawvalues flag in aitf-ql script in 1.0.3beta2017-09-26T20:52:12ZJessica BraunError using --rawvalues flag in aitf-ql script in 1.0.3betaI tried using the "--rawrange" flag and could not get it to work in any case with the quicklooks. I tried using one file (a different one each time), as well as the directory, or a subset of selected files. I kept getting this error.
...I tried using the "--rawrange" flag and could not get it to work in any case with the quicklooks. I tried using one file (a different one each time), as well as the directory, or a subset of selected files. I kept getting this error.
[jbraun@clarke ait]$ /data/users/jbraun/cspp-geo-aitf-1.0.3beta/libexec/aitf-ql --rawrange -o ql-raw FW_ABI-L2-LSTF-M3_G16_s20172491700354_e20172491711132_c20172491711187.nc
2017-09-25 16:54:01.124 (INFO) : -------------------------------------
2017-09-25 16:54:01.125 (INFO) : Plotting quicklooks for file: /data1/users/jbraun/aitf-work/09-06-17/ait/FW_ABI-L2-LSTF-M3_G16_s20172491700354_e20172491711132_c20172491711187.nc
2017-09-25 16:54:01.135 (INFO) : Plotting variable: LST
/data/users/jbraun/cspp-geo-aitf-1.0.3beta/lib/ShellB3/lib/python3.6/site-packages/matplotlib-2.0.2-py3.6-linux-x86_64.egg/matplotlib/ticker.py:1685: UserWarning: Steps argument should be a sequence of numbers
increasing from 1 to 10, inclusive. Behavior with
values outside this range is undefined, and will
raise a ValueError in future versions of mpl.
warnings.warn('Steps argument should be a sequence of numbers\n'
Traceback (most recent call last):
File "/data/users/jbraun/cspp-geo-aitf-1.0.3beta/lib/python/ql/aitf_ql.py", line 409, in <module>
sys.exit(main())
File "/data/users/jbraun/cspp-geo-aitf-1.0.3beta/lib/python/ql/aitf_ql.py", line 385, in main
version_txt=version_msg,
File "/data1/users/jbraun/cspp-geo-aitf-1.0.3beta/lib/python/ql/plotting_util.py", line 330, in create_mapped_figure
tick_positions=tick_positions, tick_labels=tick_labels,)
File "/data1/users/jbraun/cspp-geo-aitf-1.0.3beta/lib/python/ql/plotting_util.py", line 160, in _add_colorbar
if ( ((valid_range[1] - valid_range[0] > 5000.0) and (orientation_temp == 'horizontal')) or
TypeError: 'NoneType' object is not subscriptableEva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/cspp_geo/AIT_FW_Quicklooks/-/issues/4Quicklook images in V1.0.7beta have offset map2018-01-23T20:58:22ZJessica BraunQuicklook images in V1.0.7beta have offset mapSee attached image. Satellite is now located at -75.2, map is offset.
![FW_ABI-L2-LSTF-M3_G16_s20173590245443_e20173590256222_c20173590256273.nc.LST](/uploads/1b28e2959c5bbe3b70c0aafc023f8fec/FW_ABI-L2-LSTF-M3_G16_s20173590245443_e2017...See attached image. Satellite is now located at -75.2, map is offset.
![FW_ABI-L2-LSTF-M3_G16_s20173590245443_e20173590256222_c20173590256273.nc.LST](/uploads/1b28e2959c5bbe3b70c0aafc023f8fec/FW_ABI-L2-LSTF-M3_G16_s20173590245443_e20173590256222_c20173590256273.nc.LST.png)Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/cspp_geo/AIT_FW_Quicklooks/-/issues/5Fix sweep direction when new cartopy available2021-07-12T16:39:27ZEva SchifferFix sweep direction when new cartopy availablePreviously cartopy did not have a way to pass in the sweep direction when using a geostationary projection. For GOES-R we need to use sweep = x for the ABI instrument (the default in the layer under cartopy is unfortunately y rather than...Previously cartopy did not have a way to pass in the sweep direction when using a geostationary projection. For GOES-R we need to use sweep = x for the ABI instrument (the default in the layer under cartopy is unfortunately y rather than x).
The folks working on cartopy have apparently made the fix, so when they release a new version we need to make the change in our QL to set the sweep correctly and make sure that we are able to package the program with the newer, fixed version of cartopy.CSPP Geo GRB 1.1Nick Bearsonnickb@ssec.wisc.eduNick Bearsonnickb@ssec.wisc.eduhttps://gitlab.ssec.wisc.edu/cspp_geo/AIT_FW_Quicklooks/-/issues/6revise QL colorbars based on clavr-x plots2018-03-08T22:31:55ZEva Schifferrevise QL colorbars based on clavr-x plotsLook over what Nick is using for colorbars in clavr-x and see if we can change ours accordingly.
Nick says the relevant code to look at is here:
https://gitlab.ssec.wisc.edu/nickb/cspp-clavrx/blob/master/quicklooks/ql_clavrx.pyLook over what Nick is using for colorbars in clavr-x and see if we can change ours accordingly.
Nick says the relevant code to look at is here:
https://gitlab.ssec.wisc.edu/nickb/cspp-clavrx/blob/master/quicklooks/ql_clavrx.pyhttps://gitlab.ssec.wisc.edu/cspp_geo/AIT_FW_Quicklooks/-/issues/7Revise CMI QL labels2021-07-12T17:08:13ZEva SchifferRevise CMI QL labelsCurrently the CMI QL labels say "L1 reflectance" and "L1 brightness temperature", but they are actually considered a L2+ product.
From looking at the long names in the CMI example files that I have, those appear to correctly identify t...Currently the CMI QL labels say "L1 reflectance" and "L1 brightness temperature", but they are actually considered a L2+ product.
From looking at the long names in the CMI example files that I have, those appear to correctly identify the data as L2+, so the easiest fix here is to use the long names more directly when titling the QL plots.https://gitlab.ssec.wisc.edu/cspp_geo/AIT_FW_Quicklooks/-/issues/8Add command line option to select only one variable/product from a file to plot2021-07-12T16:41:18ZEva SchifferAdd command line option to select only one variable/product from a file to plotMany of the files we're processing only have one product per file that we care about, but some like aerosols have more. Add an optional command line argument that will let the user only plot one product present in a given file.Many of the files we're processing only have one product per file that we care about, but some like aerosols have more. Add an optional command line argument that will let the user only plot one product present in a given file.https://gitlab.ssec.wisc.edu/cspp_geo/AIT_FW_Quicklooks/-/issues/9add command line option to specify the valid range when plotting2021-07-12T16:41:46ZEva Schifferadd command line option to specify the valid range when plottingThis would be intended to allow users to specify a more limited subset than just the valid_range listed in the file.This would be intended to allow users to specify a more limited subset than just the valid_range listed in the file.CSPP Geo GRB 1.1Nick Bearsonnickb@ssec.wisc.eduNick Bearsonnickb@ssec.wisc.eduhttps://gitlab.ssec.wisc.edu/cspp_geo/AIT_FW_Quicklooks/-/issues/10review and revise QL style2021-07-12T16:43:12ZEva Schifferreview and revise QL styleReview the look and style of the colormaps with people who have a better idea what scientists will need from them (preferably including people like Kathy, Jess, and Liam) and revise based on their recommendations.
This could include ch...Review the look and style of the colormaps with people who have a better idea what scientists will need from them (preferably including people like Kathy, Jess, and Liam) and revise based on their recommendations.
This could include changing things like grid and boundary colors, grid positioning (sparsity of lines), light weights, colormaps used for the different products, and anything else they think might need revision.https://gitlab.ssec.wisc.edu/cspp_geo/AIT_FW_Quicklooks/-/issues/11when converting L1 files in QL, change variable name used to tag the end of t...2021-07-12T16:43:25ZEva Schifferwhen converting L1 files in QL, change variable name used to tag the end of the output fileRight now when QL converts L1 files it will still put something like ".Rad.png" on the end of the file name. Change this to something that represents the converted data like ".Refl.png" or ".BT.png".Right now when QL converts L1 files it will still put something like ".Rad.png" on the end of the file name. Change this to something that represents the converted data like ".Refl.png" or ".BT.png".CSPP Geo GRB 1.1Tommy JasminTommy Jasminhttps://gitlab.ssec.wisc.edu/cspp_geo/AIT_FW_Quicklooks/-/issues/12Use the attribute with the FW version that is now in the FW output to stamp o...2021-07-12T16:43:53ZEva SchifferUse the attribute with the FW version that is now in the FW output to stamp on the QL imagesThe attribute should show up on FW and CMI output and is "AIT_SoftwareVersion".The attribute should show up on FW and CMI output and is "AIT_SoftwareVersion".https://gitlab.ssec.wisc.edu/cspp_geo/AIT_FW_Quicklooks/-/issues/14Investigate black ring in Clear Sky Mask quicklooks2019-11-22T22:02:02ZJessica BraunInvestigate black ring in Clear Sky Mask quicklooksI compared software versions 1.0.13 and 1.0.14 by creating quicklooks of both L2 output files using the quicklooks from 1.0.14 for both.
This is created with 1.0.13, ql created with 1.0.14.
![CG_ABI-L2-ACMF-M3_G16_s20180801900446_e2018...I compared software versions 1.0.13 and 1.0.14 by creating quicklooks of both L2 output files using the quicklooks from 1.0.14 for both.
This is created with 1.0.13, ql created with 1.0.14.
![CG_ABI-L2-ACMF-M3_G16_s20180801900446_e20180801911225_c20180801911280.nc.BCM](/uploads/5d0c1efe689f5bd1a34d4ae5abdfa257/CG_ABI-L2-ACMF-M3_G16_s20180801900446_e20180801911225_c20180801911280.nc.BCM.png)
This is created with 1.0.14, ql created with 1.0.14. Mask is properly set to missing in space, but a black ring is around the edge of the cloud mask.
![CG_ABI-L2-ACMF-M3_G16_s20180811900446_e20180811911225_c20180811911281.nc.BCM](/uploads/cbdb26ca463e307e46fa634258c3f8d7/CG_ABI-L2-ACMF-M3_G16_s20180811900446_e20180811911225_c20180811911281.nc.BCM.png)
It's not the greatest image, but this is the display in Panoply.
![Screen_Shot_2018-03-23_at_11.50.16_AM](/uploads/2dec95014f49e78e8fbd8eb7771c187b/Screen_Shot_2018-03-23_at_11.50.16_AM.png)
Looking at the data in Panoply, I only see the fill value, 0 and 1. Would definitely appreciate another eye to see if this is a quicklook issue between space/Earth as the ring is only within the Earth boundaries, or if it's something else.
Just as a note, I did not include, but when re-running the test case with 1.0.14, I see this same this with data from back in January.Jessica BraunJessica Braunhttps://gitlab.ssec.wisc.edu/cspp_geo/AIT_FW_Quicklooks/-/issues/15Downloading ne_110m_land.zip and ne_110m_ocean.zip2021-07-14T15:35:42ZAlan De SmetDownloading ne_110m_land.zip and ne_110m_ocean.zipThe first time (and only the first time) I run the quicklooks script, I get the following messages:
```
/INSTALL_PATH/cspp-geo-aitf-1.0.15beta/lib/ShellB3/lib/python3.6/site-packages/Cartopy-0.16.0-py3.6-linux-x86_64.egg/cartopy/io/__in...The first time (and only the first time) I run the quicklooks script, I get the following messages:
```
/INSTALL_PATH/cspp-geo-aitf-1.0.15beta/lib/ShellB3/lib/python3.6/site-packages/Cartopy-0.16.0-py3.6-linux-x86_64.egg/cartopy/io/__init__.py:260: DownloadWarning: Downloading: http://naciscdn.org/naturalearth/110m/physical/ne_110m_land.zip
warnings.warn('Downloading: {}'.format(url), DownloadWarning)
/INSTALL_PATH/cspp-geo-aitf-1.0.15beta/lib/ShellB3/lib/python3.6/site-packages/Cartopy-0.16.0-py3.6-linux-x86_64.egg/cartopy/io/__init__.py:260: DownloadWarning: Downloading: http://naciscdn.org/naturalearth/110m/physical/ne_110m_ocean.zip
warnings.warn('Downloading: {}'.format(url), DownloadWarning)
```
This 1. seems like useless noise, and 2. suggests that if a third party site is down, there might be problems. Is this something that can and should be in ShellB3? Have I borken something in how I'm using/installing quicklooks?Alan De SmetAlan De Smethttps://gitlab.ssec.wisc.edu/cspp_geo/AIT_FW_Quicklooks/-/issues/17Suppress warnings from underlying libraries2021-07-12T16:46:22ZEva SchifferSuppress warnings from underlying librariesRight now the software generates several warnings from the underlying software that aren't actually hazardous. Suppress those warnings unless the user is on a more verbose level of logging like debug.
For example, when converting L1 dat...Right now the software generates several warnings from the underlying software that aren't actually hazardous. Suppress those warnings unless the user is on a more verbose level of logging like debug.
For example, when converting L1 data into imagery:
```
2018-07-06 11:33:00.307 (INFO) : -------------------------------------
2018-07-06 11:33:00.307 (INFO) : Plotting quicklooks for file: /Users/evas/Dev/CSPP FW Quicklooks/test data/2018-02_AIT-Delivery_output/L1b/CONUS/OR_ABI-L1b-RadC-M3C02_G16_s20180012232199_e20180012234572_c20180012235015.nc
2018-07-06 11:33:00.340 (INFO) : Plotting variable: Rad
/Users/evas/Dev/CSPP FW Quicklooks/AIT_FW_Quicklooks/lvl1_data.py:373: UserWarning: Warning: converting a masked element to nan.
'fk1': float(nc['planck_fk1'][:]),
/Users/evas/Dev/CSPP FW Quicklooks/AIT_FW_Quicklooks/lvl1_data.py:374: UserWarning: Warning: converting a masked element to nan.
'fk2': float(nc['planck_fk2'][:]),
/Users/evas/Dev/CSPP FW Quicklooks/AIT_FW_Quicklooks/lvl1_data.py:375: UserWarning: Warning: converting a masked element to nan.
'bc1': float(nc['planck_bc1'][:]),
/Users/evas/Dev/CSPP FW Quicklooks/AIT_FW_Quicklooks/lvl1_data.py:376: UserWarning: Warning: converting a masked element to nan.
'bc2': float(nc['planck_bc2'][:]),
2018-07-06 11:33:03.495 (WARNING) : Square root scaling is configured for this product and the product data overflows below the lower limit of the range to be plotted. But matplotlib currently cannot display colorbars with an arrow indicating minimum overflow due to a bug. Therefore the minimum overflow indication will not be displayed in the plot for this data.
2018-07-06 11:33:06.028 (INFO) : -------------------------------------
```
I'll need to run some test cases to figure out if there are other common cases we'll need to catch. There may be some matplotlib and cartopy edge cases.CSPP Geo GRB 1.1Graeme MartinGraeme Martinhttps://gitlab.ssec.wisc.edu/cspp_geo/AIT_FW_Quicklooks/-/issues/18Update help text in aitf_ql.py2021-07-12T16:46:33ZJessica BraunUpdate help text in aitf_ql.pyWhile testing GRB 1.0.12, Jay noticed a spelling error in the help text:
```
--onlyvar ONLYVAR Process only the variable name given. No other
variables from the **intput** files will be used to make
...While testing GRB 1.0.12, Jay noticed a spelling error in the help text:
```
--onlyvar ONLYVAR Process only the variable name given. No other
variables from the **intput** files will be used to make
quicklooks. If your variable name has spaces use
single quotes around it.
```
Update "intput" to "input". :)Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/cspp_geo/AIT_FW_Quicklooks/-/issues/19test averaging on input2023-03-01T16:04:11ZGraeme Martintest averaging on inputCapturing the results of Eva and my discussion last week...
Currently some quicklooks appear somewhat speckly, e.g. isolated light pixels on a dark background. I think this is caused by the tendency of striding to occasionally select ou...Capturing the results of Eva and my discussion last week...
Currently some quicklooks appear somewhat speckly, e.g. isolated light pixels on a dark background. I think this is caused by the tendency of striding to occasionally select outlier pixels (effectively we are seeing noise due to sub-sampling). Have observed that reducing the stride changes the pattern of speckliness, but does not noticeably improve it. Suspect that providing larger arrays to the plotting routines does not help because some kind of striding is done internally in the plotting routines.
Task is to try averaging discrete blocks of pixels on ingest, instead of striding. Block size would be chosen similar to current auto-stride, to give the smallest array that is larger than the image size. One complication is that missing values will have to be screened out before averaging (one good value should be enough). Also data may have to be discarded near the edges to avoid using under-sized blocks.
If averaging produces significantly better images, should assess impact on plotting times and consider adding for a future release. Could offer command line option controlling striding vs sampling.
Here is an example that looks speckly to me (though note there is nothing special about this example):
![OR_ABI-L1b-RadF-M3C03_G16_s20181501415468_e20181501426235_c20181501426284.nc.Refl](/uploads/25b451d7eb87a3d56091924a27c7b932/OR_ABI-L1b-RadF-M3C03_G16_s20181501415468_e20181501426235_c20181501426284.nc.Refl.png)CSPP Geo GRB 1.1Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/cspp_geo/AIT_FW_Quicklooks/-/issues/20Consider offering an option to specify image size in pixels2021-07-12T16:47:27ZGraeme MartinConsider offering an option to specify image size in pixelsCurrently (image_size in inches) * DPI = (image size in pixels). Might be clearer for users to specify pixels directly.
I think this would be an improvement but should get input from Jess and Kathy on this before implementing.Currently (image_size in inches) * DPI = (image size in pixels). Might be clearer for users to specify pixels directly.
I think this would be an improvement but should get input from Jess and Kathy on this before implementing.CSPP Geo GRB 1.1Graeme MartinGraeme Martinhttps://gitlab.ssec.wisc.edu/cspp_geo/AIT_FW_Quicklooks/-/issues/21Explicitly state default verbosity level in usage message (looks like -vv?)2021-07-12T16:47:35ZGraeme MartinExplicitly state default verbosity level in usage message (looks like -vv?)Based on Jay Heinzelman's comments from testing GRB 1.0.
>>>
1. Some extraneous python messages are being printed by default when I run the quicklook script. Looks like -vv [INFO] output?
[jayh@chai product]$ cspp-geo-grb-ql.sh OR_AB...Based on Jay Heinzelman's comments from testing GRB 1.0.
>>>
1. Some extraneous python messages are being printed by default when I run the quicklook script. Looks like -vv [INFO] output?
[jayh@chai product]$ cspp-geo-grb-ql.sh OR_ABI-L1b-RadC-M3C13_G16_s20181851542221_e20181851545005_c20181851545035.nc --image_size 20.0 16.0 --stride 1 -O JAY
2018-07-30 19:11:14.254 (INFO) : -------------------------------------
2018-07-30 19:11:14.254 (INFO) : Plotting quicklooks for file: /data/users/jayh/grb_output/product/OR_ABI-L1b-RadC-M3C13_G16_s20181851542221_e20181851545005_c20181851545035.nc
2018-07-30 19:11:14.272 (INFO) : Plotting variable: Rad
/home/jayh/cspp-geo-grb-1.0/quicklooks/lvl1_data.py:377: UserWarning: Warning: converting a masked element to nan.
'kappa0': float(nc['kappa0'][:])
/home/jayh/cspp-geo-grb-1.0/quicklooks/lvl1_data.py:321: RuntimeWarning: invalid value encountered in log
return (fk2 / numpy.log((fk1 / L) + 1.0) - bc1) / bc2
>>>
My reply:
>>>
There is a note in section 4.2 that the warnings can be ignored, and note that we are planning to suppress or redirect those messages in a future release. Might be worth a caveat in the doc (there is a Caveats / Known Issues section toward the end). As far as the info messages, I would say that is close to the typical level of verbosity from a CSPP package. The main program in the GRB package is an exception because it runs as a server, so all the output has to go to a log. Personally when I run a program to process data at the command line, I like to see some indication of what it is doing, without being overly verbose, so I think this is about right. Let us know if you have other ideas.
Could add a note to the usage message that -vv is the default, if that is indeed the case.
>>>
Note this is related to #17.CSPP Geo GRB 1.1Graeme MartinGraeme Martinhttps://gitlab.ssec.wisc.edu/cspp_geo/AIT_FW_Quicklooks/-/issues/22Clarify striding in usage message2021-07-12T16:48:12ZGraeme MartinClarify striding in usage messageComment from Jay Heinzelman from testing GRB 1.0:
>>>
2. --stride? How does this work? Do the resolutions of the data really not lend to this showing much difference? I didn't find a combination of image size and stride that yielde...Comment from Jay Heinzelman from testing GRB 1.0:
>>>
2. --stride? How does this work? Do the resolutions of the data really not lend to this showing much difference? I didn't find a combination of image size and stride that yielded visible or file size differences. I tested with -S 1 and -S -1. Those seemed like the only two options from how I read the documentation. -S 1 will used the full resolution, and -S any negative number will try to reduce the number of lines in the processed dependent on the quick look image size. (At first I thought if I used -6 I'd get every sixth line/element, but after testing and reading the doc closer I didn't think that was the case.) I wanted to point this out in case there is a problem here.
>>>
Graeme's reply (second paragraph is relevant to this ticket):
>>>
You can specify a positive stride to override the default auto-stride, but in my testing there was little effect on the plots until you actually exceed the auto-stride value and the image gets coarser. My theory is that the Python plotting routines are doing striding internally, so as far as image quality it doesn't really matter how much you stride on input (up to the point where you have fewer input pixels than the image size).
Maybe the action here is to clarify in the usage message that stride should be specified as a positive number, with a special value of -1 indicating auto-stride.
>>>CSPP Geo GRB 1.1Graeme MartinGraeme Martin