UW-Glance issueshttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues2024-01-11T14:28:06Zhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/97Persistent memory leak issues2024-01-11T14:28:06ZEva SchifferPersistent memory leak issuesKatie Pitts from NOAA reports that Glance is still having persistent, problematic memory issues:
> I'm reopening this thread just to say that I continue to have memory errors when running Glance on directories of GOES-R imagery files th...Katie Pitts from NOAA reports that Glance is still having persistent, problematic memory issues:
> I'm reopening this thread just to say that I continue to have memory errors when running Glance on directories of GOES-R imagery files that contain channel 2 datasets. Once my EC2 machine's memory was increased from 64GB to 128GB I was able to run through the entire set of files in my directory without error. I was watching the top statistics in the terminal while it was running, and it appears that memory is not always released when moving from one file comparison to a new file comparison, which could be why it keeps dying once it gets to a larger channel 2 file in the list. At one point it was using a whopping 111GB of memory. I attached the traceback of the memory error when the run dies, as well as at the bottom of that text file a runtime warning that showed up on the Glance run that finally made it through all the files.
> I'll note that since I last emailed I have installed the latest version of Glance using the latest installation recommendation of doing so using MambaForge.
> Since we've found a workaround of increasing the EC2 machine's memory when needed to run these Glance reports, we're okay for now. I'm mainly bringing this to your attention so you can address memory handling in any future version of Glance.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/96Move Glance to python 3.102023-10-02T14:25:21ZEva SchifferMove Glance to python 3.10Move Glance to working in python 3.10. Currently Glance is tested and functioning in 3.6, but things have moved on, so the supporting software we rely on isn't all available in 3.6 anymore. This transision will probably require a lot of ...Move Glance to working in python 3.10. Currently Glance is tested and functioning in 3.6, but things have moved on, so the supporting software we rely on isn't all available in 3.6 anymore. This transision will probably require a lot of small clean up. The list below is what I know about so far and will be expanded as I investigate.
- [x] Review changes in python from 3.6 to 3.10 and make fixes in Glance as needed
- [x] Move from easy_install to a newer installation procedure (probably pip based?)
- [x] Rewrite the installation instructions on the wiki (note: include move from conda to mambaforge)
- [x] Do some minimal testing of Glance to make sure things don't break
- [x] numpy.bool causes a fatal AttributeError if you run the guiEva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/95More detailed histograms when only a small amount of data is outliers2023-03-17T18:34:36ZEva SchifferMore detailed histograms when only a small amount of data is outliersThis is a suggestion from Rob Levy.
> Where, there are big differences at a few points, but 99% of points are within ±0.01. This is where signed(exponent(logged(differences)))) would be extremely helpful (your logit scale?)
>
> X = ...This is a suggestion from Rob Levy.
> Where, there are big differences at a few points, but 99% of points are within ±0.01. This is where signed(exponent(logged(differences)))) would be extremely helpful (your logit scale?)
>
> X = Max(Absolute difference)
> A = Min(absolute difference) that is not zero
> Range = log(X) – log(A)
> Divvy up range into 25 (or fewer is probably okay) intervals of equal log
> Final plot will have 25 intervals on left (<0.0), 25 intervals on right (>0.0), plus one with exactly 0.0.
> Now you can have histogram!Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/94Allow custom colormaps2023-03-17T18:31:14ZEva SchifferAllow custom colormapsGive the user some way of setting up custom colormaps in the configuration file. These colormaps would apply to the main variable data, not the comparison plots.Give the user some way of setting up custom colormaps in the configuration file. These colormaps would apply to the main variable data, not the comparison plots.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/93Configure plots of the main data to appear in log scale2023-03-17T18:34:57ZEva SchifferConfigure plots of the main data to appear in log scaleFor some types of data it would be useful for users to be able to plot them with log scale.
Rob also asked: "It looks like one can make histograms log scaled (in the GUI), It would just be awesome to have “negative” log (e.g. absolute v...For some types of data it would be useful for users to be able to plot them with log scale.
Rob also asked: "It looks like one can make histograms log scaled (in the GUI), It would just be awesome to have “negative” log (e.g. absolute value in log space, but negative sign to plot)."
(This second part may get deferred to a different issue later, but I want to note it here so I don't forget.)Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/92More clear color bar for the differences2022-08-31T21:47:45ZEva SchifferMore clear color bar for the differencesIt would be good to use a colorbar that more clearly highlights the differences in the difference plots. So something that has a clear turning point in the middle that we can place 0 difference at. Rob Levy suggested RedBu, the Spectral ...It would be good to use a colorbar that more clearly highlights the differences in the difference plots. So something that has a clear turning point in the middle that we can place 0 difference at. Rob Levy suggested RedBu, the Spectral Rainbow (with yellow as 0), or something similar.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/91investigate case sensitive issues with data scaling2023-05-30T15:54:39ZEva Schifferinvestigate case sensitive issues with data scalingMy notes on this are very old, but I need to double check that Glance is handling any data scaling attributes in a case-insensitive manner. I know I'm handling some attributes appropriately, but I should investigate just in case I missed...My notes on this are very old, but I need to double check that Glance is handling any data scaling attributes in a case-insensitive manner. I know I'm handling some attributes appropriately, but I should investigate just in case I missed some cases (as I didn't take detailed notes of why I was worried about this problem).Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/90data reversing filter2022-08-17T15:31:12ZEva Schifferdata reversing filterAdd a filter that will allow you to reverse the data along one index of a multi-dimensional set.Add a filter that will allow you to reverse the data along one index of a multi-dimensional set.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/89Problem saving images when using threads on a Mac2022-08-17T16:29:51ZEva SchifferProblem saving images when using threads on a MacI have some notes that say Glance is now unable to save images when you use multi-threading on a Mac (there was a warning before when this was a problem and now there is none).
I need to reinvestigate this and either warn the user or d...I have some notes that say Glance is now unable to save images when you use multi-threading on a Mac (there was a warning before when this was a problem and now there is none).
I need to reinvestigate this and either warn the user or disallow the setting in the appropriate environment.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/88Make sure summary report works properly with missing variables2022-04-07T14:57:43ZEva SchifferMake sure summary report works properly with missing variablesNow that the main report includes variables which were requested but not found when the user asks to be warned about missing variables, the summary report will need to handle that more gracefully.Now that the main report includes variables which were requested but not found when the user asks to be warned about missing variables, the summary report will need to handle that more gracefully.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/87Glance Unit Tests2022-03-07T21:22:59ZEva SchifferGlance Unit TestsGlance doesn't have unit testing built into it. Some parts of the program are going to be very hard to test (because it's got a lot of file access and complex output) but I want to include more unit tests where I can to cut down on the c...Glance doesn't have unit testing built into it. Some parts of the program are going to be very hard to test (because it's got a lot of file access and complex output) but I want to include more unit tests where I can to cut down on the chances things will break and not be detected when I make changes.
I'll put in some more notes and detailed plans once I've done some evaluation of where it will be least painful to start this effort.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/86stats should respect configuration files2022-03-03T19:35:05ZEva Schifferstats should respect configuration filesRight now calls to stats ignore configuration files that you specify on the command line. stats should load and respect the relevant information from a provided configuration file.Right now calls to stats ignore configuration files that you specify on the command line. stats should load and respect the relevant information from a provided configuration file.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/85Glance crashes when opening a str variable in the GUI (or stats)2022-03-01T19:14:54ZEva SchifferGlance crashes when opening a str variable in the GUI (or stats)If you open a string type variable in the GUI or stats Glance will crash. We should handle this more gracefully.
I've attached an example file. The variable that triggers the problem is AMVChannelName.
[GOES17_ABI_2KM_MESO_2021056_194...If you open a string type variable in the GUI or stats Glance will crash. We should handle this more gracefully.
I've attached an example file. The variable that triggers the problem is AMVChannelName.
[GOES17_ABI_2KM_MESO_2021056_1942_25_WINDS_AMV_EN-14-CT.nc](/uploads/c449d0fa6d1b7e82ff1c8d1fd63e2e16/GOES17_ABI_2KM_MESO_2021056_1942_25_WINDS_AMV_EN-14-CT.nc)https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/84Make more things configurable in the config file2022-02-25T19:46:19ZEva SchifferMake more things configurable in the config fileRight now Glance is getting some things from the command line that don't have an equivalent in the config file. I want to keep the output path and input file paths command line only, but for all the others, we should just have a way to s...Right now Glance is getting some things from the command line that don't have an equivalent in the config file. I want to keep the output path and input file paths command line only, but for all the others, we should just have a way to specify them in the config file.
Make a way to set the following in the config file
- [x] the **--nolonlat** option (maybe by setting the lat_lon_info to None?)
- [x] the **--warnmissingvars** option (maybe in the settings?)
I will also need to
- [x] Change the associated documentation to describe how these new settings work
- [x] Alter or add example configuration files to demonstrate how to use these
Note: the work to handle this for the **--doPassFail** option is being handled in the ticket for the pass/fail redesign: https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/83Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/83the --doPassFail command line option is not working properly (and needs some ...2022-02-24T16:11:35ZEva Schifferthe --doPassFail command line option is not working properly (and needs some redesign)At the moment the --doPassFail command line option turns on pass/fail testing for the purposes of the return codes, but it doesn't actually control how reports generate at the moment.
On a higher level, I think I want this setting to b...At the moment the --doPassFail command line option turns on pass/fail testing for the purposes of the return codes, but it doesn't actually control how reports generate at the moment.
On a higher level, I think I want this setting to be on by default and I need to create an appropriate config file setting that will let you do the same thing.
- [x] Turn pass/fail testing on by default
- [x] Make a command line option to turn pass/fail testing off
- [x] Make the current command line option (that turns pass/fail testing on) invisible in the docs (because it won't be needed anymore) but leave it in the system so people's existing calls don't break
- [x] Make a config file setting to turn pass/fail on and off (probably in the settings dictionary)
- [x] review report and stats code to make sure it will respect the pass/fail setting for all things (returns and how variables are analyzed)
- [x] Change the documentation to reflect the new way pass/fail testing works
- [x] Update that one config file that's intended to turn pass/fail testing offEva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/82Test and maintain the AERI part of the io library2022-01-28T17:13:28ZEva SchifferTest and maintain the AERI part of the io libraryI need to get set up to test and maintain the AERI part of the io library. Ray originally wrote the class for this type of file and I never got myself setup to test it.
As a side thing, I also need to revise the display_string provided...I need to get set up to test and maintain the AERI part of the io library. Ray originally wrote the class for this type of file and I never got myself setup to test it.
As a side thing, I also need to revise the display_string provided by the aeri class. Right now it's very bare bones.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/81Consider revising the io module to use Rasterio instead of gdal2022-01-26T22:06:28ZEva SchifferConsider revising the io module to use Rasterio instead of gdalDave recommended that the Rasterio python library may be better maintained and more pythonic, so at some point I should look into how much work it will be to move to using that instead of gdal to handle our GeoTIFFs.
For reference:
[The...Dave recommended that the Rasterio python library may be better maintained and more pythonic, so at some point I should look into how much work it will be to move to using that instead of gdal to handle our GeoTIFFs.
For reference:
[The Rasterio docs](https://rasterio.readthedocs.io/en/latest/)Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/80loading attributes in some files with complex group structure fails2022-01-20T21:12:42ZEva Schifferloading attributes in some files with complex group structure failsGlance is able to open several types of files with hierarchical groups. In the gui Glance seems to be unable to load variable attributes for files that have their variables inside more complex group structures.
I've seen this bug so fa...Glance is able to open several types of files with hierarchical groups. In the gui Glance seems to be unable to load variable attributes for files that have their variables inside more complex group structures.
I've seen this bug so far with an hdf4 file and an h5 file. It seems to work ok in the netcdf file I have with groups.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/79Fix GeoTIFF plotting in the gui2022-01-11T19:41:39ZEva SchifferFix GeoTIFF plotting in the guiAt one point Glance was able to plot GeoTIFFs in the gui as rgb images.
I think I broke that while trying to keep other errors from causing crashes when plotting. Right now if you try to plot a GeoTIFF as an rgb image, you get a messag...At one point Glance was able to plot GeoTIFFs in the gui as rgb images.
I think I broke that while trying to keep other errors from causing crashes when plotting. Right now if you try to plot a GeoTIFF as an rgb image, you get a message like:
"Two dimensional data is required for this plot type. The provided variable data is (1500, 1500, 3) shaped."Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/78rethink the info command2022-01-28T17:10:34ZEva Schifferrethink the info commandRight now the info command just gives you a list of available variables in a file. We can pretty easily organize a lot more basic information about the file contents (without actually looking at the data).
- show file info (dimensions,...Right now the info command just gives you a list of available variables in a file. We can pretty easily organize a lot more basic information about the file contents (without actually looking at the data).
- show file info (dimensions, global attributes)
- show variable info (size/type, dims, attributes)
This will need to be implemented/tested for all the types of files Glance supports:
- [x] hdf
- [x] netCDF (with and without groups)
- [x] h5
- [ ] AERI
- [x] GeoTIFFEva SchifferEva Schiffer