UW-Glance issueshttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues2024-01-11T14:28:06Zhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/97Persistent memory leak issues2024-01-11T14:28:06ZEva SchifferPersistent memory leak issuesKatie Pitts from NOAA reports that Glance is still having persistent, problematic memory issues:
> I'm reopening this thread just to say that I continue to have memory errors when running Glance on directories of GOES-R imagery files th...Katie Pitts from NOAA reports that Glance is still having persistent, problematic memory issues:
> I'm reopening this thread just to say that I continue to have memory errors when running Glance on directories of GOES-R imagery files that contain channel 2 datasets. Once my EC2 machine's memory was increased from 64GB to 128GB I was able to run through the entire set of files in my directory without error. I was watching the top statistics in the terminal while it was running, and it appears that memory is not always released when moving from one file comparison to a new file comparison, which could be why it keeps dying once it gets to a larger channel 2 file in the list. At one point it was using a whopping 111GB of memory. I attached the traceback of the memory error when the run dies, as well as at the bottom of that text file a runtime warning that showed up on the Glance run that finally made it through all the files.
> I'll note that since I last emailed I have installed the latest version of Glance using the latest installation recommendation of doing so using MambaForge.
> Since we've found a workaround of increasing the EC2 machine's memory when needed to run these Glance reports, we're okay for now. I'm mainly bringing this to your attention so you can address memory handling in any future version of Glance.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/95More detailed histograms when only a small amount of data is outliers2023-03-17T18:34:36ZEva SchifferMore detailed histograms when only a small amount of data is outliersThis is a suggestion from Rob Levy.
> Where, there are big differences at a few points, but 99% of points are within ±0.01. This is where signed(exponent(logged(differences)))) would be extremely helpful (your logit scale?)
>
> X = ...This is a suggestion from Rob Levy.
> Where, there are big differences at a few points, but 99% of points are within ±0.01. This is where signed(exponent(logged(differences)))) would be extremely helpful (your logit scale?)
>
> X = Max(Absolute difference)
> A = Min(absolute difference) that is not zero
> Range = log(X) – log(A)
> Divvy up range into 25 (or fewer is probably okay) intervals of equal log
> Final plot will have 25 intervals on left (<0.0), 25 intervals on right (>0.0), plus one with exactly 0.0.
> Now you can have histogram!Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/94Allow custom colormaps2023-03-17T18:31:14ZEva SchifferAllow custom colormapsGive the user some way of setting up custom colormaps in the configuration file. These colormaps would apply to the main variable data, not the comparison plots.Give the user some way of setting up custom colormaps in the configuration file. These colormaps would apply to the main variable data, not the comparison plots.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/93Configure plots of the main data to appear in log scale2023-03-17T18:34:57ZEva SchifferConfigure plots of the main data to appear in log scaleFor some types of data it would be useful for users to be able to plot them with log scale.
Rob also asked: "It looks like one can make histograms log scaled (in the GUI), It would just be awesome to have “negative” log (e.g. absolute v...For some types of data it would be useful for users to be able to plot them with log scale.
Rob also asked: "It looks like one can make histograms log scaled (in the GUI), It would just be awesome to have “negative” log (e.g. absolute value in log space, but negative sign to plot)."
(This second part may get deferred to a different issue later, but I want to note it here so I don't forget.)Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/92More clear color bar for the differences2022-08-31T21:47:45ZEva SchifferMore clear color bar for the differencesIt would be good to use a colorbar that more clearly highlights the differences in the difference plots. So something that has a clear turning point in the middle that we can place 0 difference at. Rob Levy suggested RedBu, the Spectral ...It would be good to use a colorbar that more clearly highlights the differences in the difference plots. So something that has a clear turning point in the middle that we can place 0 difference at. Rob Levy suggested RedBu, the Spectral Rainbow (with yellow as 0), or something similar.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/90data reversing filter2022-08-17T15:31:12ZEva Schifferdata reversing filterAdd a filter that will allow you to reverse the data along one index of a multi-dimensional set.Add a filter that will allow you to reverse the data along one index of a multi-dimensional set.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/89Problem saving images when using threads on a Mac2022-08-17T16:29:51ZEva SchifferProblem saving images when using threads on a MacI have some notes that say Glance is now unable to save images when you use multi-threading on a Mac (there was a warning before when this was a problem and now there is none).
I need to reinvestigate this and either warn the user or d...I have some notes that say Glance is now unable to save images when you use multi-threading on a Mac (there was a warning before when this was a problem and now there is none).
I need to reinvestigate this and either warn the user or disallow the setting in the appropriate environment.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/87Glance Unit Tests2022-03-07T21:22:59ZEva SchifferGlance Unit TestsGlance doesn't have unit testing built into it. Some parts of the program are going to be very hard to test (because it's got a lot of file access and complex output) but I want to include more unit tests where I can to cut down on the c...Glance doesn't have unit testing built into it. Some parts of the program are going to be very hard to test (because it's got a lot of file access and complex output) but I want to include more unit tests where I can to cut down on the chances things will break and not be detected when I make changes.
I'll put in some more notes and detailed plans once I've done some evaluation of where it will be least painful to start this effort.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/82Test and maintain the AERI part of the io library2022-01-28T17:13:28ZEva SchifferTest and maintain the AERI part of the io libraryI need to get set up to test and maintain the AERI part of the io library. Ray originally wrote the class for this type of file and I never got myself setup to test it.
As a side thing, I also need to revise the display_string provided...I need to get set up to test and maintain the AERI part of the io library. Ray originally wrote the class for this type of file and I never got myself setup to test it.
As a side thing, I also need to revise the display_string provided by the aeri class. Right now it's very bare bones.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/81Consider revising the io module to use Rasterio instead of gdal2022-01-26T22:06:28ZEva SchifferConsider revising the io module to use Rasterio instead of gdalDave recommended that the Rasterio python library may be better maintained and more pythonic, so at some point I should look into how much work it will be to move to using that instead of gdal to handle our GeoTIFFs.
For reference:
[The...Dave recommended that the Rasterio python library may be better maintained and more pythonic, so at some point I should look into how much work it will be to move to using that instead of gdal to handle our GeoTIFFs.
For reference:
[The Rasterio docs](https://rasterio.readthedocs.io/en/latest/)Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/80loading attributes in some files with complex group structure fails2022-01-20T21:12:42ZEva Schifferloading attributes in some files with complex group structure failsGlance is able to open several types of files with hierarchical groups. In the gui Glance seems to be unable to load variable attributes for files that have their variables inside more complex group structures.
I've seen this bug so fa...Glance is able to open several types of files with hierarchical groups. In the gui Glance seems to be unable to load variable attributes for files that have their variables inside more complex group structures.
I've seen this bug so far with an hdf4 file and an h5 file. It seems to work ok in the netcdf file I have with groups.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/74Investigate Plotting across the anti-meridian2023-03-17T18:48:15ZEva SchifferInvestigate Plotting across the anti-meridianMy tests with the GUI suggest that I am probably providing the correct coordinates but when I try to plot across the anti-meridian in cartopy things get decidedly weird. I only have a couple of very old test cases that I can use to test ...My tests with the GUI suggest that I am probably providing the correct coordinates but when I try to plot across the anti-meridian in cartopy things get decidedly weird. I only have a couple of very old test cases that I can use to test this at the moment, so I should spend some time making a simpler example data and testing if I can plot it correctly or if cartopy always makes it look super wrong.
The file I've been using for testing:
[grid_48hr_20120129.nc](/uploads/9ecc9019d9ffdafc68f5b27e475004c8/grid_48hr_20120129.nc)https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/70Highlight differing attributes2021-11-16T17:45:24ZEva SchifferHighlight differing attributesIt would be helpful if Glance could somehow highlight which attributes that are present in both files have different values. Right now we show both global and variable attributes on the reports (and in the GUI), but the user just has to ...It would be helpful if Glance could somehow highlight which attributes that are present in both files have different values. Right now we show both global and variable attributes on the reports (and in the GUI), but the user just has to look at them to see if they're the same.https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/69Variable comparison where data sizes don't match2021-11-16T17:44:10ZEva SchifferVariable comparison where data sizes don't matchAt the moment Glance can't compare variables with mismatching data sizes. Obviously there's a limit to how much you can compare when you can't match data points up, but it would be nice to still have some sort of variable report/stats wi...At the moment Glance can't compare variables with mismatching data sizes. Obviously there's a limit to how much you can compare when you can't match data points up, but it would be nice to still have some sort of variable report/stats with information about the two data sets. This might include stuff like their individual min/max values, other stats that don't require matching data, and plots of each individual set of data (much like the plots in the inspection variable report, but for both files).https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/63Test Map Projection Selection2021-11-16T16:45:15ZEva SchifferTest Map Projection SelectionRight now Glance only offers one map projection. Do some tests to see if allowing users to select the map projection in the config file or in the gui is practical or useful with cartopy.Right now Glance only offers one map projection. Do some tests to see if allowing users to select the map projection in the config file or in the gui is practical or useful with cartopy.https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/62Allow the lon/lat to be loaded from a different file in the GUI2021-11-16T16:37:56ZEva SchifferAllow the lon/lat to be loaded from a different file in the GUIRight now if a file doesn't have longitude and latitude information in it you can't plot it on a map in the GUI. Add controls and support on the settings tab to let the user load the navigation from a different file (probably this would ...Right now if a file doesn't have longitude and latitude information in it you can't plot it on a map in the GUI. Add controls and support on the settings tab to let the user load the navigation from a different file (probably this would need to be able to differentiate for the A and B nav loading from different places).https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/61Teach Glance about valid ranges2021-11-16T17:24:20ZEva SchifferTeach Glance about valid rangesRight now Glance mostly ignores any valid range information in the attributes of a file (it will show you the attributes, but it analyzes the data without regard for whether it's in the valid range or not). It would be good to teach Glan...Right now Glance mostly ignores any valid range information in the attributes of a file (it will show you the attributes, but it analyzes the data without regard for whether it's in the valid range or not). It would be good to teach Glance about valid ranges so it can take them into account and give warnings about data outside the range.
- [ ] teach Glance how to get valid range information from file attributes
- [ ] give Glance some keys to get valid range information from config files (valid_min, valid_max, valid_min_b, valid_max_b, etc.)
- [ ] attach valid range information to data inside Glance
- [ ] emit some sorts of warnings when there is data outside the valid range
- [ ] add information to reports about the details of the data that is outside the valid range
- [ ] change how plots handle data outside the valid range (special plot to show it, don't put it in the main plots)
I will add more tasks if I think of them.
**Note: Once this ticket is completed, revisit the idea of possibly moving epsilon percent to be a percentage of the valid range instead of it's current meaning.**Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/45Glance sometimes terminates with an error code when nothing is wrong when run...2020-10-23T18:52:00ZEva SchifferGlance sometimes terminates with an error code when nothing is wrong when run on the command line with a config fileThis came from Mike Walters. I haven't seen this one in my command line tests with a config file, so there may be some subtle issue with how forking is working differently (if I can't find a way to reproduce on my machine I should do som...This came from Mike Walters. I haven't seen this one in my command line tests with a config file, so there may be some subtle issue with how forking is working differently (if I can't find a way to reproduce on my machine I should do some tests on SSEC servers). Mike said:
> Glance terminates abnormally, with the message "Aborting". This occurs when I run reportGen on the command line with a configuration file, which is how I normally run Glance. As far as I can tell, Glance generates the report and performs all of the requested operations successfully. The exit status should be 0 for success, but instead it's 134, which indicates failure. This is a bit of a problem because our cluster software tries to rerun failed jobs, so Glance jobs get run over and over. I can create a workaround if necessary, but that will have side effects. Anyway, I tracked the issue back to the os.fork() and os.waitpid() functions in the plot.py module. I'm not very familiar with forking in Python, so I wasn't able to narrow it down more than that.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/43Add control for details of the difference plots to the configuration file2021-11-11T15:08:16ZEva SchifferAdd control for details of the difference plots to the configuration fileRequest from Michael Walters at NOAA:
"FYI, some members of our group would like to be able to customize the range and color palette for the difference plots. That doesn't appear to be possible right now, because the colorMap and dataRa...Request from Michael Walters at NOAA:
"FYI, some members of our group would like to be able to customize the range and color palette for the difference plots. That doesn't appear to be possible right now, because the colorMap and dataRanges options aren't passed along to the mappedPlottingFunction (line 424 of plotcreatefns.py)."
Note from Eva:
Those range and color options are specifically meant to apply to the main data plots. Instead of repurposing them, We should find a way to add control of the difference plot settings specifically.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/40Geoplotting large HDF data with cartopy causes crash2021-11-09T16:15:43ZAlan De SmetGeoplotting large HDF data with cartopy causes crash(This is part of #23 Use cartopy instead of basemap._
Eva has some test data that causes glance to crash when trying to geoplot some data. I've been able to reproduce it.
**TEST DATA: [40-crash.nc](/uploads/3037860ef9025fc467f752d0cf...(This is part of #23 Use cartopy instead of basemap._
Eva has some test data that causes glance to crash when trying to geoplot some data. I've been able to reproduce it.
**TEST DATA: [40-crash.nc](/uploads/3037860ef9025fc467f752d0cfd128f8/40-crash.nc)**
I'm using these libraries at the moment, all from Anaconda:
```
# packages in environment at /home/adesmet/src/glance/anaconda:
#
# Name Version Build Channel
hdf4 4.2.13 h3ca952b_2
hdf5 1.10.4 hb1b8bf9_0
libnetcdf 4.6.1 h11d0813_2
netcdf4 1.4.2 py27h808af73_0
numpy 1.16.4 py27h7e9f1db_0
numpy-base 1.16.4 py27hde5b4d6_0
pyhdf 0.10.1 py27h3a4e923_1 conda-forge
python 2.7.16 h9bab390_0
```
I'm working with UW-Glance ad359acf0991528fab048a9c0feb0006f2d80727, which is the current head of `tocartopy`.
The failure:
```console
$ glance inspectreport 40-crash.nc --outputpath=out --longitude pixel_longitude --latitude pixel_latitude ^baseline_cmask_seviri_cloud_mask$
ERROR:glance.plotcreatefns:Someone has instantiated PlottingFunctionFactory. This class is NOT meant to be instantiated.
ERROR:glance.plotcreatefns:Someone has instantiated PlottingFunctionFactory. This class is NOT meant to be instantiated.
lat type: <type 'numpy.ndarray'>
lon type: <type 'numpy.ndarray'>
ERROR:shapely.geos:TopologyException: side location conflict at -109 50
Traceback (most recent call last):
File "/home/adesmet/src/glance/anaconda/bin/glance", line 11, in <module>
load_entry_point('uwglance', 'console_scripts', 'glance')()
File "/home/adesmet/src/glance/pyglance/glance/compare.py", line 1613, in main
rc = lower_locals[args[0].lower()](*args[1:])
File "/home/adesmet/src/glance/pyglance/glance/compare.py", line 1479, in inspectReport
return inspect_library_call(clean_path(args[0]), args[1:], tempOptions)
File "/home/adesmet/src/glance/pyglance/glance/compare.py", line 601, in inspect_library_call
histRange=varRunInfo[HISTOGRAM_RANGE_KEY] if HISTOGRAM_RANGE_KEY in varRunInfo else None)
File "/home/adesmet/src/glance/pyglance/glance/plot.py", line 333, in plot_and_save_comparison_figures
makeSmall, doFork, shouldClearMemoryWithThreads, fullDPI=fullDPI, thumbDPI=thumbDPI)
File "/home/adesmet/src/glance/pyglance/glance/plot.py", line 98, in _log_spawn_and_wait_if_needed
fullDPI=fullDPI, thumbDPI=thumbDPI)
File "/home/adesmet/src/glance/pyglance/glance/plot.py", line 55, in _handle_fig_creation_task
figure_obj = child_figure_function()
File "/home/adesmet/src/glance/pyglance/glance/plotcreatefns.py", line 1455, in <lambda>
units=units_a)),
File "/home/adesmet/src/glance/pyglance/glance/figures.py", line 613, in create_mapped_figure
img_temp = maps.contourf_on_map(longitudeClean, latitudeClean, data, axes_obj, in_projection, **kwargs)
File "/home/adesmet/src/glance/pyglance/glance/graphics.py", line 263, in contourf_on_map
img_temp = axes_object.contourf(lon, lat, data, levelsToUse, transform=in_projection, **kwargs)
File "/home/adesmet/src/glance/anaconda/lib/python2.7/site-packages/cartopy/mpl/geoaxes.py", line 1397, in contourf
if col.get_paths()])
File "/home/adesmet/src/glance/anaconda/lib/python2.7/site-packages/matplotlib/collections.py", line 196, in get_datalim
paths = [transform.transform_path_non_affine(p) for p in paths]
File "/home/adesmet/src/glance/anaconda/lib/python2.7/site-packages/matplotlib/transforms.py", line 2492, in transform_path_non_affine
return self._a.transform_path_non_affine(path)
File "/home/adesmet/src/glance/anaconda/lib/python2.7/site-packages/cartopy/mpl/geoaxes.py", line 189, in transform_path_non_affine
getattr(self, 'force_path_ccw', False))
File "/home/adesmet/src/glance/anaconda/lib/python2.7/site-packages/cartopy/mpl/patch.py", line 212, in path_to_geos
collection[-1][0].contains(geom.exterior)):
File "/home/adesmet/src/glance/anaconda/lib/python2.7/site-packages/shapely/geometry/base.py", line 691, in contains
return bool(self.impl['contains'](self, other))
File "/home/adesmet/src/glance/anaconda/lib/python2.7/site-packages/shapely/predicates.py", line 18, in __call__
self._check_topology(err, this, other)
File "/home/adesmet/src/glance/anaconda/lib/python2.7/site-packages/shapely/topology.py", line 38, in _check_topology
self.fn.__name__, repr(geom)))
shapely.errors.TopologicalError: The operation 'GEOSContains_r' could not be performed. Likely cause is invalidity of the geometry <shapely.geometry.polygon.Polygon object at 0x7fbf8927a790>
```
Trying to plot interactively with `glance gui` generated similar output.Alan De SmetAlan De Smet