UW-Glance issueshttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues2021-07-30T16:18:59Zhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/5Transfer documentation2021-07-30T16:18:59ZEva SchifferTransfer documentationTransfer the Glance Documentation from the Groups page over to the Wiki associated with this repository.
The documentation is currently at: https://groups.ssec.wisc.edu/groups/goes-r/algorithm-working-group/resources-and-recipes/glance-...Transfer the Glance Documentation from the Groups page over to the Wiki associated with this repository.
The documentation is currently at: https://groups.ssec.wisc.edu/groups/goes-r/algorithm-working-group/resources-and-recipes/glance-large-dataset-comparison-tools-1
It needs a lot of revision to capture things that have changed since I was updating the documentation more regularly. It probably also needs some reorganization for brevity and clarity.
This is not a complete list of tasks, just things that I had notes about and didn't want to forget. I will probably add more tasks as I review my notes and transfer the documentation.
* [x] Review and rewrite the installation instructions
* [x] Transfer release notes
* [x] Look over the code that processes the config files and make sure all the options are captured in the documentation
* [x] Make more example plots and configuration files for reference
* [x] Examples for plotDiffs in the basic usage section
* [x] At least a handful of example config files to illustrate the config usage
* [x] Document the regex syntax for specifying variables on the command line more clearly
* ~~[ ] Change the format of the filters.extract_bit_from_packed_mask parameters to match the new format~~
* [x] Move Advanced functionality sections
* [x] Configuration Files
* [x] Built in Data Filters
* [x] (Colocating Data)
* [x] The Glance RPM building instructions
* ~~[ ] Write up details of how to use the Glance GUI~~
* [x] Example code for how to configure vector plots in a config file (on the Using Configuration Files page)
* [x] Modify the Using Configuration Files page to use the glance.constants instead of hard coded stringshttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/6Slow GUI operations should show WaitCursor2021-08-27T16:26:22ZAlan De SmetSlow GUI operations should show WaitCursorSlow GUI operations should show the wait cursor until the operation completes. Buttons to cover:
* [x] Display Data
* [x] Display Statistics
* [x] Display Plot
* [x] Load
* [x] Display Global Attrs.
This may be a bit complex, as many ...Slow GUI operations should show the wait cursor until the operation completes. Buttons to cover:
* [x] Display Data
* [x] Display Statistics
* [x] Display Plot
* [x] Load
* [x] Display Global Attrs.
This may be a bit complex, as many (all?) of the buttons are hooked up to a register/listener pattern. While calling into the listener is _probably_ blocking, it doesn't seem like something that would be guaranteed.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/4Improper scaling of ABI L1b/L2 products2017-08-24T00:02:18ZWilliam StrakaImproper scaling of ABI L1b/L2 productsJust copying the issue here in the ticketing system
File -
changeo:/data/users/wstraka/DSI_glance/OR_ABI-L2-DSIF-M4_G16_s20170151527081_e20170151531484_c2017015153
This is using the LI product in the above file.
For the soun...Just copying the issue here in the ticketing system
File -
changeo:/data/users/wstraka/DSI_glance/OR_ABI-L2-DSIF-M4_G16_s20170151527081_e20170151531484_c2017015153
This is using the LI product in the above file.
For the soundings file from G-16, at pixel x,y = 612, 83 (highlighted in both attached images. Note: HDFview is 0
based, hence why it is off by one). The (signed) value is -8835. scale factor = 7.6301E-4, add offset = -10.0.
So, following the steps to unscale the data, per Harris's writeup
1). _Unsigned = True, so convert to unscaled value:x = (-8835) + 65535 = 56700
2). Apply scale factor/add offset (mx + b) = 33.2627.
However, as can be seen in the plot dump in glance, that point is being dumped out as 23.26343. I wonder if this
has to deal with how the _Unsigned flag is being applied within Glance![Glance_LI_dump](/uploads/47b59ff1eb9fabba4fd41f33b3ff43c6/Glance_LI_dump.png)![Glance_LI_dump](/uploads/735d7ee87e3bb2b3fed58ffd4b230bf0/Glance_LI_dump.png)![HDFview_LI_dump](/uploads/93e98715e248a9ff058e2b00ea520709/HDFview_LI_dump.png)Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/3Change matching behavior of variable names specified at the command line2017-08-24T00:02:18ZGraeme MartinChange matching behavior of variable names specified at the command lineAs of version 0.3.2, when I specify a variable name at the command line after the filenames, it will match any variable that starts with that string. For example, the string
```
solar_array_current
```
will match both of these variables:...As of version 0.3.2, when I specify a variable name at the command line after the filenames, it will match any variable that starts with that string. For example, the string
```
solar_array_current
```
will match both of these variables:
```
solar_array_current
solar_array_current_channel_index_label
```
In the situation above, as far as I know there is no way to specify that the first variable should be compared and the second should not. In this particular case it was an issue because the software crashed on the "_label" variable (see issue #2), but in general it would be nice to have a way to reliable select individual variables at the command line.
Suggest that variable names should not be treated as roots, and instead multiple variables can be specified using a standard matching syntax, with patterns provided in quotes to prevent shell expansion.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/2crash when processing NetCDF string array variables2017-08-24T00:02:18ZGraeme Martincrash when processing NetCDF string array variablesglance, version 0.3.2
Glance crashed when processing variables that are arrays of strings. The other variables in the file worked fine, so I was able to generate a report by omitting the string variables.
This is not holding me up, but...glance, version 0.3.2
Glance crashed when processing variables that are arrays of strings. The other variables in the file worked fine, so I was able to generate a report by omitting the string variables.
This is not holding me up, but it would be nice to have it fixed in a future version. Would be nice to be able to compare the strings, but in the near term even something like printing a warning and skipping the string variables would be an improvement.
Reduced it to a simple case of comparing a variable to itself. The test case is on the "apollo" lustre drive.
```
glance reportGen --nolonlat /apollo/ait/glance_test_cases/20170307_seiss/OR_SEIS-L1b-EHIS_G16_s20170511738060_e20170511738060_c20170511743066.nc /apollo/ait/glance_test_cases/20170307_seiss/OR_SEIS-L1b-EHIS_G16_s20170511738060_e20170511738060_c20170511743066.nc element_label
Traceback (most recent call last):
File "/home/graemem/opt/glance.env/bin/glance", line 9, in <module>
load_entry_point('uwglance==0.3.2', 'console_scripts', 'glance')()
File "/home/graemem/opt/glance.env/lib/python2.7/site-packages/uwglance-0.3.2-py2.7.egg/glance/compare.py", line 1487, in main
rc = lower_locals[args[0].lower()](*args[1:])
File "/home/graemem/opt/glance.env/lib/python2.7/site-packages/uwglance-0.3.2-py2.7.egg/glance/compare.py", line 1301, in reportGen
return reportGen_library_call(a_path, b_path, args[2:], tempOptions)
File "/home/graemem/opt/glance.env/lib/python2.7/site-packages/uwglance-0.3.2-py2.7.egg/glance/compare.py", line 764, in reportGen_library_call
fileDescriptionForDisplay = "file A")
File "/home/graemem/opt/glance.env/lib/python2.7/site-packages/uwglance-0.3.2-py2.7.egg/glance/load.py", line 262, in load_variable_data
raise exceptionToRaise
ValueError: Unable to retrieve element_label data. The variable name may not exist in this file or an error may have occured while attempting to access the data. Details of file access error observed: could not convert string to float: B
```Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/1Variables with all missing values are reported as a mismatch2017-08-24T00:02:18ZGraeme MartinVariables with all missing values are reported as a mismatchglance, version 0.3.2
Observed the problem when generating reports comparing two NetCDF files that contained several variables with all-missing data in both files. The report showed these variables as a mismatch.
Reduced the problem to...glance, version 0.3.2
Observed the problem when generating reports comparing two NetCDF files that contained several variables with all-missing data in both files. The report showed these variables as a mismatch.
Reduced the problem to a simple case, comparing an all-missing variable to itself.
glance reportGen --nolonlat /apollo/ait/glance_test_cases/20170307_seiss/OR_SEIS-L1b-EHIS_G16_s20170511738060_e20170511738060_c20170511743066.nc /apollo/ait/glance_test_cases/20170307_seiss/OR_SEIS-L1b-EHIS_G16_s20170511738060_e20170511738060_c20170511743066.nc CNO5MinuteDifferentialFluxes
The test data is on the apollo lustre drive:
/apollo/ait/glance_test_cases/20170307_seissEva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/7empty data causes a crash when trying to use a shared range2018-03-20T18:52:38ZEva Schifferempty data causes a crash when trying to use a shared rangeI got the following report from Mike Walters at NOAA. I still need to try to reproduce this locally.
> First, I've noticed that when comparing variables with all fill/missing
> values, a fatal error can occur when this setting is used:...I got the following report from Mike Walters at NOAA. I still need to try to reproduce this locally.
> First, I've noticed that when comparing variables with all fill/missing
> values, a fatal error can occur when this setting is used:
>
> settings['useSharedRangeForOriginal'] = True
>
> The error message looks like this:
>
Traceback (most recent call last):
File "/data/data089/usr/compilers/python-2.7/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/data/data089/usr/compilers/python-2.7/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/data/data089/usr/compilers/python-2.7/lib/python2.7/site-packages/uwglance-0.3.3.2-py2.7.egg/glance/compare.py", line 1538, in <module>
sys.exit(main())
File "/data/data089/usr/compilers/python-2.7/lib/python2.7/site-packages/uwglance-0.3.3.2-py2.7.egg/glance/compare.py", line 1532, in main
rc = lower_locals[args[0].lower()](*args[1:])
File "/data/data089/usr/compilers/python-2.7/lib/python2.7/site-packages/uwglance-0.3.3.2-py2.7.egg/glance/compare.py", line 1323, in reportGen
return reportGen_library_call(a_path, b_path, args[2:], tempOptions)
File "/data/data089/usr/compilers/python-2.7/lib/python2.7/site-packages/uwglance-0.3.3.2-py2.7.egg/glance/compare.py", line 923, in reportGen_library_call
units_b= varRunInfo[VAR_UNITS_B_KEY] if VAR_UNITS_B_KEY in varRunInfo else None,
File "/data/data089/usr/compilers/python-2.7/lib/python2.7/site-packages/uwglance-0.3.3.2-py2.7.egg/glance/plot.py", line 317, in plot_and_save_comparison_figures
histRange=histRange
File "/data/data089/usr/compilers/python-2.7/lib/python2.7/site-packages/uwglance-0.3.3.2-py2.7.egg/glance/plotcreatefns.py", line 1101, in create_plotting_functio
ns
shouldUseSharedRangeForOriginal)
File "/data/data089/usr/compilers/python-2.7/lib/python2.7/site-packages/uwglance-0.3.3.2-py2.7.egg/glance/plotcreatefns.py", line 95, in _make_shared_range
data_b=bData, valid_b_mask=goodInBMask)
File "/data/data089/usr/compilers/python-2.7/lib/python2.7/site-packages/uwglance-0.3.3.2-py2.7.egg/glance/figures.py", line 81, in _make_range
minVal -= offset_to_range
TypeError: unsupported operand type(s) for -=: 'NoneType' and 'float'
>
> I've been setting 'useSharedRangeForOriginal' to False to work around
> the problem, but that can sometimes make the resulting plots harder to read.https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/8Glance sometimes choses the wrong config file when there are multiple valid c...2018-03-20T19:03:03ZEva SchifferGlance sometimes choses the wrong config file when there are multiple valid config files in a directoryI learned about this bug from Mike Walters at NOAA. It may have also been biting Hong intermittently, but we were never able to reproduce her problems consistently to figure out what was going on. If Mike's analysis is right, it should o...I learned about this bug from Mike Walters at NOAA. It may have also been biting Hong intermittently, but we were never able to reproduce her problems consistently to figure out what was going on. If Mike's analysis is right, it should only affect people who have multiple config files in the directory where the one they're trying to use is stored, and even then it will only happen intermittently. He noticed it because he was running large batches of comparisons through HTCondor and was getting inconsistent results on which variables Glance would analyze in a given run.
I should note, Glance was still copying the requested config file (rather than the one that was actually loaded) into the report directory, so it was difficult to see what was actually happening. On top of fixing the bug, I need to add better debug logging to the code that selects the variables based on the config file.
Here is Mike's final summary of his investigation:
> The problem lies on on lines 374-375 of config_organizer.py:
>
> glanceRunConfig = imp.load_module(fileBaseName, file(requestedConfigFile, 'U'),
> filePath, ('.py' , 'U', 1))
>
> The third argument passed to the imp.load_module function is filePath. Earlier, on line 365, filePath is set to the directory where the config file is located:
>
> (filePath, fileName) = os.path.split(requestedConfigFile)
>
> The Glance documentation says that the third argument is supposed to be the "pathname" of the module file. The wording is rather vague, but I'm fairly certain that this is supposed to be the full path of the module file, including the file name, and not just the directory name. This only became clear to me when I looked at examples I found online, like this one:
>
> http://effbot.org/librarybook/imp.htm
>
> When only the directory path is given, Python is able to guess the correct file name most of the time. Sometimes it guesses wrong though, and loads the wrong config file. When the wrong config file is loaded, the variable list doesn't match the input files, which explains why some or all of the variables were missing from the final report. I figured this out by adding debug statements to show what was actually being loaded. I would imagine that most people only have a few config files in their directory, which probably makes it easier to find the right config. Hence, they never encounter the bug. On the other hand, I have 70+ config files in my directory and I'm running thousands of Glance jobs, so it's not surprising that I ran into the bug.
>
> The problem can be fixed by changing the third argument of imp.load_module from filePath to requestedConfigFile:
>
> glanceRunConfig = imp.load_module(fileBaseName, file(requestedConfigFile, 'U'),
> requestedConfigFile, ('.py' , 'U', 1))
>
> Making this change solved the problem for me 100% of the time (so far, anyway).https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/9Support --nolatlon as synonym for --nolonlat2018-11-29T19:39:35ZAlan De SmetSupport --nolatlon as synonym for --nolonlatIt would be convenient for my laziness if `--nolatlon` was handled as a synonym for `--nolonlat`. I somehow always manage to pick the wrong one.It would be convenient for my laziness if `--nolatlon` was handled as a synonym for `--nolonlat`. I somehow always manage to pick the wrong one.https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/10More information than just "Unable to create plot."2018-11-29T22:15:59ZAlan De SmetMore information than just "Unable to create plot."In `_handle_fig_creation_function`, if `child_figure_function()` returns `None`, a warning is omitted in the form:
```
LOG.warn("Unable to create plot.")
```
When comparing all variables in two files, I frequently generate a lot of the...In `_handle_fig_creation_function`, if `child_figure_function()` returns `None`, a warning is omitted in the form:
```
LOG.warn("Unable to create plot.")
```
When comparing all variables in two files, I frequently generate a lot of these warnings. Unfortunately, there is no information to tell me _which_ variables had problems. Ideally more information would be made available.https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/11numpy(?) deprecation warnings in glance output2018-11-29T22:55:15ZAlan De Smetnumpy(?) deprecation warnings in glance outputGlance version 0.3.5, from Git.
```
$ glance reportgen --nolonlat --outputpath tmp glance-problems.nc glance-problems.nc --quiet ^x$
/home/adesmet/src/glance/UW-Glance/pyglance/glance/io.py:440: FutureWarning: comparison to `None` will ...Glance version 0.3.5, from Git.
```
$ glance reportgen --nolonlat --outputpath tmp glance-problems.nc glance-problems.nc --quiet ^x$
/home/adesmet/src/glance/UW-Glance/pyglance/glance/io.py:440: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
missing_mask[scaled_data_copy == missing_val] = True
/home/adesmet/src/glance/UW-Glance/pyglance/glance/io.py:440: VisibleDeprecationWarning: using a boolean instead of an integer will result in an error in the future
missing_mask[scaled_data_copy == missing_val] = True
```
Those seem bad?
The file in question: [glance-problems.nc.gz](/uploads/4c8763c229164890340a3b8df8adb550/glance-problems.nc.gz). You'll need to uncompress it.https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/12Glance hates Unicode (UTF-8?)2021-08-23T21:10:38ZAlan De SmetGlance hates Unicode (UTF-8?)```
glance reportgen --nolonlat --outputpath tmp glance-problems.nc glance-problems.nc --quiet ^x$
Traceback (most recent call last):
File "/home/adesmet/bin/glance", line 11, in <module>
load_entry_point('uwglance', 'console_scrip...```
glance reportgen --nolonlat --outputpath tmp glance-problems.nc glance-problems.nc --quiet ^x$
Traceback (most recent call last):
File "/home/adesmet/bin/glance", line 11, in <module>
load_entry_point('uwglance', 'console_scripts', 'glance')()
File "/home/adesmet/src/glance/UW-Glance/pyglance/glance/compare.py", line 1607, in main
rc = lower_locals[args[0].lower()](*args[1:])
File "/home/adesmet/src/glance/UW-Glance/pyglance/glance/compare.py", line 1388, in reportGen
return reportGen_library_call(a_path, b_path, args[2:], tempOptions)
File "/home/adesmet/src/glance/UW-Glance/pyglance/glance/compare.py", line 1014, in reportGen_library_call
globalAttrs=globalAttrInfo,)
File "/home/adesmet/src/glance/UW-Glance/pyglance/glance/report.py", line 173, in generate_and_save_summary_report
_make_and_save_page((outputPath + "/" + reportFileName), 'mainreport.txt', **kwargs)
File "/home/adesmet/src/glance/UW-Glance/pyglance/glance/report.py", line 41, in _make_and_save_page
fileToWrite.write(tempTemplate.render(**kwargs))
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2018' in position 6913: ordinal not in range(128)
```
(I've omitted some deprecation warnings that I think are a different issue. They're in #11.)
The file in question: [glance-problems.nc.gz](/uploads/4c8763c229164890340a3b8df8adb550/glance-problems.nc.gz). You'll need to uncompress it.
`u'\u2018'` is `‘` (left single quotation mark), and indeed it's present in the file's global attributes. Specifically the `summary` attribute is:
```
"The Land Surface (Skin) Temperature product consists of pixels containing the skin temperatures for each ‘clear’ or ‘probably clear’ land surface pixel. This product is generated from a regression algorithm that linearly combines ABI surface emissivity data, brightness temperature, and brightness temperature differences derived from top of atmosphere radiances from ABI bands with wavelengths 11.2 and 12.3 um. Product data is generated both day and night."
```
There are "smart" single quotes around "clear" and "probably clear". It may be wrong for those characters to be in the file in the first place, but ideally glance would handle it anyway.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/13Use same range for coloring paired original input in reports2018-09-19T19:50:04ZAlan De SmetUse same range for coloring paired original input in reportsWhen generating a report between two files, the pairs of "Original Data" images will use different ranges for the colors if the data ranges are different. This can make it difficult to visually compare the two.
For example, today:
<a h...When generating a report between two files, the pairs of "Original Data" images will use different ranges for the colors if the data ranges are different. This can make it difficult to visually compare the two.
For example, today:
<a href="/uploads/1c6e4f951612e27338145e7f2549428a/orig-data-example.png"><img width="800" src="/uploads/1c6e4f951612e27338145e7f2549428a/orig-data-example.png"></a>
I would expect the image on the right to map 0 (which is _all_ of the data it has) to blue, not pale green.
Perhaps the two images should use the same range, the smallest range that includes both files ranges.
(This may also make sense when plotting the original data in any case when you have two files, although it may be surprising in the GUI.)https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/14Use actual ✔ and ✘ instead of graphics of them in reports2018-11-29T22:00:35ZAlan De SmetUse actual ✔ and ✘ instead of graphics of them in reportsCurrently reports use pass.gif and fail.gif to indicate variable which did or did not match. The images in question can be reproduced using just HTML/CSS. This slightly simplifies things, makes them easier to change, and makes them somet...Currently reports use pass.gif and fail.gif to indicate variable which did or did not match. The images in question can be reproduced using just HTML/CSS. This slightly simplifies things, makes them easier to change, and makes them something you can easily copy to other places.https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/16Tighten up the GUI layout2021-04-08T17:38:51ZAlan De SmetTighten up the GUI layoutTighten up the GUI layout:
* Only columns that make sense to stretch are allowed to stretch. This frees up more space for fields that can contain a lot of information.
* Short tables (settings and filters) are compacted to the top.
* The...Tighten up the GUI layout:
* Only columns that make sense to stretch are allowed to stretch. This frees up more space for fields that can contain a lot of information.
* Short tables (settings and filters) are compacted to the top.
* The section labels on the filters tab now overhang their settings, instead of being next to them.
Work is in !1
| Before | After |
| ------ | ----- |
|![basic-before](/uploads/6dd6bd03ddbeb813dcd026ee026214e1/basic-before.png)|![basic-after](/uploads/48168822b231d8682d13343776ab7a61/basic-after.png)|
|![filter-before](/uploads/f50eb6cef0b0497a1fae9271fec2bb1a/filter-before.png)|![filter-after](/uploads/ddf2eb5dcd3b9789e48c6f7f4a5d5eec/filter-after.png)|
Warning: If you merge both this request and !2, there is a small conflict that causes the "A simple filtering" on the filters tab to not overhang properly. To fix it, edit GlanceGUIView._add_simple_filter_controls, around line 438, and change
```
# label the specific section
tempLabel = add_right_aligned_label(grid_layout, str(file_prefix)+" simple filtering:", current_row)
```
to
```
# label the specific section
tempLabel = QtGui.QLabel(str(file_prefix)+" simple filtering:")
```
(The merge doesn't see a conflict, and is stomping !1 to apply !2; this above just restores the line from !1.)https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/17No output if variable missing2021-11-23T15:46:49ZAlan De SmetNo output if variable missingIf a variable is not present in one or both of the input files, Glance doesn't emit any message at all, silently skipping it. This is true for `glance stats` and I suspect other commands as well.
If the variable is only present in one o...If a variable is not present in one or both of the input files, Glance doesn't emit any message at all, silently skipping it. This is true for `glance stats` and I suspect other commands as well.
If the variable is only present in one of the two files, I would expect an warning message something like:
```
Skipping VARNAME: present in file1.nc but not file2.nc
```
If the variable isn't in either:
```
Skipping VARNAME: not present in either file.
```
The same for `glance inspectStats` when the only file does not contain the named variable.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/18Pretty up report format2021-04-08T17:36:09ZAlan De SmetPretty up report formatBecause I'm a bad person, this proposed merge mixes several things up:
- Force font to be sans-serif. (Most people probably won't see any change!)
- Display statistics in a table instead of a list.
- Show statistics that are calculated ...Because I'm a bad person, this proposed merge mixes several things up:
- Force font to be sans-serif. (Most people probably won't see any change!)
- Display statistics in a table instead of a list.
- Show statistics that are calculated for each file on a single line instead of two different lines.
- Display integers with thousands separators (so instead of "1234567" you'll see "1,234,567")
- Zebra stripe tables
Merge is available as !9
<table>
<tr>
<th>Before</th>
<th>After</th>
</tr>
<tr>
<td>
<a href="/uploads/00f098dd733dc34bac7579e73952ccc6/before.png"><img src="/uploads/00f098dd733dc34bac7579e73952ccc6/before.png" width="300"></a>
</td>
<td>
<a href="/uploads/ad9ca03edbd9dfe6f2cabf1283a9c3f8/after.png"><img src="/uploads/ad9ca03edbd9dfe6f2cabf1283a9c3f8/after.png" width="300"></a>
</td>
</tr>
</table>https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/19RuntimeWarning with stats when stddev==0.02018-11-29T19:11:11ZAlan De SmetRuntimeWarning with stats when stddev==0.0`glance stats` and other paths that generate stats will generate a warning if the standard deviation for a variable is 0.0. (I believe only if the _second_ file has a stddev of 0.0.) This is because the Pearson R calculation is not valid...`glance stats` and other paths that generate stats will generate a warning if the standard deviation for a variable is 0.0. (I believe only if the _second_ file has a stddev of 0.0.) This is because the Pearson R calculation is not valid in that case.
The results are correct, but the warning is undesirable noise.
```
$ glance stats output.nc output.nc
/usr/lib/python2.7/dist-packages/scipy/stats/stats.py:3003: RuntimeWarning: invalid value encountered in float_scalars
r = r_num / r_den
```
[Reference output.nc](/uploads/446d3676ed317a8d847c567dcc3f2261/output.nc). It has one variable "zeros2d", which is full of zeroes.
In theory this can happen on any value (or even a set of values), but in practice it's tricky, since floating point rounding errors can cause the standard deviation to be larger than 0.0.https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/20Data with 0 range generates warnings in reportGen2018-11-29T19:51:06ZAlan De SmetData with 0 range generates warnings in reportGenIf a variable has a range of 0 (that is, all the same value), when we try to plot it `min_value == max_value`, which causes matplotlib to re-set the range to -0.1 through 0.1. More importantly, it emits useless warnings. We might as well...If a variable has a range of 0 (that is, all the same value), when we try to plot it `min_value == max_value`, which causes matplotlib to re-set the range to -0.1 through 0.1. More importantly, it emits useless warnings. We might as well set the range ourselves and eliminate the warnings.
```
$ glance reportgen --nolonlat --outputpath /tmp/eraseme output.nc output.nc
/usr/lib/python2.7/dist-packages/matplotlib/axes/_base.py:2787: UserWarning: Attempting to set identical left==right results
in singular transformations; automatically expanding.
left=0.0, right=0.0
'left=%s, right=%s') % (left, right))
/usr/lib/python2.7/dist-packages/matplotlib/axes/_base.py:3045: UserWarning: Attempting to set identical bottom==top results
in singular transformations; automatically expanding.
bottom=0.0, top=0.0
'bottom=%s, top=%s') % (bottom, top))
```
[Reference output.nc](/uploads/446d3676ed317a8d847c567dcc3f2261/output.nc). It has one variable "zeros2d", which is full of zeroes.https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/21Python 3 support2020-06-12T19:08:27ZAlan De SmetPython 3 supportGlance should support Python 3.x. Ideally it should support 2.recent and 3.x simultaneously.
As of November 19, 2018, [the py3port branch](https://gitlab.ssec.wisc.edu/evas/UW-Glance/tree/py3port) allows Glance to run under Python 3 and...Glance should support Python 3.x. Ideally it should support 2.recent and 3.x simultaneously.
As of November 19, 2018, [the py3port branch](https://gitlab.ssec.wisc.edu/evas/UW-Glance/tree/py3port) allows Glance to run under Python 3 and should also support Python 2. _However_, basemap doesn't support Python 3! As a result, it's basically untested beyond the most superficial `glance help inspectreport` works.
You can patch basemap with [basemap-py3-hackery.patch](/uploads/ed4be5cb28b7308b03cf30aabc73a29d/basemap-py3-hackery.patch) as a temporary workaround. The `dedent` change should be pretty safe. The `is_scalar` change may have unexpected repercussions, and until investigated should not be trusted.https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/22RuntimeWarning in reportGen: "More than 20 figures have been opened."2018-11-29T19:40:03ZAlan De SmetRuntimeWarning in reportGen: "More than 20 figures have been opened."When running `glance reportgen` on files that have a "large" number of variables that cannot be scatter plotted, you'll get this warning:
```
/usr/lib/python2.7/dist-packages/matplotlib/pyplot.py:516: RuntimeWarning: More than 20 figure...When running `glance reportgen` on files that have a "large" number of variables that cannot be scatter plotted, you'll get this warning:
```
/usr/lib/python2.7/dist-packages/matplotlib/pyplot.py:516: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam `figure.max_open_warning`).
```
The problem is that `create_complex_scatter_plot` in /pyglance/glance/figures.py around line 277 has a failure path that doesn't `plt.close` the created but unused figure.
[22-too-many-figures](https://gitlab.ssec.wisc.edu/evas/UW-Glance/compare/master...22-too-many-figures) has a one-line patch to solve this; it calls `plt.close(figure)`.https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/23Use cartopy instead of basemap2020-06-12T19:07:39ZEva SchifferUse cartopy instead of basemapRight now glance uses basemap for plotting data on a map. Unfortunately basemap is not really supported anymore, so we should move to using cartopy instead. This should also help with eventually transitioning to python 3.Right now glance uses basemap for plotting data on a map. Unfortunately basemap is not really supported anymore, so we should move to using cartopy instead. This should also help with eventually transitioning to python 3.https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/24Give Glance a shiny new icon2018-12-21T19:44:44ZAlan De SmetGive Glance a shiny new iconGlance could benefit from a shiny new icon. Proposed is an inequality sign in a magnifying glass.Glance could benefit from a shiny new icon. Proposed is an inequality sign in a magnifying glass.https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/25Add a way to display global file attributes in the GUI2021-08-23T21:11:10ZEva SchifferAdd a way to display global file attributes in the GUIRight now you can see the variable attributes in the GUI display when you load each variable, but there is no way to see the global attributes from the file. This is something people would only need sometimes, so it might be best to give...Right now you can see the variable attributes in the GUI display when you load each variable, but there is no way to see the global attributes from the file. This is something people would only need sometimes, so it might be best to give them the option of a pop up window or to put the global attributes on a different tab of the GUI.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/26Move (almost) all argument parsing logic into config_organizer.py2021-04-08T17:41:19ZAlan De SmetMove (almost) all argument parsing logic into config_organizer.pyCurrently the argument parsing logic is split between [compare.py](https://gitlab.ssec.wisc.edu/evas/UW-Glance/blob/master/pyglance/glance/compare.py) and [config_organizer.py](https://gitlab.ssec.wisc.edu/evas/UW-Glance/blob/master/pygl...Currently the argument parsing logic is split between [compare.py](https://gitlab.ssec.wisc.edu/evas/UW-Glance/blob/master/pyglance/glance/compare.py) and [config_organizer.py](https://gitlab.ssec.wisc.edu/evas/UW-Glance/blob/master/pyglance/glance/config_organizer.py). Move everything into config_organizer.py for simplicity.
Work is on the branch [argparse-cleanup](https://gitlab.ssec.wisc.edu/evas/UW-Glance/tree/argparse-cleanup) and is merge request !15.https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/27Move "Available commands in Glance" to argparse epilog2021-08-24T20:03:32ZAlan De SmetMove "Available commands in Glance" to argparse epilog`glance help` emits this:
```
Available commands in Glance:
info list information about a list of files
stats summarize the statistical differences between two files
help print ...`glance help` emits this:
```
Available commands in Glance:
info list information about a list of files
stats summarize the statistical differences between two files
help print help for a specific command or list of commands
gui start the Glance graphical user interface
plotDiffs create a set of images comparing two files
inspectReport create a report to inspect the contents of one file
colocateData colocate data from two files
reportGen create a report comparing two files
inspectStats create statistics summary of one file
```
It is automatically appended to `glance typotypo` or bare `glance`. It does _not_ get appended to `glance --help`. These should be unified, ideally by moving it into the argparse logic.
1. [x] In compare.py's main(), move the logic for collecting the commands _above_ the call to `config_organizer.parse_arguments()`
2. [x] Move logic for generating help message out of compare.py's main's help() into generating a string to pass to `config_organizer.parse_arguments()`
3. [x] Have config_organizer.parse_arguments() pass that string in as an epilog. (This will probably require `formatter=argparse.RawDescriptionHelpFormatter` in the call to `argparse.ArgumentParser`! Otherwise the text will be reflowed!)
4. [ ] Modify compare.py's main's help() to call options.print_help() for the `print_all_summary` case. (This may require some cleverness, as `help()` is defined _before_ we get options back!)
In the long run we'll want to do more clever things with subparsers, potentially eliminating the need for this logic at all. But this is a step forward.https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/28Make the command a first tier argument2021-08-24T19:47:07ZAlan De SmetMake the command a first tier argumentThe command ("info", "reportGen", etc) should be a first tier argument, not just squirreled away as the first entry in arguments (or options.misc after #26).
I _think_ it should be as straightforward as adding
```
parser.add_argume...The command ("info", "reportGen", etc) should be a first tier argument, not just squirreled away as the first entry in arguments (or options.misc after #26).
I _think_ it should be as straightforward as adding
```
parser.add_argument('command', metavar="COMMAND", help="command to invoke", nargs=1, type=str.lower)
```
(The `type=str.lower` makes it case insensitive.)
_This will cause glance to automatically fail with an error if no command is provided._ The error will look something like:
```
usage: [-h] COMMAND
glance: error: too few arguments
```
I think that's a feature.
We'd need to modify
```
parser.add_argument('misc', metavar='COMMAND [FILE [VARIABLE]]+', nargs='*')
```
to remove the `COMMAND ` bit.
The logic in main in compare.py should change from
```
# if what the user asked for is not one of our existing functions, print the help
if ((not args) or (args[0].lower() not in lower_locals)):
if options.version:
return 0
options.print_help()
help()
return 9
else:
# call the function the user named, given the arguments from the command line, lowercase the request to ignore case
rc = lower_locals[args[0].lower()](*args[1:])
return 0 if rc is None else rc
```
to
```
# if what the user asked for is not one of our existing functions, print the help
if ((options.command is None or options.command == "") or (options.command.lower() not in lower_locals)):
if options.version:
return 0
options.print_help()
help()
return 9
else:
# call the function the user named, given the arguments from the command line, lowercase the request to ignore case
rc = lower_locals[options.command.lower()](*args)
return 0 if rc is None else rc
```
For bonus points, have `main()` pass a list of valid commands in to `parse_arguments()`, and use that to set a list of `choices=` so argparse can automatically filter invalid commands.https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/29Show defaults in --help where appropriate2021-04-08T17:42:42ZAlan De SmetShow defaults in --help where appropriate"glance --help" should include defaults for options where it's meaningful.
It's _not_ meaningful when `action='store_true'`, as it will just say "(default: False)", which is silly. It's mostly meaningful when the option takes an argumen..."glance --help" should include defaults for options where it's meaningful.
It's _not_ meaningful when `action='store_true'`, as it will just say "(default: False)", which is silly. It's mostly meaningful when the option takes an argument. So for `--outputpath`, it might say "set path to the output directory (default: ./)"
In theory you can use `formatter=argparse.ArgumentDefaultsHelpFormatter`, but that will include the silly defaults, like "(default: False)". So you need to specify it on a case-by-case basis.
This work is available as merge request !16 from branch [help-defaults](https://gitlab.ssec.wisc.edu/evas/UW-Glance/tree/help-defaults).https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/30Display icon on MacOSX2018-12-20T22:46:37ZAlan De SmetDisplay icon on MacOSXThe icon isn't displayed on MacOSX. I think it's because it needs to be set in QApplication, not the top level QWidget.
A proposed _but untested on MacOSX_ is on [iconfix](https://gitlab.ssec.wisc.edu/evas/UW-Glance/tree/iconfix), and i...The icon isn't displayed on MacOSX. I think it's because it needs to be set in QApplication, not the top level QWidget.
A proposed _but untested on MacOSX_ is on [iconfix](https://gitlab.ssec.wisc.edu/evas/UW-Glance/tree/iconfix), and is available at !17. You can [see the diff directly](https://gitlab.ssec.wisc.edu/evas/UW-Glance/merge_requests/17/diffs)https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/31Support for NetCDF4 Groups2022-01-27T16:39:46ZZach GriffithSupport for NetCDF4 GroupsCurrently everything is expected to be top-level variables in NetCDF files. It would be nice to be able to use this tool with files using [NetCDF4 groups](http://unidata.github.io/netcdf4-python/#section2).Currently everything is expected to be top-level variables in NetCDF files. It would be nice to be able to use this tool with files using [NetCDF4 groups](http://unidata.github.io/netcdf4-python/#section2).Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/32Early exit without ERROR explaination in reportgen lacking longitude/latitude2021-08-24T18:52:30ZAlan De SmetEarly exit without ERROR explaination in reportgen lacking longitude/latitudeIf you try to `glance reportGen foo.nc foo.nc` where there is no longitude and latitude data in foo.nc, the output will look like this:
```
WARNING:glance.compare:Error while loading longitude or latitude:
WARNING:glance.compare:Unable...If you try to `glance reportGen foo.nc foo.nc` where there is no longitude and latitude data in foo.nc, the output will look like this:
```
WARNING:glance.compare:Error while loading longitude or latitude:
WARNING:glance.compare:Unable to retrieve pixel_longitude data. The variable name may not exist in this file or an error may have occured while attempting to access the data. Details of file access error observed: 'pixel_longitude'
```
While it says WARNING, it appears to be being treated as a fatal ERROR. I erroneously assumed that it carried on anyway and was surprised that no report was generated.https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/33GUI's "Display Statistics" should be nicely formatted similar to HTML2021-11-08T18:38:08ZAlan De SmetGUI's "Display Statistics" should be nicely formatted similar to HTMLThe HTML reports from reportGen and inspectReport are prettied up in #18. While a _bit_ of the formatting was done in the GUI's statistics, it would be nice to get some more.
<table>
<tr>
<th>Before #18</th>
<th>After #18</th>
<th>Goal...The HTML reports from reportGen and inspectReport are prettied up in #18. While a _bit_ of the formatting was done in the GUI's statistics, it would be nice to get some more.
<table>
<tr>
<th>Before #18</th>
<th>After #18</th>
<th>Goal</th>
</tr>
<tr>
<td>
<a href="/uploads/08a8265b75ad47a0c9d2021e71493ab7/guireport-before.png"><img src="/uploads/08a8265b75ad47a0c9d2021e71493ab7/guireport-before.png" width="300"></a>
</td>
<td>
<a href="/uploads/4c34cffab47e64fa2789a5032f37117f/guireport-after.png"><img src="/uploads/4c34cffab47e64fa2789a5032f37117f/guireport-after.png" width="300"></a>
</td>
<td sty=e"vertical-align: top">
<a href="/uploads/50d0e9356018d8cca2befe3f90f4df46/guireport-3.png"><img src="/uploads/50d0e9356018d8cca2befe3f90f4df46/guireport-3.png" width="300"></a>
<br>There are missing column headings: "Both", "File A", "File B"
<br>No zebra striping in this example!
</td>
</tr>
</table>
(Dark colors are my local configuration, not in the Glance code!)
Goals:
- [x] Group paired A/B values into a single line
1. [x] Move `group_stats_by_file` from report.py to stats.py
2. [x] In `sendStatsInfo` in gui_statsprovider.py, add something like:
```
kwargs = { 'runInfo': tempInfo,
'statGroups': tempAnalysis.dictionary_form(),
GROUPED_STATS_INFO_DICT_KEY: group_stats_by_file(tempAnalysis.dictionary_form()) }
```
- [x] Right align numbers
1. For the HTML reports, we're setting `class="n"` (for **n**umber). That doesn't appear to work for QTextEdit, so we'll need to use `align=right`.
- [x] Right align labels (?)
1. Same as right aligning numbers
- [x] Zebra stripe rows (?)
1. We're using CSS to accomplish this. This may not be possible; QTextEdit only supports CSS 2.1, so we can't use `:nth-child(even)`. We can add explicit CSS classes for striping, or just drop the color directly in with `<td bgcolor="SOMETHING">`.
Documentation on HTML/CSS support in QTextEdit: https://doc.qt.io/archives/qt-4.8/richtext-html-subset.htmlEva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/34Investigate Glance exhausting system memory on "simple" reportGen2019-02-05T20:54:16ZAlan De SmetInvestigate Glance exhausting system memory on "simple" reportGen**Alan is investigating this.**
Here's a failed run with a 32GB virtual memory limit:
```
$ ulimit -v 33554432
$ glance --nolonlat --outputpath scratch reportGen ./M6-hackery/output/day/GOESR_ABI_FD_2018352_1800_33_CLOUD_MASK_BL.nc ./2...**Alan is investigating this.**
Here's a failed run with a 32GB virtual memory limit:
```
$ ulimit -v 33554432
$ glance --nolonlat --outputpath scratch reportGen ./M6-hackery/output/day/GOESR_ABI_FD_2018352_1800_33_CLOUD_MASK_BL.nc ./2019-01-04/output/day/GOESR_ABI_FD_2018352_1800_33_CLOUD_MASK_BL.nc CloudMaskFlag -vvvvv
INFO:glance.config_organizer:Using Command Line Settings
INFO:glance.compare:Processing File A:
INFO:glance.data:Opening /home/adesmet/data/star-test-data/M6-hackery/output/day/GOESR_ABI_FD_2018352_1800_33_CLOUD_MASK_BL.nc
INFO:glance.data:File md5sum: a806b506c79b2d655f6bc319dd596276
INFO:glance.data:File was last modified: Mon Feb 4 13:22:03 2019
INFO:glance.compare:Processing File B:
INFO:glance.data:Opening /home/adesmet/data/star-test-data/2019-01-04/output/day/GOESR_ABI_FD_2018352_1800_33_CLOUD_MASK_BL.nc
INFO:glance.data:File md5sum: 1d925c9ee2ebe3c6c824405db04de280
INFO:glance.data:File was last modified: Mon Jan 14 15:58:40 2019
INFO:glance.compare:analyzing: CloudMaskFlag
Traceback (most recent call last):
File "/home/adesmet/bin/glance", line 11, in <module>
load_entry_point('uwglance', 'console_scripts', 'glance')()
File "/home/adesmet/src/glance/pyglance/glance/compare.py", line 1616, in main
rc = lower_locals[args[0].lower()](*args[1:])
File "/home/adesmet/src/glance/pyglance/glance/compare.py", line 1397, in reportGen
return reportGen_library_call(a_path, b_path, args[2:], tempOptions)
File "/home/adesmet/src/glance/pyglance/glance/compare.py", line 849, in reportGen_library_call
varRunInfo[EPSILON_KEY], varRunInfo[EPSILON_PERCENT_KEY])
File "/home/adesmet/src/glance/pyglance/glance/stats.py", line 883, in withSimpleData
new_object._create_stats(diffInfo)
File "/home/adesmet/src/glance/pyglance/glance/stats.py", line 910, in _create_stats
self.comparison = NumericalComparisonStatistics(diffInfoObject)
File "/home/adesmet/src/glance/pyglance/glance/stats.py", line 756, in __init__
self.correlation = delta.compute_correlation(aData, bData, valid_in_both) if not noData else np.nan
File "/home/adesmet/src/glance/pyglance/glance/delta.py", line 122, in compute_correlation
toReturn = compute_r_function(good_x_data, good_y_data)[0]
File "/usr/lib/python2.7/dist-packages/scipy/stats/stats.py", line 3018, in pearsonr
xm, ym = x - mx, y - my
MemoryError
```
Here's one with a 20GB virtual memory limit:
```
$ ulimit -v 20971520
$ glance --nolonlat --outputpath scratch reportGen ./M6-hackery/output/day/GOESR_ABI_FD_2018352_1800_33_CLOUD_MASK_BL.nc ./2019-01-04/output/day/GOESR_ABI_FD_2018352_1800_33_CLOUD_MASK_BL.nc CloudMaskFlag -vvvvv
INFO:glance.config_organizer:Using Command Line Settings
INFO:glance.compare:Processing File A:
INFO:glance.data:Opening /home/adesmet/data/star-test-data/M6-hackery/output/day/GOESR_ABI_FD_2018352_1800_33_CLOUD_MASK_BL.nc
INFO:glance.data:File md5sum: a806b506c79b2d655f6bc319dd596276
INFO:glance.data:File was last modified: Mon Feb 4 13:22:03 2019
INFO:glance.compare:Processing File B:
INFO:glance.data:Opening /home/adesmet/data/star-test-data/2019-01-04/output/day/GOESR_ABI_FD_2018352_1800_33_CLOUD_MASK_BL.nc
INFO:glance.data:File md5sum: 1d925c9ee2ebe3c6c824405db04de280
INFO:glance.data:File was last modified: Mon Jan 14 15:58:40 2019
INFO:glance.compare:analyzing: CloudMaskFlag
Traceback (most recent call last):
File "/home/adesmet/bin/glance", line 11, in <module>
load_entry_point('uwglance', 'console_scripts', 'glance')()
File "/home/adesmet/src/glance/pyglance/glance/compare.py", line 1616, in main
rc = lower_locals[args[0].lower()](*args[1:])
File "/home/adesmet/src/glance/pyglance/glance/compare.py", line 1397, in reportGen
return reportGen_library_call(a_path, b_path, args[2:], tempOptions)
File "/home/adesmet/src/glance/pyglance/glance/compare.py", line 849, in reportGen_library_call
varRunInfo[EPSILON_KEY], varRunInfo[EPSILON_PERCENT_KEY])
File "/home/adesmet/src/glance/pyglance/glance/stats.py", line 881, in withSimpleData
epsilonValue=epsilon, epsilonPercent=epsilon_percent)
File "/home/adesmet/src/glance/pyglance/glance/data.py", line 327, in __init__
epsilonValue, epsilonPercent)
File "/home/adesmet/src/glance/pyglance/glance/data.py", line 419, in analyze
raw_diff = np.zeros(shape, dtype=sharedType)
MemoryError
```
Relevant files are on my laptop.Alan De SmetAlan De Smethttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/36Use internal hashlib.md5 instead of external md5sum2021-04-08T18:32:41ZAlan De SmetUse internal hashlib.md5 instead of external md5sum`/pyglance/glance/data.py` in `FileInfo/__init__` around line 501 calls out to `md5sum` to generate an MD5 checksum for a file. This is a bit silly, as Python can do it natively, is probably a bit slower (since a new process is invoked),...`/pyglance/glance/data.py` in `FileInfo/__init__` around line 501 calls out to `md5sum` to generate an MD5 checksum for a file. This is a bit silly, as Python can do it natively, is probably a bit slower (since a new process is invoked), and may cause problems on platforms without md5sum (Windows, some embedded Linux systems).
Solution: replace it with a native Python solution.https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/37Some (all?) command line arguments are silently ignored when using config file2021-10-11T20:35:13ZAlan De SmetSome (all?) command line arguments are silently ignored when using config fileWhen using a configuration file (`--configfile`), Glance silently ignores at least some command line arguments. I know that a list of variable names is definitely ignored. This may be reasonable (the configuration file needs to include t...When using a configuration file (`--configfile`), Glance silently ignores at least some command line arguments. I know that a list of variable names is definitely ignored. This may be reasonable (the configuration file needs to include the list of variable names, or an explicit empty dict to indicate "all"), but _silently_ ignoring the command line seems bad. This probably needs at least a warning and perhaps it should be an error.
The logic in question appears to be in `load_config_or_options` in `/pyglance/glance/config_organizer.py` around line 316.https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/38Make reports use less disk space2021-10-07T19:06:06ZAlan De SmetMake reports use less disk spaceReports with images can be pretty big, especially when comparing 100+ variables. A comparison of some GOES-R level 2 outputs, 104 variables total, is about 73MiB.
The bulk of the space is going to images. Here's a typical single variabl...Reports with images can be pretty big, especially when comparing 100+ variables. A comparison of some GOES-R level 2 outputs, 104 variables total, is about 73MiB.
The bulk of the space is going to images. Here's a typical single variable:
| **Size (KiB)** | **File** |
| --: | :----------------------- |
| 1,452 | **TOTAL** |
| 217 | A.png |
| 202 | B.png |
| 135 | Hex.png |
| 88 | Diff.png |
| 80 | AbsDiff.png |
| 76 | small.A.png |
| 74 | small.AbsDiff.png |
| 73 | small.B.png |
| 72 | Mismatch.png |
| 72 | Hist.png |
| 70 | small.Diff.png |
| 64 | DensityScatter.png |
| 62 | small.DensityScatter.png |
| 60 | small.Mismatch.png |
| 47 | small.Hex.png |
| 27 | small.Hist.png |
| 9.6 | index.html |
These are big enough to be a nuisance to hand around. If we could shrink them, it would be helpful.
Possible techniques to make things smaller:
- [x] Make a command line version of the option to DO_IMAGES_ONLY_ON_FAIL_KEY.
- [x] Don't generate `small.*` images; use the originals directly and rescale in browser. It will use more browser memory, CPU, and bandwidth., but I don't think it's significant. This might save about 30% of the space.
- ~~Make our images indexed color instead of RGBA.~~ It looks like there isn't a way to do this conversion in matplotlib. We could probably do it with PIL but my back of envelope tests suggest it would perceptibly damage the image, so I'm not going to do it for now.
- ~~Higher compression levels. perhaps `compress_level=9, optimize=True`? (default is `compress_level=6, optimize=False`).~~ Probably not; Going from compression 6 to 9 saves about 2%, turning on optimization saves nothing (indeed, it grows a bit!).
Comparison of compression levels:
| Size (B) | Time (s) | compress_level | optimize |
| --------: | ------: | -: | :---: |
| 1,590,874 | 2.74 | 1 | False |
| 1,447,641 | 4.44 | 6 | False |
| 1,414,188 | 20.09 | 9 | False |
| 1,416,435 | 19.97 | 9 | True |Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/41False positives - glance 0.3.5 occasionally misses ordering differences in GO...2020-10-22T20:30:07ZNick Bearsonnickb@ssec.wisc.eduFalse positives - glance 0.3.5 occasionally misses ordering differences in GOES-R L1bUsing `glance reportGen --nolonlat $file1 $file2`, glance 0.3.5 reports these datasets as matching, however, an inspection with ncdump shows that they do have ordering differences:
# 1
https://www.ssec.wisc.edu/~nickb/grb-comparison-re...Using `glance reportGen --nolonlat $file1 $file2`, glance 0.3.5 reports these datasets as matching, however, an inspection with ncdump shows that they do have ordering differences:
# 1
https://www.ssec.wisc.edu/~nickb/grb-comparison-reports/s2020054160-glance0.3.5/OR_SEIS-L1b-MPSH_G16_s20200541609300_e20200541609590_c20200541610007.nc/L1a_SciData_TimeStamp/index.html
```
(glance.env)[nickb@espresso s2020054160]$ /opt/netcdf4/4.4.1-gcc-6.1/bin/ncdump -v L1a_SciData_TimeStamp /data/users/nickb/grb/comparisons/s2020054160/grbman/OR_SEIS-L1b-MPSH_G16_s20200541609300_e20200541609590_c20200541610007.nc
L1a_SciData_TimeStamp = 635746170, 635746171, 635746172, 635746173,
635746174, 635746175, 635746176, 635746177, 635746178, 635746179,
635746180, 635746181, 635746182, 635746183, 635746184, 635746185,
635746186, 635746187, 635746188, 635746189, 635746190, 635746191,
635746192, 635746193, 635746194, 635746195, 635746196, 635746197,
635746198, 635746199 ;
(glance.env)[nickb@espresso s2020054160]$ /opt/netcdf4/4.4.1-gcc-6.1/bin/ncdump -v L1a_SciData_TimeStamp /data/users/nickb/grb/comparisons/s2020054160/CLASS_PDA/OR_SEIS-L1b-MPSH_G16_s20200541609300_e20200541609590_c20200541610007.nc
L1a_SciData_TimeStamp = 635746170, 635746171, 635746172, 635746173,
635746174, 635746175, 635746176, 635746177, 635746178, 635746179,
635746180, 635746181, 635746182, 635746183, 635746184, 635746185,
635746187, 635746186, 635746188, 635746189, 635746190, 635746191,
635746192, 635746193, 635746194, 635746195, 635746196, 635746197,
635746198, 635746199 ;
```
`635746186, 635746187` flips to `635746187, 635746186`.
# 2
https://www.ssec.wisc.edu/~nickb/grb-comparison-reports/s2020054160-glance0.3.5/OR_SEIS-L1b-SGPS_G16_s20200541609000_e20200541609590_c20200541610004.nc/L1a_SciData_TimeStamp/index.html
```
(glance.env)[nickb@espresso s2020054160]$ /opt/netcdf4/4.4.1-gcc-6.1/bin/ncdump -v L1a_SciData_TimeStamp grbman/OR_SEIS-L1b-SGPS_G16_s20200541609000_e20200541609590_c20200541610004.nc
data:
L1a_SciData_TimeStamp =
635746140, 635746140,
635746141, 635746141,
635746142, 635746142,
635746143, 635746143,
635746144, 635746144,
635746145, 635746145,
635746146, 635746146,
635746147, 635746147,
635746148, 635746148,
635746149, 635746149,
635746150, 635746150,
635746151, 635746151,
635746152, 635746152,
635746153, 635746153,
635746154, 635746154,
635746155, 635746155,
635746156, 635746156,
635746157, 635746157,
635746158, 635746158,
635746159, 635746159,
635746160, 635746160,
635746161, 635746161,
635746162, 635746162,
635746163, 635746163,
635746164, 635746164,
635746165, 635746165,
635746166, 635746166,
635746167, 635746167,
635746168, 635746168,
635746169, 635746169,
635746170, 635746170,
635746171, 635746171,
635746172, 635746172,
635746173, 635746173,
635746174, 635746174,
635746175, 635746175,
635746176, 635746176,
635746177, 635746177,
635746178, 635746178,
635746179, 635746179,
635746180, 635746180,
635746181, 635746181,
635746182, 635746182,
635746183, 635746183,
635746184, 635746184,
635746185, 635746185,
635746186, 635746186,
635746187, 635746187,
635746188, 635746188,
635746189, 635746189,
635746190, 635746190,
635746191, 635746191,
635746192, 635746192,
635746193, 635746193,
635746194, 635746194,
635746195, 635746195,
635746196, 635746196,
635746197, 635746197,
635746198, 635746198,
635746199, 635746199 ;
(glance.env)[nickb@espresso s2020054160]$ /opt/netcdf4/4.4.1-gcc-6.1/bin/ncdump -v L1a_SciData_TimeStamp CLASS_PDA/OR_SEIS-L1b-SGPS_G16_s20200541609000_e20200541609590_c20200541610004.nc
L1a_SciData_TimeStamp =
635746140, 635746140,
635746141, 635746141,
635746142, 635746142,
635746143, 635746143,
635746144, 635746144,
635746145, 635746145,
635746146, 635746146,
635746147, 635746147,
635746148, 635746148,
635746149, 635746149,
635746150, 635746150,
635746151, 635746151,
635746152, 635746152,
635746153, 635746153,
635746154, 635746154,
635746155, 635746155,
635746156, 635746156,
635746157, 635746157,
635746158, 635746158,
635746159, 635746159,
635746160, 635746160,
635746161, 635746161,
635746162, 635746162,
635746163, 635746163,
635746164, 635746164,
635746165, 635746165,
635746167, 635746167,
635746166, 635746166,
635746168, 635746168,
635746169, 635746169,
635746170, 635746170,
635746171, 635746171,
635746172, 635746172,
635746173, 635746173,
635746174, 635746174,
635746175, 635746175,
635746176, 635746176,
635746177, 635746177,
635746178, 635746178,
635746179, 635746179,
635746180, 635746180,
635746181, 635746181,
635746182, 635746182,
635746183, 635746183,
635746184, 635746184,
635746185, 635746185,
635746186, 635746186,
635746187, 635746187,
635746188, 635746188,
635746189, 635746189,
635746190, 635746190,
635746191, 635746191,
635746192, 635746192,
635746193, 635746193,
635746194, 635746194,
635746195, 635746195,
635746196, 635746196,
635746197, 635746197,
635746198, 635746198,
635746199, 635746199 ;
```
The second file is out of order:
```
635746165, 635746165,
635746167, 635746167,
635746166, 635746166,
```
---
Note that it did not miss all ordering differences in these dataset, such as:
https://www.ssec.wisc.edu/~nickb/grb-comparison-reports/s2020054160-glance0.3.5/OR_SEIS-L1b-SGPS_G16_s20200541609000_e20200541609590_c20200541610004.nc/ECEF_Z/index.html
---
I've also put this data on `/apollo/csppgeo/verification/s2020054160`Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/42glance gui fails to run under conda install2020-07-13T19:27:35ZWilliam Strakaglance gui fails to run under conda installFollowing the instructions under https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/wikis/installation for a anaconda install, when trying to run glance gui, it fails because
PackagesNotFoundError: The following packages are not available fr...Following the instructions under https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/wikis/installation for a anaconda install, when trying to run glance gui, it fails because
PackagesNotFoundError: The following packages are not available from current channels:
- pyqt=4.11.4
Current channels:
- https://repo.anaconda.com/pkgs/main/osx-64
- https://repo.anaconda.com/pkgs/main/noarch
- https://repo.anaconda.com/pkgs/r/osx-64
- https://repo.anaconda.com/pkgs/r/noarch
This means I get
*** Unable to import PyQt4. Please install PyQt4 and add it to your PYTHONPATH in order to use the Glance GUI. ***
Traceback (most recent call last):
File "/Users/wstraka/anaconda2/envs/glance/bin/glance", line 11, in <module>
load_entry_point('uwglance==0.3.5', 'console_scripts', 'glance')()
File "/Users/wstraka/anaconda2/envs/glance/lib/python2.7/site-packages/pkg_resources/__init__.py", line 489, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/Users/wstraka/anaconda2/envs/glance/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2852, in load_entry_point
return ep.load()
File "/Users/wstraka/anaconda2/envs/glance/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2443, in load
return self.resolve()
File "/Users/wstraka/anaconda2/envs/glance/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2449, in resolve
module = __import__(self.module_name, fromlist=['__name__'], level=0)
File "/Users/wstraka/anaconda2/envs/glance/lib/python2.7/site-packages/uwglance-0.3.5-py2.7.egg/glance/compare.py", line 24, in <module>
import glance.gui_controller as gui_control
File "/Users/wstraka/anaconda2/envs/glance/lib/python2.7/site-packages/uwglance-0.3.5-py2.7.egg/glance/gui_controller.py", line 12, in <module>
from PyQt4 import QtGui
ImportError: No module named PyQt4
What is the work around?Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/44Fix mistaken use of the PlottingFunctionFactory parent class ___init__ function2021-04-08T18:36:28ZEva SchifferFix mistaken use of the PlottingFunctionFactory parent class ___init__ functionThis came from Mike Walters. I've been seeing the behavior in the newer version of Glance too, but hadn't looked into it yet. Mike said:
> I keep getting error messages like this:
>
> 2020-08-11 22:56:51 ERROR: (plotcreatefns) Someo...This came from Mike Walters. I've been seeing the behavior in the newer version of Glance too, but hadn't looked into it yet. Mike said:
> I keep getting error messages like this:
>
> 2020-08-11 22:56:51 ERROR: (plotcreatefns) Someone has instantiated PlottingFunctionFactory. This class is NOT meant to be instantiated.
>
> The problem here is that the subclasses of the PlottingFunctionFactory class don't have their own __init__() methods, so they inherit the __init__() method of PlottingFunctionFactory, which logs the error message. In spite of the error message, Glance runs normally. So this isn't a show stopper - it's just annoying.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/46SyntaxWarning: "is" with a literal. Did you mean "=="?2020-10-23T17:38:37ZAlan De SmetSyntaxWarning: "is" with a literal. Did you mean "=="?Python 3.8 adds a new SyntaxWarning if `is` or `is not` is used where `==` or `!=` is correct. It's generated only when createding the pyc file, so typically you'll only ever see the error once. ([This is a good overview of why `is`/`is ...Python 3.8 adds a new SyntaxWarning if `is` or `is not` is used where `==` or `!=` is correct. It's generated only when createding the pyc file, so typically you'll only ever see the error once. ([This is a good overview of why `is`/`is not` can cause bugs](https://adamj.eu/tech/2020/01/21/why-does-python-3-8-syntaxwarning-for-is-literal/).) Glance generates these warnings in a variety of places:
```
$ # Ensure the warnings are generated
$ rm -rf pyglance/glance/__pycache__
$ glance
/data/users/adesmet/glance-testing/glance/pyglance/glance/compare.py:558: SyntaxWarning: "is" with a literal. Did you mean "=="?
elif (len(aData.shape) is 1) :
/data/users/adesmet/glance-testing/glance/pyglance/glance/compare.py:562: SyntaxWarning: "is" with a literal. Did you mean "=="?
elif (len(aData.shape) is 2) :
/data/users/adesmet/glance-testing/glance/pyglance/glance/compare.py:912: SyntaxWarning: "is" with a literal. Did you mean "=="?
elif (len(aData.shape) is 1) :
/data/users/adesmet/glance-testing/glance/pyglance/glance/compare.py:916: SyntaxWarning: "is" with a literal. Did you mean "=="?
elif (len(aData.shape) is 2) :
/data/users/adesmet/glance-testing/glance/pyglance/glance/plot.py:50: SyntaxWarning: "is not" with a literal. Did you mean "!="?
isParent = not (pid is 0)
/data/users/adesmet/glance-testing/glance/pyglance/glance/plot.py:101: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if not (pid is 0) :
/data/users/adesmet/glance-testing/glance/pyglance/glance/plotcreatefns.py:739: SyntaxWarning: "is" with a literal. Did you mean "=="?
assert(len(aData.shape) is 1)
/data/users/adesmet/glance-testing/glance/pyglance/glance/config_organizer.py:114: SyntaxWarning: "is" with a literal. Did you mean "=="?
if (len(requestedNames) is 0) :
/data/users/adesmet/glance-testing/glance/pyglance/glance/config_organizer.py:220: SyntaxWarning: "is" with a literal. Did you mean "=="?
if (len(requestedNames) is 0) :
/data/users/adesmet/glance-testing/glance/pyglance/glance/util.py:93: SyntaxWarning: "is" with a literal. Did you mean "=="?
if totalDataPts is 0 :
```
The should probably be corrected to ensure correct future behavior.https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/47FutureWarning: Conversion of the second argument of issubdtype from `float` t...2020-10-22T19:52:28ZAlan De SmetFutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated.IN glance/data.py, around line 371 is this:
```
# if we're looking at float or complex data, use a nan
if (np.issubdtype(type_to_return, np.float) or
np.issubdtype(type_to_return, np.complex)) :
...IN glance/data.py, around line 371 is this:
```
# if we're looking at float or complex data, use a nan
if (np.issubdtype(type_to_return, np.float) or
np.issubdtype(type_to_return, np.complex)) :
fill_value_to_return = np.nan
LOG.info("float or complex: using "+str(fill_value_to_return))
```
`np.float` in that test is treated as `np.floating`.
As of NumPy 1.18.5, this is deprecated:
```
[...]glance/data.py:371: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
if (np.issubdtype(type_to_return, np.float) or
```
As of NumPy 1.19.1, this _silently_ fails; `float32` is _not_ considered a subtype of `np.float`, `fill_value_to_return` is left None, and later on glance complains that it doesn't know what the fill value is:
```
Traceback (most recent call last):
File "/home/adesmet/src/glance/anaconda/bin/glance", line 11, in <module>
load_entry_point('uwglance', 'console_scripts', 'glance')()
File "/home/adesmet/src/glance/pyglance/glance/compare.py", line 1613, in main
rc = lower_locals[args[0].lower()](*args[1:])
File "/home/adesmet/src/glance/pyglance/glance/compare.py", line 1394, in reportGen
return reportGen_library_call(a_path, b_path, args[2:], tempOptions)
File "/home/adesmet/src/glance/pyglance/glance/compare.py", line 843, in reportGen_library_call
variable_stats = statistics.StatisticalAnalysis.withSimpleData(aData, bData,
File "/home/adesmet/src/glance/pyglance/glance/stats.py", line 880, in withSimpleData
diffInfo = dataobj.DiffInfoObject(aDataObject, bDataObject,
File "/home/adesmet/src/glance/pyglance/glance/data.py", line 326, in __init__
self.diff_data_object = DiffInfoObject.analyze(aDataObject, bDataObject,
File "/home/adesmet/src/glance/pyglance/glance/data.py", line 425, in analyze
assert(fill_data_value is not None)
AssertionError
```
Changing `np.float` to `np.floating` solves the problem.https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/48Glance needs to handle data with no fill value gracefully.2020-10-22T19:52:28ZEva SchifferGlance needs to handle data with no fill value gracefully.When you try to analyze a variable that has no fill value defined Glance will crash with an assertion error. This is not a graceful way to handle data analysis, so regardless of whether we consider this to be faulty data, we shouldn't be...When you try to analyze a variable that has no fill value defined Glance will crash with an assertion error. This is not a graceful way to handle data analysis, so regardless of whether we consider this to be faulty data, we shouldn't be crashing.
Here is an example assertion error traceback from Ray:
~~~
Traceback (most recent call last):
File "/opt/conda/envs/glance/bin/glance", line 33, in <module>
sys.exit(load_entry_point('uwglance==0.4.2', 'console_scripts', 'glance')())
File "/opt/conda/envs/glance/lib/python3.8/site-packages/uwglance-0.4.2-py3.8.egg/glance/compare.py", line 1613, in main
rc = lower_locals[args[0].lower()](*args[1:])
File "/opt/conda/envs/glance/lib/python3.8/site-packages/uwglance-0.4.2-py3.8.egg/glance/compare.py", line 1394, in reportGen
return reportGen_library_call(a_path, b_path, args[2:], tempOptions)
File "/opt/conda/envs/glance/lib/python3.8/site-packages/uwglance-0.4.2-py3.8.egg/glance/compare.py", line 843, in reportGen_library_call
variable_stats = statistics.StatisticalAnalysis.withSimpleData(aData, bData,
File "/opt/conda/envs/glance/lib/python3.8/site-packages/uwglance-0.4.2-py3.8.egg/glance/stats.py", line 880, in withSimpleData
diffInfo = dataobj.DiffInfoObject(aDataObject, bDataObject,
File "/opt/conda/envs/glance/lib/python3.8/site-packages/uwglance-0.4.2-py3.8.egg/glance/data.py", line 326, in __init__
self.diff_data_object = DiffInfoObject.analyze(aDataObject, bDataObject,
File "/opt/conda/envs/glance/lib/python3.8/site-packages/uwglance-0.4.2-py3.8.egg/glance/data.py", line 416, in analyze
assert(fill_data_value is not None)
AssertionError
~~~Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/49Add config file processing for stats2022-03-03T19:15:05ZEva SchifferAdd config file processing for statsRight now stats is the only main function of glance that will just ignore your config file. It would be good to fix that so stats has a few more knobs.
Notes on what stuff in the config file is meaningful for stats:
- nothing in the se...Right now stats is the only main function of glance that will just ignore your config file. It would be good to fix that so stats has a few more knobs.
Notes on what stuff in the config file is meaningful for stats:
- nothing in the settings matters
- we don't have any use for the lon/lat info
- [x] we need to respect many of the default values, including
- [x] the epsilon and fill value
- [x] the various failure configurations matter if we're going to do pass fail for the return code (I'm not sure I want to do that on this ticket, but a thing to think about)
- we don't really care about any of the plot configuration stuff including the various bin/tuple and vector plotting stuff
- [x] In the setOfVariables we'll need to respect similar stuff to the settings
- [x] technical vs display names
- [x] epsilon and fill value
- [x] the failure configuration stuff
- [x] the basic and variable dependant filtering
Stuff to fix:
- [x] one input stats
- [x] handle the technical and display names correctly
- [x] respect fill value from command line or config (default or var specific)
- [x] handle basic and variable dependent filtering
- [x] two input stats
- [x] handle the technical and display names correctly
- [x] respect epsilon and fill value from the command line or config (default or var specific)
- [x] handle basic and variable dependant filtering
- [x] do pass/fail testing from command line or config when needed and change return codes based on that (from default or var specific in config)Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/50Bugs when running single file variable stats in the gui2021-08-19T19:17:14ZEva SchifferBugs when running single file variable stats in the guiWhen you run statistics for only one file in the gui the stats will still be labeled as "comparison" stats in the display window. Also if you run stats for the B file with no A file loaded, the stats will be labeled as being for "None" v...When you run statistics for only one file in the gui the stats will still be labeled as "comparison" stats in the display window. Also if you run stats for the B file with no A file loaded, the stats will be labeled as being for "None" variable (the name is not pulled from the B, and it gets None from the not-loaded A file somehow).Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/51Spatial mismatch plots are broken with new cartopy plotting2021-10-21T18:56:01ZEva SchifferSpatial mismatch plots are broken with new cartopy plottingNot sure why yet, but when the system tries to make one it just makes an empty white rectangle image.Not sure why yet, but when the system tries to make one it just makes an empty white rectangle image.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/52Write up details on how to use the Glance GUI2021-10-05T19:25:23ZEva SchifferWrite up details on how to use the Glance GUIFill in the page describing how to use the GUI to examine or compare data.
The stub page is here: https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/wikis/The-Glance-GUIFill in the page describing how to use the GUI to examine or compare data.
The stub page is here: https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/wikis/The-Glance-GUIhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/53Get Colocation working again2021-08-02T20:00:18ZEva SchifferGet Colocation working againThe colocation code hasn't been tested in a very long time. Now that we have some GOES R series winds, try to get that working again.The colocation code hasn't been tested in a very long time. Now that we have some GOES R series winds, try to get that working again.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/54Trim excess white borders on generated graphs2021-12-07T19:57:44ZAlan De SmetTrim excess white borders on generated graphsGenerated graphs have lots of whitespace around them because matplotlib loves whitespace. However, I'm less of a fan of lots of whitespace, especially when I'm trying to pack a bunch of charts into a small space to easily skim them. For ...Generated graphs have lots of whitespace around them because matplotlib loves whitespace. However, I'm less of a fan of lots of whitespace, especially when I'm trying to pack a bunch of charts into a small space to easily skim them. For example:
![before](/uploads/24e78bd91a528a1722660fd7fa618147/before.png)
Less margin would be better. Perhaps something like:
![after](/uploads/d6baa0efbf50a8e941970d3d5dd3c81d/after.png)
One possible solution is to use [matplotlib.pyplot.savefig](https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.savefig.html)'s `bbox_inches` and optionally `pad_inches`. Setting `bbox_inches='tight'` will cause matplotlib to remove as much border as possible, then re-add pad_inches (which defaults to 0.1). This is what generated the second image above. You can trim even tighter, all the way down to `pad_inches=0`, although that gets very cramped feeling when looking at the image by itself.
So in pyglance/glance/plot.py in _handle_fig_creation_task around line 60, you could change
```
figure_obj.savefig(os.path.join(outputPath, fullFigName), dpi=fullDPI)
```
to
```
figure_obj.savefig(os.path.join(outputPath, fullFigName), dpi=fullDPI, bbox_inches='tight', pad_inches=0.1)
```
There are other calls to savefig in Glance; I don't know if they should have similar arguments applied.
One potential down side: the resulting images will not have a consistent size! Long labels on the left side, the presence or absence of a color bar on the right, multiple lines of title, and other things can change the size of the "core" chart. So if consistent sizes are important... this might be a problem.https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/55Reintegrate glance-tools functionality into base Glance2021-11-08T18:53:48ZEva SchifferReintegrate glance-tools functionality into base GlanceRight now @adesmet is maintaining https://gitlab.ssec.wisc.edu/adesmet/glance-tools that work on top of base glance. We should just move this functionality inside of glance.
* [x] compare directories, automatically matching file pairs a...Right now @adesmet is maintaining https://gitlab.ssec.wisc.edu/adesmet/glance-tools that work on top of base glance. We should just move this functionality inside of glance.
* [x] compare directories, automatically matching file pairs and generating comparison reports for all paired files
* [x] let the user exclude parts of names when auto-matching file pairs
* [x] make a summary page for runs where you're comparing multiple sets of filesAlan De SmetAlan De Smethttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/56Zero pad report numbers when running multiple reports so the sort correctly o...2021-10-07T15:26:04ZEva SchifferZero pad report numbers when running multiple reports so the sort correctly on the summary pageRight now if you use the report command to compare directories, they output will be put into directories like "report-0-CG_ABI-L2-CODDM1-M6_G16_s20191471800485_e20191471800554_c20201761959040" or "report-16-CG_ABI-L2-NAVM1-M6_G16_s201914...Right now if you use the report command to compare directories, they output will be put into directories like "report-0-CG_ABI-L2-CODDM1-M6_G16_s20191471800485_e20191471800554_c20201761959040" or "report-16-CG_ABI-L2-NAVM1-M6_G16_s20191471800485_e20191471800554_c20201761959040". Because my numbers are not zero padded, if you compare more than 9 reports, the resulting directories will sort badly on the summary.html page.
If we zero pad the number in the directory name, they should sort more gracefully. Ideally we should zero pad based on how many total reports we expect to make (so only add zeros to pad when needed).Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/57When you fail to plot Scatter plots in the GUI we should show the user some s...2021-10-13T15:32:33ZEva SchifferWhen you fail to plot Scatter plots in the GUI we should show the user some sort of messageScatter plots will sometimes not be plotted because your data is too big. When we do this in the GUI interface we should tell the user why we didn't plot anything (right now there is a command line message but no message in the GUI, so i...Scatter plots will sometimes not be plotted because your data is too big. When we do this in the GUI interface we should tell the user why we didn't plot anything (right now there is a command line message but no message in the GUI, so it just looks like your request to plot silently failed).Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/58Better color maps for the GUI2021-10-06T21:38:56ZEva SchifferBetter color maps for the GUIOur set of available color maps in the gui is... not great. We should provide some more perceptually useful options instead of just a rainbow of terrible rainbow options.Our set of available color maps in the gui is... not great. We should provide some more perceptually useful options instead of just a rainbow of terrible rainbow options.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/59Document why unable to create HistA.png2021-10-26T16:11:27ZAlan De SmetDocument why unable to create HistA.pngUnder a variety of cases, glance will be unable to create HistA.png for a given variable. It would be useful to know why that was. For example:
```
$ glance report --nolatlon test.nc
WARNING:glance.plot:Unable to create plot His...Under a variety of cases, glance will be unable to create HistA.png for a given variable. It would be useful to know why that was. For example:
```
$ glance report --nolatlon test.nc
WARNING:glance.plot:Unable to create plot HistA.png (saving image of histogram of the values in band_wavelength).
WARNING:glance.plot:Unable to create plot HistA.png (saving image of histogram of the values in x_image).
```
In this particular case, `band_wavelength` is declared as having a length of `band`, which is `1`. `x_image` is not declared as having any length at all (simple scalar?). So I might expect output like:
```
$ glance report --nolatlon test.nc
WARNING:glance.plot:Unable to create plot HistA.png (saving image of histogram of the values in band_wavelength; only 1 data point found, at least XX are required).
WARNING:glance.plot:Unable to create plot HistA.png (saving image of histogram of the values in x_image; single scalar value found, an array with at least XX data points is required).
```https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/60The GUI should handle 3+ dimensional data gracefully when plotting2021-11-08T19:07:05ZEva SchifferThe GUI should handle 3+ dimensional data gracefully when plottingRight now if you try to plot variable data that is 3 or more dimensions in the GUI it will crash. We should handle this case more gracefully (optimally with a warning to the user and no crashing).Right now if you try to plot variable data that is 3 or more dimensions in the GUI it will crash. We should handle this case more gracefully (optimally with a warning to the user and no crashing).Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/64Double check that the range colors and labels are working properly2021-12-09T22:03:57ZEva SchifferDouble check that the range colors and labels are working properlyI have some old reports that the range colors and labels that are configurable via the config file are not working anymore. Investigate if there is actually a problem (try cloud mask maybe?).I have some old reports that the range colors and labels that are configurable via the config file are not working anymore. Investigate if there is actually a problem (try cloud mask maybe?).Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/65Treat units of "1" as not having units for the purposes of plot displays2021-12-10T20:10:11ZEva SchifferTreat units of "1" as not having units for the purposes of plot displaysWhen labeling the color bar with units, if the units are 1, treat that as if there were no units by not putting a unit label on the plot displays.When labeling the color bar with units, if the units are 1, treat that as if there were no units by not putting a unit label on the plot displays.https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/66Give more basic variable information on the main page of reports2022-02-15T19:50:57ZEva SchifferGive more basic variable information on the main page of reportsOn the main page of the report, give shape information for variables and where variables have only a single value (there are a few ways, see the GUI single value display) display that value. This information is currently available in var...On the main page of the report, give shape information for variables and where variables have only a single value (there are a few ways, see the GUI single value display) display that value. This information is currently available in variable reports, but should be super easy to display clearly on the main summary page as well.
- [x] give shape information for variables on the main page of the report
- [x] where variables have only a single value display that value on the main page of the reportEva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/67Give some of the labels in the GUI tooltips to explain them2021-11-23T16:08:17ZEva SchifferGive some of the labels in the GUI tooltips to explain themSome things like the epsilon or how the fill value overriding works might benefit from tool tips or help text of some kind. See what we can do with the tools we have in PyQt.Some things like the epsilon or how the fill value overriding works might benefit from tool tips or help text of some kind. See what we can do with the tools we have in PyQt.https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/68GUI is selecting the wrong Longitude variable on file load2021-11-23T16:19:16ZEva SchifferGUI is selecting the wrong Longitude variable on file loadOn the settings tab the GUI tries to auto select Longitude and Latitude variables when you open a file. Right now it's generally getting the Latitude right, but it's having difficulty with the Longitude variable. (I can see this problem ...On the settings tab the GUI tries to auto select Longitude and Latitude variables when you open a file. Right now it's generally getting the Latitude right, but it's having difficulty with the Longitude variable. (I can see this problem in my artificially bad nav example, and I expect it's happening with other files as well.)Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/71Update doc page for comparison reports2021-12-01T18:11:46ZEva SchifferUpdate doc page for comparison reportsWe've moved to a tabular format for the statistics and this changed some of the statistic names. For example, we used to have "median_a" and "median_b" but now we only have "median" with entries for A and B.
The doc page has the old st...We've moved to a tabular format for the statistics and this changed some of the statistic names. For example, we used to have "median_a" and "median_b" but now we only have "median" with entries for A and B.
The doc page has the old stat names and descriptions, so it needs to be updated to the new ones.https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/72Update how the documentation is shown on reports2021-12-01T20:48:14ZEva SchifferUpdate how the documentation is shown on reportsVariable reports have asterisks that link to the glossary. Improvements: A small "?", probably superscripted, it a more common idiom. Ideally the link should use an #anchor to link directly to the exact glossary entry. A wee bit of ja...Variable reports have asterisks that link to the glossary. Improvements: A small "?", probably superscripted, it a more common idiom. Ideally the link should use an #anchor to link directly to the exact glossary entry. A wee bit of javascript (or maybe just pure CSS) could also make it so that hovering or clicking the "?" would cause the glossary entry to appear inline.
Note: whatever we do here we will need to make sure we don't break the GUI display of stats in the process.
- [x] change the link to be a small superscripted "?"
- [x] #anchor directly to the glossary entry for the term
- [x] create a tooltip with the definition over the linkEva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/73When histogram creation fails for single value variables the report still dis...2021-11-23T18:01:47ZEva SchifferWhen histogram creation fails for single value variables the report still displays a broken imageWhen I create an inspection report for a single value variable it can't make a histogram, which is fine. What's not fine is that the report has a broken image link in it, rather than just no image at all.When I create an inspection report for a single value variable it can't make a histogram, which is fine. What's not fine is that the report has a broken image link in it, rather than just no image at all.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/75Revise reports based on Mike Walters' examples2022-02-18T21:04:52ZEva SchifferRevise reports based on Mike Walters' examplesMike Walters did some reworking of the reports locally and I think some of the stuff he did may generalize to making our reports more clear and useful.
- [x] add a command line option to turn on variables missing that were specified in...Mike Walters did some reworking of the reports locally and I think some of the stuff he did may generalize to making our reports more clear and useful.
- [x] add a command line option to turn on variables missing that were specified in the config file being an error (done in https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/17 )
- [x] add a line to the comparison results table for each variable that was requested (in the config or on the command line as appropriate) but could not be found or compared for whatever reason (only when '--warnmissingvars' is used on the command line)
- [x] add a summary line for the compared variables with how many passed and failed (+ errored)
- [x] a table based display of the variable comparison details in the "Compared Variables" section
~~- [ ] next/back/up links in the header to make quickly navigating the whole report easy~~
- [x] on the variable page, note whether or not that variable passed comparison
Here are some example reports from Mike for reference:
[index__1_.html](/uploads/51e33c6a0f143814cf1594d29169c3e9/index__1_.html)
[index__2_.html](/uploads/b4d3e5f50293a06a14373b2354f0fbf9/index__2_.html)
[index.html](/uploads/40e97ffab590924e429c322151663108/index.html)
_Mike has also requested the ability to have Glance use configurable custom report templates with the same input data as my regular templates. I'm not doing that in this issue, but this is a note to myself to reassess how difficult that's going to be once I've done the other stuff here._
Note to self: I added a '--warnmissingvars' command line option to warn about missing variables on https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/17 but I still need to add the changes to put that info in the main report page on this ticket.
Edit: I've decided not to do the next/back/up links, since a batch of reports can more easily be navigated from the new summary page.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/76Move to using "collocate" instead of "colocateData" as our top level command2021-11-23T15:42:40ZEva SchifferMove to using "collocate" instead of "colocateData" as our top level commandGlance has been moving to one word commands where possible and based on a search of my email most people at SSEC spell it collocate rather than colocate, so we should move to using the name "collocate" for the command that was "colocateD...Glance has been moving to one word commands where possible and based on a search of my email most people at SSEC spell it collocate rather than colocate, so we should move to using the name "collocate" for the command that was "colocateData" and deprecate the old name.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/77If you start up the gui with a non-existent file name it will crash2021-12-16T18:14:42ZEva SchifferIf you start up the gui with a non-existent file name it will crashNormally if you give the gui command line a file name it will try to load it. If you give it a file name that doesn't exist it will correctly report that it could not be opened because it doesn't exist, but then try to access stuff from ...Normally if you give the gui command line a file name it will try to load it. If you give it a file name that doesn't exist it will correctly report that it could not be opened because it doesn't exist, but then try to access stuff from the file anyway and crash. It should just load up the gui with no file loaded at all.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/78rethink the info command2022-01-28T17:10:34ZEva Schifferrethink the info commandRight now the info command just gives you a list of available variables in a file. We can pretty easily organize a lot more basic information about the file contents (without actually looking at the data).
- show file info (dimensions,...Right now the info command just gives you a list of available variables in a file. We can pretty easily organize a lot more basic information about the file contents (without actually looking at the data).
- show file info (dimensions, global attributes)
- show variable info (size/type, dims, attributes)
This will need to be implemented/tested for all the types of files Glance supports:
- [x] hdf
- [x] netCDF (with and without groups)
- [x] h5
- [ ] AERI
- [x] GeoTIFFEva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/79Fix GeoTIFF plotting in the gui2022-01-11T19:41:39ZEva SchifferFix GeoTIFF plotting in the guiAt one point Glance was able to plot GeoTIFFs in the gui as rgb images.
I think I broke that while trying to keep other errors from causing crashes when plotting. Right now if you try to plot a GeoTIFF as an rgb image, you get a messag...At one point Glance was able to plot GeoTIFFs in the gui as rgb images.
I think I broke that while trying to keep other errors from causing crashes when plotting. Right now if you try to plot a GeoTIFF as an rgb image, you get a message like:
"Two dimensional data is required for this plot type. The provided variable data is (1500, 1500, 3) shaped."Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/83the --doPassFail command line option is not working properly (and needs some ...2022-02-24T16:11:35ZEva Schifferthe --doPassFail command line option is not working properly (and needs some redesign)At the moment the --doPassFail command line option turns on pass/fail testing for the purposes of the return codes, but it doesn't actually control how reports generate at the moment.
On a higher level, I think I want this setting to b...At the moment the --doPassFail command line option turns on pass/fail testing for the purposes of the return codes, but it doesn't actually control how reports generate at the moment.
On a higher level, I think I want this setting to be on by default and I need to create an appropriate config file setting that will let you do the same thing.
- [x] Turn pass/fail testing on by default
- [x] Make a command line option to turn pass/fail testing off
- [x] Make the current command line option (that turns pass/fail testing on) invisible in the docs (because it won't be needed anymore) but leave it in the system so people's existing calls don't break
- [x] Make a config file setting to turn pass/fail on and off (probably in the settings dictionary)
- [x] review report and stats code to make sure it will respect the pass/fail setting for all things (returns and how variables are analyzed)
- [x] Change the documentation to reflect the new way pass/fail testing works
- [x] Update that one config file that's intended to turn pass/fail testing offEva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/84Make more things configurable in the config file2022-02-25T19:46:19ZEva SchifferMake more things configurable in the config fileRight now Glance is getting some things from the command line that don't have an equivalent in the config file. I want to keep the output path and input file paths command line only, but for all the others, we should just have a way to s...Right now Glance is getting some things from the command line that don't have an equivalent in the config file. I want to keep the output path and input file paths command line only, but for all the others, we should just have a way to specify them in the config file.
Make a way to set the following in the config file
- [x] the **--nolonlat** option (maybe by setting the lat_lon_info to None?)
- [x] the **--warnmissingvars** option (maybe in the settings?)
I will also need to
- [x] Change the associated documentation to describe how these new settings work
- [x] Alter or add example configuration files to demonstrate how to use these
Note: the work to handle this for the **--doPassFail** option is being handled in the ticket for the pass/fail redesign: https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/83Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/85Glance crashes when opening a str variable in the GUI (or stats)2022-03-01T19:14:54ZEva SchifferGlance crashes when opening a str variable in the GUI (or stats)If you open a string type variable in the GUI or stats Glance will crash. We should handle this more gracefully.
I've attached an example file. The variable that triggers the problem is AMVChannelName.
[GOES17_ABI_2KM_MESO_2021056_194...If you open a string type variable in the GUI or stats Glance will crash. We should handle this more gracefully.
I've attached an example file. The variable that triggers the problem is AMVChannelName.
[GOES17_ABI_2KM_MESO_2021056_1942_25_WINDS_AMV_EN-14-CT.nc](/uploads/c449d0fa6d1b7e82ff1c8d1fd63e2e16/GOES17_ABI_2KM_MESO_2021056_1942_25_WINDS_AMV_EN-14-CT.nc)https://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/86stats should respect configuration files2022-03-03T19:35:05ZEva Schifferstats should respect configuration filesRight now calls to stats ignore configuration files that you specify on the command line. stats should load and respect the relevant information from a provided configuration file.Right now calls to stats ignore configuration files that you specify on the command line. stats should load and respect the relevant information from a provided configuration file.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/88Make sure summary report works properly with missing variables2022-04-07T14:57:43ZEva SchifferMake sure summary report works properly with missing variablesNow that the main report includes variables which were requested but not found when the user asks to be warned about missing variables, the summary report will need to handle that more gracefully.Now that the main report includes variables which were requested but not found when the user asks to be warned about missing variables, the summary report will need to handle that more gracefully.Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/91investigate case sensitive issues with data scaling2023-05-30T15:54:39ZEva Schifferinvestigate case sensitive issues with data scalingMy notes on this are very old, but I need to double check that Glance is handling any data scaling attributes in a case-insensitive manner. I know I'm handling some attributes appropriately, but I should investigate just in case I missed...My notes on this are very old, but I need to double check that Glance is handling any data scaling attributes in a case-insensitive manner. I know I'm handling some attributes appropriately, but I should investigate just in case I missed some cases (as I didn't take detailed notes of why I was worried about this problem).Eva SchifferEva Schifferhttps://gitlab.ssec.wisc.edu/evas/UW-Glance/-/issues/96Move Glance to python 3.102023-10-02T14:25:21ZEva SchifferMove Glance to python 3.10Move Glance to working in python 3.10. Currently Glance is tested and functioning in 3.6, but things have moved on, so the supporting software we rely on isn't all available in 3.6 anymore. This transision will probably require a lot of ...Move Glance to working in python 3.10. Currently Glance is tested and functioning in 3.6, but things have moved on, so the supporting software we rely on isn't all available in 3.6 anymore. This transision will probably require a lot of small clean up. The list below is what I know about so far and will be expanded as I investigate.
- [x] Review changes in python from 3.6 to 3.10 and make fixes in Glance as needed
- [x] Move from easy_install to a newer installation procedure (probably pip based?)
- [x] Rewrite the installation instructions on the wiki (note: include move from conda to mambaforge)
- [x] Do some minimal testing of Glance to make sure things don't break
- [x] numpy.bool causes a fatal AttributeError if you run the guiEva SchifferEva Schiffer