|
|
|
This page includes notes about the changes in released versions of Glance. Newer notes will be added at the top of the page as versions are released.
|
|
|
|
|
|
|
|
------------
|
|
|
|
|
|
|
|
**version 0.2.6.24**
|
|
|
|
|
|
|
|
the glance io module can now open AERI data files; this includes cxs, rnc, cxv, csv, spc, sum, uvs, and aeri data files
|
|
|
|
|
|
|
|
if the 'units' of a variable are described in the file, they will now be displayed on reports generated by glance
|
|
|
|
|
|
|
|
|
|
|
|
new plots for bin-tuple mode:
|
|
|
|
each witness plot also has a raw difference plot (A - B)
|
|
|
|
the bin-tuple mode will now plot all the simple comparisons over the whole data set
|
|
|
|
(this includes the scatter plot, the hex plot, and the histogram)
|
|
|
|
a new type of scatter plot will be included with the points for each of the bins colored in a different color
|
|
|
|
|
|
|
|
new get_sounding_profile_at_index(profile_data_3d, index_desired) filter in glance.filters, allows selection of one level of the sounding temperature or pressure profile
|
|
|
|
|
|
|
|
new 'thumb_DPI' and 'detail_DPI' keywords that can be used in the settings section of the configuration file to change the size of the images generated by glance
|
|
|
|
|
|
|
|
fixed link to the configuration file on main and variable report pages
|
|
|
|
|
|
|
|
------------
|
|
|
|
|
|
|
|
**version 0.2.6.23**
|
|
|
|
|
|
|
|
Fixed a bug that adversely affected plotting of original images while using an alternate missing value for the B file
|
|
|
|
|
|
|
|
------------
|
|
|
|
|
|
|
|
**version 0.2.6.22**
|
|
|
|
|
|
|
|
Added new hex plot to the comparison plots available in Glance reports
|
|
|
|
|
|
|
|
Corrected bug that caused the glossary link to break
|
|
|
|
|
|
|
|
------------
|
|
|
|
|
|
|
|
**version 0.2.6.21**
|
|
|
|
|
|
|
|
Added new method to the compare module, reportGen_raw_data_simple_call, that will allow simple calls to generate reports and images using static data blocks rather than files
|
|
|
|
|
|
|
|
------------
|
|
|
|
|
|
|
|
**version 0.2.6.20**
|
|
|
|
|
|
|
|
Added 'total_data_failure_tolerance' that can be defined in the config file to limit the percentage of data which can cumulatively fail the epsilon and non-finite tests
|
|
|
|
|
|
|
|
Added the ability to use the 'useSharedRangeForOriginal' setting when generating simple images without mapped correlation
|
|
|
|
|
|
|
|
------------
|
|
|
|
|
|
|
|
**version 0.2.6.19**
|
|
|
|
|
|
|
|
Fixed stats section labeling that sometimes caused a section to be mis-labeled or not shown.
|
|
|
|
|
|
|
|
Fixed error in pass/fail calculations that caused many pass/fail results to be incorrect
|
|
|
|
|
|
|
|
Added assert to simplify error shown when fill value is not defined in while attempting colocation
|
|
|
|
|
|
|
|
Corrected bug in how internal defaults are provided for fill values when casting data to different types for compatibility
|
|
|
|
|
|
|
|
Corrected broken doc link on variable pages with more than one subdirectory in their path (generally seen in h5 files)
|
|
|
|
|
|
|
|
Fixed links to configuration file copy on the report pages (also reformatted other configuration file related information that shows up on the report to appear near the link to the configuration file)
|
|
|
|
|
|
|
|
Added code to check if a configuration file requested by the user is present; Glance will now stop execution with a warning if the file could not be found
|
|
|
|
|
|
|
|
------------
|
|
|
|
|
|
|
|
**version 0.2.6.18**
|
|
|
|
|
|
|
|
Added ability to analyze with 'epsilon_percent', which can be set in the config file, and causes glance to calculate the acceptable epsilon value as a percentage of the A file data value (ie. assumes the A file is the 'truth' data set)
|
|
|
|
|
|
|
|
Added the r-squared correlation coefficient to the statistics calculated in stats and reportGen
|
|
|
|
|
|
|
|
Added 'minimum_acceptable_squared_correlation_coefficient' value, which can be defined in the config file to allow a pass / fail test for variables based on a minimum definition of the r squared correlation coefficient. If this value is defined as None or left undefined the r squared correlation coefficient will not be tested.
|
|
|
|
|
|
|
|
Corrected a bug where some special path characters were not correctly escaped in configuration file path
|
|
|
|
|
|
|
|
Massive internal refactoring to the code in the delta.py module, splitting it into delta.py, stats.py, data.py, and collocation.py
|
|
|
|
|
|
|
|
------------
|
|
|
|
|
|
|
|
**version 0.2.6.17**
|
|
|
|
|
|
|
|
corrected critical bugs related to running colocations on the command line
|
|
|
|
|
|
|
|
------------
|
|
|
|
|
|
|
|
**version 0.2.6.16**
|
|
|
|
|
|
|
|
additional code to allow attributes for the colocation statistics to be added to the file; also added code to write several of these statistics (related to how points were colocated) into the files during colocation
|
|
|
|
|
|
|
|
added the ability to use remote files when using the colocation feature of glance
|
|
|
|
|
|
|
|
additional data filtering now available on colocated data sets so that missing value data points will be considered invalid
|
|
|
|
|
|
|
|
the output files for colocation now include a -colocation suffix to avoid overwriting the original files
|
|
|
|
|
|
|
|
added code to allow vector data to be plotted using the 'magnitudeName', 'directionName', 'magnitudeBName' and 'directionBName' settings in a config file; the plots generated for this feature are preliminary and subject to change in the future
|
|
|
|
|
|
|
|
corrected formula for magnitude, direction to UV vector calculations
|
|
|
|
|
|
|
|
|
|
|
|
corrected colocation counts of unmatched points to ignore invalid data points
|
|
|
|
|
|
|
|
initial release of bin / tuple analysis for large data sets including
|
|
|
|
|
|
|
|
* new 'binIndex', 'binName', 'tupleIndex', and 'tupleName' values that can be set in your config file (the names are optional for display)
|
|
|
|
* new set of histogram plots that includes RMS difference analysis separated along the selected bin dimension
|
|
|
|
* new set of line plots that are generated from randomly selected cases for each section of the aforementioned histogram plots
|
|
|
|
* each set of related histogram and line plots will be separated by a horizontal rule in the report for clarity
|
|
|
|
* all bins will be described in terms of the index along that "bin" dimension
|
|
|
|
* all cases will be described in terms of the original indexes needed to find that sub section of your data (so you can examine that case separately by filtering and rerunning glance)
|
|
|
|
|
|
|
|
corrected loading of missing value for hdf5 files
|
|
|
|
|
|
|
|
corrected bug where invalid file selections in calls to info would crash glance
|
|
|
|
|
|
|
|
fixed minor bug in quiver plotting function
|
|
|
|
|
|
|
|
------------
|
|
|
|
|
|
|
|
**version 0.2.6.15**
|
|
|
|
|
|
|
|
added the colocateData option that allows you to colocate data sets across two files and create modified files containing the colocated data
|
|
|
|
|
|
|
|
added additional basic capabilities to the glance's io module to support the modification of netcdf files
|
|
|
|
|
|
|
|
added the ability to filter on an additional variable data set using the 'variable_to_filter_on_a'/'variable_to_filter_on_b' and 'variable_based_filter_a'/'variable_based_filter_b' settings in a config file (this might be used for something like limiting the winds data to that above a minimum Quality Index value)
|
|
|
|
|
|
|
|
added filter_based_on_additional_data_set_min_max_bounds to the glance.filters
|
|
|
|
|
|
|
|
------------
|
|
|
|
|
|
|
|
**version 0.2.6.14**
|
|
|
|
|
|
|
|
added the ability to compare files without defining lon/lat variables and removed corresponding sections from reports in this case (see --nolonlat option at command line)
|
|
|
|
|
|
|
|
added new image show methods when plotting 2D data without lon/lat variables
|
|
|
|
|
|
|
|
added new line plot methods when plotting 1D data without lon/lat variables (this was intended primarily for IPOPP, but should work for general 1D data as well)
|
|
|
|
|
|
|
|
added filter to collapse a multi-dimensional data set down to one dimension
|
|
|
|
|
|
|
|
added filter to organize ipopp CrIS data into a pseudo "image" for inspection (the organization of the data has not be fully confirmed to match the usage of the instruments; please use as an additional diagnostic tool with care)
|
|
|
|
|
|
|
|
added link to configuration file copy to report (in case where there was one to copy)
|
|
|
|
|
|
|
|
corrected a bug in the "plot only on fail" feature"
|
|
|
|
|
|
|
|
corrected bug where an alternative variable name in the second file would cause glance to incorrectly reject pairs of variable names where one of the names was present in both files
|
|
|
|
|
|
|
|
changed the format of thumbnail names for simplicity
|
|
|
|
|
|
|
|
added some foundation work for vector data plotting (may be available in next release depending on demand)
|
|
|
|
|
|
|
|
rewrote and reorganized plotting code to improve clarity and simplify future maintenance
|
|
|
|
|
|
|
|
reorganized ordering in report/image generation to reduce memory usage and simplify code for future maintenance
|
|
|
|
|
|
|
|
------------
|
|
|
|
|
|
|
|
**version 0.2.6.13**
|
|
|
|
|
|
|
|
added ability to load longitude and latitude data from a different set of files; specified in the config file
|
|
|
|
|
|
|
|
added ability to use the same range when potting the original data sets for a variable; specified in the config file
|
|
|
|
|
|
|
|
fixed algorithm that detects invalid data in the lat/lon data to correctly detect NaN data
|
|
|
|
|
|
|
|
------------
|
|
|
|
|
|
|
|
**version 0.2.6.12**
|
|
|
|
|
|
|
|
Added the ability to control which figures are plotted individually (per variable or for the whole report) using a configuration file
|
|
|
|
|
|
|
|
Added a new filter to allow you to mask out input data that is non-finite or outside of a given set of bounds
|
|
|
|
|
|
|
|
Ploting more than 1,000,000 trouble points will now be done as a contour rather than as points in order to improve speed
|
|
|
|
|
|
|
|
Added ability to clean up memory bloating related to figure generation using additional threads, controlled via config file (Note: This feature can be used in most Unix like machines, but not Mac OSX owning to a threading bug in python in OSX)
|
|
|
|
|
|
|
|
Refactored Glance to present external library functions for stats and reportGen (primarily for use by GeoKennel, but may be used by others in the future)
|
|
|
|
|
|
|
|
------------
|
|
|
|
|
|
|
|
**version 0.2.6.11**
|
|
|
|
|
|
|
|
glance now gets the data type that will be used for scaled data from the type of the loaded offset or scale
|
|
|
|
|
|
|
|
made data type comparison more flexible; types will automatically be upcast if it's possible to do so without losing precision
|
|
|
|
|
|
|
|
add an additional filter to filters.py for rotating the index order in a 3D array
|
|
|
|
|
|
|
|
moved trouble pt statistics to a different set for logical consistency in groupings
|
|
|
|
|
|
|
|
rewrote trouble point statistics' definitions for clarity and to reflect an earlier change in meaning
|
|
|
|
|
|
|
|
moved the output of variable reports into their own directories named with the user defined display names so that variable reports can be generated for virtual data sets from the same variable without conflicting image names
|
|
|
|
|
|
|
|
added greater support for hdf5 files (scaling data in hdf5 is still only partly supported)
|
|
|
|
|
|
|
|
|
|
|
|
(testing possible use of fork to control memory bloating when plotting, this feature may be user accessible in future versions and could be used in Linux only)
|
|
|
|
|
|
|
|
------------
|
|
|
|
|
|
|
|
**version 0.2.6.10**
|
|
|
|
|
|
|
|
fixed a bug that resulted in scaling of missing values (which caused them not to be detected as missing)
|
|
|
|
|
|
|
|
changed to using resizing to create thumbnail images rather than of re-saving in order to improve performance
|
|
|
|
|
|
|
|
corrected some of the statistics tools to handle data sets that are completely filled with missing values
|
|
|
|
|
|
|
|
added check to make sure we were able to find valid bounds around the longitude/latitude before attempting to plot
|
|
|
|
|
|
|
|
swapped to generating reports before figures when using reportGen
|
|
|
|
|
|
|
|
changed scaling formula from pyhdf standard to CF compliant formula
|
|
|
|
|
|
|
|
changed finite tolerance to only consider the differences in finiteness between the two files
|
|
|
|
|
|
|
|
added reporting on % of data matchinf in finite-ness between the files to the main report (since this is the second thing that can cause a variable to fail it's comparison)
|
|
|
|
|
|
|
|
increased precision on the percentages reported for the compared variables on the main report in order to greatly decrease cases where a difference will be rounded up to 100% (and this it looks like there isn't any divergent data when there is some)
|
|
|
|
|
|
|
|
corrected interaction between settings version of shouldIncludeImages and the new plot only on fail feature
|
|
|
|
|
|
|
|
------------
|
|
|
|
|
|
|
|
**version 0.2.6.9**
|
|
|
|
|
|
|
|
in the config file, you can now mark a variable as not intended to include images and it will not be compared to the lon/lat or plotted (this allows any arbitrary variable with matching size between A and B to be compared statistically)
|
|
|
|
|
|
|
|
added 'only_plot_on_fail' per-variable setting for the config file that will cause variables to only generate image comparisons when they fail their tolerance tests
|
|
|
|
|
|
|
|
corrected a bug in the parsing of missing values when using config files
|
|
|
|
|
|
|
|
created a graphics.py for map plotting in glance and removed dependency on Keoni
|
|
|
|
|
|
|
|
refactored use of basemap and calculation of bounding area for map plot to improve speed of figure generation
|
|
|
|
|
|
|
|
added clear/close of figures when they are no longer in use in order to improve memory footprint
|
|
|
|
|
|
|
|
|
|
|
|
Note: With this Glance update Keoni is no longer required for Glance.
|
|
|
|
|
|
|
|
------------
|
|
|
|
|
|
|
|
**version 0.2.6.8**
|
|
|
|
|
|
|
|
glance now clears all figures after use (to save memory and increase performance)
|
|
|
|
|
|
|
|
corrected a bug which caused glance to crash when trying to get fill values from files which did not have defined fill values (also cleaned up some leaked objects in the process)
|
|
|
|
|
|
|
|
changed configuration file so that variables to be analyzed are uniquely identified by their display names (rather than the names they show up in the files with); this will allow multiple virtual data sets derived from one variable to be present in the same configuration file
|
|
|
|
|
|
|
|
a custom color range for plotting a variable can now be defined in the config file
|
|
|
|
|
|
|
|
Note: This version of glance was released concurrently with a new version of the keoni package (0.6.7). The new version of the keoni package includes support for plotting of 1D data arrays, which glance will need in order to handle winds data.
|
|
|
|
|
|
|
|
------------
|
|
|
|
|
|
|
|
**version 0.2.6.7**
|
|
|
|
|
|
|
|
added note when data was filtered to report
|
|
|
|
|
|
|
|
added feature to save a copy of the configuration file with the report to preserve information on data filters that were used
|
|
|
|
|
|
|
|
created set of default filters (located in the new glance.filters)
|
|
|
|
filters to trim the top/bottom/left/right of a data set
|
|
|
|
a filter to take a slice out of a 3D data set
|
|
|
|
a filter to unpack one bit from a packed mask
|
|
|
|
a filter to sort data into ordered bins
|
|
|
|
|
|
|
|
added -f or --fork command line option (available in the config file as settings['doFork']) to allow you to turn on multiprocessing; by default it is now off
|
|
|
|
|
|
|
|
changed format of some settings in config file to simplify logical grouping and code
|
|
|
|
|
|
|
|
added logging details about data sizes when comparison is disqualified on size
|
|
|
|
|
|
|
|
------------
|
|
|
|
|
|
|
|
**version 0.2.6.6**
|
|
|
|
|
|
|
|
added additional statistics for min and max values in both data sets to the general statistics set
|
|
|
|
|
|
|
|
made sure that trouble plot figure handles mismatch of data existence between A and B gracefully
|
|
|
|
|
|
|
|
added ability to define arbitrary filtering functions in the config file for the longitude and latitude of each file and for the data coming from each file on a per-variable basis (these could also be defined as defaults in order to affect all variables)
|
|
|
|
|
|
|
|
added the ability to define display ranges in the config file in order to limit data display when plotting (this doesn't filter data directly but it will change how the contours show in the images)
|
|
|
|
|
|
|
|
added the ability to label the boundaries between data ranges (in the future it may be possible to label the ranges themselves as well)
|
|
|
|
|
|
|
|
cut back on the number of processes that will be generated at any one time; now all the plotting for one variable will be handled in parallel before moving on to a different variable
|
|
|
|
|
|
|
|
refactored delta.diff to correct bug in interaction of ignore mask and plotting individual data sets with mismatching invalid areas; also added more organized output to provide comprehensive mask info (since we've already calculated it anyway) and used this to simplify some code and remove extraneous calculations
|
|
|
|
|
|
|
|
moved the min/max with mask functions into delta.py
|
|
|
|
|
|
|
|
documented investigation notes related to view problems with conical projections
|
|
|
|
|
|
|
|
_create_mapped_figure now finds it's own bounding axes
|
|
|
|
|
|
|
|
corrected possible divide by zero error when getting percentage related information on a mask
|
|
|
|
|
|
|
|
updated glance's longitude and latitude related defaults to function more like it's other defaults (programatically, users should see no change externally) |
