Excessive memory usage
A NetCDF4 byte-sized variable of size 5424×5424×33 would take about 0.9GB, 3.6GB if unpacked to 32-bit integers. Trying to run glance stats
needs about 44GB, which is far more than expected. Similarly large files can use seemingly excessive amounts of memory trying to plot.
Here's a simple test case: test.nc (6.7K) . Despite the small size, it has a 5424×5424×33. glance stats
peaks at about 44GB on my system.
If you set a 32GB virtual memory limit, it fails on my system:
$ ulimit -v 33554432
$ glance stats ./test.nc ./test.nc
--------------------------------
testvar
Traceback (most recent call last):
File "/home/adesmet/bin/glance", line 11, in <module>
load_entry_point('uwglance', 'console_scripts', 'glance')()
File "/home/adesmet/src/glance/pyglance/glance/compare.py", line 1613, in main
rc = lower_locals[args[0].lower()](*args[1:])
File "/home/adesmet/src/glance/pyglance/glance/compare.py", line 1314, in stats
output_channel=toPrintTo)
File "/home/adesmet/src/glance/pyglance/glance/compare.py", line 1083, in stats_library_call
variable_stats = statistics.StatisticalAnalysis.withSimpleData(aData, bData, amiss, bmiss, epsilon=epsilon)
File "/home/adesmet/src/glance/pyglance/glance/stats.py", line 883, in withSimpleData
new_object._create_stats(diffInfo)
File "/home/adesmet/src/glance/pyglance/glance/stats.py", line 910, in _create_stats
self.comparison = NumericalComparisonStatistics(diffInfoObject)
File "/home/adesmet/src/glance/pyglance/glance/stats.py", line 756, in __init__
self.correlation = delta.compute_correlation(aData, bData, valid_in_both) if not noData else np.nan
File "/home/adesmet/src/glance/pyglance/glance/delta.py", line 122, in compute_correlation
toReturn = compute_r_function(good_x_data, good_y_data)[0]
File "/usr/lib/python2.7/dist-packages/scipy/stats/stats.py", line 3018, in pearsonr
xm, ym = x - mx, y - my
MemoryError
I ran into this on real AIT_Framework GOES-16 output data. Trying to run glance reportGen
on my 65GB RAM laptop was failing (and wedging my laptop in the process).
There are more examples (but without the output data!) at #34 (closed), where I did my initial investigation.
My system is Ubuntu 18.04.1 LTS (64-bit), running on a 4-core x86-64 Intel CPU with 64GB of memory.