Investigate Glance exhausting system memory on "simple" reportGen
Alan is investigating this.
Here's a failed run with a 32GB virtual memory limit:
$ ulimit -v 33554432
$ glance --nolonlat --outputpath scratch reportGen ./M6-hackery/output/day/GOESR_ABI_FD_2018352_1800_33_CLOUD_MASK_BL.nc ./2019-01-04/output/day/GOESR_ABI_FD_2018352_1800_33_CLOUD_MASK_BL.nc CloudMaskFlag -vvvvv
INFO:glance.config_organizer:Using Command Line Settings
INFO:glance.compare:Processing File A:
INFO:glance.data:Opening /home/adesmet/data/star-test-data/M6-hackery/output/day/GOESR_ABI_FD_2018352_1800_33_CLOUD_MASK_BL.nc
INFO:glance.data:File md5sum: a806b506c79b2d655f6bc319dd596276
INFO:glance.data:File was last modified: Mon Feb 4 13:22:03 2019
INFO:glance.compare:Processing File B:
INFO:glance.data:Opening /home/adesmet/data/star-test-data/2019-01-04/output/day/GOESR_ABI_FD_2018352_1800_33_CLOUD_MASK_BL.nc
INFO:glance.data:File md5sum: 1d925c9ee2ebe3c6c824405db04de280
INFO:glance.data:File was last modified: Mon Jan 14 15:58:40 2019
INFO:glance.compare:analyzing: CloudMaskFlag
Traceback (most recent call last):
File "/home/adesmet/bin/glance", line 11, in <module>
load_entry_point('uwglance', 'console_scripts', 'glance')()
File "/home/adesmet/src/glance/pyglance/glance/compare.py", line 1616, in main
rc = lower_locals[args[0].lower()](*args[1:])
File "/home/adesmet/src/glance/pyglance/glance/compare.py", line 1397, in reportGen
return reportGen_library_call(a_path, b_path, args[2:], tempOptions)
File "/home/adesmet/src/glance/pyglance/glance/compare.py", line 849, in reportGen_library_call
varRunInfo[EPSILON_KEY], varRunInfo[EPSILON_PERCENT_KEY])
File "/home/adesmet/src/glance/pyglance/glance/stats.py", line 883, in withSimpleData
new_object._create_stats(diffInfo)
File "/home/adesmet/src/glance/pyglance/glance/stats.py", line 910, in _create_stats
self.comparison = NumericalComparisonStatistics(diffInfoObject)
File "/home/adesmet/src/glance/pyglance/glance/stats.py", line 756, in __init__
self.correlation = delta.compute_correlation(aData, bData, valid_in_both) if not noData else np.nan
File "/home/adesmet/src/glance/pyglance/glance/delta.py", line 122, in compute_correlation
toReturn = compute_r_function(good_x_data, good_y_data)[0]
File "/usr/lib/python2.7/dist-packages/scipy/stats/stats.py", line 3018, in pearsonr
xm, ym = x - mx, y - my
MemoryError
Here's one with a 20GB virtual memory limit:
$ ulimit -v 20971520
$ glance --nolonlat --outputpath scratch reportGen ./M6-hackery/output/day/GOESR_ABI_FD_2018352_1800_33_CLOUD_MASK_BL.nc ./2019-01-04/output/day/GOESR_ABI_FD_2018352_1800_33_CLOUD_MASK_BL.nc CloudMaskFlag -vvvvv
INFO:glance.config_organizer:Using Command Line Settings
INFO:glance.compare:Processing File A:
INFO:glance.data:Opening /home/adesmet/data/star-test-data/M6-hackery/output/day/GOESR_ABI_FD_2018352_1800_33_CLOUD_MASK_BL.nc
INFO:glance.data:File md5sum: a806b506c79b2d655f6bc319dd596276
INFO:glance.data:File was last modified: Mon Feb 4 13:22:03 2019
INFO:glance.compare:Processing File B:
INFO:glance.data:Opening /home/adesmet/data/star-test-data/2019-01-04/output/day/GOESR_ABI_FD_2018352_1800_33_CLOUD_MASK_BL.nc
INFO:glance.data:File md5sum: 1d925c9ee2ebe3c6c824405db04de280
INFO:glance.data:File was last modified: Mon Jan 14 15:58:40 2019
INFO:glance.compare:analyzing: CloudMaskFlag
Traceback (most recent call last):
File "/home/adesmet/bin/glance", line 11, in <module>
load_entry_point('uwglance', 'console_scripts', 'glance')()
File "/home/adesmet/src/glance/pyglance/glance/compare.py", line 1616, in main
rc = lower_locals[args[0].lower()](*args[1:])
File "/home/adesmet/src/glance/pyglance/glance/compare.py", line 1397, in reportGen
return reportGen_library_call(a_path, b_path, args[2:], tempOptions)
File "/home/adesmet/src/glance/pyglance/glance/compare.py", line 849, in reportGen_library_call
varRunInfo[EPSILON_KEY], varRunInfo[EPSILON_PERCENT_KEY])
File "/home/adesmet/src/glance/pyglance/glance/stats.py", line 881, in withSimpleData
epsilonValue=epsilon, epsilonPercent=epsilon_percent)
File "/home/adesmet/src/glance/pyglance/glance/data.py", line 327, in __init__
epsilonValue, epsilonPercent)
File "/home/adesmet/src/glance/pyglance/glance/data.py", line 419, in analyze
raw_diff = np.zeros(shape, dtype=sharedType)
MemoryError
Relevant files are on my laptop.