Katie Pitts from NOAA reports that Glance is still having persistent, problematic memory issues:
I'm reopening this thread just to say that I continue to have memory errors when running Glance on directories of GOES-R imagery files that contain channel 2 datasets. Once my EC2 machine's memory was increased from 64GB to 128GB I was able to run through the entire set of files in my directory without error. I was watching the top statistics in the terminal while it was running, and it appears that memory is not always released when moving from one file comparison to a new file comparison, which could be why it keeps dying once it gets to a larger channel 2 file in the list. At one point it was using a whopping 111GB of memory. I attached the traceback of the memory error when the run dies, as well as at the bottom of that text file a runtime warning that showed up on the Glance run that finally made it through all the files.
I'll note that since I last emailed I have installed the latest version of Glance using the latest installation recommendation of doing so using MambaForge.
Since we've found a workaround of increasing the EC2 machine's memory when needed to run these Glance reports, we're okay for now. I'm mainly bringing this to your attention so you can address memory handling in any future version of Glance.
Move Glance to working in python 3.10. Currently Glance is tested and functioning in 3.6, but things have moved on, so the supporting software we rely on isn't all available in 3.6 anymore. This transision will probably require a lot of small clean up. The list below is what I know about so far and will be expanded as I investigate.
I sorted out most of this and released Glance version 0.7.1 last week.
Eva Schiffer (fc46b4d2) at 29 Sep 16:54
changing version for release
Eva Schiffer (4514e284) at 29 Sep 15:34
correcting manifest file
Eva Schiffer (c72658d2) at 29 Sep 15:31
correcting manifest file
Eva Schiffer (4670ed62) at 29 Sep 15:20
adding MANIFEST.in so the non python files will be included in my b...
... and 1 more commit
Eva Schiffer (bfd520fe) at 28 Sep 16:39
moving to importlib.resources as we move to python 3.11
Eva Schiffer (3441448e) at 28 Sep 13:30
updating glance version
Eva Schiffer (731164ba) at 28 Sep 13:21
updating some logging messages to use a more appropriate level
Eva Schiffer (c13dd4b7) at 14 Sep 14:16
replacing usages of pkg_resources with importlib_resources
Eva Schiffer (971d602f) at 13 Sep 16:44
fixing some extranious warnings and moving from os.path to mostly u...
Eva Schiffer (8bc384e4) at 09 Sep 01:26
moving gui code into a subpackage and fixing report template usage
Eva Schiffer (05f69496) at 09 Sep 00:03
cleaning up numpy usage