Glance sometimes choses the wrong config file when there are multiple valid config files in a directory
I learned about this bug from Mike Walters at NOAA. It may have also been biting Hong intermittently, but we were never able to reproduce her problems consistently to figure out what was going on. If Mike's analysis is right, it should only affect people who have multiple config files in the directory where the one they're trying to use is stored, and even then it will only happen intermittently. He noticed it because he was running large batches of comparisons through HTCondor and was getting inconsistent results on which variables Glance would analyze in a given run.
I should note, Glance was still copying the requested config file (rather than the one that was actually loaded) into the report directory, so it was difficult to see what was actually happening. On top of fixing the bug, I need to add better debug logging to the code that selects the variables based on the config file.
Here is Mike's final summary of his investigation:
The problem lies on on lines 374-375 of config_organizer.py:
glanceRunConfig = imp.load_module(fileBaseName, file(requestedConfigFile, 'U'), filePath, ('.py' , 'U', 1))
The third argument passed to the imp.load_module function is filePath. Earlier, on line 365, filePath is set to the directory where the config file is located:
(filePath, fileName) = os.path.split(requestedConfigFile)
The Glance documentation says that the third argument is supposed to be the "pathname" of the module file. The wording is rather vague, but I'm fairly certain that this is supposed to be the full path of the module file, including the file name, and not just the directory name. This only became clear to me when I looked at examples I found online, like this one:
http://effbot.org/librarybook/imp.htm
When only the directory path is given, Python is able to guess the correct file name most of the time. Sometimes it guesses wrong though, and loads the wrong config file. When the wrong config file is loaded, the variable list doesn't match the input files, which explains why some or all of the variables were missing from the final report. I figured this out by adding debug statements to show what was actually being loaded. I would imagine that most people only have a few config files in their directory, which probably makes it easier to find the right config. Hence, they never encounter the bug. On the other hand, I have 70+ config files in my directory and I'm running thousands of Glance jobs, so it's not surprising that I ran into the bug.
The problem can be fixed by changing the third argument of imp.load_module from filePath to requestedConfigFile:
glanceRunConfig = imp.load_module(fileBaseName, file(requestedConfigFile, 'U'), requestedConfigFile, ('.py' , 'U', 1))
Making this change solved the problem for me 100% of the time (so far, anyway).