Skip to content

WIP: Add "--fields" option for glance info

Alan De Smet requested to merge adesmet/UW-Glance:more-info into master

This is a work in progress, do not merge Based on later experience, I want to rethink my design, including if it should be integrated with stats and inspectStats.

--fields is a comma seperated list of fields to be output. By default, it's "filename,variable", which is the old output. You can use any of:

  • filename
  • variable
  • shape - data shape of the variable
  • ATTRIBUTE_NAME - any attribute associated with the variable. If missing, will be an empty string.
  • stats(STAT_NAME) - any statistic as seen in glance inspectStats. The names are exactly as shown there, so "stats(min)" and "stats(finite_count)" are options.

I'd regularly wanted "shape" and the attribute "long_name" for use while exploring new data. A need to create a report of all of our output data drove generalizing the problem and adding stats(STAT_NAME).

Examples:

filename,variable is identical to using glance info without any --fields option at all.

$ glance info FILE.nc
FILE.nc: Latitude
         Longitude
         SensorAzimuth
         SensorZenith
         SolarAzimuth
         SolarZenith
         SpaceMask
$ glance info --fields filename,variable FILE.nc
FILE.nc: Latitude
         Longitude
         SensorAzimuth
         SensorZenith
         SolarAzimuth
         SolarZenith
         SpaceMask

If you aren't using --parsable, it will insert filename at the front if not already present. This is a side effect of striving for maximum backward compatibility.

$ glance info --fields long_name,filename FILE.nc 
FILE.nc: Latitude FILE.nc
         Longitude FILE.nc
         Sensor azimuth angle FILE.nc
         Sensor zenith angle FILE.nc
         Solar azimuth angle FILE.nc
         Solar zenith angle FILE.nc
         Space Mask FILE.nc

I use this command when exploring new files. shape is calculated by glance. units and long_name attributes associated with each variable.

$ glance info --parsable --fields variable,shape,units,long_name FILE.nc 
Latitude        (5424, 5424)    degrees_north   Latitude
Longitude       (5424, 5424)    degrees_east    Longitude
SensorAzimuth   (5424, 5424)    degrees Sensor azimuth angle
SensorZenith    (5424, 5424)    degrees Sensor zenith angle
SolarAzimuth    (5424, 5424)    degrees Solar azimuth angle
SolarZenith     (5424, 5424)    degrees Solar zenith angle
SpaceMask       (5424, 5424)    1       Space Mask

For a recent report, I needed a pile of the calculated statistics. The names are the same names that appear in the statistics reports. Here's a simple example:

$ glance info --parsable --fields 'variable,stats(min),stats(max)' FILE.nc 
Latitude        -81.1488        81.1469
Longitude       -156.195        6.19885
SensorAzimuth   0.0190026       359.981
SensorZenith    0.0151067       90.0
SolarAzimuth    0.000601057     360.0
SolarZenith     22.2221 179.96
SpaceMask       0.0     1.0

Here's the actual command I ran:

glance info --parsable --fields "filename,variable,long_name,units,shape,"\
"stats(min),stats(max),stats(mean),stats(std_val),stats(finite_count),"\
"stats(finite_fraction),stats(missing_count),stats(missing_fraction)" \
*.[nN][Cc] > variable-report.tsv

(This was migrated from https://github.com/evas-ssec/uwglance/pull/13 )

Merge request reports