aeri_quality_control

Installation

1. Install python 3 with anaconda
2. Install dmv from wheel
3. conda install numpy scipy pandas scikit-learn netcdf4
4. Install aeri_tools
5. python setup.py install

Invocation

$ python run_aeri_qc.py

usage: run_aeri_qc.py [-h] [--hint HINT] ftp [sci]

positional arguments:
  ftp
  sci

optional arguments:
  -h, --help   show this help message and exit
  --hint HINT  inspect this file and others in similar time period more
               closely

Visualization

quick_vis.py creates a png representation of the QC output under the ending 'VIS.png' within each day's directory. a text representation listing all tests that are <90% valid is also made in each day's directory under the ending 'REP.txt'. the checks that qualify to be put onto that list also have a corresponding png made of the variables used in the check for easy manual confirmation

to run quick_vis.py on files without running run_aeri_qc.py, invoke in this manner:

$ python quick_vis.py filepath [-s, --skip_number]

positional arguments:
  filepath    can be a QC.nc file, a directory with QC.nc file(s), or a directory of directories with QC.nc files

optional arguments:
  -h, --help    show this help message and exit
  -s, --skip_number   specify number of QC.nc files to skip

Test Documentation

Cooler Check

detector_temp_check

• description: uses power spectral density analysis to find periodicity in 30 minute time series
• short name: Detector Stability
• long name: Cooler Test: Test for strong periodicity in temperature
• parameters: value in return to replace "True" with
• output value: {0,0.4}
• triggered: detector_temp fluctuates in a sinusoidal manner in 30 minute increments

Electronic Check

hbb_temp_outlier_flag

• description: outlier in HBB temperature discovered
• short name: HBB Temp Outlier
• long name:
• parameters: delta_thresh (threshold of how much of a change triggers check)
• output value: {0,1}
• triggered: change in temperature is >0.001 Kelvin from the last non-outlier temperature abb_temp_outlier_flag:
• description: outlier in ABB temperature discovered
• short name: ABB Temp Outlier
• long name:
• parameters: delta_thresh (threshold of how much of a change triggers check)
• output value: {0,1}
• triggered: change in temperature is >0.01 Kelvin from the last non-outlier temperature bst_temp_outlier_flag:
• description: outlier in BB support structure temperature discovered
• short name: BST Temp Outlier
• long name:
• parameters: delta_thresh (threshold of how much of a change triggers check)
• output value: {0,1}
• triggered: change in temperature is >1 Kelvin from the last non-outlier temperature air_interferometer_outlier_flag:
• description: outlier in the air near the interferometer discovered
• short name: Air Interferometer Outlier Check
• long name:
• parameters: delta_thresh (threshold of how much of a change triggers check)
• output value: {0,1}
• triggered: change in temperature is >0.5 Kelvin from the last non-outlier temperature

Global Check

missing_data_flag

• description: record is missing
• short name: Missing Data Flag
• long name:
• parameters:
• output value: {0,1}
• triggered: missingDataFlag is NaN or is 1

Radiometric Check

imaginary_radiance_flag

• description: sky view imaginary radiance is out of range
• short name: Imaginary Radiance
• long name:
• parameters: threshold
• output value: {0,1}
• triggered: skyViewImaginaryRadiance2510_2515 > imaginary_radiance_threshold hbb_std_dev_flag
• description: HBB radiance has too much variance
• short name: HBB Standard Deviation
• long name:
• parameters: number multiplied by standard deviation
• output value: {0,1}
• triggered: HBBviewStdDevRadiance985_990 is greater than 3 standard deviations from its mean hbb_lw_nen_check
• description: Longwave HBB radiance NEN too high
• short name: HBB LW Nen
• long name:
• parameters: bounds of yellow and red zones
• output value: {0, 0.1, 1}
• triggered: 0 if LWW_HBB_NEN inside green zone, 0.1 if LWW_HBB_NEN outside green zone but inside yellow zone, 1 if LWW_HBB_NEN outside both green and yellow zones hbb_sw_nen_check
• description: Shortwave HBB radiance NEN too high
• short name: HBB SW Nen
• long name:
• parameters: bounds of yellow and red zones
• output value: {0, 0.1, 1}
• triggered: 0 if SW_HBB_NEN inside green zone, 0.1 if SW_HBB_NEN outside green zone but inside yellow zone, 1 if SW_HBB_NEN outside both green and yellow zones lw_responsivity_flag
• description: Longwave responsivity outlier
• short name: LW Responsivity
• long name:
• parameters: number of standard deviations used for threshold
• output value: {0,1}
• triggered: zscore of LWresponsivity is greater than 6 sw_responsivity_flag
• description: Shortwave responsivity outlier
• short name: SW Responsivity
• long name:
• parameters: number of standard deviations used for threshold
• output value: {0,1}
• triggered: zscore of sWresponsivity is greater than 6 sky_brightness_temp_spectral_averages_ch1_flag
• description: brightness temp @675cm-1 within 5K of BBsupportStructureTemp
• short name: Sky Brightness Temp Spectral Averages Ch1
• long name:
• parameters: difference between surfaceLayerAirTemp675_680 and BBsupportStructureTemp needed to trigger
• output value: {0,1}
• triggered: surfaceLayerAirTemp675_680 is 20 Kelvin higher or lower than BBsupportStructureTemp or airNearBBsTemp sky_brightness_temp_spectral_averages_ch2_flag
• description: brightness temp @2300cm-1 within 5K of BBsupportStructureTemp
• short name: Sky Brightness Temp Spectral Averages Ch2
• long name:
• parameters: difference between surfaceLayerAirTemp2295_2300 and BBsupportStructureTemp needed to trigger
• output value: {0,1}
• triggered: surfaceLayerAirTemp2295_2300 is 20 Kelvin higher or lower than BBsupportStructureTemp or airNearBBsTemp

Scene Check

hatch_flag

• description: hatch is closed
• short name: Hatch
• long name:
• parameters:
• output value: {0,1}
• triggered: hatch is not open (hatchOpen != 1) and mirror position is not at 'H' or 'A' blackbody scenes safing_flag
• description: hatch moving during calibration view, mirror might have safed
• short name: Safing
• long name:
• parameters:
• output value: {0,1}
• triggered: hatch is in the process of closing (hatchOpen == -3) and the mirror position is at 'H' or 'A' blackbody scenes encoder_check
• description: mirror position not in mirror.beg
• short name: Encoder
• long name:
• parameters:
• output value: {0,1}
• triggered: mirror.beg is not in its designated directory or mirror position is in mirror.beg

State Check

detector_check

• description: detector temp too high
• short name: Detector Temp
• long name:
• parameters: softmax value, upper and lower bounds
• output value: [0,1]
• triggered: between 0 and 1 when temp>80 and temp<90, 1 when temp>90, the rising and falling parts of spikes don't trigger hbb_thermistor_flag
• description: HBB thermistors outside range
• short name: HBB Thermistor
• long name:
• parameters: upper and lower bounds
• output value: {0,1}
• triggered: HBBapexTemp, HBBbottomTemp, or HBBtopTemp is <331 Kelvin or >335 Kelvin abb_thermistor_flag
• description: ABB thermistors outside range
• short name: ABB Thermistor
• long name:
• parameters: upper and lower bounds
• output value: {0,1}
• triggered: ABBapexTemp, ABBbottomTemp, or ABBtopTemp is <200 Kelvin or >318 Kelvin hbb_stable_flag
• description: HBB temperature is changing too quickly
• short name: HBB Stable
• long name:
• parameters: hsr
• output value: {0,1}
• triggered: change in HBBapexTemp, HBBbottomTemp, or HBBtopTemp per second is greater than 0.002

Thermal Check

hbb_covariance_flag

• description: HBB thermistors disagree
• short name: HBB Covariance
• long name:
• parameters:
• output value: {0,1}
• triggered: HBBapexTemp, HBBbottomTemp, HBBtopTemp stray too far from their designated means sce_temp_deviation_check
• description: absolute deviation of temperature too high
• short name: SCE Temp Deviation
• long name:
• parameters:
• output value: [0,0.4]
• triggered: rolling deviation is >0.5 cross_correlation_check
• description: test for correlation between SCEtemp and radiance
• short name: Cross Correlation
• long name:
• parameters:
• output value: [0.05,0.5]
• triggered: SCEtemp and radiance correlate to each other