Skip to content
Snippets Groups Projects
Select Git revision
  • c828cd65f8e73253b4258496feb7e7125bd72b58
  • master default protected
  • use_flight_altitude
  • distribute
4 results

modules

Name Last commit Last update
..
aeri_qc
aeri_qc_web
testing
README.md
package-list.txt
quick_vis.py
run_aeri_qc.py
setup.py
README.md

Installation

  1. Install python 3 with anaconda
  2. Install dmv from wheel
  3. conda install numpy scipy pandas scikit-learn netcdf4
  4. Install aeri_tools
  5. python setup.py install

Invocation

$ python run_aeri_qc.py

usage: run_aeri_qc.py [-h] [--hint HINT] ftp [sci]

positional arguments:
  ftp
  sci

optional arguments:
  -h, --help   show this help message and exit
  --hint HINT  inspect this file and others in similar time period more
               closely

Old Visualization

If you just want to view the results of the QC process, you'll need the ".SUM" and "QC.nc" files

  1. conda install mpld3 flask matplotlib pandas numpy
  2. Download the sum and netcdf files for all the days you'll want to review into an "ftp" style directory tree. e.g rsync -r --progress --include=\*.SUM --include=\*.nc --exclude=\*/\* metobs01::AERI/SSEC\_Rooftop/AE160928 /tmp/ftp/
  3. cd aeri\_qc\_web
  4. python interpret_qc.py <path/to/ftp>
  5. Direct your web browser to http://localhost:5000

New Visualization

quick_vis.py is automatically run when run_aeri_qc.py is run. quick_vis.py creates a png representation of the QC output under the ending 'VIS.png' within each day's directory. a text representation listing all tests that are <90% valid is also made in each day's directory under the ending 'REP.txt'. the checks that qualify to be put onto that list also have a corresponding png made of the variables used in the check for easy manual confirmation

to run quick_vis.py on files without running run_aeri_qc.py, invoke in this manner:

$ python quick_vis.py filepath [-s, --skip_number]

positional arguments:
  filepath    can be a QC.nc file, a directory with QC.nc file(s), or a directory of directories with QC.nc files

optional arguments:
  -h, --help    show this help message and exit
  -s, --skip_number   specify number of QC.nc files to skip

Test Documentation

Cooler Check

detector_temp_check

  • thresholds: {0,0.4}
  • triggered: detector_temp fluctuates in a sinusoidal manner in 30 minute increments

Electronic Check

hbb_temp_outlier_check

  • thresholds: {0,1}
  • triggered: change in temperature is >0.001 Kelvin from the last non-outlier temperature abb_temp_outlier_check:
  • thresholds: {0,1}
  • triggered: change in temperature is >0.01 Kelvin from the last non-outlier temperature bst_temp_outlier_check:
  • thresholds: {0,1}
  • triggered: change in temperature is >1 Kelvin from the last non-outlier temperature air_interferometer_outlier_check:
  • thresholds: {0,1}
  • triggered: change in temperature is >0.5 Kelvin from the last non-outlier temperature

Global Check

missing_data_flag_check

  • thresholds: {0,1}
  • triggered: missingDataFlag is NaN or is 1

Radiometric Check

imaginary_radiance_check

  • thresholds: {0,1}
  • triggered: skyViewImaginaryRadiance2510_2515 > imaginary_radiance_threshold hbb_std_dev_check
  • thresholds: {0,1}
  • triggered: HBBviewStdDevRadiance985_990 is greater than 3 standard deviations from its mean hbb_lw_nen_check
  • thresholds: {0, 0.1, 1}
  • triggered: 0 if LWW_HBB_NEN inside green zone, 0.1 if LWW_HBB_NEN outside green zone but inside yellow zone, 1 if LWW_HBB_NEN outside both green and yellow zones hbb_sw_nen_check
  • thresholds: {0, 0.1, 1}
  • triggered: 0 if SW_HBB_NEN inside green zone, 0.1 if SW_HBB_NEN outside green zone but inside yellow zone, 1 if SW_HBB_NEN outside both green and yellow zones lw_responsivity_check
  • thresholds: {0,1}
  • triggered: zscore of LWresponsivity is greater than 6 sw_responsivity_check
  • thresholds: {0,1}
  • triggered: zscore of sWresponsivity is greater than 6 sky_brightness_temp_spectral_averages_ch1_check
  • thresholds: {0,1}
  • triggered: surfaceLayerAirTemp675_680 is 20 Kelvin higher or lower than BBsupportStructureTemp or airNearBBsTemp sky_brightness_temp_spectral_averages_ch2_check
  • thresholds: {0,1}
  • triggered: surfaceLayerAirTemp2295_2300 is 20 Kelvin higher or lower than BBsupportStructureTemp or airNearBBsTemp

Scene Check

hatch_check

  • thresholds: {0,1}
  • triggered: hatch is not open (hatchOpen != 1) and mirror position is not at 72 or 65 degrees safing_check
  • thresholds: {0,1}
  • triggered: hatch is in the process of closing (hatchOpen == -3) and the mirror position is at 72 or 65 degrees encoder_check
  • thresholds: {0,1}
  • triggered: mirror.beg is not in its designated directory or mirror position is in mirror.beg

State Check

detector_check

  • thresholds: [0,1]
  • triggered: between 0 and 1 when temp>80 and temp<90, 1 when temp>90, the rising and falling parts of spikes don't trigger hbb_thermister_check
  • thresholds: {0,1}
  • triggered: HBBapexTemp, HBBbottomTemp, or HBBtopTemp is <331 Kelvin or >335 Kelvin abb_thermistor_check
  • thresholds: {0,1}
  • triggered: ABBapexTemp, ABBbottomTemp, or ABBtopTemp is <200 Kelvin or >318 Kelvin hbb_stable_check
  • thresholds: {0,1}
  • triggered: change in HBBapexTemp, HBBbottomTemp, or HBBtopTemp per second is greater than 0.002

Thermal Check

hbb_covariance_check

  • thresholds: {0,1}
  • triggered: HBBapexTemp, HBBbottomTemp, HBBtopTemp stray too far from their designated means sce_temp_deviation_check
  • thresholds: [0,0.4]
  • triggered: rolling deviation is >0.5 cross_correlation_check
  • thresholds: [0.05,0.5]
  • triggered: SCEtemp and radiance correlate to each other