Skip to content
Snippets Groups Projects
Select Git revision
  • feature-spectral2025
  • master default protected
  • feature-maerisst2024
  • readme_update
  • rocky_linux8
  • new_compare
  • submodule_migration
  • tmp-test-ci
  • maerisst_wrapper
  • buildx-test
  • util_refactoring
  • test_ci
  • segfault_example
  • feature-dmv-ncdf-tests
  • centos-miniconda
  • bbcal_env
  • parser_refactor
  • NBBTemp_test
  • test_large_files
  • rayg-bugfix-maerisst-m64
  • 0.19
  • v0.0
  • 0.18
  • v__test__
  • 0.17
  • 0.16
  • v0.15
  • v0.14
  • v0.13
  • v0.12
  • v0.11
  • v0.1
  • 32bit_stable
33 results
Compare
Denny Hackel's avatar
Merge branch 'cal_val' into 'master'
Denny Hackel authored 
Cal val

See merge request aeri/aeri_armory!3
c59683fb
History
Denny Hackel's avatar c59683fb
History
Name Last commit Last update
.ssh
dev
emissivity
lib
scripts
vips
.gitattributes
Dockerfile
README.md
data.zip
README.md

use_ifg is the preffered branch for 32bit work

master is the preffered branch for 64bit work

Note: " and ' are slightly different in bash. When copying and pasting code, some editors will replace one with the other.

Remove current image

docker rmi aeri_armory

NOTE: Every time that an update is made, you must pull, remove, and remake the current image!

Build image from Dockerfile

docker build -t aeri_armory .

Mount volume that scripts will be used on

docker run --rm -itv <source>:<destination> aeri_armory

source: Absolute path of directory on your local machine in which to store products

destination: Absolute path of in docker container. Note that this does not need to match <source>.

Be careful when copying and pasting; there is a difference between `, ', and ".

Run aeri quality control and visualize results

run_aeri_qc <FILEPATH>

Files are written in FILEPATH unless specified by --qc_dir.

FILEPATH: The directory where the AEYYMMDD ftp subdirectories with the .CXS and .SUM files are located.

quick_vis <FILEPATH>

Files are written in FILEPATH.

FILEPATH: The directory where the .nc file is stored. This will be there the AEYYMMDD ftp subdirectories are located or the filepath specified by --qc_dir.

Interactive shell example:

$ docker run --rm -itv /home/aeri_armory/dev/test_data/aeri_qc:/aeri_qc aeri_armory
# ls /aeri_qc
AE190201  test_output
# run_aeri_qc /aeri_qc
# ls /aeri_qc
190201QC.nc  AE190201  test_output
# quick_vis /aeri_qc
# ls /aeri_qc
190201QC.nc  190201REP.txt  190201VIS.png  AE190201  test_output
# exit

Noninteractive shell example:

$ ls /home/aeri_armory/dev/test_data/aeri_qc
AE190129 test_output
$ docker run --rm -itv /home/aeri_armory/dev/test_data/aeri_qc:/aeri_qc aeri_armory run_aeri_qc /aeri_qc
$ docker run --rm -itv /home/aeri_armory/dev/test_data/aeri_qc:/aeri_qc aeri_armory quick_vis /aeri_qc
$ ls /home/aeri_armory/dev/test_data/aeri_qc
190201QC.nc  190201REP.txt  190201VIS.png  AE190201  test_output

You can pass multiple shell commands to docker using bash -c 'command1; command2; ...':

$ docker run --rm -itv /home/aeri_armory/dev/test_data/aeri_qc:/aeri_qc aeri_armory bash -c 'run_aeri_qc /aeri_qc; quick_vis /aeri_qc'

You can test aeri quality control output by doing the following:

$ docker run --rm -itv /home/aeri_armory/dev/test_data/aeri_qc:/aeri_qc aeri_armory bash -c 'diff <(ncdump /aeri_qc/190201QC.nc) <(ncdump /aeri_qc/test_output/190201QC.nc)'
$ docker run --rm -itv /home/aeri_armory/dev/test_data/aeri_qc:/aeri_qc aeri_armory diff /aeri_qc/190201REP.txt /aeri_qc/test_output/190201REP.txt
$ docker run --rm -itv /home/aeri_armory/dev/test_data/aeri_qc:/aeri_qc aeri_armory diff /aeri_qc/190201VIS.png /aeri_qc/test_output/190201VIS.png

maerisst

Interactive shell example:

$ docker run --rm -itv /home/aeri_armory/dev/test_data/maerisst:/maerisst aeri_armory
# cd /maerisst
# maerisst.exe
 ...
 ----------------------------------------------------
 $Id: maerisst.for,v 1.2 2005-07-12 15:23:06 benh Exp $
 ----------------------------------------------------
 ...
 Enter date (YYMMDD or YYYYMMDD): 960318
 Enter main-program screen-output parameter.
 (1="Display screen messages in MAERISST") 1
# exit

Noninteractive shell example:

$ docker run --rm -itv /home/aeri_armory/dev/test_data/maerisst:/maerisst aeri_armory bash -c 'cd /maerisst; maerisst.exe'
 ...
 ----------------------------------------------------
 $Id: maerisst.for,v 1.2 2005-07-12 15:23:06 benh Exp $
 ----------------------------------------------------
 ...
 Enter date (YYMMDD or YYYYMMDD): 960318
 Enter main-program screen-output parameter.
 (1="Display screen messages in MAERISST") 1

You can test maerisst output by doing the following:

$ docker run --rm -itv /home/aeri_armory/dev/test_data/maerisst:/maerisst aeri_armory diff /maerisst/asciiout.mlt /maerisst/test_output/asciiout.mlt

Run cal val

usage: cal_val_plots.py [-h] [-w WORK_DIR] [-o OUTPUT_DIR] [-r RECORD_RANGE]
                        [-e EXCLUDED_RECORDS] [-s] [-n] [-v]
                        input

Generate cal val plots for IBB and NBB. The first record is record 1

positional arguments:
  input                 Channel 1 input filename in HHMMSSC1.RNC format.
                        Realtive or absolute path

optional arguments:
  -h, --help            show this help message and exit
  -w WORK_DIR, --work-dir WORK_DIR
                        Directory containing bbcal.in files. Defaults to
                        current directory. Relative or absolute path
  -o OUTPUT_DIR, --output-dir OUTPUT_DIR
                        Output file directory. It will be created if it does
                        not exist. Defaults to current directory. Relative or
                        absolute path
  -r RECORD_RANGE, --record-range RECORD_RANGE
                        Record range. Either a number or a string of two
                        numbers separated by a space. Defaults to all records
  -e EXCLUDED_RECORDS, --excluded-records EXCLUDED_RECORDS
                        Records to exclude. Either a number or a string of
                        numbers separated by spaces
  -s, --separate        Plot each record individually. Also generates IBB.avg
                        and NBB.avg files
  -n, --no-spectral     Skip generation of spectral plots and only plot
                        thermistor temperatures and diffmeans.
  -v, --verbose         Each occurrence increases verbosity 1 level through
                        ERROR-WARNING-INFO-DEBUG.

Interactive shell example:

$ docker run --rm -it aeri_armory
# cd /cal_val
# cal_val_plots.py 191010C1.RNC -r "1 4" -o /out -vv
[INFO: 2020-09-10 19:07:34 : __main__] Creating directory: /out
[INFO: 2020-09-10 19:07:34 : __main__] start through end records: 1 - 4
[INFO: 2020-09-10 19:07:34 : __main__] excluded records: 
[INFO: 2020-09-10 19:07:34 : __main__] Record 1 scene mirror position = 83
[INFO: 2020-09-10 19:07:34 : __main__] Record 4 scene mirror position = 84
[root@b0b03380d819 cal_val]# ls /out
aeri_119_3rdbody_191010.png  bbcal.mlt1  bbcal.sum1  bbcal3p.lo1
aeri_119_icebody_191010.png  bbcal.mlt2  bbcal.sum2  bbcal3p.lo2

Noninteractive shell example:

$ docker run --rm -it aeri_armory bash -c 'cd /cal_val; cal_val_plots.py 191010C1.RNC -o /out -r "1 4" -vv; ls /out'
[INFO: 2020-09-10 19:07:34 : __main__] Creating directory: /out
[INFO: 2020-09-10 19:07:34 : __main__] start through end records: 1 - 4
[INFO: 2020-09-10 19:07:34 : __main__] excluded records: 
[INFO: 2020-09-10 19:07:34 : __main__] Record 1 scene mirror position = 83
[INFO: 2020-09-10 19:07:34 : __main__] Record 4 scene mirror position = 84
aeri_119_3rdbody_191010.png  bbcal.mlt1  bbcal.sum1  bbcal3p.lo1
aeri_119_icebody_191010.png  bbcal.mlt2  bbcal.sum2  bbcal3p.lo2

If you are not in the directory containing the bbcal files, you can specify it as the working directory (/cal_val).

$ docker run --rm -itv /home/AE200909_3body:/AE200909_3body aeri_armory bash -c 'cd /AE200909_3body; ls; cal_val_plots.py 200909C1.RNC -w /cal_val -o out'; ls /home/AE200909_3body/out
200909.PAR    200909B1.UVS  200909C2.RNC  200909F2.CSV	RADIANCE.SCR
200909.QC     200909B2.CXS  200909F1.CSV  200909F2.CXS	SUMARY.SCR
200909.SUM    200909B2.UVS  200909F1.CXS  200909F2.UVS
200909B1.CXS  200909C1.RNC  200909F1.UVS  AESITTER.SCR
aeri_120_3rdbody_200909.png	bbcal.mlt2			bbcal.sum2			bbcal3p.lo2
bbcal.mlt1			bbcal.sum1			bbcal3p.lo1

Note that if bbcal runs into a value error with nan values, it will be skipped and a warning will be displayed (use -v to show warning message). Output will still be generated, but may be invalid.

$ docker run --rm -it aeri_armory bash -c 'cd /cal_val; cal_val_plots.py 191010C1.RNC -v'
[WARNING: 2020-09-10 18:45:36 : __main__] Records flagged for missing data: 119, 120. Adding them to excluded records

When plotting, if nans are found in the long or short waves, a warning will be printed with which wave numbers contain nan values. If there is part of the graph with all nans, that file will not be written.

$ docker run --rm -itv /home/aeri_armory/out:/out aeri_armory bash -c 'cd /cal_val; cal_val_plots.py 191010C1.RNC -r "118 120" -v'
[WARNING: 2020-09-10 18:46:45 : __main__] Records flagged for missing data: 119, 120. Adding them to excluded records
[WARNING: 2020-09-10 18:46:46 : __main__] All nans in NBB observed longwave, records 118-118. Skipping plot generation
[WARNING: 2020-09-10 18:46:46 : __main__] nan values found in IBB observed longwave, records 118-118, in the following wave numbers: 1558.2998046875, 1700.0510864257812, 1733.3192443847656

The following commands have the same result:

$ docker run --rm -itv /home/AE200909_3body:/AE200909_3body aeri_armory bash -c 'cd /AE200909_3body; cal_val_plots.py 200909C1.RNC -o out -w /cal_val'
$ docker run --rm -itv /home/AE200909_3body:/AE200909_3body aeri_armory bash -c 'cal_val_plots.py /AE200909_3body/200909C1.RNC -o /AE200909_3body/out -w /cal_val'
$ docker run --rm -itv /home/AE200909_3body:/AE200909_3body aeri_armory bash -c 'cd /cal_val; cal_val_plots.py /AE200909_3body/200909C1.RNC -o /AE200909_3body/out'

Full use of flags:

$ docker run --rm -itv /home/AE200909_3body:/AE200909_3body aeri_armory bash -c 'cd /AE200909_3body; cal_val_plots.py 200909C1.RNC -o out -w /cal_val -r "1 10" -e "$(seq 2 2 10)" -snvv'
[INFO: 2020-09-10 19:34:36 : __main__] start through end records: 1 - 10
[INFO: 2020-09-10 19:34:36 : __main__] excluded records: 2, 4, 6, 8, 10
[INFO: 2020-09-10 19:34:36 : __main__] Record 1 scene mirror position = 83
[INFO: 2020-09-10 19:34:38 : __main__] Record 3 scene mirror position = 83
[INFO: 2020-09-10 19:34:40 : __main__] Record 5 scene mirror position = 83
[INFO: 2020-09-10 19:34:42 : __main__] Record 7 scene mirror position = 83
[INFO: 2020-09-10 19:34:44 : __main__] Record 9 scene mirror position = 83