Skip to content
Snippets Groups Projects
Select Git revision
  • master default protected
  • feature-maerisst2024
  • readme_update
  • rocky_linux8
  • new_compare
  • submodule_migration
  • tmp-test-ci
  • maerisst_wrapper
  • buildx-test
  • util_refactoring
  • test_ci
  • segfault_example
  • feature-dmv-ncdf-tests
  • centos-miniconda
  • bbcal_env
  • parser_refactor
  • NBBTemp_test
  • test_large_files
  • rayg-bugfix-maerisst-m64
  • mx32
  • 0.19
  • v0.0
  • 0.18
  • v__test__
  • 0.17
  • 0.16
  • v0.15
  • v0.14
  • v0.13
  • v0.12
  • v0.11
  • v0.1
  • 32bit_stable
33 results
Compare
Name Last commit Last update
.ssh
calibration_validation
tests
utils
.gitattributes
.gitlab-ci.sh
.gitlab-ci.yml
Dockerfile
README.md
README.md

Overview

AERI Armory is a suite of software tools for working with AERI data. The tools are publicly available to all AERI users, and are containerized in Docker so that they can be ran on any platform supporting Docker

Feedback

Please contact us with any feedback, including bug reports, feature requests, or software contributions: https://www.ssec.wisc.edu/aeri/contact-us

Installation Procedure

For Users

  1. Install Docker: https://www.docker.com/products/docker-desktop
  2. Start Docker and confirm that it’s running: $ docker --version
  3. Pull the latest version of the AERI Armory and tag the docker image with the specified name below:
    • $ docker pull gitlab.ssec.wisc.edu:5555/aeri/aeri_armory
    • $ docker image tag gitlab.ssec.wisc.edu:5555/aeri/aeri_armory aeri_armory

AERI Armory is now installed and ready to run.

For Developers

  1. Verify that git, git lfs, and docker are installed:
    • $ git --version; git lfs --version; docker --version
  2. Clone the aeri_armory gitlab repository:
    • $ git clone https://gitlab.ssec.wisc.edu/aeri/aeri_armory.git
  3. Build from the Dockerfile:
    • $ docker build -t aeri_armory aeri_armory

Software Update

  1. Untag the previous docker image of AERI Armory:
    • $ docker rmi aeri_armory
  2. Pull and tag the latest version of the AERI Armory:
    • $ docker pull gitlab.ssec.wisc.edu:5555/aeri/aeri_armory
    • $ docker image tag gitlab.ssec.wisc.edu:5555/aeri/aeri_armory aeri_armory

AERI Armory is now updated and ready to run.

Uninstallation

Remove and untag the docker images of AERI Armory:

  • $ docker rmi aeri_armory
  • $ docker rmi gitlab.ssec.wisc.edu:5555/aeri/aeri_armory

General Information

This README can be found in the following locations:

  1. Inside the docker container: /README.md
  2. https://gitlab.ssec.wisc.edu/aeri/aeri_armory

Notes

  • Every time that an update is made, you must pull or rebuild the current image and remake any current containers!
  • Be careful when copying and pasting; there is a difference between `, ', and ".
  • " and ' are slightly different in bash. When copying and pasting code, some editors will replace one with the other.
  • If on Windows, use %cd% in place of $PWD.
  • If git cloning the aeri_armory repository, git lfs MUST be installed or else test data files get corrupted.
  • Set the PYTHONWARNINGS variable equal to default to see hidden errors.
  • The preferred branch for 32bit work is use_ifg.
  • The preferred branch for 64bit work is master.

Docker Information

Docker is a product that uses virtualization to deliver software in packages called containers. A container is a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another. Every container is an instance of a pre-built image, which describes the environment for a container to run in.

Key Concepts

  • A container is spawned by the docker run command.
  • Each container is an isolated entity which is not tied to local directories by default.
    • The flag -v is used to mount a local directory to a docker container.
    • You can mount multiple local directories.
  • If inside an interactive docker container, type "exit" and enter.
  • You can pass environment variables using the -e flag.
  • You can set the default working directory using the -w flag.

Useful Commands

  • List all docker images: $ docker image ls -a
  • List all docker containers: $ docker container ls -a
  • The following command will clear all docker objects, which useful if you want to start from a clean slate: $ docker system prune --volumes -af
  • Pass multiple shell commands to docker: bash -c 'command1; command2; ...'

Quick Start

$ docker run --rm -it -v <data_dir>:<destination> aeri_armory UTILITY_SCRIPT ...
  • <data_dir>: Absolute path of directory on your local machine to read data from and write data to.
  • <destination>: Absolute path of <data_dir> in docker container. This does not need to match <data_dir>.

If input and output are not located in the same directory, you can mount both, then specify the output location in the script you decide to run:

$ docker run --rm -it -v <input_dir>:/input -v <output_dir>:/output aeri_armory UTILITY_SCRIPT ...

You are not restricted to /input and /output, they're just intuitive:

  • <input_dir>: Absolute path of directory on your local machine to read data from.
  • <output_dir>: Absolute path of directory on your local machine to write data to.

Quality Control and Visualization

Summary

Run the AERI Quality Control algorithm on AERI data and visualize the results.

Input Requirements

  • Directory containing AEYYMMDD subdirectories. Each AEYYMMDD subdirectory must contain:
    • YYMMDDB1.CXS
    • YYMMDD.SUM

Output

  • YYYYMMDDQC.nc files with the QC results
  • YYYYMMDDVIS.png files with the QC visualization
  • YYYYMMDDREP.txt files

quality_control.py

usage: quality_control.py [-h] [--output OUTPUT] [--hint HINT] [-f] [--skip-quick-vis] [--skip-qc] [--exclude EXCLUDE] [-v] input_dir [sci]

positional arguments:
  input_dir             The AEYYMMDD directory or the directory containing AEYYMMDD subdirectories where the .CXS and .SUM files are located
  sci

optional arguments:
  -h, --help            show this help message and exit
  --output OUTPUT, -o OUTPUT
                        Directory where QC netcdf files are stored. Defaults to the same directory as input_dir
  --hint HINT           inspect this file and others in similar time period more closely
  -f, --force           Overwrite any existing QC files in output directory
  --skip-quick-vis      Skip the generation of quick vis images
  --skip-qc             Skip the generation of quality control files. Changes input_dir to take a directory containing QC netcdf output
  --exclude EXCLUDE, -e EXCLUDE
                        A string of YYMMDD separated by spaces to exclude from running qc and/or quick_vis.
  -v, --verbose         Each occurrence increases verbosity 1 level through ERROR-WARNING-INFO-DEBUG.

Shell Example

Produce both QC and visual results:

$ docker run --rm -it -v <data_dir>:/aeri_qc aeri_armory quality_control.py /aeri_qc/AEYYMMDD -o /aeri_qc/output -vv -f
[INFO: 2021-12-21 16:47:38 : aeri_qc.main] Performing quality control for /aeri_qc/AEYYMMDD
[INFO: 2021-12-21 16:48:23 : plotting.quick_vis] Running quick vis on /aeri_qc/output/20YYMMDDQC.nc

Produce only QC output:

$ docker run --rm -it -v <data_dir>:/aeri_qc aeri_armory quality_control.py /aeri_qc/AEYYMMDD -o /aeri_qc/output -vv -f --skip-quick-vis
[INFO: 2021-12-21 16:47:38 : aeri_qc.main] Performing quality control for /aeri_qc/AEYYMMDD

Produce visual results from QC files:

$ docker run --rm -it -v <data_dir>:/aeri_qc aeri_armory quality_control.py /aeri_qc/output -vv --skip-qc
[INFO: 2021-12-21 16:48:23 : plotting.quick_vis] Running quick vis on /aeri_qc/output/20YYMMDDQC.nc

MAERISST

Summary

Sea surface skin temperature retrieval for Marine-AERI observations.

Input Requirements

  • ftp directory of M-AERI data
  • mirror.beg file

Output

  • .mlt file with the retrieval results
  • .ASC (ASCII) file with the retrieval results

Shell Example

$ docker run --rm -it -v <data_dir>:/maerisst aeri_armory \
    bash -c 'cd /maerisst; maerisst.exe; \
    mkdir output; mv asciiout.mlt TSKINDMP.ASC output'
 ...
 ----------------------------------------------------
 $Id: maerisst.for,v 1.2 2005-07-12 15:23:06 benh Exp $
 ----------------------------------------------------
 ...
 Enter date (YYMMDD or YYYYMMDD): 960318
 Enter main-program screen-output parameter.
 (1="Display screen messages in MAERISST") 1

Run cal val

Summary

Validate AERI calibration results.

Input Requirements

  • AEYYMMDD
    • YYMMDDC1.RNC
    • YYMMDDC2.RNC

Output

  • .png files containing plots of the calibration
  • bbcal files containing raw data and debug information

cal_val.py

usage: cal_val.py [-h] [-o OUTPUT_DIR] [-r RECORD_RANGE] [-e EXCLUDED_RECORDS] [-s] [-n] [-v] input

Generate cal val plots for IBB and NBB. The first record is record 1

positional arguments:
  input                 Channel 1 input filename in HHMMSSC1.RNC format. Realtive or absolute path

optional arguments:
  -h, --help            show this help message and exit
  -o OUTPUT_DIR, --output-dir OUTPUT_DIR
                        Output file directory. It will be created if it does not exist. Defaults to current directory. Relative or absolute path
  -r RECORD_RANGE, --record-range RECORD_RANGE
                        Record range. Either a number or a string of two numbers separated by a space. Defaults to all records
  -e EXCLUDED_RECORDS, --excluded-records EXCLUDED_RECORDS
                        Records to exclude. Either a number or a string of numbers separated by spaces
  -s, --separate        Plot each record individually. Also generates IBB.avg and NBB.avg files
  -n, --no-spectral     Skip generation of spectral plots and only plot thermistor temperatures and diffmeans.
  -v, --verbose         Each occurrence increases verbosity 1 level through ERROR-WARNING-INFO-DEBUG.

Shell Examples

$ docker run --rm -it -v <data_dir>:/cal_val aeri_armory cal_val.py /cal_val/AEYYMMDD -o /cal_val/output -r "1 4" -vv
[INFO: 2021-09-28 18:44:56 : __main__] start through end records: 1 - 4
[INFO: 2021-09-28 18:44:56 : __main__] excluded records: 
[INFO: 2021-09-28 18:44:56 : __main__] Record 1 scene mirror position = 83
[INFO: 2021-09-28 18:44:56 : __main__] Record 4 scene mirror position = 84

Note that if bbcal runs into a value error with nan values, it will be skipped and a warning will be displayed (use -v to show warning message). Output will still be generated, but may be invalid.

$ docker run --rm -it -v <data_dir>:/cal_val aeri_armory cal_val.py /cal_val/AEYYMMDD -v
[WARNING: 2020-09-10 18:45:36 : __main__] Records flagged for missing data: 119, 120. Adding them to excluded records

When plotting, if nans are found in the long or short waves, a warning will be printed with which wave numbers contain nan values. If there is part of the graph with all nans, that file will not be written.

$ docker run --rm -it -v <data_dir>:/cal_val aeri_armory cal_val.py /cal_val/AEYYMMDD -r "118 120" -v
[WARNING: 2020-09-10 18:46:45 : __main__] Records flagged for missing data: 119, 120. Adding them to excluded records
[WARNING: 2020-09-10 18:46:46 : __main__] All nans in NBB observed longwave, records 118-118. Skipping plot generation
[WARNING: 2020-09-10 18:46:46 : __main__] nan values found in IBB observed longwave, records 118-118, in the following wave numbers: 1558.2998046875, 1700.0510864257812, 1733.3192443847656

The following commands have the same result:

$ docker run --rm -it -v <data_dir>:/cal_val aeri_armory cal_val.py /cal_val/AEYYMMDD -o /cal_val/output
$ docker run --rm -it -v <data_dir>:/cal_val -w /cal_val aeri_armory cal_val.py AEYYMMDD -o /cal_val/output

Full use of flags:

$ docker run --rm -it -v <data_dir>:/cal_val aeri_armory cal_val.py /cal_val/AEYYMMDD -o /cal_val/output -r "1 10" -e "$(seq 2 2 10)" -snvv
[INFO: 2020-09-10 19:34:36 : __main__] start through end records: 1 - 10
[INFO: 2020-09-10 19:34:36 : __main__] excluded records: 2, 4, 6, 8, 10
[INFO: 2020-09-10 19:34:36 : __main__] Record 1 scene mirror position = 83
[INFO: 2020-09-10 19:34:38 : __main__] Record 3 scene mirror position = 83
[INFO: 2020-09-10 19:34:40 : __main__] Record 5 scene mirror position = 83
[INFO: 2020-09-10 19:34:42 : __main__] Record 7 scene mirror position = 83
[INFO: 2020-09-10 19:34:44 : __main__] Record 9 scene mirror position = 83

Detector Nonlinearity Calculation

Summary

Calculate the nonlinearity parameter for the channel 1 detector, for use in nlapp.sip.

Input Requirements

  • YYMMDDB1.CXS
  • YYMMDDF1.CXS
  • aeriemis.asc
  • contrl4.in
  • contrl5.in

Output

  • ARRAYYMMDD
    • b1
      • contrl-b1.in
      • NLAPP.PAR
      • plots
        • CXS
          • .png files
        • IFG
          • .png files
        • responsivity
          • .png files
    • f1
      • contrl-f1.in
      • NLAPP.PAR
      • plots
        • CXS
          • .png files
        • IFG
          • .png files
        • responsivity
          • .png files
    • NLAPP.SIP

nonlinearity.py

Usage: 
nonlinearity.py

Purpose: This program computes the amplitude of non-linear terms contributing to instrument responsivity.
Updated for AERI-01

Usage: nonlinearity.py [options] [paramfile]

Based on Fortran NLI Version 2.7 from 27 Jan 97


Options:
  -h, --help      show this help message and exit
  -o OUTPUT_DIR   Specify output directory to use or be created. Defaults to
                  working directory
  -u              Use UW-AERI configuration
  -v              Enable verbose output
  -a              Average NLAPP.PAR values
  --plotall       Plot everything
  -p PLOT1        Plot a single data set [i(fg), c(xs), r(esponsivity)]
  --zpd1          Use erroneous ft2ifg settings from pre- r2949 ft2ifg.for.
  --scale=SCALE   Scale plot output from default values

Shell Example

$ docker run --rm -it -v <data_dir>:/nli aeri_armory nonlinearity.py -o /nli/output -a --plotall /nli/AEYYMMDD -v

DMV to netCDF converter

Summary

Convert AERI DMV data files into NetCDF format.

Input Requirements

One of:

  • .CXS
  • .SUM
  • .RNC

Output

  • converted .nc file

dmv_to_ncdf.py

usage: dmv_to_ncdf.py [-h] [-v] [-o OUTPUT_DIR] [-e EXTENSIONS] input

Convert DMV files to netCDF4 format.

positional arguments:
  input                 The file, AEYYMMDD directory, or directory containing AEYYMMDD subdirectories to convert to netCDF4 format

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         show output to terminal
  -o OUTPUT_DIR, --output-dir OUTPUT_DIR
                        output directory name
  -e EXTENSIONS, --extensions EXTENSIONS
                        A string of DMV file extensions, separated by spaces, to convert. Defaults to "CSX SUM RNC"

Shell Example

$ docker run --rm -it -v <data_dir>:/dmv_to_ncdf aeri_armory dmv_to_ncdf.py /dmv_to_ncdf/AEYYMMDD -o /dmv_to_ncdf/output -vv
[INFO: 2021-12-21 17:54:09 : __main__] Processing /dmv_to_ncdf/AE190923/190923F1.CXS
STOP *** end dmvtocdf.exe ***

Tests

Test Data Location

Currently, test data is only located in the gitlab repository: aeri_armory/tests

Summary

Run test_suite.sh on aeri_armory test data. This generates user output inside the test data directory which is then compared to the expected output. Log files can also be found inside the test data directory containing runtime output and/or error messages (if applicable).

Note that test_suite.sh uses PWD by default, so it must be run inside the test data directory unless another directory is passed as an argument like test_suite.sh /tests

The test data directory must have this structure to generate the user_output folder in each tests sub-directory

--- tests
  |--- aeri_qc
  |  |--- expected_output
  |  |--- input_data
  |
  |--- cal_val
  |  |--- expected_output
  |  |--- input_data
  |
  |--- nli
  |  |--- expected output
  |  |--- input_data
  |
  |--- dmv_to_ncdf
  |  |--- expected_output
  |  |--- input_data

You can skip tests by assigning the variables below with any non-empty values respectively:

  • SKIP_AERI_QC
  • SKIP_CAL_VAL
  • SKIP_NLI
  • SKIP_DMV_TO_NCDF

Shell Example

$ docker run --rm -it -v <test_dir>:/tests aeri_armory test_suite.sh /tests
running aeri qc...
running quick vis...
running nc diff...
running REP.txt diff...
running png diff...
SUCCESS: aeri_qc
running cal_val...
running diff...
SUCCESS: cal_val
running nli...
running diff...
SUCCESS: nli
running dmv to ncdf...
running diff...
SUCCESS: dmv_to_ncdf

To skip aeri_qc and cal_val:

$ docker run --rm -e SKIP_AERI_QC=foo -e SKIP_CAL_VAL=bar -it -v <test_dir>:/tests aeri_armory test_suite.sh /tests
running nli...
running diff...
SUCCESS: nli
running dmv to ncdf...
running diff...
SUCCESS: dmv_to_ncdf