README.md 15.2 KB
Newer Older
1
# Overview
2
3
4

AERI Armory is a suite of software tools for working with AERI data. The tools are publicly available to all AERI users,
and are containerized in Docker so that they can be ran on any platform supporting Docker
5
6

# Feedback
7
8
9

Please contact us with any feedback, including bug reports, feature requests, or software
contributions: https://www.ssec.wisc.edu/aeri/contact-us
10
11

# Installation Procedure
William Roberts's avatar
William Roberts committed
12

William Roberts's avatar
William Roberts committed
13
### For Users
14

15
16
17
18
19
1. Install Docker: https://www.docker.com/products/docker-desktop
2. Start Docker and confirm that it’s running: `$ docker --version`
3. Pull the latest version of the AERI Armory and tag the docker image with the specified name below:
    - `$ docker pull gitlab.ssec.wisc.edu:5555/aeri/aeri_armory`
    - `$ docker image tag gitlab.ssec.wisc.edu:5555/aeri/aeri_armory aeri_armory`
William Roberts's avatar
William Roberts committed
20

21
AERI Armory is now installed and ready to run.
William Roberts's avatar
William Roberts committed
22

William Roberts's avatar
William Roberts committed
23
### For Developers
24

25
26
27
28
29
30
1. Verify that git, git lfs, and docker are installed:
    - `$ git --version; git lfs --version; docker --version`
2. Clone the aeri_armory gitlab repository:
    - `$ git clone https://gitlab.ssec.wisc.edu/aeri/aeri_armory.git`
3. Build from the Dockerfile:
    - `$ docker build -t aeri_armory aeri_armory`
31

32
# Software Update
33

34
35
36
37
38
1. Untag the previous docker image of AERI Armory:
    - `$ docker rmi aeri_armory`
2. Pull and tag the latest version of the AERI Armory:
    - `$ docker pull gitlab.ssec.wisc.edu:5555/aeri/aeri_armory`
    - `$ docker image tag gitlab.ssec.wisc.edu:5555/aeri/aeri_armory aeri_armory`
39

40
41
AERI Armory is now updated and ready to run.

42
# Uninstallation
43

44
Remove and untag the docker images of AERI Armory:
45

46
47
- `$ docker rmi aeri_armory`
- `$ docker rmi gitlab.ssec.wisc.edu:5555/aeri/aeri_armory`
48

49
# General Information
50
51
52
53
54
55

This README can be found in the following locations:

1. Inside the docker container: /README.md
2. https://gitlab.ssec.wisc.edu/aeri/aeri_armory

William Roberts's avatar
William Roberts committed
56
### Notes
57

William Roberts's avatar
William Roberts committed
58
- Every time that an update is made, you must pull or rebuild the current image and remake any current containers!
59
60
61
62
- Be careful when copying and pasting; there is a difference between `, ', and ".
- `"` and `'` are slightly different in bash. When copying and pasting code, some editors will replace one with the
  other.
- If on Windows, use `%cd%` in place of `$PWD`.
63
- If git cloning the aeri_armory repository, git lfs MUST be installed or else test data files get corrupted.
64
65
66
67
- Set the `PYTHONWARNINGS` variable equal to `default` to see hidden errors.
- The preferred branch for 32bit work is `use_ifg`.
- The preferred branch for 64bit work is `master`.

William Roberts's avatar
William Roberts committed
68
# Docker Information
69
70
71
72
73
74

Docker is a product that uses virtualization to deliver software in packages called containers. A container is a
standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably
from one computing environment to another. Every container is an instance of a pre-built image, which describes the
environment for a container to run in.

William Roberts's avatar
William Roberts committed
75
### Key Concepts
76
77
78
79
80

- A container is spawned by the `docker run` command.
- Each container is an isolated entity which is not tied to local directories by default.
    - The flag `-v` is used to mount a local directory to a docker container.
    - You can mount multiple local directories.
William Roberts's avatar
William Roberts committed
81
- If inside an interactive docker container, type "exit" and enter.
82
83
84
- You can pass environment variables using the `-e` flag.
- You can set the default working directory using the `-w` flag.

William Roberts's avatar
William Roberts committed
85
### Useful Commands
86
87
88
89
90
91
92
93
94
95

- List all docker images: `$ docker image ls -a`
- List all docker containers: `$ docker container ls -a`
- The following command will clear all docker objects, which useful if you want to start from a clean
  slate: `$ docker system prune --volumes -af`
- Pass multiple shell commands to docker: `bash -c 'command1; command2; ...'`

# Quick Start

```console
William Roberts's avatar
William Roberts committed
96
$ docker run --rm -it -v <data_dir>:<destination> aeri_armory UTILITY_SCRIPT ...
97
98
99
100
101
```

- `<data_dir>`: Absolute path of directory on your local machine to read data from and write data to.
- `<destination>`: Absolute path of `<data_dir>` in docker container. This does not need to match `<data_dir>`.

William Roberts's avatar
William Roberts committed
102
103
If input and output are not located in the same directory, you can mount both, then specify the output location in the
script you decide to run:
104
105

```console
William Roberts's avatar
William Roberts committed
106
$ docker run --rm -it -v <input_dir>:/input -v <output_dir>:/output aeri_armory UTILITY_SCRIPT ...
107
108
109
110
111
112
```

You are not restricted to `/input` and `/output`, they're just intuitive:

- `<input_dir>`: Absolute path of directory on your local machine to read data from.
- `<output_dir>`: Absolute path of directory on your local machine to write data to.
113
114
115

# Quality Control and Visualization

116
117
### Summary

118
119
Run the AERI Quality Control algorithm on AERI data and visualize the results.

William Roberts's avatar
William Roberts committed
120
### Input Requirements
121
122
123
124

- Directory containing AEYYMMDD subdirectories. Each AEYYMMDD subdirectory must contain:
    - YYMMDDB1.CXS
    - YYMMDD.SUM
125

126
### Output
127

128
129
130
131
132
- YYYYMMDDQC.nc files with the QC results
- YYYYMMDDVIS.png files with the QC visualization
- YYYYMMDDREP.txt files

### run_aeri_qc.py
133

William Roberts's avatar
William Roberts committed
134
```console
William Roberts's avatar
William Roberts committed
135
usage: run_aeri_qc.py [-h] [--output OUTPUT] [--hint HINT] [-f] [-v] input_dir [sci]
136

William Roberts's avatar
William Roberts committed
137
positional arguments:
William Roberts's avatar
William Roberts committed
138
  input_dir             The AEYYMMDD directory or the directory containing AEYYMMDD subdirectories where the .CXS and .SUM files are located
William Roberts's avatar
William Roberts committed
139
  sci
140

William Roberts's avatar
William Roberts committed
141
142
143
optional arguments:
  -h, --help            show this help message and exit
  --output OUTPUT, -o OUTPUT
William Roberts's avatar
William Roberts committed
144
                        Directory where QC netcdf files are stored. Defaults to the same directory as input_dir
William Roberts's avatar
William Roberts committed
145
146
147
148
  --hint HINT           inspect this file and others in similar time period more closely
  -f, --force           Overwrite any existing QC files in output directory
  -v, --verbose         Each occurrence increases verbosity 1 level through ERROR-WARNING-INFO-DEBUG.
```
149

150
151
### quick_vis.py

William Roberts's avatar
William Roberts committed
152
153
```console
usage: quick_vis.py [-h] [--output OUTPUT] [--exclude EXCLUDE] filepath
154

William Roberts's avatar
William Roberts committed
155
156
positional arguments:
  filepath              path to directory containing QC netcdf output
157

William Roberts's avatar
William Roberts committed
158
159
160
161
162
163
164
optional arguments:
  -h, --help            show this help message and exit
  --output OUTPUT, -o OUTPUT
                        Directory to write images to. Defaults to directory containg QC netcdf files provided
  --exclude EXCLUDE, -e EXCLUDE
                        QC filenames to exlude from quick_vis. A string of QC filenames separated by spaces
```
165

William Roberts's avatar
William Roberts committed
166
### Shell Example
167
168

```console
Marco Kurzynski's avatar
Marco Kurzynski committed
169
$ docker run --rm -it -v <data_dir>:/aeri_qc aeri_armory run_aeri_qc.py /aeri_qc/AEYYMMDD -o /aeri_qc/output -vv -f
170
INFO:aeri_qc.main:Performing quality control for /aeri_qc/AEYYMMDD/YYMMDDB1.CXS
William Roberts's avatar
William Roberts committed
171
$ docker run --rm -it -v <data_dir>:/aeri_qc aeri_armory quick_vis.py /aeri_qc/output
172
173
174
175
```

Or equivalently

Marco Kurzynski's avatar
Marco Kurzynski committed
176
```console
Marco Kurzynski's avatar
Marco Kurzynski committed
177
$ docker run --rm -it -v <data_dir>:/aeri_qc aeri_armory run_aeri_qc.py bash -c 'run_aeri_qc.py /aeri_qc/AEYYMMDD -o /aeri_qc/output -vv -f; quick_vis.py /aeri_qc/output'
178
INFO:aeri_qc.main:Performing quality control for /aeri_qc/AEYYMMDD/YYMMDDB1.CXS
William Roberts's avatar
William Roberts committed
179
```
William Roberts's avatar
William Roberts committed
180

181
182
# MAERISST

183
184
### Summary

185
186
Sea surface skin temperature retrieval for Marine-AERI observations.

187
188
189
190
### Input Requirements

- ftp directory of M-AERI data
- mirror.beg file
191

192
193
194
195
### Output

- .mlt file with the retrieval results
- .ASC (ASCII) file with the retrieval results
196

William Roberts's avatar
William Roberts committed
197
### Shell Example
198

Marco Kurzynski's avatar
Marco Kurzynski committed
199
```console
William Roberts's avatar
William Roberts committed
200
$ docker run --rm -it -v <data_dir>:/maerisst aeri_armory \
201
    bash -c 'cd /maerisst; maerisst.exe; \
202
    mkdir output; mv asciiout.mlt TSKINDMP.ASC output'
203
204
205
206
207
208
209
210
 ...
 ----------------------------------------------------
 $Id: maerisst.for,v 1.2 2005-07-12 15:23:06 benh Exp $
 ----------------------------------------------------
 ...
 Enter date (YYMMDD or YYYYMMDD): 960318
 Enter main-program screen-output parameter.
 (1="Display screen messages in MAERISST") 1
William Roberts's avatar
William Roberts committed
211
212
```

William Roberts's avatar
William Roberts committed
213
214
# Run cal val

215
216
217
218
### Summary

Validate AERI calibration results.

William Roberts's avatar
William Roberts committed
219
### Input Requirements
220

Marco Kurzynski's avatar
Marco Kurzynski committed
221
222
223
- AEYYMMDD
    - YYMMDDC1.RNC
    - YYMMDDC2.RNC
224
225
226

### Output

227
- .png files containing plots of the calibration
Marco Kurzynski's avatar
Marco Kurzynski committed
228
- bbcal files containing raw data and debug information
229
230
231

### cal_val_plots.py

Marco Kurzynski's avatar
Marco Kurzynski committed
232
```console
233
usage: cal_val_plots.py [-h] [-o OUTPUT_DIR] [-r RECORD_RANGE] [-e EXCLUDED_RECORDS] [-s] [-n] [-v] input
William Roberts's avatar
William Roberts committed
234

William Roberts's avatar
William Roberts committed
235
Generate cal val plots for IBB and NBB. The first record is record 1
William Roberts's avatar
William Roberts committed
236
237

positional arguments:
238
  input                 Channel 1 input filename in HHMMSSC1.RNC format. Realtive or absolute path
William Roberts's avatar
William Roberts committed
239
240
241
242

optional arguments:
  -h, --help            show this help message and exit
  -o OUTPUT_DIR, --output-dir OUTPUT_DIR
243
                        Output file directory. It will be created if it does not exist. Defaults to current directory. Relative or absolute path
William Roberts's avatar
William Roberts committed
244
  -r RECORD_RANGE, --record-range RECORD_RANGE
245
                        Record range. Either a number or a string of two numbers separated by a space. Defaults to all records
246
  -e EXCLUDED_RECORDS, --excluded-records EXCLUDED_RECORDS
247
248
249
250
                        Records to exclude. Either a number or a string of numbers separated by spaces
  -s, --separate        Plot each record individually. Also generates IBB.avg and NBB.avg files
  -n, --no-spectral     Skip generation of spectral plots and only plot thermistor temperatures and diffmeans.
  -v, --verbose         Each occurrence increases verbosity 1 level through ERROR-WARNING-INFO-DEBUG.
William Roberts's avatar
William Roberts committed
251
```
252

William Roberts's avatar
William Roberts committed
253
### Shell Examples
254

255
```console
Marco Kurzynski's avatar
Marco Kurzynski committed
256
$ docker run --rm -it -v <data_dir>:/cal_val aeri_armory cal_val_plots.py /cal_val/AEYYMMDD -o /cal_val/output -r "1 4" -vv
257
258
259
260
[INFO: 2021-09-28 18:44:56 : __main__] start through end records: 1 - 4
[INFO: 2021-09-28 18:44:56 : __main__] excluded records: 
[INFO: 2021-09-28 18:44:56 : __main__] Record 1 scene mirror position = 83
[INFO: 2021-09-28 18:44:56 : __main__] Record 4 scene mirror position = 84
261
```
William Roberts's avatar
William Roberts committed
262

263
264
Note that if bbcal runs into a value error with nan values, it will be skipped and a warning will be displayed
(use -v to show warning message). Output will still be generated, but may be invalid.
265

Marco Kurzynski's avatar
Marco Kurzynski committed
266
```console
Marco Kurzynski's avatar
Marco Kurzynski committed
267
$ docker run --rm -it -v <data_dir>:/cal_val aeri_armory cal_val_plots.py /cal_val/AEYYMMDD -v
268
[WARNING: 2020-09-10 18:45:36 : __main__] Records flagged for missing data: 119, 120. Adding them to excluded records
William Roberts's avatar
William Roberts committed
269
270
```

271
272
273
When plotting, if nans are found in the long or short waves, a warning will be printed with which wave numbers contain
nan values. If there is part of the graph with all nans, that file will not be written.

Marco Kurzynski's avatar
Marco Kurzynski committed
274
```console
Marco Kurzynski's avatar
Marco Kurzynski committed
275
$ docker run --rm -it -v <data_dir>:/cal_val aeri_armory cal_val_plots.py /cal_val/AEYYMMDD -r "118 120" -v
276
277
278
[WARNING: 2020-09-10 18:46:45 : __main__] Records flagged for missing data: 119, 120. Adding them to excluded records
[WARNING: 2020-09-10 18:46:46 : __main__] All nans in NBB observed longwave, records 118-118. Skipping plot generation
[WARNING: 2020-09-10 18:46:46 : __main__] nan values found in IBB observed longwave, records 118-118, in the following wave numbers: 1558.2998046875, 1700.0510864257812, 1733.3192443847656
279
280
```

William Roberts's avatar
William Roberts committed
281
The following commands have the same result:
282

Marco Kurzynski's avatar
Marco Kurzynski committed
283
```console
Marco Kurzynski's avatar
Marco Kurzynski committed
284
285
$ docker run --rm -it -v <data_dir>:/cal_val aeri_armory cal_val_plots.py /cal_val/AEYYMMDD -o /cal_val/output
$ docker run --rm -it -v <data_dir>:/cal_val -w /cal_val aeri_armory cal_val_plots.py AEYYMMDD -o /cal_val/output
William Roberts's avatar
William Roberts committed
286
287
```

William Roberts's avatar
William Roberts committed
288
Full use of flags:
289

Marco Kurzynski's avatar
Marco Kurzynski committed
290
```console
Marco Kurzynski's avatar
Marco Kurzynski committed
291
$ docker run --rm -it -v <data_dir>:/cal_val aeri_armory cal_val_plots.py /cal_val/AEYYMMDD -o /cal_val/output -r "1 10" -e "$(seq 2 2 10)" -snvv
292
293
294
295
296
297
298
[INFO: 2020-09-10 19:34:36 : __main__] start through end records: 1 - 10
[INFO: 2020-09-10 19:34:36 : __main__] excluded records: 2, 4, 6, 8, 10
[INFO: 2020-09-10 19:34:36 : __main__] Record 1 scene mirror position = 83
[INFO: 2020-09-10 19:34:38 : __main__] Record 3 scene mirror position = 83
[INFO: 2020-09-10 19:34:40 : __main__] Record 5 scene mirror position = 83
[INFO: 2020-09-10 19:34:42 : __main__] Record 7 scene mirror position = 83
[INFO: 2020-09-10 19:34:44 : __main__] Record 9 scene mirror position = 83
William Roberts's avatar
William Roberts committed
299
```
300

301
302
# Detector Nonlinearity Calculation

303
304
### Summary

305
306
Calculate the nonlinearity parameter for the channel 1 detector, for use in nlapp.sip.

307
308
### Input Requirements

309
310
311
312
313
314
- YYMMDDB1.CXS
- YYMMDDF1.CXS
- aeriemis.asc
- contrl4.in
- contrl5.in

315
### Output
316

William Roberts's avatar
William Roberts committed
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
- ARRAYYMMDD
    - b1
        - contrl-b1.in
        - NLAPP.PAR
        - plots
            - CXS
                - .png files
            - IFG
                - .png files
            - responsivity
                - .png files
    - f1
        - contrl-f1.in
        - NLAPP.PAR
        - plots
            - CXS
                - .png files
            - IFG
                - .png files
            - responsivity
                - .png files
    - NLAPP.SIP
339

340
### nli.py
William Roberts's avatar
William Roberts committed
341
342

```console
Marco Kurzynski's avatar
Marco Kurzynski committed
343
344
345
346
347
348
Usage: 
Nli.py

Purpose: This program computes the amplitude of non-linear terms contributing to instrument responsivity.
Updated for AERI-01

William Roberts's avatar
William Roberts committed
349
350
351
352
353
354
Usage: nli.py [options] [paramfile]

Based on Fortran NLI Version 2.7 from 27 Jan 97


Options:
Marco Kurzynski's avatar
Marco Kurzynski committed
355
356
357
358
359
360
361
362
363
364
  -h, --help      show this help message and exit
  -o OUTPUT_DIR   Specify output directory to use or be created. Defaults to
                  working directory
  -u              Use UW-AERI configuration
  -v              Enable verbose output
  -a              Average NLAPP.PAR values
  --plotall       Plot everything
  -p PLOT1        Plot a single data set [i(fg), c(xs), r(esponsivity)]
  --zpd1          Use erroneous ft2ifg settings from pre- r2949 ft2ifg.for.
  --scale=SCALE   Scale plot output from default values
William Roberts's avatar
William Roberts committed
365
366
```

William Roberts's avatar
William Roberts committed
367
### Shell Example
368

Marco Kurzynski's avatar
Marco Kurzynski committed
369
```console
370
$ docker run --rm -it -v <data_dir>:/nli aeri_armory nli.py -o /nli/output -a --plotall /nli/AEYYMMDD
371
```
Marco Kurzynski's avatar
Marco Kurzynski committed
372

373
374
# DMV to netCDF converter

375
376
### Summary

377
378
Convert AERI DMV data files into NetCDF format.

379
380
381
382
383
384
385
### Input Requirements

One of:

- .CXS
- .SUM
- .RNC
386

387
388
389
### Output

- converted .nc file
Marco Kurzynski's avatar
Marco Kurzynski committed
390

William Roberts's avatar
William Roberts committed
391
### Shell Example
392

Marco Kurzynski's avatar
Marco Kurzynski committed
393
```console
William Roberts's avatar
William Roberts committed
394
$ docker run --rm -it -v <data_dir>:/dmv_to_ncdf aeri_armory dmv_to_ncdf.py /dmv_to_ncdf/AEYYMMDD -o /dmv_to_ncdf/output
Marco Kurzynski's avatar
Marco Kurzynski committed
395
```
Marco Kurzynski's avatar
Marco Kurzynski committed
396
397
398

# Tests

399
### Test Data Location
Marco Kurzynski's avatar
typos    
Marco Kurzynski committed
400

401
Currently, test data is only located in the gitlab repository: `aeri_armory/tests`
Marco Kurzynski's avatar
typos    
Marco Kurzynski committed
402

403
### Summary
Marco Kurzynski's avatar
Marco Kurzynski committed
404

405
406
407
408
Run `test_suite.sh` on aeri_armory test data. This generates user output inside the test data directory which is then
compared to the expected output. Log files can also be found inside the test data directory containing runtime output
and/or error messages (if applicable).

Marco Kurzynski's avatar
Marco Kurzynski committed
409
Note that `test_suite.sh` uses PWD by default, so it must be run inside the test data directory unless another directory is passed as an argument like `test_suite.sh /tests`
410
411

**The test data directory must have this structure to generate the user_output folder in each tests sub-directory**
Marco Kurzynski's avatar
typos    
Marco Kurzynski committed
412

Marco Kurzynski's avatar
Marco Kurzynski committed
413
414
415
416
```
--- tests
  |--- aeri_qc
  |  |--- expected_output
Marco Kurzynski's avatar
Marco Kurzynski committed
417
  |  |--- input_data
Marco Kurzynski's avatar
Marco Kurzynski committed
418
419
420
  |
  |--- cal_val
  |  |--- expected_output
Marco Kurzynski's avatar
Marco Kurzynski committed
421
  |  |--- input_data
Marco Kurzynski's avatar
Marco Kurzynski committed
422
423
424
  |
  |--- nli
  |  |--- expected output
Marco Kurzynski's avatar
Marco Kurzynski committed
425
  |  |--- input_data
Marco Kurzynski's avatar
Marco Kurzynski committed
426
427
428
  |
  |--- dmv_to_ncdf
  |  |--- expected_output
Marco Kurzynski's avatar
Marco Kurzynski committed
429
  |  |--- input_data
Marco Kurzynski's avatar
Marco Kurzynski committed
430
```
Marco Kurzynski's avatar
Marco Kurzynski committed
431

432
You can skip tests by assigning the variables below with any non-empty values respectively:
Marco Kurzynski's avatar
Marco Kurzynski committed
433

434
435
436
437
- `SKIP_AERI_QC`
- `SKIP_CAL_VAL`
- `SKIP_NLI`
- `SKIP_DMV_TO_NCDF`
Marco Kurzynski's avatar
Marco Kurzynski committed
438

William Roberts's avatar
William Roberts committed
439
### Shell Example
Marco Kurzynski's avatar
Marco Kurzynski committed
440

Marco Kurzynski's avatar
Marco Kurzynski committed
441
```bash
442
$ docker run --rm -it -v <test_dir>:/tests aeri_armory test_suite.sh /tests
Marco Kurzynski's avatar
Marco Kurzynski committed
443
444
445
446
447
448
running aeri qc...
running quick vis...
running nc diff...
running REP.txt diff...
running png diff...
SUCCESS: aeri_qc
Marco Kurzynski's avatar
Marco Kurzynski committed
449
450
451
452
453
454
455
456
457
running cal_val...
running diff...
SUCCESS: cal_val
running nli...
running diff...
SUCCESS: nli
running dmv to ncdf...
running diff...
SUCCESS: dmv_to_ncdf
Marco Kurzynski's avatar
Marco Kurzynski committed
458
```
459

460
To skip aeri_qc and cal_val:
Marco Kurzynski's avatar
Marco Kurzynski committed
461
462

```bash
463
$ docker run --rm -e SKIP_AERI_QC=foo -e SKIP_CAL_VAL=bar -it -v <test_dir>:/tests aeri_armory test_suite.sh /tests
William Roberts's avatar
William Roberts committed
464
465
466
467
468
469
running nli...
running diff...
SUCCESS: nli
running dmv to ncdf...
running diff...
SUCCESS: dmv_to_ncdf
Marco Kurzynski's avatar
Marco Kurzynski committed
470
```