Skip to content
GitLab
Commit 3d6d22ec authored by William Roberts's avatar William Roberts
Browse files

Merge branch 'marco' into 'master' 

Utilities accept the same input using file_finder util

See merge request aeri/aeri_armory!11
parents fef8306c c4ddaf8f
Show whitespace changes
Inline Side-by-side
Dockerfile
View file @ 3d6d22ec
# mambaforge base image in centos
FROM centos:7 as centos-mambaforge
RUN touch /etc/sysconfig/64bit_strstr_via_64bit_strstr_sse2_unaligned
RUN yum -y update
......@@ -31,9 +30,9 @@ RUN chmod -R 700 /root/.ssh
# Get source code.
RUN git clone --depth 1 -b m64_r20200617 git@gitlab.ssec.wisc.edu:aeri/ifg_ret.git
RUN git clone --depth 1 -b m64_r20210721 git@gitlab.ssec.wisc.edu:aeri/ifg_tools.git
RUN git clone --depth 1 -b m64_r20210927 git@gitlab.ssec.wisc.edu:cphillips/aeri_quality_control.git
RUN git clone --depth 1 -b m64_r20211123 git@gitlab.ssec.wisc.edu:cphillips/aeri_quality_control.git
RUN git clone --depth 1 -b m64_r20200630 git@gitlab.ssec.wisc.edu:cphillips/aeri_tools.git
RUN git clone --depth 1 -b py2to3 git@gitlab.ssec.wisc.edu:aeri/ifg_aeri.git
RUN git clone --depth 1 -b m64_r20211123 git@gitlab.ssec.wisc.edu:aeri/ifg_aeri.git
# The following directories just need to be in the same directory.
RUN mv /ifg_aeri/py/nli /root/nli
......@@ -81,12 +80,15 @@ RUN /root/aeri_armory_env/bin/conda-unpack
# deployment image
FROM centos:7
MAINTAINER Denny Hackel (denny.hackel@ssec.wisc.edu)
ARG AERI_ARMORY_VERSION
RUN touch /etc/sysconfig/64bit_strstr_via_64bit_strstr_sse2_unaligned
# setup python env
ENV VIRTUAL_ENV="/aeri_armory_env"
ENV PATH="$VIRTUAL_ENV/bin:${PATH}"
ENV PYTHONWARNINGS=ignore
# Must be in vX.X format to get an image made.
ENV AERI_ARMORY_VERSION="${AERI_ARMORY_VERSION}"
RUN yum -y update
RUN yum -y install epel-release
......
README.md
View file @ 3d6d22ec
......@@ -78,6 +78,7 @@ environment for a container to run in.
- Each container is an isolated entity which is not tied to local directories by default.
- The flag `-v` is used to mount a local directory to a docker container.
- You can mount multiple local directories.
- If inside an interactive docker container, type "exit" and enter.
- You can pass environment variables using the `-e` flag.
- You can set the default working directory using the `-w` flag.
......@@ -92,7 +93,7 @@ environment for a container to run in.
# Quick Start
```console
$ docker run --rm -it -v <data_dir>:<destination> aeri_armory
$ docker run --rm -it -v <data_dir>:<destination> aeri_armory UTILITY_SCRIPT ...
```
- `<data_dir>`: Absolute path of directory on your local machine to read data from and write data to.
......@@ -102,7 +103,7 @@ If input and output are not located in the same directory, you can mount both, t
script you decide to run:
```console
$ docker run --rm -it -v <input_dir>:/input -v <output_dir>:/output aeri_armory
$ docker run --rm -it -v <input_dir>:/input -v <output_dir>:/output aeri_armory UTILITY_SCRIPT ...
```
You are not restricted to `/input` and `/output`, they're just intuitive:
......@@ -131,16 +132,16 @@ Run the AERI Quality Control algorithm on AERI data and visualize the results.
### run_aeri_qc.py
```console
usage: run_aeri_qc.py [-h] [--output OUTPUT] [--hint HINT] [-f] [-v] ftp [sci]
usage: run_aeri_qc.py [-h] [--output OUTPUT] [--hint HINT] [-f] [-v] input_dir [sci]
positional arguments:
ftp The directory where the AEYYMMDD ftp subdirectories with the .CXS and .SUM files are located
input_dir The AEYYMMDD directory or the directory containing AEYYMMDD subdirectories where the .CXS and .SUM files are located
sci
optional arguments:
-h, --help show this help message and exit
--output OUTPUT, -o OUTPUT
Directory where QC netcdf files are stored. Defaults to the same directory as ftp
Directory where QC netcdf files are stored. Defaults to the same directory as input_dir
--hint HINT inspect this file and others in similar time period more closely
-f, --force Overwrite any existing QC files in output directory
-v, --verbose Each occurrence increases verbosity 1 level through ERROR-WARNING-INFO-DEBUG.
......@@ -165,7 +166,7 @@ optional arguments:
### Shell Example
```console
$ docker run --rm -it -v <data_dir>:/aeri_qc aeri_armory run_aeri_qc.py /aeri_qc -o /aeri_qc/output -vv -f
$ docker run --rm -it -v <data_dir>:/aeri_qc aeri_armory run_aeri_qc.py /aeri_qc/AEYYMMDD -o /aeri_qc/output -vv -f
INFO:aeri_qc.main:Performing quality control for /aeri_qc/AEYYMMDD/YYMMDDB1.CXS
$ docker run --rm -it -v <data_dir>:/aeri_qc aeri_armory quick_vis.py /aeri_qc/output
```
......@@ -173,7 +174,7 @@ $ docker run --rm -it -v <data_dir>:/aeri_qc aeri_armory quick_vis.py /aeri_qc/o
Or equivalently
```console
$ docker run --rm -it -v <data_dir>:/aeri_qc aeri_armory run_aeri_qc.py bash -c 'run_aeri_qc.py /aeri_qc -o /aeri_qc/output -vv -f; quick_vis.py /aeri_qc/output'
$ docker run --rm -it -v <data_dir>:/aeri_qc aeri_armory run_aeri_qc.py bash -c 'run_aeri_qc.py /aeri_qc/AEYYMMDD -o /aeri_qc/output -vv -f; quick_vis.py /aeri_qc/output'
INFO:aeri_qc.main:Performing quality control for /aeri_qc/AEYYMMDD/YYMMDDB1.CXS
```
......@@ -217,20 +218,14 @@ Validate AERI calibration results.
### Input Requirements
- YYMMDDC1.RNC
- YYMMDDC2.RNC
- AEYYMMDD
- YYMMDDC1.RNC
- YYMMDDC2.RNC
### Output
- aeri_XXX_3rdbody_YYMMDD.png
- aeri_XXX_icebody_YYMMDD.png
- aeri_XXX_therm_temps_YYMMDD.png
- bbcal.mlt1
- bbcal.mlt2
- bbcal.sum1
- bbcal.sum2
- bbcal.lo1
- bbcal.lo2
- .png files containing plots of the calibration
- bbcal files containing raw data and debug information
### cal_val_plots.py
......@@ -258,7 +253,7 @@ optional arguments:
### Shell Examples
```console
$ docker run --rm -it -v <data_dir>:/cal_val aeri_armory cal_val_plots.py /cal_val/YYMMDDC1.RNC -o /cal_val/output -r "1 4" -vv
$ docker run --rm -it -v <data_dir>:/cal_val aeri_armory cal_val_plots.py /cal_val/AEYYMMDD -o /cal_val/output -r "1 4" -vv
[INFO: 2021-09-28 18:44:56 : __main__] start through end records: 1 - 4
[INFO: 2021-09-28 18:44:56 : __main__] excluded records:
[INFO: 2021-09-28 18:44:56 : __main__] Record 1 scene mirror position = 83
......@@ -269,7 +264,7 @@ Note that if bbcal runs into a value error with nan values, it will be skipped a
(use -v to show warning message). Output will still be generated, but may be invalid.
```console
$ docker run --rm -it -v <data_dir>:/cal_val aeri_armory cal_val_plots.py /cal_val/YYMMDDC1.RNC -v
$ docker run --rm -it -v <data_dir>:/cal_val aeri_armory cal_val_plots.py /cal_val/AEYYMMDD -v
[WARNING: 2020-09-10 18:45:36 : __main__] Records flagged for missing data: 119, 120. Adding them to excluded records
```
......@@ -277,7 +272,7 @@ When plotting, if nans are found in the long or short waves, a warning will be p
nan values. If there is part of the graph with all nans, that file will not be written.
```console
$ docker run --rm -it -v <data_dir>:/cal_val aeri_armory cal_val_plots.py /cal_val/YYMMDDC1.RNC -r "118 120" -v
$ docker run --rm -it -v <data_dir>:/cal_val aeri_armory cal_val_plots.py /cal_val/AEYYMMDD -r "118 120" -v
[WARNING: 2020-09-10 18:46:45 : __main__] Records flagged for missing data: 119, 120. Adding them to excluded records
[WARNING: 2020-09-10 18:46:46 : __main__] All nans in NBB observed longwave, records 118-118. Skipping plot generation
[WARNING: 2020-09-10 18:46:46 : __main__] nan values found in IBB observed longwave, records 118-118, in the following wave numbers: 1558.2998046875, 1700.0510864257812, 1733.3192443847656
......@@ -286,14 +281,14 @@ $ docker run --rm -it -v <data_dir>:/cal_val aeri_armory cal_val_plots.py /cal_v
The following commands have the same result:
```console
$ docker run --rm -it -v <data_dir>:/cal_val aeri_armory cal_val_plots.py /cal_val/YYMMDDC1.RNC -o /cal_val/output
$ docker run --rm -it -v <data_dir>:/cal_val aeri_armory bash -c 'cd /cal_val; cal_val_plots.py YYMMDDC1.RNC -o output'
$ docker run --rm -it -v <data_dir>:/cal_val aeri_armory cal_val_plots.py /cal_val/AEYYMMDD -o /cal_val/output
$ docker run --rm -it -v <data_dir>:/cal_val -w /cal_val aeri_armory cal_val_plots.py AEYYMMDD -o /cal_val/output
```
Full use of flags:
```console
$ docker run --rm -it -v <data_dir>:/cal_val aeri_armory cal_val_plots.py /cal_val/YYMMDDC1.RNC -o /cal_val/output -r "1 10" -e "$(seq 2 2 10)" -snvv
$ docker run --rm -it -v <data_dir>:/cal_val aeri_armory cal_val_plots.py /cal_val/AEYYMMDD -o /cal_val/output -r "1 10" -e "$(seq 2 2 10)" -snvv
[INFO: 2020-09-10 19:34:36 : __main__] start through end records: 1 - 10
[INFO: 2020-09-10 19:34:36 : __main__] excluded records: 2, 4, 6, 8, 10
[INFO: 2020-09-10 19:34:36 : __main__] Record 1 scene mirror position = 83
......@@ -311,11 +306,46 @@ Calculate the nonlinearity parameter for the channel 1 detector, for use in nlap
### Input Requirements
- YYMMDDB1.CXS
- YYMMDDF1.CXS
- aeriemis.asc
- contrl4.in
- contrl5.in
### Output
- ARRAYYMMDD
- b1
- contrl-b1.in
- NLAPP.PAR
- plots
- CXS
- .png files
- IFG
- .png files
- responsivity
- .png files
- f1
- contrl-f1.in
- NLAPP.PAR
- plots
- CXS
- .png files
- IFG
- .png files
- responsivity
- .png files
- NLAPP.SIP
### nli.py
```console
Usage:
Nli.py
Purpose: This program computes the amplitude of non-linear terms contributing to instrument responsivity.
Updated for AERI-01
Usage: nli.py [options] [paramfile]
Based on Fortran NLI Version 2.7 from 27 Jan 97
......@@ -323,6 +353,8 @@ Based on Fortran NLI Version 2.7 from 27 Jan 97
Options:
-h, --help show this help message and exit
-o OUTPUT_DIR Specify output directory to use or be created. Defaults to
working directory
-u Use UW-AERI configuration
-v Enable verbose output
-a Average NLAPP.PAR values
......@@ -335,7 +367,7 @@ Options:
### Shell Example
```console
$ docker run --rm -it -v <data_dir>:/nli aeri_armory bash -c 'cd /nli; nli.py -o /nli/output -a --plotall contrl4.in contrl5.in'
$ docker run --rm -it -v <data_dir>:/nli aeri_armory nli.py -o /nli/output -a --plotall /nli/AEYYMMDD
```
# DMV to netCDF converter
......@@ -359,36 +391,7 @@ One of:
### Shell Example
```console
$ docker run --rm -it -v <data_dir>:/dmv_to_ncdf aeri_armory \
bash -c 'cd /dmv_to_ncdf; dmv-ncdf; \
mkdir output; mv YYMMDDF1_cxs.nc YYMMDDF1_cxs.log output'
------------------------------------------------------
$Id: dmv-ncdf.for,v 2.25 2009-04-17 18:01:53 dennyh Exp $
------------------------------------------------------
Input file, in "DMV/RGD" format:
YYMMDDF1.CXS
First input record number:
1
Last input record number:
99999
Number of scene types (up to 10) to be extracted
(Enter "0" to extract ALL types!)
0
Output (NetCDF) file:
YYMMDDF1_cxs.nc
Comment file, or comments:
Comments:
Metadata configuration file
Log (text output) file
YYMMDDF1_cxs.log
Opening NEW log file YYMMDDF1_cxs.log
...
...
Number of records output: 223
$ docker run --rm -it -v <data_dir>:/dmv_to_ncdf aeri_armory dmv_to_ncdf.py /dmv_to_ncdf/AEYYMMDD -o /dmv_to_ncdf/output
```
# Tests
......@@ -403,8 +406,7 @@ Run `test_suite.sh` on aeri_armory test data. This generates user output inside
compared to the expected output. Log files can also be found inside the test data directory containing runtime output
and/or error messages (if applicable).
Note that `test_suite.sh` must be run inside the test data directory unless another directory is pointed to
by `TEST_DIR`.
Note that `test_suite.sh` uses PWD by default, so it must be run inside the test data directory unless another directory is passed as an argument like `test_suite.sh /tests`
**The test data directory must have this structure to generate the user_output folder in each tests sub-directory**
......@@ -437,7 +439,7 @@ You can skip tests by assigning the variables below with any non-empty values re
### Shell Example
```bash
$ docker run --rm -w /tests -it -v <test_dir>:/tests aeri_armory test_suite.sh
$ docker run --rm -it -v <test_dir>:/tests aeri_armory test_suite.sh /tests
running aeri qc...
running quick vis...
running nc diff...
......@@ -458,7 +460,7 @@ SUCCESS: dmv_to_ncdf
To skip aeri_qc and cal_val:
```bash
$ docker run --rm -w /tests -e SKIP_AERI_QC=foo -e SKIP_CAL_VAL=bar -it -v <test_dir>:/tests aeri_armory test_suite.sh
$ docker run --rm -e SKIP_AERI_QC=foo -e SKIP_CAL_VAL=bar -it -v <test_dir>:/tests aeri_armory test_suite.sh /tests
running nli...
running diff...
SUCCESS: nli
......
cal_val/__init__.py deleted 100644 → 0
View file @ fef8306c
import os
def set_env():
# Chan 1
os.environ.setdefault('C1_IDNBB', 'S')
os.environ.setdefault('C1_IDIBB', 'T')
os.environ.setdefault('C1_NBEMFL', 'nbb-emis.asc')
os.environ.setdefault('C1_IBEMFL', 'ibb-emis.asc')
os.environ.setdefault('C1_CAVFAC2', '39.0')
os.environ.setdefault('C1_WTNB', '0.811 0.189 0.0')
os.environ.setdefault('C1_TOSNB', '0.0 0.0 0.0')
os.environ.setdefault('C1_WTIB', '0.811 0.0945 0.0945')
os.environ.setdefault('C1_TOSIB', '0.0 0.0 0.0')
os.environ.setdefault('C1_NRNGS', '1')
# Chan 1 redeclared for Chan 2
os.environ.setdefault('C2_IDNBB', 'S')
os.environ.setdefault('C2_IDIBB', 'T')
os.environ.setdefault('C2_NBEMFL', 'nbb-emis.asc')
os.environ.setdefault('C2_IBEMFL', 'ibb-emis.asc')
os.environ.setdefault('C2_CAVFAC2', '39.0')
os.environ.setdefault('C2_WTNB', '0.811 0.189 0.0')
os.environ.setdefault('C2_TOSNB', '0.0 0.0 0.0')
os.environ.setdefault('C2_WTIB', '0.811 0.0945 0.0945')
os.environ.setdefault('C2_TOSIB', '0.0 0.0 0.0')
os.environ.setdefault('C2_NRNGS', '1')
# Unique Chan 1
os.environ.setdefault('C1_LOGFIL', 'bbcal3p.lo1')
os.environ.setdefault('C1_MLTFILE', 'bbcal.mlt1')
os.environ.setdefault('C1_SUMFILE', 'bbcal.sum1')
os.environ.setdefault('C1_WN', '900 1100')
# Unique Chan 2
os.environ.setdefault('C2_LOGFIL', 'bbcal3p.lo2')
os.environ.setdefault('C2_MLTFILE', 'bbcal.mlt2')
os.environ.setdefault('C2_SUMFILE', 'bbcal.sum2')
os.environ.setdefault('C2_WN', '2100 2200')
set_env()
cal_val/cal_val/__init__.py
View file @ 3d6d22ec
import os
def set_env():
# Chan 1
os.environ.setdefault('C1_IDNBB', 'S')
os.environ.setdefault('C1_IDIBB', 'T')
os.environ.setdefault('C1_NBEMFL', 'nbb-emis.asc')
os.environ.setdefault('C1_IBEMFL', 'ibb-emis.asc')
os.environ.setdefault('C1_CAVFAC2', '39.0')
os.environ.setdefault('C1_WTNB', '0.811 0.189 0.0')
os.environ.setdefault('C1_TOSNB', '0.0 0.0 0.0')
os.environ.setdefault('C1_WTIB', '0.811 0.0945 0.0945')
os.environ.setdefault('C1_TOSIB', '0.0 0.0 0.0')
os.environ.setdefault('C1_NRNGS', '1')
# Chan 1 redeclared for Chan 2
os.environ.setdefault('C2_IDNBB', 'S')
os.environ.setdefault('C2_IDIBB', 'T')
os.environ.setdefault('C2_NBEMFL', 'nbb-emis.asc')
os.environ.setdefault('C2_IBEMFL', 'ibb-emis.asc')
os.environ.setdefault('C2_CAVFAC2', '39.0')
os.environ.setdefault('C2_WTNB', '0.811 0.189 0.0')
os.environ.setdefault('C2_TOSNB', '0.0 0.0 0.0')
os.environ.setdefault('C2_WTIB', '0.811 0.0945 0.0945')
os.environ.setdefault('C2_TOSIB', '0.0 0.0 0.0')
os.environ.setdefault('C2_NRNGS', '1')
os.environ.setdefault('IDNBB', 'S')
os.environ.setdefault('IDIBB', 'T')
os.environ.setdefault('NBEMFL', 'nbb-emis.asc')
os.environ.setdefault('IBEMFL', 'ibb-emis.asc')
os.environ.setdefault('CAVFAC2', '39.0')
os.environ.setdefault('WTNB', '0.811 0.189 0.0')
os.environ.setdefault('TOSNB', '0.0 0.0 0.0')
os.environ.setdefault('WTIB', '0.811 0.0945 0.0945')
os.environ.setdefault('TOSIB', '0.0 0.0 0.0')
os.environ.setdefault('NRNGS', '1')
os.environ.setdefault('IBBTEMP', '273.182')
# Unique Chan 1
os.environ.setdefault('C1_LOGFIL', 'bbcal3p.lo1')
os.environ.setdefault('C1_MLTFILE', 'bbcal.mlt1')
os.environ.setdefault('C1_SUMFILE', 'bbcal.sum1')
os.environ.setdefault('C1_WN', '900 1100')
# Unique Chan 2
os.environ.setdefault('C2_LOGFIL', 'bbcal3p.lo2')
os.environ.setdefault('C2_MLTFILE', 'bbcal.mlt2')
os.environ.setdefault('C2_SUMFILE', 'bbcal.sum2')
os.environ.setdefault('C2_WN', '2100 2200')
set_env()
cal_val/cal_val/bbcal.py 100755 → 100644
View file @ 3d6d22ec
......@@ -48,10 +48,18 @@ def interpolate(xin, yin, xout):
return val
def process(c1_file, c2_file, records, NBBtemp=None, IBBtemp=None, abb_breakout=None, output_path=''):
return process_helper(1, c1_file, records, NBBtemp, IBBtemp, abb_breakout, output_path), process_helper(2, c2_file, records, NBBtemp, IBBtemp, abb_breakout, output_path)
def process(directory, records, abb_breakout=None, output_path=''):
try:
NBBtemp = float(str.strip(os.environ.get('NBBTEMP')))
except TypeError:
NBBtemp = None
try:
IBBtemp = float(str.strip(os.environ.get('IBBTEMP')))
except TypeError:
IBBtemp = None
return process_helper(1, directory, records, NBBtemp, IBBtemp, abb_breakout, output_path), process_helper(2, directory, records, NBBtemp, IBBtemp, abb_breakout, output_path)
def process_helper(channel_number, channel_file, records, NBBtemp=None, IBBtemp=None, abb_breakout=None, output_path=''):
def process_helper(channel_number, directory, records, NBBtemp=None, IBBtemp=None, abb_breakout=None, output_path=''):
if isinstance(records, numbers.Number):
records = [records]
......@@ -61,40 +69,37 @@ def process_helper(channel_number, channel_file, records, NBBtemp=None, IBBtemp
# Maximum number of ranges
MAXRNGS = 10
if channel_number == 1:
prefix = 'C1_'
elif channel_number == 2:
prefix = 'C2_'
else:
prefix = ''
if channel_number != 1 and channel_number != 2:
logger.error("Invalid channel_number, must be 1 or 2")
exit(1)
# line[5] Log file
LOGFIL = os.path.join(output_path, str.strip(os.environ.get(prefix+'LOGFIL')))
try:
LOGFILE = str.strip(os.environ.get(f'C{channel_number}_LOGFILE'))
except TypeError:
LOGFILE = f'bbcal3p_{os.path.basename(directory)[2:]}.lo{channel_number}'
LOGFILE = os.path.join(output_path, LOGFILE)
# Open logfile
lf = open(LOGFIL, 'w')
lf = open(LOGFILE, 'w')
lf.write(" ---------------- bbcal3p.py -----------------------\n")
lf.write(" -----------------------------------------------------\n")
# Input File
INFILE = channel_file
INFILE = os.path.join(directory, f"{os.path.basename(directory)[2:8]}C{channel_number}.RNC")
lf.write("INPUT FILE: %s\n" % INFILE)
lf.write("INPUT RECORD RANGE: {0}\n".format(records))
# NBB & IBB Scene IDs
IDNBB = str.strip(os.environ.get(prefix+'IDNBB'))
IDNBB = str.strip(os.environ.get('IDNBB'))
lf.write("NBB identifier: %s\n" % IDNBB)
IDIBB = str.strip(os.environ.get(prefix+'IDIBB'))
IDIBB = str.strip(os.environ.get('IDIBB'))
lf.write("IBB identifier: %s\n" % IDIBB)
# NBB & IBB Emissivity Files
NBEMFL = str.strip(os.environ.get(prefix+'NBEMFL'))
NBEMFL = str.strip(os.environ.get('NBEMFL'))
lf.write("NBB EMISSIVITY FILE: %s\n" % NBEMFL)
IBEMFL = str.strip(os.environ.get(prefix+'IBEMFL'))
IBEMFL = str.strip(os.environ.get('IBEMFL'))
lf.write("IBB EMISSIVITY FILE: %s\n" % IBEMFL)
# Read NBB & IBB Emissivity Files
......@@ -125,50 +130,58 @@ def process_helper(channel_number, channel_file, records, NBBtemp=None, IBBtemp
imfp.close()
# BB Cavity Factor
CAVFAC2 = float(str.strip(os.environ.get(prefix+'CAVFAC2')))
CAVFAC2 = float(str.strip(os.environ.get('CAVFAC2')))
lf.write("BB CAVITY FACTOR: %f\n" % CAVFAC2)
RCF = 1.0 / CAVFAC2
# NBB Apex, Bottom and Top weightings
WTNBA = float(str.split(str.strip(os.environ.get(prefix+'WTNB')))[0])
WTNBB = float(str.split(str.strip(os.environ.get(prefix+'WTNB')))[1])
WTNBT = float(str.split(str.strip(os.environ.get(prefix+'WTNB')))[2])
WTNBA = float(str.split(str.strip(os.environ.get('WTNB')))[0])
WTNBB = float(str.split(str.strip(os.environ.get('WTNB')))[1])
WTNBT = float(str.split(str.strip(os.environ.get('WTNB')))[2])
# Write Weighting to Logfile
lf.write("NBB Weights: %1.5f %1.5f %1.5f\n" % (WTNBA, WTNBB, WTNBT))
# NBB Apex, Bottom and Top offsets
TOSNBA = float(str.split(str.strip(os.environ.get(prefix+'TOSNB')))[0])
TOSNBB = float(str.split(str.strip(os.environ.get(prefix+'TOSNB')))[1])
TOSNBT = float(str.split(str.strip(os.environ.get(prefix+'TOSNB')))[2])
TOSNBA = float(str.split(str.strip(os.environ.get('TOSNB')))[0])
TOSNBB = float(str.split(str.strip(os.environ.get('TOSNB')))[1])
TOSNBT = float(str.split(str.strip(os.environ.get('TOSNB')))[2])
# Write Offsets to Logfile
lf.write("NBB Offsets: %1.5f %1.5f %1.5f\n" % (TOSNBA, TOSNBB, TOSNBT))
# IBB Apex, Bottom and Top weightings
WTIBA = float(str.split(str.strip(os.environ.get(prefix+'WTIB')))[0])
WTIBB = float(str.split(str.strip(os.environ.get(prefix+'WTIB')))[1])
WTIBT = float(str.split(str.strip(os.environ.get(prefix+'WTIB')))[2])
WTIBA = float(str.split(str.strip(os.environ.get('WTIB')))[0])
WTIBB = float(str.split(str.strip(os.environ.get('WTIB')))[1])
WTIBT = float(str.split(str.strip(os.environ.get('WTIB')))[2])
# Write Weighting to Logfile
lf.write("IBB Weights: %1.5f %1.5f %1.5f\n" % (WTIBA, WTIBB, WTIBT))
# IBB Apex, Bottom and Top offsets
TOSIBA = float(str.split(str.strip(os.environ.get(prefix+'TOSIB')))[0])
TOSIBB = float(str.split(str.strip(os.environ.get(prefix+'TOSIB')))[1])
TOSIBT = float(str.split(str.strip(os.environ.get(prefix+'TOSIB')))[2])
TOSIBA = float(str.split(str.strip(os.environ.get('TOSIB')))[0])
TOSIBB = float(str.split(str.strip(os.environ.get('TOSIB')))[1])
TOSIBT = float(str.split(str.strip(os.environ.get('TOSIB')))[2])
# Write Offsets to Logfile
lf.write("IBB Offsets: %1.5f %1.5f %1.5f\n" % (TOSIBA, TOSIBB, TOSIBT))
# MLT file
MLTFILE = os.path.join(output_path, str.strip(os.environ.get(prefix+'MLTFILE')))
try:
MLTFILE = str.strip(os.environ.get(f'C{channel_number}_MLTFILE'))
except TypeError:
MLTFILE = f'bbcal_{os.path.basename(directory)[2:]}.mlt{channel_number}'
MLTFILE = os.path.join(output_path, MLTFILE)
# Write MLT file to Log file
lf.write("ASCII Output (MLT) file: % s\n" % MLTFILE)
# Summary File
SUMFILE = os.path.join(output_path, str.strip(os.environ.get(prefix+'SUMFILE')))
try:
SUMFILE = str.strip(os.environ.get(f'C{channel_number}_SUMFILE'))
except TypeError:
SUMFILE = f'bbcal_{os.path.basename(directory)[2:]}.sum{channel_number}'
SUMFILE = os.path.join(output_path, SUMFILE)
lf.write("TEXT OUTPUT (SUMMARY) file: %s\n" % SUMFILE)
# Array of Metadata required
......@@ -449,17 +462,17 @@ def process_helper(channel_number, channel_file, records, NBBtemp=None, IBBtemp
mlt.close()
# Read number of ranges
NRNGS = int(str.strip(os.environ.get(prefix+'NRNGS')))
NRNGS = int(str.strip(os.environ.get('NRNGS')))
# FIXME: Do something in the future if there is more then 1 range
if NRNGS >= 2:
logger.error("Add more code :)")
WN = str.strip(os.environ.get(f'C{channel_number}_WN'))
# Lower bound of spectral range
WNLO = int(str.split(str.strip(os.environ.get(prefix+'WN')))[0])
WNLO = int(str.split(WN)[0])
# Higher bound of spectral range
WNHI = int(str.split(str.strip(os.environ.get(prefix+'WN')))[1])
WNHI = int(str.split(WN)[1])
XMIN = interx[0]
XMAX = interx[len(interx) - 1]
......@@ -494,7 +507,7 @@ def process_helper(channel_number, channel_file, records, NBBtemp=None, IBBtemp
jhms = format_time(TIME2)
# jhms=ihms
dateYYMMDD = channel_file[-12:-6]
dateYYMMDD = os.path.basename(directory)[2:8]
with open(SUMFILE, 'w') as sf:
sf.write("%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%\n\n\n")
sf.write("Mean and Standard deviation of Observed minus Predicted BT\n\n\n")
......
cal_val/cal_val/cal_val_plots.py
View file @ 3d6d22ec
#!/usr/bin/env python
#!/usr/bin/env python3.8
from cal_val import AERI_lib
from cal_val import bbcal
import matplotlib
matplotlib.use('Agg')
import faulthandler
import matplotlib.pylab as plt
import numpy as np
import re
import argparse
import os, logging, math, numbers
from textwrap import wrap
from datetime import datetime
from utils import file_finder
faulthandler.enable()
matplotlib_logger = logging.getLogger('matplotlib')
# set WARNING for Matplotlib.
......@@ -19,12 +21,6 @@ matplotlib_logger.setLevel(logging.WARNING)
logger = logging.getLogger(__name__)
# Metadata override or another bbcal.in1 & bbcal.in2 files
# If the data is in another file set to 0 & create a
# bbcal.in3 & bbcal.in4 files
# NBBtemp=317.925
IBBtemp = 273.182
def to_int(arg):
try:
......@@ -508,72 +504,32 @@ def make_IBB_plot(c1_file, lw, sw, records, output_path):
plt.close('all')
def main():
#
# Values to change with different data sets in box below
#
######################################################################
# Argument Parsing
parser = argparse.ArgumentParser(description='Generate cal val plots for IBB and NBB. The first record is record 1')
parser.add_argument("input", help="Channel 1 input filename in HHMMSSC1.RNC format. Realtive or absolute path")
parser.add_argument("-o", '--output-dir',
help="Output file directory. It will be created if it does not exist. Defaults to current directory. Relative or absolute path")
parser.add_argument("-r", '--record-range',
help="Record range. Either a number or a string of two numbers separated by a space. Defaults to all records")
parser.add_argument('-e', '--excluded-records',
help='Records to exclude. Either a number or a string of numbers separated by spaces')
parser.add_argument('-s', '--separate', action='store_true',
help='Plot each record individually. Also generates IBB.avg and NBB.avg files')
parser.add_argument('-n', '--no-spectral', action='store_true',
help='Skip generation of spectral plots and only plot thermistor temperatures and diffmeans.')
parser.add_argument('-v', '--verbose', action="count", default=0,
help='Each occurrence increases verbosity 1 level through ERROR-WARNING-INFO-DEBUG.')