Fusion Matlab README

Contents

¶ ↑

• Introduction

• Building

  - Build requirements
  - Build recommendations

• Installation

  - Runtime requirements
  - Runtime recommendations
  - Environment setup

• Running

  - Required inputs
  - Exit status codes
  - Expected outputs
  - Example

• Troubleshooting

• FAQ

• Maintainers

Introduction

¶ ↑

• Software Description

This software package provides codes and ancillary data necessary to produce synthetic infrared (IR) bands by fusing imager and sounder radiance and geolocation data. While a sensor like MODIS has multiple IR absorption bands, the current VIIRS imager has only IR window bands. The software provides synthetic MODIS-like bands from VIIRS and CrIS data. With this approach applications that require IR absorption bands (e.g. retrieval of cloud properties) can be continued over generations of weather satellite sensors. The data fusion method consists of two main steps. The first step is performing a nearest neighbor search using a k-d tree algorithm on both high-spatial and low-spatial (i.e., geographic avaraged) resolution imager split-window 11 and 12 micron radiances. In the second step, the new so-called fusion radiances (at original high spatial resolution) are obtained by computing the mean of convolved sounder radiances for the filed-of-views (FOVs) selected in step 1. Details of the data fusion process can be found in Weisz, E., B. A. Baum, and W. P. Menzel, 2017: Fusion of Satellite-Based Imager and Sounder Data to Construct Supplementary High Spatial Resolution Narrowband IR Radiances. In press, J. Appl. Remote Sens.

• Package Description

This package has the following directory structure:

fusion_matlab

├── dist
│   ├── bin
│   │   ├── imagersounderfusion_M
│   │   ├── imagersounderfusion_V
│   │   ├── run_imagersounderfusion_M.sh
│   │   └── run_imagersounderfusion_V.sh
│   ├── env
│   │   ├── bin
│   │   ├── conda-meta
│   │   ├── examples
│   │   ├── include
│   │   ├── lib
│   │   ├── share
│   │   └── ssl
│   └── luts
│       ├── NG_VIIRS_NPP_RSR_filtered_Oct2011_BA
│       ├── L2.chan_prop.2005.03.01.v9.5.1.txt
│       ├── modis_aqua.srf.nc
│       └── modis_conv_error_2005.mat
├── source
│   ├── fusion_glue
│   ├── modis-airs-fusion
│   ├── polar2grid
│   └── viirs-cris-fusion
├── test
│   ├── c_sh
│   │   └── run_fusion_1gran.sh
│   ├── inputs
│   │   ├── AIRS.2015.04.17.145.L1B.AIRS_Rad.v5.0.23.0.G15173170714.hdf
│   │   ├── AIRS.2015.04.17.146.L1B.AIRS_Rad.v5.0.23.0.G15173170305.hdf
│   │   ├── colloc.airs_aqua.modis_aqua.20150417T142930_143500.nc
│   │   ├── colloc.airs_aqua.modis_aqua.20150417T143530_143500.nc
│   │   ├── colloc.cris_snpp.viirs_m_snpp.20180202T183600_183600.nc
│   │   ├── MYD021KM.A2015107.1435.006.2015108162652.hdf
│   │   ├── MYD03.A2015107.1435.006.2015108161923.hdf
│   │   ├── SNDR.SNPP.CRIS.20180202T1836.m06.g187.L1B.std.v2_0_15.T.180203004403.nc
│   │   ├── VNP02MOD.A2018033.1836.001.2018033235822.uwssec.nc
│   │   └── VNP03MOD.A2018033.1836.001.2018033235737.uwssec.nc
│   ├── outputs
│   │   ├── MYD02FSN.A2015107.1435.006.2018053182304.nc
│   │   └── VNP02FSN.A2018033.1836.001.2018054223818.nc
│   ├── py
│   │   ├── log_common.py
│   │   ├── run_fusion_1gran.py
│   │   └── utils.py
│   └── run.sh
├── ANCILLARY.txt
├── HISTORY.txt
├── README.txt
└── VERSION.txt

The source/ directory contains the git repos modis-airs-fusion and viirs-cris-fusion, which are matlab source codes for the MODIS/AIRS and VIIRS/CRIS fusion processes respectively. The fusion_glue and polar2grid git repos contain the source code for the Matlab-to-HDF4 and Matlab-to-NetCDF4 conversions.

The dist/bin directory contains the compiled Matlab executables (and supporting scripts), while dist/luts contain the static LUTS required by the Matlab executables:

modis_aqua.srf.nc                   : Aqua MODIS Spectral Response Functions (SRFs)
modis_conv_error_2005.mat           : MAT-file of convolution corrections to be used when convolved AIRS
                                      data to MODIS bands.
                                      This correction accounts for the gaps within the AIRS spectrum.
                                      Provided by Dave Tobin (SSEC).
L2.chan_prop.2005.03.01.v9.5.1.txt  : AIRS channel property list (lists channel frequencies, bad_flags etc)
NG_VIIRS_NPP_RSR_filtered_Oct2011_BA: This is a directory containing the VIIRS Spectral Response Functions.
                                      Here the NPP_VIIRS_NG_M*_filtered_Oct2011f_BA.dat-files are used.

The dist/env contains a complete python environment required to run the python elements of the fusion_matlab package.

The test/ directory contains a set of sample AIRS/MODIS and CrIS VIIRS inputs, the associated outputs generated by the fusion_matlab package, and the script run.sh, which runs the package on the sample inputs.

To invoke the matlab executables (e.g., within a bash script) use :

./run_imagersounderfusion_M.sh  input1 input2 input3 inpu4 input5 input6

where
    input1 : The directory containing the Matlab runtime (i.e.: /opt/matlab/2015b/)
    input2 : The MODIS or VIIRS geolocation file
    input3 : The MODIS or VIIRS level-1b file
    input4 : The AIRS or CrIS sounder level-1b files
    input5 : The AIRS/MODIS or CrIS/VIIRS collocation files (must be same number of these files as sounder level-1b files)
    input6 : The directory containing the luts required for the Matlab executables.

Building and Installation

¶ ↑

• Matlab Code

The required Matlab codes are:

- MODIS/AIRS Fusion (https://gitlab.ssec.wisc.edu/eweisz/modis-airs-fusion.git)
    * Commit: 869bbcd39660a1c31549691eac298746fad53ea7
- VIIRS/CrIS Fusion (https://gitlab.ssec.wisc.edu/eweisz/viirs-cris-fusion.git)
    * Commit: 8e2c7887016d31fe453b4c945618014557f4c44b

To be able to run the Matlab executables without a Matlab license, the source codes need to be compiled using the “mcc” command to create the matlab binaries (executables) ‘imagersounderfusion_M’ and ‘imagersounderfusion_V’ as well as scripts ‘run_imagersounderfusion_M.sh’ and ‘run_imagersounderfusion_M.sh’. A Matlab license is required to build the Matlab binaries.

• Python Environment and CF-compliance converter

To create the python environment contained in dist/env, we create the python environment “python-matlab” (assuming that a version of the Anaconda python distro is installed) by:

conda create --copy -n fusion-matlab h5py
source activate fusion-matlab
sips=http://sips.ssec.wisc.edu/conda
conda install --copy h5py netcdf4 scipy pyproj
conda install --copy -c $sips python-hdf4
conda install --copy -c $sips viirsmend

Remaining in the “python-matlab” python environment, we now install two Polar2Grid modules, polar2grid_core and polar2grid_modis:

* Polar2Grid (https://github.com/davidh-ssec/polar2grid.git),
    - Commit: 6b7816ae62db43e74fcf823bd51ada4af73509ec
    - polar2grid/py/polar2grid_core/setup.py
        - requires: numpy
    - polar2grid/py/polar2grid_modis/setup.py
        - requires: numpy, python-hdf4, polar2grid.core

We begin by checking out the correct revision of Polar2Grid:

cd fusion_matlab/source/polar2grid
git checkout 6b7816ae62db43e74fcf823bd51ada4af73509ec

We next install the polar2grid_core module:

pip install py/polar2grid_core

The polar2grid_modis module requires modification of its setup.py script to use the correct version of pyhdf, so we apply the following diff:

> git diff py/polar2grid_modis/setup.py
diff --git a/py/polar2grid_modis/setup.py b/py/polar2grid_modis/setup.py
index 67dcf56..c3eef00 100644
--- a/py/polar2grid_modis/setup.py
+++ b/py/polar2grid_modis/setup.py
@@ -82,7 +82,7 @@ setup(
   namespace_packages=["polar2grid"],
   include_package_data=True,
   zip_safe=True,
-    install_requires=['numpy', 'pyhdf', 'polar2grid.core'],
+    install_requires=['numpy', 'python-hdf4', 'polar2grid.core'],
   entry_points={
       'console_scripts': [],
       'polar2grid.frontend_class': ['modis=polar2grid.modis:Frontend'],

and install the polar2grid_modis module:

pip install py/polar2grid_modis

Lastly, we install the fusion_glue module:

* Fusion Glue (https://gitlab.ssec.wisc.edu/sounder-imager-fusion/fusion_glue.git)
    - Commit: c36966d00bc86c39c3744ed8b16971aa3ce775e3
    - requires: netCDF4, python-hdf4, numpy, scipy, polar2grid.core, polar2grid.modis
    - provides: l1b-fusion-modis-airs, l1b-fusion-viirs-cris

cd fusion_matlab/source/fusion_glue
git checkout develop
pip install py/fusion_l1b

The required python environment should now be present in your conda environment location ($HOME/.conda/envs/fusion-matlab for example). We finish by copying the python environment into our package:

cp -R $HOME/.conda/envs/fusion-matlab fusion_matlab/dist/env/

Requirements

---

Requirements to build the software:

• Continuum Analytics Anaconda for python 2.7 (www.continuum.io/downloads#linux)

• Matlab (2015b), and associated license

Recommendations

---

None

Environment Setup

---

Describe the setup of the environment required to run the package, including:

• Environment Variables:

  LD_LIBRARY_PATH=fusion_matlab/dist/env/lib
  PATH=fusion_matlab/dist/env/bin:/usr/bin:/bin

• Configuration files: None

Troubleshooting

¶ ↑

• Describe any common issues and their potential solutions.

• Where to look for relevant output or logs

• etc …

FAQ

¶ ↑

Frequently asked questions

Maintainers

¶ ↑

Elisabeth Weisz <elisabeth.weisz@ssec.wisc.edu>

- Matlab code

Ray Keoni Garcia <rayg@ssec.wisc.edu>

- CF-Compliance converter

Geoff Cureton <geoff.cureton@ssec.wisc.edu>

- fusion_matlab packaging